Categories
Discussions

"dontCreate" option for "listItemSplitter" problem

brunettonbrunetton
edited December 2016 in General Support


Hi all,


I've a doubdt about the meaning of the "dontCreate" option for "listItemSplitter". I thought it prevents for creating list items when they do not exists; as there is a global existingRecordPolicy in imports options section in import mapping file that controls the whole row creation.

I'm right ?

If yes, there's a problem with the use of that param: I used it in Refinery parameters like this:

{
    "list": "test_vocab",
    "dontCreate": "1"
}

The "test_vocab" list exists in the profile I choosed while installation (v1.6.3). It only containts one element: "Foo".

Here's my "data.csv", since csv file type upload isn't allowed on wiki:

id, voca
10, Foo
11, FooBar


After import, "FooBar" was added to the "test_vocab" list. This is not what I excepted since I passed the dontCreate option to listItemSplitter.
Also, the two objects were created (this make me think that this option doesn't control objects creation)

Thanks for reading !



xls
xls
import_mapping.xls
12K
Share on Twitter

Comments

  • JuliaJulia
    The creation of objects is governed entirely by existingRecordPolicy (which you have set to none, so it's correct that the objects were created). dontCreate should only effect the listItem creation. Did you include other parameters, such as relationshipType or anything else in the listItemSplitter?

    Share on Twitter
  • brunettonbrunetton
    Hi Julia, thanks for your answer.

    > The creation of objects is governed entirely by existingRecordPolicy (which you have set to none, so it's correct that the objects were created). 
    Yes I agree with that, I want my object to be created, no problem so far.

    > dontCreate should only effect the listItem creation. 
    Indeed, I understood it like that, I'll ask for a wiki login and make the sentence a little more clear maybe.

    > Did you include other parameters, such as relationshipType or anything else in the listItemSplitter?
    All parameters are here in the attached file. I didn't add anything. Is relationshipType needed in that case ? I don't really understand why.
    But I noticed a line about that in log:
    2016-12-22 12:33:09 - WARN --> [listItemSplitter] No listItem type is set for listItem FooBar
    As import seems ok then, I didn't payed much attention to that, and anyway I don't really know what to give to this param.

    I just re-re-tested to be sure; and dive onto DB to check what I can see in user interface. Here's the before and after import:

    BEFORE

    Elements in test_vocab list:

    mysql root@localhost:providence_nu> SELECT idno FROM ca_list_items WHERE list_id = 50 AND deleted = 0;
    +--------------------------+
    | idno                     |
    |--------------------------|
    | Root node for test_vocab |
    | Foo                      |
    +--------------------------+
    2 rows in set

    This is confirmed in interface: only "Foo" is listed.

    AFTER

    Elements in test_vocab list:

    mysql root@localhost:providence_nu> SELECT idno FROM ca_list_items WHERE list_id = 50 AND deleted = 0;
    +--------------------------+
    | idno                     |
    |--------------------------|
    | Root node for test_vocab |
    | Foo                      |
    | FooBar                   |
    +--------------------------+
    3 rows in set

    This is confirmed in interface: "Foo" and "FooBar" are listed.

    The list item creation is also confirmed by log (sorry, in French here):

    2016-12-22 12:33:09 - INFO --> Nouvel item de liste FooBar/FooBar/FooBar créé dans la liste 50
    2016-12-22 12:33:09 - DEBUG --> Création de l'arbre de contenu pour 25 terminé en    0.0594 secondes [3]

    So I really think there's a bug here: dontCreate doesn't seems to be taken in account.

    What do you think about it ?

    Thanks again 
    Share on Twitter
  • sethseth
    The warning for [listItemSplitter] No listItem type is set for listItem FooBar warning is there because a type for the new list item is not  explicitly set. Unlike most other record types list items don't require a type. It can be null. But we still issue the warning across the board. You can ignore it, and maybe we should stop issuing it in all cases.

    "dontCreate" should indeed prevent new list items from being created by the splitter, which it seems like it's not accomplishing in your case. One thing to try is run the import with log level set to "debug". "_dontCreate" should appear in the content tree with the list item values to which it is bound. 

    If you could share your mapping worksheet I'd like to look at it.

    Thanks,
    Seth

    Share on Twitter
  • brunettonbrunetton
    Thanks Seth,
    I shared the mapping in my first post ! Can't you see it ?

    > The warning for [listItemSplitter] No listItem type is set for listItem FooBar warning is there because a type for the new list item is not  explicitly set. Unlike most other record types list items don't require a type. It can be null. But we still issue the warning across the board. You can ignore it, and maybe we should stop issuing it in all cases.
    Exactly what I deduced :)

    > "dontCreate" should indeed prevent new list items from being created by the splitter, which it seems like it's not accomplishing in your case. One thing to try is run the import with log level set to "debug". "_dontCreate" should appear in the content tree with the list item values to which it is bound. 
    I always run with DEBUG loglevel. Here's the complete import log:

    2016-12-22 10:58:52 - INFO --> Started import of /home/bruno/dev/ee/collectiveaccess/import/problèmes/1/1.csv using mapping test_import
    2016-12-22 10:58:52 - DEBUG --> Finished reading input source at    0.0297 seconds
    2016-12-22 10:58:52 - DEBUG --> Found 3 rows in input source
    2016-12-22 10:58:52 - DEBUG --> Finished analyzing mapping at    0.0358 seconds
    2016-12-22 10:58:52 - DEBUG --> Skipped initial row 0 of 1
    2016-12-22 10:58:52 - DEBUG --> Started reading row 2 at    0.0000 seconds
    2016-12-22 10:58:52 - WARN --> [listItemSplitter] No listItem type is set for listItem Foo
    2016-12-22 10:58:52 - DEBUG --> Found existing list item Foo (member of list 50) in DataMigrationUtils::getListItemID() using singular label Foo
    2016-12-22 10:58:52 - DEBUG --> Finished building content tree for 10 at    0.1726 seconds [2]
    2016-12-22 10:58:52 - DEBUG --> Content tree is
    Array
    (
        [ca_objects] => Array
            (
                [0] => Array
                    (
                        [debug_test_voca] => Array
                            (
                                [debug_test_voca] => 209
                                [_errorPolicy] => ignore
                            )
                    )
            )
    )

    2016-12-22 10:58:52 - DEBUG --> Created idno 10 at    0.2550 seconds
    2016-12-22 10:58:52 - DEBUG --> Finished inserting content tree for 10 at    0.3098 seconds into database
    2016-12-22 10:58:52 - INFO --> [10] Imported  as ca_objects 
    2016-12-22 10:58:52 - DEBUG --> Started reading row 3 at    0.0000 seconds
    2016-12-22 10:58:52 - WARN --> [listItemSplitter] No listItem type is set for listItem FooBar
    2016-12-22 10:58:52 - INFO --> Created new list item FooBar/FooBar/FooBar in list 50
    2016-12-22 10:58:52 - DEBUG --> Finished building content tree for 11 at    0.0691 seconds [3]
    2016-12-22 10:58:52 - DEBUG --> Content tree is
    Array
    (
        [ca_objects] => Array
            (
                [0] => Array
                    (
                        [debug_test_voca] => Array
                            (
                                [debug_test_voca] => 210
                                [_errorPolicy] => ignore
                            )
                    )
            )
    )

    2016-12-22 10:58:52 - DEBUG --> Created idno 11 at    0.1117 seconds
    2016-12-22 10:58:52 - DEBUG --> Finished inserting content tree for 11 at    0.1799 seconds into database
    2016-12-22 10:58:52 - INFO --> [11] Imported  as ca_objects 
    2016-12-22 10:58:52 - INFO --> Import of /home/bruno/dev/ee/collectiveaccess/import/problèmes/1/1.csv completed using mapping test_import: 2 imported/0 skipped/0 errors

    Indeed the dontCreate param isn't passed to the content tree hash (errorPolicy is set to "ignore" while I set it to "stop", but this is another story (or maybe not, maybe this is linked to the same problem))

    Maybe it's the good time to fill a bug ? 

    Thanks again
    Share on Twitter
  • sethseth
    Can you send me your mapping file? 
    Share on Twitter
  • yeltoyelto
    The mapping is attached to the first message of this thread.
    Share on Twitter
  • sethseth
    There are no attachments displaying for me. Odd.
    Share on Twitter
  • brunettonbrunetton
    Hi Seth,
    Weired that you don't see the attached file.
    Here's the direct link: http://www.collectiveaccess.org/support/forum/uploads/FileUpload/a3/5af4ef15e981725049b31b4f64ef06.xls
    Share on Twitter
  • yeltoyelto
    Maybe you don't see it because the icon for attachments (supposedly http://www.collectiveaccess.org/support/forum/support/forum/plugins/FileUpload/images/file.png ) is missing?
    Share on Twitter
  • brunettonbrunetton
    edited December 2016
    Indeed Yelto, depending on browser used, there may be or not a replacement icon for missing icon.

    Anyway, I filled a bug for this as I approximately found the problem in the code:


    I suggest technical discussion to be continued in Jira
    Share on Twitter
  • sethseth
    Ok found the problem. Patches are in Github/develop.

    The issue with the attachments isn't specific to my browser – I see other attachments. I'll need to investigate that... this forum software isn't the greatest.
    Share on Twitter
  • brunettonbrunetton
    Thanks, I'll apply that to my local version, check and close the issue.
    See you at next one ;)
    Share on Twitter
Sign In or Register to comment.