existingRecordPolicy ignored in data import

jairomelo

I have spent several hours trying to update my data to add relationships between objects and entities. I have made several attempts to perform a mapping that allows me to add a type of relationship between objects and entities without success.

I have my installation configured to avoid duplicates and have tried several policies for existing data without success:

merge_on_preferred_labels_with_replace

merge_on_idno_with_replace

merge_on_idno_and_preferred_labels_with_replace

I have tried with another map and a dataset that includes the preferred labels of the objects, but it did not work either.

Exploring in the forums they said that it could be a bad declaration of the type identifiers, but "item" is correct for object and "ind" is equally correct for entities.

Any idea what is happening?

(our installed version is 1.8)

seth

I assume the idno's match existing data? I just ran a subset of this on the demo.collectiveaccess.org site with success. (I changed the idno's on a couple of existing records to match the first two rows of your data). Data and mapping I used are attached. Changes in the mapping are highlighted in yellow. There are two changes in the mapping:

1. Set existing record policy to merge_on_idno. This assumes that column 1 in your data matches idno's in existing records in your system.
2. Added the ignoreTypeForExistingRecordPolicy setting to your mapping with value=1. By default, the existing record policy will match on idno (and/or preferred label) as well as type. This ensures that matching is accurate when identifiers are reused across different types of records, which happens a lot. You can disable this using ignoreTypeForExistingRecordPolicy and just do straight matching on idno. For my test on demo.collectiveaccess.org this was required (there's no "item" type on that system, and I didn't feel like adding on). I'm guessing that in your system, the record you're matching aren't actually of type "item". Perhaps they are, I have no way of knowing, but give it a try with this option set and see if helps.

Run this mapping with debugging-level logging enabled. If this doesn't work, take a close look at the logs. If you can't discern the problem from them post them here or if you don't want them public send them to support@collectiveaccess.org.

seth

Also, here's a starter importer template with all 1.8 mapping level settings in place and described. Perhaps that will be helpful.

jairomelo

Thanks for your help Seth. You bring me a lot of insights to make my import. Unfortunately it was unsuccessful :( I cannot make it work with any configuration possible. I enabled the debugging-level logging but there's no messages (no errors, no warnings....) At this momment, I'm afraid this could be our installation. Could be possible, maybe, that our Schema revision is 180?

seth

Being at revision 180 is not the problem. I'd have to look at your setup to tell you anything more.

jairomelo

Sure :) I'm sharing with you the configuration as it is exported from the ExportConfiguration tool and a screenshot of our ConfigurationCheck

I hope you can see something that's causing this problem.

jairomelo

Well, I thought it could be the locale. I changed to en_US in the importer but it has no effect :(

mbutel

I've had the same problem in 1.8 while i was trying to import relationships between objects and entities using a "^3" for the relationship type.

The mapping: https://docs.google.com/spreadsheets/d/1CPZCzS1WoQxKX-8PtXycakapZS4P6bZ0AnVP1UyVO3Y/edit#gid=2147344053

The test data:

idno,ca_entities.preferred_label,type
1,Leo Isaias Kremer,owner

The log: https://docs.google.com/document/d/1FmORKPCHqT7KKkUwoWKuAIMrVvxpR9rLxys_EIdI5O4/edit?usp=share_link

mbutel

This part of the log is strange:

[ca_entities] => Array
        (
            [0] => Array
                (
                    [_type] => [individual]
                    [_relationship_type] => owner
                    [preferred_labels] => Array
                        (
                            [surname] => owner
                            [forename] => 
                            [middlename] => 
                            [displayname] => owner
                            [prefix] => 
                            [suffix] => 
                        )

                    [idno] => owner
                    [_matchOn] => Array
                        (
                            [0] => labels
                        )

                    [_errorPolicy] => ignore
                )

It reads the relationshipType following ^3 but then it is used for other elements.

seth

It looks like you're mismapping something. I'll try to look into this further soon.