Adding individuals to a group with import

Get your CiviCRM questions answered here!

Adding individuals to a group with import

Postby J3landon » Tue Mar 15, 2011 3:51 pm

I have 50 individual names who are former board members that I want to add to a group called "Board Member - Former". Some of the individual names already exist in the database and some do not. When I import using Contacts/Import Contacts, 50 new "individual" records are created for both the pre-existing names and those not previously in the database. I get identical results if I use Skip, update, or fill. Shouldn't I be able to do this without creating duplicate records?
John
J3landon
 
Posts: 1
Joined: Tue Mar 15, 2011 3:28 pm
 

Re: Adding individuals to a group with import

Postby Conrad » Wed Apr 20, 2011 10:03 am

I'm working on a more complete answer. For the moment, I'm posting links to pages that discuss the 'Import' and 'Dedupe' process.

Dedupe Rules: http://civicrm.org/node/201
Default Dedupe Rules: http://civicrm.org/node/385
Importing Data: http://wiki.civicrm.org/confluence/disp ... rting+Data
Preparing MicroSoft Exel for Import: http://wiki.civicrm.org/confluence/disp ... +to+import
Importing Data: http://en.flossmanuals.net/CiviCRM/Cont ... Importing/
Importing Relationships: http://blogs.terrorware.com/geoff/2009/ ... o-civicrm/
Migrating Raiser's Edge: http://civicrm.org/blogs/dharmatech/mig ... ge-civicrm
Import Tips: http://www.grahamnott.com/2010/07/civic ... acts-tips/
Importing Data: http://en.flossmanuals.net/CiviCRM/DevelopImport/
Importing Contacts: http://www.booki.cc/civicrm-1/_v/1.0/ad ... importing/
Importing Data: http://www.gingerfeet.net/civicrm-userg ... rting-data

After I do an import & dedupe with 'real' data, I'll report what I discover here.

Conrad
Conrad
Site Admin
 
Posts: 54
Joined: Tue Feb 15, 2011 4:20 pm
Location: GingerFeet LLC
 

Re: Adding individuals to a group with import

Postby Conrad » Sat May 07, 2011 2:16 pm

Hello John,

I finally had time to set up a duplicate of your database and run the import.

The issue is setting the strict/fuzzy dedupe rules.

These rules are found under:

Contacts -> Find and Merge Duplicate Contacts

Which will open up to a listing of the rules.

You can edit the rules with the 'edit' link to the right.

Strict rules are automatically used by CiviCRM for imports.
Fuzzy rules are automatically used by CiviCRM for deduping existing data.

So you would need to edit the strict rule for individuals prior to importing.

Rules.

Rules have up to 5 'Fields'/'Lengths'/'Weights' and 1 'Threshold Weight'.

Field = what field is being compared against in the current CiviCRM db.
Length = how long a string do we compare (and ignore the remainder)
Weight = how much importance do we attach to the comparison

Note that 'Weight' is more understandable if you think of it as 'points'

Threshold Weight = how much weight the fields need to be considered matching

Note that 'Threshold Weight' is more understandable if you think of it as 'total points'

The following examples ignore Length.
They are just Field, Weight, Threshold

Example 1:

Email 3
First Name 3
Last Name 3

Threshold 10 (that is Total Points)

Result: you will never have a match. All imported entries will be considered brand new. 3+3+3=9, you will never exceed the threshold. CiviCRM will not skip, will not update, but it will create a new entry.

Example 2:

Email 10
First Name 3
Last Name 3

Threshold 10

Result: the only thing that matters is email. If email matches, the import will consider it a duplicate. It will either update or skip depending on which radio button you selected during the import. Email=10 which meets threshold. Email:10+First Name:3=13 and is greater than threshold, but is totally irrelevant. Who cares that the name also matches, when you said that it was good enough for the email to match? Also, if First name & Last name match, they will never, ever add up to 10, so they will never, ever be considered a match.

Example 3:

Email 5
First Name 3
Last Name 3

Threshold 8

Result: You must have either a match on the email+first name OR a match on email+last name. If First & Last name match, you only added up to a total of 6 points, and do not pass threshold. So you are saying: email is really important, and a name is used to help verify. But it can be either first or last.

Example 4:

Email 10

Threshold 10

Result: The only thing you are checking is email. If your import data does not contain an email address, you will never, ever match. If the import data does contain an email address, and it is identical to one found in the existing CiviCRM database, it will either skip, or update that record.

OK, we're starting to get a grip on the situation. There are rules that we can set up to control how a decision is made about matching.

- The rule must match against a field (or fields) in the existing CiviCRM that contains data (no data = no match)

- The rule must match against a field (or fields) in the .csv file to be imported (no data = no match)

- If there is no match, CiviCRM does not skip, does not update. It creates a new entry

Ah. Your problem is the import creates new entries. It does not skip. It does not update. It probably means that CiviCRM is not finding any matches...

Your .csv file looks like this:

"last name","first name",,"Comments"
"Jake"," Jakaroo"," Jr.","not in database"

The only possible things you can seriously consider for matching are
Last name
First name
Last name & First name

And of those, it is obvious that you need both the first name and the last name.

(Edit your .csv file to add 'suffix' to the first row / third column. It helps you and CiviCRM during the import setup.)

Go to:
Contacts -> Find and Merge Duplicate Contacts

Click on the 'Edit Rule' link for Individual / Strict (remember that strict rules are used for import, and you want to do an import).

You will see that your rule is to match email. And nothing else. Same as Example 4. So you can import First name, Last name without an email address all you want, and it will never find a match, even though you have contacts in the db with the same first/last combination.

Now
add a First name field and give it a weight of 5
add a Last name field and give it a weight of 5

Because your threshold is at 10, when the first/last combination hits, CiviCRM will consider it a match and either skip or update.

Click the 'Save' button.

You should be good to go.

The lesson: your dedupe rules are not fixed, they are not constant. You must adjust your dedupe rules depending on what the fields are that you are importing. If your rules are checking against a field that does not exist in your import, it will never match.

Conrad
Conrad
Site Admin
 
Posts: 54
Joined: Tue Feb 15, 2011 4:20 pm
Location: GingerFeet LLC
 
 

Return to CiviCRM How To

Who is online

Users browsing this forum: No registered users and 1 guest

cron