missing valid contacts & cycling very clowly

[MarshCastle]

Hi Vitor,
Using this plugin on a D6 site, it's taken over two months to check arounf 8000 contacts.
here's a screensshot of the last 2 days... this is repetitive since installation (nov 2014).

There are around 10000 contacts on the site, and around 14 contacts have been overlooked ie:

[ccjs_civicrm]> select id, first_name, last_name, is_deleted
-> from civicrm_contact
-> where contact_type = "Individual"
-> AND (last_name = BINARY LOWER(last_name) OR last_name = BINARY LOWER(last_name))
-> AND is_deleted = 0;
| 1577 | chaxxxx | weixxxxx | 0 |
| 5909 | Amxx | kexxx | 0 |
| 7306 | suxxxxx | ferxxxxxx | 0 |
| 7463 | axxxxx | mexxx | 0 |
| 7694 | agxxxx | burxxx | 0 |
...
.... more with greater cid's...

all with a lower cid than the "offset" of 8006 shown in the logs (sorry about the x's - they replace lower case letters).
These contacts are not deleted (there are around 35 missed (a few upper & 28 lower case)

You know of any reason for this not to work?
Had a dev look over it and he says yr code is good and should work, but for some reason it's falling over. You have any suggestions why this may be the case?

Thanks,
Martin

[futurefirst]

Hi Martin,

I've got a few ideas:

1. Deadlocks http://forum.civicrm.org/index.php?topic=35409.0 on updates?
2. Automatic deduping?

The other thing is, perhaps the display_name isn't being updated upon a
first_name / last_name change via the API calls we use? I think I
encountered this bug on our View and Edit Contact Screen. I can't remember
what the resolution was though.

Thanks for raising this. If you could continue to assist then that'd be
ace. It is a neat extension,

John

On Wed, Feb 4, 2015 at 4:13 PM, marsh-circle [email protected]
wrote:

Hi Vitor,
Using this plugin on a D6 site, it's taken over two months to check arounf
8000 contacts.
here's a screensshot of the last 2 days... this is repetitive since
installation (nov 2014).
[image: caps_concersionlogs]
https://cloud.githubusercontent.com/assets/10851325/6043889/9a4d13a4-ac87-11e4-9507-38fe0e5bf17c.png

There are around 10000 contacts on the site, and around 14 contacts have
been overlooked ie:

[ccjs_civicrm]> select id, first_name, last_name, is_deleted
-> from civicrm_contact
-> where contact_type = "Individual"
-> AND (last_name = BINARY LOWER(last_name) OR last_name = BINARY
LOWER(last_name))
-> AND is_deleted = 0;
| 1577 | chaxxxx | weixxxxx | 0 |
| 5909 | Amxx | kexxx | 0 |
| 7306 | suxxxxx | ferxxxxxx | 0 |
| 7463 | axxxxx | mexxx | 0 |
| 7694 | agxxxx | burxxx | 0 |
...
.... more with greater cid's...

all with a lower cid than the "offset" of 8006 shown in the logs (sorry
about the x's - they replace lower case letters).
These contacts are not deleted (there are around 35 missed (a few upper &
28 lower case)

You know of any reason for this not to work?
Had a dev look over it and he says yr code is good and should work, but
for some reason it's falling over. You have any suggestions why this may be
the case?

Thanks,
Martin

—
Reply to this email directly or view it on GitHub
#3.

John Prescott
Lead Developer
Future First
[email protected]

[futurefirst]

Hello Martin,

Thank you for using our extension and reporting this issue.

For what I can see on the screenshot there appears to be a problem indeed
because the number of contacts scanned that turn didn't reach 500.
This could be either because of an unknown error or that the cron job
reached the end of 'Individual' contacts on the database.
Please check how many 'Individual' contacts and see if it matches the cron
job offset.

It is indeed a bug we haven't seen this before. My suggestion would be to
take a look at the Civi/Drupal logs and see if there is any deadlock or
error like John suggested.

Regarding the contacts that haven't been corrected the only reasons for
that I could think of are:

• They were deduped;
• Or manually edited after being corrected.
  Note that the capscorrector will only correct a contact once. Once it
  reaches the end of contacts it will only correct new contacts.

I hope that I could be of any help. Please keep in touch so we can help you
with this issue.

Thank you,

On 4 February 2015 at 16:13, marsh-circle [email protected] wrote:

Hi Vitor,
Using this plugin on a D6 site, it's taken over two months to check arounf
8000 contacts.
here's a screensshot of the last 2 days... this is repetitive since
installation (nov 2014).
[image: caps_concersionlogs]
https://cloud.githubusercontent.com/assets/10851325/6043889/9a4d13a4-ac87-11e4-9507-38fe0e5bf17c.png

There are around 10000 contacts on the site, and around 14 contacts have
been overlooked ie:

[ccjs_civicrm]> select id, first_name, last_name, is_deleted
-> from civicrm_contact
-> where contact_type = "Individual"
-> AND (last_name = BINARY LOWER(last_name) OR last_name = BINARY
LOWER(last_name))
-> AND is_deleted = 0;
| 1577 | chaxxxx | weixxxxx | 0 |
| 5909 | Amxx | kexxx | 0 |
| 7306 | suxxxxx | ferxxxxxx | 0 |
| 7463 | axxxxx | mexxx | 0 |
| 7694 | agxxxx | burxxx | 0 |
...
.... more with greater cid's...

all with a lower cid than the "offset" of 8006 shown in the logs (sorry
about the x's - they replace lower case letters).
These contacts are not deleted (there are around 35 missed (a few upper &
28 lower case)

You know of any reason for this not to work?
Had a dev look over it and he says yr code is good and should work, but
for some reason it's falling over. You have any suggestions why this may be
the case?

Thanks,
Martin

—
Reply to this email directly or view it on GitHub
#3.

Vitor Nóbrega

Developer

*Future First *

[image: NEW LOGO SMALL]

020 7239 8933 | [email protected] [email protected] |
www.futurefirst.org.uk | @futurefirstorg

Future First, 338 City Road, London, EC1V 2PY

Future First is the public name for The Camden Future First Network, a
registered charity, number 1135638.

Standard disclaimer: this email and any attachments are confidential and
may also be privileged. If you are not the addressee, do not disclose,
copy, circulate or in any other way use or rely on the information
contained in this email or any attachments. If received in error, notify
the sender immediately and delete this email and any attachments from your
system. Emails cannot be guaranteed to be secure or error free as the
message and any attachments could be intercepted, corrupted, lost, delayed,
incomplete or amended. Future First does not accept legal responsibility
for the contents of this message or liability for damage caused by this
email or any attachments. Any views or opinions presented are only those of
the author and not those of Future First.

[MarshCastle]

Hi Guys,

the output contained first_name and last_name , so I don't think this relates to updating the display_name (as the code seems to afftect first_name & last_name columns.

As I said above - there are over 10,000 contacts on the site - so we've not run out of contacts yet.

You have an idea of how to reset the plugin / offset to cid 1 so that we can start again? (is there a variable to reset or removal & re-installing - :( ?)

Also - in what way would de-duping cause issues?

I'm going to set the #contacts to process to 100 and see if it can handle that any better.

Thanks in advance
M.

[futurefirst]

Hello Martin,

To restart/reset the offset you have to reinstall the extension.

Deduping a contact with another that doesn't have the name corrected may
cause that ('dirty') name to be the new contact name.

Also note that the offset is the number of 'Individual' contacts that the
corrector has scanned. It does not match directly to any cid.

Please do keep in touch with details about this issue.

Thank you,
Vitor

On 4 February 2015 at 16:43, marsh-circle [email protected] wrote:

Hi Guys,

the output contained first_name and last_name , so I don't think this
relates to updating the display_name (as the code seems to afftect
first_name & last_name columns.

As I said above - there are over 10,000 contacts on the site - so we've
not run out of contacts yet.

You have an idea of how to reset the plugin / offset to cid 1 so that we
can start again? (is there a variable to reset or removal & re-installing -
:( ?)

Also - in what way would de-duping cause issues?

I'm going to set the #contacts to process to 100 and see if it can handle
that any better.

Thanks in advance
M.

—
Reply to this email directly or view it on GitHub
#3 (comment)
.

Vitor Nóbrega

Developer

*Future First *

[image: NEW LOGO SMALL]

020 7239 8933 | [email protected] [email protected] |
www.futurefirst.org.uk | @futurefirstorg

Future First, 338 City Road, London, EC1V 2PY

Future First is the public name for The Camden Future First Network, a
registered charity, number 1135638.

Standard disclaimer: this email and any attachments are confidential and
may also be privileged. If you are not the addressee, do not disclose,
copy, circulate or in any other way use or rely on the information
contained in this email or any attachments. If received in error, notify
the sender immediately and delete this email and any attachments from your
system. Emails cannot be guaranteed to be secure or error free as the
message and any attachments could be intercepted, corrupted, lost, delayed,
incomplete or amended. Future First does not accept legal responsibility
for the contents of this message or liability for damage caused by this
email or any attachments. Any views or opinions presented are only those of
the author and not those of Future First.

[MarshCastle]

Hi Vitor.
Thanks for your responses / support - very much appreciated.
I've solved the issue, and the extension runs like a charm ;)

the problem came down to a permissions issue. The sites user needed permission to access /usr/share/pear/api/v3:

• Warning: is_dir(): open_basedir restriction in effect. File(/usr/share/pear/api/v3) is not within the allowed path(s): (/var/www:/var/sites/[sitename]:/tmp:/usr/share/pear/api/v3:) in civicrm_api3_entity_get() (line 16 of /var/www/drupal734civi447/sites/all/modules/civicrm/api/v3/Entity.php).

this led to a similar problem for the /usr/share/php/api/v3 folder. fixed this for the webserver too.

So -in order for the extension to work, the webservers system user for the site must have [open_basedir] access to /usr/share/php/api/v3 and /usr/share/pear/api/v3

Thanks,
M.