High "Batch items size" breaks CiviCRM Queue Runner

[sluc23]

Ok, this is weird and took me some time to find it. And very likely it's related with CiviCRM Queue / PHP limitations, rather than CSV_Import extension.

When setting a relatively "high" amount of items to be processed in each queue task item run, the queue gets stuck and this error is given in the AJAX call:

Error: Call to a member function run() on boolean in CRM_Queue_Runner->runNext() (line 214 of <>/civicrm/CRM/Queue/Runner.php).

After some research I've found that the field data in civicrm_queue_item table, gets truncated and it doesn't store more than ~ 65500 characters, so the serialized data object is broken.

I could replicate that behavior in an internal extension we have, which implements CiviCRM Queue, and got same results, when data field is big.. doesn't store more than approx 65520 characters.

data is type of text in MySQL , so no limitations there.

I'll keep on investigating this, just wanted to point out in case anybody faced the same issue

[sluc23]

Looks like class CRM_Queue_DAO_QueueItem is the one that doesn't persist properly the data value in the MySQL table

[vinuvarshith]

@sluc23 You are right. The issue is that 'civicrm_queue_item' table which stores the data is of type 'text' which has a limit of '65,535 characters'.
The workaround is to use lesser number of items per batch or to manually update the data type of 'data' to some other type that supports more content. 'civicrm_batch' table which has a 'data' column which uses 'longtext' type for example. This should be raised as an improvement to civicrm core.

[eileenmcnaughton]

Interesting - I expect it affects built in imports too? Or do they not use the queue runner like that

[sluc23]

built in imports don't use queue runner at all (which would be very nice to have), which leads to classic php timeouts error when importing relative big files.

i.e.:
https://github.com/civicrm/civicrm-core/blob/master/CRM/Contact/Import/Parser.php#L209