Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloud Drive synchronization generates lot of indexation errors in 5.0.x #69

Open
vsellier opened this issue Mar 6, 2018 · 3 comments
Open

Cloud Drive synchronization generates lot of indexation errors in 5.0.x #69

vsellier opened this issue Mar 6, 2018 · 3 comments

Comments

@vsellier
Copy link

vsellier commented Mar 6, 2018 •
edited
Loading

During the synchronization, there is a lot of errors like this on the logs in 5.0.0-RC11 version :

2018-03-06 15:20:11,261 | ERROR | Error while indexing file fbaf0a669e4534520bdd52edca277631 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-13>]
javax.jcr.ItemNotFoundException: Node not found fbaf0a669e4534520bdd52edca277631 at collaboration
        at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:159) ~[commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:320) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.0.0-RC11.jar:5.0.0-RC11]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]

There is a lot of exception like this one relative to different ids, but no other errors before.
I don't know if it's due to my drive content or a problem with some transactions concurrency.
@pnedonosko
Copy link
Contributor

This problem can be a normal behaviour as indexer works asynchronously and some nodes may gone to reflect the remote changes.

@vsellier
Copy link
Author

It doesn't seems so normal, for example, with an empty dropbox account (only the 2 default files), when I link it, 4 exception are logged :

2018-08-27 11:35:53,257 | INFO  | Dropbox - test vse connect started. [o.e.clouddrive.rest.ConnectService<http-nio-127.0.0.1-10180-exec-9>]
2018-08-27 11:35:56,321 | ERROR | Error while indexing file 7abd585495ca548f56d0ebb80f13c515 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-21>]
javax.jcr.ItemNotFoundException: Node not found 7abd585495ca548f56d0ebb80f13c515 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:323) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,372 | ERROR | Error while indexing file 7abd638c95ca548f4cdfd5ce6490bc24 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-25>]
javax.jcr.ItemNotFoundException: Node not found 7abd638c95ca548f4cdfd5ce6490bc24 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:323) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,372 | ERROR | Error while indexing file 7abd585495ca548f56d0ebb80f13c515 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-25>]
javax.jcr.ItemNotFoundException: Node not found 7abd585495ca548f56d0ebb80f13c515 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:377) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,373 | ERROR | Error while indexing file 7abd585495ca548f56d0ebb80f13c515 [o.e.s.w.s.c.FileindexingConnector<ExoScheduler_Worker-25>]
javax.jcr.ItemNotFoundException: Node not found 7abd585495ca548f56d0ebb80f13c515 at collaboration
	at org.exoplatform.services.jcr.impl.core.SessionImpl.getNodeByIdentifier(SessionImpl.java:743) ~[exo.jcr.component.core-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.create(FileindexingConnector.java:124) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.services.wcm.search.connector.FileindexingConnector.update(FileindexingConnector.java:176) ~[ecms-core-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.es.client.ElasticContentRequestBuilder.getCreatePipelineDocumentRequestContent(ElasticContentRequestBuilder.java:163) ~[commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processCUD(ElasticIndexingOperationProcessor.java:377) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.processBulk(ElasticIndexingOperationProcessor.java:212) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.index.impl.ElasticIndexingOperationProcessor.process(ElasticIndexingOperationProcessor.java:150) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.exoplatform.commons.search.job.BulkIndexingJob.execute(BulkIndexingJob.java:41) [commons-search-5.1.0-RC09.jar:5.1.0-RC09]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.2.jar:na]
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.2.jar:na]
2018-08-27 11:36:01,581 | INFO  | Dropbox - test vse successfully connected. [o.e.clouddrive.rest.ConnectService<clouddrive-thread-1>]

It seems at least these exceptions should be catched correctly to avoid to generate such noise or at worst, this ES indexation does not support clouddrive documents as if I'm not wrong they are not present on the plf storage.

@pnedonosko
Copy link
Contributor

Several things should be taken in account: connected files have content with zero length, in some cases (like Dropbox) file originally uploaded will be renamed by a connector to follow hierarchy rules specific to the provider.
As we see in the stacktraces, it's not the add-on code but a job of the indexer - it's where we need a fix of an empty docs and file rename (e.g. ignore not found and rely on a next indexing job for a renamed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vsellier @pnedonosko