Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrowdKernel Time Limit Error #8

Open
iamzoltan opened this issue May 15, 2024 · 1 comment
Open

CrowdKernel Time Limit Error #8

iamzoltan opened this issue May 15, 2024 · 1 comment

Comments

@iamzoltan
Copy link

t/getQuery
minionworker-1       | [2024-05-15 21:26:45,860: WARNING/ForkPoolWorker-1] ########## Finished namespace:5c9d71061565ca92b097ff494c969b_UncertaintySampling,  job_uid=28fcb8a2bb9ae4716a46e4b6a58e63d6,  time_enqueued=0.001793,  execution_time=0.06032109260559082 ##########
minionworker-1       | [2024-05-15 21:26:45,874: WARNING/ForkPoolWorker-1] #### Finished getQuery,  time_enqueued=0.001527,  execution_time=0.053614139556884766 ####
nextbackenddocker-1  | [2024-05-15 21:26:46 +0000] [7] [DEBUG] POST /api/experiment/processAnswer
minionworker-1       | [2024-05-15 21:26:46,688: WARNING/ForkPoolWorker-1] #### Finished processAnswer,  time_enqueued=0.004867,  execution_time=0.03965330123901367 ####
minionworker-1       | [2024-05-15 21:26:46,688: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_RandomSampling,  job_uid=2bb003efb24bb9353c5d7b9372af0715,  time_enqueued=0.00165 <<<<<<<<<
nextbackenddocker-1  | [2024-05-15 21:26:46 +0000] [7] [DEBUG] POST /api/experiment/getQuery
minionworker-1       | [2024-05-15 21:26:46,790: WARNING/ForkPoolWorker-1] #### Finished getQuery,  time_enqueued=0.002666,  execution_time=0.05387735366821289 ####
minionworker-1       | [2024-05-15 21:26:46,809: WARNING/ForkPoolWorker-1] ########## Finished namespace:5c9d71061565ca92b097ff494c969b_RandomSampling,  job_uid=2bb003efb24bb9353c5d7b9372af0715,  time_enqueued=0.00165,  execution_time=0.08316993713378906 ##########
nextbackenddocker-1  | [2024-05-15 21:26:47 +0000] [7] [DEBUG] POST /api/experiment/processAnswer
minionworker-1       | [2024-05-15 21:26:47,452: WARNING/ForkPoolWorker-1] #### Finished processAnswer,  time_enqueued=0.00487,  execution_time=0.006047248840332031 ####
nextbackenddocker-1  | [2024-05-15 21:26:47 +0000] [7] [DEBUG] POST /api/experiment/getQuery
minionworker-1       | [2024-05-15 21:26:47,528: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[54b761dd-7806-4974-b0b2-1befc4cefbce]
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       | [2024-05-15 21:26:47,529: WARNING/ForkPoolWorker-2]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 21:26:47,530: WARNING/ForkPoolWorker-2] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=8703e697993eb09dddde937b621b02ff,  time_enqueued=109.411547 <<<<<<<<<
minionworker-1       | [2024-05-15 21:26:47,551: WARNING/ForkPoolW
@iamzoltan
Copy link
Author

ceeded()
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:14,924: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:14,926: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=c487e70e3d9b0d3404bd6ed37a34c69e,  time_enqueued=1497.313422 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:19,859: WARNING/ForkPoolWorker-1] ########## Finished namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=c487e70e3d9b0d3404bd6ed37a34c69e,  time_enqueued=1497.313422,  execution_time=4.914435148239136 ##########
minionworker-1       | [2024-05-15 22:10:19,860: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=f8937f28325a1badfc80cc85f6ecf96f,  time_enqueued=1501.996853 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:24,861: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[afce0f3e-3b28-49c2-b2a9-ea14f9df1750]
minionworker-1       | [2024-05-15 22:10:24,861: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2552, in norm
minionworker-1       |     sqnorm = x.dot(x)
minionworker-1       |              ^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2552, in norm
minionworker-1       |     sqnorm = x.dot(x)
minionworker-1       |              ^^^^^^^^
minionworker-1       | [2024-05-15 22:10:24,862: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:24,864: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=b047e418c61c5076bf8295d0fa57384b,  time_enqueued=1506.169664 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:29,866: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[e3532eff-4269-48e0-ac98-b9294cc2daaf]
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2546, in norm
minionworker-1       |     x = x.ravel(order='K')
minionworker-1       |         ^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 167, in getCrowdKernelTripletProbability
minionworker-1       |     ca = norm(c-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2546, in norm
minionworker-1       |     x = x.ravel(order='K')
minionworker-1       |         ^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:29,867: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:29,869: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=946a918166627ce902e142eda4ac18c8,  time_enqueued=1509.985539 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[9cc99a61-3f6c-472d-9d58-5cff778d3370]
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,870: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       | [2024-05-15 22:10:34,871: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:34,872: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=e01b2df9e2aeb66f1e8173fe4008b3bb,  time_enqueued=1513.741367 <<<<<<<<<
minionworker-1       | [2024-05-15 22:10:39,872: WARNING/MainProcess] Soft time limit (5s) exceeded for next.broker.celery_app.tasks.apply_sync_by_namespace[229ba661-5919-4cbd-919a-b7a53199fa6a]
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1] tasks Exception: SoftTimeLimitExceeded() Traceback (most recent call last):
minionworker-1       |   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       |   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       |   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/next/broker/celery_app/tasks.py", line 201, in apply_sync_by_namespace
minionworker-1       |     response, dt = next.utils.timeit(getattr(next_alg, task_name))(butler, args)
minionworker-1       |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/next/utils.py", line 219, in timed
minionworker-1       |     result = f(*args, **kw)
minionworker-1       |              ^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/myAlg.py", line 124, in incremental_embedding_update
minionworker-1       |     tau = utilsCrowdKernel.getCrowdKernelTauDistribution(X,S,mu)
minionworker-1       |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 208, in getCrowdKernelTauDistribution
minionworker-1       |     tau[a,i] = tau[a,i] + log( getCrowdKernelTripletProbability(X[q[0]], X[q[1]], X[i], mu=mu))
minionworker-1       |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/next_backend/apps/PoolBasedTripletMDS/algs/CrowdKernel/utilsCrowdKernel.py", line 168, in getCrowdKernelTripletProbability
minionworker-1       |     ba = norm(b-a)
minionworker-1       |          ^^^^^^^^^
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/numpy/linalg/linalg.py", line 2379, in _norm_dispatcher
minionworker-1       |     def _norm_dispatcher(x, ord=None, axis=None, keepdims=None):
minionworker-1       |
minionworker-1       | [2024-05-15 22:10:39,873: WARNING/ForkPoolWorker-1]   File "/usr/local/lib/python3.11/site-packages/billiard/pool.py", line 228, in soft_timeout_sighandler
minionworker-1       |     raise SoftTimeLimitExceeded()
minionworker-1       | [2024-05-15 22:10:39,874: WARNING/ForkPoolWorker-1] >>>>>>>> Starting namespace:5c9d71061565ca92b097ff494c969b_Test,  job_uid=bb59a54cd5234049af11ba6d9cae44ea,  time_enqueued=1517.899519 <<<<<<<<<

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant