You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the end of a fairly long run of Evaluation.evaluate (approx. 8000 examples), I get a peculiar error that does not appear for shorter runs.
Exception in thread Thread-1 (_process_batches):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/lib/python3.12/site-packages/weave/trace_server/async_batch_processor.py", line 57, in _process_batches
self.processor_fn(current_batch)
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/weave/trace_server/remote_http_trace_server.py", line 166, in _flush_calls
r.raise_for_status()
File "/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 413 Client Error: Request Entity Too Large for url: https://trace.wandb.ai/call/upsert_batch
Evaluation summary
<omitted for conciseness>
Exception in thread Thread-1 (_process_batches):
Traceback (most recent call last):
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/lib/python3.12/site-packages/weave/trace_server/async_batch_processor.py", line 57, in _process_batches
self.processor_fn(current_batch)
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/weave/trace_server/remote_http_trace_server.py", line 166, in _flush_calls
r.raise_for_status()
File "/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 413 Client Error: Request Entity Too Large for url: https://trace.wandb.ai/call/upsert_batch
The text was updated successfully, but these errors were encountered:
Hi @ryanrudes thanks for reporting this. My apologies you ran into this issue. Do you mind giving us a bit more information about your environment? Specifically, the weave version you are using, as well as your wandb username. It is possible that this issue has already been resolved in master (here), but if you are logging large images or very large (30+MB) traces this notably does not account for that. Thanks!
Hi, my trace is most definitely larger than 30 MB. I was not aware that that would be considered large. I am noting that these runs are also appearing as ongoing in the portal despite having finished. Also, although all of the data I uploaded appears in the web portal, some of it is missing when I export and download the tables.
I am using weave 0.50.13 and wandb 0.17.6. My wandb username is ryanrudes-dtxplus.
@ryanrudes Both of those issues, large traces erroring and the export not containing the full data, are ticketed and under active development. They should be resolved in the near future; this thread will be updated when they are merged and deployed. Thanks for raising these.
Also, a new version of the weave SDK was released yesterday, so updating the weave package will include the changes I mentioned that might alleviate your upsert_batch issue.
At the end of a fairly long run of
Evaluation.evaluate
(approx. 8000 examples), I get a peculiar error that does not appear for shorter runs.The text was updated successfully, but these errors were encountered: