error in response_body for s3 uploads using json encoder

[sadath-12]

I'm following https://github.com/tensorlakeai/indexify/blob/main/examples/readme/readme_example.py excpet that I'm I set encoder="json" in ever function .

When I run it I get

{"timestamp":"2024-11-18T07:57:14.465179Z","level":"INFO","message":"Checking for error in response_body: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<CompleteMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Location>https://s3.us-east-1.amazonaws.com/indexify_bucket/afa2c059-7e42-4f3a-8e62-</Location><Bucket>indexify_bucket</Bucket><Key>afa2c059-7e42-4f3a</Key><ETag>\"a11d6ef96cf2fa93a2d521b31d1ce2c8-1\"</ETag></CompleteMultipartUploadResult>","target":"object_store::client::retry","span":{"matched_path":"/namespaces/:namespace/compute_graphs/:compute_graph/invoke_object","method":"POST","uri":"/namespaces/default/compute_graphs/sequence_summer/invoke_object?block_until_finish=true","name":"request"}}

And the task is hanged up there .

My environment

I used k8s to deploy the indexify-server and executor with values.yaml

blobStore:
  allowHTTP: true
    #endpoint: http://blob-store:9000
  credentialSecret: blob-creds
  config:
    backend: s3
    path: s3://indexify_bucket
    dynamodb_table: graph_ctx
    s3:
      bucket: indexify_bucket
      region: us-east-1
      accessKey: 123
      secretKey: 123

And used indexify version
Version: 0.2.29

[sadath-12]

The uploads to s3 increases latency and I was running into timeouts . Increasing timeouts was the workaround

[seriousben]

Thanks for reporting this.

More info:

When uploading large fields or lots of different function outcomes, the executor's Task Report can be stuck in a retry loop that will never succeed.

Error seen:

task-reporter failed to report task outcome timed out.

╭──────────────────────────────────── Reporting Error ─────────────────────────────────────╮
│ Failed to report task 902492df-006e-4c63-8c2d-660101f7f12e                               │
│ Exception:timed out   Retrying...                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────╯

[seriousben]

Reopening to also track the invocation aspect that this ticket mentioned.