JSONDecodeError in Client and Pydantic Schema Error on Server during Pipeline Deployment

[kkmarv]

Problem Definition

I have tried to deploy two different pipeline definitions using the SentenceTransformersTextEmbedder of which both fail with errors in server and client code. Furthermore, hayhooks status does not seem to be aware of the deployment failure.

Expected Behaviour

• Pipelines deploy without errors, and the client retrieves a valid response.
• hayhooks status returns no deployed pipelines.

Observed Behaviour

• Client: JSONDecodeError
• Server: PydanticSchemaGenerationError
• hayhooks status says the pipelines was successfully deployed despite not being the case.

Client stacktrace

$ hayhooks deploy MinimalRetrievalPipe.yaml 
Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".venv/bin/hayhooks", line 8, in <module>
    sys.exit(hayhooks())
             ^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/hayhooks/cli/deploy/__init__.py", line 23, in deploy
    click.echo(f"Error deploying pipeline: {resp.json().get('detail')}")
                                            ^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Server stacktrace

$ hayhooks run
INFO:     Pipelines dir set to: pipelines.d
INFO:     Started server process [1001376]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://localhost:1416 (Press CTRL+C to quit)
INFO:     127.0.0.1:34380 - "POST /deploy HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File ".venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File ".venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File ".venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File ".venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File ".venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File ".venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File ".venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File ".venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/hayhooks/server/handlers/deploy.py", line 7, in deploy
    return deploy_pipeline_def(app, pipeline_def)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/hayhooks/server/utils/deploy_utils.py", line 21, in deploy_pipeline_def
    PipelineRunResponse = get_response_model(pipeline_def.name, pipe.outputs())
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/hayhooks/server/pipelines/models.py", line 60, in get_response_model
    response_model[component_name] = (create_model("ComponentParams", **component_model, __config__=config), ...)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/main.py", line 1673, in create_model
    return meta(
           ^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py", line 226, in __new__
    complete_model_class(
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_model_construction.py", line 658, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/main.py", line 697, in __get_pydantic_core_schema__
    return handler(source)
           ^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 881, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 693, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1073, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1261, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2051, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2032, in inner_handler
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 988, in match_type
    return self._match_generic_type(obj, origin)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1020, in _match_generic_type
    return self._list_schema(self._get_first_arg_or_any(obj))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 369, in _list_schema
    return core_schema.list_schema(self.generate_schema(items_type))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 984, in match_type
    return self._dataclass_schema(obj, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1765, in _dataclass_schema
    args = sorted(
           ^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1766, in <genexpr>
    (self._generate_dc_field_schema(k, v, decorators) for k, v in fields.items()),
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1090, in _generate_dc_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1261, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2051, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
    schema = self._handler(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 2032, in inner_handler
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 988, in match_type
    return self._match_generic_type(obj, origin)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1016, in _match_generic_type
    return self._union_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 1323, in _union_schema
    choices.append(self.generate_schema(arg))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
    return self.match_type(obj)
           ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 997, in match_type
    return self._unknown_type_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/pydantic/_internal/_generate_schema.py", line 515, in _unknown_type_schema
    raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'pandas.core.frame.DataFrame'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.

If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.

For further information visit https://errors.pydantic.dev/2.10/u/schema-for-unknown-type

Steps to reproduce

$ hayhooks run

$ hayhooks deploy MinimalRetrievalPipeQdrant.yaml

MinimalRetrievalPipeQdrant.yaml

components:
  document_embedder:
    init_parameters:
      batch_size: 32
      config_kwargs: null
      device:
        device: cpu
        type: single
      model: sentence-transformers/paraphrase-MiniLM-L3-v2
      model_kwargs: null
      normalize_embeddings: false
      precision: float32
      prefix: ''
      progress_bar: true
      suffix: ''
      token:
        env_vars:
        - HF_API_TOKEN
        - HF_TOKEN
        strict: false
        type: env_var
      tokenizer_kwargs: null
      truncate_dim: null
      trust_remote_code: false
    type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
  document_retriever:
    init_parameters:
      document_store:
        init_parameters:
          api_key: null
          embedding_dim: 384
          force_disable_check_same_thread: false
          grpc_port: 6334
          hnsw_config: null
          host: null
          https: null
          index: Document
          init_from: null
          location: null
          metadata: {}
          on_disk: false
          on_disk_payload: null
          optimizers_config: null
          path: null
          payload_fields_to_index: null
          port: 6333
          prefer_grpc: false
          prefix: null
          progress_bar: true
          quantization_config: null
          recreate_index: false
          replication_factor: null
          return_embedding: false
          scroll_size: 10000
          shard_number: null
          similarity: cosine
          sparse_idf: false
          timeout: null
          url: http://localhost:6333
          use_sparse_embeddings: false
          wait_result_from_api: true
          wal_config: null
          write_batch_size: 100
          write_consistency_factor: null
        type: haystack_integrations.document_stores.qdrant.document_store.QdrantDocumentStore
      filter_policy: replace
      filters: null
      group_by: null
      group_size: null
      return_embedding: false
      scale_score: false
      score_threshold: null
      top_k: 3
    type: haystack_integrations.components.retrievers.qdrant.retriever.QdrantEmbeddingRetriever
connections:
- receiver: document_retriever.query_embedding
  sender: document_embedder.embedding
max_runs_per_component: 100
metadata: {}

MinimalRetrievalPipeSentenceTransfomer.yaml

components:
  document_embedder:
    init_parameters:
      batch_size: 32
      config_kwargs: null
      device:
        device: cpu
        type: single
      model: sentence-transformers/paraphrase-MiniLM-L3-v2
      model_kwargs: null
      normalize_embeddings: false
      precision: float32
      prefix: ''
      progress_bar: true
      suffix: ''
      token:
        env_vars:
        - HF_API_TOKEN
        - HF_TOKEN
        strict: false
        type: env_var
      tokenizer_kwargs: null
      truncate_dim: null
      trust_remote_code: false
    type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
  document_retriever:
    init_parameters:
      document_store:
        init_parameters:
          bm25_algorithm: BM25L
          bm25_parameters: {}
          bm25_tokenization_regex: (?u)\b\w\w+\b
          embedding_similarity_function: dot_product
          index: b39f1fea-7c83-4fdc-a9e0-928e3d5e4ae7
        type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
      filter_policy: replace
      filters: null
      return_embedding: false
      scale_score: false
      top_k: 3
    type: haystack.components.retrievers.in_memory.embedding_retriever.InMemoryEmbeddingRetriever
connections:
- receiver: document_retriever.query_embedding
  sender: document_embedder.embedding
max_runs_per_component: 100
metadata: {}

Versions

"colorama~=0.4.6",
"hayhooks~=0.0.18",
"haystack-ai~=2.8.0",
"haystack-experimental~=0.3.0",
"huggingface_hub~=0.24.2",
"minio~=7.2.9",
"pymupdf~=1.24.7",
"pymupdf4llm~=0.0.8",
"qdrant-haystack~=7.0.0",