httpx migration problems

[xdgrulez]

Description

Seems that the Python client with the latest update to 2.6.2 is broken...

All kinds of modules probably required for httpx are seemingly not listed in the dependencies of confluent_kafka and when you import confluent_kafka, you get all kinds of errors like this:

ModuleNotFoundError: No module named 'httpx'
ModuleNotFoundError: No module named 'cachetools'
ModuleNotFoundError: No module named 'google.type'

etc.

How to reproduce

import confluent_kafka
from confluent_kafka.schema_registry.avro import AvroDeserializer
Traceback (most recent call last):
File "", line 1, in
File "/tmp/bla/venv/lib/python3.10/site-packages/confluent_kafka/schema_registry/init.py", line 20, in
from .schema_registry_client import (
File "/tmp/bla/venv/lib/python3.10/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 26, in
import httpx
ModuleNotFoundError: No module named 'httpx'
from confluent_kafka.schema_registry.json_schema import JSONDeserializer
Traceback (most recent call last):
File "", line 1, in
File "/tmp/bla/venv/lib/python3.10/site-packages/confluent_kafka/schema_registry/init.py", line 20, in
from .schema_registry_client import (
File "/tmp/bla/venv/lib/python3.10/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 26, in
import httpx
ModuleNotFoundError: No module named 'httpx'
from confluent_kafka.schema_registry.protobuf import ProtobufDeserializer
Traceback (most recent call last):
File "", line 1, in
File "/tmp/bla/venv/lib/python3.10/site-packages/confluent_kafka/schema_registry/init.py", line 20, in
from .schema_registry_client import (
File "/tmp/bla/venv/lib/python3.10/site-packages/confluent_kafka/schema_registry/schema_registry_client.py", line 26, in
import httpx
ModuleNotFoundError: No module named 'httpx'

...and if you install httpx, you get other missing dependencies...

I think these dependencies should be part of confluent_kafka and should not have to be installed manually by the user...

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.6.2
• Apache Kafka broker version: doesn't matter here
• Client configuration: doesn't matter here
• Operating system: doesn't matter here
• Provide client logs: see above
• Provide broker log excerpts: doesn't matter here
• Critical issue: I think so...

[ealap]

Same issue as #1879.

They suddenly decided on a patch version to make requirements for schema registry and other serializer/deserializers optional.

From updated README, you now have to install it as

pip install confluent-kafka[schemaregistry]

# or if you need one of serializers/deserializers like avro

pip install confluent-kafka[avro,schemaregistry]

[Nemeczek]

It actually crashed our Databricks jobs... It definately should no be smuggled in as a minor version patch

[scholer]

@Nemeczek - crashed some of our Databricks jobs as well (although not any production jobs, fortunately). We have previously allowed non-production jobs to not pin minor and/or patch versions for dependencies to allow for security updates. But I guess going forward we will use bots to update package dependencies as needed for these jobs as well.

[tribeiro]

We have a CI build that is failing the same way with version 2.8 of the library.

Installed with conda:

    python-confluent-kafka:           2.8.0-py311h9ecbd09_0           conda-forge      

Error:

../_test_env_placehold_placehold_placehold_p/lib/python3.11/site-packages/confluent_kafka/schema_registry/__init__.py:20: in <module>
    from .schema_registry_client import (
../_test_env_placehold_placehold_placehold_p/lib/python3.11/site-packages/confluent_kafka/schema_registry/schema_registry_client.py:26: in <module>
    import httpx
E   ModuleNotFoundError: No module named 'httpx'

[emasab]

Thanks for filing this issue, I want to point out that while the extra/optional dependencies were documented in 2.6.2, they were available in current form since 1.4.1.

Optional dependencies are inevitable for SR and serialization features as we don't want all users have to install for example the protobuf dependencies if they're only using avro, and they must install the third party dependencies through this Python feature.