refactor: Lazy import builders, embedders, (chat)generators, retrievers

[julian-risch]

Related Issues

• fixes Importing one component of a certain family/module leads to importing all components of the same family/module #8650

Proposed Changes:

• Add lazy imports for builders, embedders, (chat)generators, retrievers

How did you test it?

Notes for the reviewer

I just added the lazy import for a small selection of components in this draft. All components used in our "first RAG pipeline" tutorial are covered, which comes down to:

from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
from datasets import load_dataset
from haystack import Document
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import ChatPromptBuilder, AnswerBuilder
from haystack.dataclasses import ChatMessage
import os
from getpass import getpass
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack import Pipeline

Tests are failing with TypeErrors. We also need to check whether type hints or auto completion in IDEs are affected in a bad way by the lazy import. For example, when add an AnswerBuilder, I get two suggestions about importing either the class or the function AnswerBuilder

Here is a comparison first without and then with lazy imports. Created with

 python -X importtime 27_first_rag_pipeline.py 2> lazy-import.log
 tuna lazy-import.log

without lazy imports

with lazy imports

An alternative to lazy imports would be to require users to know and always use the full path of the component they want to import.
from haystack.components.generators.hugging_face_local import HuggingFaceLocalGenerator for example, instead of just from haystack.components.generators import HuggingFaceLocalGenerator

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I documented my code
• I ran pre-commit hooks and fixed any issue