Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FIX] Install correct clip dep for llama-index-embeddings-clip #10759

Merged
merged 4 commits into from
Feb 15, 2024
Merged

[FIX] Install correct clip dep for llama-index-embeddings-clip #10759

merged 4 commits into from
Feb 15, 2024

Conversation

nerdai
Copy link
Contributor

@nerdai nerdai commented Feb 15, 2024 •
edited
Loading

Description

  • Previously installed the incorrect "clip" package
  • This PR installs the correct one
  • Version has been patched, and published to pypi

Fixes #10756

Type of Change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • I stared at the code and made sure it makes sense

@dosubot dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Feb 15, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 15, 2024
@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Feb 15, 2024
@nerdai nerdai enabled auto-merge (squash) February 15, 2024 19:21
@nerdai nerdai merged commit 036ff28 into main Feb 15, 2024
8 checks passed
@nerdai nerdai deleted the nerdai/install-correct-clip-library branch February 15, 2024 19:27
humpydonkey added a commit to humpydonkey/llama_index that referenced this pull request Feb 19, 2024
@humpydonkey
[FIX]Fix a NameError for llama-index-embeddings-clip
0f0ca48 
A small bug fix for a NameError: `name 'torch' is not defined`, a patch for run-llama#10759

The fix re-imports the `torch` module at the local function scope.
We don't need a try-except as the constructor already checks it.

Detailed stack trace
```
NameError                                 Traceback (most recent call last)
Cell In[11], line 2
      1 documents = image_documents
----> 2 index = MultiModalVectorStoreIndex.from_documents(
      3     documents,
      4     storage_context=storage_context,
      5 )

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/base.py:142, in BaseIndex.from_documents(cls, documents, storage_context, show_progress, callback_manager, transformations, service_context, **kwargs)
    133     docstore.set_document_hash(doc.get_doc_id(), doc.hash)
    135 nodes = run_transformations(
    136     documents,  # type: ignore
    137     transformations,
    138     show_progress=show_progress,
    139     **kwargs,
    140 )
--> 142 return cls(
    143     nodes=nodes,
    144     storage_context=storage_context,
    145     callback_manager=callback_manager,
    146     show_progress=show_progress,
    147     transformations=transformations,
    148     service_context=service_context,
    149     **kwargs,
    150 )

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/multi_modal/base.py:101, in MultiModalVectorStoreIndex.__init__(self, nodes, index_struct, embed_model, storage_context, use_async, store_nodes_override, show_progress, image_vector_store, image_embed_model, is_image_to_text, is_image_vector_store_empty, is_text_vector_store_empty, service_context, **kwargs)
     97     storage_context.add_vector_store(SimpleVectorStore(), self.image_namespace)
     99 self._image_vector_store = storage_context.vector_stores[self.image_namespace]
--> 101 super().__init__(
    102     nodes=nodes,
    103     index_struct=index_struct,
    104     embed_model=embed_model,
    105     service_context=service_context,
    106     storage_context=storage_context,
    107     show_progress=show_progress,
    108     use_async=use_async,
    109     store_nodes_override=store_nodes_override,
    110     **kwargs,
    111 )

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py:74, in VectorStoreIndex.__init__(self, nodes, use_async, store_nodes_override, embed_model, insert_batch_size, objects, index_struct, storage_context, callback_manager, transformations, show_progress, service_context, **kwargs)
     67 self._embed_model = (
     68     resolve_embed_model(embed_model, callback_manager=callback_manager)
     69     if embed_model
     70     else embed_model_from_settings_or_context(Settings, service_context)
     71 )
     73 self._insert_batch_size = insert_batch_size
---> 74 super().__init__(
     75     nodes=nodes,
     76     index_struct=index_struct,
     77     service_context=service_context,
     78     storage_context=storage_context,
     79     show_progress=show_progress,
     80     objects=objects,
     81     callback_manager=callback_manager,
     82     transformations=transformations,
     83     **kwargs,
     84 )

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/base.py:91, in BaseIndex.__init__(self, nodes, objects, index_struct, storage_context, callback_manager, transformations, show_progress, service_context, **kwargs)
     89 if index_struct is None:
     90     nodes = nodes or []
---> 91     index_struct = self.build_index_from_nodes(
     92         nodes + objects  # type: ignore
     93     )
     94 self._index_struct = index_struct
     95 self._storage_context.index_store.add_index_struct(self._index_struct)

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py:307, in VectorStoreIndex.build_index_from_nodes(self, nodes, **insert_kwargs)
    299 if any(
    300     node.get_content(metadata_mode=MetadataMode.EMBED) == "" for node in nodes
    301 ):
    302     raise ValueError(
    303         "Cannot build index from nodes with no content. "
    304         "Please ensure all nodes have content."
    305     )
--> 307 return self._build_index_from_nodes(nodes, **insert_kwargs)

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py:279, in VectorStoreIndex._build_index_from_nodes(self, nodes, **insert_kwargs)
    277     run_async_tasks(tasks)
    278 else:
--> 279     self._add_nodes_to_index(
    280         index_struct,
    281         nodes,
    282         show_progress=self._show_progress,
    283         **insert_kwargs,
    284     )
    285 return index_struct

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/multi_modal/base.py:395, in MultiModalVectorStoreIndex._add_nodes_to_index(self, index_struct, nodes, show_progress, **insert_kwargs)
    390     self._is_text_vector_store_empty = True
    392 if len(image_nodes) > 0:
    393     # embed image nodes as images directly
    394     # check if we should use text embedding for images instead of default
--> 395     image_nodes = self._get_node_with_embedding(
    396         image_nodes,
    397         show_progress,
    398         is_image=True,
    399     )
    400     new_img_ids = self.storage_context.vector_stores[self.image_namespace].add(
    401         image_nodes, **insert_kwargs
    402     )
    403 else:

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/multi_modal/base.py:210, in MultiModalVectorStoreIndex._get_node_with_embedding(self, nodes, show_progress, is_image)
    207 id_to_text_embed_map = None
    209 if is_image:
--> 210     id_to_embed_map = embed_image_nodes(
    211         nodes,
    212         embed_model=self._image_embed_model,
    213         show_progress=show_progress,
    214     )
    216     # text field is populate, so embed them
    217     if self._is_image_to_text:

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/indices/utils.py:173, in embed_image_nodes(nodes, embed_model, show_progress)
    170     else:
    171         id_to_embed_map[node.node_id] = node.embedding
--> 173 new_embeddings = embed_model.get_image_embedding_batch(
    174     images_to_embed, show_progress=show_progress
    175 )
    177 for new_id, img_embedding in zip(ids_to_embed, new_embeddings):
    178     id_to_embed_map[new_id] = img_embedding

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/embeddings/multi_modal_base.py:119, in MultiModalEmbedding.get_image_embedding_batch(self, img_file_paths, show_progress)
    110 if (
    111     idx == len(img_file_paths) - 1
    112     or len(cur_batch) == self.embed_batch_size
    113 ):
    114     # flush
    115     with self.callback_manager.event(
    116         CBEventType.EMBEDDING,
    117         payload={EventPayload.SERIALIZED: self.to_dict()},
    118     ) as event:
--> 119         embeddings = self._get_image_embeddings(cur_batch)
    120         result_embeddings.extend(embeddings)
    121         event.on_end(
    122             payload={
    123                 EventPayload.CHUNKS: cur_batch,
    124                 EventPayload.EMBEDDINGS: embeddings,
    125             },
    126         )

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/embeddings/multi_modal_base.py:76, in MultiModalEmbedding._get_image_embeddings(self, img_file_paths)
     70 """
     71 Embed the input sequence of image synchronously.
     72 
     73 Subclasses can implement this method if batch queries are supported.
     74 """
     75 # Default implementation just loops over _get_image_embedding
---> 76 return [
     77     self._get_image_embedding(img_file_path) for img_file_path in img_file_paths
     78 ]

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/core/embeddings/multi_modal_base.py:77, in <listcomp>(.0)
     70 """
     71 Embed the input sequence of image synchronously.
     72 
     73 Subclasses can implement this method if batch queries are supported.
     74 """
     75 # Default implementation just loops over _get_image_embedding
     76 return [
---> 77     self._get_image_embedding(img_file_path) for img_file_path in img_file_paths
     78 ]

File ~/miniconda3/envs/zillow/lib/python3.10/site-packages/llama_index/embeddings/clip/base.py:134, in ClipEmbedding._get_image_embedding(self, img_file_path)
    133 def _get_image_embedding(self, img_file_path: ImageType) -> Embedding:
--> 134     with torch.no_grad():
    135         image = (
    136             self._preprocess(Image.open(img_file_path))
    137             .unsqueeze(0)
    138             .to(self._device)
    139         )
    140         return self._model.encode_image(image).tolist()[0]

NameError: name 'torch' is not defined
```
humpydonkey added a commit to humpydonkey/llama_index that referenced this pull request Feb 19, 2024
@humpydonkey
[FIX]Fix a NameError for llama-index-embeddings-clip
88f27bf 
A small bug fix for a NameError: `name 'torch' is not defined`.

We could also import torch lazily. But there are  

This is likely introduced by  run-llama#10759
@humpydonkey humpydonkey mentioned this pull request Feb 19, 2024
Merged
15 tasks
Dominastorm pushed a commit to uptrain-ai/llama_index that referenced this pull request Feb 28, 2024
@nerdai @Dominastorm
[FIX] Install correct clip dep for llama-index-embeddings-clip (run-l…
578850d 
…lama#10759)

* remove wrong clip; add correct one

* CHANGELOG

* remove clip as can't publish pypi packwage with git dep

* remove clip as can't publish pypi packwage with git dep
Izukimat pushed a commit to Izukimat/llama_index that referenced this pull request Mar 29, 2024
@nerdai
[FIX] Install correct clip dep for llama-index-embeddings-clip (run-l…
1ba9e78 
…lama#10759)

* remove wrong clip; add correct one

* CHANGELOG

* remove clip as can't publish pypi packwage with git dep

* remove clip as can't publish pypi packwage with git dep
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hatianzhang hatianzhang hatianzhang approved these changes

@logan-markewich logan-markewich Awaiting requested review from logan-markewich

Assignees
No one assigned
Labels
lgtm This PR has been approved by a maintainer size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Installs wrong CLIP library
2 participants
@nerdai @hatianzhang