feat: allow vectorizer to read documents via smart-open parse via pymupdf #431
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chunking needs a special logic for document processing, given that we're no longer expecting a column name that should exist in the source table, but what's going to be chunked is the content of the file stored in the cloud object storage. Co-authored-by: Sergio Moya <[email protected]>
fixes some tests while working on the new chunking strategy. Co-authored-by: Adol Rodriguez <[email protected]>
Co-authored-by: Adol Rodriguez <[email protected]>
Co-authored-by: Sergio Moya <[email protected]>
pgai v0.8.0 was released so it's time to rebase this branch Co-authored-by: Sergio Moya <[email protected]>
Co-authored-by: Adol Rodriguez <[email protected]>
disable ollama tests and include new content on expected files. remove unrequired asserts from vectorizer test and add statement timeout to speed feedback on transient local failures.
…orage and parse it
Askir
force-pushed
the
jascha/vectorizer-s3-document
branch
from
577957e to
220aafa
Compare
Askir
force-pushed
the
jascha/vectorizer-s3-document
branch
from
9af8756 to
f0b2e2b
Compare
Askir
force-pushed
the
jascha/vectorizer-s3-document
branch
from
f0b2e2b to
f364ca6
Compare
cevian
reviewed
Feb 5, 2025
Comment on lines
+771
to
780
| if self.vectorizer.config.loader is not None: | ||
| file_content = self.vectorizer.config.loader.load(item) | ||
| markdown_content = self.vectorizer.config.parser.parse_file_content( | ||
| file_content | ||
| ) | ||
| chunks = self.vectorizer.config.chunking.into_chunks(markdown_content) | ||
| else: | ||
| chunks = self.vectorizer.config.chunking.into_chunks(item) | ||
| for chunk_id, chunk in enumerate(chunks, 0): | ||
| formatted = self.vectorizer.config.formatting.format(chunk, item) |
There was a problem hiding this comment.
I think it may be best to conceive of this as a processing pipeline where most processing nodes alter the row
Suggested change
| if self.vectorizer.config.loader is not None: | |
| file_content = self.vectorizer.config.loader.load(item) | |
| markdown_content = self.vectorizer.config.parser.parse_file_content( | |
| file_content | |
| ) | |
| chunks = self.vectorizer.config.chunking.into_chunks(markdown_content) | |
| else: | |
| chunks = self.vectorizer.config.chunking.into_chunks(item) | |
| for chunk_id, chunk in enumerate(chunks, 0): | |
| formatted = self.vectorizer.config.formatting.format(chunk, item) | |
| if self.vectorizer.config.loader is not None: | |
| # loader reads a "url" column and loads binary data into item['file_contents'] | |
| self.vectorizer.config.loader.load(item) | |
| if self.vectorizer.config.parser is not None: | |
| # reads some column (default: file_contents) and puts result in item['parsed_file_content'] | |
| self.vectorizer.config.parser.parse_file_content(item) | |
| chunks = self.vectorizer.config.chunking.into_chunks(item) | |
| for chunk_id, chunk in enumerate(chunks, 0): | |
| formatted = self.vectorizer.config.formatting.format(chunk, item) | |
| records_without_embeddings.append(pk + [chunk_id, formatted]) |
jgpruitt
requested changes
Feb 5, 2025
Comment on lines
+2
to
+3
| -- loader_file_loader | ||
| create or replace function ai.loader_from_document |
There was a problem hiding this comment.
naming considerations:
- our other configs for vectorizer end in
ing:embedding,formatting,chunking. Should we follow the same convention? - would
object_storagebe more appropriate thandocument? We should describe "where" instead of "what". We may load documents from other places in the future
Comment on lines
+2
to
+3
| -- parser_auto | ||
| create or replace function ai.parser_auto() returns pg_catalog.jsonb |
There was a problem hiding this comment.
parsing vs parser to follow the existing convention?
| @@ -2,7 +2,7 @@ | |||
| ------------------------------------------------------------------------------- | |||
| -- chunking_character_text_splitter | |||
| create or replace function ai.chunking_character_text_splitter | |||
| ( chunk_column pg_catalog.name | |||
| ( chunk_column pg_catalog.name default '' | |||
There was a problem hiding this comment.
i think this column will go away completely when we have a loader_row configuration, but in case it doesn't the default should be null rather than ''
|
Closing in favor of the WIP work in #442 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses our agreed upon API spec number 4.
Works end to end for local files and s3 (credential loading also works, see tests)
What's missing?