Support Spatial Types for PostGIS #1927
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Pipboyguy
added
enhancement
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
# Conflicts: # poetry.lock # pyproject.toml
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
burnash
reviewed
Nov 1, 2024
burnash
reviewed
Nov 1, 2024
There was a problem hiding this comment.
Thanks for the PR, @Pipboyguy
I have a couple of questions, please see my comments.
# Conflicts: # .github/workflows/test_doc_snippets.yml # poetry.lock
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
rudolfix
requested changes
Nov 25, 2024
| ) and not file_path.endswith("insert_values"): | ||
| # Only insert_values load jobs supported for geom types. | ||
| # TODO: This isn't actually true, can make it work with geoarrow! | ||
| raise TerminalValueError( |
There was a problem hiding this comment.
is it really true? if column is text and contains WKT data it should get inserted. did you try it?
There was a problem hiding this comment.
I added csv loader cases to the geo test, and it all passes. Your intuition was entirely correct!
dlt/common/data_writers/escape.py
Outdated
| @@ -56,7 +67,12 @@ def escape_postgres_literal(v: Any) -> Any: | |||
| if isinstance(v, (list, dict)): | |||
| return _escape_extended(json.dumps(v)) | |||
| if isinstance(v, bytes): | |||
| return f"'\\x{v.hex()}'" | |||
| if is_valid_wkb(v): | |||
There was a problem hiding this comment.
this is very impractical :/ we have a dependence on shapely and there may be false positives (there are no magic numbers in wkb). passing expected type to escape...literal will slow this down
we could use a special binary type but user would need to wrap binary anyway. so my take is to drop wkb and just keep wkb_hex.
There was a problem hiding this comment.
Agreed, removed wkb support, will update docs too
# Conflicts: # poetry.lock
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Signed-off-by: Marcel Coetzee <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds support for PostGIS GEOMETRY type in PostgreSQL. It introduces a logical type for GEOMETRY on binary or text fields.
Users can declare geometry columns via the
postgres_adapter, enabling direct insertion from WKT, WKB and WKB hex representations, in addition to GeoPandas integration with automatic geometry column detection and conversion.Related Issues
Limitations
LinearRingonly works with wkb type (python binary type)Typical usage pattern: