-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: Centralize JSON SerDe into helper functions #2259
refactor: Centralize JSON SerDe into helper functions #2259
Conversation
CodSpeed Performance ReportMerging #2259 will not alter performanceComparing Summary
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2259 +/- ##
==========================================
+ Coverage 89.23% 89.41% +0.18%
==========================================
Files 54 58 +4
Lines 4782 4779 -3
Branches 935 933 -2
==========================================
+ Hits 4267 4273 +6
+ Misses 359 353 -6
+ Partials 156 153 -3 ☔ View full report in Codecov by Sentry. |
@edgarrmondragon when you get a chance, please take a look and let me know what you think of this idea. I was working on the |
ed8f39f
to
b9faa7f
Compare
So we currently have this $ rg -. serde singer_sdk
singer_sdk/helpers/_flattening.py
13:from singer_sdk._singerlib.serde import serialize_json
singer_sdk/contrib/batch_encoder_jsonl.py
9:from singer_sdk._singerlib.serde import serialize_json
singer_sdk/_singerlib/messages.py
11:from singer_sdk._singerlib.serde import serialize_json
singer_sdk/tap_base.py
13:from singer_sdk._singerlib.serde import serialize_json
singer_sdk/io_base.py
14:from singer_sdk._singerlib.serde import deserialize_json
singer_sdk/connectors/sql.py
15:from singer_sdk._singerlib import CatalogEntry, MetadataMapping, Schema, serde
1167: return serde.serialize_json(obj)
1183: return serde.deserialize_json(json_str)
singer_sdk/sinks/core.py
19:from singer_sdk._singerlib.serde import deserialize_json And I think not all of these need to factor out SerDe. In particular, The others, I'm not sure currently:
These four cases still architecturally benefit from having a centralized approach that was missing previously, and it was causing inconsistent handling of whitespace and decimals. |
804537f
to
dc5c7a1
Compare
ff38602
to
96525f1
Compare
This way we can catch something like `msgspec.DecodeError` and raise the more generic `InvalidInputLine`.
946a0cc
to
5d98422
Compare
5d98422
to
8f390f6
Compare
20a75cd
to
a5f67a5
Compare
These are the only breaking API changes in this PR according to griffe:
I think these are acceptable. The plugin class methods weren't really removed, but the superclass now lives in the private |
Thanks @BuzzCutNorman! |
I was wondering if it made sense to move all the serialize and deserialize functions into a helper library that we could reference in other areas. I thought
singer_sdk.helper._util
could be a place for it.