All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- #1410 Tap template: fix style, types and imports; and also catch more errors by building from replay files in CI -- Thanks @flexponsive!
- #1428 Tap template: cover all REST authentication cases, and one GraphQL case -- Thanks @flexponsive!
- #1365 Add
strptime_to_utcandstrftimefunctions to_singerlib.utils-- Thanks @menzenski! - #1394 Refactor SQLConnector connection handling -- Thanks @qbatten!
- #1241 Support declaring variant for use in package name
- #1109 Support
requests.authauthenticators
- #1380 Move tests in cookiecutters to project root to support
pytest_plugins - #1406 Use a version of
isortcompatible with Python 3.8 - #1385 SQL Targets ignore collation when evaluating column data types -- Thanks @BuzzCutNorman!
- #1342 Remove SQLSink snakecase conform in favor of simpler transformations
- #1364 TapDiscoveryTest remove catalog if one is passed
- #1390 Add incremental replication example -- Thanks @flexponsive!
- #1171 Improve included tap and target tests in
singer_sdk.testing
- #1345 Remove tox dependency from tap/target template
- #1358 Fix typo in
if __name__ ==example
- #1283 Automatic catalog selection of replication keys
- #1335 Stream maps example for adding property with hardcoded string value
- #1308 Replace hyphens with underscores when generating expected env var name
<PLUGIN_NAME>_LOGLEVEL-- Thanks @adherr! - #887 Make
conform_record_data_typeswork on nested objects and arrays -- Thanks @Jack-Burnett! - #1287 Targets to fail gracefully when schema message is missing the
propertieskey -- Thanks @visch!
- #1262 Support string
"__NULL__"whereever null values are allowed in stream maps configuration
- #1281 Apply Version bump commit file perms with sudo
- #1280 Set repo file perms after checkout in Version bump workflow
- #1214 Avoid duplicate entries in
requiredarray of JSON schema helpers
- #1188 Support boolean
additional_propertiesin JSON schema helper objects - #1237 Catch and retry
ConnectionResetErrorexceptions in HTTP taps - #1087 S3 batch storage -- Thanks @jamielxcarter!
- #1197 Support
patternPropertiesin JSON schema helpers - #1157 Built-in handling of
default-target-schemafor SQL Targets -- Thanks @BuzzCutNorman!
- #1238 Ensure metric tags coming from stream context can be JSON-serialized
- #1233 Add level and logger name to default log format
- #1219 Schema passthrough for whitelisted fields
- #1174 Do not emit log message if no record properties were ignored
- #1192 Change max record age for emitting state messages to 5 instead of 30 mins -- Thanks @spacecowboy!
- #1196 Improve performance of record message serialization -- Thanks @Jack-Burnett!
- #1243 Document inherited
PluginBaseattributes and methods - #1209 Fix argument descriptions for
OAuthAuthenticator
- #1175 Add
datetimefunctions to simpleeval env in stream maps -- Thanks @qbatten!
- #1182 Update
SQLConnectorimport for SQL target cookiecutter -- Thanks @radbrt! - #1168
SQLConnector.table_exists()to use separatetable_nameandschema_nameinstead of fully qualified name -- Thanks @BuzzCutNorman! - #1164 Write a valid final state message at the end of each stream sync -- Thanks @laurentS!
- #1114 Make DDL overridable for column
ADD,ALTER, andRENAMEoperations
- #1126 Resolve failure in
_increment_stream_state()for cases whenreplication_methodisLOG_BASED - #1111 Push
_MAX_RECORDS_LIMITdown into SQL
- #1091 Move SQLConnector into a separate module, for use by both SQLStream and SQLSink
- #1098 Add JSON Schema
Propertyhelpers forallowed_values(enum) andexamples - #1096 Add secrets support for tap and target config, via
Property(..., secret=True) - #1039 Support conforming singer property names to target identifier constraints in SQL sinks
- #1093 Add environment support to the cookie cutter for
meltano.yml - #1036 Create schema and table on
add_sink
- #1032 Support stream property selection push-down in SQL streams
- #978 Allow configuring a dedicated metrics logger
- #1043 Batch storage
split_urlto work with Windows paths -- Thanks @BuzzCutNorman! - #826 Remove Poetry version pin for GitHub Actions -- Thanks @visch!
- #1001 Use column name in
allow_column_altererror message
- #1060 Add explanation and recommendations for context usage
- #1074 Document an example implementation and usage of
BaseHATEOASPaginator - #1020 Fixed typo in
docs/stream_maps.md-- Thanks @spacecowboy! - #1006 Add links to Meltano install/tut
- #999 Absolute file paths created by taps running in BATCH mode can't be processed by the Sink
- Change
targetUrlof semantic PR title check to point to SDK docs
- #968 Added cookiecutter support and docs for VSCode debugging
- #904 Add support for new
BATCHmessage type in taps and targets
- #972 Resolve issue where TypeError is thrown by SQLConnector cookiecutter implementation due to super() references
- #829 Add checks for primary keys, replication keys and state partitioning keys to standard tap tests -- Thanks @laurentS!
- #732 Implement reference paginators.
- #898 Fix SQL type merging for pre-existing target tables -- Thanks @BuzzCutNorman!
- #856 Fix typo RecordsWitoutSchemaException -> RecordsWithoutSchemaException.
- Use
__future__.annotationsonsinger_sdk.helpers._singer
- #950 Document missing initializers for authentication and pagination helpers.
- #947 Remove stale autodoc page for RecordsWitoutSchemaException.
- #942 Add docs preview links to PR description.
- #842 Allow authenticating more generic requests
- #919 add
ConnectionErrorto list of backoff exceptions for auto-retry
- #917 Allow Singer schemas to include the
requiredandenumfields - #759 Use recent start_date as starting_replication_value
- #908 Allow overriding the bulk insert statement in
SQLSink
- #784 Update return type for
backoff_max_triesto reflect it accepts a callable that returns an integer - #874 Singer metrics are now properly emitted in JSON format --Thanks, @Jack-Burnett!
- #869 Cleanup whitespace in backoff code samples
- #785 Output full URL path in error messages
- #815 Generate correct SQL target project from cookiecutter
- #782 Allow lists and dictionaries as types for default JSON values
- #823 Add link to the sdk for README generation regarding Stream Maps
- #813 Fix PyPI trove classifiers
- #783 Document using pipx inject for nox-poetry
- Fix missing typing-extensions for Python<3.10 (#776)
- Tap and Target SDK: Adds the ability to override the logging level via
LOGLEVELenvironment variables. (!300) - Thanks, Eric Boucher! - Tap SDK: Allow sort checking to be disabled for taps with un-sortable replication keys. (#729, #730) - Thanks, Ilkka!
- Tap SDK: Add
Stream.calculate_sync_costhook to gather "cost" (in terms of number of requests, etc.) for each stream. (#348, #704) - Thanks, Laurent Savaëte! - Target SDK: Add end-of-pipe clean up hooks for
Sinkimplementations (#750) - Thanks, Alexander Butler!
- Tap SDK: Bump minimum version of
PyJWTto2.4(#396, !302). - Tap SDK: Compare replication key values when incrementing state, only if the
check_sortedflag is set to True (#736, #737).
- Tap and Target SDK: Resolves an issue where SQLAlchemy logs would be printed to
STDOUT. (!303) - Thanks, Andrey Tatarinov! - Tap SDK: Reduces number of log lines printed when unmapped properties are ignored from the source API (!300) - Thanks, Eric Boucher!
- Tap and Target SDK: Remove trailing parenthesis from logged version (#766, #767).
- Tap and Target SDK: The
--config=ENVoption now also considers environment variables from a dotenv (.env) file if present (#384, !293).
-
Target SDK: Use generic SQLALchemy markers for parameterized queries in SQL targets (#376, !287) - Thanks, Thomas Briggs!
-
Target SDK: Explicitly specify column names when inserting rows in SQL targets (#385, !294) - Thanks, Thomas Briggs!
-
Target SDK: Fixes bug where an
activate_versionmessage caused an error when destination table does not exist (#372, !285) - Thanks, Thomas Briggs! -
Tap and Target SDK: Do not fail
--aboutoption if tap or target is not configured (#379, !291).
- Tap SDK: Improved helpers for handling rate limits, backoff and retries (#137, #140, !277) - Thanks, Fred O'Loughlin!
- Remove "errors" from non-error log line to improve log searchability (!279) - Thanks, Laurent Savaëte!
- Tap and Target SDK: Fixed a bug where setting stream map property to
nulldid not remove it from SCHEMA message (#370, !286) - Thanks, Ryan Whitten! - Tap and Target SDK: Fixed a bug where flattening resulted in an invalid SCHEMA message (!286) - Thanks, Ryan Whitten!
- Target SDK: Use
maxLengthin SQL targets for string fields if the schema provides it (#371, !284) - Thanks, Thomas Briggs!
- Tap Cookiecutter: Fixed a bug where the wrong key was used to select the appropriate
Tapclass for SQL taps (#353, !275) - Thanks, Dan Norman! - Tap Cookiecutter: Fixed a bug where
client.pyfor SQL taps was overwritten during initialization (#354, !276) - Thanks, Dan Norman! - Tap SDK: Fixed a bug where a parent stream emitted schema messages when it's not selected, but at least one of its child streams is (#366, !280)
- Tap SDK: Bump
pyjwtdependency to~=2.3(!281) - Thanks, Eric Boucher!
- Raise more descriptive exceptions when wrapped JSON typing classes needs to be instantiated (#55, #360, !270).
- Support JSONPath extensions in
records_jsonpathandnext_page_token_jsonpath(#361, !271).
- Fixed a bug where setting a stream map property to null did not remove the property (#352, !263).
- Avoid SQLAlchemy deprecation warning caused by using
Engine.has_table()(#341, !264) - Resolve issue where "falsey" defaults like '0', '', and 'False' would not be properly applied to tap settings config. (#357, !265)
- Return to stable
poetry-coreversion in cookiecutter templates (#338, !260)
- Define all built-in JSON Schema string formats as separate types (#336, !250) - Thanks, Reuben Frankel!
- Enable JSONPath for GraphQL record extraction (#327, !247) - Thanks, Fred O'Loughlin!
- Add record and schema flattening in Stream Maps (!236),
- Resolve issues when aliasing stream maps using the keywords
__alias__,__source__, or__else__(#301, #302, !243)
- Always sync one record per stream when invoking with
--testor--test=all(#311, !241) - Add
--test=schemaoption to emit tap SCHEMA messages only (!218) - Thanks, Laurent Savaëte!
- Add support for SQL Taps (#74, !44)
- Add support for SQL Targets (#263, !44, !200, !239)
- Added Licence tracking to SDK GitLab Project (#166, !237)
- Inline Mapper SDK: Support for creation of inline mapper plugins (#257, !234)
- Tap and Target Cookiecutter: Support editable pip installs (
pip install -e .) with PEP 660 (#238, !231)
- Tap Cookiecutter: Add output directory for
target-jsonl(!228) -- Thanks, Niall Woodward! - Tap SDK: Make the
expires_inproperty optional in OAuth response (#297, !232) -- Thanks, Daniel Ferguson!
- Tap SDK: Add configurable timeout for HTTP requests (#287, !217, !225) -- Thanks, Josh Lloyd!
- Tap and Target SDK: Adds support for Python 3.10 (#293, !224)
- Resolve lint errors when ArrayType is used to wrap other types (!223) -- Thanks, David Wallace!
- Tap SDK: Fix datelike type parsing bug with nested schemas (#283, !219)
- Tap SDK: Resolved bug in
--testwhich caused child streams to not use record limiting (#268, !204, !220) -- Thanks, Derek Visch!
- Tap SDK: Fixed mapped
__key_properties__not being passed to the emitted schema message (#281, !209) - Tap SDK: Fixed missing schema during development causing sync to fail #284, !212 -- Thanks, Fred Reimer!
- Tap SDK: New method
RESTStream.validate_responsefor custom validation of HTTP responses (#207, !195) - Tap SDK: New method
RESTStream.request_decoratorfor custom back-off and retry parameters (#137, !195)
- Target SDK: Document options for the target CLI and accept multiple config files as input (!183)
- Target SDK: CLI flag for targets to read messages from a file instead of stdin (#249, !190) -- Thanks, Charles Julian Knight!
- Target SDK: Add target mock classes and tap-to-target scenario tests (#198, !138)
- Tap and Target SDK: Create expanded list of capabilities (#186, !141)
- Tap and Target SDK: Fix markdown table formatting in
--aboutfor multi-line settings descriptions (#240, !185) - Tap SDK: Clarify undocumented feature of filtering
Nonefrompost_process()(#233, !187) - Tap and Target SDK: Add
dataclassesas an explicit third-party dependency for Python 3.6 (#245, !189) - Tap and Target SDK: Allows
--discoverand--aboutexecution without requiring settings validation (#235, !188)
- Tap and Target SDK: Adds capability to print markdown docs with
--about --format=markdown(!172, !180) -- Thanks, Nick Müller!
- Tap and Target SDK: Autogenerated docstrings for arguments, return types, and exceptions raised (!166).
- Tap and Target SDK: Support Black by default by bumping min Python version to 3.6.2. (#224, !169)
- Fixes a bug where tox invocations after initial setup failed (!179) -- Thanks, Jon Watson!.
- Tap SDK: Fixes a bug in
Stream.get_starting_timestamp()andStream.get_starting_replication_key_value()calls where results where not cached breaking stream sorting (!157)
- Tap and Target SDK: Prevents the leaking of sensitive configuration values when JSON schema validation fails (!173) -- Thanks, Kevin Mullins!.
- Add description attribute to
Propertyclass for JSON schemas (#159, !164) -- Thanks, Stephen Bailey!
- Tap SDK: Set
key_properties = []instead ofnullper the Singer spec (!160) -- Thanks, Niall Woodward!
- Tap SDK: Fixes issue where stream map schema generation fails when overriding the value of an existing property. (#196, !165)
- Tap and Target SDK: Resolves
2to3compatibility issues when installed withsetuptools>=58.0. - Resolve issue preventing repo from being cloned on Windows.
- Tap and Target SDK: Added compatibility with PEP 561 and
mypytype checking (#212, !150) -- Thanks, Laurent Savaëte!
- Tap SDK: Improved record parsing and validation performance, especially with large record objects (#161, !146)
- Tap SDK: Changed the signature of
Stream.apply_catalogto reflect new catalog parsing flow (#161, !146)
- Tap and Target SDK: Adds support for Python 3.9 (#66, !38)
- Tap SDK: Added support for new authenticator classes:
BasicAuthenticator,BearerTokenAuthenticator, andAPIKeyAuthenticator(#185, !128) -- Thanks, Stephen Bailey!
- Tap and Target SDK: Bumps
clicklibrary version to 8.0 (#178, !140). - Target SDK: Improves
BatchSinkperformance by reducing the frequency by which batches are processed. (#172, !137)
- Tap SDK: Improves CLI
--helpoutput (#177, !140). - Tap SDK: Fixes a bug in state tracking where timezone-aware timestamps are appended again with
+0:00(#176, !142) -- Thanks, Joshua Adeyemi! - Tap SDK: Improve performance by reusing a single authenticator instance (#168, #173, !136)
- Tap SDK: Fixed a bug where not using a catalog file resulted in all streams being selected but all properties being removed from the schema and records (#190, !132)
- Tap SDK: Added full support for selection metadata as specified by the Singer Spec, including metadata for
selected,selected-by-default, andinclusion(!121)
- Target SDK: Improved performance for Batch Sinks by skipping extra drain operations when newly recieved STATE messages are unchanged from the prior received STATE (#172, !125) -- Thanks, Pat Nadolny!
- Target SDK: Fixed a bug where would emit an invalid STATE payload (#188, !130) -- Thanks, Pat Nadolny!
- Tap SDK: Fixed a bug where replication key signposts were not correctly applied for streams which defined them (#180, !129)
- Added JSONPath for taps to handle record extraction from API responses (!77)
- Resolve batch
contextnot being reset between batches (#164, !117)
- Removed unused
DEFAULT_BATCH_SIZE_ROWSin favor ofmax_sizeforBatchSinkimplementations (#163, !118)
- Resolve stream map duplicates not aliased correctly (#154, !114)
- Added target support for
add_record_metadataconfig (#157, !111)
- Resolve target failures when dates are parsed prior to JSON Schema validation (#156, !110)
- Resolve target failures when
default_sink_classis not used (#153, !109) - Improved tap log messages when child property's selection metadata defaults to the parent's (#131, !91)
- Added Stream Map feature for inline transformation and filtering capabilities (#63, !4, !92, !103)
- Added Target SDK, components and templates for building Singer targets (#96, !4)
- Removed methods deprecated in v0.2.0:
get_partition_state()andget_stream_or_partition_state(). Affected developers should replace these with references toget_context_state(). (#152, !107)
- Added support for parent-child streams (#97, !79)
- Added support for configurable metrics logging (#91, !79)
- Added ability to use fewer state bookmarks by setting
Stream.state_partitioning_keysto a subset of available context keys (!79)
- Renamed the optional
partitiondictionary arg in method signatures to the more genericcontext(!79)
- The methods
Stream.get_partition_state()andStream.get_stream_or_partition_state()have been deprecated in favor of the new and simplerget_context_state()(!79)
- Code coverage is now tracked and available as a tool for SDK contributors to further improve overall stability and help prioritize unit test development. (#39, !89)
Stability and bugfix release. No breaking changes.
- Resolved excessive logging during selection filtering. (#125, !83)
- Resolved issue where deselected sub-fields were incorrectly included in stream records. (#126, !85) -- Thanks, Alex Levene!
- Added improved type hints for developers, including mypy code compliance for improved stability. (#127, !86)
Bugfix release. No breaking changes.
- Resolved tap failure when a sorted stream has non-unique replication keys. (#120, !82)
Significant release with newly added features. No breaking changes.
- Added support for GraphQL query variables (#115, !78)
- Added selection rules support for record and schema messages (#7, !26)
- Improved cookiecutter template coverage, resolved readability issues. (#116, #119, !75)
- Resolved tap failure when a stream is missing from the input catalog. (#105, !80)
- Resolved bug where unsorted streams did not properly advance state bookmarks for incremental streams. (#118, !74)
Significant release with newly added features. No breaking changes.
- Added
is_sortedstream property, which enables long-running incremental streams to be resumed if interrupted. (!61) - Added signpost feature to prevent bookmarks from advancing beyond the point where all records have been streamed. (!61)
- Added
get_replication_key_signpost()stream method which defaults to the current time for timestamp-based replication keys. (!61)
- Fixed a scenario where unsorted incremental streams would generate incorrect STATE bookmarks. (!61) -- Thanks, Egi Gjevori!
- Fixed a problem where CI pipelines would fail when run from a fork. (!71) -- Thanks, Derek Visch!
- Fixed fatal error when running from the cookiecutter shell script (#102, !64)
Fixes bug in state handling, adds improvements to documentation.
- Streamlined Dev Guide (!56)
- Added Code Samples page, including dynamic schema discovery examples (#33, !56)
- Added links to external sdk-based taps (#32, !56)
- Added static/dynamic property documentation (#86, !56)
- Added "implementation" docs for debugging and troubleshooting (#71, !41)
- Fixes bug in
Stream.get_starting_timestamp()using incorrect state key (#94, !58)
Documentation and cookiecutter template improvements.
- Added 'admin_name' field in cookiecutter, streamline poetry setup (!25)
- Added meltano integration and testing options (#47, !52)
- Added new cookiecutter
.shscript to ease testing during development (!52)
- Improved cookiecutter readme template with examples (#76, !53)
First official SDK release. Numerous changes and improvements implemented, with the goal of stabilizing the SDK and making it broadly available to the community.
- Added this CHANGELOG.md file (#68, !43)
- Added standardized tap tests (!36, #78, !46)
- Added SDK testing matrix for python versions 3.6, 3.7, 3.8 (#61, !33)
- Added support for multiple
--config=inputs, combining one or more config.json files (#53, !27) - Added new CLI
--testoption to perform connection test on all defined streams (#14, !28) - Added default value support for plugin configs (!12) -- Contributed by: Ken Payne
- Promote
singer_sdk.helpers.typingtosinger_sdk.typing(#84) - Modified environment variable parsing logic for arrays (#82)
- Renamed
http_headersinAuthenticatorclass toauth_headers(#75, !47) - Expect environment variables in all caps (
<PLUGIN>_<SETTING>) (#59, !34) - Parse environment variables only if
--config=ENVis passed (#53, !27)
- OAuth no longer applies
client_emailautomatically ifclient_idis missing (#83) - Resolved issue on Python 3.6:
cannot import 'metadata' from 'importlib'(#58) - Fixed issue reading from JSON file (!11) -- Contributed by: Edgar R. Mondragón
- Look only for valid plugin settings in environment variables (!21) -- Contributed by: Edgar R. Mondragón
- Fixed bug in
STATEhandling (!13) -- Contributed by: Ken Payne
- Remove parquet sample (#81,!48)
Initial prerelease version for review and prototyping.