Tap-Airbyte-Wrapper

tap-airbyte is a Singer tap that wraps all Airbyte sources implicitly. This adds over 250 immediately usable extractors to the broader Singer ecosystem. This opens up high quality connectors for an expanded audience further democratizing ELT and encourages contributions upstream where system experts using an airbyte source via this wrapper may be inclined to contribute to the connector source, open issues, etc if it is the better option than what's available in the Singer catalog alone.

Built with the Meltano Tap SDK for Singer Taps.

Configuration 📝

Setting	Required	Default	Description
airbyte_spec	True	None	Specification for the Airbyte source connector. This is a JSON object minimally containing the image key. The tag key is optional and defaults to latest.
airbyte_config	False	None	Configuration to pass through to the Airbyte source connector, this can be gleaned by running the the tap with the --about flag and the --config flag pointing to a file containing the airbyte_spec configuration. This is a JSON object.
docker_mounts	False	None	Docker mounts to mount to the container. Expects a list of maps containing source, target, and type as is documented in the docker --mount documentation
stream_maps	False	None	Config object for stream maps capability. For more information check out Stream Maps.
stream_map_config	False	None	User-defined config values to be used within map expressions.
flattening_enabled	False	None	'True' to enable schema flattening and automatically expand nested properties.
flattening_max_depth	False	None	The max depth to flatten schemas.

Configure using environment variables ✏️

OCI_RUNTIME can be set to override the default of docker. This lets the tap work with podman, nerdctl, colima, and so on.

OCI_RUNTIME=nerdctl meltano run tap-pokeapi target-jsonl

This Singer tap will automatically import any environment variables within the working directory's .env if the --config=ENV is provided, such that config values will be considered if a matching environment variable is set either in the terminal context or in the .env file.

Source Authentication and Authorization 👮🏽‍♂️

First, configure your tap by creating a configuration json file. In this example we will call it github.json since this tap may use many configurations for different sources.

❗️ Remember the required keys for airbyte_config can be dumped to stdout by running tap-airbyte --about --config /path/to/FILE where FILE minimally contains just the airbyte_spec.image value

{
  "airbyte_spec": {
    "image": "source-github"
  },
  "airbyte_config": {
    "credentials": {
      "access_token": "..."
    },
    "repositories": "z3z1ma/*",
  }
}

Run the built in Airbyte connection test to validate your configuration like this where github.json represents the above config (note the choice of file name is purely for illustration):

tap-airbyte --config ./github.json --test

The --test flag will validate your configuration as being able to access the configured data source! Be sure to use it. With meltano, configuration is implicitly passed based on what's in your meltano.yml configuration which simplifies it to just meltano invoke tap-airbyte --test

See more configuration examples in the sync tests

Usage 👷‍♀️

You can easily run tap-airbyte by itself or in a pipeline using Meltano.

Executing the Tap Directly 🔨

tap-airbyte --version
tap-airbyte --help
tap-airbyte --config CONFIG --discover > ./catalog.json

Developer Resources 👩🏼‍💻

Follow these instructions to contribute to this project.

Initialize your Development Environment

pipx install poetry
poetry install

Create and Run Tests 🧪

Create tests within the tap_airbyte/tests subfolder and then run:

poetry run pytest

You can also test the tap-airbyte CLI interface directly using poetry run:

poetry run tap-airbyte --help

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-airbyte
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-airbyte --version
# OR run a test `elt` pipeline:
meltano elt tap-airbyte target-jsonl

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.