-brightgreen){kind=icon}
-1,052-c0c0c4){kind=icon}
-10-c0c0c4){kind=icon}
target-elasticsearch is a Singer target for Elastic.
Some target development because I am currently hacking together something together with a bash script Built with the Meltano Tap SDK for Singer Taps.
| Setting | Required | Default | Description |
|---|---|---|---|
| scheme | True | http | http scheme used for connecting to elasticsearch |
| host | True | localhost | host used to connect to elasticsearch |
| port | True | 9200 | port use to connect to elasticsearch |
| username | False | None | basic auth username as per elastic documentation |
| password | False | None | basic auth password as per elastic documentation |
| bearer_token | False | None | Bearer token for bearer authorization as per elastic documentation |
| api_key_id | False | None | api key id for auth key authorization as per elastic documentation |
| api_key | False | None | api key for auth key authorization as per elastic documentation |
| ssl_ca_file | False | None | location of the the SSL certificate for cert verification ie. /some/path as per elastic documentation |
| index_format | False | ecs-{{ stream_name }}-{{ current_timestamp_daily }} | can be used to handle custom index formatting such as specifying -latest index. Default options: Daily {{ current_timestamp_daily }}, Monthly {{ current_timestamp_monthly }}, or Yearly {{ current_timestamp_yearly }}. You should use fields specified in index_schema_fields such as {{ _id }} or {{ timestamp }} . There are also helper fuctions such as {{ to_daily(timestamp) }}`. |
| index_schema_fields | False | None | this id map allows you to specify specific record values via jsonpath from the stream to be used in index formulation. |
| metadata_fields | false | None | this should be used to pull out specific fields via jsonpath to be used on for ecs metadata patters |
A full list of supported settings and capabilities is available by running: target-elasticsearch --about
git clone [email protected]:dtmirizzi/target_elasticsearch.git
cd target_elasticsearch
pipx install .A full list of supported settings and capabilities for this tap is available by running:
target_elasticsearch --aboutYou can easily run target-elasticsearch by itself or in a pipeline using Meltano.
There are 4 types of auth supported by this target:
- Basic Auth
- Access Token
- Bearer Token
- SSL Based
target_elasticsearch --version
target_elasticsearch --help
target_elasticsearch --config CONFIG --discover > ./catalog.jsonpipx install poetry
poetry install
poetry run pre-commit install- Create tests within the
target_elasticsearch/testssubfolder - To Run tests, first set environment variables in your current shell for all required settings from the Settings section above. Example for UNIX base systems running
export Target_elasticsearch_USER_ID="xxxxx"then run:
poetry run pytestYou can also test the target-elasticsearch CLI interface directly using poetry run:
poetry run target_elasticsearch --helpTesting with Meltano
Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Your project comes with a custom meltano.yml project file already created.
Next, install Meltano (if you haven't already) and any needed plugins:
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd target-elasticsearch
meltano install# You can spin up elastic locally
make local-esNow you can test and orchestrate using Meltano:
meltano install
# Test invocation:
meltano invoke target-elasticsearch --version
# test configuration
meltano config target-elasticsearch set --interactive
# OR run a test `elt` pipeline:
meltano elt tap-smoke-test target-elasticsearchSee the dev guide for more instructions on how to use the SDK to develop your own taps and targets.