Add Lime CRM Destination

airbyte-integrations/connectors/destination-lime-crm/Dockerfile

@@ -0,0 +1,37 @@
+FROM python:3.11 AS base
+
+# build and load all requirements
+FROM base AS builder
+WORKDIR /airbyte/integration_code
+
+# upgrade pip to the latest version
+RUN apt-get update && apt-get -y upgrade \
+    && pip install --upgrade pip
+
+COPY pyproject.toml ./
+COPY README.md ./
+COPY main.py ./
+COPY destination_lime_crm ./destination_lime_crm
+
+
+# install necessary packages to a temporary folder
+RUN ls -l
+RUN pip install -v --prefix=/install .
+
+# build a clean environment
+FROM base
+WORKDIR /airbyte/integration_code
+
+# # copy all loaded and built libraries to a pure basic image
+COPY --from=builder /install /usr/local
+# # add default timezone settings
+COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
+RUN echo "Etc/UTC" > /etc/timezone
+
+COPY main.py ./
+
+ENV AIRBYTE_ENTRYPOINT="python /airbyte/integration_code/main.py"
+ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
+
+LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.name=airbyte/destination_lime_crm

airbyte-integrations/connectors/destination-lime-crm/README.md

@@ -0,0 +1,103 @@
+# Lime Crm Destination
+
+This is the repository for the Lime Crm destination connector, written in Python.
+For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/destinations/lime-crm).
+
+## Local development
+
+### Prerequisites
+
+* Python (`^3.9`)
+* Poetry (`^1.7`) - installation instructions [here](https://python-poetry.org/docs/#installation)
+
+
+
+### Installing the connector
+
+From this connector directory, run:
+```bash
+poetry install --with dev
+```
+
+
+#### Create credentials
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/destinations/lime-crm)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `destination_lime_crm/spec.json` file.
+Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
+See `integration_tests/sample_config.json` for a sample config file.
+
+**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `destination lime-crm test creds`
+and place them into `secrets/config.json`.
+
+### Locally running the connector
+```
+poetry run destination-lime-crm spec
+poetry run destination-lime-crm check --config secrets/config.json
+poetry run destination-lime-crm write --config secrets/config.json --catalog sample_files/configured_catalog.json
+```
+
+### Running tests
+
+To run tests locally, from the connector directory run:
+
+```
+poetry run pytest tests
+```
+
+### Building the docker image
+
+1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
+2. Run the following command to build the docker image:
+```bash
+airbyte-ci connectors --name=destination-lime-crm build
+```
+
+An image will be available on your host with the tag `airbyte/destination-lime-crm:dev`.
+
+### Running as a docker container
+
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/destination-lime-crm:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-lime-crm:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-lime-crm:dev write --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+### Running our CI test suite
+
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
+
+```bash
+airbyte-ci connectors --name=destination-lime-crm test
+```
+
+### Customizing acceptance Tests
+
+Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
+If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+
+### Dependency Management
+
+All of your dependencies should be managed via Poetry.
+To add a new dependency, run:
+
+```bash
+poetry add <package-name>
+```
+
+Please commit the changes to `pyproject.toml` and `poetry.lock` files.
+
+## Publishing a new version of the connector
+
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-lime-crm test`
+2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
+    - bump the `dockerImageTag` value in in `metadata.yaml`
+    - bump the `version` value in `pyproject.toml`
+3. Make sure the `metadata.yaml` content is up to date.
+4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/destinations/lime-crm.md`).
+5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
+6. Pat yourself on the back for being an awesome contributor.
+7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
+8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry.

airbyte-integrations/connectors/destination-lime-crm/destination_lime_crm/__init__.py

@@ -0,0 +1,8 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+
+from .destination import DestinationLimeCrm
+
+__all__ = ["DestinationLimeCrm"]

airbyte-integrations/connectors/destination-lime-crm/destination_lime_crm/destination.py

@@ -0,0 +1,90 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+import datetime
+import logging
+
+import requests
+
+import urllib.parse
+from typing import Any, Iterable, Mapping
+import uuid
+
+from airbyte_cdk.destinations import Destination
+from airbyte_cdk.models import AirbyteConnectionStatus, AirbyteMessage, ConfiguredAirbyteCatalog, Status, Type
+from collections import defaultdict
+
+logger = logging.getLogger(__name__)
+
+
+class DestinationLimeCrm(Destination):
+    def write(
+        self, config: Mapping[str, Any], configured_catalog: ConfiguredAirbyteCatalog, input_messages: Iterable[AirbyteMessage]
+    ) -> Iterable[AirbyteMessage]:
+        """
+        Reads the input stream of messages, config, and catalog to write data to the destination.
+
+        This method returns an iterable (typically a generator of AirbyteMessages via yield) containing state messages received
+        in the input message stream. Outputting a state message means that every AirbyteRecordMessage which came before it has been
+        successfully persisted to the destination. This is used to ensure fault tolerance in the case that a sync fails before fully completing,
+        then the source is given the last state message output from this method as the starting point of the next sync.
+
+        :param config: dict of JSON configuration matching the configuration declared in spec.json
+        :param configured_catalog: The Configured Catalog describing the schema of the data being received and how it should be persisted in the
+                                    destination
+        :param input_messages: The stream of input messages received from the source
+        :return: Iterable of AirbyteStateMessages wrapped in AirbyteMessage structs
+        """
+        streams = {s.stream.name for s in configured_catalog.streams}
+        logger.info(f"Starting write to Lime CRM with {len(configured_catalog.streams)} streams")
+
+        buffer = defaultdict(list)
+
+        for message in input_messages:
+
+            logger.info(f"Received message: {message}")
+
+            match message.type:
+                case Type.RECORD:
+                    logger.info(f"Received record: {message.record}")
+                    stream = message.record.stream
+                    if stream not in streams:
+                        logger.debug(f"Stream {stream} was not present in configured streams, skipping")
+                        continue
+                    buffer[stream].append((str(uuid.uuid4()), datetime.datetime.now().isoformat(), message))
+                case Type.STATE:
+                    logger.info(f"Received state: {message.state}")
+                    for stream_name in buffer.keys():
+                        logger.info(f"Flushing buffer for stream: {stream_name}")
+                        logger.info("TODO: Persisting records to Lime CRM...")
+                        for message in buffer[stream_name]:
+                            yield message
+                    buffer.clear()
+                case _:
+                    logger.info(f"Message type {message.type} not supported, skipping")
+
+            for stream_name in buffer.keys():
+                logger.info(f"Flushing remaining streams in buffer: {stream_name}")
+                logger.info("TODO: Persisting records to Lime CRM...")
+            buffer.clear()
+
+    def check(self, logger: logging.Logger, config: Mapping[str, Any]) -> AirbyteConnectionStatus:
+        """
+        Tests if the input configuration can be used to successfully connect to the destination with the needed permissions
+            e.g: if a provided API token or password can be used to connect and write to the destination.
+
+        :param logger: Logging object to display debug/info/error to the logs
+            (logs will not be accessible via airbyte UI if they are not passed to this logger)
+        :param config: Json object containing the configuration of this destination, content of this json is as specified in
+        the properties of the spec.json file
+
+        :return: AirbyteConnectionStatus indicating a Success or Failure
+        """
+        try:
+            url_root = config.get("url_root")
+            url = urllib.parse.urljoin(url_root, "api/v1/")
+            requests.get(url, timeout=3, headers={"x-api-key": config.get("api_key")}).raise_for_status()
+            return AirbyteConnectionStatus(status=Status.SUCCEEDED)
+        except Exception as e:
+            return AirbyteConnectionStatus(status=Status.FAILED, message=f"An exception occurred: {repr(e)}")

airbyte-integrations/connectors/destination-lime-crm/destination_lime_crm/run.py

@@ -0,0 +1,13 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+
+import sys
+
+from airbyte_cdk.entrypoint import launch
+from .destination import DestinationLimeCrm
+
+def run():
+    destination = DestinationLimeCrm()
+    destination.run(sys.argv[1:])

airbyte-integrations/connectors/destination-lime-crm/destination_lime_crm/spec.json

@@ -0,0 +1,25 @@
+{
+  "documentationUrl": "https://docs.airbyte.com/integrations/destinations/lime-crm",
+  "supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
+  "supportsIncremental": true,
+  "connectionSpecification": {
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "title": "Destination Lime Crm",
+    "type": "object",
+    "required": [],
+    "additionalProperties": false,
+    "properties": {
+      "api_key": {
+        "title": "API Key",
+        "type": "string",
+        "description": "API Key for Lime CRM",
+        "airbyte_secret": true
+      },
+      "url_root": {
+        "title": "URL",
+        "type": "string",
+        "description": "URL for Lime CRM"
+      }
+    }
+  }
+}

airbyte-integrations/connectors/destination-lime-crm/integration_tests/catalog.json

@@ -0,0 +1,36 @@
+{
+    "streams": [
+      {
+        "stream": {
+          "name": "airbyte_acceptance_table",
+          "json_schema": {
+            "type": "object",
+            "properties": {
+              "column1": {
+                "type": "string"
+              },
+              "column2": {
+                "type": "integer"
+              },
+              "column3": {
+                "type": "string",
+                "format": "date-time"
+              },
+              "column4": {
+                "type": "number"
+              },
+              "column5": {
+                "type": "array",
+                "items": {
+                  "type": ["integer", "null"]
+                }
+              }
+            }
+          },
+          "supported_sync_modes": ["full_refresh", "incremental"]
+        },
+        "sync_mode": "incremental",
+        "destination_sync_mode": "append"
+      }
+    ]
+  }

airbyte-integrations/connectors/destination-lime-crm/integration_tests/config.json

@@ -0,0 +1,4 @@
+{
+    "api_key": "TODO - ADD API KEY HERE!",
+    "url_root": "TODO"
+}

airbyte-integrations/connectors/destination-lime-crm/integration_tests/input_messages.jsonl

@@ -0,0 +1,9 @@
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type": "STATE", "state": {"data": {"last_synced_timestamp": "2022-06-20T18:56:18", "last_synced_id": 12345}}}  
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}
+{"type":"RECORD","record":{"stream":"airbyte_acceptance_table","emitted_at":1664705198575,"data":{"column1":"test","column2":222,"column3":"2022-06-20T18:56:18","column4":33.33,"column5":[1,2,null]}}}

airbyte-integrations/connectors/destination-lime-crm/integration_tests/integration_test.py

@@ -0,0 +1,63 @@
+#
+# Copyright (c) 2024 Airbyte, Inc., all rights reserved.
+#
+
+from destination_lime_crm import DestinationLimeCrm
+import json
+import pathlib
+import sys
+import logging
+from airbyte_cdk.models import (
+    ConfiguredAirbyteCatalog,
+    AirbyteMessage,
+)
+
+
+logger = logging.getLogger()
+
+# Ensure logs are printed to stdout
+handler = logging.StreamHandler(sys.stdout)
+logger.addHandler(handler)
+logger.setLevel(logging.INFO)
+
+
+def integration_test():
+    with open("integration_tests/config.json", "r") as file:
+        config = json.load(file)
+
+    if not pathlib.Path("integration_tests/secret_api_key").exists():
+        print("Please provide a integration_tests/secret_api_key file")
+        sys.exit(1)
+
+    with open("integration_tests/secret_api_key", "r") as file:
+        secret_api_key = file.read().strip()
+
+    config["api_key"] = secret_api_key
+
+    dest = DestinationLimeCrm()
+
+    input_messages, configured_catalog = None, None
+    with open("integration_tests/catalog.json", "r") as file:
+        configured_catalog = json.load(file)
+
+    with open("integration_tests/input_messages.jsonl", "r") as file:
+        input_messages = [x.strip() for x in file.readlines()]
+
+    configured_catalog = ConfiguredAirbyteCatalog.parse_obj(configured_catalog)
+    input_messages = (
+        AirbyteMessage.parse_obj(json.loads(x)) for x in input_messages
+    )
+
+    messages = dest.write(
+        config=config,
+        configured_catalog=configured_catalog,
+        input_messages=input_messages
+    )
+
+    [_ for _ in messages]
+
+    print("Integration test passed")
+
+
+if __name__ == "__main__":
+    integration_test()