[airbyte-ci] new commands for migration to base image (#30520)

airbyte-ci/connectors/pipelines/README.md

@@ -90,7 +90,9 @@ At this point you can run `airbyte-ci` commands from the root of the repository.
 - [`connectors publish` command](#connectors-publish-command)
 - [Examples](#examples)
 - [Options](#options-2)
-  * [What it runs](#what-it-runs-1)
+- [`connectors bump-version` command](#connectors-bump-version)
+- [`connectors upgrade-base-image` command](#connectors-upgrade-base-image)
+- [`connectors migrate-to-base-image` command](#connectors-migrate-to-base-image)
 - [`metadata` command subgroup](#metadata-command-subgroup)
 - [`metadata validate` command](#metadata-validate-command)
   * [Example](#example)
@@ -342,6 +344,48 @@ flowchart TD
     validate-->check-->build-->upload_spec-->push-->pull-->upload_metadata
 ```
 
+
+### <a id="connectors-bump-version"></a>`connectors bump-version` command
+Bump the version of the selected connectors.
+
+### Examples
+Bump source-openweather: `airbyte-ci connectors --name=source-openweather bump-version patch <pr-number> "<changelog-entry>"`
+
+#### Arguments
+| Argument              | Description                                                            |
+| --------------------- | ---------------------------------------------------------------------- |
+| `BUMP_TYPE`           | major, minor or patch                                                  |
+| `PULL_REQUEST_NUMBER` | The GitHub pull request number, used in the changelog entry            |
+| `CHANGELOG_ENTRY`     | The changelog entry that will get added to the connector documentation |
+
+### <a id="connectors-upgrade-base-image"></a>`connectors upgrade-base-image` command
+Modify the selected connector metadata to use the latest base image version.
+
+### Examples
+Upgrade the base image for source-openweather: `airbyte-ci connectors --name=source-openweather upgrade-base-image`
+
+### Options
+| Option                  | Required | Default | Mapped environment variable | Description                                                                                                     |
+| ----------------------- | -------- | ------- | --------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| `--docker-hub-username` | True     |         | `DOCKER_HUB_USERNAME`       | Your username to connect to DockerHub. It's used to read the base image registry.                               |
+| `--docker-hub-password` | True     |         | `DOCKER_HUB_PASSWORD`       | Your password to connect to DockerHub. It's used to read the base image registry.                               |
+| `--set-if-not-exists`   | False    | True    |                             | Whether to set or not the baseImage metadata if no connectorBuildOptions is declared in the connector metadata. |
+
+### <a id="connectors-migrate-to-base-image"></a>`connectors migrate-to-base-image` command
+Make a connector using a Dockerfile migrate to the base image by:
+* Removing its Dockerfile
+* Updating its metadata to use the latest base image version
+* Updating its documentation to explain the build process
+* Bumping by a patch version
+
+### Examples
+Migrate source-openweather to use the base image: `airbyte-ci connectors --name=source-openweather migrate-to-base-image`
+
+### Arguments
+| Argument              | Description                                                            |
+| --------------------- | ---------------------------------------------------------------------- |
+| `PULL_REQUEST_NUMBER` | The GitHub pull request number, used in the changelog entry            |
+
 ### <a id="metadata-validate-command-subgroup"></a>`metadata` command subgroup
 
 Available commands:
@@ -381,11 +425,12 @@ This command runs the Python tests for a airbyte-ci poetry package.
 ## Changelog
 | Version | PR                                                         | Description                                                                                               |
 | ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
-| 1.8.0   | [#31369](https://github.com/airbytehq/airbyte/pull/31369)  | Remove metadata test commands                                                                             |
-| 1.7.2   | [#31343](https://github.com/airbytehq/airbyte/pull/31343)  | Bind Pytest integration tests to a dockerhost.                                                                  |
-| 1.7.1   | [#31332](https://github.com/airbytehq/airbyte/pull/31332)  | Disable Gradle step caching on source-postgres.                                                                  |
-| 1.7.0   | [#30526](https://github.com/airbytehq/airbyte/pull/30526)  | Implement pre/post install hooks support.                                                                  |
-| 1.6.0   | [#30474](https://github.com/airbytehq/airbyte/pull/30474)  | Test connector inside their containers.                                                                  |
+| 1.9.0   | [#30520](https://github.com/airbytehq/airbyte/pull/30520)  | New commands: `bump-version`, `upgrade-base-image`, `migrate-to-base-image`.                              |
+| 1.8.0   | [#30520](https://github.com/airbytehq/airbyte/pull/30520)  | New commands: `bump-version`, `upgrade-base-image`, `migrate-to-base-image`.                              |
+| 1.7.2   | [#31343](https://github.com/airbytehq/airbyte/pull/31343)  | Bind Pytest integration tests to a dockerhost.                                                            |
+| 1.7.1   | [#31332](https://github.com/airbytehq/airbyte/pull/31332)  | Disable Gradle step caching on source-postgres.                                                           |
+| 1.7.0   | [#30526](https://github.com/airbytehq/airbyte/pull/30526)  | Implement pre/post install hooks support.                                                                 |
+| 1.6.0   | [#30474](https://github.com/airbytehq/airbyte/pull/30474)  | Test connector inside their containers.                                                                   |
 | 1.5.1   | [#31227](https://github.com/airbytehq/airbyte/pull/31227)  | Use python 3.11 in amazoncorretto-bazed gradle containers, run 'test' gradle task instead of 'check'.     |
 | 1.5.0   | [#30456](https://github.com/airbytehq/airbyte/pull/30456)  | Start building Python connectors using our base images.                                                   |
 | 1.4.6   | [ #31087](https://github.com/airbytehq/airbyte/pull/31087) | Throw error if airbyte-ci tools is out of date                                                            |

airbyte-ci/connectors/pipelines/pipelines/actions/environments.py

@@ -87,7 +87,7 @@ def with_testing_dependencies(context: PipelineContext) -> Container:
     )
 
 
-def with_git(dagger_client, ci_github_access_token_secret, ci_git_user) -> Container:
+def with_git(dagger_client, ci_git_user: str = "octavia") -> Container:
     return (
         dagger_client.container()
         .from_("alpine:latest")
@@ -102,7 +102,6 @@ def with_git(dagger_client, ci_github_access_token_secret, ci_git_user) -> Conta
                 ]
             )
         )
-        .with_secret_variable("GITHUB_TOKEN", ci_github_access_token_secret)
         .with_workdir("/ghcli")
         .with_exec(
             sh_dash_c(

airbyte-ci/connectors/pipelines/pipelines/bases.py

@@ -586,7 +586,7 @@ async def to_html(self) -> str:
     async def save(self) -> None:
         local_html_path = await self.save_local(self.html_report_file_name, await self.to_html())
         absolute_path = await local_html_path.resolve()
-        if self.pipeline_context.is_local:
+        if self.pipeline_context.open_report_in_browser:
             self.pipeline_context.logger.info(f"HTML report saved locally: {absolute_path}")
             if self.pipeline_context.open_report_in_browser:
                 self.pipeline_context.logger.info("Opening HTML report in browser.")

airbyte-ci/connectors/pipelines/pipelines/commands/groups/connectors.py

@@ -15,13 +15,20 @@
 from pipelines import main_logger
 from pipelines.bases import ConnectorWithModifiedFiles
 from pipelines.builds import run_connector_build_pipeline
+from pipelines.connector_changes.base_image_version_migration import (
+    run_connector_base_image_upgrade_pipeline,
+    run_connector_migration_to_base_image_pipeline,
+)
+from pipelines.connector_changes.version_bump import run_connector_version_bump_pipeline
 from pipelines.contexts import ConnectorContext, ContextState, PublishConnectorContext
-from pipelines.format import run_connectors_format_pipelines
+from pipelines.format import run_connector_format_pipeline
 from pipelines.github import update_global_commit_status_check_for_tests
 from pipelines.pipelines.connectors import run_connectors_pipelines
 from pipelines.publish import reorder_contexts, run_connector_publish_pipeline
 from pipelines.tests import run_connector_test_pipeline
 from pipelines.utils import DaggerPipelineCommand, get_connector_modified_files, get_modified_connectors
+from rich.table import Table
+from rich.text import Text
 
 # HELPERS
 
@@ -535,13 +542,188 @@ def format_code(ctx: click.Context) -> bool:
     ]
 
     anyio.run(
-        run_connectors_format_pipelines,
+        run_connectors_pipelines,
+        connectors_contexts,
+        run_connector_format_pipeline,
+        "Format connectors pipeline",
+        ctx.obj["concurrency"],
+        ctx.obj["dagger_logs_path"],
+        ctx.obj["execute_timeout"],
+    )
+
+    return True
+
+
+@connectors.command(cls=DaggerPipelineCommand, help="Upgrades the base image version used by the selected connectors..")
+@click.option("--set-if-not-exists", default=True)
+@click.option(
+    "--docker-hub-username",
+    help="Your username to connect to DockerHub to read the registries.",
+    type=click.STRING,
+    required=True,
+    envvar="DOCKER_HUB_USERNAME",
+)
+@click.option(
+    "--docker-hub-password",
+    help="Your password to connect to DockerHub to read the registries.",
+    type=click.STRING,
+    required=True,
+    envvar="DOCKER_HUB_PASSWORD",
+)
+@click.pass_context
+def upgrade_base_image(ctx: click.Context, set_if_not_exists: bool, docker_hub_username: str, docker_hub_password: str) -> bool:
+    """Upgrades the base image version used by the selected connectors."""
+
+    connectors_contexts = [
+        ConnectorContext(
+            pipeline_name=f"Upgrade base image versions of connector {connector.technical_name}",
+            connector=connector,
+            is_local=ctx.obj["is_local"],
+            git_branch=ctx.obj["git_branch"],
+            git_revision=ctx.obj["git_revision"],
+            ci_report_bucket=ctx.obj["ci_report_bucket_name"],
+            report_output_prefix=ctx.obj["report_output_prefix"],
+            use_remote_secrets=ctx.obj["use_remote_secrets"],
+            gha_workflow_run_url=ctx.obj.get("gha_workflow_run_url"),
+            dagger_logs_url=ctx.obj.get("dagger_logs_url"),
+            pipeline_start_timestamp=ctx.obj.get("pipeline_start_timestamp"),
+            ci_context=ctx.obj.get("ci_context"),
+            ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
+            ci_git_user=ctx.obj["ci_git_user"],
+            ci_github_access_token=ctx.obj["ci_github_access_token"],
+            open_report_in_browser=False,
+            docker_hub_username=docker_hub_username,
+            docker_hub_password=docker_hub_password,
+        )
+        for connector in ctx.obj["selected_connectors_with_modified_files"]
+    ]
+
+    anyio.run(
+        run_connectors_pipelines,
+        connectors_contexts,
+        run_connector_base_image_upgrade_pipeline,
+        "Upgrade base image pipeline",
+        ctx.obj["concurrency"],
+        ctx.obj["dagger_logs_path"],
+        ctx.obj["execute_timeout"],
+        set_if_not_exists,
+    )
+
+    return True
+
+
+@connectors.command(cls=DaggerPipelineCommand, help="Bump a connector version: update metadata.yaml and changelog.")
+@click.argument("bump-type", type=click.Choice(["patch", "minor", "major"]))
+@click.argument("pull-request-number", type=str)
+@click.argument("changelog-entry", type=str)
+@click.pass_context
+def bump_version(
+    ctx: click.Context,
+    bump_type: str,
+    pull_request_number: str,
+    changelog_entry: str,
+) -> bool:
+    """Bump a connector version: update metadata.yaml and changelog."""
+
+    connectors_contexts = [
+        ConnectorContext(
+            pipeline_name=f"Upgrade base image versions of connector {connector.technical_name}",
+            connector=connector,
+            is_local=ctx.obj["is_local"],
+            git_branch=ctx.obj["git_branch"],
+            git_revision=ctx.obj["git_revision"],
+            ci_report_bucket=ctx.obj["ci_report_bucket_name"],
+            report_output_prefix=ctx.obj["report_output_prefix"],
+            use_remote_secrets=ctx.obj["use_remote_secrets"],
+            gha_workflow_run_url=ctx.obj.get("gha_workflow_run_url"),
+            dagger_logs_url=ctx.obj.get("dagger_logs_url"),
+            pipeline_start_timestamp=ctx.obj.get("pipeline_start_timestamp"),
+            ci_context=ctx.obj.get("ci_context"),
+            ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
+            ci_git_user=ctx.obj["ci_git_user"],
+            ci_github_access_token=ctx.obj["ci_github_access_token"],
+            open_report_in_browser=False,
+        )
+        for connector in ctx.obj["selected_connectors_with_modified_files"]
+    ]
+
+    anyio.run(
+        run_connectors_pipelines,
         connectors_contexts,
-        ctx.obj["ci_git_user"],
-        ctx.obj["ci_github_access_token"],
-        ctx.obj["git_branch"],
-        ctx.obj["is_local"],
+        run_connector_version_bump_pipeline,
+        "Version bump pipeline pipeline",
+        ctx.obj["concurrency"],
+        ctx.obj["dagger_logs_path"],
+        ctx.obj["execute_timeout"],
+        bump_type,
+        changelog_entry,
+        pull_request_number,
+    )
+
+    return True
+
+
+@connectors.command(
+    cls=DaggerPipelineCommand,
+    help="Make the selected connectors use our base image: remove dockerfile, update metadata.yaml and update documentation.",
+)
+@click.argument("pull-request-number", type=str)
+@click.option(
+    "--docker-hub-username",
+    help="Your username to connect to DockerHub to read the registries.",
+    type=click.STRING,
+    required=True,
+    envvar="DOCKER_HUB_USERNAME",
+)
+@click.option(
+    "--docker-hub-password",
+    help="Your password to connect to DockerHub to read the registries.",
+    type=click.STRING,
+    required=True,
+    envvar="DOCKER_HUB_PASSWORD",
+)
+@click.pass_context
+def migrate_to_base_image(
+    ctx: click.Context,
+    pull_request_number: str,
+    docker_hub_username: str,
+    docker_hub_password: str,
+) -> bool:
+    """Bump a connector version: update metadata.yaml, changelog and delete legacy files."""
+
+    connectors_contexts = [
+        ConnectorContext(
+            pipeline_name=f"Upgrade base image versions of connector {connector.technical_name}",
+            connector=connector,
+            is_local=ctx.obj["is_local"],
+            git_branch=ctx.obj["git_branch"],
+            git_revision=ctx.obj["git_revision"],
+            ci_report_bucket=ctx.obj["ci_report_bucket_name"],
+            report_output_prefix=ctx.obj["report_output_prefix"],
+            use_remote_secrets=ctx.obj["use_remote_secrets"],
+            gha_workflow_run_url=ctx.obj.get("gha_workflow_run_url"),
+            dagger_logs_url=ctx.obj.get("dagger_logs_url"),
+            pipeline_start_timestamp=ctx.obj.get("pipeline_start_timestamp"),
+            ci_context=ctx.obj.get("ci_context"),
+            ci_gcs_credentials=ctx.obj["ci_gcs_credentials"],
+            ci_git_user=ctx.obj["ci_git_user"],
+            ci_github_access_token=ctx.obj["ci_github_access_token"],
+            open_report_in_browser=False,
+            docker_hub_username=docker_hub_username,
+            docker_hub_password=docker_hub_password,
+        )
+        for connector in ctx.obj["selected_connectors_with_modified_files"]
+    ]
+
+    anyio.run(
+        run_connectors_pipelines,
+        connectors_contexts,
+        run_connector_migration_to_base_image_pipeline,
+        "Migration to base image pipeline",
+        ctx.obj["concurrency"],
+        ctx.obj["dagger_logs_path"],
         ctx.obj["execute_timeout"],
+        pull_request_number,
     )
 
     return True

airbyte-ci/connectors/pipelines/pipelines/connector_changes/__init__.py

@@ -0,0 +1,3 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#