Skip to content

1.8
Compare
Choose a tag to compare
@csun-cpointe csun-cpointe released this 05 Aug 19:31
· 255 commits to dev since this release
aissemble-root-1.8.0
4fab51d

Major Additions

Python Code Linter

We incorporated PyLint as the approach to linting Python modules to detect errors. This will enable developers to identify and resolve errors during a project build. By default, PyLint checkers classfied as errors are flagged and can be configured through Habushu's configuration.

There is a known issue with using PyLint and importing modules from the Behave package that will require a modification to Python test scripts. For existing projects, you will need to change the imports by:

- from behave import *
+ from behave import given, when, then  # pylint: disable=no-name-in-module

Transition from Orphedomos to Fabric8

aiSSEMBLE has changed its Docker management plugin from the orphedomos-maven-plugin to Fabric8's docker-maven-plugin. This will enable developers to leverage a Docker management tool that is maintained and supported by a broader community.

Helm Migration

The following Helm charts have been migrated to the v2 structure. To migrate your helm charts to use the v2 pattern, please follow the instruction in the technical documentation.

  • Spark Operator (projects created after 1.2 will have this chart by default)

Breaking Changes

Note: instructions for adapting to these changes are outlined in the upgrade instructions below.

  • Transitioning from orphedomos to fabric8 may result in breaking changes, see the Upgrade Steps for Projects with Customized Orphedomos Configurations for further details.
  • The following docker modules have been deprecated. You need to continue to use the 1.7.0 versions rather than
    upgrade them. Please note, these will only be breaking IF you try to use the 1.8.0 versions, which would require manual
    intervention. The helm charts referencing these containers have already been updated to use 1.7.0.
    • aissemble-airflow
    • aissemble-kafka
    • aissemble-mlflow
  • Upgraded Pydantic v1.10.x to 2.8.x to incorporate performance improvements and incorporate the availability of new features for future. See here for the guide on how to migrate pydantic V1 into V2.

Known Issues

  • There is currently a bug with the Sagemaker training Docker image generated by the aissemble-sagemaker-training-docker Fermenter profile. The installation of the logistic-training module's requirements.txt fails, due to an unresolvable set of dependencies.
  • Running a downstream project build with -Pintegration-test is currently broken due to incompatibilities that have remained from before the open-sourcing.
  • If you have updated your project to include the docker registry in image names when built locally, the orphedomos-to-fabric8-migration may remove the registry. It can simply be added back without issue.
  • ArgoCD apps fail to utilize the current deploy job branch parameter, see the Upgrade Steps for ArgoCD Branch Deployment below for resolution.
  • For project deploying the Spark Operator V2 chart with ArgoCd, there is an issue with metadata being too long resulting in the CRD failing to deploy. See the Upgrade Steps for Spark Operator v2 ArgoCD CRD Deployment section below for resolution.

Known Vulnerabilities

Date
identified		 Vulnerability Severity Package Affected
versions		 CVE Fixed
in

How to Upgrade

The following steps will upgrade your project to 1.8. These instructions consist of multiple phases:

  • Automatic Upgrades - no manual action required
  • Precondition Steps - needed in all situations
  • Conditional Steps (e.g., Python steps, Java steps, if you use Metadata, etc)
  • Final Steps - needed in all situations

Automatic Upgrades

To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the build-parent version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

Migration Name Description
upgrade-tiltfile-aissemble-version-migration Updates the aiSSEMBLE version within your project's Tiltfile
upgrade-v2-chart-files-aissemble-version-migration Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
upgrade-v1-chart-files-aissemble-version-migration Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE
python-linting-migration Updates Habushu configuration in root pom.xml to disable build failures when linting issues are detected
orphedomos-to-fabric8-migration Updates orphedomos-maven-plugin usages to leverage fabric8's docker-maven-plugin
aiops-reference-python-migration Updates the python packages which were renamed from aiops to aissemble.
aiops-reference-pdp-python-migration Updates the policy decision point python packages which were renamed from aiops to aissemble.
aiops-reference-java-migration Updates the java packages which were renamed from aiops to aissemble.
add-fabric8-to-build-migration Ensures all Docker modules have the Docker Maven plugin defined in project/build/plugins.
fabric8-location-migration For all aggregator projects (packaging type pom), moves plugin definitions for the Docker Maven plugin from build/plugins to build/pluginManagement/plugins.

To deactivate any of these migrations, add the following configuration to the baton-maven-plugin within your root pom.xml:

    <plugin>
        <groupId>org.technologybrewery.baton</groupId>
        <artifactId>baton-maven-plugin</artifactId>
        <dependencies>
            <dependency>
                <groupId>com.boozallen.aissemble</groupId>
                <artifactId>foundation-upgrade</artifactId>
                <version>${version.aissemble}</version>
            </dependency>
        </dependencies>
+        <configuration>
+             <deactivateMigrations>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+                 <deactivateMigration>NAME_OF_MIGRATION</deactivateMigration>
+             </deactivateMigrations>
+        </configuration>
    </plugin>

Precondition Steps - Required for All Projects

Beginning the Upgrade

To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.8.0 version of the build-parent:

<parent>
    <groupId>com.boozallen.aissemble</groupId>
    <artifactId>build-parent</artifactId>
    <version>1.8.2</version>
</parent>

Conditional Steps

Upgrade Steps for Projects Leveraging aiSSEMBLE Inference Pipeline

With the enablement of linting, there is a known issue with the Pylint package when linting on Python modules using the Pydantic package or Python files generated by Protobuff. As a result, when linting is enabled, the project build will error.

To resolve this error, within the <project-name>-pipelines/<pipeline-name>/<inference-step-name>/pyproject.toml file, add the configuration:

[tool.pylint.'MESSAGES CONTROL']
extension-pkg-whitelist = "pydantic"

[tool.pylint.MASTER]
ignore-patterns = '.*pb2[\S]*.py'

Upgrade Steps for Projects with Customized Orphedomos Configurations

If any extra configurations were added to the orphedomos-maven-plugin in addition to the generated defaults, executing the migration will result in loss of these extra configurations. To facilitate the upgrade, the following steps should be taken:

  1. Before executing the baton migration, it is recommended to ensure some form of version control is in place to preserve your existing orphedomos-maven-plugin configurations.
  2. After executing the baton migration, please see the docker-maven-plugin docs to add back any extra configurations, via corresponding configuration analogs.

Steps for Projects leveraging Pipeline Invocation Service

For projects that downgraded the Pipeline Invocation Service to 1.6.1 to workaround the known issue with the 1.7.0 service, the workaround can be removed to upgrade the service to 1.8.0.

  1. In the root pom.xml, the following baton migrations can be removed from the deactivateMigrations list:

    <deactivateMigrations>
    <deactivateMigration>upgrade-v2-chart-files-aissemble-version-migration</deactivateMigration>
    <deactivateMigration>upgrade-v1-chart-files-aissemble-version-migration</deactivateMigration>
    <deactivateMigration>upgrade-helm-chart-names-migration</deactivateMigration>
</deactivateMigrations>

  2. Update the pipeline-invocation-service Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy). Replace the following values and names accordingly:

    Current Value Required Value
    aissemble-pipeline-invocation aissemble-pipeline-invocation-chart
    aissemble-pipeline-invocation-lib aissemble-pipeline-invocation-lib-chart
    aissemble-quarkus aissemble-quarkus-chart
    https://nexus.boozallencsn.com/repository/aiops-helm-internal/ oci://ghcr.io/boozallen

Upgrade Steps for ArgoCD Branch Deployment

Due to the issues with ArgoCD apps not respecting the deploy job branch parameter, any test deployments completed prior to merging into dev will fail to include modifications made to your helm charts. To ensure ArgoCD is using the correct branch when deploying, add the following to your argocdDeploy method within devops/jenkinsPipelineSteps.groovy:

def argocdDeploy(argocdAppName, argocdUrl, argocdDestinationServer, gitRepo, argocdBranch, argocdNamespace, values) {
    String valuesParam = "--values " + values.join(" --values ")
    sh "/usr/local/bin/argocd app create ${argocdAppName} --grpc-web \
            --server ${argocdUrl} \
            --dest-namespace ${argocdNamespace} \
            --dest-server ${argocdDestinationServer} \
            --repo ${gitRepo} \
            --path video-processing-deploy/src/main/resources --revision ${argocdBranch} \
-           ${valuesParam}"
+           ${valuesParam} \
+           --helm-set spec.targetRevision=${argocdBranch}"

Upgrade Steps for Spark Operator v2 ArgoCD CRD Deployment

When ArgoCD deploys a resource, it utilizes a client-side kubectl apply which injects an annotation containing a complete copy of the YAML being deployed. This results in the metadata being too long for the Spark Operator v2 CRD. To resolve this, you can instruct ArgoCD to use a server side deployment instead by adding the following syncPolicy to YOUR-PROJECT-deploy/src/main/resources/templates/spark-operator.yaml

spec:
+  syncPolicy:
+   syncOptions:
+      - ServerSideApply=true

Final Steps - Required for All Projects

Finalizing the Upgrade

  1. Run ./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate to apply the automatic migrations
  2. Run ./mvnw clean install and resolve any manual actions that are suggested
    • NOTE: This will update any aiSSEMBLE dependencies in 'pyproject.toml' files automatically
  3. Repeat the previous step until all manual actions are resolved

What's Changed

New Contributors

Full Changelog: aissemble-root-1.7.0...aissemble-root-1.8.0

Contributors

peter-mcclonski, liangyun123, and 10 other contributors
Assets 2
Loading