2. Quick start

Navigation:

Syndicate

AWS-Syndicate is an Amazon Web Services deployment framework written in Python. It facilitates the deployment of serverless applications using resource descriptions.

2.1 Installing the Framework

Prerequisites

Python 3.10 version;
Package manager PIP 22.0 or higher version;
Apache Maven 3.3.9 or higher version (for Java projects).
SDK 8.0.402 or higher version (for .NET projects).
npm 11.1.0 or higher version (for NodeJS projects).

Installation from PyPI (RECOMMENDED)

pip install aws-syndicate

Install aws-syndicate from PyPI

Note: In case you've encountered difficulties with installation via PyPI, you can try to install AWS-syndicate from the source code below

Installation from source code

Installation on macOS

Detailed guide how to install Python you can find here. Also here you can find detailed guild how to install the latest Apache Maven. .NET installation guide can be found here, it is needed if you plan to work with runtime .NET.

Pull the project

git clone https://github.com/epam/aws-syndicate.git

Create virtual environment:

python3 -m venv syndicate_venv

Activate your virtual environment:

source syndicate_venv/bin/activate

Install Syndicate framework with pip from GitHub:

pip3 install aws-syndicate/.

Set up a Syndicate Java plugin:

mvn install -f aws-syndicate/plugin/

Go to the 2. Usage guide

Installation on Linux

Detailed guide how to install Python you can find here. Also here you can find detailed guild how to install the latest Apache Maven. .NET installation guide can be found here, it is needed if you plan to work with runtime .NET.

Pull the project

git clone https://github.com/epam/aws-syndicate.git

Create virtual environment:

python3 -m venv syndicate_venv

Activate your virtual environment:

source syndicate_venv/bin/activate

Install Syndicate framework with pip from GitHub:

pip3 install aws-syndicate/.

Set up a Syndicate Java plugin:

mvn install -f aws-syndicate/plugin/

Go to the 2. Usage guide

Installation on Windows

Detailed guide how to install Python you can find here. Also here you can find detailed guild how to install the latest Apache Maven. .NET installation guide can be found here, it is needed if you plan to work with runtime .NET.

Pull the project

git clone https://github.com/epam/aws-syndicate.git

Create virtual environment:

python3 -m venv syndicate_venv

Activate your virtual environment:

syndicate_venv\Scripts\activate.bat

Install Syndicate framework with pip from GitHub:

pip3 install aws-syndicate/.

Set up a Syndicate Java plugin:

mvn install -f aws-syndicate/plugin/

2.2 Usage guide

2.2.1 Creating Project files

Execute syndicate generate project command to generate the project with all the necessary components and in a right folders/files hierarchy to start developing in a min. Command example:

syndicate generate project 
    --name $project_name
    --path $path_to_project

All the provided information is validated. After the project folder will be generated the command will return the following message:

    Project name: $project_name
    Project path: $path_to_project

The following files will be created in this folder: .gitignore, .syndicate, CHANGELOG.md, deployment_resources.json, README.md.

Command sample:

syndicate generate project --name DemoSyndicateJava && cd DemoSyndicateJava

CLICK HERE if you work with Java Project

After creating a new Java project, reopen it separately in your IDE to ensure all imports and dependencies are correctly configured.

For more details please execute syndicate generate project --help

2.2.2 Creating configuration files for environment

Execute the syndicate generate config command to create Syndicate configuration files. Command example:

syndicate generate config
    --name                      $configuration_name   [required]
    --region                    $region_name          [required]
    --bundle_bucket_name        $s3_bucket_name       [required]
    --access_key                $access_key  
    --secret_key                $secret_key   
    --config_path               $path_to_store_config
    --project_path              $relative_path_to_project
    --prefix                    $prefix
    --suffix                    $suffix
    --extended_prefix           $extended_prefix_mode
    --use_temp_creds            $use_temp_creds #Specify,if use mfa or access_role
    --access_role               $role_name
    --serial_number             $serial_number
    --tags                      $KEY:VALUE
    --iam_permissions_boundary  $ARN

All the provided information is validated.

Note: you may not specify --access_key and --secret_key params. In this case Syndicate will try to find your credentials by the path ~/.aws or from environment variables.

Note: You can force Syndicate to generate temporary credentials and use them for deployment. For such cases, set use_temp_creds parameter to True and specify serial number if IAM user which will be used for deployment has a policy that requires MFA authentication. Syndicate will prompt you to enter MFA code to generate new credentials, save and use them until expiration.

Setup the SDCT_CONF environment variable pointing to the folder with syndicate.yml file

After the configuration files will be generated the command will return the following message:

 Syndicate initialization has been completed. 
 Set SDCT_CONF:
 Unix: export SDCT_CONF=$path_to_store_config
 Windows: setx SDCT_CONF $path_to_store_config

Just copy one of the last two lines, depending on your OS, and execute the command. The commands set the environment variable SDCT_CONF required by aws-syndicate to operate.

Note: The default syndicate_aliases.yaml file has been generated. Your application may require additional aliases to be deployed - please add them to the file.

Command sample:

SYNDICATE_AWS_ACCESS_KEY=# enter your aws_access_key_id here
SYNDICATE_AWS_SECRET_KEY=# enter your aws_secret_access_key here
syndicate generate config --name dev --region eu-central-1 --bundle_bucket_name syndicate-artifacts-eu-central-1 --access_key $SYNDICATE_AWS_ACCESS_KEY --secret_key $SYNDICATE_AWS_SECRET_KEY --config_path $(pwd) --prefix syn- --suffix -dev --tags ENV:DEV

For more details please execute syndicate generate config --help

2.2.3 Extended Prefix Mode

Extended Prefix Mode offers a straightforward way to manage the naming conventions of resources during deployment.

You can enable this feature when setting up your project configuration by using the syndicate generate config command.

This command allows you to add a consistent prefix and/or suffix to all resources created during deployment by specifying --extended_prefix True along with --prefix and --suffix options.

Alternatively, you can manually add these settings to your project's configuration file, syndicate.yml:

extended_prefix_mode: true
resources_prefix: demo-
resources_suffix: -test

With these settings, every resource that Syndicate deploys will have its name prefixed with demo- and suffixed with -test. This feature is especially useful for maintaining clarity and consistency in environments where multiple development stages coexist, or where specific naming rules are required for organizational or regulatory compliance.

2.2.4 Creating lambda files

Execute syndicate generate lambda command to generate required environment for lambda function except business logic. Command example:

syndicate generate lambda
    --name $lambda_name_1
    --runtime python|java|nodejs|dotnet
    --project_path $project_path

All the provided information is validated. Different environments will be created for different runtimes:

for Python

    .
    ├── $project_path
    │   └── src
    │       ├── commons
    │       │   ├── __init__.py
    │       │   ├── abstract_lambda.py
    │       │   ├── exception.py
    │       │   └── log_helper.py
    │       └── lambdas
    │           ├── $lambda_name_1
    │           │   ├── __init__.py
    │           │   ├── deployment_resources.json
    │           │   ├── handler.py
    │           │   ├── lambda_config.json
    │           │   ├── local_requirements.txt
    │           │   └── requirements.txt
    │           ├── $lambda_name_2
    │           │   ├── __init__.py
    │           │   ├── deployment_resources.json
    │           │   ├── handler.py
    │           │   ├── lambda_config.json
    │           │   ├── local_requirements.txt
    │           │   └── requirements.txt
    │           ├── __init__.py
    │           └── ...
    └── ...

for Java

    .
    ├── $project_path
    │   └── jsrc
    │       └── main
    │           └── java
    │               └── com
    │                   └── $projectpath
    │                       ├── $lambda_name_1.java
    │                       └── $lambda_name_2.java
    └── ...

for NodeJS

    .
    ├── $project_path
    │   └── app
    │       └── lambdas
    │           ├── $lambda_name_1
    │           │   ├── deployment_resources.json
    │           │   ├── lambda_config.json
    │           │   ├── index.js
    │           │   ├── package.json
    │           │   └── package-lock.json
    │           └── $lambda_name_2
    │               ├── deployment_resources.json
    │               ├── lambda_config.json
    │               ├── index.js
    │               ├── package.json
    │               └── package-lock.json
    └── ...

for .NET

   .
    ├── $project_path
    │   └── dnapp
    │       └── lambdas
    │           └── $lambda_name
    │               ├── deployment_resources.json
    │               ├── lambda_config.json
    │               ├── Function.cs
    │               └── Function.csproj
    └── ...

Command sample:

syndicate generate lambda --name DemoLambda --runtime java --project_path $(pwd)

For more details please execute syndicate generate lambda --help

CLICK HERE if you use TypeScript instead of plain Node.js

If you've decided to use TypeScript for your AWS Lambda function instead of plain Node.js, follow the steps below to set up your project.

These steps are mandatory for TypeScript users to ensure proper compilation and deployment of your TypeScript code to AWS Lambda.

Step 1: Create Required TypeScript Files

Create an index.ts file: This will be your main handler function file.

It should be located in the lambda folder, next to an autogenerated index.js file.

If your project structure differs, adjust the "func_name" parameter in lambda_config.json to match your setup.

Step 2: Configure TypeScript

Create a tsconfig.json file in the root of your project with the following content:

{
    "compilerOptions": {
        "target": "es6",
        "module": "commonjs",
        "outDir": "./dist",
        "rootDir": "./app",
        "strict": true,
        "esModuleInterop": true,
        "moduleResolution": "node",
        "allowSyntheticDefaultImports": true
    },
    "include": ["app/**/*.ts", "app/lambdas/**/*.ts"],
    "exclude": ["node_modules"]
}

Step 3: Install TypeScript Compiler

Install TypeScript globally via npm:

npm install -g typescript

Verify the installation by checking the TypeScript compiler version:

tsc -v

Step 4: Compile TypeScript to JavaScript

Navigate to your project directory where tsconfig.json is located.

Run the TypeScript compiler:

tsc

This compiles your .ts files to .js files based on the configuration specified in tsconfig.json.

Step 5: Organize Compiled JavaScript Files

Locate the compiled .js files in the ./dist directory (or whichever directory you specified as the "outDir" in your tsconfig.json).
Move or copy the .js files next to their corresponding .ts files in the project structure, particularly within the lambda folder.

By following these steps, you'll have set up your project for AWS Lambda, including compiling TypeScript files into JavaScript and organizing them according to your project's structure.

2.2.5 Verbose mode

The Syndicate framework supports a verbose logging mode designed to provide users with more detailed information during command execution. This feature is invaluable for troubleshooting, debugging, and gaining deeper insights into the operational processes of Syndicate commands.

To enable verbose logging, use the --verbose flag or its alias -v with any Syndicate command.

How to Use Verbose Logging

To activate verbose logging, append --verbose or -v to the end of any Syndicate command. This flag instructs Syndicate to produce detailed logs of the command's execution, offering insights into the process and helping identify any potential issues.

Example:

syndicate deploy --verbose

Or using the alias:

syndicate deploy -v

2.2.6 Add other infrastructure components

This step is optional and could be skipped while getting familiar with syndicate.

All the resources the syndicate works with could be generated in the same way as lambda. Invoke the syndicate generate meta --help command to find out which resources are available.

2.2.7 Open project in your IDE

Now the project is ready to be adjusted. Consider opening your favourite IDE and observe the files created by syndicate.

2.3 Deployment

2.3.1 Create an S3 bucket for aws-syndicate artifacts:

syndicate create_deploy_target_bucket

2.3.2 Build project artifacts

syndicate build

2.3.3 Deploy project resources to AWS account

syndicate deploy

Now the DemoLambda is created and available to be tested.

2.3.4 Update resources

In order to be sure your latest changes works well on the AWS account the application should be deployed to the AWS account. To do this use the following commands set:

syndicate build
syndicate update

2.3.5 Lambdas invocations metrics

syndicate profiler

2.3.6 Clean up project resources from AWS Account

syndicate clean

2.3.7 Observing the environment manipulation history

syndicate status # this shows the general CLI dashboard where latest modification, locks state, latest event, project resources are shown
syndicate status --events # this returns all the history of what happened to the environment

Uniqueness of Regional Resource Naming

It is important to note that while Amazon Web Services (AWS) allows the creation of certain resources with identical names (e.g., API Gateway, Cognito User Pool), AWS Syndicate requires that all resource names be unique. This uniqueness policy is aimed at ensuring a clear and error-free organization of our resources.

Deployment Policy

When deploying a new resource, AWS Syndicate first checks whether a resource with the same name already exists. If such a resource exists, the creation of a new resource with the same name will be skipped. This approach helps prevent naming conflicts and ensures compliance with our security and management standards.

2.4 Export

The command is aimed at exporting resource configurations

Parameters:

resource_type: The type of resource to export configuration
dsl: Output domain-specific language

2.4.1 API Gateway OpenAPI specification

syndicate export
 --resource_type api_gateway
 --dsl oas_v3

By default, the specification will be saved in the export directory inside the project root directory as <api_id>_oas_v3.json

2.5 Examples

If you are just getting familiar with the functionality, you can use one of the pre-prepared examples that contain a minimum set of AWS resources and lambdas.

The aws-syndicate/examples folder contains structure examples for different runtimes. Go to any example you like best and set the environment variable SDCT_CONF=$path_to_the_selected_example.

Add your account details to syndicate.yml file - account id, secret access key, access key and bucket name for deployment. To syndicate_aliases.yml add your account id, region name (eu-central-1, us-west-1, etc.) and other values in the file that start with a $ sign.

Syndicate project examples

Documentation

You can find a detailed documentation here

Getting Help

We use GitHub issues for tracking bugs and feature requests. You can find our public backlog here. If it turns out that you may have found a bug, please open an issue with some of existing templates.

Default label for bugs, improvements and feature requests is To-Think-About, it defines that ticket requires additional information about what should be done in scope of this issue. To-Do label should be added only for tickets with clear and reviwed issue scope.

But before creating new issues - check existing, it may cover you problem or question. For increasing issue priority - just add "+1" comment.