This project is a proof-of-concept (POC) for an A/B-testing architecture for Alexa Skills. There's a Medium article that describes the background and architecture of this POC in greater detail, whereas this repository serves more as a blueprint for how to set up the system for yourself.

Here's a graphical representation of the POC's architecture:

This repo covers the following of these components:

• Alexa Skill frontent (manifest and language model)
• The router Lambda function (henceforth abbreviated as 'router') implemented in plain Node.js
• A deployment and testing script for the router (as bash scripts)
• The Alexa Skill backend (using the Jovo framework) that can be deployed as the version A and B Lambda function
  • To keep this POC simple, the difference in their execution will come from different environment parameters
• Project and deployment information for the Alexa Skill's backend and frontend (using the Jovo CLI)
• A setup script for the DynamoDB database that serves as the result store, using Node.js
• For the sake of simplicity, the Skill backend uses no other resources like DynamoDB, S3, Google Spreadsheet or other

AWS CloudFormation setup

AWS CloudFormation is an 'infrastructure as code' tool that helps you provision, deploy, update and delete AWS stacks of arbitrary complexity.

This repository contains the template for setting up the POC described here. Please be aware that it can currently only be used in the us-east-1 region (because that's where the S3 bucket with the pre-packaged Lambda deployment packages is), but I'm working on something more portable. You can still use it to quickly set up and check out the system.

Deploying the stack

1. In your AWS account, change to the region "North Virginia" (us-east-1) and go to CloudFormation.
2. What you'll see here depends on whether you already have CloudFormation stacks. In either case, find the button Create stack
3. In step 1 of the dialog, choose 'Template is ready' and 'Upload a template file', then find the cloudFormation.yamlfile, select it for upload and click 'Next':
   

4. In step 2 of the dialog, you must give the stack a name (e.g. `abtesting-poc`), and then you can review the stack's parameters and change them as you see fit. Again, click 'Next' when you're finished 5. You can leave the settings in step 3 at their defaults, and hit 'Next' 6. In step 4 you can review your stack's settings. One box you must check before you proceed is the one for 'Capabilities' at the very bottom. It wants you to acknowledge that the stack creates IAM resources - To be precise, the policies and roles required for the system's Lambda functions to work. I took great care that these policies have super tight privileges!

7. After you hit 'Create stack', CloudFormation creates and configures the required resources. You can follow along by clicking the refresh icon in the 'Events' tab. The entire process should take about 2 minutes. 8. After your stack is in the status 'CREATE_COMPLETE', you can switch to the 'Outputs' tab and copy the value of the `EndpointARN` key

Updating your Alexa Skill

1. Navigate into the folder ./demoApp, and set up your .env file by typing
   cp .env.example .env
2. In the file ./demoApp/.env, replace the dummy value of LAMBDA_ARN_ROUTER with the one your copied from the outputs of your CloudFormation console.
3. Now build and deploy the language model and Skill manifest:
   jovo build --stage console --deploy
   

The result should look like this:

3. Now copy the Skill ID and paste it as the value of SKILL_ID in your .env file for future reference.

Congrats, you have completely set up this A/B-testing POC!

You can skip the following manual setup instructions, since you have now deployed this relatively complex system in less than 5 minutes.

If your want to delete this POC system, you can easily do so from the CloudFront console: Just find your stack and hit the 'Delete' button. It will remove all the resources (Lambda, DynamoDB and IAM) that were created for this project.

Now I recommend to jump to the 'Seeing it in action' section.

Manual setup instructions

We're going through these parts step by step. We start with the result store, then continue with the Alexa Skill frontend and backend, and finish with the router.

This script assumes that you have an AWS account, your AWS CLI and ASK CLI configured and set up, and the Jovo CLI installed.

Setting up the result store

Setting up a simple DynamoDB table like we need for the result store is very easy, but to make it even easier I provided a setup script that you can use if you like. In order for it to access your AWS account and create the table, you'll need to paste the credentials of your AWS CLI account into the environment variables.

1. Navigate into the folder ./demoApp, and set up your .env file by typing
   cp .env.example .env
2. If you have your AWS CLI configured and want to use these credentials for the setup script, you can open your AWS CLI credentials file by typing
   cat ~/.aws/credentials
3. From there, you can copy the access key ID and secret access key and paste them into the values of the parameters AWS_CREDENTIALS_ACCESS_KEY_ID and AWS_CREDENTIALS_SECRET_ACCESS_KEY of your .env file
4. If your want to name your DynamoDB table anything else than ABTESTING_DATA, your can change that in the value of the parameter DYNAMODB_TABLE_NAME in your .env file. Same with your AWS region and the parameter AWS_REGION.
5. If your AWS CLI user has the right IAM policies attached (you need at least dynamodb:CreateTable), you can now run your setup script by typing
   node setup.js

As a result, you should see about the following (with a list of your existing tables in the selected region above):

Congrats, you have already set up the results store DynamoDB. If you'd run the scipt again, it would notice that the table already exists, and not attempt to create it again.

Setting up the A and B version of the Alexa Skill backend

To do this, you'll need two Lambda functions. Unfortunately these can't easily be created with a setup script, because they need a IAM role ARN, for which you'll need to log into the console anyways.

I assume you have a at least some familiarity with setting up Lambda functions, so I won't guide you step by step. Instead, I'm giving you the config of your Lambda, and then we'll continue afterwards.

1. Log into the AWS Console and go to the Lambda function overview for the region your selected before. Create a new function with the following config:
   • Name: Could be anything, by I recommend abTesting_version_a
   • Runtime: Node.js 8.10 or 10.x
   • Permissions: Use (or create) a role that allows logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents and dynamodb:PutItem
   • Increase the timeout from 3 to 5 seconds
2. Copy some of the parameters from .env into the 'Environment variables' section of the Lambda:
   • DYNAMODB_TABLE_NAME with the same value as in your local file
   • TESTING_CONTENT with the value A. This environment variable is the only difference between versions A and B!
3. Now copy the Lambda ARN and paste it into your .env file as the value of LAMBDA_ARN_VERSION_A
4. Repeat steps 1 to 3 for new Lambda function, except this time you're substituting A by B
   (This applies to the Lambda name, TESTTING_CONTENT and LAMBDA_ARN_VERSION_B)

Interestingly, you don't need to configure any trigger for these two Lambdas! The reason is that they are invoked by the AWS SDK, which apparently doens't count as a triggering service.

5. Now that you have set up the Lambdas for version A and B, you can deploy the demo voice app into them right away. Before, you need to install the project's dependencies:
   npm run install
6. To deploy to Lambda, simply type:
   jovo build --stage versionA --deploy and
   jovo build --stage versionB --deploy.
   

The result should look like this:

Setting up the router

The router is a Lambda function that manages the split of the users for the A/B-test and routes the Alexa request to the according A or B version's Lambda.

1. Go back to the Lambda function overview for your selected region, and create a new function with the following config:
   • Name: Could be anything, by I recommend abTesting_router
   • Runtime: Node.js 8.10 or 10.x
   • Permissions: Use (or create) a role that allows logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents and lambda:InvokeFunction
   • Increase the timeout from 3 to 8 seconds (just in case!)
   • Add 'Alexa Skills Kit' as a trigger
2. Now configure the Lambda's environment variables:
   • LAMBDA_ARN_VERSION_A and LAMBDA_ARN_VERSION_B with the respective values from your .env file
   • TESTING_SPLIT with a comma-separated list of the percentages you want for your split, e.g. 80,20 for 80% A and 20% B
   • TESTING_SALT with any value you can just randomly generate. If you don't feel creative, you can just take today's date in a format of your choosing.
3. Now copy the Lambda's ARN and paste it as the value of LAMBDA_ARN_ROUTER in the .env file (remember, this is still the .env file of the demoApp folder!)
4. You need a separate .env in the router folder to deploy your router, so you can either copy it or make a link (since they share some environment variables):
   cp ../demoApp/.env ../router/.env
5. Navigate into the router folder if you haven't already.
6. The last things that's missing for you to be able to deploy is the name of your AWS CLI profile, which you need to paste as the value of AWS_PROFILE in your .env variable.
7. The other last thing to do before being able to deploy is to give the deployment script execution privileges:
   chmod +x deploy.sh
   (and chmod +x invoke.sh, if you also want to invoke your router from the CLI)
8. Finally, you can deploy:
   ./deploy.sh
   Optionally, you can already try and invoke your router function:
   ./invoke.sh

The result should look like this:

Something interesting to note here is that the duration was more than 2000 ms, which is definitely a lot! But given that in this case both Lambdas were cold, I think it's tolerable. If you invoke the function a couple of times, you can bring the duration down to few hundred milliseconds.

Setting up the Alexa Skill frontend

We've waited with this until the end because we need to have an endpoint ARN that is configured to accept triggers from the Alexa Skills Kit, which we just set up in the previous step.

Now this will be the easiest to set up:

1. Navigate back into the demoApp folder, and now build and deploy the language model and Skill manifest:
   jovo build --stage console --deploy

The result should look like this:

2. Now copy the Skill ID and paste it as the value of SKILL_ID in your .env file for future reference.

Congrats, you have completely set up this A/B-testing POC!

Seeing it in action!

Now let's trs it out. Please don't expect something spectacular, it's just a POC. But I hope it makes it easy to imagine more interesting use cases for it!

Here's the dialog you can expect when you invoke the Skill:

If you invoke it a couple of times, you should get a mix of A and B, depending on the split ration you configured for the router. This behavior wouldn't make sense for a live application, where one user should consistently see the same version - And that's how the router is actually built. Just for demonstration purposes (i.e. to populate the result store with interesting data), this POC assigns a version depending on the session ID instead of the user ID. You can change this behavior by uncommenting line 20 of the router's index.js file.

Now let's look inside the result store DynamoDB:

For a real-world application, you would define your custom metrics here, and could then do inferential statistics with these data.

Conclusion

If you followed along, you should have gotten a functional POC for A/B-testing an Alexa Skill. If you ran into any issues during setup, please let me know!

I personally see a huge potential for optimizing voice apps with A/B-testing, and hope that you can adapt this POC for your specific purposes. It's worth noting that while the Jovo framework and CLI are great for development and deployment, this system works just as well with any other framework or SDK that runs on Lambda.

Thanks for reading. If this has been useful to you, please give this repo a star and/or follow me on Twitter.