aws-exec

why

building services on lambda should be easy and fun.

what

a project scaffold for a backend service on aws with an infrastructure set ready-to-deploy with libaws.

the project scaffold makes it easy to:

• authenticate callers.

• implement fast synchronous apis that return all results immediately.

• implement slow asynchronous apis with streaming logs, exit code, and 15 minutes max duration.

• use the web admin interface, even from a phone.

• use the cli admin interface, executing locally or on lambda.

• use the api interface, calling efficiently from other backend services.

how

synchronous apis are normal http on lambda.

asynchronous apis are a http post that triggers an async lambda which invokes a command via rpc or subprocess and stores the results in s3.

• each invocation creates 3 objects in s3:

  • log: all stdout and stderr, updated in its entirety every second.
  • exit: the exit code of the command, written once.
  • size: the size in bytes of the log after the final update, written once, written last.

• objects are stored in either:

  • aws-exec private s3.
  • presigned s3 put urls provided by the caller.

• to follow invocation status, the caller:

  • polls the log object with increasing range-start.
  • stops when the size object exists and range-start equals size.
  • returns the exit object.

there are three ways to invoke an asynchronous api:

• api invoke via rpc, this is faster.
• cli invoke via subprocess, this is slower.
• web invoke via subprocess, this is slower.

add a new synchronous functionality

add to api/.

duplicate the httpExecGet or httpExecPost handler and modify it to introduce new functionality.

add a new asynchronous functionality

add to cmd/.

duplicate the listdir command and modify it to introduce new functionality.

web demo

cli demo

api demo

mobile demo

dependencies

use the included Dockerfile or install the following dependencies:

• npm
• jdk
• go
• bash
• entr
• libaws

aws prerequisites

• aws route53 has the domain or its parent from env.sh

• aws acm has a wildcard cert for the domain or its parent from env.sh

usage

go install github.com/nathants/libaws@latest
export PATH=$PATH:$(go env GOPATH)/bin

cp env.sh.template env.sh # update values
bash bin/check.sh         # lint
bash bin/preview.sh       # preview changes to aws infra
bash bin/ensure.sh        # ensure aws infra
bash bin/dev.sh           # iterate on backend and frontend
bash bin/logs.sh          # tail the logs
bash bin/delete.sh        # delete aws infra
bash bin/cli.sh -h        # interact with the service via the cli

usage with bad upload bandwidth:

# bash bin/dev.sh         # this needs upload bandwidth
bash bin/dev_frontend.sh  # iterate on localhost frontend
bash bin/relay.sh         # iterate on backend via ec2 relay

usage with docker

cp env.sh.template env.sh # update values
docker build -t aws-exec:latest .
docker run -it --rm \
    -v $(pwd):/code \
    -e AWS_DEFAULT_REGION \
    -e AWS_ACCESS_KEY_ID \
    -e AWS_SECRET_ACCESS_KEY \
    aws-exec:latest \
    bash -c '
        cd /code
        bash bin/ensure.sh
    '

create auth

bash bin/cli.sh auth-new test-user

install and use cli

go install github.com/nathants/aws-exec@latest
export PATH=$PATH:$(go env GOPATH)/bin

export AUTH=$AUTH
export PROJECT_DOMAIN=$DOMAIN
aws-exec exec -- whoami

install and use api

go get github.com/nathants/aws-exec@latest

package cmd

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	awsexec "github.com/nathants/aws-exec/exec"
)

func main() {
	val, err := json.Marshal(map[string]interface{}{
		"path": ".",
	})
	if err != nil {
	    panic(err)
	}
	exitCode, err := awsexec.Exec(context.Background(), &awsexec.Args{
		Url:     "https://%s" + os.Getenv("PROJECT_DOMAIN"),
		Auth:    os.Getenv("AUTH"),
		RpcName: "listdir",
		RpcArgs: string(val),
		LogDataCallback: func(logs string) {
			fmt.Print(logs)
		},
	})
	if err != nil {
		panic(err)
	}
	os.Exit(exitCode)
}