Skip to content

danjenson/motoko

BranchesTags

Repository files navigation

motoko

motoko motorcycle

I'll have my AI analyze the data.

Graduate Schools

The following are links to relevant parts of codebase:

Overview

Motoko is an app that can plot, model, preview, and analyze datasets from iOS or Android.

The frontend is written in Dart using the Flutter framework and can be compiled to iOS, Android, Linux, and the Web.

The backend is written in Rust and Python. Rust is used for database management and AWS Lambda functions. Python is used for plotting and machine learning models.

An AWS RDS database (PostgreSQL) stores all of the data.

Local setup, data models, deployment, authentication and authorization, app signing, and network topography are detailed below.

There is a also a binary written in Rust that performs the various administrative and deployment commands called motoko, which can be installed with ./install_motoko_command.

Screenshots

TODO

Product

  • Marketplace for canned analyses?
  • Data adapters?
  • Tools like hedonometer

Local Setup

General

  • motoko command
    • curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
    • ./install_motoko_command
  • lambdas:
    • macOS:
      • brew install filosottile/musl-cross/musl-cross
      • ln -s /usr/local/bin/x86_64-linux-musl-gcc /usr/local/bin/musl-gcc
    • arch:
      • sudo pacman -S musl # required to compile rust lambdas
  • AWS:
    • motoko install aws
    • aws configure
    • sudo pacman -S docker # required for SAM
    • systemctl enable docker && systemctl start docker
    • sudo usermod -a -G docker danj # requires reboot
  • flutter:
    • motoko run setup-android-keystore
    • motoko install flutter
    • android
      • android-studio # install SDKs
        • install command line tools
        • install dart and flutter plugins
        • flutter doctor
          • flutter config --android-studio-dir /opt/android-studio
    • iOS
      • install XCode
  • add sqlx cli:
    • cargo install --version=0.2.0 sqlx-cli --no-default-features --features postgres
  • postgres:
    • macOS:
      • brew install postgres
      • enable connections from Docker using AWS SAM:
        • vim /usr/local/var/postgres/postgresql.conf
          • listen_addresses = '*'
          • port = 5432
        • vim /usr/local/var/postgres/pg_hba.conf
          • host all all 172.17.0.1/16 trust
      • psql -d postgres
      • CREATE USER postgres SUPERUSER
      • pg_ctl restart -D /usr/local/var/postgres
      • brew services postgres start
    • arch:
      • pacman -S postgresql
      • sudo -iu postgres
      • initdb -D /var/lib/postgres/data
      • enable connections from Docker using AWS SAM:
        • vim /var/lib/postgres/data/postgresql.conf
          • listen_addresses = '*'
        • vim /var/lib/postgres/data/pg_hba.conf
          • host all all 172.17.0.1/16 trust
      • systemctl enable postgresql
      • systemctl start postgresql
      • psql -U postgres or backend/rs/graphql/connect_to_db.sh
    • CREATE USER motoko WITH PASSWORD '<password>' # check .env file
    • CREATE DATABASE motoko_data
    • CREATE DATABASE motoko_meta
    • GRANT ALL PRIVILEGES ON DATABASE motoko_data TO motoko;
    • GRANT ALL PRIVILEGES ON DATABASE motoko_meta TO motoko;
    • motoko run reset-databases (local|remote)
  • run AWS SAM from motoko/backend:
    • sam local start-lambda
      • if you get [Errno 28] No space left on device, you need to increase your /tmp directory as detailed here
      • can also temporarily run: sudo mount -o remount,size=20G,noatime /tmp
    • Note that on Linux, the host IP is 172.17.0.1; calls from a rust test must call lambda functions at 127.0.0.1:3001, but calls back to host from those lambda functions must go through 172.17.0.1
    • To test locally, run motoko build sam; the lambda functions need to be built in order to be called by other testing scripts

Databases

AWS RDS creation:
  • Create new db here
  • Make sure public accessibility option is set to Yes
  • Make sure you assign a security group that permits TCP traffic to port 5432, like this

Simulators

Android
  • motoko run emulator {android,ios,web}
  • frontend $ flutter run
  • if you get PlatformException...ApiException: 10, see App Signing section
  • if you get an out of date Google Play Services:
    • sign in on device
    • go to Google Play
    • Settings
    • click Play Store version to update

Infrastructure

  • Frontend uses flutter, which is written in dart, here is the style guide
  • Backend uses API Gateway and Lambda Functions; most code is written in rust or python

Data Models

model graph

Creating a Migration

  • sqlx migrate add <name> # fill out in migrations/<name>.sql
  • cargo sqlx prepare -- --lib # recompile static type checking

Deployment

Authentication and Authorization

App Signing

  • android requires a keystore to sign the release app:
    • to setup building locally using the release keys, run motoko run setup-android-keystore, which does the following:
      • downloads the android keystore to ~/.keys/motoko/android/signing_key.jks
      • creates the file frontend/android/key.properties, which contains the password to unlock the keystore (also from AWS Secrets Manager) and is used when building by gradle; do not add either of these files to the code repo
    • make sure that before installing the version with the new keystore, you have uninstalled the old version
    • to reset the keystore in AWS Secrets Manager, run motoko run reset-android-keystores, which does the following:
      • generates a new keystore and uploads it to AWS Secrets Manager with the key android_{release,debug}_keystore along with the password under the key android_keystore_password
      • runs the same commands as motoko run setup-android-keystores to setup the local environment to use the new keys
      • NOTE: after a reset, you will need to run ./gradlew signingReport from the frontend/android directory and copy the debug and release SHA1 hashes into the OAuth2 clients configs: motoko-android-debug and motoko-android-release; this lets google login know that builds using these signatures are legitimate; if the hashes are incorrect, google will reject attempted logins and return a PlatformException with error code 10

Topography

  • Route 53:
    • NS Records:
      • mapped Nameservers from Namecheap to Route 53 Nameservers above
      • when validating ownership with AWS, remove name suffix motoko.ai for CNAME records because Namecheap automatically appends it
    • A Records:
      • motoko.ai:
        • mapped to this CloudFront distribution
          • re-routes traffic from motoko.ai/graphql to the prod stage of API Gateway's motoko-api
          • re-routes traffic from motoko.ai/_ and motoko.ai/install/_ to prod/_ and prod/install/_ in the S3 bucket
            • allows access by OAI (Origin Access Identity) to CloudFront distribution in bucket policy
      • dev.motoko.ai:
        • mapped to this CloudFront distribution
          • re-routes traffic from dev.motoko.ai/graphql to the dev stage of API Gateway's motoko-api
          • re-routes traffic from motoko.ai/_ and motoko.ai/install/_ to dev/_ and dev/install/_ in the S3 bucket
            • allows access by OAI (Origin Access Identity) to CloudFront distribution in bucket policy

GOTCHAS:

  • when using sam build on macOS, be careful about installing binaries; see backend/py/model/Makefile, which installs psycopg2_binary separately from the rest of the dependencies so it can specify a platform version
  • use docker run -it --rm busybox to debug networking from a docker container
  • Integration Requests from API Gateway to Lambda Functions must have Proxy enabled.
  • CloudFront Forwarding should have TTL set to 0 under Behaviors for requests that shouldn't be cached
  • If you re-upload a lambda that had environment variables, you will need to manually put them back in or you will get an internal server error
  • If you get this error Error: NotPresent, it's because you're missing an environment variable
  • Lambdas require python 3.8 [2020-12-01], which can be install from AUR on Arch Linux; if you are running 3.9 and use --python-version 3.8, it requires setting --only-binary :all:, and (1) some packages don't have binaries, (2) this leads to a larger deployment, which, incidentally, also blows out the 128MB lambda function memory as well; better to install python3.8 as the default version until AWS upgrades to 3.9
  • Make sure you set the timeout on the Lambdas to be sufficiently long, especially on things like motoko-uri-to-sql-db
  • If a lambda request is taking a long time, check to see whether it is using all it's memory and upgrade it if necessary
  • If you get weird errors like this, try upgrading gradle or android build tools
  • If lambdas start failing, check that you haven't used all database connections:
    • select usename from pg_stat_activity;
    • select pg_terminate_backend(pid) from pg_stat_activity where usename='motoko';
  • Don't use double quotes in psycopg2 queries

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages