Skip to content

Latest commit

 

History

History
64 lines (50 loc) · 1.8 KB

README.md

File metadata and controls

64 lines (50 loc) · 1.8 KB

vm setup

set up ssh connection

First we connet to the vm instances via ssh connections. Follow the following steps to set up ssh

  1. create ssh key on your local environment

  2. add the generated public key to the project metadata

    • copy the contents of the .pub file
    • follow the guide here

  3. create a config file in the .ssh directory

    touch ~/.ssh/config

  4. paste the text below into the config file and edit accordingly

     Host kafka-vm
     HostName <External IP Address>
     User <username>
     IdentityFile <~/.ssh/private_keyfile>
     LocalForward 9021 localhost:9021

 Host spark-master-node
     HostName <External IP Address Of Master Node>
     User <username>
     IdentityFile <~/.ssh/private_keyfile>
     LocalForward 4040 localhost:4040

 Host airflow-vm
     HostName <External IP Address>
     User <username>
     IdentityFile <~/.ssh/private_keyfile>
     LocalForward 8080 localhost:8080

  5. connect to the vms in separate terminal windows

     ssh kafka-vm
     ssh spark-master-node
     ssh airflow-vm

setup vms

  1. clone git repo and change directory to kafka

    git clone https://github.com/topefolorunso/musicaly-project.git ~/musicaly-project

    The following set up only applies to the kafka and airflow vms. The spark vm is managed by GCP so the necessary installation has been handled upon provisioning.

  2. install python (anaconda dist), docker and docker-compose in the vm

    bash ~/musicaly-project/vm_setup/vm_setup.sh && \
exec newgrp docker