This project is responsible for generating booking data and then saving generated csv files into hdfs.
##Configuration
There are many parameters that can be pass as environment variables (option -e, example -e HDFS_URL='hadoop') to docker:
-
HDFS_URL- the url of hdfs file system or ip of docker image with hadoop, if it is not passed, the default value hdfs://hdfs:8020 is used -
INFINITE_GENERATE- if this flag is set to true value, generator runs indefinitely, otherwise it will generate data only once -
SAVE_TO_DATABASE- if this flag is set to true value, generator saves the data into database defined in database.properties -
BATCH_SIZE- set the size of one batch of generated flights -
FLIGHTS_NUMBER- number of all flights to generate -
BOOKING_FILENAME- name of file where are saved generated bookings -
SERVICE_FILENAME- name of file where are saved generated services -
SAVE_LOCALLY- flag, if want to save files locally, not in hdfs
##Building
-
Clone git projects and put in into this project structure
uc4 ├── uc4-ancillarypricingcalculator └── uc4-AncillaryPricingDataGenerator
-
Go into uc4-AncillaryPricingDataGenerator/scripts (cd uc4-AncillaryPricingDataGenerator/scripts)
-
Change build script permissions (chmod +x build.sh)
-
Then run script (./build.sh)
###build.sh script steps
- Generator code is compiled and docker image with output jar file is created by maven plugin.
- Docker image with calculator code is build
##Running
- Execute building steps
- Go into uc4-AncillaryPricingDataGenerator/scripts (cd uc4-AncillaryPricingDataGenerator/scripts)
- Change build script permissions (chmod +x run.sh)
- Then run script (./run.sh)
###run.sh script steps
- Running images are stopped.
- Docker network is created, if it is needed.
- Docker image with hadoop is started.
- The generator image is started and it connects to hadoop container using docker network to save generated data.
- Calculator docker image starts after termination of docker with generator. It reads data from the same hadoop docker, then saves output results in newly created directory ‘response’
All dockers are connected together within network created in step 2.