FW-527: AWS upload

[lewis-chambers]

Makes the changes needed to track the latest data sent to AWS and send only new data.

Changes

• Implemented file upload step
  • When run it checks the state file for the last data sent, falling back to 3 hours ago if not in the state file yet. Then it retrieves all data rows after the given time and converts them into CR1000X datalogger style payloads. The payloads are then dropped into the requested s3 bucket and prefix. On successful writes, if the written data timestamp is newer than in the state file, the file is updated (they may be written out of order).
• Wrote unit tests
• Wrote end to end tests
• Implemented a simple CLI for running the code
• Added logging
• Added localstack

[richjam]

test_db, test_messaging, and test_devices are being skipped because it requires config.cfg to exist. Is this still valid, given the docs tell user to create the config file as oracle.cfg?

[lewis-chambers]

test_db, test_messaging, and test_devices are being skipped because it requires config.cfg to exist. Is this still valid, given the docs tell user to create the config file as oracle.cfg?

The config file was used to link into the Cosmos database for testing and comes from a time when I didn't know I could mimick the connections through mocking. It shouldn't impact the live-cosmos sub-app. They get skipped because the CI wouldn't be able to connect to the database anyway. I'm not keen to patch it because the cosmos swarm will be abandoned soon.

[richjam]

In liveupload.py, add fallback time to logger message on line 84

logger.debug(f"site {site} not in state. Using fallback time {self._fallback_time}.")

[richjam]

In liveupload.py, add fallback time to logger message on line 84

logger.debug(f"site {site} not in state. Using fallback time {self._fallback_time}.")

Should/could the fallback time be a command line arg, for testing purposes (and if we wanted to send a bigger chunk of data?) Or the timedelta hours?

[richjam]

Suggested change

	site: A list of sites to upload from. Grabs sites from the Oracle database if
	sites: A list of sites to upload from. Grabs sites from the Oracle database if

[lewis-chambers]

This one is supposed to be site so that you add sites in the CLI like --site SITE1 --site SITE2. It felt odd to have them be plural each time

[lewis-chambers]

In liveupload.py, add fallback time to logger message on line 84
logger.debug(f"site {site} not in state. Using fallback time {self._fallback_time}.")

Should/could the fallback time be a command line arg, for testing purposes (and if we wanted to send a bigger chunk of data?) Or the timedelta hours?

This has been added as an argument now