Skip to content

Scalable Network-Based RL Environment for High-Level EV Natural Disaster Response Simulation

License

Notifications You must be signed in to change notification settings

Natemen96/ECEMasterProject

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

65 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ECE Master Project - Scalable Network-Based RL Environment for High-Level EV Natural Disaster Response Simulation

Purpose

In situations of natural disasters/phenomenons, blackouts can occur. To combat this, EV (electric vehicles) drivers can use their vehicles as batteries for customers/ clients. To simulate the difficulty of this challenge of getting to as many customers as possible until the blackout period is over, a RL environment was created. This environment features random scenarios based on node network size, amount of the houses (can't be larger than node size), blackout data, and housing models to simulate a blackout response in a high level matter.

Demo

Demo of RL environment + random agent in action

Example of an random agent in action.

Load represents the power needed at the moment from the node.

Current Charge represents the power the agent (or car) has.

Values on edges represents the miles of the edge cost to go through.

Blue node represents buffers. Green node represents homes. Grey node represent blackout out node.

More details can be found in Action & State Space section.

Run Locally

Clone the project

  git clone https://github.com/Natemen96/ECEMasterProject.git

Go to the project directory

  cd ECEMasterProject/RL
  python main.py

Installation

WARNING: This is a large install (~2GB) and will take approximately 30 mins depending on internet speed. Depending on your set up, you may find it easier to install the package yourself. The list can be found in the conda_env folder. See 'conda_env/rl_window.yml' for the most up to date package list. Works best on Windows.

For easy installing, using a conda first make sure you have conda installed.

After conda is installed do the following to set up the environment

  conda env create -f conda_env/rl_window.yml --name rl_env  #or any name you prefer

  conda activate rl_env #verify it work 

make sure to run the program within the conda env.

Housing Data can be download from pecan street/dataport

Task for Agent

Start at "Home" (a node) travel to each location in exchange for power (main resource for EV), unload power in exchange for reward, travel back to home in exchange for power. The agent can restore power at "Home" and earn more reward depending on how much power it got back. The more power it restores for itself the more it's rewarded.

Once power is restored in a node with a power charging station, the agent can restore energy there as well. Some nodes will be empty depending on parameters.

Environment Overview

graph_env: Randomly generated a fully connected graph n nodes

Args:
    n (int, optional): [number of nodes]. Defaults to 5.
    numofhome (int, optional): [number of homes for agent starting 
    point]. Defaults to 1.
    numofblackout (int, optional): [number of houses with blackout]. 
    Defaults to 1.
    numofblackoutws (int, optional): [number of blackout houses with 
    solarpower]. Defaults to 0.
    numofchargingstation (int, optional): [number of charging 
    station]. Defaults to 0.
    max_actions (int, optional): [max number of action can be done 
    times 7.  By default 3 (7) = 21]. Defaults to 3.
    blackout_str (str, optional): [Reads Blackout data from npy 
    file. Blackout data from the 2021 Texas Power Crisis, a file for 
    each country]. Defaults to 'Brazoria'.
    agents (list, optional): [Agent objects]. Defaults to [None].
    nonsolarhouse_data_paths (list, optional): [Path for npy file 
    with nonsolar data info. Added to a node per path.]. Defaults to 
    [None].
    solarhouse_data_paths (list, optional): [Path for npy file with 
    solarhouse data info. Added to a node per path.]. Defaults to 
    [None].
    data_sample (int, optional): [how many data sample it take from 
    data in paths, samples taken evenly throughout data. By default 
    (0/6. 1/6 ... 6/6 0/6 ...) sample]. Defaults to 6.

Agent Overview

BasicAgent: Basic Agent that act as skeleton for other agent. Won't work by itself.

Args:
    car ([dict]): [car information in the form of a python
     dictionary. Required dict information: model, make, year,
     avg_energy_consumption(kWh/mile), battery_capacity(kWh)]
SmartQLAgent: Smart Agent that uses Qtable to make decisions

Args:
    car ([dict]): [car information in the form of a python 
    dictionary. Required dict information: model, make, year,
     avg_energy_consumption(kWh/mile), battery_capacity(kWh)]
    sample_rate (int, optional): [decides how often a sample of the
    qtable is taken. If left to default it's every 1/100 of total ep]
    . Defaults to 100.
    qtabletraining (bool, optional): [Flag for turning qtable 
    training on]. Defaults to True.
    quiet (bool, optional): [Flag for turning data collection on]. 
    Defaults to True.
RandomDataAgent: Used for data collection of random agent. Doesn't 
use Observation.

Args:
    car ([dict]): [car information in the form of a python 
    dictionary. Required dict information: model, make, year,
     avg_energy_consumption(kWh/mile), battery_capacity(kWh)]

Reward

Reward given for unloading, recharging. Diversity of power restoration will result in more reward. Reward lost for running out of energy. Episode is over when EV is out of energy or network is completely powered.

Action & State Space

    Each Agent will have 24 actions per ep by default.

    e+2 Actions:
        e - Move (at cost) to Node if possible
        1 - Unload (at cost)
        1 - Recharge (no cost, expect an action)

    2*n + 6*n + 300*n + c States:
        n - Statuses of Each Node
            Key - Assignment
            0 - Buffer (Blue)
            1 - Blackout (Darkgrey)
            2 - Blackout with Solar (Lightgrey)
            3 - Powered (Blue)
            4 - Charging Station (Yellow)
            5 - Home (Green)

        n - Cost of each edge

        n - Is the path reachable or not? 
            0 - No 
            1 - Yes
        
        c - Charge State (400 possible values by default)

    where n is the number of nodes states, e is the number of edges, and c number of is Charge states 

Logic

Each nodes was designed to be based on a house dataset. Some houses that are solar powered will need less power from EVs compared to non-solar powered homes. The cost of house node can be derived from the housing data.

Percentage of blackout based on texas blackout data, as well as change per day.

At a minimum a Q-Learning was used to find the best solutions.

Scalability

Can be scaled to multiple EV (MARL would be needed), & a larger network. MARL has not been fully tested and is likely very buggy.

Optimizations

During a test run of 50k Eps the smart agent that was support with Qtables had 263.13% improvement compared to a random agent (that does random actions).

Data Overview

scripts/Data/TexasBlackout.csv
    -csv file of data collected from https://poweroutage.us/area/state/texas during the 2021 Texas Power Crisis with the following  
    - Metrics: ID,County,Customers Tracked,Customers Out,Outage %,DateTime

ECEMasterProject/RL/Data/
    Avg_House_0/
        - npy files of average daily power per every 15 mins of various houses provide by Dataport

    Avg_House_1/
        - npy files of next day regression of Avg_House_0 per house
    
    Avg_House_n/
        - npy files of concatenation of Avg_House_0 adn Avg_House_1 per houses

    County/
        - npy files of Texas County that were effect the most during the 2021 Texas Power Crisis

RL/agent/Sim_Data
    - csv file result from various RL training, only stored reward. 

RL/agent/agent_jsons/
    -various json file of EV information 
    -Metrics: model, make, year, avg_energy_consumption(kWh/mile), battery_capacity(kWh), source

Model Overview

models/

    KM/ 
        -Classifier models trained using Time Series K Means stored in h5

    LSTM/ 
        -Old LSTM regression model trained on one house stored in h5

    LSTM_nonsolar/
        -LSTM regression model trained on non solar data stored in h5

    LSTM_solar/
        -LSTM regression model trained on solar data stored in h5

    Qtables/
        -Qtables from RL training stored in h5

NoteBooks Overview

notebooks/

    analysis_c0.ipynb 
        -Analysis of classifier and regression model on solar houses data

    analysis_c1.ipynb 
        -Analysis of classifier and regression model on nonsolar houses data

    cluster.ipynb
        -Creation and Analysis of classifier that can distinguish between solar and nonsolar data

    compare_qtable_analyis.ipynb
        -Comparing Qtable Smart Agent with Random Agent Baseline

    D_LSTM_HousingData.ipynb
        -Creation and analysis of regression on avg housing data where the original data sample occurs every 15 mins -> 98 sample per day

    Dp1_analysis.ipynb
        -Creation and analysis of regression on one house

    EveryNyHouseAvg.ipynb
        -Creation and analysis of regression on all house samples with model trained on one house (house 1)

    LSTM_cluster0.ipynb
        -Creation and analysis of regression on one house using most of the solar housing data 

    LSTM_cluster1.ipynb
        -Creation and analysis of regression on one house using most of the nonsolar housing data

    qtable_analyis_4_19.ipynb
        -Analysis of Smart Agents performance

    qtable_analyis_4_22.ipynb
        -Analysis of Smart Agents performance

    qtable_analyis.ipynb
        -Analysis of Smart Agents performance

    random_agent_analyis.ipynb
        -Analysis of Random Agents performance

    Texas_Blackout.ipynb
        -Analysis of recorded Blackout Data

Extras Ideas

Some edges can be blocked off and restored later as days pass (simulate for natural disasters)

Add testing

DRL and Classic Optimization techniques

Debug for MARL

Convert to OpenAI environment

Lessons Learned

Learned more about making DL/ML models.

Learned more about data analysis from exploring data and results from DL/ML training.

Learned about networkx and it's capabilities for graph theory related for projects.

Learned more about making reinforcement learning environments and agents.

Learned more about data scraping as I needed to scrap blackout data during the 2021 Texas Power Crisis

Learned how to make a gif using images in python.

Acknowledgements

Thank you to Professor Qin of my alma mater, Purdue University for his amazing support and sharing his expertise for this project.

Thank you to DataPort and Pecan Street for providing housing data to build models with.

Thank you to Matt Leacock the creator of the board game, Pandemic for inspiring this Reinforcement Learning Environment.

Authors

License and Badges

GPLv3 License DOI

About

Scalable Network-Based RL Environment for High-Level EV Natural Disaster Response Simulation

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages