Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get data #36

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open

Get data #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
f22a51c
get types
JrtPec Dec 18, 2017
80d95cb
`get data` and `get sensor data`
JrtPec Dec 18, 2017
ea18ed1
`get data` and `get sensor data`
JrtPec Dec 18, 2017
87906df
typo
JrtPec Dec 18, 2017
80162cf
Merge branch 'master' into get_data
JrtPec Dec 18, 2017
7f25759
Merge branch 'master' into get_data
JrtPec Dec 18, 2017
8a1e386
better demo
JrtPec Dec 21, 2017
be07b60
deal with sensors without data
JrtPec Dec 21, 2017
a2e0fd9
get unit
JrtPec Dec 21, 2017
6693b92
catch empty string unit
JrtPec Dec 21, 2017
574ede0
Catch empty list of series before attempting concatenation
JrtPec Mar 1, 2018
63df831
use generators to do lazy loading and be more memory efficient
JrtPec Mar 1, 2018
11e0d93
replace empty list check with try catch, because it is a generator so…
JrtPec Mar 1, 2018
5731656
add option to automatically get the diff for counters
JrtPec Apr 12, 2018
847c12f
get types
JrtPec Dec 18, 2017
0db4738
`get data` and `get sensor data`
JrtPec Dec 18, 2017
cb2b351
`get data` and `get sensor data`
JrtPec Dec 18, 2017
ec6da77
typo
JrtPec Dec 18, 2017
67cfe18
better demo
JrtPec Dec 21, 2017
1c941e4
deal with sensors without data
JrtPec Dec 21, 2017
8d7daf3
get unit
JrtPec Dec 21, 2017
45d7545
catch empty string unit
JrtPec Dec 21, 2017
50ba56a
Catch empty list of series before attempting concatenation
JrtPec Mar 1, 2018
9117aae
use generators to do lazy loading and be more memory efficient
JrtPec Mar 1, 2018
1c9ebc9
replace empty list check with try catch, because it is a generator so…
JrtPec Mar 1, 2018
b5f353a
add option to automatically get the diff for counters
JrtPec Apr 12, 2018
52e1519
Merge remote-tracking branch 'origin/get_data' into get_data
JrtPec Jun 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
235 changes: 200 additions & 35 deletions tmpo.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,21 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline"
"import pandas as pd\n",
"import os\n",
"import tmpo\n",
"\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"token = \"b371402dc767cc83e41bc294b63f9586\"\n",
Expand All @@ -27,17 +28,19 @@
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Init session, add sensors and sync"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import tmpo\n",
"\n",
"s = tmpo.Session(path=os.getcwd())\n",
"for room in house:\n",
" for sensor in house[room]:\n",
Expand All @@ -47,9 +50,7 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"s.debug = True"
Expand All @@ -58,50 +59,78 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"s.sync()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get info about sensor"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"start = pd.Timestamp('20160101')"
"sensor = house[room][\"electricity\"]\n",
"\n",
"type, data_type, subtype = s.get_types(sensor)\n",
"unit = s.get_unit(sensor)\n",
"\n",
"print(type, data_type, subtype, unit)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get raw data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"metadata": {},
"outputs": [],
"source": [
"s.debug = False"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = s.dataframe([house[room][\"electricity\"]], head=start)"
"start = pd.Timestamp('20160101')\n",
"end = pd.Timestamp('20160501')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = s.dataframe(sids=[house[room][\"electricity\"]], head=start, tail=end)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"room = \"cellar\"\n",
"plt.figure(figsize=(16,10))\n",
"plt.figure(figsize=(16,5))\n",
"ax1=plt.subplot()\n",
"ax1.grid()\n",
"ax1.set_ylabel(u'Wh')\n",
Expand All @@ -110,11 +139,147 @@
]
},
{
"cell_type": "code",
"execution_count": null,
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Get resampled data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The difference between raw and resampled data is that raw data has data with an uneven time axis. This makes it difficult to calculate power values or flow rates."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"df = s.get_data(sids=[house[room][\"electricity\"]], head=start, tail=end, resolution='1h')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As resolution you can pick from these resolutions, and you can precede them with a number:\n",
"* B business day frequency\n",
"* C custom business day frequency (experimental)\n",
"* D calendar day frequency\n",
"* W weekly frequency\n",
"* M month end frequency\n",
"* BM business month end frequency\n",
"* MS month start frequency\n",
"* BMS business month start frequency\n",
"* Q quarter end frequency\n",
"* BQ business quarter endfrequency\n",
"* QS quarter start frequency\n",
"* BQS business quarter start frequency\n",
"* A year end frequency\n",
"* BA business year end frequency\n",
"* AS year start frequency\n",
"* BAS business year start frequency\n",
"* H hourly frequency\n",
"* T minutely frequency\n",
"* S secondly frequency\n",
"* L milliseconds\n",
"* U microseconds"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"room = \"cellar\"\n",
"plt.figure(figsize=(16,5))\n",
"ax1=plt.subplot()\n",
"ax1.grid()\n",
"ax1.set_ylabel(u'Wh')\n",
"plt.title(room + \" electricity\")\n",
"df.plot(ax=ax1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now you can easily calculate power by taking the difference between values."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"P = df.diff()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"room = \"cellar\"\n",
"plt.figure(figsize=(16,5))\n",
"ax1=plt.subplot()\n",
"ax1.grid()\n",
"ax1.set_ylabel(u'W')\n",
"plt.title(room + \" electricity\")\n",
"P.plot(ax=ax1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Mind your dimensions! Since we had Wh values resampled to hourly basis, the `diff` gets us Wh/h, or W! If you choose another resolution you need to bring a factor into account."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Export to CSV, Excel..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"P.to_csv('tmpo_example.csv')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"P.to_excel('tmpo_example.xlsx')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Pandas has many other export methods, see https://pandas.pydata.org/pandas-docs/stable/io.html"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
Expand All @@ -139,5 +304,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 0
"nbformat_minor": 1
}
Loading