Fixing bug with running test_nem in timezones ahead of Melbourne

[ADGEfficiency]

Reworked how start_datetime is calculated in testing - now uses a 10 hour offset from UTC rather than assuming user is in Melbourne.

[jufemaiz]

Thinking out loud, how would people think about introducing NEMTIME:

# nemweb/timezone.py
from dateutil.tz import tzrange

NEMTIME = tzrange('NEM', 36000)

This would then provide:

import datetime
from nemweb import timezone as tz

time = datetime.datetime.now().astimezone(tz.NEMTIME)

t.strftime('%Z')
# 'NEM'

Ref: https://dateutil.readthedocs.io/en/stable/tz.html#dateutil.tz.tzrange

[rockinrobstar]

I am currently building additional changes so that data can also be pulled from http://data.wa.aemo.com.au/. Hopefully should have a first PR this weekend.

As part of those changes - I needed to start using specific numpy/panda dtypes for the tables to speed up the processing, as the data is in fewer large csv files as opposed to many smaller ones as it is for the NEM. Unfortunately, many of the columns are datetimes - but as we are only using SQLite at the moment - I have just been treating these as strings, but panda does have support for datetimes that includes the timezone, so I see a common interest in defining it. (Tangent: was wondering if its worth defining the dtypes for the columns in a new db table to get around the SQLite limitations).

In saying that I haven't really heard of AEMO time before; I have heard of NEM Time however (eg http://v6.nem-review.info/use/enjoy/data/datasets/glossary/index.aspx) so maybe could I suggest just calling it NEMTZ instead? (AEMO also manage the SWIS and possibly the NWIS soon, so AEMO time doesn't really make much sense to me. (Note: I am a biased West Australian who wants his fair share of AEMO as much as the GST :-P ))

[dylanjmcconnell]

Hey @rockinrobstar - I agree with you re: NEM tz vs AEMO tz (...I think 'NEM time' is fairly common term with in the industry).

Just a note on SQLite - the backend of the actual openNEM website / the version of this package that is in production does actually use mysql. Basically the current version on github is a minimum version that doesn't require someone to set up a proper sql server (am currently refactoring some of the sql code so you can choose either, if you so wish).

P.s. - if you are interested, there is a slack channel I could invite you to?

[jufemaiz]

NEMTIME!

We had to do that for Origin Energy with integration at $LastJob. https://github.com/sjauld/nemtime

[rockinrobstar]

hey @dylanjmcconnell. Yes, would love to join the slack channel to discuss things. My slack user id is: robinp