Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EIA client #109

Merged
merged 58 commits into from
Jun 2, 2017
Merged

EIA client #109

merged 58 commits into from
Jun 2, 2017

Conversation

andydevlinsmith
Copy link
Contributor

@andydevlinsmith andydevlinsmith commented Feb 20, 2017

This addresses #97.

This is my first PR for this project, so constructive feedback would be welcome.

Caveats:

  • We may need to add an EIA key to the CI tool for the tests to pass. I can give you mine or we can set one up for Watttime and use that.
  • I hit some issues pulling Canada/Mexico data, so this just includes US BAs for now.
  • Some BAs don’t consistently return some data, as documented in the problem_bas lists in the tests. This probably needs more graceful handling.
  • The setup for tests was different enough that I found it easiest to put everything up in test_eia rather than incorporate them into test_load/test_genmix/test_trade.
  • Generally speaking, I'm sure this can improve but figured I would get some input before going further.

Thanks!

Copy link
Contributor

@ndavis6 ndavis6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andydevlinsmith Thanks for this great PR. I have provided some feedback in comments that I hope is helpful. Let me know if anything doesn't make sense. The test suite fails on JAE for me, did you have that issue?

'DOPD': {'class': 'EIACLIENT', 'module': 'eia_esod'},
'DUK': {'class': 'EIACLIENT', 'module': 'eia_esod'},
'EEI': {'class': 'EIACLIENT', 'module': 'eia_esod'},
'ELE': {'class': 'SVERIClient', 'module': 'sveri'},
Copy link
Contributor

@ndavis6 ndavis6 Feb 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kudos for following our existing naming conventions, but I think this is starting to get confusing. Maybe a better way would be to just have an EIA client that took the name of the balancing authority as an argument to the methods? Or had a set_BA function? @marcpare thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a set_BA function to the class.

LOGGER.error('No end_at date provided')
raise ValueError('You must specify an end_at date.')

def handle_options(self, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of this is done by base.handle_options that can be called with super(EIAClient, self).handle_options(**kwargs). Take a look at caiso.handle_options for how we have handled this in the past. If base.handle_options doesn't do everything, you can add the extra handling in here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think I have this fairly well trimmed down now.

@@ -0,0 +1,645 @@
import unittest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for a great test suite! Comments and answers to some of your questions below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ndavis6, unless I'm overlooking something I think I've addressed all the comments here. Please let me know if you see any other issues, particularly with the set_ba function I added to eia_esod.py. Thanks.

# return
return data

# this is repeated- super it?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super it. Super all of them. You could merge these two tests with one extra argument. It looks like the differences are the set of keys ['load_MW', ... vs ['ba_name',... and the keys 'load_MW' and 'net_exp_MW' when checking for numeric value.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done for _run_test and _run_null_response_test. I'll see if there are others I can combine to clean things up a bit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combined a few more tests here.

data = c.get_trade(**kwargs)

# test number
self.assertGreaterEqual(len(data), 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a msg='BA is %s' % ba_name to the assert arguments. If you add ``self.longMessage = TrueinTestEIA.setUp``` it adds this message to the original assertion message, That let's the user know which BA is failing (or at least which is the first BA to fail), so that it is clear if a single BA is causing all the failures.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!


if self.options['data'] == 'gen':
if self.options['forecast']:
LOGGER.error('Forecast generation data error for %s' % self.NAME)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would change Logger.error message here and line 221 to "forecast not supported for generation' to match the msg in ValueError.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

from pyiso import LOGGER


class EIACLIENT(BaseClient):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, but we only capitalize the first letter of non-acronyms (EIAClient)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@andydevlinsmith
Copy link
Contributor Author

Many thanks for the comments! I'll get these in and update the PR.

@ndavis6 ndavis6 merged commit 3db23c5 into WattTime:master Jun 2, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants