-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
daily processing pipeline #405
base: master
Are you sure you want to change the base?
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #405 +/- ##
==========================================
- Coverage 75.83% 71.52% -4.31%
==========================================
Files 68 74 +6
Lines 7254 7678 +424
==========================================
- Hits 5501 5492 -9
- Misses 1753 2186 +433 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks ok to me.
I'm concerned about out use of context mangers and multiple threads - I don't think they are natively thread aware. Might not be an issue for this PR but brought to my attention based on our conversation the other day and this PR
# self.cur.execute('''CREATE INDEX if not exists processed_flare_products_idx ON | ||
# processed_flare_products (flareid, flarelist, version, name, | ||
# level, type)''') | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this need to be uncommented?
if __name__ == '__main__': | ||
flm = SDCFlareListManager(Path("./flarelist.scv"), update=False) | ||
print(flm) | ||
print(flm.flarelist) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if __name__ == '__main__': | |
flm = SDCFlareListManager(Path("./flarelist.scv"), update=False) | |
print(flm) | |
print(flm.flarelist) |
@@ -30,6 +30,7 @@ install_requires = | |||
sunpy>=4.0.1 | |||
watchdog | |||
requests | |||
stixdcpy @ git+ssh://[email protected]/i4ds/stixdcpy.git |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a tag or version we can set here? This will just install what ever is on master
Also I see the tests are failing? |
create a second processing pipeline to handle not TM packets triggered processing steps
this pipeline runs on a cron bases (once a day?) and performs product creations (fits files)
first included processing is HK-L+ to ANC-ASPECT files