Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RTDS provider fixes and improvements #12

Merged
merged 36 commits into from
Apr 30, 2024

Conversation

GhostofGoes
Copy link
Collaborator

@GhostofGoes GhostofGoes commented Sep 28, 2023

Updates to the RTDS provider.

  • Initial work to support UDP for C37.118. Unfortunately, pyPMU doesn't have proper UDP support yet, though I think it's sititng in a branch on there. I just need to test and merge it at some point.
    • new config field: rtds-pmu-protocols (for now, only tcp does anything)
  • Update method of getting the bennu version for the new way of versioning by commit ID. This goes into the Elasticsearch data in agent.version.
  • Elasticsearch: fix index dating at midnight boundary by using the date from the message's timestamp
  • Workaround for RTDS time drift issue
  • Ensure all timestamps are in UTC timezone and use timezone-aware datetime.datetime objects
  • Performance optimizations and tweaks
    • GTNET-SKT: Don't log every GTNET-SKT tag update
    • GTNET-SKT: Log all GTNET-SKT values, only when debugging
    • GTNET-SKT: calculate all the typecasts, THEN update the state all at once (instead of taking the lock 20 times and incrementally updating the state...)
    • Move a lot of logging to if statements so we avoid function call
    • asynchronous logging so log calls don't block threads (especially for file writes)
  • Improve type annotations
  • Minor tweaks to documentation, comments, and code cleanup

The license file was already included in the source tree.
However, it wasn't listed in NOTICE.
- Remove PMU sequence numbers.
They ended up being wildly inaccurate thanks to threading.
- Apply a field type mapping when creating Elasticsearch indices
- Handle Elastic bulk ingest error messages
- Add 'elastic-index-basename' option to configure
the Elasticsearch index name.
- Fix PMU connection rebuilding.
If RSCAD case is stopped, the provider no longer needs to be restarted and will properly restart the PMU connections.
This fixes a long-standing issue that's been around since HARMONIE.
- Improve PMU handling.
RTDS provider is now able to be started AFTER the RSCAD case is started.
- Change gtnet-skt to continually send data over UDP
- Add 'gtnet-skt-tcp-retry-delay' config option
- Add 'gtnet-skt-udp-write-rate' config option
- Move Elasticsearch processing and pushing into a thread.
This prevents it from blocking the core functionality of
sending data to SCEPTRE.
- Improve how boolean config options are handled
- Move RTDS data processing into a thread
tox hasn't been used by pybennu for a very, very long time
Git hashes can't be used for the Version field in package metadata
It isn't used anywhere. It's a pyinstaller spec but pyinstaller hasn't been used in a long time.
- Remove unused variables and stages
- Fix indentation
- Consolidate and update help messages
- Remove requirement for root from several commands
- Add 'rtds-pmu-protocols' to config.ini
- Add ability to use UDP for PMUs in the RTDS
- Totally untested right now, will be tested soon
- GTNET-SKT: Don't log every GTNET-SKT tag update
- GTNET-SKT: Log all GTNET-SKT values, only when debugging
- GTNET-SKT: calculate all the typecasts, THEN update the state all at once (instead of taking the lock 20 times and incrementally updating the state...)
- Move a lot of logging to if statements so we avoid function call
- asynchronous logging so log calls don't block threads (especially for file writes)
@GhostofGoes GhostofGoes marked this pull request as ready for review April 22, 2024 19:05
@GhostofGoes GhostofGoes self-assigned this Apr 22, 2024
@GhostofGoes GhostofGoes changed the title RTDS new development RTDS provider fixes and improvements Apr 22, 2024
@mgaliar mgaliar merged commit e4cfe0e into sandialabs:main Apr 30, 2024
1 check passed
@GhostofGoes GhostofGoes deleted the rtds-udp-pmu branch May 1, 2024 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants