diff --git a/CITATION.cff b/CITATION.cff index a8ed34f..aa8b32b 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -1,8 +1,8 @@ cff-version: 1.2.0 message: "If you use this software in your research, please cite it using these metadata." title: Ziggy -version: v0.4.1 -date-released: "2023-11-21" +version: v0.5.0 +date-released: "2024-03-26" abstract: "Ziggy, a portable, scalable infrastructure for science data processing pipelines, is the child of the Transiting Exoplanet Survey Satellite (TESS) pipeline and the grandchild of the Kepler Pipeline." authors: - family-names: Tenenbaum diff --git a/README.md b/README.md index b00caa8..8166370 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ Ziggy is a collaboration between the Science Processing Group in NASA’s Advanc ## License -Copyright © 2022-2023 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved. +Copyright © 2022-2024 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved. NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline Management System for Data Analysis Pipelines, under Cooperative Agreement Nos. NNX14AH97A, 80NSSC18M0068 & 80NSSC21M0079. diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md index bc49fe9..83cd1c3 100644 --- a/RELEASE-NOTES.md +++ b/RELEASE-NOTES.md @@ -2,15 +2,181 @@ # Ziggy Release Notes -These are the release notes for Ziggy. While we're able to provide links to GitHub issues, we are not able to provide links to our internal Jira issues. +These are the release notes for Ziggy. In the change log below, we'll refer to our internal Jira key for our benefit. If the item is associated with a resolved GitHub issue or pull request, we'll add a link to that. Changes that are incompatible with previous versions are marked below. While the major version is 0, we will be making breaking changes when bumping the minor number. However, once we hit version 1.0.0, incompatible changes will only be made when bumping the major number. -## 0.2.0 +## v0.5.0: A major overhaul of the datastore, some UI improvements, and documentation for the command-line interface + +The title pretty much says it all. There was also a lot of internal refactoring to buy down more technical debt. + +### New Features + +1. Review javadoc warnings (ZIGGY-126) +1. Remove GUI code from ProcessHeartbeatManager (ZIGGY-261) +1. Move remote execution configuration from parameters to pipeline node definition (ZIGGY-280) +1. Remove timestamp from pipeline instance name when launched by event handler (ZIGGY-299) +1. Add architecture and queue information to remote execution dialog (ZIGGY-306) +1. Increase number of digits displayed by cost estimate (ZIGGY-352) +1. Functionality inherited from Spiffy (ZIGGY-364) +1. Create ZiggyCliTest and add documentation to user manual (ZIGGY-370) +1. Remove security packages (ZIGGY-371) +1. Eliminate default from unit of work machinery (ZIGGY-374) +1. Refactor Parameters a bit more (ZIGGY-378) +1. Redesign datastore and data file type APIs (ZIGGY-380) +1. Implement subqueries in ZiggyQuery (ZIGGY-381) +1. Simplify ZiggyConfiguration and reduce test fixtures (ZIGGY-382) + +### Bug Fixes + +1. Max worker count not set correctly (ZIGGY-302) +1. Timestamp fields not initialized (ZIGGY-375) +1. Reserved queue dialog issues (ZIGGY-377) +1. Exceptions running ziggy.pl with missing properties (ZIGGY-391) +1. Ziggy supervisor logging no longer goes to supervisor.log (ZIGGY-392) + +## v0.4.1: Fixed halt task and restart task commands + +As promised, the Halt tasks (formerly Kill tasks) and Restart tasks commands have been fixed, along with a handful other buglets. + +### New Features + +### Bug Fixes + +1. Remove option of UOW reuse (Incompatible change, ZIGGY-278) +1. Replace == with equals for non-primitives (ZIGGY-287) +1. Can't kill (local) tasks (ZIGGY-290) +1. Can't restart tasks (ZIGGY-291) +1. Incorrect number of downstream tasks (ZIGGY-303) +1. Increase pause after starting database (ZIGGY-354) +1. Undesired reprocessing (ZIGGY-361) +1. Tests sometimes fail in Eclipse (ZIGGY-367) +1. Unable to kill or restart tasks for first pipeline instance (ZIGGY-368) +1. Close and Cancel buttons in wrong order on resources dialogs (ZIGGY-372) +1. No transition after error resume (ZIGGY-373) + +## v0.4.0: Hibernate 6, reorganized properties, an improved UI, runjava renamed to ziggy + +Last time we said that our next release will contain the result of replacing our TESS data analysis pipeline infrastructure with Ziggy. That work continues, so we'll try to get back into a regular cadence of releases to avoid astronomically large releases, like this one. + +As promised, we reorganized our properties and eliminated the effects of 15 years of entropy. We also updated the version of Hibernate we use and updated the database schema. + +The UI witnessed a major facelift, and `runjava` was renamed to `ziggy` and the `--help` option works consistently with ziggy and its commands. The sample pipeline now uses an internal HSQLDB database, so it's even easier to try. + +The version generator was redone to avoid rebuilding the world every time. Third-party sources have been moved from `buildSrc` to `outside,` where they are still safe from `gradle clean`. + +The `Kill tasks` and `Restart tasks` commands are broken and will be fixed shortly in 0.4.1. + +### New Features + +1. Switch to Hibernate 6 (Incompatible change, ZIGGY-5) +1. Add additional queue support (ZIGGY-92) +1. Review handling of checked exceptions (ZIGGY-149) +1. Clean up of StateFile name / path management (ZIGGY-152) +1. Respond to requested changes on Ziggy remote execution dialog (ZIGGY-167) +1. Reorganization of console tabs and content (ZIGGY-169) +1. Clean up properties (Incompatible change, ZIGGY-172) +1. Replace ZiggyVersionGenerator / ZiggyVersion classes (ZIGGY-181) +1. Implement parameter set groups (ZIGGY-221) +1. Revise pipeline priorities (ZIGGY-225) +1. Changes needed for TESS-Ziggy (ZIGGY-240) +1. Implement subworkers (ZIGGY-242) +1. Add per-module thread settings (ZIGGY-243) +1. Switch sample pipeline to HSQLDB (ZIGGY-250) +1. Second Generation Messaging System (ZIGGY-259) +1. Refactor OpsInstancesPanel (ZIGGY-263) +1. Remove BeanWrappers class (ZIGGY-301) + +### Bug Fixes + +1. runjava --help should provide help for (ZIGGY-155) +1. Incorrect version of wrapper jarfile (ZIGGY-196) +1. Configuration is mutable and use of system properties is misleading (ZIGGY-201) +1. cluster init --force can't delete write-protected files (ZIGGY-231) +1. Fix Remote Execution typical wall time label (ZIGGY-232) +1. Remote parameter set not updated by remote execution dialog (ZIGGY-233) +1. Ziggy FileUtil.cleanDirectoryTree() fails with symlinks (ZIGGY-236) +1. ZiggyTable wrapping text doesn't support text color (ZIGGY-246) +1. A little time in the great outdoors, er, outside (ZIGGY-247) +1. Improve runjava console user experience (ZIGGY-251) +1. Database classes have ambiguous versioning (ZIGGY-256) +1. Remove task counts from PipelineInstanceNode (ZIGGY-262) +1. Delay task request messages until tasks are committed to database (ZIGGY-269) +1. Unable to start pipeline from selected node (ZIGGY-270) +1. Allow ZiggyCppPojo to use correct compiler for a given source file (ZIGGY-279) +1. Editing a parameter clobbers other parameters in the set (ZIGGY-286) + +## v0.3.1: Fixed CITATION.cff and some news + +This release includes a fix to our CITATION.cff courtesy of @dieghernan. + +We are now working towards replacing our [TESS](https://www.nasa.gov/tess-transiting-exoplanet-survey-satellite) data analysis pipeline infrastructure with Ziggy. Our next release will contain the result of that work. It's a large effort and we expect it to take at least two months if not longer. + +We are still planning to reorganize our properties and eliminate the effects of 15 years of entropy. We'll also be updating the version of Hibernate we use and anticipate updating the database schema as a result. You have been warned! This will occur before the 1.0 release to maximize our chances of stability after that. + +### New Features + +### Bug Fixes + +1. Correct conference section in CITATION.cff (ZIGGY-241, [pull \#1](https://github.com/nasa/ziggy/pull/1)) + +## v0.3.0: Java 17, Gradle 7, and a new event manager + +This release includes an update to the Gradle build system and an upgrade of Java to Java 17. It introduces an event manager system so that Ziggy can respond automatically to external events. The user manual was expanded to cover this feature and a few others. + +We are planning to reorganize our properties and eliminate the effects of 15 years of entropy. We'll also be updating the version of Hibernate we use and anticipate updating the database schema as a result. You have been warned! This will occur before the 1.0 release to maximize our chances of stability after that. + +### New Features + +1. Upgrade to Java 17 (Incompatible change, ZIGGY-22) +1. Add static analysis to build (ZIGGY-30) +1. Create prototype event manager system (ZIGGY-119) +1. Clean up unit test execution on Gradle (ZIGGY-180) +1. Create user manual, part II (ZIGGY-193) +1. Remove Sockets from Subtask Server and Client (ZIGGY-199) +1. Update copyright (ZIGGY-220) + +### Bug Fixes + +1. Fix build warnings in both src and buildSrc (ZIGGY-142) +1. Delete setPosixPermissionsRecursively with a single permission (ZIGGY-197) +1. Ziggy "keep-up" processing fails (ZIGGY-204) +1. Correct P1 bugs identified by SpotBugs (ZIGGY-207) +1. NAS does not support Java 17 (Incompatible change, ZIGGY-234) + +## v0.2.2: More documentation goodness + +This release adds Previous, Next, and Up buttons to the user manual to make it easier to read cover to cover. We also added a CITATION.cff file to make it easier for you to cite Ziggy in your own work. Finally, we changed some 3-byte quotes to ASCII as these quotes could not be compiled if LANG is C. + +We have updated the Gradle build system and Java to Java 17. This change will appear in version 0.3.0. + +### New Features + +1. Add "Prev", "Next", and "Up" buttons to user manual articles (ZIGGY-160) +1. Update GitHub documentation (ZIGGY-200) + +### Bug Fixes + +1. Fix funny quotes in NASA notices (ZIGGY-188) + +## v0.2.1: Add GitHub info to docs + +Once we uploaded our first version to GitHub, we could fill in some documentation TODOs like how to download the code. + +We are still in the process of updating the build system Gradle and Java to at least Java 11 and possibly Java 17. We'll take advantage of post-Java 8 versions at that time. This change will appear in version 0.3.0. + +### New Features + +1. Ziggy reuse (ZIGGY-134) +1. Update GitHub info in manual (ZIGGY-192) + +### Bug Fixes + +## v0.2.0: Initial release This is the first Ziggy release to appear on GitHub. The major number of 0 indicates that we're still refactoring the Kepler and TESS codebases and reserve the right to make breaking changes from time to time as we make the pipeline more widely useful. However, the general pipeline infrastructure has been in production use since Kepler's launch in 2009. -### Resolved issues +This is the first Ziggy release. -In future releases, this section will contain a list of GitHub/Jira issues that were resolved and incorporated into the release. If the resolution for an issue introduced a breaking change, it will be described so that you can update your properties files or pipeline configurations in advance. +We are in the process of updating the build system Gradle and Java to at least Java 11. It's possible that we will take advantage of post-Java 8 versions at that time. This change will appear in version 0.3.0. ## 0.1.0 diff --git a/build.gradle b/build.gradle index 835ef2f..ec54f50 100644 --- a/build.gradle +++ b/build.gradle @@ -45,21 +45,21 @@ dependencies { implementation 'com.github.librepdf:openpdf:1.3.+' implementation 'com.github.spotbugs:spotbugs-annotations:4.7.+' - implementation 'com.google.guava:guava:23.+' + implementation 'com.google.guava:guava:23.6.+' implementation 'com.jgoodies:jgoodies-forms:1.9.+' implementation 'com.jgoodies:jgoodies-looks:2.7.+' implementation 'commons-cli:commons-cli:1.5.+' - implementation 'commons-codec:commons-codec:1.+' + implementation 'commons-codec:commons-codec:1.16.+' implementation 'commons-io:commons-io:2.11.+' - implementation 'jakarta.xml.bind:jakarta.xml.bind-api:3.0+' - implementation 'org.apache.commons:commons-collections4:4.+' - implementation 'org.apache.commons:commons-compress:1.+' + implementation 'jakarta.xml.bind:jakarta.xml.bind-api:3.0.+' + implementation 'org.apache.commons:commons-collections4:4.4' + implementation 'org.apache.commons:commons-compress:1.25.+' implementation 'org.apache.commons:commons-configuration2:2.9.+' implementation 'org.apache.commons:commons-csv:1.9.+' - implementation 'org.apache.commons:commons-exec:1.+' + implementation 'org.apache.commons:commons-exec:1.3' implementation 'org.apache.commons:commons-lang3:3.12.+' implementation 'org.apache.commons:commons-math3:3.6.+' - implementation 'org.apache.commons:commons-text:1.+' + implementation 'org.apache.commons:commons-text:1.11.+' implementation 'org.apache.logging.log4j:log4j-core:2.20.+' implementation 'org.apache.logging.log4j:log4j-slf4j2-impl:2.20.+' implementation 'org.hibernate.orm:hibernate-ant:6.2.+' @@ -67,10 +67,15 @@ dependencies { implementation 'org.javassist:javassist:3.29.2-GA' implementation 'org.jfree:jfreechart:1.0.+' implementation 'org.jsoup:jsoup:1.16.+' - implementation 'org.netbeans.api:org-netbeans-swing-outline:+' + implementation 'org.netbeans.api:org-netbeans-swing-outline:RELEASE121' // see note below implementation 'org.slf4j:slf4j-api:2.0.+' implementation 'org.tros:l2fprod-properties-editor:1.3.+' + // The NetBeans library started emitting the error, + // "No SVG loader available for ... columns.svg" + // at version 122 (through version 200). Hold version at 121 until + // this error is fixed or a workaround is discovered. + // Configuration2 declares the following as optional [1]. It's not, so it's added here. // Occasionally, comment out this line--if the tests pass, delete it. // 1. https://github.com/apache/commons-configuration/blob/master/pom.xml @@ -85,11 +90,10 @@ dependencies { // Needed to compile unit tests. testImplementation 'junit:junit:4.13.+' - testImplementation 'org.hamcrest:hamcrest:2.+' + testImplementation 'org.hamcrest:hamcrest:2.2' testImplementation 'org.mockito:mockito-core:3.12.+' // Needed at runtime. - runtimeOnly 'jakarta.xml.bind:jakarta.xml.bind-api:3.0+' runtimeOnly 'org.hibernate.orm:hibernate-hikaricp:6.2.+' runtimeOnly 'org.postgresql:postgresql:42.6.+' @@ -201,6 +205,8 @@ javadoc { options.addBooleanOption("Xdoclint:-missing", true) } +check.dependsOn javadoc + // The SpotBugs plugin adds spotbugsMain and spotbugsTest to the check task. spotbugs { // The SMP requires that all high priority problems are addressed before testing can commence. diff --git a/doc/user-manual/alerts.md b/doc/user-manual/alerts.md index fea19f9..5cebdf7 100644 --- a/doc/user-manual/alerts.md +++ b/doc/user-manual/alerts.md @@ -12,7 +12,7 @@ Ziggy uses alerts to tell the pipeline operator that something has happened that There are two flavors of alert that you're likely to see: warnings and errors. Warnings will turn the alerts stoplight yellow, errors turn it red. The alerts panel shows which task generated the alert, when it happened, and a hopefully-useful message. If there are no alerts, the stoplight will be green. -Sadly, in this case it tells you pretty much what you already knew: task 12 blew up. +Sadly, in this case it tells you pretty much what you already knew: tasks 8 and 9 blew up. ### Acknowledging Alerts diff --git a/doc/user-manual/configuring-pipeline.md b/doc/user-manual/configuring-pipeline.md index 7940071..c3acae6 100644 --- a/doc/user-manual/configuring-pipeline.md +++ b/doc/user-manual/configuring-pipeline.md @@ -32,7 +32,7 @@ The issues described above are collectively the "pipeline configuration." This i [Module Parameters](module-parameters.md) -[Data File Types](data-file-types.md) +[The Datastore](datastore.md) [Pipeline Definition](pipeline-definition.md) @@ -128,8 +128,6 @@ In this case the algorithm code doesn't return anything because it writes its ou Anyway, moving on to the last chunk of the Python-side "glue" code, we see this: -Anyway, moving on to the last chunk of the Python-side "glue" code, we see this: - ```python # Sleep for a user-specified interval. This is here just so the # user can watch execution run on the pipeline console. diff --git a/doc/user-manual/console-cli.md b/doc/user-manual/console-cli.md new file mode 100644 index 0000000..00a460b --- /dev/null +++ b/doc/user-manual/console-cli.md @@ -0,0 +1,100 @@ + + +[[Previous]](event-handler-labels.md) +[[Up]](user-manual.md) +[[Next]](dusty-corners.md) + +## The Console Command-line Interface (CLI) + +The command-line interface for the console contains enough functionality to start, stop, and view pipelines. Let's start by displaying the help for the console. + +```console +$ ziggy console --help +usage: ZiggyConsole command [options] + +Commands: +cancel Cancel running pipelines +config --configType TYPE [--instance ID | --pipeline NAME] + Display pipeline configuration +display [[--displayType TYPE] --instance ID | --task ID] + Display pipeline activity +log --task ID | --errors + Request logs for the given task(s) +reset --resetType TYPE --instance ID + Put tasks in the ERROR state so they can be restarted +restart --task ID ... Restart tasks +start PIPELINE [NAME [START_NODE [STOP_NODE]]] + Start the given pipeline and assign its name to NAME + (default: NAME is the current time, and the NODES are + the first and last nodes of the pipeline respectively) +version Display the version (as a Git tag) + +Options: + -c,--configType Configuration type (data-model-registry | instance | pipeline | + pipeline-nodes) + -d,--displayType Display type (alerts | errors | full | statistics | statistics-detailed) + -e,--errors Selects all failed tasks + -h,--help Show this help + -i,--instance Instance ID + -p,--pipeline Pipeline name + -r,--resetType Reset type (all | submitted) + -t,--task Task ID +``` + +### Commands + +We'll cover each command in turn. + +**cancel** + +This command is currently broken. It will be renamed to halt and given the same semantics as the halt commands in the GUI in a future version. + +**config --configType TYPE [--instance ID | --pipeline NAME]** + +Display pipeline configuration. The four configuration types that can be displayed are `data-model-registry`, `instance`, `pipeline`, and `pipeline-nodes`. The `data-model-registry` type displays the content of the known models. The `instance` type displays details for all of the pipeline instances, including parameter sets and module definitions. Use the `--instance` option to limit the display to the given instance. The `pipeline` type displays details for all of the pipeline definitions, including parameter sets and module definitions. Use the `--pipeline` option to limit the display to the given pipeline. Finally, the `pipeline-nodes` type displays a short list of the nodes for the pipeline named with the `--pipeline` option. + +```console +$ ziggy console config --configType pipeline --pipeline sample +``` + +**display [[--displayType TYPE] --instance ID | --task ID]** + +Display pipeline activity. When the command appears by itself, a table of instances is shown. If an instance ID is provided, then instance and task summaries are shown. Use the `displayType` option to increase the level of detail or to show other elements. The `alerts` and `errors` types will show those elements associated with the given instance respectively. The `full` option adds a table of the tasks. The `statistics` option shows timing information for each task. The `statistics-detailed` option is similar, but a PDF is generated. + +```console +$ while true; do ziggy console display --instance 2 --displayType full; sleep 15; done +``` + +**log --task ID | --errors** + +Request logs for the given task(s). This command is not yet implemented. + +**reset --resetType TYPE --instance ID** + +This command is currently broken. It will be renamed to halt and given the same semantics as the halt commands in the GUI in a future version. + +**restart restart --task ID ...** + +Restart tasks. Multiple `--task options` may be given. Tasks are started from the beginning. This command only has effect on tasks in the ERROR state. + +```console +$ ziggy console restart --task 2 --task 3 +``` + +**start PIPELINE [NAME [START_NODE [STOP_NODE]]]** + +Start the given PIPELINE and assign its name to NAME. If NAME is omitted, the pipeline will be named with the current time. The start and stop nodes can be provided, but if they are not, the first and last nodes of the pipeline are used instead. + +```console +$ ziggy console start sample "Test 1" +$ ziggy console start sample "Test 2" permuter flip +``` + +**version** + +Display the version (as a Git tag). + + +[[Previous]](event-handler-labels.md) +[[Up]](user-manual.md) +[[Next]](dusty-corners.md) diff --git a/doc/user-manual/contact-us.md b/doc/user-manual/contact-us.md index c65b6cd..61c04ec 100644 --- a/doc/user-manual/contact-us.md +++ b/doc/user-manual/contact-us.md @@ -1,6 +1,6 @@ -[[Previous]](edit-pipeline.md) +[[Previous]](nicknames.md) [[Up]](user-manual.md) [[Next]](properties.md) @@ -15,6 +15,6 @@ If you just want to send an email, we are: Peter Tenenbaum (but you can call him PT)
Bill Wohler -[[Previous]](edit-pipeline.md) +[[Previous]](nicknames.md) [[Up]](user-manual.md) [[Next]](properties.md) diff --git a/doc/user-manual/data-file-types.md b/doc/user-manual/data-file-types.md deleted file mode 100644 index 1821433..0000000 --- a/doc/user-manual/data-file-types.md +++ /dev/null @@ -1,141 +0,0 @@ - - -[[Previous]](module-parameters.md) -[[Up]](configuring-pipeline.md) -[[Next]](pipeline-definition.md) - -## Data File Types - -As the user, one of your jobs is to define, for Ziggy, the file naming patterns that are used for the inputs and outputs for each algorithm, and the file name patterns that are used for instrument models. The place for these definitions is in data file type XML files. These have names that start with "pt-" (for "Pipeline Data Type"); in the sample pipeline, the data file type definitions are in [config/pt-sample.xml](../../sample-pipeline/conf/pt-sample.xml). - -Note that when we talk about data file types, we're not talking about data file formats (like HDF5 or geoTIFF). Ziggy doesn't care about data file formats; use whatever you like, as long as the algorithm software can read and write that format. - -### The Datastore and the Task Directory - -Before we get too deeply into the data file type definitions, we need to have a brief discussion about two directories that Ziggy uses: the datastore, on the one hand, and the task directories, on the other. - -#### The Datastore - -"Datastore" here is just a $10 word for an organized directory tree where Ziggy keeps the permanent copies of its various kinds of data files. Files from the datastore are provided as inputs to the algorithm modules; when the modules produce results, those outputs are transferred back to the datastore. - -Who defines the organization of the datastore? You do! The organization is implicitly defined when you define the data file types that go into, and come out of, the datastore. This will become clear in a few paragraphs (at least I hope it's clear). - -#### The Task Directory - -Each processing activity has its own directory, known as the "task directory." The task directory is where the algorithm modules look to find the files they operate on, and it's where they put the files they produce as results. Unlike the datastore, these directories are transient; once processing is complete, you can feel free to delete them at some convenient time. In addition, there are some other uses that benefit from the task directory. First, troubleshooting. In the event that a processing activity fails, you have in one place all the inputs that the activity uses, so it's easy to inspect files, watch execution, etc. In fact, you can even copy the task directory to some other system (say, your laptop) if that is a more convenient place to do the troubleshooting! Second, and relatedly, the algorithm modules are allowed to write files to the task directory that aren't intended to be persisted in the datastore. This means that the task directory is a logical place to put files that are used for diagnostics or troubleshooting or some other purpose, but which you don't want to save for posterity in the datastore. - -#### And My Point Is? - -The key point is this: the datastore can be, and generally is, heirarchical; the task directory is flat. Files have to move back and forth between these two locations. The implications of this are twofold. First, **the filenames used in the datastore and the task directory generally can't be the same.** You can see why: because the datastore is heirarchical, two files that sit in different directories can have the same name. If those two files are both copied to the task directory, one of them will overwrite the other unless the names are changed when the files go to the task directory. - -Second, and relatedly, **the user has to provide Ziggy with some means of mapping the two filenames to one another.** Sorry about that; but the organization of the datastore is a great power, and with great power comes great responsibility. - -### Mission Data - -Okay, with all that throat-clearing out of the way, let's take a look at some sample data file type definitions. - -```xml - - - -``` - -Each data file type has a name, and that name can have whitespace in it. That much makes sense. - -#### fileNameRegexForTaskDir - -This is how we define the file name that's used in the task directory. This is a [Java regular expression](https://docs.oracle.com/en/java/javase/12/docs/api/java.base/java/util/regex/Pattern.html) (regex) that the file has to conform to. For `raw data`, for example, a name like `some-kinda-name-set-1-file-9.png` would conform to this regular expression, as would `another_NAME-set-4-file-3.png`, etc. - -#### fileNameWithSubstitutionsForDatastore - -Remember that the task directory is a flat directory, while the datastore can be heirarchical. This means that each part of the path to the file in the datastore has to be available somewhere in the task directory name, and vice-versa, so that the two can map to each other. - -In the `fileNameWithSubstitutionsForDatastore`, we accomplish this mapping. The way that this is done is that each "group" (one of the things in parentheses) is represented with $ followed by the group number. Groups are numbered from left to right in the file name regex, starting from 1 (group 0 is the entire expression). In raw data, we see a value of `$2/L0/$1-$3.png`. This means that group 2 is used as the name of the directory under the datastore root; `L0` is the name of the next directory down; and groups 1 and 3 are used to form the filename. Thus, `some-kinda-name-set-1-file-9.png` in the task directory would translate to `set-1/L0/some-kinda-name-file-9.png` in the datastore. - -Looking at the example XML code above, you can (hopefully) see what we said about how you would be organizing the datastore. From the example, we see that the directories immediately under the datastore root will be `set-0, set-1`, etc. Each of those directories will then have, under it, an `L0` directory and an `L1` directory. Each of those directories will then contain PNG files. - -Notice also that the filenames of `raw data` files and `permuted colors` files in the datastore can potentially be the same! This is allowed because the `fileNameWithSubstitutionsForDatastore` values show that the files are in different locations in the datastore, and the `fileNameRegexForTaskDir` values show that their names in the task directory will be different, even though their names in the datastore are the same. - -### Instrument Model Types - -Before we can get into this file type definition, we need to answer a question: - -#### What is an Instrument Model, Anyway? - -Instrument models are various kinds of information that are needed to process the data. These can be things like calibration constants; the location in space or on the ground that the instrument was looking at when the data was taken; the timestamp that goes with the data; etc. - -Generally, instrument models aren't the data that the instrument acquired (that's the mission data, see above). This is information that is acquired in some other way that describes the instrument properties. Like mission data, instrument models can use any file format that the algorithm modules can read. - -#### Instrument Model Type Definition - -Here's our sample instrument model type definition: - -​ `` - -As with the data file types, model types are identified by a string (in this case, the `type` attribute) that can contain whitespace, and provides a regular expression that can be used to determine whether any particular file is a model of the specified type. In this case, in a fit of no-imagination, the regex is simply a fixed name of `sample-model.txt`. Thus, any processing algorithm that needs the `dummy model` will expect to find a file named `sample-model.txt` in its task directory. - -#### Wait, is That It? - -Sadly, no. Let's talk about model names and how they fit into all of this. - -##### Datastore Model Names - -Ziggy permanently stores every model of every kind that is imported into it. This is necessary because someday you may need to figure out what model was used for a particular processing activity, but on the other hand it may be necessary to change the model as time passes -- either because the instrument itself changes with time, or because your knowledge of the instrument changes (hopefully it improves). - -But -- in the example above, the file name "regex" is a fixed string! This means that the only file name that Ziggy can possibly recognize as an instance of `dummy model` is `sample-model.txt`. So when I import a new version of `sample-model.txt` into the datastore, what happens? To answer that, let's take a look at the `dummy model` subdirectory of the `models` directory in the datastore: - -```console -models$ ls "dummy\ model" -2022-10-31.0001-sample-model.txt -models$ -``` - -(Yes, I broke my own strongly-worded caution against using whitespace in names, and in a place where it matters a lot -- a directory name! Consistency, hobgoblins, etc.) - -As you can see, the name of the model in the datastore isn't simply `sample-model.txt`. It's had the date of import prepended, along with a version number. By making these changes to the name, Ziggy can store as many versions of a model as it needs to, even if the versions all have the same name at the time of the import. - -##### Task Directory Model Names - -Ziggy also maintains a record of the name the model file had at the time of import. When the model is provided to the task directory so the algorithms can use it, this original name is restored. This way, the user never needs to worry about Ziggy's internal renaming conventions; the algorithms can use whatever naming conventions the mission uses for the model files, even if the mission reuses the same name over and over again. - -##### Which Version is Sent to the Algorithms? - -The most recent version of each model is the one provided to the algorithms at runtime. If there were 9 different models in `dummy model`, the one with version number `0009` would be the one that is copied to the task directories. If, some time later, a tenth version was imported, then all subsequent processing would use version `0010`. - -##### What Happens if the Actual Model Changes? - -Excellent question! Imagine that, at some point in time, one or more models change -- not your knowledge of them, the actual, physical properties of your instrument change. Obviously you need to put a new model into the system to represent the new properties of the instrument. But equally obviously, if you ever go back and reprocess data taken prior to the change, you need to use the model that was valid at that time. How does Ziggy handle that? - -Answer: Ziggy always, *always* provides the most recent version of the model file. If you go and reprocess, the new processing will get the latest model. In order to properly represent a model that changes with time, **the changes across time must be reflected in the most recent model file!** Also, and relatedly, **the algorithm code must be able to pull model for the correct era out of the model file!** - -In practice, that might mean that your model file contains multiple sets of information, each of which has a datestamp; the algorithm would then go through the file contents to find the set of information with the correct datestamp, and use it. Or, it might mean that the "model" is values measured at discrete times that need to be interpolated by the algorithm. How the time-varying information is provided in the model file is up to you, but if you want to have a model that does change in time, this is how you have to do it. - -##### Model Names with Version Information - -The above example is kind of unrealistic because in real life, a mission that provides models that get updated will want to put version information into the file name; if for no other reason than so that when there's a problem and we need to talk about a particular model version, we can refer to the one we're concerned about without any confusion ("Is there a problem with sample model?" "Uh, which version of sample model?" "Dunno, it's just called sample model."). Thus, the file name might contain a timestamp, a version number, or both. - -If the model name already has this information, it would be silly for Ziggy to prepend its own versioning; it should use whatever the mission provides. Fortunately, this capability is provided: - -```xml - -``` - -In this case, the XML attribute `versionNumberGroup` tells Ziggy which regex group it should use as the version number, and the attribute `timestampGroup` tells it which to use as the file's timestamp. When Ziggy stores this model in the `versioned-model` directory, it won't rename the file; it will keep the original file name, because the original name already has a timestamp and a version number. - -In general, the user can include in the filename a version number; a timestamp; or both; or neither. Whatever the user leaves out, Ziggy will add to the filename for internal storage, and then remove again when providing the file to the algorithms. - -##### Models Never Get Overwritten in the Datastore - -One thing about supplying timestamp and version information in the filename is that it gives some additional protection against accidents. **Specifically: Ziggy will never import a model that has the same timestamp and version number as one already in the datastore.** Thus, you can never accidentally overwrite an existing model with a new one that's been accidentally given the same timestamp and version information. - -For models that don't provide that information in the filename, there's no protection against such an accident because there can't be any such protection. If you accidentally re-import an old version of `sample-model.txt`, Ziggy will assume it's a new version and store it with a new timestamp and version number. When Ziggy goes to process data, this version will be provided to the algorithms. - -[[Previous]](module-parameters.md) -[[Up]](configuring-pipeline.md) -[[Next]](pipeline-definition.md) diff --git a/doc/user-manual/data-receipt-display.md b/doc/user-manual/data-receipt-display.md index 9b722a7..1ecea15 100644 --- a/doc/user-manual/data-receipt-display.md +++ b/doc/user-manual/data-receipt-display.md @@ -12,7 +12,7 @@ The console has the ability to display data receipt activities. Select the `Data Double-clicking a row in the table brings up a display of all the files in the dataset: - + Note that the file names are the datastore names. diff --git a/doc/user-manual/data-receipt-execution.md b/doc/user-manual/data-receipt-execution.md index 437f87a..fcd7cf2 100644 --- a/doc/user-manual/data-receipt-execution.md +++ b/doc/user-manual/data-receipt-execution.md @@ -13,7 +13,7 @@ At the highest level, the purpose of data receipt is to take files delivered fro - No files showed up that are not expected. - The files that showed up were not somehow corrupted in transit. - Whoever it was that delivered the files may require a notification that there were no problems with the delivery, so data receipt needs to produce something that can function as the notification. -- The data receipt process needs to clean up after itself. In a nutshell, this means that there is no chance that a future data receipt operation fails because of some debris left from a prior data receipt operation, and that there is no chance that a future data receipt operation will inadvertently re-import files that were already imported. +- The data receipt process needs to clean up after itself. This means that there is no chance that a future data receipt operation fails because of some debris left from a prior data receipt operation, and that there is no chance that a future data receipt operation will inadvertently re-import files that were already imported. The integrity of the delivery is supported by an XML file, the *manifest*, that lists all of the delivered files and contains size and checksum information for each one. After a successful import, Ziggy produces an XML file, the *acknowledgement*, that can be used as a notification to the source of the files that the files were delivered and imported without incident. The cleanup is managed algorithmically by Ziggy. @@ -25,33 +25,51 @@ The sample pipeline's data receipt directory uses a copy of the files from the ` ```console sample-pipeline$ ls data -nasa_logo-set-1-file-0.png -nasa_logo-set-1-file-3.png -nasa_logo-set-2-file-2.png +models sample-pipeline-manifest.xml -nasa_logo-set-1-file-1.png -nasa_logo-set-2-file-0.png -nasa_logo-set-2-file-3.png -nasa_logo-set-1-file-2.png -nasa_logo-set-2-file-1.png +set-1 +set-2 +sample-pipeline$ +``` + +Look more closesly and you'll see that only sample-pipeline-manifest.xml is a regular file. The other files are all directories. Let's look into them and see what's what: + +```bash +sample-pipeline$ ls data/set-1/L0 +nasa-logo-file-0.png +nasa-logo-file-1.png +nasa-logo-file-2.png +nasa-logo-file-3.png +sample-pipeline$ +``` + +If we look at the `set-2` directory, we'll see something analogous. Meanwhile, the `models` directory looks like this: + +```bash +sample-pipeline$ ls data/models sample-model.txt sample-pipeline$ ``` -Most of these files are obviously the files that get imported. But what about the manifest? Here's the contents of the manifest: +From looking at this, you've probably already deduced the two rules of data receipt layout: + +1. The mission data must be in a directory tree that matches the datastore, such that each file's location in the data receipt directory tree matches its destination in the datastore. +2. All model files must be in a `models` directory within the data receipt directory. + +Now let's look at the manifest file: ```xml - - - - - - - - - + + + + + + + + + ``` @@ -107,63 +125,20 @@ In the interest of completeness, here's the content of the acknowledgement file ```xml - - - - - - - - - + + + + + + + + + ``` Note that the manifest file must end with "`-manifest.xml`", and the acknowledgement file will end in "`-manifest-ack.xml`", with the filename prior to these suffixes being the same for the two files. -### Systems that Treat Directories as Data Files - -There may be circumstances in which it's convenient to put several files into a directory, and then to use a collection of directories of that form as "data files" for the purposes of data processing. For example, consider a system where there's a data file with an image, and then several files that are used to background-subtract the data file. Rather than storing each of those files separately, you might put the image file and its background files into a directory; import that directory, as a whole, into the datastore; then supply that directory, as a whole, as an input for a subtask. - -In that case, the manifest still needs to have an entry for each regular file, but in this case the name of the file includes the directory it sits in. Here's what that looks like in this example: - -```xml - - - - - - - - - - - - - -``` - -Now the only remaining issue is how to tell Ziggy to import the files in such a way that each of the `data-#####` directories is imported and stored as a unit. To understand how that's accomplished, let's look back at the data receipt node in `pd-sample.xml`: - -```xml - - - - -``` - -Meanwhile, the definition of the raw data type is in `pt-sample.xml`: - -```xml - -``` - -Taken together, these two XML snippets tell us that data receipt's import is going to import files that match the file name convention for the `raw data` file type. We can do the same thing when the "file" to import is actually a directory. If you define a data file type that has `fileNameRegexForTaskDir` set to `data-[0-9]{5}`, Ziggy will import directory `data-00001` and all of its contents as a unit and store that unit in the datastore, and so on. - -Note that the manifest ignores the fact that import of data is going to treat the `data-#####` directories as the "files" it imports, and the importer ignores that the manifest validates the individual files even if they are in these subdirectories. - ### Generating Manifests Ziggy also comes with a utility to generate manifests from the contents of a directory. Use `ziggy generate-manifest`. This utility takes 3 command-line arguments: diff --git a/doc/user-manual/datastore-regexp.md b/doc/user-manual/datastore-regexp.md new file mode 100644 index 0000000..a718309 --- /dev/null +++ b/doc/user-manual/datastore-regexp.md @@ -0,0 +1,33 @@ + + +[[Previous]](edit-pipeline.md) +[[Up]](ziggy-gui.md) +[[Next]](intermediate-topics.md) + +## The Datastore Control Panel + +If you click on the `Datastore` link in the console's navigation panel, you'll see this: + + + +The first two columns make good sense: we have one `DatastoreRegexp` instance, with name `dataset` and value `set-[0-9]`. The last two columns aren't self-explanatory, but to find out what they are and how they work, double-click the `dataset` row. You'll see the following dialog box: + + + +It looks like you can enter text into these boxes, and indeed you can: + + + +If you now press `Save`, here's what you see back on the main panel: + + + +If you were now to run the sample pipeline, you would notice something interesting: Ziggy only creates one task for each module, and that task is the `set-1` task! What you have done by adding `set-1` as an include regexp is you've added a condition to the `dataset` DatastoreRegexp: when it sweeps through the directories to generate units of work, the `dataset` level directories need to match the `dataset` value but also match its include regexp. + +The exclude regexp, by symmetry, does the opposite: only `dataset` level directories that do not match this regular expression can be included. Rather than setting the include to `set-1`, we could have left the include blank and set the exclude to `set-[02-9]`. + +Going back to the ludicrous example from [the Instances Panel article](instances-panel.md), we can now see how we would go about limiting the pipeline to running only tasks where `guitar` equals `reeves` and `album` equals either `outside` or `stardust`. We would go to the regular expressions panel and set the `guitar` `DatastoreRegexp` include value to `reeves`; we would then set the `album` include regexp to `outside|stardust`. + +[[Previous]](edit-pipeline.md) +[[Up]](ziggy-gui.md) +[[Next]](intermediate-topics.md) \ No newline at end of file diff --git a/doc/user-manual/datastore-task-dir.md b/doc/user-manual/datastore-task-dir.md index 801de3e..ee0a767 100644 --- a/doc/user-manual/datastore-task-dir.md +++ b/doc/user-manual/datastore-task-dir.md @@ -2,7 +2,7 @@ [[Previous]](intermediate-topics.md) [[Up]](intermediate-topics.md) -[[Next]](task-configuration.md) +[[Next]](rdbms.md) ## The Datastore and the Task Directory @@ -28,50 +28,54 @@ datastore$ tree │   └── 2022-10-31.0001-sample-model.txt ├── set-1 │   ├── L0 -│   │   ├── nasa_logo-file-0.png -│   │   ├── nasa_logo-file-1.png -│   │   ├── nasa_logo-file-2.png -│   │   └── nasa_logo-file-3.png +│   │   ├── nasa-logo-file-0.png +│   │   ├── nasa-logo-file-1.png +│   │   ├── nasa-logo-file-2.png +│   │   └── nasa-logo-file-3.png │   ├── L1 -│   │   ├── nasa_logo-file-0.png -│   │   ├── nasa_logo-file-1.png -│   │   ├── nasa_logo-file-2.png -│   │   └── nasa_logo-file-3.png +│   │   ├── nasa-logo-file-0.perm.png +│   │   ├── nasa-logo-file-1.perm.png +│   │   ├── nasa-logo-file-2.perm.png +│   │   └── nasa-logo-file-3.perm.png │   ├── L2A -│   │   ├── nasa_logo-file-0.png -│   │   ├── nasa_logo-file-1.png -│   │   ├── nasa_logo-file-2.png -│   │   └── nasa_logo-file-3.png +│   │   ├── nasa-logo-file-0.fliplr.png +│   │   ├── nasa-logo-file-1.fliplr.png +│   │   ├── nasa-logo-file-2.fliplr.png +│   │   └── nasa-logo-file-3.fliplr.png │   ├── L2B -│   │   ├── nasa_logo-file-0.png -│   │   ├── nasa_logo-file-1.png -│   │   ├── nasa_logo-file-2.png -│   │   └── nasa_logo-file-3.png +│   │   ├── nasa-logo-file-0.flipud.png +│   │   ├── nasa-logo-file-1.flipud.png +│   │   ├── nasa-logo-file-2.flipud.png +│   │   └── nasa-logo-file-3.flipud.png │   └── L3 -│   └── averaged-image.png +│   └── nasa-logo-averaged.png └── set-2 - datastore$ +datastore$ ``` Summarizing what we see: - a `models` directory, with a `dummy model` subdirectory and within that a sample model. - A `set-1` directory and a `set-2` directory. The `set-2` directory layout mirrors the layout of `set-1`; take a look if you don't believe me, I didn't bother to expand set-2 in the interest of not taking up too much space. -- Within `set-1` we see a directory `L0` with some PNG files in it, a directory `L1` with some PNG files, and then `L2A`, `L2B`, and `L3` directories which (again, trust me or look for yourself) contain additional PNG files. +- Within `set-1` we see a directory `L0` with some PNG files in it, a directory `L1` with some PNG files, and then `L2A`, `L2B`, and `L3` directories which contain additional PNG files. Where did all this come from? Let's take a look again at part of the `pt-sample.xml` file: ```xml - - - + + + + + ``` -If you don't remember how data file type definitions worked, feel free to [go to the article on Data File Types](data-file-types.md) for a quick refresher course. In any event, you can probably now see what we meant when we said that the data file type definitions implicitly define the structure of the datastore. The `set-1/L0` and `set-2/L0` directories come from the `fileNameWithSubstitutionsForRegex` value for raw data; similarly the permuted colors data type defines the `set-1/L1` and `set-2/L2` directories. +If you don't remember how data file type definitions worked, feel free to [go to the article on The Datastore](datastore.md) for a quick refresher course. What you can see is that, as advertised, data file type `raw data` has a location that points to the `L0` directory in the datastore, and files with the name convention `"nasa-logo-file-[0-9]\.png"`. Similarly, the files in the `L1` directory have file names that match the `fileNameRegexp` for the `permuted colors` data file type. #### Model Names in the Datastore @@ -125,30 +129,34 @@ PBS_JOB_FINISH.1667003320029 permuter-inputs.h5 st-2 PBS_JOB_START.1667003280021 st-0 st-3 1-2-permuter/st-0: -SUB_TASK_FINISH.1667003287519 nasa_logo-set-2-file-0-perm.png permuter-inputs-0.h5 sample-model.txt -SUB_TASK_START.1667003280036 nasa_logo-set-2-file-0.png permuter-stdout-0.log +SUB_TASK_FINISH.1667003287519 nasa_logo-file-0.perm.png permuter-inputs.h5 +sample-model.txt +SUB_TASK_START.1667003280036 nasa_logo-file-0.png permuter-stdout.log 1-2-permuter/st-1: -SUB_TASK_FINISH.1667003294982 nasa_logo-set-2-file-1-perm.png permuter-inputs-0.h5 sample-model.txt -SUB_TASK_START.1667003287523 nasa_logo-set-2-file-1.png permuter-stdout-0.log +SUB_TASK_FINISH.1667003294982 nasa_logo-file-1.perm.png permuter-inputs.h5 +sample-model.txt +SUB_TASK_START.1667003287523 nasa_logo-file-1.png permuter-stdout.log 1-2-permuter/st-2: -SUB_TASK_FINISH.1667003302619 nasa_logo-set-2-file-2-perm.png permuter-inputs-0.h5 sample-model.txt -SUB_TASK_START.1667003294987 nasa_logo-set-2-file-2.png permuter-stdout-0.log +SUB_TASK_FINISH.1667003302619 nasa_logo-file-2.perm.png permuter-inputs.h5 +sample-model.txt +SUB_TASK_START.1667003294987 nasa_logo-file-2.png permuter-stdout.log 1-2-permuter/st-3: -SUB_TASK_FINISH.1667003310303 nasa_logo-set-2-file-3-perm.png permuter-inputs-0.h5 sample-model.txt -SUB_TASK_START.1667003302623 nasa_logo-set-2-file-3.png permuter-stdout-0.log +SUB_TASK_FINISH.1667003310303 nasa_logo-file-3.perm.png permuter-inputs.h5 +sample-model.txt +SUB_TASK_START.1667003302623 nasa_logo-file-3.png permuter-stdout.log 1-2-permuter$ ``` At the top level there's some stuff we're not going to talk about now. What's interesting is the contents of the subtask directory, st-0: - The sample model is present with its original (non-datastore) name, `sample-model.txt`. -- The inputs file for this subtask is present, also with its original (non-datastore) name, `nasa-logo-set-2-file-0.png`. -- The outputs file for this subtask is present: `nasa-logo-set-2-file-0-perm.png`. -- The HDF5 file that contains filenames is present: `permuter-inputs-0.h5`. -- There's a file that contains all of the standard output (i.e., printing) from the algorithm: `permuter-stdout-0.log`. +- The inputs file for this subtask is present: `nasa-logo-file-0.png`. +- The outputs file for this subtask is present: `nasa-logo-file-0.perm.png`. +- The HDF5 file that contains filenames is present: `permuter-inputs.h5`. +- There's a file that contains all of the standard output (i.e., printing) from the algorithm: `permuter-stdout.log`. - There are a couple of files that show the Linux time that the subtask started and completed processing. ### The Moral of this Story @@ -158,43 +166,40 @@ So what's the takeaway from all this? Well, there's actually a couple: - Ziggy maintains separate directories for its permanent storage in the datastore and temporary storage for algorithm use in the task directory. - The task directory, in turn, contains one directory for each subtask. - The subtask directory contains all of the content that the subtask needs to run. This is convenient if troubleshooting is needed: you can copy a subtask directory to a different computer to be worked on, rather than being forced to work on it on the production file system used by Ziggy. -- There's some name mangling between the datastore and the task directory. +- There's some name mangling of models between the datastore and the task directory. - You can put anything you want into the subtask or task directory; Ziggy only pulls back the results it's been told to pull back. This means that, if you want to dump a lot of diagnostic information into each subtask directory, which you only use if something goes wrong in that subtask, feel free; Ziggy won't mind. -### Postscript: Copies vs. Symbolic Links +### Postscript: Copies vs. Links -If you look closely at the figure that shows the task directory, you'll notice something curious: the input and output "files" aren't really files. They're symbolic links. Specifically, they're symbolic links to files in the datastore. Looking at an example: +Are the files in the datastore and the task directory really copies of one another? Well, that depends. -```console -st-0$ ls -l -total 64 --rw-r--r-- 1 0 Oct 31 16:01 SUB_TASK_FINISH.1667257285445 --rw-r--r-- 1 0 Oct 31 16:01 SUB_TASK_START.1667257269376 -lrwxr-xr-x 1 104 Oct 31 16:01 nasa_logo-set-2-file-0-perm.png -> ziggy/sample-pipeline/build/pipeline-results/datastore/set-2/L1/nasa_logo-file-0.png -lrwxr-xr-x 1 104 Oct 31 16:01 nasa_logo-set-2-file-0.png -> ziggy/sample-pipeline/build/pipeline-results/datastore/set-2/L0/nasa_logo-file-0.png --rw-r--r-- 1 25556 Oct 31 16:01 permuter-inputs-0.h5 --rw-r--r-- 1 174 Oct 31 16:01 permuter-stdout-0.log -lrwxr-xr-x 1 126 Oct 31 16:01 sample-model.txt -> ziggy/sample-pipeline/build/pipeline-results/datastore/models/dummy model/2022-10-31.0001-sample-model.txt -st-0$ -``` +Most modern file systems offer a facility known as a "link" or a "hard link." The way a link works is as follows: rather than copy a file from Directory A to Directory B, the file system creates a new entry in Directory B for the file, and points it at the spot in the file system that holds the file you care about in Directory A. The file has, in effect, two names: one in Directory A and one in Directory B; but that file still only takes up the space of one file on the file system (rather than two, which is what you get when you copy a file). -Ziggy allows the user to select whether to use actual copies of the files or symbolic links. This is configured in -- yeah, you got it -- the properties file: +A great property of the link system is that, if we start with a file in Directory A, create a link to that file in Directory B, and then delete the file in Directory A, as far as Directory B is concerned that file is still there and can be accessed, modified, etc. In other words, as long as a file has multiple names (via the link system), "deleting the file" in one place only deletes that reference to the file, not the actual content of the file. The content of the file isn't deleted until the last such reference is removed. In other words, when you "delete" the file from Directory A, the file is still there on the file system, but the only way to find it now is via the name it has in Directory B. When you delete the file from Directory B, there are no longer any directories that have a reference to that file, so the actual content of the file is deleted. -``` -ziggy.pipeline.useSymlinks = true -``` +There are two limitations to hard links as implemented on typical file systems: + +- Only regular files can be linked; directories cannot. +- File links only work within a file system. + +What does Ziggy do? By default, Ziggy always uses links if it can; that is to say, it does so if the file system in question supports links and if the requested link is on the same file system as the original file. If Ziggy is asked to "copy" a directory from one place on a file system to another, Ziggy will create a new directory at the destination and then fill it with links to the files in the source directory. + +If the file system doesn't support links, or if the datastore and the task directory are on separate file systems, Ziggy will use ordinary file copying rather than linking. + +Why would a person ever want to put the datastore and the task directory on separate systems, given all of the aforementioned advantages of co-locating them? Turns out that there are security benefits to putting the datastore on a file system that's not networked all over the place, but rather is directly connected to a single computer (i.e., the one that's running Ziggy for you). By putting the task files on networked file systems, you can use all the other computers that mount that file system for processing data; when you then copy results back to the datastore on the direct-mounted file system, you've eliminated a risk that some other computer is going to come along and mess up your datastore. On the other hand, actually copying files creates performance issues because copying is extremely slow compared to linking, and it means that, at least temporarily, you have two copies of all your files taking up space (the task directory copy and the datastore copy). We report, you decide. -The way this works is obvious for the input files: Ziggy puts a symlink in the working directory, and that's all there is to it. For the outputs file, what happens is that the algorithm produces an actual file of results; when Ziggy goes to store the outputs file, it moves it to the datastore and replaces it in the working directory with a symlink. This is a lot of words to say that you can turn this feature on or off at will and your code doesn't need to do anything different either way. +#### Why not Symlinks? -The advantages of the symlinks are fairly obvious: +The same file systems that provide links also provide a different way to avoid copying files within a file system: symbolic links, also known as "symlinks" or (somewhat harshly) "slimelinks." Symlinks are somewhat more versatile than hard links: you can symlink to a directory, and you can have symlinks from one file system to another. Meanwhile, they give the same advantages in speed and disk space as hard links. Why doesn't Ziggy use them? -- Symbolic links take up approximately zero space on the file system. If you use symbolic links you avoid having multiple copies of every file around (one in the datastore, one in the subtask directory). For large data volumes, this can be valuable. -- Similarly, symbolic links take approximately zero time to instantiate. Copies take actual finite time. Again, for large data volumes, it can be a lot better to use symlinks than copies in terms of how much time your processing needs. +There are a few disadvantages of symlinks that were decisive in our thinking on this issue. Specifically: -There are also situations in which the symlinks may not be a good idea: +- A symlink can target a file on another file system, but it doesn't change the way that the file systems are mounted. Consider a system in which there's a datastore file system that's not networked and a task directory file system that is networked. The Ziggy server creates symlinks on the task directory file system that target files on the datastore file system, then hands execution over to another computer. That computer tries to open the file on the task directory, but it's not really there. It's really on the datastore file system, which the algorithm computer can't read from. Boom! Execution fails. +- Symlinks create a potential data-loss hazard. Imagine that you have a symlink that targets a directory in the datastore. Meanwhile, the actual files in that directory aren't symlinks; they're real files. Now imagine a user `cd`'s into the symlink directory. When that user accesses the files, they're accessing the files that are in the datastore, not files that are in some other directory. This means that if that user `cd`'s into the directory (which is a symlink), they can `rm` datastore files without realizing it! +- Because symlinks can target directories as well as regular files, you can wind up with an extremely complicated system in which you have a directory tree that contains a mixture of symlinks and real files / real directories, and in each and every case you need to decide how to handle them. This can quickly become a quagmire from which one will have a lot of trouble escaping. -- It may be the case that you're using one computer to run the supervisor, workers, and database, and a different one to run the algorithms. In this situation, the datastore can be on a file system that's mounted on the supervisor machine but not the compute machine, in which case the symlink solution won't work (the compute node can't see the datastore, so it can't follow the link). +For all these reasons we decided to stick with hard links and eschew symlinks. [[Previous]](intermediate-topics.md) [[Up]](intermediate-topics.md) -[[Next]](task-configuration.md) +[[Next]](rdbms.md) diff --git a/doc/user-manual/datastore.md b/doc/user-manual/datastore.md new file mode 100644 index 0000000..9747e0d --- /dev/null +++ b/doc/user-manual/datastore.md @@ -0,0 +1,250 @@ + + +[[Previous]](module-parameters.md) +[[Up]](configuring-pipeline.md) +[[Next]](pipeline-definition.md) + +## The Datastore + +"The Datastore" is a $10 word for an organized directory tree where Ziggy keeps the permanent copies of its various kinds of data files. These include the actual files of mission data, data product files, and a particular kind of metadata known as "instrument model files." + +As the user, one of your jobs is to define the following for Ziggy: + +- The layout of the datastore directory tree. +- The datastore locations and file name conventions for all of the data files used as inputs or outputs for your algorithms. +- The types of model files that your algorithms need, and the file name conventions for each. + +The place for these definitions is in data file type XML files. These have names that start with "pt-" (for "Pipeline Data Type"); in the sample pipeline, the data file type definitions are in [config/pt-sample.xml](../../sample-pipeline/conf/pt-sample.xml). + +Note that when we talk about data file types, we're not talking about data file formats (like HDF5 or geoTIFF). Ziggy doesn't care about data file formats; use whatever you like, as long as the algorithm software can read and write that format. + +### The Datastore Directory Tree + +Once you've spent a bit of time thinking about your algorithms and their inputs and outputs, you've probably got some sense of how you want to organize the directory tree for all those files. It's probably a bit intuitive and hard to put into words, but it's likely that you have some directory levels where there's just one directory with a fixed name, and others where you can have several directories with different names. If you have a directory "foo" that has subdirectories "bar" and "baz", the "foo" directory is an example of a fixed-name, all-by-itself-at-a-directory-level directory, while "bar" and "baz" are examples of a directory level where the directories can have one of a variety of different names. + +The way that Ziggy puts these into words (and code) is that every level of a directory is a `DatastoreNode`, and `DatastoreNodes` can use another kind of object, a `DatastoreRegexp`, to define different names that a `DatastoreNode` can take on. + +To make this more concrete (it could hardly be less concrete so far), let's consider the section of pt-sample.xml that defines the datastore directory tree: + +```xml + + + + + + + + + + + +``` + +The first thing you see is an example of a `DatastoreRegexp`. It has a `name` (`"dataset"`) and a `value` (`"set-[0-9]"`). The value is a *Java regular expression* (`"regexp"`). In this case, the regular expression will match "`set-0"`, `"set-1"`, etc. -- anything that's a combination of `"set-"` and a digit. + +The next thing you see is a `DatastoreNode`, also named `"dataset".` It has an attribute, `isRegexp`, which is true. What does this mean? It means that there's a top-level directory under the datastore root directory which can have as its name anything that matches the value of the `"dataset"` `DatastoreRegexp`. More generally, it means that any directory under the datastore root that matches that value is a valid directory in the datastore! Thus, the "`dataset"` `DatastoreNode` means, "put as many directories as you like here, as long as they match the `dataset` regular expression, and I'll know how to access them when the time comes." + +The `"dataset"` `DatastoreNode` also has another attribute: `nodes`, which has a value of `"L0, L1, L2A, L2B, L3"`. This tells Ziggy, "You should expect that any of these `dataset` directories will have subdirectories given by the `L0, L1, L2A, L2B`, and `L3` `DatastoreNode` instances." The `"dataset"` `DatastoreNode` then has elements that are themselves `DatastoreNode` instances, specifically the `"L0"`, `"L1"`, `"L2A"`, `"L2B"`, and `"L3"` nodes. + +None of these 5 `DatastoreNode` instances has an `isRegexp` attribute. That means that none of them references any `DatastoreRegexp` instances; which in turn means that each of them represents a plain old directory with a fixed name. + +Anyway, the point of this is that, at the top level of the datastore, we can have directories `set-1`, `set-2`, etc.; and each of those can have in it subdirectories named `L0`, `L1`, etc. + +#### A More Complicated Example: Deeper Nesting + +Let's consider our datastore layout again, but instead of putting all of the L* directories under the "dataset" directory, let's nest the directories, so that you wind up with directories like `set-0/L0`, `set-0/L0/L1`, etc. The obvious way to do that is like this: + +```xml + + + + + + + + + + + + + + + +``` + +This is a perfectly valid way to set up the datastore, but it's kind of a mess. There's a lot of nesting and a lot of `datastoreNode` closing tags, and between them it makes the layout hard to read and understand. For that reason, a better way to do it is like this: + +```xml + + + + + + + + + + + +``` + +Better, right? + +#### An Even More Complicated Example + +Now let's do something even more perverse: let's say that we want another L0 level under L2A but above L2B. That is to say, we want to make a directory like `set-0/L0/L1/L2A/L0` part of the datastore. Based on the example above, you might think that you could do this: + +```xml + + + + + + + + + + + + +``` + +In this case, though, you would be wrong! This won't work. + +Why not? + +The reason is that **every `DatastoreNode` within a parent `DatastoreNode` must have a unique name.** In this case, the `"dataset"` node contains two `"L0"` nodes, which is not allowed. If you wanted to do something like this, here's how you'd assemble the XML: + +```xml + + + + + + + + + + + + + +``` + +This works because, although there are two nodes named `"L0"`, they are sub-nodes of different parents: one is under `"dataset"`, the other is under `"L1"`. The first one is the only `"L0"` that has `"dataset"` as its parent; the second one is the only `"L0"` that has `"L1"` as its parent. + +Although the sample pipeline uses a pretty simple datastore layout, it's possible to implement extremely sophisticated layouts with the use of additional `DatastoreRegexp` instances, and so on. + +### Mission Data + +Now that we have the datastore layout defined, let's look at the next thing in the pt-sample file: data file type definitions. We'll just look at the first two: + +```xml + + + + + +``` + +A data file type declaration has three pieces of information: a `name,` a `location`, and a `fileNameRegexp`. + +The `name` is hopefully self-explanatory. + +What is a `location`? It's a valid, er, location in the datastore, as defined by the `DatastoreNode` instances. In the case of `raw data`, the `location` is `dataset/L0`. This means that `raw data` files can be found in directories `set-1/L0`, `set-2/L0`, etc. Note that the separator used in `location` instances is always the slash character. This is true even when the local file system uses some other character as its file separator in file path definitions. + +The `fileNameRegexp` uses a Java regular expression to define the naming convention for files of the `raw data` type. For raw-data, the regular expression is `"(nasa-logo-file-[0-9])\.png"`. This means that `nasa-logo-file-0.png`, `nasa-logo-file-1.png`, etc., are valid names for `raw data` files. Note the backslash character before the "." character: this is necessary because "." has a special meaning in Java regular expressions. If you don't want it to have that meaning, but instead just want it to be a regular old period, you put the backslash character before the period. + +Anyway, if you put it all together, this `DataFileType` is telling you that `raw data` files are things like `set-1/L0/nasa-logo-file-0.png`, `set-2/L0/nasa-logo-file-1.png`, and so on. + +### Instrument Model Types + +Before we can get into this file type definition, we need to answer a question: + +#### What is an Instrument Model, Anyway? + +We've given a lot of thought to how to define an instrument model. Here's the formal definition: + +**Instrument models are various kinds of information that are needed to process the data. These can be things like calibration constants; the location in space or on the ground that the instrument was looking at when the data was taken; the timestamp that goes with the data; etc.** + +The foregoing is not very intuitive. Here's a more colloquial definition: + +**Instrument models are any kinds of mission information that you're tempted to hard-code into your algorithms.** + +Think about it: when you write code to process data from an experiment, there's always a bunch of constants, coefficients, etc., that you need in order to perform your analysis. Unlike the data, these values don't change very often, so your first thought would be to just put them right into the code (or at least to hard-code the name and directory of the file that has the information). Anything that you'd treat that way is a model. + +Our opinion is that model files are a better way to handle this type of information, rather than hard-coding. For one thing, Ziggy provides explicit tracking of model versions and supports model updates in a way that's superior to receiving a new file and then either copying and pasting its contents into your source code or putting the file into version control and changing a hard-coded file name in the source. It also supports models that can't easily be put into a repository, either because they're too big, because they're in a non-text format, or both. + +#### Instrument Model Type Definition + +Behold our sample instrument model type definition: + +​ `` + +As with the data file types, model types are identified by a string (in this case, the `type` attribute) that can contain whitespace, and provides a regular expression that can be used to determine whether any particular file is a model of the specified type. In this case, in a fit of no-imagination, the regex is simply a fixed name of `sample-model.txt`. Thus, any processing algorithm that needs the `dummy model` will expect to find a file named `sample-model.txt` in its task directory. + +#### Wait, is That It? + +Sadly, no. Let's talk about model names and how they fit into all of this. + +##### Datastore Model Names + +Ziggy permanently stores every model of every kind that is imported into it. This is necessary because someday you may need to figure out what model was used for a particular processing activity, but on the other hand it may be necessary to change the model as time passes -- either because the instrument itself changes with time, or because your knowledge of the instrument changes (hopefully it improves). + +But -- in the example above, the file name "regex" is a fixed string! This means that the only file name that Ziggy can possibly recognize as an instance of `dummy model` is `sample-model.txt`. So when I import a new version of `sample-model.txt` into the datastore, what happens? To answer that, let's take a look at the `dummy model` subdirectory of the `models` directory in the datastore: + +```console +models$ ls "dummy\ model" +2022-10-31.0001-sample-model.txt +models$ +``` + +(Yes, I broke my own strongly-worded caution against using whitespace in names, and in a place where it matters a lot -- a directory name! Consistency, hobgoblins, etc.) + +As you can see, the name of the model in the datastore isn't simply `sample-model.txt`. It's had the date of import prepended, along with a version number. By making these changes to the name, Ziggy can store as many versions of a model as it needs to, even if the versions all have the same name at the time of the import. + +Note also that model type definitions don't require a defined `location`. Ziggy creates a subdirectory to the datastore root, `models`, and puts under that a subdirectory for every model type. So that's one set of decisions you don't need to make. + +When a model is provided to an algorithm that needs it, the models infrastructure does the following: + +First, it finds the most recent model of the specified type (which has the highest model number and also the most recent date stamp); then, it copies the file to the algorithm's working directory, but in the process it renames the file from the name it uses for storage (in this example, `2022-10-31.0001-sample-model.txt`) to the name it had when it was imported (in this example, `sample-model.txt`). In this way, Ziggy uses a name-mangling scheme to keep multiple model versions in a common directory, but then un-mangles the name for the algorithm, so the algorithm developers don't need to know anything about name-mangling; the name you expect the file to have is the name it actually will have. + +##### What Happens if the Actual Model Changes? + +Excellent question! Imagine that, at some point in time, one or more models change -- not your knowledge of them, the actual, physical properties of your instrument change. Obviously you need to put a new model into the system to represent the new properties of the instrument. But equally obviously, if you ever go back and reprocess data taken prior to the change, you need to use the model that was valid at that time. How does Ziggy handle that? + +Answer: Ziggy always, *always* provides the most recent version of the model file. If you go and reprocess, the new processing will get the latest model. In order to properly represent a model that changes with time, **the changes across time must be reflected in the most recent model file!** Also, and relatedly, **the algorithm code must be able to pull model for the correct era out of the model file!** + +In practice, that might mean that your model file contains multiple sets of information, each of which has a datestamp; the algorithm would then go through the file contents to find the set of information with the correct datestamp, and use it. Or, it might mean that the "model" is values measured at discrete times that need to be interpolated by the algorithm. How the time-varying information is provided in the model file is up to you, but if you want to have a model that does change in time, this is how you have to do it. + +##### Model Names with Version Information + +The above example is kind of unrealistic because in real life, a mission that provides models that get updated will want to put version information into the file name; if for no other reason than so that when there's a problem and we need to talk about a particular model version, we can refer to the one we're concerned about without any confusion ("Is there a problem with sample model?" "Uh, which version of sample model?" "Dunno, it's just called sample model."). Thus, the file name might contain a timestamp, a version number, or both. + +If the model name already has this information, it would be silly for Ziggy to prepend its own versioning; it should use whatever the mission provides. Fortunately, this capability is provided: + +```xml + +``` + +In this case, the XML attribute `versionNumberGroup` tells Ziggy which regex group it should use as the version number, and the attribute `timestampGroup` tells it which to use as the file's timestamp. When Ziggy stores this model in the `versioned-model` directory, it won't rename the file; it will keep the original file name, because the original name already has a timestamp and a version number. + +In general, the user can include in the filename a version number; a timestamp; or both; or neither. Whatever the user leaves out, Ziggy will add to the filename for internal storage, and then remove again when providing the file to the algorithms. + +##### Models Never Get Overwritten in the Datastore + +One thing about supplying timestamp and version information in the filename is that it gives some additional protection against accidents. **Specifically: Ziggy will never import a model that has the same timestamp and version number as one already in the datastore.** Thus, you can never accidentally overwrite an existing model with a new one that's been accidentally given the same timestamp and version information. + +For models that don't provide that information in the filename, there's no protection against such an accident because there can't be any such protection. If you accidentally re-import an old version of `sample-model.txt`, Ziggy will assume it's a new version and store it with a new timestamp and version number. When Ziggy goes to process data, this version will be provided to the algorithms. + +[[Previous]](module-parameters.md) +[[Up]](configuring-pipeline.md) +[[Next]](pipeline-definition.md) diff --git a/doc/user-manual/dusty-corners.md b/doc/user-manual/dusty-corners.md index 72c8e7e..02cddc9 100644 --- a/doc/user-manual/dusty-corners.md +++ b/doc/user-manual/dusty-corners.md @@ -1,6 +1,6 @@ -[[Previous]](event-handler-labels.md) +[[Previous]](console-cli.md) [[Up]](user-manual.md) [[Next]](more-rdbms.md) @@ -24,20 +24,16 @@ How to conveniently package a collection of parameter changes. What to do if you realize that you need to change the configuration of a pipeline. -### [The Edit Pipeline Dialog Box](edit-pipeline.md) - -Additional features on a dialog box we've already used. - ### [Creating Ziggy Nicknames](nicknames.md) Make it easier to run those Java programs you've written. +[[Previous]](console-cli.md) +[[Up]](user-manual.md) +[[Next]](more-rdbms.md) + - -[[Previous]](event-handler-labels.md) -[[Up]](user-manual.md) -[[Next]](more-rdbms.md) diff --git a/doc/user-manual/edit-pipeline.md b/doc/user-manual/edit-pipeline.md index efef228..2b44407 100644 --- a/doc/user-manual/edit-pipeline.md +++ b/doc/user-manual/edit-pipeline.md @@ -1,14 +1,14 @@ -[[Previous]](parameter-overrides.md) -[[Up]](dusty-corners.md) -[[Next]](nicknames.md) +[[Previous]](organizing-tables.md) +[[Up]](ziggy-gui.md) +[[Next]](datastore-regexp.md) ## The Edit Pipeline Dialog Box -The Edit Pipeline dialog box is used to edit pipeline parameter sets and modules. +The Edit Pipeline dialog box is used to edit pipeline parameter sets and modules, and to configure the quantity of resources each pipeline module in a given pipeline can use. -To get to this dialog box, open the pipelines panel and double-click the pipeline you're interested in. You'll get this dialog box: +To get to this dialog box, open the pipelines panel and double-click the pipeline you're interested in. You'll see this: @@ -28,7 +28,7 @@ The `Priority` field takes a little more explanation. We've discussed in the pas So how to tasks get assigned a priority? -All tasks that are running for the first time get assigned a priority equal to the priority of the parent pipeline. In this example, the sample pipeline has a priority of NORMAL, meaning that all tasks for this pipeline will have the lowest possible priority on their first pass through the system. Tasks that are being persisted (which happens on a separate pass through the task management system) do so with priority HIGH, so persisting results takes precedence over starting new tasks. Tasks that are being rerun or restarted do so with priority HIGHEST, which means exactly what it sounds like. +All tasks that are running for the first time get assigned a priority equal to the priority of the parent pipeline. In this example, the sample pipeline has a priority of NORMAL, meaning that all tasks for this pipeline will have the a moderate priority level on their first pass through the system. Tasks that are being persisted (which happens on a separate pass through the task management system) do so with priority HIGH, so persisting results takes precedence over starting new tasks. Tasks that are being rerun or restarted do so with priority HIGHEST, which means exactly what it sounds like. All pipelines, in turn, are initially created with priority NORMAL, meaning that all pipelines will, by default, produce tasks at priority NORMAL. Thus, all tasks from all pipelines compete for workers with a "level playing field," if you will. Usually this is the situation that most users want. @@ -36,6 +36,10 @@ One case where this isn't true is missions that have occasional need for much fa Finally, the read-only `Valid?` checkbox is ticked after the `Validate` button is pressed, presuming all went well. +#### Processing mode + +The `Processing mode` radio button section has two options: `Process all data` versus `Process new data`. This option is pretty much exactly what it sounds like. Specifically: the `Process all data` option tells Ziggy that each pipeline module should process all the data it finds, whether that data has already been processed or not; the `Process new data` only processes data files that have never before been processed. For a mission that's currently acquiring data, it's likely that most of the time you'll want to set the `Process new data` option, since it will save time by not processing data that's already been processed. At intervals, the mission may decide to do a uniform reprocessing of all data (to take advantage of algorithm improvements, etc.) For this activity, `Process all data` is the correct option. + ### Pipeline Parameter Sets Section Say that five times fast. @@ -54,15 +58,41 @@ The display shows the modules in the pipeline, sorted in execution order. You ca #### Task Information Button -This button produces a table of the tasks that Ziggy will produce for the specified module if you start the pipeline. This takes into account whether the module is configured for "keep-up" processing or reprocessing, the setting of the taskDirectoryRegex string (which allows the user to specify that only subsets of the datastore should be run through the pipeline). For each task, the task's unit of work description and number of subtasks are shown. If the table is empty, it means that the relevant files in the datastore are missing. The datastore is populated by [Data Receipt](data-receipt.md); that article will help you ingest your data into the datastore so that the task information table can calculate the number of tasks and subtasks the input data will generate. +This button produces a table of the tasks that Ziggy will produce for the specified module if you start the pipeline. This takes into account whether the module is configured to process all data or to process only new data; the setting of the taskDirectoryRegex string (which allows the user to specify that only subsets of the datastore should be run through the pipeline). For each task, the task's unit of work description and number of subtasks are shown. If the table is empty, it means that the relevant files in the datastore are missing. The datastore is populated by [Data Receipt](data-receipt.md); that article will help you ingest your data into the datastore so that the task information table can calculate the number of tasks and subtasks the input data will generate. #### Resources Button If you look back at [the article on running the cluster](running-pipeline.md), you'll note that we promised that there was a way to set a different limit on the number of workers for each pipeline module. This button is that way! -More specifically, if you press the `Resources` button, you'll get the `Worker resources` dialog box that displays a table of the modules and the current max workers and heap size settings. To change these settings from the default, either double-click on a module or use the context menu and choose the `Edit` command. This brings up the `Edit worker resources` dialog box where you can uncheck the Default checkboxes and enter new values for the number of workers or the heap size for that module. Note that the console won't let you enter more workers than cores on your machine, which is found in the the tooltip for this field. Henceforth, Ziggy will use those values when deciding on the maximum number of workers to spin up for that module and how much memory each should be given. Typically, as you increase the number of workers on a single host, you'll need to reduce the amount of heap space for each worker so that the total memory will fit within the memory available on the machine. +More specifically, if you select a module and press the `Resources` button, you'll get the `Edit worker resources` dialog box that displays a number of resource settings: + + + +Let's take these in order, again from top to bottom: + +##### Maximum workers + +This allows you to set the number of worker processes each pipeline module can spin up. Spinning up more can allow more tasks to run in parallel to one another, but may also cause the tasks to consume more memory than is available. The `Default` check box tells Ziggy to use the default value for the maximum worker processes on this module. The default is the value of the `ziggy.worker.count` property in [the properties file](properties.md), unless you overrode this value by using the `--workerCount` option when you [started the cluster](running-pipeline.md). + +##### Maximum heap size + +This allows you to set the maximum total Java heap size used by the workers for this pipeline module. As described before, Ziggy will take the maximum heap size for a module and divide it up evenly between the worker processes. Thus, in this case the default of 2 workers and 12 GB heap size means that every worker gets 6 GB of Java heap. As with the `Maximum workers` option, the user can use the `Default` check box to get the default value, or uncheck it to enter a custom value: + + + +As with the worker count, the default heap size is the value specified by the `ziggy.worker.heapSize` property unless it has been overridden by using the `--workerHeapSize` option when you started the cluster. + +##### Maximum failed subtasks + +As a pipeline module executes its assorted subtasks, it is possible that not every subtask will run to completion. Most vexingly, it is possible that some of the subtasks will fail due to various errors in the code or features of the data, while others complete successfully. Under ordinary circumstances, if even one subtask fails, the entire task will be marked as failed and the pipeline will halt until the issue is addressed. + +The `Maximum failed subtasks` tells Ziggy that, in the event that some subtasks do fail, if the number of subtasks falls below the value of `Maximum failed subtasks`, Ziggy should mark the task as complete rather than failed. Note that this can be set after the fact! Say for example that a task has 100 subtasks, of which 95 succeed and 5 fail. If the mission decides to not try to rescue the 5 failed subtasks right now, you can set `Maximum failed subtasks` to 5 and then resubmit the task. Ziggy will detect that the number of failed subtasks is below the limit and will mark the task as completed. + +##### Maximum automatic resubmits + +Another vexing occurrence is when a task, or some of its subtasks, fail even though in principle they should all have been able to complete successfully. This can be due to various transient system problems (a brief glitch in a network file system, for example), or because the task ran out of wall time before all the subtasks had completed. In these cases, it can be useful for Ziggy to automatically resubmit any tasks that fail. By setting the `Maximum automatic resubmits`, you can control this behavior in Ziggy. -Alternately, you may want to do the reverse: take a module that has user-set maximum workers or heap size values and tick the Default checkboxes to go back to using the defaults. +Note that this option can potentially be dangerous. In particular, if a task fails because it has subtasks that fail due to algorithm or data problems, then each time Ziggy resubmits the task those same subtasks will fail again, until the automatic resubmits are exhausted. Use with caution! #### Parameters Button @@ -83,6 +113,6 @@ The points I'm trying to make here are twofold: 1. Anything you do after you launch the `Edit pipeline` dialog box can be discarded, and will only be preserved when you press `Save`. 2. The `Save` and `Cancel` buttons on the `Edit pipeline` dialog box also apply to changes made on the `Edit remote execution parameters` dialog box, the `Edit parameter sets` dialog box, etc. -[[Previous]](parameter-overrides.md) -[[Up]](dusty-corners.md) -[[Next]](nicknames.md) +[[Previous]](organizing-tables.md) +[[Up]](ziggy-gui.md) +[[Next]](datastore-regexp.md) diff --git a/doc/user-manual/event-handler-examples.md b/doc/user-manual/event-handler-examples.md index 384fc98..f0d4845 100644 --- a/doc/user-manual/event-handler-examples.md +++ b/doc/user-manual/event-handler-examples.md @@ -91,7 +91,7 @@ Before we do that, though, let's reconfigure the pipeline a bit. First, we need Second, let's tell the pipeline that we only want it to process new data that's never been processed, and it should leave alone any data that's been successfully processed before this. To do so, select the `Multiple subtask configuration` and the `Single subtask configuration` parameter sets, and uncheck the reprocess box: - + Now return to the instances panel, and finally create the ready files. Remember that you need two ready files because we are simulating a complete delivery from the first source, and it's delivering to the `sample-1` and `sample-2` directories. diff --git a/doc/user-manual/event-handler-labels.md b/doc/user-manual/event-handler-labels.md index 99f7acc..54ca35f 100644 --- a/doc/user-manual/event-handler-labels.md +++ b/doc/user-manual/event-handler-labels.md @@ -2,7 +2,7 @@ [[Previous]](event-handler-examples.md) [[Up]](event-handler.md) -[[Next]](dusty-corners.md) +[[Next]](console-cli.md) ## Sending Event Information to Algorithms @@ -22,4 +22,4 @@ When a task is created by a pipeline that started in response to an event, the ` [[Previous]](event-handler-examples.md) [[Up]](event-handler.md) -[[Next]](dusty-corners.md) +[[Next]](console-cli.md) diff --git a/doc/user-manual/images/architecture-diagram.png b/doc/user-manual/images/architecture-diagram.png index af65af9..1522187 100644 Binary files a/doc/user-manual/images/architecture-diagram.png and b/doc/user-manual/images/architecture-diagram.png differ diff --git a/doc/user-manual/images/data-receipt-display.png b/doc/user-manual/images/data-receipt-display.png index e26a30f..7e7e050 100644 Binary files a/doc/user-manual/images/data-receipt-display.png and b/doc/user-manual/images/data-receipt-display.png differ diff --git a/doc/user-manual/images/data-receipt-list.png b/doc/user-manual/images/data-receipt-list.png index c27a441..e93a909 100644 Binary files a/doc/user-manual/images/data-receipt-list.png and b/doc/user-manual/images/data-receipt-list.png differ diff --git a/doc/user-manual/images/data-receipt-use-subdirs.png b/doc/user-manual/images/data-receipt-use-subdirs.png index 1cbb8ef..1444018 100644 Binary files a/doc/user-manual/images/data-receipt-use-subdirs.png and b/doc/user-manual/images/data-receipt-use-subdirs.png differ diff --git a/doc/user-manual/images/datastore-display-1.png b/doc/user-manual/images/datastore-display-1.png new file mode 100644 index 0000000..0bd9f62 Binary files /dev/null and b/doc/user-manual/images/datastore-display-1.png differ diff --git a/doc/user-manual/images/datastore-display-2.png b/doc/user-manual/images/datastore-display-2.png new file mode 100644 index 0000000..20b3910 Binary files /dev/null and b/doc/user-manual/images/datastore-display-2.png differ diff --git a/doc/user-manual/images/disable-reprocess.png b/doc/user-manual/images/disable-reprocess.png index 4dddea4..947747e 100644 Binary files a/doc/user-manual/images/disable-reprocess.png and b/doc/user-manual/images/disable-reprocess.png differ diff --git a/doc/user-manual/images/edit-datastore-regexp-1.png b/doc/user-manual/images/edit-datastore-regexp-1.png new file mode 100644 index 0000000..319344d Binary files /dev/null and b/doc/user-manual/images/edit-datastore-regexp-1.png differ diff --git a/doc/user-manual/images/edit-datastore-regexp-2.png b/doc/user-manual/images/edit-datastore-regexp-2.png new file mode 100644 index 0000000..8a07f54 Binary files /dev/null and b/doc/user-manual/images/edit-datastore-regexp-2.png differ diff --git a/doc/user-manual/images/edit-pipeline.png b/doc/user-manual/images/edit-pipeline.png index 092b469..3d3651d 100644 Binary files a/doc/user-manual/images/edit-pipeline.png and b/doc/user-manual/images/edit-pipeline.png differ diff --git a/doc/user-manual/images/event-handler-display-1.png b/doc/user-manual/images/event-handler-display-1.png index e69760f..f93900e 100644 Binary files a/doc/user-manual/images/event-handler-display-1.png and b/doc/user-manual/images/event-handler-display-1.png differ diff --git a/doc/user-manual/images/event-handler-instances-1.png b/doc/user-manual/images/event-handler-instances-1.png index e6afea9..20ba686 100644 Binary files a/doc/user-manual/images/event-handler-instances-1.png and b/doc/user-manual/images/event-handler-instances-1.png differ diff --git a/doc/user-manual/images/event-handler-instances-2.png b/doc/user-manual/images/event-handler-instances-2.png index 1631332..3acf6da 100644 Binary files a/doc/user-manual/images/event-handler-instances-2.png and b/doc/user-manual/images/event-handler-instances-2.png differ diff --git a/doc/user-manual/images/exception-1.png b/doc/user-manual/images/exception-1.png index 9786684..60df261 100644 Binary files a/doc/user-manual/images/exception-1.png and b/doc/user-manual/images/exception-1.png differ diff --git a/doc/user-manual/images/exception-2.png b/doc/user-manual/images/exception-2.png deleted file mode 100644 index 1b98a72..0000000 Binary files a/doc/user-manual/images/exception-2.png and /dev/null differ diff --git a/doc/user-manual/images/flip-tasks.png b/doc/user-manual/images/flip-tasks.png index 2dbf6a5..2708b82 100644 Binary files a/doc/user-manual/images/flip-tasks.png and b/doc/user-manual/images/flip-tasks.png differ diff --git a/doc/user-manual/images/gui-start-end-adjusted.png b/doc/user-manual/images/gui-start-end-adjusted.png index 762beb1..302227b 100644 Binary files a/doc/user-manual/images/gui-start-end-adjusted.png and b/doc/user-manual/images/gui-start-end-adjusted.png differ diff --git a/doc/user-manual/images/gui.png b/doc/user-manual/images/gui.png index 4b99487..cc4c6a1 100644 Binary files a/doc/user-manual/images/gui.png and b/doc/user-manual/images/gui.png differ diff --git a/doc/user-manual/images/halt-alert.png b/doc/user-manual/images/halt-alert.png index 6248965..fe46d69 100644 Binary files a/doc/user-manual/images/halt-alert.png and b/doc/user-manual/images/halt-alert.png differ diff --git a/doc/user-manual/images/halt-in-progress.png b/doc/user-manual/images/halt-in-progress.png index 5aa0efd..3a41788 100644 Binary files a/doc/user-manual/images/halt-in-progress.png and b/doc/user-manual/images/halt-in-progress.png differ diff --git a/doc/user-manual/images/halt-task-menu-item.png b/doc/user-manual/images/halt-task-menu-item.png index acba20b..4da39c6 100644 Binary files a/doc/user-manual/images/halt-task-menu-item.png and b/doc/user-manual/images/halt-task-menu-item.png differ diff --git a/doc/user-manual/images/instances-running.png b/doc/user-manual/images/instances-running.png index 7ff982d..5d1c0e7 100644 Binary files a/doc/user-manual/images/instances-running.png and b/doc/user-manual/images/instances-running.png differ diff --git a/doc/user-manual/images/monitor-processes.png b/doc/user-manual/images/monitor-processes.png index acd5044..0745eca 100644 Binary files a/doc/user-manual/images/monitor-processes.png and b/doc/user-manual/images/monitor-processes.png differ diff --git a/doc/user-manual/images/monitoring-alerts.png b/doc/user-manual/images/monitoring-alerts.png index c903fd6..03ec711 100644 Binary files a/doc/user-manual/images/monitoring-alerts.png and b/doc/user-manual/images/monitoring-alerts.png differ diff --git a/doc/user-manual/images/monitoring-worker-2.png b/doc/user-manual/images/monitoring-worker-2.png index 116fc83..e8f29db 100644 Binary files a/doc/user-manual/images/monitoring-worker-2.png and b/doc/user-manual/images/monitoring-worker-2.png differ diff --git a/doc/user-manual/images/monitoring-worker.png b/doc/user-manual/images/monitoring-worker.png index 24990b1..887e40e 100644 Binary files a/doc/user-manual/images/monitoring-worker.png and b/doc/user-manual/images/monitoring-worker.png differ diff --git a/doc/user-manual/images/param-import-dialog-box.png b/doc/user-manual/images/param-import-dialog-box.png index 38cfa32..0be846c 100644 Binary files a/doc/user-manual/images/param-import-dialog-box.png and b/doc/user-manual/images/param-import-dialog-box.png differ diff --git a/doc/user-manual/images/param-lib-all-groups-expanded.png b/doc/user-manual/images/param-lib-all-groups-expanded.png index 8346831..cf38e63 100644 Binary files a/doc/user-manual/images/param-lib-all-groups-expanded.png and b/doc/user-manual/images/param-lib-all-groups-expanded.png differ diff --git a/doc/user-manual/images/param-lib-context-menu.png b/doc/user-manual/images/param-lib-context-menu.png index a97617d..e39e422 100644 Binary files a/doc/user-manual/images/param-lib-context-menu.png and b/doc/user-manual/images/param-lib-context-menu.png differ diff --git a/doc/user-manual/images/param-lib-group-assigned.png b/doc/user-manual/images/param-lib-group-assigned.png index e69cbe0..0af5ee5 100644 Binary files a/doc/user-manual/images/param-lib-group-assigned.png and b/doc/user-manual/images/param-lib-group-assigned.png differ diff --git a/doc/user-manual/images/param-lib-modified.png b/doc/user-manual/images/param-lib-modified.png index 27608b5..413b0e6 100644 Binary files a/doc/user-manual/images/param-lib-modified.png and b/doc/user-manual/images/param-lib-modified.png differ diff --git a/doc/user-manual/images/param-lib-used.png b/doc/user-manual/images/param-lib-used.png index 8ff7d4a..04cea6c 100644 Binary files a/doc/user-manual/images/param-lib-used.png and b/doc/user-manual/images/param-lib-used.png differ diff --git a/doc/user-manual/images/parameter-library.png b/doc/user-manual/images/parameter-library.png index e51c253..c68b338 100644 Binary files a/doc/user-manual/images/parameter-library.png and b/doc/user-manual/images/parameter-library.png differ diff --git a/doc/user-manual/images/permuter-tasks.png b/doc/user-manual/images/permuter-tasks.png index c3d9015..9b18222 100644 Binary files a/doc/user-manual/images/permuter-tasks.png and b/doc/user-manual/images/permuter-tasks.png differ diff --git a/doc/user-manual/images/pipeline-done.png b/doc/user-manual/images/pipeline-done.png index 89979f2..cc23a6a 100644 Binary files a/doc/user-manual/images/pipeline-done.png and b/doc/user-manual/images/pipeline-done.png differ diff --git a/doc/user-manual/images/pipelines-panel.png b/doc/user-manual/images/pipelines-panel.png index 8388062..cea8e1c 100644 Binary files a/doc/user-manual/images/pipelines-panel.png and b/doc/user-manual/images/pipelines-panel.png differ diff --git a/doc/user-manual/images/remote-dialog-1.png b/doc/user-manual/images/remote-dialog-1.png index 4585b49..f8ea849 100644 Binary files a/doc/user-manual/images/remote-dialog-1.png and b/doc/user-manual/images/remote-dialog-1.png differ diff --git a/doc/user-manual/images/remote-dialog-2.png b/doc/user-manual/images/remote-dialog-2.png index 7a39a4e..7f8a41e 100644 Binary files a/doc/user-manual/images/remote-dialog-2.png and b/doc/user-manual/images/remote-dialog-2.png differ diff --git a/doc/user-manual/images/remote-dialog-3.png b/doc/user-manual/images/remote-dialog-3.png index 10e10c1..a95fc65 100644 Binary files a/doc/user-manual/images/remote-dialog-3.png and b/doc/user-manual/images/remote-dialog-3.png differ diff --git a/doc/user-manual/images/remote-dialog-4.png b/doc/user-manual/images/remote-dialog-4.png deleted file mode 100644 index 734f481..0000000 Binary files a/doc/user-manual/images/remote-dialog-4.png and /dev/null differ diff --git a/doc/user-manual/images/remote-dialog-5.png b/doc/user-manual/images/remote-dialog-5.png deleted file mode 100644 index 357bffa..0000000 Binary files a/doc/user-manual/images/remote-dialog-5.png and /dev/null differ diff --git a/doc/user-manual/images/resources-initial.png b/doc/user-manual/images/resources-initial.png new file mode 100644 index 0000000..aa07162 Binary files /dev/null and b/doc/user-manual/images/resources-initial.png differ diff --git a/doc/user-manual/images/resources-updated.png b/doc/user-manual/images/resources-updated.png new file mode 100644 index 0000000..c8b200e Binary files /dev/null and b/doc/user-manual/images/resources-updated.png differ diff --git a/doc/user-manual/images/tasks-done.png b/doc/user-manual/images/tasks-done.png index d3f272f..1bf5a4a 100644 Binary files a/doc/user-manual/images/tasks-done.png and b/doc/user-manual/images/tasks-done.png differ diff --git a/doc/user-manual/images/tasks-menu.png b/doc/user-manual/images/tasks-menu.png index 16cbedd..993316f 100644 Binary files a/doc/user-manual/images/tasks-menu.png and b/doc/user-manual/images/tasks-menu.png differ diff --git a/doc/user-manual/instances-panel.md b/doc/user-manual/instances-panel.md index 24fe040..0a7fe20 100644 --- a/doc/user-manual/instances-panel.md +++ b/doc/user-manual/instances-panel.md @@ -52,51 +52,148 @@ The parameter set that Ziggy uses to figure out how to divide work up into tasks #### Can You be a Bit More Specific About That? -Sure! Let's look again at the definition of the permuter node from [The Pipeline Definition article](pipeline-definition.md): +Sure! Let's take a look again at the input data file type for permuter: ```xml - - - - - - - + ``` -The definition of the node includes a parameter set, `Multiple subtask configuration`, which is an instance of the `TaskConfigurationParameters`. From [the article on The Task Configuration Parameter Sets](task-configuration.md), we see that it looks like this: +While we're at it, let's look at the definition of the `dataset` datastore node: ```xml - - - - - - - - + + ``` -The `taskDirectoryRegex` parameter is `set-([0-9]{1})`. In plain (but New York accented) English, what this means is, "Go to the datastore and find every directory that matches the regex. Every one of those, you turn into a task. You got a problem with that?" Thus you wind up with a task for `set-1` and another for `set-2`. +When Ziggy goes to generate `permuter` tasks, the first thing it does is go to the datastore and say, "Give me all the directories that match the location of this `raw data` data file type." When it gets back "`set-1/L0`" and "`set-2/L0`", it says to them, "Congratulations, you two define the two units of work for this module." -Meanwhile, the `taskDirectoryRegex` has a regex group in it, `([0-9]{1})`. This tells Ziggy to take that part of the directory name (i.e., a digit) and make it the name of the unit of work on the tasks table. If I had been smarter and written the `taskDirectoryRegex` as `(set-[0-9]{1})`, the UOW display would have shown `set-1` and `set-2` instead of `1` and `2`. +The next thing Ziggy has to do is give those tasks names, or in Ziggy lingo, "brief states." The way it does that is by grabbing the `set-1/L0` location and the `set-2/L0` location and asking, "What parts of these locations are different from one to the next?" Seeing that it's the "set-#" part, it then says, "Congratulations, your brief states are `[set-1]` and `[set-2]`." + +##### A More Complicated Example + +Now let's imagine a less trivial datastore configuration: + +```xml + + + + + + + + + + + +``` + +When Ziggy goes to make units of work, and then tasks, there will be 36 of them total! The brief states will be things like, "`[reeves;omar;earthling]`", "`[reeves;woody;earthling]`", "`[carlos;woody;hours]`", etc. In other words, 1 task for each possible combination of `guitar`, `drums`, and `album`. + +Notice that, although all the data directories have a `spider` element in the path, none of the brief states include `spider`. This is because it's common to all the tasks, which means it's not interesting to put into the brief state. The brief state only includes the path elements that vary between tasks. + +##### But What if I Don't Want to Run All 36 Tasks? + +So in our real sample pipeline, what do you do if you only want to run the `set-1` tasks? Or, in our more complicated example, what if we want to run the tasks where guitar is set to `reeves` and album is set to either `outside` or `stardust`? + +It can be done! But not here. If you want to know how this is handled, check out [the article on the Datastore Regular Expression control panel](datastore-regexp.md). #### What About Subtask Definition? -Now we've seen how Ziggy uses the `TaskConfigurationParameters` instance to define multiple tasks for a given pipeline node. How do subtasks get defined? This uses a combination of 2 things: the `TaskConfigurationParameters` and the definition of input data file types for the node. Let's look at how that works. +Now we've seen how Ziggy uses the inputs data file type to define multiple tasks for a given pipeline node. How do subtasks get defined? + +The default is for Ziggy to create a subtask for each input data file. When creating a task, Ziggy finds all of the data files in the appropriate datastore directory: in the case of `permuter`, `raw-data`, and `set-1/L0`, we find `nasa-logo-file-0.png`, `nasa-logo-file-1.png`, `nasa-logo-file-2.png,` `nasa-logo-file-3.png`. Ziggy goes ahead and creates a subtask for each of these data files. Presto! Four subtasks for `permuter`. -In TaskConfigurationParameters, there's a boolean parameter, `singleSubtask`. This does what it says: if set to `true`, Ziggy creates one and only one subtask for each task, and copies all the inputs into that subtask's directory. When set to false, as here, it generates multiple subtasks for the task. +##### Tasks with Multiple Input Data File Types -The way it does this is to create a subtask for each input data file. If we look at how the inputs to the permuter are defined in [the article on Data File Types](data-file-types.md), we see this: +At the end of the pipeline, we have the `averaging` pipeline module, which averages together a bunch of PNG files. Let's see how it's defined in pd-sample.xml: ```xml - + + + + + ``` -The file names in the datastore are going to be things like `set-1/L0/nasa-logo-file-0.png`, `set-1/L0/nasa-logo-file-1.png`, and so on. So -- cool! All the files in `set-1/L0` will be processed in the task for `set-1` data; there will be a subask for `nasa-logo-file-0.png`, another for `nasa-logo-file-1.png`, and so on. +This module has two input file types! How does Ziggy generate subtasks for that? + +Let's look again at the `left-right flipped` and `up-down flipped` data file type definitions: + +```xml + + + + + +``` + +Notice the part of the fileNameRegexp that's inside the parentheses: In Java regular expressions, this is called a "group." Java has tools that will take a string, match it to a regular expression, and extract the values of the groups for futher perusal. + +You've probably already guessed what this is leading up to: + +**When Ziggy has multiple input data file types for a pipeline module, it figures out which files go together in a subtask by their regular expression groups. Specifically, two files go together if the values of all their regular expression groups match.** + +Of course, if that was the only requirement, we could simply give both data file types the fileNameRegexp value of `(nasa-logo-file-[0-9])\.png`. There is one other requirement, which, again, you've probably already guessed: + +Because all of the inputs for a given subtask get copied to the subtask working directory, they must all have unique names. If we used `(nasa-logo-file-[0-9])\.png`for both fileNameRegexp values, then either the up-down flipped file would overwrite the left-right flipped one, or vice versa. + +In fact, the rule is stricter than that, because output files get written into the subtask working directory as well, and, again, we can't abide by files overwriting each other. Thus the general rule: + +**All of the inputs and outputs files for a pipeline module must have unique names. Inputs files for a given pipeline module must have names that match up to the first period (".") character.** + +##### Tasks with Only One Subtask + +There are also cases in which you won't want Ziggy to automatically build a bunch of subtasks. Consider the `averaging` pipeline module: we want to average together all 4 of the up-down flipped and all 4 of the right-left flipped images into one final averaged image. We obviously can't do that if `averaging` creates 4 subtasks and each one averages together one up-down flipped file with one left-right flipped one. In this situation -- where Ziggy is tempted to make multiple subtasks but you want it to resist that temptation -- you can tell this to Ziggy by adding the `singleSubtask="true"` attribute to the pipeline node definition. If you look back up at the definition of the `averaging` node a few paragraphs back, you can see that the node is marked to produce a single subtask, and indeed that's exactly what it does. + +##### Data Files Needed By All Subtasks + +Now let's consider an even more complex situation: imagine that the `permuter` pipeline module needs some other flavor of data file that's in the datastore. Let's imagine that there's a directory, description, that's at the same level of the datastore as the L0, etc., directories, and it contains a bunch of files with miscellaneous names and file type XML: + +```xml + + + + + + + + + + + + +``` + +Imagine that permuter needs the files from description to do its work; but, unlike in the case of the PNG files, every subtask needs every description file. That is, the `set-1` subtask `st-0` needs one file from the `set-1/L0` directory and all of the files in `set-1/description`; the st-1 subtask needs a different file from `set-1/L0`, but it also needs all of the files in `set-1/description`; and so on. What do we do? + +There's an XML attribute (of course) that tells Ziggy to resist the temptation to slice up the files for a data file type into different subtasks. It's the `includeAllFilesInAllSubtasks` attribute, which is applied to the data file type definition: + +```xml + +``` + +The node definition above tells Ziggy that it should form subtasks using files of the `raw data` type, but then give every subtask copies of all of the files of the `description` type. + +##### What If I Need a Data File Type That's Sometimes `includeAllFilesInAllSubtasks`, and Sometimes Not? + +In that case, what you do is define ... two data file types. They need to have different `name`s and values of the `includeAllFilesInAllSubtasks` attribute, but matching `location` and `fileNameRegexp` attributes: + +```xml + + +``` + +Note that we're taking advantage of the fact that `includeAllFilesInAllSubtasks` is an optional attribute that defaults to `false`. + +At this point you're probably thoroughly sick of the entire topic of subtasks, so let's go back to the console. ### Pipeline States @@ -106,14 +203,14 @@ The instances panel also has numerous indicators named `State` and `p-State` tha The possible states for a pipeline instance are described below. -| State | Description | -| -------------- | ------------------------------------------------------------ | +| State | Description | +| ----- | ----------- | | INITIALIZED | Instance has been created but Ziggy hasn't yet gotten around to running any of its tasks. | | PROCESSING | Tasks in this instance are being processed, and none have failed yet. | | ERRORS_RUNNING | Tasks in this instance are being processed, but at least 1 has failed. | -| ERRORS_STALLED | Processing has stopped because of task failures. | -| STOPPED | Not currently used. | -| COMPLETED | All done! | +| ERRORS_STALLED | Processing has stopped because of task failures. | +| STOPPED | Not currently used. | +| COMPLETED | All done! | About ERRORS_RUNNING and ERRORS_STALLED: as a general matter, tasks that are running the same algorithm in parallel are totally independent, so if one fails the others can keep running; this is the ERRORS_RUNNING state. However: once all tasks for a given algorithm are done, if one or more has failed, it's not guaranteed that the next algorithm can run. After all, a classic pipeline has the outputs from one task become the inputs of the next, and in this case some of the outputs from some of the tasks aren't there. In this case, the instance goes to ERRORS_STALLED, and nothing more will happen until the operator addresses whatever caused the failure. @@ -121,35 +218,35 @@ About ERRORS_RUNNING and ERRORS_STALLED: as a general matter, tasks that are run The possible states for a pipeline task are described below. -| State | Description | -| ----------- | ------------------------------------------------------------ | -| INITIALIZED | Task has been created and is waiting for some kind of attention. | -| SUBMITTED | The task will run as soon as the supervisor has available resources to devote to it. | -| PROCESSING | The task is running. | -| ERROR | All subtasks have run, and at least one subtask has failed. | +| State | Description | +| ----- | ----------- | +| INITIALIZED | Task has been created and is waiting for some kind of attention.| +| SUBMITTED | The task will run as soon as the supervisor has available resources to devote to it.| +| PROCESSING | The task is running. | +| ERROR | All subtasks have run, and at least one subtask has failed. | | COMPLETED | All subtasks completed successfully and results were copied back to the datastore. | -| PARTIAL | Not currently used. | +| PARTIAL | Not currently used. | | #### Pipeline Task Processing States (p-States) When a task is in the `PROCESSING` state, it's useful to have a more fine-grained sense of what it's doing, where it is in the process, etc. This is the role of the processing state, or `P-state`, of the task. Each `P-state` has an abbreviation that's shown in the last column of the tasks table. The `P-states` are shown below. -| P-state | Abbreviation | Description | -| -------------------- | ------------ | ------------------------------------------------------------ | +| P-state | Abbreviation | Description | +| ------- | ------------ | ----------- | | INITIALIZING | I | Nothing has happened yet, the task is still in the state it was in at creation time. | -| MARSHALING | M | The inputs for the task are being assembled. | +| MARSHALING | M | The inputs for the task are being assembled. | | ALGORITHM_SUBMITTING | As | The task is ready to run and is being sent to whatever system is in charge of scheduling its execution. | | ALGORITHM_QUEUED | Aq | In the case of execution environments that use a batch system, the task is waiting in the batch queue to run. | -| ALGORITHM_EXECUTING | Ae | The algorithm is running, data is getting processed. | -| ALGORITHM_COMPLETE | Ac | The algorithm is done running. | +| ALGORITHM_EXECUTING | Ae | The algorithm is running, data is getting processed. | +| ALGORITHM_COMPLETE | Ac | The algorithm is done running. | | STORING | S | Ziggy is storing results in the datastore. Sometimes referred to as "persisting." | -| COMPLETE | C | The results have been copied back to the datastore. | +| COMPLETE | C | The results have been copied back to the datastore. | ### Worker The `Worker` column on the tasks table shows which worker is managing task execution. -Right now, the workers all run on the same system as the supervisor, which is also the same system that runs the console. As a result, all the workers are "localhost" workers. At some point this may change, and the supervisor, workers, and console can conceivably run on different systems. For this reason we've left the "localhost" part of the display, in an effort to future-proof it. +Right now, the workers all run on the same system as the supervisor, which is also the same system that runs the console. As a result, all the workers are "localhost" workers. At some point this may change, and the supervisor, workers, and console can conceivably run on different systems. For this reason we've left the "localhost" part of the display, in an effort to future-proof it. Recall from the discussion on [Running the Cluster](running-pipeline.md) that the supervisor can create multiple worker processes that can execute in parallel. The worker number tells you which of these is occupied with a given task. diff --git a/doc/user-manual/intermediate-topics.md b/doc/user-manual/intermediate-topics.md index 9c0fbd6..4a83169 100644 --- a/doc/user-manual/intermediate-topics.md +++ b/doc/user-manual/intermediate-topics.md @@ -1,6 +1,6 @@ -[[Previous]](organize-tables.md) +[[Previous]](datastore-regexp.md) [[Up]](user-manual.md) [[Next]](datastore-task-dir.md) @@ -12,10 +12,10 @@ Now that you've [successfully run a pipeline](start-pipeline.md), there are some The permanent, and temporary, storage areas for data and other files. -### [The Task Configuration Parameter Sets](task-configuration.md) +### [Setting up a Relational Database Management System (RDBMS)](rdbms.md) -How Ziggy knows how to divide work up into tasks, and much, much more. +How to give Ziggy the database it needs for managing its information. -[[Previous]](organize-tables.md) +[[Previous]](datastore-regexp.md) [[Up]](user-manual.md) [[Next]](datastore-task-dir.md) diff --git a/doc/user-manual/module-parameters.md b/doc/user-manual/module-parameters.md index 29728ce..67085ba 100644 --- a/doc/user-manual/module-parameters.md +++ b/doc/user-manual/module-parameters.md @@ -2,7 +2,7 @@ [[Previous]](configuring-pipeline.md) [[Up]](configuring-pipeline.md) -[[Next]](data-file-types.md) +[[Next]](datastore.md) ## Module Parameters @@ -14,19 +14,7 @@ Module parameters are organized into groups known as "parameter sets." When Zigg Ziggy expects its parameter set XML files to start with "pl-" (for "Parameter Library"). In the case of the sample pipeline, take a look at [config/pl-sample.xml](../sample-pipeline/config/pl-sample.xml). Note that you also don't need to confine yourself to a single parameter library file; you can have as many as you like. -### Execution Control Parameters - -Ziggy provides two flavors of pre-defined parameter sets that are used by Ziggy itself to control parts of its execution: the remote parameters, that control execution on a high-performance computing system; and the task configuration parameters, which define how Ziggy subdivides data for execution in chunks. - -For now, we're not going to talk further about these kinds of parameters. They'll be discussed at greater length in sections on running the pipeline. However, if you can't bear the suspense, see the following articles: - -[Remote Parameters](remote-parameters.md) - -[Task Configuration Parameters](task-configuration.md) - -### Parameters Used by Algorithm Modules - -This is more what you need to worry about as you're designing your algorithms. In `pl-sample.xml`, swim down to the last parameter set: +Because the sample pipeline is extremely simple, we have only one parameter library file and it, in turn, has only one parameter set: ```xml @@ -45,4 +33,4 @@ Note that both the parameter set name and the parameter name can have whitespace [[Previous]](configuring-pipeline.md) [[Up]](configuring-pipeline.md) -[[Next]](data-file-types.md) +[[Next]](datastore.md) diff --git a/doc/user-manual/monitoring.md b/doc/user-manual/monitoring.md index 1fcab7b..00f64a1 100644 --- a/doc/user-manual/monitoring.md +++ b/doc/user-manual/monitoring.md @@ -26,7 +26,7 @@ Why are there no workers? Workers are created as needed, and each worker is assigned to perform a particular pipeline task (or a portion of a particular task). Once the worker's job is done, the worker is closed down, and if the supervisor later needs a worker process, it will create one on-the-fly. Thus when you run a pipeline that has a total of 6 tasks, and has a worker count of 2, the supervisor will create a total of 6 workers over the course of execution, but only 2 will be running at any given time. -Thus the blank workers display: when there are no tasks running, there are no workers running; and when a worker isn't running, it ceases to exist. If you were to start a new pipeline with 2 workers, after the delay mentioned in the first point you'd see something like this: +Thus the blank workers display: when there are no tasks running, there are no workers running; and when a worker isn't running, it ceases to exist. If you were to start the sample pipeline with the default maximum number of workers set to 6, after the delay mentioned in the first point you'd see that only 2 workers would be started to process the two tasks. diff --git a/doc/user-manual/nicknames.md b/doc/user-manual/nicknames.md index f5b42b4..4400e20 100644 --- a/doc/user-manual/nicknames.md +++ b/doc/user-manual/nicknames.md @@ -6,7 +6,7 @@ ## Creating Ziggy Nicknames -Ziggy nicknames were introduced in the article (Running the Pipeline)[running-pipeline.md]. Those nicknames are defined by properties in `ziggy.properties` with `ziggy.nickname.` prefixes. There is another property called `ziggy.default.jvm.args` that is added to any JVM arguments that appear in those properties. +Ziggy nicknames were introduced in the article [Running the Pipeline](running-pipeline.md). Those nicknames are defined by properties in `ziggy.properties` with `ziggy.nickname.` prefixes. There is another property called `ziggy.default.jvm.args` that is added to any JVM arguments that appear in those properties. You can add your own nicknames to your own property file that is referred to by `PIPELINE_CONFIG_PATH`. You can find examples of the format in `ziggy.properties`, which is this: diff --git a/doc/user-manual/organizing-tables.md b/doc/user-manual/organizing-tables.md index 8f57ee3..a3b7b83 100644 --- a/doc/user-manual/organizing-tables.md +++ b/doc/user-manual/organizing-tables.md @@ -2,11 +2,11 @@ [[Previous]](change-param-values) [[Up]](ziggy-gui.md) -[[Next]](intermediate-topics.md) +[[Next]](edit-pipeline.md) ## Organizing Pipelines and Parameter Sets -The sample pipeline, as discussed earlier, is a pretty trivial example of what you can do with Ziggy. There's only one pipeline defined (`sample`), and only 5 parameter sets. +The sample pipeline, as discussed earlier, is a pretty trivial example of what you can do with Ziggy. There's only one pipeline defined (`sample`), and only 1 parameter set. In real life, you may well wind up with a configuration that is, shall we say, somewhat richer. For example, the current TESS science data processing system has over a dozen pipelines and over 100 parameter sets! There are pipelines and parameter sets that are only used in system testing; pipelines and parameter sets that are targeted to one of the three types of flight data acquired by the instruments; pipelines and parameter sets that were used earlier in the mission but are now obsolete; and so on. @@ -18,7 +18,7 @@ Let's take another look at the parameter library display: -The display shows all of the parameter sets under an icon labeled ``. If you right-click on a parameter set, you get the following context menu: +The display shows the single parameter set under an icon labeled ``. If you right-click on a parameter set, you get the following context menu: @@ -36,7 +36,7 @@ If you now press the `OK` button, you'll see the following change to the paramet -The `Single subtask configuration` parameter set is no longer visible, but there's a spot in the table for the `Test` group defined previously. If you click the `+` button next to `Test` to expand the group, you see this: +The `Algorithm Parameters` parameter set is no longer visible, but there's a spot in the table for the `Test` group defined previously. If you click the `+` button next to `Test` to expand the group, you see this: @@ -51,8 +51,8 @@ With this wisdom in hand, let's look again at the `Pipelines` panel: -Based on our experience with parameter sets, it's now clear that we can group pipelines in the same way that we can group parameter sets. Given that there's only one pipeline in the sample, it would be really ludicrous to do so, but someday when you've got a couple of dozen actual pipelines you'll thank us for this. +Based on our experience with parameter sets, it's now clear that we can group pipelines in the same way that we can group parameter sets. Given that there's only one pipeline in the sample, it would be as ludicrous as our example above, but someday when you've got a couple of dozen actual pipelines you'll thank us for this. [[Previous]](change-param-values) [[Up]](ziggy-gui.md) -[[Next]](intermediate-topics.md) \ No newline at end of file +[[Next]](edit-pipeline.md) \ No newline at end of file diff --git a/doc/user-manual/pipeline-definition.md b/doc/user-manual/pipeline-definition.md index 13dd320..4900863 100644 --- a/doc/user-manual/pipeline-definition.md +++ b/doc/user-manual/pipeline-definition.md @@ -1,6 +1,6 @@ -[[Previous]](data-file-types.md) +[[Previous]](datastore.md) [[Up]](configuring-pipeline.md) [[Next]](building-pipeline.md) @@ -40,23 +40,21 @@ Note that data receipt is the only pre-defined module in Ziggy. There's more inf #### Back to the Pipeline Definition -The next chunk of the pipeline definition is thus: +The next chunk of the pipeline definition is thus (minus comments, which I removed in the interest of brevity): ```xml - - - - - - - - - - - - + + + + + + + + + + ``` Each step in the pipeline is a node. The `node` specifies the name of the module for that node and the name of any nodes that execute next, as `childNodeNames`. Here we see the `data-receipt` node is followed by `permuter`, and `permuter` is followed by `flip`. @@ -65,9 +63,18 @@ Each step in the pipeline is a node. The `node` specifies the name of the module Parameter sets can be supplied for either the entire pipeline as a whole, or else for individual nodes. -In the text above we see a `pipelineParameter` named `Algorithm Parameters`. This means that the `Algorithm Parameters` set from `pl-sample.xml` will be provided to each and every module when it starts to execute. By contrast, the `Data receipt configuration` parameter set is provided as a `moduleParameter` to the `data-receipt` node. This means that the data receipt module will get the `Data receipt configuration` parameter set provided, but the permuter module will not. +In the text above we see a `pipelineParameter` named `Algorithm Parameters`. This means that the `Algorithm Parameters` set from `pl-sample.xml` will be provided to each and every module when it starts to execute. On the other hand, it's possible to imagine that the permuter module would have some parameters that it needs but which aren't used by the other modules. To do this, we would put a `moduleParameter` element into the `permuter` dode definition. Here's what that would look like: + +```xml + + + + + + +``` -A given parameter set can be provided as a `moduleParameter` to any number of nodes. For example, the `Multiple subtask configuration` parameter set is provided as a `moduleParameter` to `permuter` and also to `flip` (not shown above, but it's in the XML file, check it out). This provides fairly fine-grained control in terms of which parameter sets go to which nodes. +A given parameter set can be provided as a `moduleParameter` to any number of nodes. For example, if we wanted to provide `Some other parameter set` to both `permuter` and `flip`, but not to `data-receipt` or `average`, we could simply copy the `moduleParameter` element from the `permuter` node definition into the `flip` node definition. ##### Data File and Model Types @@ -75,6 +82,6 @@ Each node can have `inputDataFileType`, `outputDataFileType`, and `modelType` el A node can have multiple output types (see for example the `flip` node in `pd-sample.xml`) or multiple input types (as in the `averaging` node). Each node can use any combination of model types it requires, and each model type can be provided to as many nodes as need it. -[[Previous]](data-file-types.md) +[[Previous]](datastore.md) [[Up]](configuring-pipeline.md) [[Next]](building-pipeline.md) diff --git a/doc/user-manual/properties.md b/doc/user-manual/properties.md index 0b6a3c6..030f88b 100644 --- a/doc/user-manual/properties.md +++ b/doc/user-manual/properties.md @@ -29,6 +29,8 @@ The properties manager can also reach out to the environment and pull in the val hibernate.connection.username = ${env:USER} ``` +See also the `ziggy.pipeline.environment` property. + ### Pipeline Properties vs Ziggy Properties Ziggy actually uses two properties files. @@ -76,7 +78,7 @@ The default value is either defined by code or by `ziggy.properties`. If the def | ziggy.pipeline.data.receipt.validation.maxFailurePercentage | Maximum percentage of files that can fail validation before DR throws an exception | 100 | | ziggy.pipeline.datastore.dir | Root directory for datastore | None | | ziggy.pipeline.definition.dir | Location for XML files that define the pipeline | None | -| ziggy.pipeline.environment | Comma-separated list of name-value pairs of environment variables that should be provided to the algorithm at runtime. | "" | +| ziggy.pipeline.environment | Comma-separated list of name-value pairs of environment variables that should be provided to the algorithm at runtime. Note that whitespace around the commands is not allowed. | "" | | ziggy.pipeline.home.dir | Top-level directory for the pipeline code. | None | | ziggy.pipeline.libPath | Colon-separated list of directories to search for shared libraries such as files with .so or .dylib suffix (LD_LIBRARY_PATH is ignored by Ziggy) | "" | | ziggy.pipeline.mcrRoot | Location of the MATLAB Compiler Runtime (MCR), including the version, if MATLAB algorithm executables are used | "" | diff --git a/doc/user-manual/rdbms.md b/doc/user-manual/rdbms.md index d9ef5dc..3a7525a 100644 --- a/doc/user-manual/rdbms.md +++ b/doc/user-manual/rdbms.md @@ -1,6 +1,6 @@ -[[Previous]](task-configuration.md) +[[Previous]](datastore-task-dir.md) [[Up]](user-manual.md) [[Next]](troubleshooting.md) @@ -119,6 +119,6 @@ As with a PostgreSQL system database, a database that uses some other RDBMS appl 1 HSQLDB can be configured to run from on-disk catalogs and can theoretically handle 64 TB of data but that is beyond the scope of this document. -[[Previous]](task-configuration.md) +[[Previous]](datastore-task-dir.md) [[Up]](user-manual.md) [[Next]](troubleshooting.md) diff --git a/doc/user-manual/redefine-pipeline.md b/doc/user-manual/redefine-pipeline.md index e547d4e..5889b64 100644 --- a/doc/user-manual/redefine-pipeline.md +++ b/doc/user-manual/redefine-pipeline.md @@ -2,7 +2,7 @@ [[Previous]](parameter-overrides.md) [[Up]](dusty-corners.md) -[[Next]](edit-pipeline.md) +[[Next]](nicknames.md) ## Redefining a Pipeline @@ -40,4 +40,4 @@ And that's all there is to it! [[Previous]](parameter-overrides.md) [[Up]](dusty-corners.md) -[[Next]](edit-pipeline.md) +[[Next]](nicknames.md) diff --git a/doc/user-manual/remote-dialog.md b/doc/user-manual/remote-dialog.md index 701914b..b06b110 100644 --- a/doc/user-manual/remote-dialog.md +++ b/doc/user-manual/remote-dialog.md @@ -1,6 +1,6 @@ -[[Previous]](remote-parameters.md) +[[Previous]](select-hpc.md) [[Up]](select-hpc.md) [[Next]](hpc-cost.md) @@ -16,15 +16,61 @@ Go to the `Pipelines` panel and double-click on the sample pipeline row in the t -A whole new dialog box we've never seen before! But actually we're just going to use it to get to yet another one. Select `permuter` from the modules list and press `Remote execution`. You'll see this: +Now select `permuter` from the modules list and press `Remote execution`. You'll see this: - + -### Using the Remote Parameters Dialog Box +### Using the Remote Execution Dialog Box -Notice that the values you've set in the `RemoteParameters` instance for permuter have been populated, as has the total number of subtasks that Ziggy found for this node, based on the contents of the datastore. In the interest of making this more realistic, change the number in `Total subtasks` to 1000, then hit the `Calculate` button. You'll see this: +The first thing to notice is that there are some parameters that are in a group labeled Required parameters, and other in a group labeled Optional parameters. Let's talk first about the ... - +#### Required Parameters + +##### Enable remote execution + +This is the one that tells Ziggy that you want to run this pipeline module on a high performance computing system of some sort. Obviously when the check box is checked, Ziggy will farm out the module execution to the appropriate batch system; when it's unchecked, Ziggy will run all the tasks on the local system (i.e., the system where the Ziggy process is running). Ziggy will not let you save the configuration via the `Close` button until all of the required parameters have been entered. + +##### Run one subtask per node + +This one's a bit tricky to explain, so bear with me. + +The usual way that Ziggy performs parallel execution on a remote system is that it starts a bunch of compute nodes and then, on each compute node, as many subtasks as possible run in parallel, depending on the number of cores and amount of RAM on the compute node. The advantage of this is that the folks who write the algorithm code don't need to do anything special to their software to take advantage of parallel execution: just plug your algorithm in and let Ziggy do the rest. + +In some cases, though, there are algorithm packages that have their own parallelization that was implemented by the subject matter experts. Typically these algorithms make use of one of many third-party concurrency libraries available for most computer languages. In these cases, you probably don't want multiple subtasks all vying for the CPUs and RAM on a compute node; you want one subtask to run on the compute node, and that subtask should farm out work using its own parallel processing capabilities. + +In this latter case, you should check the `Run one subtask per node` check box. This tells Ziggy to defer to the algorithm's parallelism and not to try to use Ziggy's subtask parallelism. + +##### Total tasks and subtasks + +The total number of tasks and subtasks. + +If the datastore already has the input files for the module you're working on, then Ziggy can figure out how many tasks will be needed, and how many subtasks each task will need (it does this by running the code that's used by Ziggy to determine the task and subtask populations). In this example, because Data Receipt has been run, Ziggy knows how to set the task and subtask count parameters for Permuter (which gets all its inputs from the files that got read in by Data Receipt). Consequently, it will do so automatically. + +On the other hand, consider the case in which the inputs to a module do not yet exist. For example, imagine that we've just run data receipt but none of the other pipeline modules. If you ask for remote execution parameters for Permuter, it can figure out the task and subtask counts. On the other hand, if you try to generate remote execution parameters for Flip, it will show 0 tasks and 0 subtasks. This is because Flip's inputs are Permuter's outputs. If Permuter hasn't yet generated its outputs, there's nothing there to let Ziggy do the task/subtask calculation for Flip. In this case, if you want to generate remote parameter estimates, you'll need to fill in estimates for the task count and subtask count yourself. + +Note that, even in the case in which Ziggy can fill in task and subtask counts, you can delete the values it comes up with and put in your own! This is helpful when you've just used a small run to determine things like the gigs per subtask, and now you want to see how performance will scale to a much larger run. + +##### Gigs per subtask + +This is the maximum amount of RAM you expect a single subtask to consume at any given time in its execution, in gigabytes. + +##### Max subtask wall time and Typical subtask wall time + +These are estimates of how much time a subtask will need in order to finish. In general, subtask wall times will have a distribution, with most tasks executing in a time X while a few stragglers will require a time Y > X. Enter the "most tasks time," X, for `Typical subtask wall time` and the "stragglers time," Y, for `Max subtask wall time`. + +##### Scale wall time by number of cores + +This option is only used when `Run one subtask per node` is enabled. + +The issue here is that the wall time needed for each subtask depends on the number of cores in a given node. If a subtask runs on a node with 32 cores, it will probably need only half as much wall time as if it runs on one with 16 cores. + +To ensure that the wall time is set correctly on any compute node, no matter how many cores it has, the user can enter in the `Max subtask wall time` and `Typical subtask wall time` the time that subtasks will take if they're given only 1 core. When the compute node architecture is selected, Ziggy will scale down the wall time by the number of cores in that architecture, assuming a simple linear scaling. + +#### Calculate the PBS Parameters + +Once the required parameters have been entered, Ziggy will convert the remote execution parameters on the left hand side of the dialog box into the parameters that will be used in the PBS submission if and when you run this module on the HPC system. In this example, the PBS parameters have not been generated because the gigs per subtask and wall time per subtask are set to zero. Note that Ziggy highlights those incomplete fields. The total tasks and total subtasks fields have been auto-filled with a value of 2 and 8 respectively, which is correct but not very interesting. For the purposes of the example, let's set `Total subtasks` to 1000, `Gigs per subtask` to 10, and both wall time parameters to 0.15 hours. You'll see this: + + The parameters that will be used in the request to PBS are shown in the `PBS parameters` section. Ziggy will ask for 84 nodes of the Haswell type, for 15 minutes each; the total cost in Standard Billing Units (SBUs) will be 16.8. @@ -32,11 +78,17 @@ A Haswell node at the NAS has 24 cores and 128 GB of RAM. Since we've asserted t What did Ziggy actually do here? Given the parameters we supplied, Ziggy looked for the architecture that would minimize the cost in SBUs, which turns out to be Haswell, and it asked for enough nodes that all of the subtasks could execute in parallel. This latter minimizes the estimated wall time, but at the expense of asking for a lot of nodes. -#### Setting the Maximum Number of Nodes +We can now tune the PBS request that Ziggy makes on our behalf by making use of ... + +#### Optional Parameters + +The optional parameters are there in case Ziggy produces a ludicrous PBS request, and you want to apply some additional limits to get the request to be less insane. -Given the above, it might be smarter to ask for fewer nodes. If we change the `Max nodes` value to 10 and press `Calculate`, this is what we see: +##### Maximum nodes per task - +Given the above, it might be smarter to ask for fewer nodes. If we change the `Max nodes` value to 10, this is what we see: + + As expected, the number of remote nodes went down and the wall time went up. What's unexpected is that the total cost also went down! What happened? @@ -48,48 +100,44 @@ In the second example, given the parameters requested, the actual wall time need That said: Once the HPC has processed all the subtasks, the jobs all exit and the nodes are returned to the HPC pool. The user is only charged for the actual usage. In the first case, what would have happened is that all the jobs would finish early, and we'd only get billed for what we actually used, which would be more like 10 SBUs than 17 SBUs. -#### Selecting a Different Optimizer - -Right now the `Optimizer` is set to `COST`, meaning that Ziggy will attempt, within the limits of its smarts, to find the compute node architecture that minimizes the cost in SBUs. There are 3 other settings available for the optimizer: `CORES`, `QUEUE_DEPTH`, and `QUEUE_TIME`. These are described in detail in the article on [Remote Parameters](remote-parameters.md). - -#### Manually Selecting an Architecture - -The `Architecture` pull-down menu allows you to manually select an architecture for the compute nodes. This in turn allows you to run the wall time and SBU calculation and see the results for each architecture. +##### Optimizer -#### Enabling or Disabling Node Sharing +When Ziggy does its calculations, its default behavior is to select an architecture that minimizes the total cost (in SBUs for NASA's supercomputer, dollars or other currency for other systems). This is reflected in the `Cost` setting of the `Optimizer`. This is the default, but there are three other options: -So far we've implicitly assumed that each compute node can run subtasks in parallel up to the limit of the number of active cores per node. That is to say, in this example a Haswell node will always have 12 permuter subtasks running simultaneously. This is the situation we call "Node Sharing." +- `Cores`: As mentioned above, depending on the amount of RAM required for each subtask, you may find that, on some or even all architectures, it's not possible to run subtasks in parallel on all the cores; in order to free up enough RAM for the subtasks, some cores must be idled. The `Cores` option minimizes the number of idled cores. +- `Queue depth`: This is one of the optimizers that tries to minimize the time spent waiting in the queue. The issue here is that some architectures are in greater demand than others. The `Queue depth` optimization looks at each architecture's queued jobs and calculates the time it would take to run all of them. The architecture that has the shortest time based on this metric wins. +- `Queue time`: This is a different optimization related to queues, but in this case it attempts to minimize the total time you spend waiting for results (the time in queue plus the time spent running the jobs). This looks at each architecture and computes the amount of queue time "overhead" that typical jobs are seeing. The architecture that produces the shortest total time (queue time plus execution time) wins. -There are cases when this won't be true -- when it won't be safe to try to force a node to run many subtasks in parallel. In those cases, you can uncheck the Node Sharing check box and run the PBS parameter calculation. You get this: +##### Architecture - +The `Architecture` pull-down menu allows you to manually select an architecture for the compute nodes rather than allowing Ziggy to try to pick one for you. The console will show you the capacity and cost of the selected architecture to the right of the combo box as well as the estimated wall time and SBU calculation for that architecture in the `PBS parameters` section. -Ziggy no longer asks about GB per subtask, because only 1 subtask will run at a time on each node, so there's an assumption that the available RAM on any architecture will be sufficient. The cost has ballooned, which is expected since now each node can only process 1 subtask at a time. Somewhat unexpectedly, the optimal architecture has changed. This is because, with each node processing 1 subtask at a time, the benefits to having a lot of cores in a compute node go away and architectures with fewer cores are potentially favored. +##### Subtasks per core -Now: hopefully, the reason you're disabling node sharing is because your algorithm program has its own, internal concurrency support, and that support spreads the work of the subtask onto all the available cores on the compute node. When this is the case, check the `Wall Time Scaling` box. When this box is checked, Ziggy assumes that the wall times provided by the user are wall times for processing on 1 core, and that the actual time will go inversely with the number of cores (i.e., the parallelization benefit is perfect). When you do this and calculate the parameters, you see this: - - - -This results in a cost even lower than what we saw before! This is because, in this configuration, Ziggy assumes that every core can be utilized regardless of how much RAM per core the architecture has. +The `Subtasks per core` option does something similar to the `Max nodes per task` option. Both of these options tell Ziggy to reduce its node request in exchange for asking for a longer wall time. The difference is that the `Max nodes per task` option does this explicitly, by setting a limit on how many nodes Ziggy can ask for. `Subtasks per core`, by contrast, applies an implicit limit. In the default request, Ziggy will ask for enough nodes that every core processes one and only one subtask, so that all the subtasks get done in one "wave," as it were. The `Subtasks per core` option tells Ziggy to tune its request so that each active core processes multiple subtasks, resulting in a number of "waves" of processing equal to the `Subtasks per core` value. #### Selecting A Batch Queue Under ordinary circumstances, it's best to leave the Queue selection blank so that it can be selected based on the required execution time resources. There are two non-ordinary circumstances in which it makes sense to select a queue manually. -The first circumstance is when you want to use either the `DEVEL` or the `DEBUG` queue on the NASA Advanced Supercomputer (NAS). These are special purpose queues that allow users to execute jobs at higher priority but which set a low limit on the number of nodes and amount of wall time that can be requested. Ziggy will never select these queues for you, but if you think you should use them you can select them yourself. +The first circumstance is when you are using a reserved queue. In this case, when you select Reserved in the Queue selection box, the Reserved queue name text field will be enabled and can enter your reservation queue, which is named with the letter R followed by a number. + +The second circumstance is when you want to use either the `Devel` or the `Debug` queue on the NASA Advanced Supercomputer (NAS). These are special purpose queues that allow users to execute jobs at higher priority but which set a low limit on the number of nodes (2 and 1 respectively) and amount of wall time that can be requested. Ziggy will never select these queues for you, but if you think you should use them you can select them yourself. + +If you select either of these two queues, the maximum number of nodes field is filled in for you. In addition, you also have to ensure that you only have 1 or 2 tasks respectively. In the case of the sample pipeline, you can change the unit of work so that a single task is generated. To do this, close the Remote execution dialog, select the `permuter` module, for example, and press the `Parameters` button. In the Edit parameter sets dialog, double-click on the `Single subtask configuration` parameter. Then change the `taskDirectoryRegex` parameter to `set-1`. When you save these settings, you'll find that when you start the Permuter module, only one task--the one associated with the set-1 directory--will start. -The second circumstance is when you are using a reserved queue. In this case, when you select RESERVED in the Queue selection box, you'll be prompted for a queue name (and you won't be able to get rid of the prompt until you enter a valid queue name). When you save the results from the remote parameters dialog box to the database, the queue name you entered will be put into the remote parameters instance for the pipeline module. +In any case, it may be instructive to select the various queues to see the maximum wall times for each queue displayed to the right of the combo box. ### Keeping or Discarding Changes -After some amount of fiddling around, you may reach a configuration that you like, and you'd like to ensure that Ziggy uses that configuration when it actually submits your jobs. Alternately, you might realize that you've made a total mess and you want to discard all the changes you've made and start over (or just go home). +After some amount of fiddling around, you may reach a configuration that you like, and you'd like to ensure that Ziggy uses that configuration when it actually submits your jobs. Alternately, you might realize that you've made a total mess and you want to discard all the changes you've made and start over (or just go home). -Let's start with the total mess case. If you press the `Reset` button, the remote parameters will be set back to their values from when the remote execution dialog opened. At this point you can try again. Alternately, if you've decided to give up on this activity completely, press the `Cancel` button: this will discard all your changes and close the remote execution dialog box. +Let's start with the total mess case. If you press the `Reset` button, the remote parameters will be set back to their values from when the Remote execution dialog opened. At this point you can try again. Alternately, if you've decided to give up on this activity completely, press the `Cancel` button: this will discard all your changes and close the Remote execution dialog box. -Alternately, you might think that your changes are pretty good and you want to hold onto them. If this is the case, press the `Close` button. This button saves your changes to the remote parameters, but **it only saves them to the Edit pipeline dialog box!** What this means is that if you hit `Close` and then press the `Remote execution` button on the Edit pipeline dialog box again, the values you see in the remote execution dialog box will be the ones you saved earlier with the `Close` button. Relatedly, if you make a bunch of changes now and decide to use the `Reset` button, the values are reset to the ones you saved via the `Close` button in your prior session with the remote execution dialog. +Alternately, you might think that your changes are pretty good and you want to hold onto them. If this is the case, press the `Close` button. This button saves your changes to the remote parameters, but **it only saves them to the Edit pipeline dialog box!** What this means is that if you hit `Close` and then press the `Remote execution` button on the Edit pipeline dialog box again, the values you see in the Remote execution dialog box will be the ones you saved earlier with the `Close` button. Relatedly, if you make a bunch of changes now and decide to use the `Reset` button, the values are reset to the ones you saved via the `Close` button in your prior session with the Remote execution dialog. -If you decide that you're so happy with your edits to remote execution that you want Ziggy to actually store them and use them when running the pipeline, you need to press the `Save` button on the Edit pipeline dialog. This will save all the changes you've made since you started the Edit pipeline dialog box. Alternately, if you realize that you've messed something up and want Ziggy to forget all about this session, you can use the `Cancel` button. +If you decide that you're so happy with your edits to remote execution that you want Ziggy to actually store them and use them when running the pipeline, you need to press the `Save` button on the Edit pipeline dialog. This will save all the changes you've made since you started the Edit pipeline dialog box. Alternately, if you realize that you've messed something up and want Ziggy to forget all about this session, you can use the `Cancel` button. -[[Previous]](remote-parameters.md) +[[Previous]](select-hpc.md) [[Up]](select-hpc.md) [[Next]](hpc-cost.md) diff --git a/doc/user-manual/remote-parameters.md b/doc/user-manual/remote-parameters.md deleted file mode 100644 index fa900d5..0000000 --- a/doc/user-manual/remote-parameters.md +++ /dev/null @@ -1,154 +0,0 @@ - - -[[Previous]](select-hpc.md) -[[Up]](select-hpc.md) -[[Next]](remote-dialog.md) - -## Remote Parameters - -The way that you set up a pipeline module to run on a remote (i.e., high-performance computing / cloud computing) system is to create a `ParameterSet` of the `RemoteParameters` class, and then make it a module parameter set for the desired node. - -Wow! What does all that mean? Let's start with the second half: "make it a module parameter set for the desired node." If you look at `pd-sample.xml`, you'll see this: - -```xml - - - - - - - -``` - -That line that says ` `tells Ziggy that there's a parameter set with the name `Remote Parameters (permute color)` that it should connect to tasks that run the `permuter` module in the sample pipeline. - -Now consider the first part of the sentence: "create a `ParameterSet` of the `RemoteParameters` class." If you look at pl-sample.xml, you'll see this: - -```xml - - - - - - - - - -``` - -This is the parameter set that, we saw above, got attached to the `permuter` node. - -Let's go back to that sentence: all we need to do to run on a supercomputer is to have one of these parameter sets in the parameter library file, and tell the relevant node to use it. Is that true? - -Yes. Pretty much. But as always, the deity is in the details. - -### RemoteParameters class - -Unlike the parameter sets you'll want to construct for use by algorithms, the parameter set above is supported by a Java class in Ziggy: `gov.nasa.ziggy.module.remote.RemoteParameters`. This means you'll need to include that XML attribute for any parameter set that you want to use to control remote execution, and that you can't make up your own parameters for the parameter set; you'll need to stick to the ones that the Java class defines (but on the other hand you won't need to specify the data types, since the definition of `RemoteParameters` does that for you). - -The `RemoteParameters` class has a lot of parameters, but there are only four that you, personally, must set. These can be set either in the parameter library file or via the [module parameters editor](change-param-values.md) on the console. The remainder can, in principle, be calculated on your behalf when Ziggy goes to submit your jobs via the Portable Batch System (PBS). In practice, it may be the case that you don't like the values that Ziggy selects when left to its own devices. For this reason, you can specify your own values for the optional parameters. Ziggy will still calculate values for the optional parameters you leave blank. In this case, Ziggy's calculated parameters will always (a) result in parameters that provide sufficient compute resources to run the job, while (b) taking into account the values you have specified for any of the optional parameters. - -Note that, rather than setting optional parameters via the module parameters editor, there's an entire separate system in Ziggy that allows you to try out parameter values, see what Ziggy calculates for the remaining optional parameters, and make changes until you are satisfied with the result. This is the [remote execution dialog](remote-dialog.md). - -Anyway, let's talk now about all those parameters. - -#### Required Parameters - -##### enabled (boolean) - -The `enabled` parameter does what it sounds like: if `enabled` is true, the node will use remote execution; if it's false, it will run locally (i.e., on the same system where the supervisor process runs). - -Note that, since you can edit the parameters, you can use this parameter to decide at runtime whether to run locally or remotely. - -##### minSubtasksForRemoteExecution (int) - -One thing about remote execution is that you may not know whether you want remote execution when you're about to submit the task. How can that happen? The main way is that you might not know at that time how many subtasks need to run! You may be in a situation where if there's only a few subtasks you'd rather run locally, but if there are more you'll use HPC. - -Rather than force you to figure out the number of subtasks and manually select or deselect `enabled`, Ziggy provides a way to override the `enabled` parameter and force it to execute locally if the number of subtasks is small enough. This is controlled by the `minSubtasksForRemoteExecution` parameter. If remote execution is enabled, Ziggy will determine the number of subtasks that need to be run in the task. If the number is smaller than `minSubtasksForRemoteExecution`, Ziggy will execute the task locally. - -The default value for this parameter is zero, meaning that Ziggy will always run remotely regardless of how many or how few subtasks need to be processed. - -##### subtaskTypicalWallTimeHours and subtaskMaxWallTimeHours (float) - -"Wall Time" is a $10 word meaning, "Actual time, as measured by a clock on the wall." This term is used to distinguish it from values like "CPU time" (which is wall time multiplied by the number of CPUs), or other compute concepts that refer to time in some way. Compute nodes in HPC systems are typically reserved in units of wall time, rather than CPU time or any other parameter. - -When Ziggy is computing the total resources needed for a particular task, it needs to know the wall time that a single, typical subtask would need to run from start to finish. With that information, and the total number of subtasks, it can figure out how many compute nodes will be needed, and how much wall time is needed for each compute node. The typical subtask start-to-finish time is specified as the `subtaskTypicalWallTimeHours`. - -That said: for some algorithms and some data, there will be subtasks that take much longer than the typical subtask to run from start to finish. Consider, for example, an algorithm that can process the typical subtask in 1 hour, but needs 10 hours for some small number of subtasks. If you ignore the handful of tasks that need 10 hours, you (or Ziggy) might be tempted to ask for a large number of compute nodes, with 1 hour of wall time for each. If you do this, the subtasks that need 10 hours won't finish. Your requested compute resources need to take into account those long-running subtasks. (This is sometimes analogized as, "If the brownie recipe says bake at 350 degrees for 1 hour, you can't bake at 700 degrees for 30 minutes instead." In this case, a more accurate analogy would be, "You can't split the brownie batter into 2 batches and bake each batch at 350 degrees for 30 minutes.") - -To address this, you specify the `subtaskMaxWallTimeHours`. Ziggy guarantees that it won't ask for a wall time less than this value, which should (fingers crossed!) ensure that all subtasks finish. - -One thing about this: Ziggy will ask for a wall time that's sufficient for the `subtaskMaxWallTimeHours` parameter, but it will ask for a total number of CPU hours that's determined by the subtask count and the `subtaskTypicalWallTimeHours`, under the assumption that the number of long-running subtasks is small compared to the number of typical ones. Imagine for example that you have 1000 subtasks, with a typical wall time of 1 hour and a max wall time of 10 hours; and you're trying to run on a system where the compute nodes have 10 CPUs each. The typical usage would be satisfied by getting 100 compute nodes for 1 hour each, but that leaves the long-running subtasks high and dry. The `subtaskMaxWallTimeHours` tells Ziggy that it needs to ask for 10 hour wall times for this task; thus it will ask for 10 hour wall times, but only 10 compute nodes total. - -##### gigsPerSubtask (float) - -This parameter tells Ziggy how many GB each subtask will need at its peak. - -The reason this needs to be specified is that the compute nodes on your typical HPC facility have some number of CPUs and some amount of RAM per compute node. By default, Ziggy would like to utilize all the cores on all the compute nodes it requests. Unfortunately, it's not guaranteed that each subtask can get by with an amount of RAM given by node total RAM / node total CPUs; if the subtasks need more than this amount of RAM, then running subtasks on all the CPUs of a compute node simultaneously will run out of RAM. Which is bad. - -The specification of `gigsPerSubtask` allows Ziggy to figure out the maximum number of CPUs on each compute node that can run simultaneously, which in turn ensures that it asks for enough compute nodes when taking into account that it may not be possible to run all the cores on the nodes simultaneously. - -#### Optional Parameters - -##### optimizer (String) - -Typical HPC systems and cloud computing facilities have a variety of compute nodes. The different flavors of compute nodes will have different numbers of cores, different amounts of RAM, different costs for use, and different levels of demand (translating to different wait times to get nodes). Ziggy will use the information about these parameters to select a node architecture. - -There are four optimizers that Ziggy allows you to use: - -**COST:** This is the default optimization. Ziggy looks for the node that will result in the lowest cost, taking into account the different per-node costs and different capabilities of the different node architectures (because the cheapest node architecture on a per-node basis might lead to a solution that needs more nodes or more hours, so the "cheapest" node may not result in the cheapest jobs). - -**CORES:** This attempts to minimize the fraction of CPUs left idled. If the subtasks need a lot of RAM, it will optimize for nodes with more RAM per CPU. If there are multiple architectures that have the same idled core fraction (for example, if all of the architectures can run 100% of their nodes), then the lowest-cost solution from the set of "semifinalist" architectures will be picked. - -**QUEUE_DEPTH:** This is one of the optimizers that tries to minimize the time spent waiting in the queue. The issue here is that some architectures are in greater demand than others. The `QUEUE_DEPTH` optimiztaion looks at each architecture's queued jobs and calculates the time it would take to run all of them. The architecture that has the shortest time based on this metric wins. - -**QUEUE_TIME:** This is a different optimization related to queues, but in this case it attempts to minimize the total time you spend waiting for results (the time in queue plus the time spent running the jobs). This looks at each architecture and computes the amount of queue time "overhead" that typical jobs are seeing. The architecture that produces the shortest total time (queue time plus execution time) wins. - -###### A Note About Queue Optimizations - -The optimization options for queue depth and queue time are only approximate, and can potentially wind up being very wrong. This is because the queue management is sufficiently complicated that the current estimates are only modestly reliable predictors of performance. - -What makes the management complicated? For one thing, the fact that new jobs are always being submitted to the queues, and there's no way to predict what gets submitted between now and when your job runs. Depending on the number, size, and priority of jobs that come in before your job runs, these might move ahead of your jobs in the queue. Relatedly, if a user decides to delete a job that's in the queue ahead of you, that represents an unpredictable occurrence that improves your waiting time. Jobs that don't take as long as their full wall time are another unpredictable effect. - -Anyway. The point is, caveat emptor. - -##### remoteNodeArchitecture (String) - -All of the foregoing is about selecting an architecture from the assorted ones that are available on your friendly neighborhood HPC facility. However: it may be the case that this isn't really a free parameter for you! For example: if you have compute nodes reserved, you probably have nodes with a specific architecture reserved. If for this reason, or any other, you want to specify an architecture, use this parameter. - -##### subtasksPerCore (float) - -Ziggy will generally gravitate to a solution in which all the subtasks in a task run simultaneously, which means it will ask for a lot of compute nodes. This parameter allows the user to force Ziggy to a solution that has fewer nodes but for more wall time. A `subtasksPerCore` of 1.0 means all the subtasks run in parallel all at the same time. A value of 2.0 means that 50% of the tasks will run in parallel and the remainder will wait for tasks in the first "wave" to finish before they can run. - -##### maxNodes (int) - -This is a more direct way to force Ziggy to a solution with a smaller number of nodes than it would ordinarily request. When this is set, Ziggy will not pursue any solution that uses more compute nodes than the `maxNodes` value. This will of course result in longer wall times than if you just ask for as many nodes as it takes to finish as fast as possible, but asking for thousands of nodes for 30 seconds each may get you talked about in the control room, and not in a good way. - -Note that Ziggy can request a number of nodes that is smaller than the value for `maxNodes` (which is why it's called `maxNodes` in the first place). This happens if the number of subtasks is small: if you only have 8 subtasks, and `maxNodes` is set to 30, it would clearly be useless to actually ask for 30 nodes, since most of them will be idle but you'll get charged for them anyway. In these sorts of situations (where even the value of `maxNodes` is too large, given the number of subtasks to process), Ziggy will select a number of nodes that ensures that none of the nodes sits idle. - -Note that Ziggy can request a number of nodes that is smaller than the value for `maxNodes` (which is why it's called `maxNodes` in the first place). This happens if the number of subtasks is small: if you only have 8 subtasks, and `maxNodes` is set to 30, it would clearly be useless to actually ask for 30 nodes, since most of them will be idle but you'll get charged for them anyway. In these sorts of situations (where even the value of `maxNodes` is too large, given the number of subtasks to process), Ziggy will select a number of nodes that ensures that none of the nodes sits idle. - -##### nodeSharing (boolean) - -In all the foregoing, we've assumed that the algorithm will permit multiple subtasks to run in parallel on a given compute node (albeit using different CPUs). In some cases, this proves to be untrue! For example, there are algorithms that have their own, internal concurrency support, but they rely on a single process having the use of all the CPUs. If this is the case for your algorithm, then set `nodeSharing` to `false`. This will tell Ziggy that each compute node can only process one subtask at a time, and that it should book nodes and wall times accordingly. - -##### wallTimeScaling (boolean) - -Related to the above: if your algorithm can use all the CPUs on a compute node, it's obviously going to run faster on compute nodes with more CPUs than on nodes with fewer CPUs. But this leads to a problem: how can Ziggy select the correct parameters when the actual run time depends on number of cores, which depends on architecture? - -To avoid having to retype the wall time parameters whenever Ziggy wants a different architecture, enter the wall time parameters that would be valid in the absence of concurrency (i.e., how long it would take to run your algorithm on 1 CPU), and set `wallTimeScaling` to true. When this is set, Ziggy knows that it has to scale the actual wall time per subtask down based on the number of CPUs in each node. Ziggy will assume a simple linear scaling, i.e.: true wall time = wall time parameter / CPUs per node. This probably isn't quite correct, but hopefully is correct enough. - -#### A Note on Setting Optional Parameters - -When you set some of the optional parameters, Ziggy will compute values for all the rest. Ziggy has 2 requirements for this process. First, it **must** use any optional parameter values you specify, it can never change those values. Second, it **must** produce a result that allows the task to run to completion. That is, it has to select enough compute nodes and enough wall time to process all of the subtasks. - -A close reading of the paragraph above reveals a potential problem: what happens if you specify a set of parameters that makes it impossible for Ziggy to satisfy that second requirement? Just as an example, if you set the maximum number of nodes, the compute node architecture, and the requested wall time, it's possible to ask for so little in total compute resources that it's impossible to run all of your subtasks! - -If Ziggy determines that it can't ask for enough resources to run your task, it will throw an exception at runtime, and your pipeline will stop. You'll then need to adjust the parameters and restart. - -The best way to avoid this outcome is to use the [remote execution dialog](remote-dialog.md) to set the optional parameters. The remote execution dialog won't allow you to save your parameter values if they result in tasks that can't finish because they're starved of compute resources. - -[[Previous]](select-hpc.md) -[[Up]](select-hpc.md) -[[Next]](remote-dialog.md) \ No newline at end of file diff --git a/doc/user-manual/select-hpc.md b/doc/user-manual/select-hpc.md index 9003ada..f2fd7f2 100644 --- a/doc/user-manual/select-hpc.md +++ b/doc/user-manual/select-hpc.md @@ -2,7 +2,7 @@ [[Previous]](halt-tasks.md) [[Up]](user-manual.md) -[[Next]](remote-parameters.md) +[[Next]](remote-dialog.md) ## High Performance Computing Overview @@ -16,46 +16,9 @@ At the moment, the NAS is the only supported HPC option for Ziggy. This is mainl In the near future the Ziggy team hopes to add cloud computing support for cloud systems that are supported by NASA. We'll keep you posted. -### The RemoteParameters Parameter Set +#### Enabling and Configuring HPC Execution -Let's look again at the XML that defined the sample pipeline. At one point, you can see this: - -```xml - - - - - - - -``` - - The parameter set `Remote Parameters (permute color)` is defined in the parameter library XML file: - -```xml - - - - - - - - - -``` - -The parameter set has a Java class associated with it, which as you've probably gathered means that it's a parameter set that Ziggy uses for its own management purposes. - -#### Enabling Remote Execution - -If you change the value of `enabled` to true, then any node that includes this parameter set will try to run on HPC. - -That's all it takes. Well, that and access to the NAS. - -Note the implications here. First, for a given pipeline the user can select which nodes will run on HPC and which will run locally (here "locally" means "on the system that hosts the supervisor process", or more generally, "on the system where the cluster runs"). Second, even for nodes that have an instance of `RemoteParameters` connected to them, you can decide at runtime whether you want to run locally or on HPC! - -The RemoteParameters class is discussed in greater detail in [the Remote Parameters article](remote-parameters.md). +See [the article on the Remote Execution dialog box](remote-dialog.md) for details on how to select whether to run remote execution, and how to configure it to your satisfaction. #### HPC Execution Flow @@ -87,14 +50,14 @@ In the event that a really small number of subtasks haven't run when your jobs e In the discussion above, we suggested that, in the event that execution times out, you can resubmit the task and the subtasks that didn't run to completion will then be run (subject to the possibility that some of them will then also time out). We also offered the option to disable remote execution if the number of subtasks gets small enough that your local system (i.e., the system that runs the supervisor process) can readily handle them. This is all true, but it can be a nuisance. For this reason, there are some additional options to consider. -First: Ziggy can automatically resubmit tasks that don't complete successfully. This is discussed in [the article on TaskConfigurationParameters](task-configuration.md). You can specify that, in the event that a task doesn't complete, Ziggy should resubmit it. In fact, you can specify the number of times that Ziggy should resubmit the task: after the number of automatic resubmits is exhausted, the task will wait in the ALGORITHM_COMPLETE processing state for you to decide what to do with it (i.e., try to resubmit it again, or fix underlying software problems, or decide that the number of completed subtasks is sufficient and that you want to move on to persisting the results). +First: Ziggy can automatically resubmit tasks that don't complete successfully. This is discussed in [the article on the Edit Pipeline dialog box](edit-dialog.md). You can specify that, in the event that a task doesn't complete, Ziggy should resubmit it. In fact, you can specify the number of times that Ziggy should resubmit the task: after the number of automatic resubmits is exhausted, the task will wait in the ALGORITHM_COMPLETE processing state for you to decide what to do with it (i.e., try to resubmit it again, or fix underlying software problems, or decide that the number of completed subtasks is sufficient and that you want to move on to persisting the results). At this point, we need to reiterate the warning in the TaskConfigurationParameters article regarding this parameter: Ziggy can't tell the difference between a task that didn't finish because it ran out of wall time and a task that didn't finish because of a bug in the algorithm code somewhere. If there's an algorithm bug, Ziggy will nonetheless resubmit an incomplete task (because Ziggy doesn't know the problem is a bug), and the task will fail again when it hits that bug. -Second: Ziggy allows you to automate the decision on whether a given number of subtasks is too small to bother with remote execution. The automation is a bit crude: as shown in [the RemoteParameters article](remote-parameters.md), there is an option, `minSubtasksForRemoteExecution`. This option does what it says: it sets the minimum number of subtasks that are needed to send a task to remote execution; if the number of subtasks is below this number, the task will run locally, **even if enabled is set to true**! +Second: Ziggy allows you to automate the decision on whether a given number of subtasks is too small to bother with remote execution. The automation is a bit crude: as shown in [the Remote Execution dialog box article](remote-dialog.md), there is an option, `minSubtasksForRemoteExecution`. This option does what it says: it sets the minimum number of subtasks that are needed to send a task to remote execution; if the number of subtasks is below this number, the task will run locally, **even if enabled is set to true**! By using these two parameters, you can, in effect, tell Ziggy in advance about your decisions about whether to resubmit a task and whether to use remote execution even if the number of subtasks to process is fairly small. [[Previous]](halt-tasks.md) [[Up]](user-manual.md) -[[Next]](remote-parameters.md) +[[Next]](remote-dialog.md) diff --git a/doc/user-manual/start-pipeline.md b/doc/user-manual/start-pipeline.md index ed86114..b84168c 100644 --- a/doc/user-manual/start-pipeline.md +++ b/doc/user-manual/start-pipeline.md @@ -32,7 +32,7 @@ A lot of options! For now, just put some kind of identifying text in the `Pipeli As soon as the dialog box disappears, select the `Instances` content menu item. The left side should look something like this1: - + Select your pipeline instance in the table. On the right you see this: diff --git a/doc/user-manual/task-configuration.md b/doc/user-manual/task-configuration.md deleted file mode 100644 index 66921e0..0000000 --- a/doc/user-manual/task-configuration.md +++ /dev/null @@ -1,148 +0,0 @@ - - -[[Previous]](datastore-task-dir.md) -[[Up]](intermediate-topics.md) -[[Next]](rdbms.md) - -## The Task Configuration Parameter Sets - -If you look back at [what happened when we ran the pipeline](start-pipeline.md), you'll note that each of the nodes after data receipt -- permuter, flip, and averaging -- had 2 tasks; and in each case there was one task with a `UOW` of 1 and another with `UOW` 2. You may have wondered: how does this work? How did Ziggy decide how to divide up the work into tasks? - -Excellent question! We will now answer that question. We'll also show you some other cool features that are all controlled by the parameter sets that manage task configuration. - -### Task Configuration in XML Files - -If you go back to pl-sample.xml, you'll see a parameter set that looks like this: - -```xml - - - - - - - - -``` - -Then, looking at pd-sample.xml, if you swim down to the spot where the permuter node of the sample pipeline is defined, you where this parameter set is referenced: - -```xml - - - - - - - -``` - -And finally, if you recall from [the datastore layout](datastore-task-dir.md), there are directories under the datastore root directory named `set-1` and `set-2`. - -Put it all together, and you can sorta-kinda see what happened. I'll spell it out for you here: - -- The permuter node uses the `Multiple subtask configuration` to, uh, configure its tasks. -- The `Multiple subtask configuration` parameter set has a regex of `set-([0-9]{1})` that is used to define task boundaries. -- Every directory that matches the regex is turned into a pipeline task. There are 2 directories in the datastore that match the regex: `set-1` and `set-2`. Thus, there's 1 task for `set-1` and another for `set-2`. -- The permuter uses `raw data` as its input data file type. Ziggy goes into the set-1 directory and looks for all the files that match the datastore file name specification. These are `set-1/L0/nasa-logo-file-0.png`, et. al. Ziggy moves copies of these files into the task directory for the `set-1` task. The `set-2` task files are procured the same way. -- The task directory regex has a regex group, `([0-9]{1})`. This is used as a label for the UOW column of the tasks table on the instances panel. Thus when the tasks show up on the instances panel, one is called `1` and the other `2`. - -Every node needs an instance of `TaskConfigurationParameters`, so you may well wind up with more than one in your `pl-*.xml` files. The sample pipeline has a total of 3. - -#### How to Run a Subset of Tasks - -What happens if I want to run the `set-1` tasks but not the `set-2` tasks? All I have to do is change the regex so that only `set-1` matches. In other words, I want `taskDirectoryRegex="set-1"`. - -This remains the case for more complicated arrangements. Consider for example what would happen if we had `set-1`, `set-2`, ... `set-9`. Now imagine that we want to run `set-1`, `set-3`, and `set-6`. In that case the regex would be `taskDirectoryRegex="set-([136]{1})"`. - -#### Datastore Directory Organization - -For the sample pipeline, we put the dataset directory at the top level, the `L0`, `L1`, etc., directories at the next level down. What if I wanted to do it the other way around? That is, what if I wanted `L0`, `L1`, etc., at the top and `set-1` and `set-2` under those? We can do it that way, sure! - -First, the datastore would need to change, which means that the `fileNameWithSubstitutionsForDatastore` entries in the data file type definitions need to change. For example: for raw data, instead of `"$2/L0/$1-$3"`, it would need to be `"L0/$2/$1-$3"`; and so on for the other data file type definitions. - -Second the `taskDirectoryRegex` values would have to change. For the permuter, the regex would need to be `"L0/set-([0-9]{1})"`. This is because the task directory regex needs to match the layout of the datastore and the naming conventions of the data file types. - -There are two major downsides to organizing the datastore by processing step / data set rather than data set / processing step: - -First, if you look at the pipeline definition, you can see that I use the same task configuration parameter set for both permuter and flip nodes. I can do that because in both cases, the regex for task generation is the same: `"set-([0-9]{1})"`. If I exchanged the order of directories, then permuter would need `"L0/set-([0-9]{1})"`, while the one for flip would be `"L1/set-([0-9]{1})"`. - -Second, consider the situation in which I want to process `set-1` but not `set-2` through both permuter and flip. With the current setup, that's easy to do because they use the same task configuration parameters: I change the one regex, and both pipeline modules are affected. If I had a separate task configuration parameter set for each pipeline node, then I'd need to adjust the regex in each set to get the same effect. It's less convenient and introduces a greater chance of pilot error. - -The moral of this story is that it's worthwhile to think a bit about the datastore organization, in particular thinking about what operations should be easy to group together. - -### All Those Other Parameters - -So far we've discussed only 1 parameter in the task configuration parameter set. What about all the others? Let's walk through them. - -#### singleSubtask - -For permuter and flip, we wanted 1 subtask for each image, so 4 subtasks in each task. The averaging algorithm is different: in that case, we're averaging together a bunch of images. That means that we want all the data files to get processed together, not in a subtask each (I mean, you can average together 1 file, but it's kind of dull). - -What this tells us is that Ziggy needs to support algorithms that get executed once per task and process all their data in that single execution (like the averaging algorithm), as well as algorithms that get executed multiple times per task and process one input file in each execution (like permuter and flip). The choice between these two modes of execution is controlled by the `singleSubtask` parameter. Setting this to true tells Ziggy that all the data files for the task are processed in 1 subtask, not multiple. Note that the averaging node has a different task configuration parameter set, and that set has `singleSubtask` set to true. - -The default for `singleSubtask` is `false`. - -#### maxFailedSubtaskCount - -Under normal circumstances, as we'll see shortly, if even a single subtask fails then the entire task is considered as failed and the operator has to intervene. However, it may be the case that the mission doesn't want to stop and deal with the issue if only a few subtasks fail. Imagine your task has 5,000 images in it, and the requirements state that the mission shall process at least 90% of all images successfully. Well, you can set `maxFailedSubtaskCount` in that case to 500. If the number of failed subtasks is less than 500, then Ziggy will declare the task successful and move on to the next processing activity; if it's 501 or more, Ziggy will declare the task failed and an operator will need to address the issue. - -The default for `maxFailedSubtaskCount` is zero. - -#### reprocess - -Most missions don't get all their data all at once. Most missions get data at some regular interval: every day, every month, whatever. - -In that case, a lot of the time you'll only want to process the new data. If you've got 5 years of data in the datastore, and you get a monthly delivery, you don't want to process the 5 years of data you've already processed; you want to process only the data you just got. However: it may be that every once in a while you'll want to go back and process every byte since the beginning of the mission. Maybe you do this because your algorithms have improved. Maybe it's time series data, and you want to generate time series across the whole 5 years. - -The `reprocess` parameter controls this behavior. When `reprocess` is true, Ziggy will try to process all the mission data, subject to the `taskDirectoryRegex` for each pipeline node. When it's set to false, Ziggy will only process new data. - -The default for `reprocess` is `false`. - -##### How does Ziggy Know Whether Data is "New" or "Old"? - -Excellent question! Here's a long answer: - -Ziggy keeps track of the "producer-consumer" information for every file in the datastore. That is, it knows which pipeline task produced each file, and it knows which pipeline tasks used each file as input ("consumed" it). Ziggy also knows what algorithm was employed by every task it's ever run. - -Thus, when Ziggy marshals inputs for a new task, and that task's `TaskConfigurationParameters` instance has `reprocess` set to `false`, the system filters out any inputs that have been, at some point in the past, processed using the same algorithm as the task that's being marshaled. Result: a task directory that has only data files that haven't been processed before. - -One subtlety: Ziggy not only tracks which pipeline tasks consumed a given file as input; it also looks to see whether they did so successfully. If a task consumes a file as input, but then the algorithm process that was running on that data bombs, Ziggy doesn't record that task as a consumer of that file. That way, the next time Ziggy processes "new" data, it will also include any "old" data that was only used in failed processing attempts. - -The default for `reprocess` is `false`. - -#### reprocessingTasksExclude - -Okay, this one's a bit complicated. - -Imagine that you ran the pipeline and everything seemed fine, everything ran to completion. But then a subject matter expert looks at the results and sees that some of them are good, some are garbage. After some effort, the algorithm code is fixed to address the problems with the ones that are garbage, and you're ready to reprocess the tasks that produced garbage outputs. - -How do you do that? - -If you select `reprocess="false"`, nothing will get processed. As far as Ziggy is concerned, all the data was used in tasks that ran to completion, so there's nothing that needs to be processed. - -If you select `reprocess="true"`, both the inputs that produced garbage and the ones that didn't will get reprocessed. If the fraction of inputs that produced garbage is small, you may not want to do this. - -What you do is set `reprocess="true"`, but then you tell Ziggy to exclude the inputs that were used in tasks that produced good output. So imagine that tasks 101 to 110 were run, and only 105 is bad. To re-run the data from 105 in a new pipeline instance, you'd set `reprocessingTasksExclude="101, 102, 103, 104, 106, 107, 108, 109, 110"`. - -In my experience, we haven't had to do this very often, but we have had to do it. So now you can, too! - -The default for `reprocessingTasksExclude` is empty (i.e., don't exclude anything). - -#### maxAutoResubmits - -It will sometimes happen that a Ziggy task will finish in such a way that you actually want to resubmit the task to the processing system without skipping a beat. There are a couple of circumstances where this may happen: - -- If you're uncertain of the settings you've put in for the amount of wall time your task should request, then it's possible that the task will reach its wall time limit before all the subtasks are processed. In this case, you would simply want to resubmit the task so that the remaining subtasks (or at least some of them) complete. -- If you're using an algorithm that has non-deterministic errors, the result will be that subtasks will "fail," but if the subtasks are re-run they will (probably) complete successfully. In this case, you would want to resubmit to get the handful of "failed" subtasks completed. - -Because of issues like these, Ziggy has the capacity to automatically resubmit a task if it has incomplete or failed subtasks. The user can limit the number of automatic resubmits that the task gets; this prevents you from getting into an infinite loop of resubmits if something more serious is going wrong. the `maxAutoResubmits` parameter tells Ziggy how many times it may resubmit a task without your involvement before it gives up and waits for you to come along and see what's broken. - -Note that this is a potentially dangerous option to use! The problem is that Ziggy can't tell the difference between a failure that a typical user would want to automatically resubmit, and a failure that is a real problem that needs human intervention (for example, a bug in the algorithm software). If the situation is in the second category, you can wind up with Ziggy resubmitting a task that's simply going to fail in exactly the same way as it did the first time it was submitted. - -The default for `maxAutoResubmits` is zero. - -[[Previous]](datastore-task-dir.md) -[[Up]](intermediate-topics.md) -[[Next]](rdbms.md) diff --git a/doc/user-manual/troubleshooting.md b/doc/user-manual/troubleshooting.md index 582e34d..71836af 100644 --- a/doc/user-manual/troubleshooting.md +++ b/doc/user-manual/troubleshooting.md @@ -14,26 +14,22 @@ On the console, go back to the `Parameter Library` panel and double-click `Algor What this does is to tell the permuter's Python-side "glue" code that it should deliberately throw an exception during the processing of subtask zero. Obviously you shouldn't put such a parameter into your own pipeline! But in this case it's useful because it allows us to generate an exception in a controlled manner and watch what happens. -Now go to the `Pipelines` panel and set up to run `permuter` and `flip` nodes in the sample pipeline. Start the pipeline and return to the `Instances` panel. After a few seconds you'll see something like this: +Now go to the `Pipelines` panel and set up to run `permuter` and `flip` nodes in the sample pipeline. Start the pipeline and return to the `Instances` panel. After a few seconds you'll see that the `permuter` module has started, but then stops with an error: -What you see is that at the moment when I took my screen shot, 2 subtasks had completed and 1 had failed in each task, which is why each has a `P-state` that includes `(4 / 2 / 1)` (i.e., total subtasks, completed subtasks, failed subtasks). After a little more time has passed, you'll see this: - - - What indications do we have that all is not well? Let me recount the signs and portents: - The instance state is `ERRORS_STALLED`, which means that Ziggy can't move on to the next pipeline node due to errors. - The task state is `ERROR`. -- The task P-states are `Ac` (algorithm complete), and subtask counts are 4 / 3 / 1, so 1 failed subtask per task, as expected. +- The task P-states are `Ac` (algorithm complete), and subtask counts are 4 / 3 / 1 (total subtasks, completed subtasks, failed subtasks), so 1 failed subtask per task, as expected. - Both the `Pi` stoplight (pipelines) and the `A` stoplight (alerts) are red. ### [Log Files](log-files.md) The first stop for understanding what went wrong is the log files. Ziggy produces a lot of these! -### [Using the Ziggy GUI for Troubleshooting](ziggy-gui-troubleshootihng.md) +### [Using the Ziggy GUI for Troubleshooting](ziggy-gui-troubleshooting.md) In most cases, you wan't want or need to resort to manually pawing through log files. In most cases, you can use Ziggy for troubleshooting and to respond to problems. diff --git a/doc/user-manual/user-manual.md b/doc/user-manual/user-manual.md index 9b29c49..e7ab7c2 100644 --- a/doc/user-manual/user-manual.md +++ b/doc/user-manual/user-manual.md @@ -24,7 +24,7 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the 5.1. [Module Parameters](module-parameters.md) - 5.2​. [Data File Types](data-file-types.md) + 5.2​. [The Datastore](datastore.md) 5.3​. [Pipeline Definition](pipeline-definition.md) @@ -44,13 +44,15 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the 8.5.​ [Organizing Pipelines and Parameter Sets](organizing-tables.md) + 8.6. [The Edit Pipeline Dialog Box](edit-pipeline.md) + + 8.7. [The Datastore Control Panel](datastore-regexp.md) + 9. [Intermediate Topics](intermediate-topics.md) 9.1. [The Datastore and the Task Directory](datastore-task-dir.md) - 9.2​. [The Task Configuration Parameter Sets](task-configuration.md) - - 9.3. [Setting up a Relational Database Management System (RDBMS)](rdbms.md) + 9.2. [Setting up a Relational Database Management System (RDBMS)](rdbms.md) 10. [Troubleshooting Pipeline Execution](troubleshooting.md) @@ -70,11 +72,9 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the 12. [High Performance Computing](select-hpc.md) - 12.1​. [Remote Parameters](remote-parameters.md) + 12.1. [The Remote Execution Dialog Box](remote-dialog.md) - 12.2​. [The Remote Execution Dialog Box](remote-dialog.md) - - 12.23. [HPC Cost Estimation](hpc-cost.md) + 12.2. [HPC Cost Estimation](hpc-cost.md) 13. Cloud Computing (TBD) @@ -82,7 +82,7 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the 14.1​. [Execution Flow](data-receipt-execution.md) - 14.2.​ [Console Display](data-receipt-display.md) + 14.2.​ [Data Receipt Display](data-receipt-display.md) 15. [Event Handler](event-handler.md) @@ -108,17 +108,17 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the --> -17. Alternative User Interface Options (TBD) +17. Alternative User Interface Options - @@ -134,11 +134,9 @@ Ziggy is "A Pipeline management system for Data Analysis Pipelines." This is the 19.4​. [Redefining a Pipeline](redefine-pipeline.md) - 19.5​. [The Edit Pipeline Dialog Box](edit-pipeline.md) - - 19.6. [Creating Ziggy Nicknames](nicknames.md) + 19.5. [Creating Ziggy Nicknames](nicknames.md) - + 20. [Contact Us](contact-us.md) diff --git a/doc/user-manual/ziggy-gui.md b/doc/user-manual/ziggy-gui.md index b933176..00361bd 100644 --- a/doc/user-manual/ziggy-gui.md +++ b/doc/user-manual/ziggy-gui.md @@ -26,6 +26,18 @@ There's no law that says you have to run the pipeline straight through from its How the user adjusts the user-adjustable parameters. +### [Organizing Pipelines and Parameter Sets](organizing-tables.md) + +Bringing order to the chaos of some of the Ziggy display and edit tables. + +### [The Edit Pipeline Dialog Box](edit-pipeline.md) + +How the user adjusts most parameters related to things like provisioning execution resources. + +### [The Datastore Control Panel](datastore-regexp.md) + +Adjust the regular expressions in your datastore definition. + [[Previous]](running-pipeline.md) [[Up]](user-manual.md) [[Next]](start-pipeline.md) diff --git a/etc/hsqldb.wrapper.conf b/etc/hsqldb.wrapper.conf index 4db93b7..3785470 100644 --- a/etc/hsqldb.wrapper.conf +++ b/etc/hsqldb.wrapper.conf @@ -14,11 +14,9 @@ wrapper.console.title = Ziggy HSQLDB Server wrapper.java.initmemory = 512 # Additional Java parameters. -# ClusterController.workerCommand() adds additional parameters starting at 5. -wrapper.java.additional.1 = -Dlog4j2.configurationFile=etc/log4j2.xml -wrapper.java.additional.2 = -Dlog4j.logfile.prefix=logs/hsqldb -wrapper.java.additional.3 = -XX:+UseCompressedOops -wrapper.java.additional.4 = -XX:-OmitStackTraceInFastThrow +# ClusterController.workerCommand() adds additional parameters starting at 3. +wrapper.java.additional.1 = -XX:+UseCompressedOops +wrapper.java.additional.2 = -XX:-OmitStackTraceInFastThrow # Disable timeouts because the workers used to get killed when # MATLAB processes consumed 100% CPU for extended periods of time. diff --git a/etc/log4j2.xml b/etc/log4j2.xml index 9527edf..845cc74 100644 --- a/etc/log4j2.xml +++ b/etc/log4j2.xml @@ -3,9 +3,9 @@ %d %-5p [%t:%C{1}.%M] %X{logStreamIdentifier} %m%n - ${sys:ziggy.logfile:-/dev/null} + ${sys:ziggy.logFile:-/dev/null} @@ -14,6 +14,7 @@ + @@ -41,6 +42,8 @@ + + diff --git a/etc/supervisor.wrapper.conf b/etc/supervisor.wrapper.conf index e80618a..419baab 100644 --- a/etc/supervisor.wrapper.conf +++ b/etc/supervisor.wrapper.conf @@ -15,11 +15,9 @@ wrapper.console.title=Ziggy Supervisor wrapper.java.initmemory=512 # Additional Java parameters. -# ClusterController.supervisorCommand() adds additional parameters starting at 5. -wrapper.java.additional.1=-Dlog4j2.configurationFile=etc/log4j2.xml -wrapper.java.additional.2=-Dlog4j.logfile.prefix=logs/supervisor -wrapper.java.additional.3=-XX:+UseCompressedOops -wrapper.java.additional.4=-XX:-OmitStackTraceInFastThrow +# ClusterController.supervisorCommand() adds additional parameters starting at 3. +wrapper.java.additional.1=-XX:+UseCompressedOops +wrapper.java.additional.2=-XX:-OmitStackTraceInFastThrow # Disable timeouts because the workers used to get killed when # MATLAB processes consumed 100% CPU for extended periods of time. diff --git a/etc/ziggy.properties b/etc/ziggy.properties index 8c5882d..2d9c5bb 100644 --- a/etc/ziggy.properties +++ b/etc/ziggy.properties @@ -7,10 +7,10 @@ hibernate.format_sql = true hibernate.jdbc.batch_size = 30 hibernate.show_sql = false hibernate.use_sql_comments = true -matlab.log4j.config = ${ziggy.home.dir}/etc/log4j-matlab-interactive.xml matlab.log4j.initialize = true ziggy.database.protocol = ziggy.default.jvm.args = -Dlog4j2.configurationFile=${ziggy.home.dir}/etc/log4j2.xml -Djava.library.path=${ziggy.home.dir}/lib +ziggy.environment = ZIGGY_HOME=${env:ZIGGY_HOME},PIPELINE_CONFIG_PATH=${env:PIPELINE_CONFIG_PATH},JAVA_HOME=${env:JAVA_HOME} ziggy.nickname.cluster = gov.nasa.ziggy.ui.ClusterController|cluster|| ziggy.nickname.compute-node-master = gov.nasa.ziggy.module.ComputeNodeMaster||-XX:ParallelGCThreads=2 -XX:+UseParallelGC -XX:OnOutOfMemoryError="kill -QUIT %p"| ziggy.nickname.console = gov.nasa.ziggy.ui.ZiggyConsole|console|-Dsun.java2d.xrender=false -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true -Xmx2G| @@ -20,10 +20,10 @@ ziggy.nickname.export-parameters = gov.nasa.ziggy.parameters.ParameterLibraryImp ziggy.nickname.export-pipelines = gov.nasa.ziggy.pipeline.definition.PipelineDefinitionCli|||-export ziggy.nickname.generate-manifest = gov.nasa.ziggy.data.management.Manifest||| ziggy.nickname.hsqlgui = org.hsqldb.util.DatabaseManagerSwing||| +ziggy.nickname.import-datastore-config = gov.nasa.ziggy.data.management.DatastoreConfigurationImporter||| ziggy.nickname.import-events = gov.nasa.ziggy.services.events.ZiggyEventHandlerDefinitionImporter||| ziggy.nickname.import-parameters = gov.nasa.ziggy.parameters.ParameterLibraryImportExportCli|||-import ziggy.nickname.import-pipelines = gov.nasa.ziggy.pipeline.definition.PipelineDefinitionCli|||-import -ziggy.nickname.import-types = gov.nasa.ziggy.data.management.DataFileTypeImporter||| ziggy.nickname.metrics = gov.nasa.ziggy.metrics.report.MetricsCli||| ziggy.nickname.perf-report = gov.nasa.ziggy.metrics.report.PerformanceReport||| ziggy.test.file.property = from.default.location diff --git a/gradle.properties b/gradle.properties index 5a254bf..5533425 100644 --- a/gradle.properties +++ b/gradle.properties @@ -7,7 +7,7 @@ org.gradle.parallel = true // The version is updated when the first release candidate is created // while following Release Branches in Appendix C of the SMP, Git // Workflow. This property is used when publishing Ziggy. -version = 0.4.1 +version = 0.5.0 // The Maven group for the published Ziggy libraries. group = gov.nasa diff --git a/licenses/licenses.md b/licenses/licenses.md index a8b2eb8..c4591fb 100644 --- a/licenses/licenses.md +++ b/licenses/licenses.md @@ -7,7 +7,7 @@ This file lists the third-party software that is used by Ziggy. Links to the sof ## Tools |Name|Requirement|Alternative| -|---|---|---|---|---| +|---|---|---| |[Git](https://git-scm.com/) [(docs)](https://git-scm.com/doc) [(license)](git-2.x)|SWE-42|Subversion| |[Gradle](https://gradle.org/) [(docs)](https://docs.gradle.org/current/userguide/userguide.html) [(license)](gradle-4.1.x)||ant| diff --git a/sample-pipeline/build-env.sh b/sample-pipeline/build-env.sh index 05dbf06..6a71ca2 100755 --- a/sample-pipeline/build-env.sh +++ b/sample-pipeline/build-env.sh @@ -21,7 +21,7 @@ mkdir -p $python_env # Create and populate the data receipt directory from the sample data data_receipt_dir=$sample_home/pipeline-results/data-receipt mkdir -p $data_receipt_dir -cp $sample_root/data/* $data_receipt_dir +cp -r $sample_root/data/* $data_receipt_dir # Build the bin directory in build. bin_dir=$sample_home/bin diff --git a/sample-pipeline/clean-env.sh b/sample-pipeline/clean-env.sh new file mode 100755 index 0000000..87fb4a4 --- /dev/null +++ b/sample-pipeline/clean-env.sh @@ -0,0 +1,17 @@ +#! /bin/bash +# +# Shell script that tears down the Python environment for the sample pipeline. +# +# The environment variable PIPELINE_CONFIG_PATH contains the path to +# the pipeline configuration file. This script uses that path to +# derive all the paths it needs. +# +# Author: PT +# Author: Bill Wohler + +etc_dir="$(dirname "$PIPELINE_CONFIG_PATH")" +sample_root="$(dirname "$etc_dir")" +sample_home=$sample_root/build + +chmod -R u+w $sample_home +rm -rf $sample_home diff --git a/sample-pipeline/config/pd-sample.xml b/sample-pipeline/config/pd-sample.xml index 8243d9d..dcb2692 100644 --- a/sample-pipeline/config/pd-sample.xml +++ b/sample-pipeline/config/pd-sample.xml @@ -8,9 +8,10 @@ and outputs, information about models, and parameter sets. Enjoy! --> - + @@ -21,27 +22,24 @@ and outputs, information about models, and parameter sets. Enjoy! --> + pipeline. --> + be a user-defined module. Ziggy provides data receipt "for free" as a + tool to get files into the datastore. The user does have to define the + data types that will be imported. The model types can be defined if + desired; if not, the assumption will be that all model types can be + imported. There's also a task configuration parameter set so that the + user can define which data receipt tasks are to be performed. --> - - - @@ -49,17 +47,15 @@ and outputs, information about models, and parameter sets. Enjoy! --> - - + is no child node listed because it's the last step in the pipeline. + Also, it uses the single-subtask configuration. --> + - diff --git a/sample-pipeline/config/pe-sample.xml b/sample-pipeline/config/pe-sample.xml index 0900b0e..d1b87fd 100644 --- a/sample-pipeline/config/pe-sample.xml +++ b/sample-pipeline/config/pe-sample.xml @@ -3,7 +3,7 @@ + enableOnClusterStart="false" + directory="${ziggy.pipeline.data.receipt.dir}"/> diff --git a/sample-pipeline/config/pl-sample.xml b/sample-pipeline/config/pl-sample.xml index a6009ff..5e1eb7f 100644 --- a/sample-pipeline/config/pl-sample.xml +++ b/sample-pipeline/config/pl-sample.xml @@ -6,66 +6,6 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + - + - + - + diff --git a/sample-pipeline/data/sample-model.txt b/sample-pipeline/data/models/sample-model.txt similarity index 100% rename from sample-pipeline/data/sample-model.txt rename to sample-pipeline/data/models/sample-model.txt diff --git a/sample-pipeline/data/sample-pipeline-manifest.xml b/sample-pipeline/data/sample-pipeline-manifest.xml index 97e2456..64b9593 100644 --- a/sample-pipeline/data/sample-pipeline-manifest.xml +++ b/sample-pipeline/data/sample-pipeline-manifest.xml @@ -1,12 +1,12 @@ - - - - - - - - - + + + + + + + + + diff --git a/sample-pipeline/data/nasa_logo-set-1-file-0.png b/sample-pipeline/data/set-1/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-1-file-0.png rename to sample-pipeline/data/set-1/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/data/nasa_logo-set-1-file-1.png b/sample-pipeline/data/set-1/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-1-file-1.png rename to sample-pipeline/data/set-1/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/data/nasa_logo-set-1-file-2.png b/sample-pipeline/data/set-1/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-1-file-2.png rename to sample-pipeline/data/set-1/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/data/nasa_logo-set-1-file-3.png b/sample-pipeline/data/set-1/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-1-file-3.png rename to sample-pipeline/data/set-1/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/data/nasa_logo-set-2-file-0.png b/sample-pipeline/data/set-2/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-2-file-0.png rename to sample-pipeline/data/set-2/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/data/nasa_logo-set-2-file-1.png b/sample-pipeline/data/set-2/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-2-file-1.png rename to sample-pipeline/data/set-2/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/data/nasa_logo-set-2-file-2.png b/sample-pipeline/data/set-2/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-2-file-2.png rename to sample-pipeline/data/set-2/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/data/nasa_logo-set-2-file-3.png b/sample-pipeline/data/set-2/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/data/nasa_logo-set-2-file-3.png rename to sample-pipeline/data/set-2/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/etc/sample.properties b/sample-pipeline/etc/sample.properties index 63939be..0cba81b 100644 --- a/sample-pipeline/etc/sample.properties +++ b/sample-pipeline/etc/sample.properties @@ -17,7 +17,7 @@ ziggy.pipeline.binPath = ${env:JAVA_HOME}/bin:/bin:/usr/bin ziggy.pipeline.data.receipt.dir = ${ziggy.pipeline.results.dir}/data-receipt ziggy.pipeline.datastore.dir = ${ziggy.pipeline.results.dir}/datastore ziggy.pipeline.definition.dir = ${pipeline.root.dir}/config -ziggy.pipeline.environment = ZIGGY_HOME=${env:ZIGGY_HOME} +ziggy.pipeline.environment = ZIGGY_ROOT=${env:ZIGGY_ROOT} ziggy.pipeline.home.dir = ${pipeline.root.dir}/build ziggy.pipeline.memdrone.enabled = false ziggy.pipeline.memdrone.sleepSeconds = 10 @@ -28,6 +28,6 @@ ziggy.remote.group = phony ziggy.remote.host = phony ziggy.remote.user = phony ziggy.root = ${env:ZIGGY_ROOT} -ziggy.worker.count = 2 +ziggy.worker.count = 6 ziggy.worker.heapSize = 12000 ziggy.worker.port = 1099 diff --git a/sample-pipeline/multi-data/sample-1/sample-1-pipeline-manifest.xml b/sample-pipeline/multi-data/sample-1/sample-1-pipeline-manifest.xml index 2fe4763..1ed7c66 100644 --- a/sample-pipeline/multi-data/sample-1/sample-1-pipeline-manifest.xml +++ b/sample-pipeline/multi-data/sample-1/sample-1-pipeline-manifest.xml @@ -1,11 +1,11 @@ - - - - - - - - + + + + + + + + diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-0.png b/sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-0.png rename to sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-1.png b/sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-1.png rename to sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-2.png b/sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-2.png rename to sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-3.png b/sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-3-file-3.png rename to sample-pipeline/multi-data/sample-1/set-3/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-0.png b/sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-0.png rename to sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-1.png b/sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-1.png rename to sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-2.png b/sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-2.png rename to sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-3.png b/sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/multi-data/sample-1/nasa_logo-set-4-file-3.png rename to sample-pipeline/multi-data/sample-1/set-4/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/multi-data/sample-2/sample-2-pipeline-manifest.xml b/sample-pipeline/multi-data/sample-2/sample-2-pipeline-manifest.xml index 0991f4f..35c5789 100644 --- a/sample-pipeline/multi-data/sample-2/sample-2-pipeline-manifest.xml +++ b/sample-pipeline/multi-data/sample-2/sample-2-pipeline-manifest.xml @@ -1,11 +1,11 @@ - - - - - - - - + + + + + + + + diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-0.png b/sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-0.png rename to sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-1.png b/sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-1.png rename to sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-2.png b/sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-2.png rename to sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-3.png b/sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-5-file-3.png rename to sample-pipeline/multi-data/sample-2/set-5/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-0.png b/sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-0.png rename to sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-1.png b/sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-1.png rename to sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-2.png b/sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-2.png rename to sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-3.png b/sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/multi-data/sample-2/nasa_logo-set-6-file-3.png rename to sample-pipeline/multi-data/sample-2/set-6/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/multi-data/sample-3/sample-3-pipeline-manifest.xml b/sample-pipeline/multi-data/sample-3/sample-3-pipeline-manifest.xml index 093578e..f070ad4 100644 --- a/sample-pipeline/multi-data/sample-3/sample-3-pipeline-manifest.xml +++ b/sample-pipeline/multi-data/sample-3/sample-3-pipeline-manifest.xml @@ -1,11 +1,11 @@ - - - - - - - - + + + + + + + + diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-0.png b/sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-0.png rename to sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-1.png b/sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-1.png rename to sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-2.png b/sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-2.png rename to sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-3.png b/sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-3.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-7-file-3.png rename to sample-pipeline/multi-data/sample-3/set-7/L0/nasa-logo-file-3.png diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-8-file-0.png b/sample-pipeline/multi-data/sample-3/set-8/L0/nasa-logo-file-0.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-8-file-0.png rename to sample-pipeline/multi-data/sample-3/set-8/L0/nasa-logo-file-0.png diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-8-file-1.png b/sample-pipeline/multi-data/sample-3/set-8/L0/nasa-logo-file-1.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-8-file-1.png rename to sample-pipeline/multi-data/sample-3/set-8/L0/nasa-logo-file-1.png diff --git a/sample-pipeline/multi-data/sample-3/nasa_logo-set-8-file-2.png b/sample-pipeline/multi-data/sample-3/set-8/L0/nasa-logo-file-2.png similarity index 100% rename from sample-pipeline/multi-data/sample-3/nasa_logo-set-8-file-2.png rename to sample-pipeline/multi-data/sample-3/set-8/L0/nasa-logo-file-2.png diff --git a/sample-pipeline/src/main/python/major_tom/major_tom.py b/sample-pipeline/src/main/python/major_tom/major_tom.py index 7c61220..60b2240 100644 --- a/sample-pipeline/src/main/python/major_tom/major_tom.py +++ b/sample-pipeline/src/main/python/major_tom/major_tom.py @@ -47,8 +47,8 @@ def permute_color(filename, throw_exception, generate_output): png_array[:,:,indices[1]] = green_image png_array[:,:,indices[2]] = blue_image - bare_filename = os.path.splitext(filename)[0]; - save_filename = bare_filename + "-perm.png" + bare_filename = filename.split(os.extsep)[0] + save_filename = bare_filename + ".perm.png" print("Saving color-permuted image to file {} in directory {}".format(save_filename, os.getcwd())) new_png_file = Image.fromarray(png_array, 'RGBA') @@ -73,8 +73,8 @@ def left_right_flip(filename): png_array[:,:,2] = np.fliplr(blue_image) png_array[:,:,3] = np.fliplr(alpha_image) - bare_filename = os.path.splitext(filename)[0]; - save_filename = bare_filename + "-lrflip.png" + bare_filename = filename.split(os.extsep)[0] + save_filename = bare_filename + ".fliplr.png" print("Saving LR-flipped image to file {} in directory {}".format(save_filename, os.getcwd())) new_png_file = Image.fromarray(png_array) @@ -99,8 +99,8 @@ def up_down_flip(filename): png_array[:,:,2] = np.flipud(blue_image) png_array[:,:,3] = np.flipud(alpha_image) - bare_filename = os.path.splitext(filename)[0]; - save_filename = bare_filename + "-udflip.png" + bare_filename = filename.split(os.extsep)[0] + save_filename = bare_filename + ".flipud.png" print("Saving UD-flipped image to file {} in directory {}".format(save_filename, os.getcwd())) new_png_file = Image.fromarray(png_array) @@ -115,7 +115,7 @@ def average_images(filenames): i_image = 0 # Extract the dataset string. - pattern="(\\S+?)-(set-[0-9])-(file-[0-9])-perm-(\\S+?).png" + pattern="nasa-logo-(file-[0-9])\\.(flipud|fliplr)\\.png" match = re.search(pattern, filenames[0]) setString = match.group(2); for filename in filenames: @@ -132,7 +132,7 @@ def average_images(filenames): i_image += 1 mean_array = sum_array // n_images - save_filename = 'averaged-image-' + setString + '.png' + save_filename = 'nasa-logo-averaged.png' print("Saving averaged image to file {} in directory {}".format(save_filename, os.getcwd())) diff --git a/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java b/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java index 9ad49cc..8e0dc4a 100644 --- a/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java +++ b/src/main/java/gov/nasa/ziggy/crud/ZiggyQuery.java @@ -11,6 +11,8 @@ import com.google.common.collect.Lists; +import gov.nasa.ziggy.module.PipelineException; +import jakarta.persistence.criteria.AbstractQuery; import jakarta.persistence.criteria.CriteriaBuilder; import jakarta.persistence.criteria.CriteriaQuery; import jakarta.persistence.criteria.Expression; @@ -18,6 +20,7 @@ import jakarta.persistence.criteria.Predicate; import jakarta.persistence.criteria.Root; import jakarta.persistence.criteria.Selection; +import jakarta.persistence.criteria.Subquery; import jakarta.persistence.metamodel.SetAttribute; import jakarta.persistence.metamodel.SingularAttribute; @@ -27,14 +30,14 @@ * query artifacts in a single class. *

* The JPA Criteria API is extremely verbose and typically requires 3 separate objects to construct - * and perform queries: the {@link CriteriaQuery}, which is defined in terms of the class that the - * query returns; the {@link Root}, which is defined in terms of the database table that is the - * target of the query; and the {@link CriteriaBuilder}, which provides the options that configure - * the query actions (sorting, filtering, projecting, etc.). Most of Ziggy's query requirements can - * be satisfied using a small fraction of the JPA system's capabilities. Thus the JPA API is - * abstracted into the {@link ZiggyQuery}, which automatically constructs those queries and the - * necessary objects and hides all the underlying verbosity and complexity from the user, who - * generally could care less. + * and perform queries: the {@link CriteriaQuery} or {@link Subquery}, which is defined in terms of + * the class that the query returns; the {@link Root}, which is defined in terms of the database + * table that is the target of the query; and the {@link CriteriaBuilder}, which provides the + * options that configure the query actions (sorting, filtering, projecting, etc.). Most of Ziggy's + * query requirements can be satisfied using a small fraction of the JPA system's capabilities. Thus + * the JPA API is abstracted into the {@link ZiggyQuery}, which automatically constructs those + * queries and the necessary objects and hides all the underlying verbosity and complexity from the + * user, who generally could care less. *

* The JPA API requires that queries include a component with a type parameter that corresponds to * the class of object returned by the query and a component that is parameterized based on the @@ -53,13 +56,12 @@ * @author PT * @author Bill Wohler */ + public class ZiggyQuery { - protected HibernateCriteriaBuilder builder; - protected CriteriaQuery criteriaQuery; - protected Root root; - protected Class databaseClass; - protected Class returnClass; + private HibernateCriteriaBuilder builder; + + private Root root; private SingularAttribute attribute; private SetAttribute set; @@ -68,15 +70,27 @@ public class ZiggyQuery { private List predicates = new ArrayList<>(); private List> selections = new ArrayList<>(); + private List> subqueries = new ArrayList<>(); + + // AbstractQuery allows this to be either CriteriaQuery or Subquery, as needed. + private AbstractQuery jpaQuery; + /** For testing only. */ private List> queryChunks = new ArrayList<>(); - public ZiggyQuery(Class databaseClass, Class returnClass, AbstractCrud crud) { + /** Constructor for {@link CriteriaQuery} instances. */ + ZiggyQuery(Class databaseClass, Class returnClass, AbstractCrud crud) { builder = crud.createCriteriaBuilder(); - criteriaQuery = builder.createQuery(returnClass); - root = criteriaQuery.from(databaseClass); - this.databaseClass = databaseClass; - this.returnClass = returnClass; + jpaQuery = builder.createQuery(returnClass); + root = jpaQuery.from(databaseClass); + } + + /** Constructor for {@link Subquery} classes. */ + ZiggyQuery(Class databaseClass, Class returnClass, HibernateCriteriaBuilder builder, + AbstractQuery parentQuery) { + this.builder = builder; + jpaQuery = parentQuery.subquery(returnClass); + root = jpaQuery.from(databaseClass); } /** @@ -93,6 +107,20 @@ public ZiggyQuery column(String columnName) { return this; } + /** + * Use when a method can take either a scalar or collection attribute. + */ + private boolean hasAttribute() { + return hasScalarAttribute() || set != null; + } + + /** + * Use when a method can take either a scalar or collection attribute. + */ + private boolean hasScalarAttribute() { + return attribute != null || columnName != null; + } + /** * Defines a column for later use in a query operation. The column is a scalar. *

@@ -121,26 +149,19 @@ public ZiggyQuery column(SetAttribute set) { return this; } - /** - * Use when a method can take either a scalar or collection attribute. - */ - private boolean hasAttribute() { - return hasScalarAttribute() || set != null; - } - - /** - * Use when a method can take either a scalar or collection attribute. - */ - private boolean hasScalarAttribute() { - return attribute != null || columnName != null; - } - /** * Applies a query constraint that the value of a column must contain the specified value. */ @SuppressWarnings("unchecked") public ZiggyQuery in(Y value) { checkState(hasScalarAttribute(), "a column has not been defined"); + if (value instanceof ZiggyQuery) { + Subquery subquery = (Subquery) ((ZiggyQuery) value).jpaQuery; + predicates.add(attribute != null + ? builder.in((Path) root.get(attribute), Set.of(subquery)) + : builder.in(root.get(columnName), Set.of(subquery))); + return this; + } predicates.add( attribute != null ? builder.in((Path) root.get(attribute), Set.of(value)) : builder.in(root.get(columnName), Set.of(value))); @@ -160,14 +181,6 @@ public ZiggyQuery in(Collection values) { return this; } - public CriteriaBuilder.In in(Expression expression, Collection values) { - return builder.in(expression, values); - } - - public CriteriaBuilder.In in(Expression expression, Y value) { - return builder.in(expression, Set.of(value)); - } - /** * Performs the action of the {@link #in(Collection)} method, but performs the query in chunks. * This allows queries in which the collection of values is too large for a single query. The @@ -189,6 +202,14 @@ public ZiggyQuery chunkedIn(Collection values) { return this; } + public CriteriaBuilder.In in(Expression expression, Collection values) { + return builder.in(expression, values); + } + + public CriteriaBuilder.In in(Expression expression, Y value) { + return builder.in(expression, Set.of(value)); + } + /** * Applies a query constraint that the value of a column must be between a specified minimum * value and a specified maximum value, inclusive. @@ -202,28 +223,6 @@ public > ZiggyQuery between(Y minValue, Y return this; } - /** - * Applies a query constraint that the results of the query must be sorted in ascending order - * based on a column value. - */ - public ZiggyQuery ascendingOrder() { - checkState(hasScalarAttribute(), "a column has not been defined"); - criteriaQuery.orderBy(attribute != null ? builder.asc(root.get(attribute)) - : builder.asc(root.get(columnName))); - return this; - } - - /** - * Applies a query constraint that the results of the query must be sorted in descending order - * based on a column value. - */ - public ZiggyQuery descendingOrder() { - checkState(hasScalarAttribute(), "a column has not been defined"); - criteriaQuery.orderBy(attribute != null ? builder.desc(root.get(attribute)) - : builder.desc(root.get(columnName))); - return this; - } - /** * Selects a column of the query target class for projection. */ @@ -253,6 +252,9 @@ public ZiggyQuery select(SetAttribute set) { } public ZiggyQuery select(Selection selection) { + if (selection instanceof ZiggyQuery) { + selections.add((Selection) ((ZiggyQuery) selection).jpaQuery); + } selections.add(selection); return this; } @@ -279,6 +281,10 @@ public > ZiggyQuery max() { /** * Applies a query constraint that projects the minimum and maximum value of a column. + *

+ * In order to use the {@link #minMax()} constraint, the user must specify a {@link ZiggyQuery} + * that returns Object[]. The minimum value will be the first element of the array, the maximum + * value will be the second value. */ public ZiggyQuery minMax() { min(); @@ -328,11 +334,14 @@ public ZiggyQuery where(Predicate predicate) { * The {@link #constructWhereClause()} must be called before the query is executed. */ public ZiggyQuery constructWhereClause() { + for (ZiggyQuery subquery : subqueries) { + subquery.constructSelectClause().constructWhereClause(); + } if (predicates.isEmpty()) { return this; } Predicate[] predicatesArray = predicates.toArray(new Predicate[0]); - criteriaQuery = criteriaQuery.where(predicatesArray); + jpaQuery = jpaQuery.where(predicatesArray); return this; } @@ -348,37 +357,95 @@ public ZiggyQuery constructWhereClause() { * The {@link #constructSelectClause()} must be called before the query is executed. */ public ZiggyQuery constructSelectClause() { + for (ZiggyQuery subquery : subqueries) { + subquery.constructSelectClause().constructWhereClause(); + } if (selections.isEmpty()) { return this; } - if (selections.size() == 1) { - // A single select is a special case. In this case, the type of the Selection instance - // must match the return type for the query, which is R. We can achieve this via a cast, - // as long as we don't mind the resulting unchecked cast warning. + // Insane as this may sound, the one method that CriteriaQuery has, and Subquery has, but + // Abstract query DOES NOT have, is select. Also, the Subquery form of where requires an + // additional cast from Selection to Expression. + jpaQuery = querySelect(jpaQuery, selections); + return this; + } + + private AbstractQuery querySelect(AbstractQuery query, List> selections) { + if (query instanceof Subquery) { + return subquerySelect((Subquery) query, selections); + } + return criteriaQuerySelect((CriteriaQuery) query, selections); + } + + private Subquery subquerySelect(Subquery query, List> selections) { + if (selections.size() > 1) { + throw new PipelineException("Subquery does not support multiselect"); + } + @SuppressWarnings("unchecked") + Expression selection = (Expression) selections.get(0); + return query.select(selection); + } + + private CriteriaQuery criteriaQuerySelect(CriteriaQuery query, + List> selections) { + + // NB: multiselect produces an (undocumented) IllegalStateException + // if the size of its argument is 1, which is why there needs to be + // a block for the single Selection case that uses select and one for + // the multiple Selection case that uses multiselect. + if (selections.size() == 1) { Selection selection = selections.get(0); @SuppressWarnings("unchecked") Selection selectionR = (Selection) selection; - criteriaQuery = criteriaQuery.select(selectionR); - return this; + return query.select(selectionR); } - criteriaQuery = criteriaQuery.multiselect(selections); - return this; + return query.multiselect(selections); } /** * Applies a query constraint that specifies whether all values are returned, or whether the * returned values are filtered to eliminate duplicates. - *

- * In order to use the {@link #minMax()} constraint, the user must specify a {@link ZiggyQuery} - * that returns Object[]. The minimum value will be the first element of the array, the maximum - * value will be the second value. */ public ZiggyQuery distinct(boolean distinct) { - criteriaQuery = criteriaQuery.distinct(distinct); + jpaQuery = jpaQuery.distinct(distinct); return this; } + /** Instructs the query to return results in descending order. */ + public ZiggyQuery ascendingOrder() { + if (jpaQuery instanceof Subquery) { + return this; + } + checkState(hasScalarAttribute(), "a column has not been defined"); + ((CriteriaQuery) jpaQuery).orderBy(attribute != null ? builder.asc(root.get(attribute)) + : builder.asc(root.get(columnName))); + return this; + } + + /** Instructs the query to return results in descending order. */ + public ZiggyQuery descendingOrder() { + if (jpaQuery instanceof Subquery) { + return this; + } + checkState(hasScalarAttribute(), "a column has not been defined"); + ((CriteriaQuery) jpaQuery).orderBy(attribute != null ? builder.desc(root.get(attribute)) + : builder.desc(root.get(columnName))); + return this; + } + + /** Generates a subquery to the current query. */ + public ZiggyQuery ziggySubquery(Class subqueryClass) { + return ziggySubquery(subqueryClass, subqueryClass); + } + + /** Generates a subquery to the current query. */ + public ZiggyQuery ziggySubquery(Class databaseClass, Class returnClass) { + ZiggyQuery subquery = new ZiggyQuery<>(databaseClass, returnClass, builder, jpaQuery); + subqueries.add(subquery); + return subquery; + } + /** * Returns the {@link CriteriaBuilder} instance in the {@link ZiggyQuery}. This allows users to * build queries that aren't directly supported by {@link ZiggyQuery} and instead require more @@ -388,15 +455,6 @@ public HibernateCriteriaBuilder getBuilder() { return builder; } - /** - * Returns the {@link CriteriaQuery} instance in the {@link ZiggyQuery}. This allows users to - * build queries that aren't directly supported by {@link ZiggyQuery} and instead require more - * direct use of the JPA API. - */ - public CriteriaQuery getCriteriaQuery() { - return criteriaQuery; - } - /** * Returns the {@link Root} instance in the {@link ZiggyQuery}. This allows users to build * queries that aren't directly supported by {@link ZiggyQuery} and instead require more direct @@ -410,6 +468,14 @@ public Path get(SingularAttribute attribute) { return root.get(attribute); } + /** Returns the {@link AbstractQuery} cast to {@link CriteriaQuery}. */ + public CriteriaQuery getCriteriaQuery() { + if (jpaQuery instanceof Subquery) { + throw new PipelineException("Subquery cannot be cast to CriteriaQuery"); + } + return (CriteriaQuery) jpaQuery; + } + /** * Maximum expressions allowed in each chunk of {@link #chunkedIn(Collection)}. Broken out into * a package-private method so that tests can reduce this value to something small enough to diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DataFileType.java b/src/main/java/gov/nasa/ziggy/data/datastore/DataFileType.java new file mode 100644 index 0000000..bd52a98 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DataFileType.java @@ -0,0 +1,125 @@ +package gov.nasa.ziggy.data.datastore; + +import java.io.Serializable; +import java.util.Objects; + +import jakarta.persistence.Entity; +import jakarta.persistence.Id; +import jakarta.persistence.Table; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlAttribute; + +/** + * Defines a data file type for a pipeline. Data file types are used as input or output file types + * for each pipeline module. + *

+ * Data file types are defined by a location in the datastore, where the location is defined as: + * + * 

+ * locationElement1/locationElement2/locationElement3.../locationElementN,
+ *
+ * + * where each locationElement contains the name of a {@link DatastoreNode} instance. If the instance + * is a reference to a {@link DatastoreRegexp}, then the locationElement can also contain a string + * to be used with the regexp, separated from the node name by a $ (i.e., if the regexp is named + * cadenceType and has valid values of "(target|ffi)", then the locationElement would be either + * "cadenceType$target" or "cadenceType$ffi"). The full path defined by the locatiionElement + * elements must correspond to the full path of a datastore node in the database. + *

+ * Ziggy uses the location of a {@link DataFileType} to identify all the directories that + * potentially have data that can be used in processing in a particular module, or to find the + * destination of any output files from a given pipeline module. + *

+ * The {@link DataFileType} also requires a String that is a regular expression for the data file + * names that correspond to this data file type. + *

+ * Each {@link DataFileType} instance has a Boolean field, + * {@link DataFileType#includeAllFilesInAllSubtasks}. This indicates whether the data file type will + * provide 1 file of the given type to each subtask (default value, false) or whether the data file + * type will provide each subtask with all the files in that type (true). The field is a Boolean + * rather than boolean because it corresponds to an optional XML attribute, which means that it + * cannot be a primitive type. + * + * @author PT + */ + +@XmlAccessorType(XmlAccessType.NONE) +@Entity +@Table(name = "ziggy_DataFileType") +public class DataFileType implements Serializable { + + private static final long serialVersionUID = 20240122L; + + @Id + @XmlAttribute(required = true) + private String name; + + @XmlAttribute(required = true) + private String location; + + @XmlAttribute(required = true) + private String fileNameRegexp; + + @XmlAttribute(required = false) + private Boolean includeAllFilesInAllSubtasks = false; + + /** + * "The JPA specification requires all Entity classes to have a default no-arg constructor. This + * can be either public or protected." + */ + protected DataFileType() { + } + + public DataFileType(String name, String location) { + this(name, location, null); + } + + public DataFileType(String name, String location, String fileNameRegexp) { + this.name = name; + this.location = location; + this.fileNameRegexp = fileNameRegexp; + } + + public String getName() { + return name; + } + + public String getLocation() { + return location; + } + + public void setFileNameRegexp(String fileNameRegexp) { + this.fileNameRegexp = fileNameRegexp; + } + + public String getFileNameRegexp() { + return fileNameRegexp; + } + + public void setIncludeAllFilesInAllSubtasks(boolean includeAllFilesInAllSubtasks) { + this.includeAllFilesInAllSubtasks = includeAllFilesInAllSubtasks; + } + + public boolean isIncludeAllFilesInAllSubtasks() { + return includeAllFilesInAllSubtasks != null ? includeAllFilesInAllSubtasks : false; + } + + @Override + public int hashCode() { + return Objects.hash(fileNameRegexp, location, name); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + DataFileType other = (DataFileType) obj; + return Objects.equals(fileNameRegexp, other.fileNameRegexp) + && Objects.equals(location, other.location) && Objects.equals(name, other.name); + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreConfigurationFile.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationFile.java similarity index 57% rename from src/main/java/gov/nasa/ziggy/data/management/DatastoreConfigurationFile.java rename to src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationFile.java index 7256ac2..f104f89 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreConfigurationFile.java +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationFile.java @@ -1,4 +1,4 @@ -package gov.nasa.ziggy.data.management; +package gov.nasa.ziggy.data.datastore; import java.util.Collection; import java.util.HashSet; @@ -15,8 +15,9 @@ import jakarta.xml.bind.annotation.XmlRootElement; /** - * Models a single XML file containing definitions of model and data file types. The file contains - * one or more {@link DataFileType} definitions and one or more {@link ModelType} definitions. + * Models a single XML file containing definitions of model and data file types, datastore directory + * regular expressions, and datastore nodes. The file contains {@link DataFileType}, + * {@link ModelType}, {@link DatastoreRegexp}, and {@link DatastoreNode} definitions. * * @author PT */ @@ -27,7 +28,9 @@ public class DatastoreConfigurationFile implements HasXmlSchemaFilename { private static final String XML_SCHEMA_FILE_NAME = "datastore-configuration.xsd"; @XmlElements(value = { @XmlElement(name = "dataFileType", type = DataFileType.class), - @XmlElement(name = "modelType", type = ModelType.class) }) + @XmlElement(name = "modelType", type = ModelType.class), + @XmlElement(name = "datastoreRegexp", type = DatastoreRegexp.class), + @XmlElement(name = "datastoreNode", type = DatastoreNode.class) }) private Set datastoreConfigurationElements = new HashSet<>(); public Set getDatastoreConfigurationElements() { @@ -37,12 +40,16 @@ public Set getDatastoreConfigurationElements() { public void setDatastoreConfigurationElements(Set datastoreConfigurationElements) { Set originalConfigurationElements = this.datastoreConfigurationElements; this.datastoreConfigurationElements = datastoreConfigurationElements; - if (getDataFileTypes().size() + getModelTypes().size() != datastoreConfigurationElements - .size()) { + if (getDataFileTypes().size() + getModelTypes().size() + getRegexps().size() + + getDatastoreNodes().size() != datastoreConfigurationElements.size()) { this.datastoreConfigurationElements = originalConfigurationElements; throw new PipelineException("Number of data file types (" + getDataFileTypes().size() - + ") and number of model types (" + getModelTypes().size() - + ") does not match number of datastoreConfigurationElements objects (" + + "), number of model types (" + getModelTypes().size() + "), number of regexps (" + + getRegexps().size() + "), number of datastore nodes (" + + getDatastoreNodes().size() + ") total " + + (getDataFileTypes().size() + getModelTypes().size() + getRegexps().size() + + getDatastoreNodes().size()) + + " does not match number of datastoreConfigurationElements objects (" + datastoreConfigurationElements.size() + ")"); } } @@ -66,6 +73,26 @@ public void setModelTypes(Collection modelTypes) { datastoreConfigurationElements.addAll(modelTypes); } + public Set getRegexps() { + return CollectionFilters.filterToSet(datastoreConfigurationElements, DatastoreRegexp.class); + } + + public void setRegexps(Collection regexps) { + CollectionFilters.removeTypeFromCollection(datastoreConfigurationElements, + DatastoreRegexp.class); + datastoreConfigurationElements.addAll(regexps); + } + + public Set getDatastoreNodes() { + return CollectionFilters.filterToSet(datastoreConfigurationElements, DatastoreNode.class); + } + + public void setDatastoreNodes(Collection datastoreNodes) { + CollectionFilters.removeTypeFromCollection(datastoreConfigurationElements, + DatastoreNode.class); + datastoreConfigurationElements.addAll(datastoreNodes); + } + @Override public String getXmlSchemaFilename() { return XML_SCHEMA_FILE_NAME; diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java new file mode 100644 index 0000000..0e38cca --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporter.java @@ -0,0 +1,698 @@ +/* + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the + * National Aeronautics and Space Administration. All Rights Reserved. + * + * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline + * Management System for Data Analysis Pipelines, under Cooperative Agreement Nos. NNX14AH97A, + * 80NSSC18M0068 & 80NSSC21M0079. + * + * This file is available under the terms of the NASA Open Source Agreement (NOSA). You should have + * received a copy of this agreement with the Ziggy source code; see the file LICENSE.pdf. + * + * Disclaimers + * + * No Warranty: THE SUBJECT SOFTWARE IS PROVIDED "AS IS" WITHOUT ANY WARRANTY OF ANY KIND, EITHER + * EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SUBJECT + * SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A + * PARTICULAR PURPOSE, OR FREEDOM FROM INFRINGEMENT, ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL BE + * ERROR FREE, OR ANY WARRANTY THAT DOCUMENTATION, IF PROVIDED, WILL CONFORM TO THE SUBJECT + * SOFTWARE. THIS AGREEMENT DOES NOT, IN ANY MANNER, CONSTITUTE AN ENDORSEMENT BY GOVERNMENT AGENCY + * OR ANY PRIOR RECIPIENT OF ANY RESULTS, RESULTING DESIGNS, HARDWARE, SOFTWARE PRODUCTS OR ANY + * OTHER APPLICATIONS RESULTING FROM USE OF THE SUBJECT SOFTWARE. FURTHER, GOVERNMENT AGENCY + * DISCLAIMS ALL WARRANTIES AND LIABILITIES REGARDING THIRD-PARTY SOFTWARE, IF PRESENT IN THE + * ORIGINAL SOFTWARE, AND DISTRIBUTES IT "AS IS." + * + * Waiver and Indemnity: RECIPIENT AGREES TO WAIVE ANY AND ALL CLAIMS AGAINST THE UNITED STATES + * GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT. IF RECIPIENT'S + * USE OF THE SUBJECT SOFTWARE RESULTS IN ANY LIABILITIES, DEMANDS, DAMAGES, EXPENSES OR LOSSES + * ARISING FROM SUCH USE, INCLUDING ANY DAMAGES FROM PRODUCTS BASED ON, OR RESULTING FROM, + * RECIPIENT'S USE OF THE SUBJECT SOFTWARE, RECIPIENT SHALL INDEMNIFY AND HOLD HARMLESS THE UNITED + * STATES GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT, TO THE + * EXTENT PERMITTED BY LAW. RECIPIENT'S SOLE REMEDY FOR ANY SUCH MATTER SHALL BE THE IMMEDIATE, + * UNILATERAL TERMINATION OF THIS AGREEMENT. + */ + +package gov.nasa.ziggy.data.datastore; + +import java.io.File; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.CommandLineParser; +import org.apache.commons.cli.DefaultParser; +import org.apache.commons.cli.HelpFormatter; +import org.apache.commons.cli.Option; +import org.apache.commons.cli.Options; +import org.apache.commons.cli.ParseException; +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.hibernate.Hibernate; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.crud.DataFileTypeCrud; +import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; +import gov.nasa.ziggy.pipeline.xml.ValidatingXmlManager; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.ZiggyStringUtils; + +/** + * Performs import of the datastore configuration. + *

+ * The datastore configuration consists of the following: + *

    + *
  1. Definition of {@link DatastoreRegexp} instances. + *
  2. Definition of {@link DatastoreNode} instances. + *
  3. Definition of {@link DataFileType} instances. + *
  4. Definition of {@link ModelType} instances. + *
+ * + * @author PT + */ +public class DatastoreConfigurationImporter { + + private static final Logger log = LoggerFactory.getLogger(DatastoreConfigurationImporter.class); + public static final String DRY_RUN_OPTION = "dry-run"; + + private List filenames; + private boolean dryRun; + + private List dataFileTypes = new ArrayList<>(); + private List modelTypes = new ArrayList<>(); + private List datastoreRegexps = new ArrayList<>(); + private List datastoreNodes = new ArrayList<>(); + private List fullPathsForNodesToRemove = new ArrayList<>(); + private Set nodesForDatabase = new HashSet<>(); + + private Map databaseDatastoreNodesByFullPath; + + private DataFileTypeCrud dataFileTypeCrud = new DataFileTypeCrud(); + private ModelCrud modelCrud = new ModelCrud(); + private DatastoreRegexpCrud datastoreRegexpCrud = new DatastoreRegexpCrud(); + private DatastoreNodeCrud datastoreNodeCrud = new DatastoreNodeCrud(); + + // The following are instantiated so that unit tests that rely on them don't fail + + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + public static void main(String[] args) { + + CommandLineParser parser = new DefaultParser(); + Options options = new Options().addOption(Option.builder("n") + .longOpt(DRY_RUN_OPTION) + .hasArg(false) + .desc("Parses and creates objects but does not persist to database") + .build()); + + CommandLine cmdLine = null; + try { + cmdLine = parser.parse(options, args); + } catch (ParseException e) { + usageAndExit(options, "", e); + } + String[] filenames = cmdLine.getArgs(); + boolean dryRun = cmdLine.hasOption(DRY_RUN_OPTION); + DatastoreConfigurationImporter importer = new DatastoreConfigurationImporter( + Arrays.asList(filenames), dryRun); + + if (!dryRun) { + DatabaseTransactionFactory.performTransaction(() -> { + importer.importConfiguration(); + return null; + }); + } else { + importer.importConfiguration(); + } + } + + private static void usageAndExit(Options options, String message, Throwable e) { + // Until we've gotten through argument parsing, emit errors to stderr. Once we start the + // program, we'll be logging and throwing exceptions. + if (options != null) { + if (message != null) { + System.err.println(message); + } + new HelpFormatter().printHelp("DatastoreConfigurationImporter [options]", + "Import datastore configuration from XML file(s)", options, null); + } else if (e != null) { + log.error(message, e); + } + + System.exit(-1); + } + + public DatastoreConfigurationImporter(List filenames, boolean dryRun) { + this.filenames = filenames; + this.dryRun = dryRun; + } + + /** + * Perform the import from all XML files. The importer will skip any file that fails to validate + * or cannot be parsed, will skip any DataFileType instance that fails internal validation, and + * will skip any DataFileType that has the name of a type that is already in the database; all + * other DataFileTypes will be imported. If any duplicate names are present in the set of + * DataFileType instances to be imported, none will be imported. The import also imports model + * definitions. + */ + @SuppressWarnings("unchecked") + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + public void importConfiguration() { + + databaseDatastoreNodesByFullPath = (Map) DatabaseTransactionFactory + .performTransaction(() -> { + Map nodes = datastoreNodeCrud().retrieveNodesByFullPath(); + for (DatastoreNode node : nodes.values()) { + Hibernate.initialize(node.getChildNodeFullPaths()); + } + return nodes; + }); + + for (String filename : filenames) { + File file = new File(filename); + if (!file.exists() || !file.isFile()) { + log.warn("File {} is not a regular file", filename); + continue; + } + + // open and read the XML file + log.info("Reading from {}", filename); + DatastoreConfigurationFile configDoc = null; + try { + configDoc = new ValidatingXmlManager<>(DatastoreConfigurationFile.class) + .unmarshal(file); + } catch (Exception e) { + log.warn("Unable to parse configuration file {}", filename, e); + continue; + } + + log.info("Importing DataFileType definitions from {}", filename); + dataFileTypes.addAll(configDoc.getDataFileTypes()); + + // Now for the models + log.info("Importing ModelType definitions from {}", filename); + modelTypes.addAll(configDoc.getModelTypes()); + + log.info("Importing datastore regexp definitions from {}", filename); + datastoreRegexps.addAll(configDoc.getRegexps()); + + log.info("Importing datastore node definitions from {}", filename); + datastoreNodes.addAll(configDoc.getDatastoreNodes()); + } + checkDefinitions(); + if (!dryRun) { + persistDefinitions(); + } else { + log.info("Dry run."); + } + } + + /** + * Validates all imports. + *

+ * This method is ordinarily called as part of {@link #importConfiguration()}. It is broken out + * as a separate, package-scoped method to support testing. + */ + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + void checkDefinitions() { + + updateRegexps(); + checkAndUpdateNodeDefinitions(); + checkDataFileTypeDefinitions(); + checkModelTypeDefinitions(); + } + + /** + * Updates {@link DatastoreRegexp} definitions that are present in the import but which are also + * present in the database. Note that DatastoreRegexp definitions are never deleted. The + * instance variable datastoreRegexps is updated to contain the new DatastoreRegexp definitions + * and the updated DatastoreRegexp definitions. In the latter case, the actual objects in the + * datastoreRegexps list are the objects retrieved from the database, since we need to modify + * the database object in order to safely use the merge() method. + *

+ * The method returns a List of DatastoreRegexp names that includes the new names, the names of + * existing instances that are updated, and the names of existing instances that are not being + * touched (i.e., it's the list of names that will be in the database after the merge). + */ + private void updateRegexps() { + List regexpsToPersist = new ArrayList<>(); + + // Get the regexps out of the database. + @SuppressWarnings("unchecked") + Map databaseRegexpsByName = (Map) DatabaseTransactionFactory + .performTransaction(() -> datastoreRegexpCrud().retrieveRegexpsByName()); + Set regexpNames = new HashSet<>(databaseRegexpsByName.keySet()); + for (DatastoreRegexp regexp : datastoreRegexps) { + + // If the regexp is new, we need to persist it; if it matches one in the database, + // we need to update the value of the database copy. + DatastoreRegexp regexpToPersist = databaseRegexpsByName.containsKey(regexp.getName()) + ? databaseRegexpsByName.get(regexp.getName()) + : regexp; + regexpToPersist.setValue(regexp.getValue()); + regexpsToPersist.add(regexpToPersist); + regexpNames.add(regexpToPersist.getName()); + if (databaseRegexpsByName.containsKey(regexp.getName())) { + log.warn( + "Datastore regexp {} already exists, updating database value from {} to {}", + regexp.getName(), databaseRegexpsByName.get(regexp.getName()).getValue(), + regexp.getValue()); + } + } + datastoreRegexps = regexpsToPersist; + } + + /** + * Ensure that datastore nodes are defined correctly. Specifically, use the XML-only fields to + * populate the database fields, and generate database-appropriate parent-child relationships + * between the nodes. + *

+ * The imported node definitions are merged with existing node definitions in the database. This + * means that the identities of the child nodes are updated and the value of isRegexp is + * updated. This process can result in DatastoreNode instances or even entire branches of the + * node becoming obsolete. These nodes / branches will be deleted from the database table that + * holds the DatastoreNode definitions. + */ + private void checkAndUpdateNodeDefinitions() { + + Set allRegexpNames = allRegexpNames(); + + // Populate the set of DatabaseNode instances that will be persisted to the + // database. + findNodesForDatabase(allRegexpNames); + + for (DatastoreNode nodeForDatabase : nodesForDatabase) { + if (!isNodeSelfConsistent(nodeForDatabase, allRegexpNames, false)) { + log.warn("Unable to store datastore nodes in database due to validation failures"); + datastoreNodes.clear(); + fullPathsForNodesToRemove.clear(); + return; + } + } + log.info("All datastore nodes successfully populated"); + } + + @SuppressWarnings("unchecked") + private Set allRegexpNames() { + Set allRegexpNames = datastoreRegexps.stream() + .map(DatastoreRegexp::getName) + .collect(Collectors.toSet()); + allRegexpNames.addAll((List) DatabaseTransactionFactory + .performTransaction(() -> datastoreRegexpCrud().retrieveRegexpNames())); + + return allRegexpNames; + } + + /** Top-level method for locating the nodes that will be persisted. */ + private void findNodesForDatabase(Set allRegexpNames) { + findNodesForDatabase(datastoreNodes, allRegexpNames, ""); + } + + /** + * Business logic for locating nodes that will be persisted. + *

+ * The method first checks to see whether a given node is a new node or is an update to an + * existing node; if the latter, the existing node is updated with content from the imported + * node. Several consistency checks are performed on the node contents. The child nodes to the + * current node are located (either from imported or from existing database nodes), and the + * {@link #findNodesForDatabase(List, Set, String)} method is called on the child nodes. + *

+ */ + private void findNodesForDatabase(List datastoreNodes, + Set allRegexpNames, String parentNodeFullPath) { + for (DatastoreNode node : datastoreNodes) { + + node.setFullPath(fullPathFromParentPath(node, parentNodeFullPath)); + DatastoreNode nodeForDatabase = selectImportedOrDatabaseNode(node); + + // If we've already encountered a problem then we can't perform the + // child node portion of the search because the child nodes may + // contain duplicates. + if (!isNodeSelfConsistent(nodeForDatabase, allRegexpNames, true)) { + continue; + } + + // Find the child nodes to the current nodes, some of which + // may be database nodes. + List childNodes = childNodes(nodeForDatabase); + + // Generate nodeForDatabase instances for the child nodes. + nodesForDatabase.add(nodeForDatabase); + findNodesForDatabase(childNodes, allRegexpNames, node.getFullPath()); + } + } + + /** + * Full path string for a {@link DatastoreNode} when the full path of its parent is taken into + * account. + */ + private static String fullPathFromParentPath(DatastoreNode node, String parentFullPath) { + return fullPathFromParentPath(node.getName(), parentFullPath); + } + + /** + * Full path string for a {@link DatastoreNode} when the full path of its parent is taken into + * account. Package scoped for test purposes. + */ + static String fullPathFromParentPath(String nodeName, String parentFullPath) { + return StringUtils.isEmpty(parentFullPath) ? nodeName + : parentFullPath + DatastoreWalker.NODE_SEPARATOR + nodeName; + } + + /** + * Returns either the {@link DatastoreNode} passed as an argument, or else the existing database + * node with the same full path. In the latter case, content from the imported node is copied to + * the database node. + */ + private DatastoreNode selectImportedOrDatabaseNode(DatastoreNode node) { + DatastoreNode nodeForDatabase = node; + if (databaseDatastoreNodesByFullPath.get(node.getFullPath()) != null) { + nodeForDatabase = databaseDatastoreNodesByFullPath.get(node.getFullPath()); + nodeForDatabase.setRegexp(node.isRegexp()); + nodeForDatabase.setNodes(node.getNodes()); + nodeForDatabase.setXmlNodes(node.getXmlNodes()); + } + return nodeForDatabase; + } + + /** + * Performs self-consistency checks on a {@link DatastoreNode} and optionally generates log + * messages in the event of any failures. + */ + private boolean isNodeSelfConsistent(DatastoreNode node, Set allRegexpNames, + boolean doLogging) { + + // Check for undefined regexp. + if (node.isRegexp() && !allRegexpNames.contains(node.getName())) { + logOptionalErrorMessage(doLogging, "Node {} is undefined regexp", node.getName()); + return false; + } + + // Check for duplicate child node names. + List childNodeNames = childNodeNames(node.getNodes()); + List duplicateChildNodeNames = ZiggyStringUtils.duplicateStrings(childNodeNames); + if (!CollectionUtils.isEmpty(duplicateChildNodeNames)) { + logOptionalErrorMessage(doLogging, "Node {} has duplicate child names: {}", + node.getFullPath(), duplicateChildNodeNames.toString()); + return false; + } + + // Check XML nodes for duplicates + List duplicateXmlNodeNames = duplicateXmlNodeNames(node.getXmlNodes()); + if (!CollectionUtils.isEmpty(duplicateXmlNodeNames)) { + logOptionalErrorMessage(doLogging, "Node {} has duplicate XML node names: {}", + node.getFullPath(), duplicateXmlNodeNames.toString()); + return false; + } + return true; + } + + private void logOptionalErrorMessage(boolean doLogging, String format, Object... args) { + if (doLogging) { + log.error(format, args); + } + } + + /** + * Returns the child nodes of a given datastore node. + *

+ * The child nodes can include nodes that are in the xmlNodes field of the current node, and can + * also include database nodes that are not included in the xmlNodes field. Other xmlNodes + * elements can be more remote descendants of the current node (grandchildren, etc.). This + * method constructs the collection of child nodes to the current node, taking the + * aforementioned factors into account, and puts any remote descendants from xmlNodes into the + * xmlNodes fields of the child nodes. + *

+ * In the process of updating the child node population, nodes that were child nodes of the + * original database node may no longer be children of that node. In that case, the obsolete + * child nodes must be removed from the database, along with all of their descendants. The node + * deletion information is also updated as part of this process. + */ + private List childNodes(DatastoreNode node) { + + // Update the full paths of any child nodes. + List childNodeNames = childNodeNames(node.getNodes()); + List originalChildNodeFullPaths = node.getChildNodeFullPaths(); + List updatedChildNodeFullPaths = new ArrayList<>(); + for (String childNodeName : childNodeNames) { + if (!StringUtils.isBlank(childNodeName)) { + updatedChildNodeFullPaths + .add(fullPathFromParentPath(childNodeName, node.getFullPath())); + } + } + + // Mark any obsolete child nodes for deletion. + node.setChildNodeFullPaths(updatedChildNodeFullPaths); + originalChildNodeFullPaths.removeAll(updatedChildNodeFullPaths); + updateFullPathsForNodesToRemove(originalChildNodeFullPaths); + + // Add full path values to the XML nodes, assuming that the XML + // nodes are all children of the current node. + for (DatastoreNode xmlNode : node.getXmlNodes()) { + xmlNode.setFullPath(fullPathFromParentPath(xmlNode, node.getFullPath())); + } + + // Obtain the child nodes (which may include nodes from the database that are + // not included in the imported nodes), and find nodes that might be more + // remote descendants of the current node. + List childNodes = locateChildNodes(node, childNodeNames); + List descendantNodes = new ArrayList<>(node.getXmlNodes()); + descendantNodes.removeAll(childNodes); + + // Add the remote descendants to the xmlNodes of all the children. + for (DatastoreNode childNode : childNodes) { + childNode.getXmlNodes().addAll(descendantNodes); + } + + // Remove the remote descendants from the current node's xmlNodes. + node.getXmlNodes().removeAll(descendantNodes); + return childNodes; + } + + /** Recursively adds datastore node full paths to the list of nodes for removal. */ + private void updateFullPathsForNodesToRemove(List fullPathsForRemoval) { + if (fullPathsForRemoval.isEmpty()) { + return; + } + for (String fullPathForRemoval : fullPathsForRemoval) { + updateFullPathsForNodesToRemove( + databaseDatastoreNodesByFullPath.get(fullPathForRemoval).getChildNodeFullPaths()); + fullPathsForNodesToRemove.add(fullPathForRemoval); + log.warn("Datastore location {} will be removed from the database", fullPathForRemoval); + } + } + + /** + * Locates the child nodes for the current database node based on their names. + *

+ * The imported nodes, stored in the current node's xmlNodes field, are searched for + * appropriately-named nodes. Any that are found are added to the child nodes collection. Any + * missing child nodes are retrieved from the existing database nodes. In this latter case, the + * database node's child node full paths have to be converted back into a String for that node's + * nodes field. + */ + private List locateChildNodes(DatastoreNode node, List childNodeNames) { + List childNodes = new ArrayList<>(); + for (String childNodeName : childNodeNames) { + DatastoreNode childNode = null; + for (DatastoreNode xmlNode : node.getXmlNodes()) { + if (xmlNode.getName().equals(childNodeName)) { + childNode = xmlNode; + continue; + } + } + if (childNode == null) { + childNode = childNodeFromDatabaseNodes( + fullPathFromParentPath(childNodeName, node.getFullPath())); + } + if (childNode != null) { + childNodes.add(childNode); + } + } + return childNodes; + } + + /** Locates a child node from the existing database nodes and updates its fields. */ + private DatastoreNode childNodeFromDatabaseNodes(String childNodeFullPath) { + DatastoreNode childNode = databaseDatastoreNodesByFullPath.get(childNodeFullPath); + if (childNode != null) { + childNode.setNodes(nodesFromChildNodeFullPaths(childNode)); + } + return childNode; + } + + /** Converts a node's child node full paths to a node string. */ + String nodesFromChildNodeFullPaths(DatastoreNode node) { + if (node.getChildNodeFullPaths().isEmpty()) { + return ""; + } + StringBuilder sb = new StringBuilder(); + for (String childNodeFullPath : node.getChildNodeFullPaths()) { + sb.append(databaseDatastoreNodesByFullPath.get(childNodeFullPath).getName()); + sb.append(DatastoreNode.CHILD_NODE_NAME_DELIMITER); + } + sb.setLength(sb.length() - 1); + return sb.toString(); + } + + /** Converts the nodes field into a {@link List} of child node names. */ + List childNodeNames(String xmlNodesAttribute) { + List childNodeNames = new ArrayList<>(); + String[] childNodeNamesArray = xmlNodesAttribute + .split(DatastoreNode.CHILD_NODE_NAME_DELIMITER); + for (String childNodeName : childNodeNamesArray) { + childNodeNames.add(childNodeName.trim()); + } + return childNodeNames; + } + +// /** Checks a {@link List} of {@link String}s for duplicates. */ +// List duplicateStrings(List childNodeNames) { +// Set allChildNodeNames = new HashSet<>(); +// return childNodeNames.stream() +// .filter(s -> !allChildNodeNames.add(s)) +// .collect(Collectors.toList()); +// } + + /** Checks a {@link List} of {@link DatastoreNode}s for duplicate names. */ + List duplicateXmlNodeNames(List xmlNodes) { + return ZiggyStringUtils.duplicateStrings( + xmlNodes.stream().map(DatastoreNode::getName).collect(Collectors.toList())); + } + + /** Ensure uniqueness of all imported data file type definitions. */ + private void checkDataFileTypeDefinitions() { + + // First check against the database definitions. + List dataFileTypesNotImported = new ArrayList<>(); + @SuppressWarnings("unchecked") + List databaseDataFileTypeNames = (List) DatabaseTransactionFactory + .performTransaction(() -> dataFileTypeCrud().retrieveAllNames()); + for (DataFileType typeXb : dataFileTypes) { + if (databaseDataFileTypeNames.contains(typeXb.getName())) { + log.warn("Not importing data file type definition \"{}\"" + + " due to presence of existing type with same name", typeXb.getName()); + dataFileTypesNotImported.add(typeXb); + continue; + } + } + dataFileTypes.removeAll(dataFileTypesNotImported); + + // Now check for duplicates within the imports. + Set uniqueDataFileTypeNames = dataFileTypes.stream() + .map(DataFileType::getName) + .collect(Collectors.toSet()); + if (dataFileTypes.size() != uniqueDataFileTypeNames.size()) { + throw new IllegalStateException( + "Unable to persist data file types due to duplicate names"); + } + } + + /** + * Check that all model type definitions are unique and that their database-only fields can be + * populated without errors. + */ + private void checkModelTypeDefinitions() { + List modelTypesNotImported = new ArrayList<>(); + @SuppressWarnings("unchecked") + List databaseModelTypes = (List) DatabaseTransactionFactory + .performTransaction(() -> modelCrud().retrieveAllModelTypes() + .stream() + .map(ModelType::getType) + .collect(Collectors.toList())); + for (ModelType modelTypeXb : modelTypes) { + try { + modelTypeXb.validate(); + } catch (Exception e) { + log.warn("Unable to validate model type definition " + modelTypeXb.getType(), e); + modelTypesNotImported.add(modelTypeXb); + continue; + } + if (databaseModelTypes.contains(modelTypeXb.getType())) { + log.warn( + "Not importing model type definition \"{}\"" + + " due to presence of existing type with same name", + modelTypeXb.getType()); + modelTypesNotImported.add(modelTypeXb); + continue; + } + } + modelTypes.removeAll(modelTypesNotImported); + log.info("Imported {} ModelType definitions from files", modelTypes.size()); + + // Now check for duplicate model names in the imports. + Set uniqueModelTypeNames = modelTypes.stream() + .map(ModelType::getType) + .collect(Collectors.toSet()); + if (modelTypes.size() != uniqueModelTypeNames.size()) { + throw new IllegalStateException("Unable to persist model types due to duplicate names"); + } + } + + /** Persist all definitions to the database. */ + private void persistDefinitions() { + DatabaseTransactionFactory.performTransaction(() -> { + log.info("Persisting to database {} DataFileType definitions", dataFileTypes.size()); + dataFileTypeCrud().persist(dataFileTypes); + log.info("Persisting to database {} model definitions", modelTypes.size()); + modelCrud().persist(modelTypes); + log.info("Persisting to database {} regexp definitions", datastoreRegexps.size()); + for (DatastoreRegexp regexp : datastoreRegexps) { + datastoreRegexpCrud().merge(regexp); + } + log.info("Deleting from database {} datastore node definitions", + fullPathsForNodesToRemove.size()); + for (String fullPathForRemoval : fullPathsForNodesToRemove) { + datastoreNodeCrud() + .remove(databaseDatastoreNodesByFullPath.get(fullPathForRemoval)); + } + log.info("Persisting to database {} datastore node definitions", + nodesForDatabase.size()); + for (DatastoreNode nodeForDatabase : nodesForDatabase) { + datastoreNodeCrud().merge(nodeForDatabase); + } + log.info("Persist step complete"); + return null; + }); + } + + DataFileTypeCrud dataFileTypeCrud() { + return dataFileTypeCrud; + } + + ModelCrud modelCrud() { + return modelCrud; + } + + DatastoreRegexpCrud datastoreRegexpCrud() { + return datastoreRegexpCrud; + } + + DatastoreNodeCrud datastoreNodeCrud() { + return datastoreNodeCrud; + } + + List getDataFileTypes() { + return dataFileTypes; + } + + List getModelTypes() { + return modelTypes; + } + + List getRegexps() { + return datastoreRegexps; + } + + Set nodesForDatabase() { + return nodesForDatabase; + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java new file mode 100644 index 0000000..af69587 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreFileManager.java @@ -0,0 +1,570 @@ +package gov.nasa.ziggy.data.datastore; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Predicate; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +import org.apache.commons.collections4.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.data.management.DatastoreProducerConsumer; +import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; +import gov.nasa.ziggy.module.AlgorithmStateFiles; +import gov.nasa.ziggy.module.SubtaskUtils; +import gov.nasa.ziggy.pipeline.definition.ModelMetadata; +import gov.nasa.ziggy.pipeline.definition.ModelRegistry; +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.uow.UnitOfWork; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.FileUtil; + +/** + * Provides services related to marshaling and persisting data files, and transporting them between + * the datastore and the task directories. These services include: + *

    + *
  1. Creating inputs and outputs subdirectories in the task directory for a given + * {@link PipelineTask}. + *
  2. Identifying the datastore directories that contain files to use as inputs for a task. + *
  3. Copying or linking the input files for a task to a subtask directory of the task directory. + *
  4. Copying either all files or all newly-created files, depending on whether the use-case is + * reprocessing or forward processing. + *
  5. Copying or moving the output files from the subtasks of the task directory to the datastore. + *
  6. Managing file permissions for the datastore: the files in the datastore are write-protected + * except when being deliberately overwritten with newer results. + *
      + * + * @author PT + */ + +public class DatastoreFileManager { + + private static final Logger log = LoggerFactory.getLogger(DatastoreFileManager.class); + + private static final Predicate WITH_OUTPUTS = AlgorithmStateFiles::hasOutputs; + private static final Predicate WITHOUT_OUTPUTS = WITH_OUTPUTS.negate(); + public static final String FILE_NAME_DELIMITER = "\\."; + public static final String SINGLE_SUBTASK_BASE_NAME = "Single Subtask"; + + private final PipelineTask pipelineTask; + private AlertService alertService = new AlertService(); + private DatastoreWalker datastoreWalker; + private final Path taskDirectory; + private PipelineDefinitionCrud pipelineDefinitionCrud = new PipelineDefinitionCrud(); + private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud = new DatastoreProducerConsumerCrud(); + private PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); + + public DatastoreFileManager(PipelineTask pipelineTask, Path taskDirectory) { + this.pipelineTask = pipelineTask; + this.taskDirectory = taskDirectory; + } + + /** + * Constructs the collection of {@link Path}s for each subtask. + *

      + * All subtasks must have one data file from each file-per-subtask data file type. Any subtask + * that is missing one or more files is omitted from the returned {@link List}. + */ + public Map> filesForSubtasks() { + + // Obtain the data file types that the module requires + Set dataFileTypes = pipelineTask.pipelineDefinitionNode() + .getInputDataFileTypes(); + // Construct a List of data file types that expect 1 file per subtask. + List filePerSubtaskDataFileTypes = dataFileTypes.stream() + .filter(s -> !s.isIncludeAllFilesInAllSubtasks()) + .collect(Collectors.toList()); + + // Construct a list of data file types for which all files need to be provided + // to all subtasks. + List allFilesAllSubtasksDataFileTypes = new ArrayList<>(dataFileTypes); + allFilesAllSubtasksDataFileTypes.removeAll(filePerSubtaskDataFileTypes); + + UnitOfWork uow = pipelineTask.uowTaskInstance(); + // Generate a Map from each file-per-subtask data file type to all the data files for + // that type; then the same for the all-files-all-subtask types. + Map> pathsByPerSubtaskDataType = pathsByDataFileType(uow, + filePerSubtaskDataFileTypes); + Map> pathsByAllSubtasksDataType = pathsByDataFileType(uow, + allFilesAllSubtasksDataFileTypes); + + // If the user wants new-data processing only, filter the data files to remove + // any that were processed already by the pipeline module that's assigned to + // this pipeline task. + if (!singleSubtask() && pipelineDefinitionCrud() + .retrieveProcessingMode( + pipelineTask.getPipelineInstance().getPipelineDefinition().getName()) + .equals(ProcessingMode.PROCESS_NEW)) { + filterOutDataFilesAlreadyProcessed(pathsByPerSubtaskDataType); + } + + // Produce the List using just the file-per-subtask data files. + Map> filesForSubtasks = filePerSubtaskFilesForSubtasks( + pathsByPerSubtaskDataType); + + // if this task will use a single subtask, it's possible that it has + // no input types that are in the one-file-per-subtask category. Handle + // that corner case now. + if (singleSubtask() && filesForSubtasks.isEmpty()) { + filesForSubtasks.put(SINGLE_SUBTASK_BASE_NAME, new HashSet<>()); + } + + // Add the all-files-all-subtasks paths to all the subtasks. + Set allFilesAllSubtasks = new HashSet<>(); + for (Set files : pathsByAllSubtasksDataType.values()) { + allFilesAllSubtasks.addAll(files); + } + for (Set files : filesForSubtasks.values()) { + files.addAll(allFilesAllSubtasks); + } + return filesForSubtasks; + } + + /** + * Produces a {@link Map} from a given {@link DataFileType} to the data files for that type, + * based on the unit of work. + */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private Map> pathsByDataFileType(UnitOfWork uow, + List dataFileTypes) { + Map> pathsByDataFileType = new HashMap<>(); + + // Obtain the Map from data file type names to UOW paths + Map pathsByDataTypeName = DirectoryUnitOfWorkGenerator + .directoriesByDataFileType(uow); + for (DataFileType dataFileType : dataFileTypes) { + Path datastoreDirectory = Paths.get(pathsByDataTypeName.get(dataFileType.getName())); + pathsByDataFileType.put(dataFileType, + FileUtil.listFiles(datastoreDirectory, dataFileType.getFileNameRegexp())); + } + return pathsByDataFileType; + } + + /** + * Filters out data files that have already been processed for situations in which the user only + * wants to process new data files (i.e., files that have not yet been processed). + *

      + * The method works by obtaining the {@link DatastoreProducerConsumer} records for all the files + * in the datastore that are going to be processed by this task. It then finds the intersection + * of the consumer task IDs for the files and IDs for tasks that share the same pipeline + * definition node. Any file that has a consumer in that intersection set must be omitted from + * processing. + */ + private void filterOutDataFilesAlreadyProcessed( + Map> pathsByPerSubtaskDataType) { + + for (Set paths : pathsByPerSubtaskDataType.values()) { + + // The names in the producer-consumer table are relative to the datastore + // root, while the values in pathsByPerSubtaskDataType are absolute. + // Generate a relativized Set now. + Set relativizedFilePaths = paths.stream() + .map(s -> DirectoryProperties.datastoreRootDir().toAbsolutePath().relativize(s)) + .map(Path::toString) + .collect(Collectors.toSet()); + + // Find the consumers that correspond to the definition node of the current task. + List consumersWithMatchingPipelineNode = pipelineTaskCrud() + .retrieveIdsForPipelineDefinitionNode(pipelineTask.pipelineDefinitionNode(), null); + + // Obtain the Set of datastore files that are in the relativizedFilePaths collection + // and which have a consumer that matches the pipeline definition node of the current + // pipeline task. + Set namesOfFilesAlreadyProcessed = datastoreProducerConsumerCrud() + .retrieveFilesConsumedByTasks(consumersWithMatchingPipelineNode, + relativizedFilePaths); + + if (CollectionUtils.isEmpty(namesOfFilesAlreadyProcessed)) { + continue; + } + + // Convert the strings back to absolute paths. + Set filesAlreadyProcessed = namesOfFilesAlreadyProcessed.stream() + .map(Paths::get) + .map(t -> DirectoryProperties.datastoreRootDir().toAbsolutePath().resolve(t)) + .collect(Collectors.toSet()); + + // Remove the files already processed from the set of paths. + paths.removeAll(filesAlreadyProcessed); + } + } + + /** + * Generates the portion of the {@link List} of {@link Set}s of {@link Paths} for each subtask + * that comes from file-per-subtask data types. Returns a {@link Map} from the data file base + * name (i.e., everything before the first "." in its name) to all the data files that have that + * base name. Each Map entry's value are the set of input files needed for a given subtask. + */ + private Map> filePerSubtaskFilesForSubtasks( + Map> pathsByPerSubtaskDataType) { + + if (singleSubtask()) { + Set allDataFiles = new HashSet<>(); + for (Set paths : pathsByPerSubtaskDataType.values()) { + allDataFiles.addAll(paths); + } + return Map.of(SINGLE_SUBTASK_BASE_NAME, allDataFiles); + } + + // Generate the mapping from regexp group values to sets of files. + Map> filePerSubtaskFilesForSubtasks = new HashMap<>(); + for (Map.Entry> entry : pathsByPerSubtaskDataType.entrySet()) { + addPathsByRegexpGroupValues(filePerSubtaskFilesForSubtasks, entry); + } + + // Check for cases that have insufficient files. These are cases in which one or more + // data file type has no file for the given subtasks, which means that these are subtasks + // that cannot run. Note that the logic of regular expressions guarantees that each data + // file type can produce no more than one file that matches a given data file type regexp. + int subtaskCount = filePerSubtaskFilesForSubtasks.size(); + int dataFileTypeCount = pathsByPerSubtaskDataType.size(); + Set regexpGroupValuesForInvalidSubtasks = new HashSet<>(); + for (Map.Entry> subtaskMapEntry : filePerSubtaskFilesForSubtasks + .entrySet()) { + if (subtaskMapEntry.getValue().size() < dataFileTypeCount) { + regexpGroupValuesForInvalidSubtasks.add(subtaskMapEntry.getKey()); + } + } + if (!regexpGroupValuesForInvalidSubtasks.isEmpty()) { + log.warn("{} subtasks out of {} missing files and will not be processed", + regexpGroupValuesForInvalidSubtasks.size(), subtaskCount); + for (String regexpGroupValuesForInvalidSubtask : regexpGroupValuesForInvalidSubtasks) { + filePerSubtaskFilesForSubtasks.remove(regexpGroupValuesForInvalidSubtask); + } + } + return filePerSubtaskFilesForSubtasks; + } + + /** + * Adds the {@link Path}s for a given {@link DataFileType} to the overall {@link Map} of paths + * by concatenated regexp group values. Each entry in the map represents a subtask in which all + * of the data files for the subtask have matching values for their regexp groups. + */ + private void addPathsByRegexpGroupValues(Map> pathsByRegexpGroupValue, + Map.Entry> pathsByDataFileType) { + Pattern dataFileTypePattern = Pattern + .compile(pathsByDataFileType.getKey().getFileNameRegexp()); + + for (Path path : pathsByDataFileType.getValue()) { + String concatenatedRegexpGroups = concatenatedRegexpGroups(dataFileTypePattern, path); + if (StringUtils.isBlank(concatenatedRegexpGroups)) { + continue; + } + if (pathsByRegexpGroupValue.get(concatenatedRegexpGroups) == null) { + pathsByRegexpGroupValue.put(concatenatedRegexpGroups, new HashSet<>()); + } + pathsByRegexpGroupValue.get(concatenatedRegexpGroups).add(path); + } + } + + /** + * Applies a {@link Pattern} to the file name element of a {@link Path}, and returns the + * concatenation of the values of the regexp groups. For example, if the pattern is + * "(\\S+)-bauhaus-(\\S+).nc" and the file name is "foo-bauhaus-baz.nc", then this method + * returns "foobaz". + */ + private String concatenatedRegexpGroups(Pattern dataFileTypePattern, Path file) { + Matcher matcher = dataFileTypePattern.matcher(file.getFileName().toString()); + if (!matcher.matches()) { + log.warn("File {} does not match regexp {}", file.getFileName().toString(), + dataFileTypePattern.pattern()); + return null; + } + StringBuilder groupValueConcatenator = new StringBuilder(); + for (int groupIndex = 1; groupIndex <= matcher.groupCount(); groupIndex++) { + groupValueConcatenator.append(matcher.group(groupIndex)); + } + return groupValueConcatenator.toString(); + } + + /** + * Returns the model files for the task. The return is in the form of a {@link Map} in which the + * datastore paths of the current models are the keys and the names of the files in the task + * directory are the values. + */ + public Map modelFilesForTask() { + Map modelFilesForTask = new HashMap<>(); + + // Get the model registry and the model types from the pipeline task. + ModelRegistry modelRegistry = pipelineTask.getPipelineInstance().getModelRegistry(); + Set modelTypes = pipelineTask.pipelineDefinitionNode().getModelTypes(); + + // Put the model location in the datastore, and its original file name, into the Map. + for (ModelType modelType : modelTypes) { + ModelMetadata metadata = modelRegistry.getModels().get(modelType); + modelFilesForTask.put(metadata.datastoreModelPath(), metadata.getOriginalFileName()); + } + return modelFilesForTask; + } + + /** Copies datastore files to the subtask directories. Both data files and models are copied. */ + public Map> copyDatastoreFilesToTaskDirectory( + Collection> subtaskFiles, Map modelFilesForTask) { + + List> subtaskFilesCopy = new ArrayList<>(subtaskFiles); + log.info("Generating subtasks..."); + // The algorithm may want one subtask per task. Handle that case now. + if (pipelineTask.getPipelineInstanceNode().getPipelineDefinitionNode().getSingleSubtask()) { + Set filesForSingleSubtask = new HashSet<>(); + for (Set files : subtaskFiles) { + filesForSingleSubtask.addAll(files); + } + subtaskFilesCopy.clear(); + subtaskFilesCopy.add(filesForSingleSubtask); + } + + Map> pathsBySubtaskDirectory = new HashMap<>(); + + // Loop over subtasks. + int subtaskIndex = 0; + int loggingIndex = Math.max(1, subtaskFilesCopy.size() / 20); + for (Set files : subtaskFilesCopy) { + Path subtaskDirectory = SubtaskUtils.createSubtaskDirectory(taskDirectory(), + subtaskIndex); + + // Copy or link the data files. + for (Path file : files) { + Path destination = subtaskDirectory.resolve(file.getFileName()); + copyOrLink(file, destination); + } + if (modelFilesForTask == null) { + continue; + } + + // Copy or link the models. + for (Map.Entry modelEntry : modelFilesForTask.entrySet()) { + Path destination = subtaskDirectory.resolve(modelEntry.getValue()); + copyOrLink(modelEntry.getKey(), destination); + } + if (subtaskIndex++ % loggingIndex == 0) { + log.info("Subtask {} of {} generated", subtaskIndex, subtaskFilesCopy.size()); + } + pathsBySubtaskDirectory.put(subtaskDirectory, files); + } + log.info("Generating subtasks...done"); + return pathsBySubtaskDirectory; + } + + /** + * Copies output files from the task directory to the datastore, returning the Set of datastore + * Paths that result from the copy operations. + */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public Set copyTaskDirectoryFilesToDatastore() { + + log.info("Copying output files to datastore..."); + Set outputDataFileTypes = pipelineTask.pipelineDefinitionNode() + .getOutputDataFileTypes(); + Map regexpValues = DatastoreDirectoryUnitOfWorkGenerator + .regexpValues(pipelineTask.uowTaskInstance()); + Map datastorePathByDataFileType = new HashMap<>(); + + // Get a Map from each data file type to its location in the datastore. Here + // we use the regexp values captured in the UOW, which in turn is captured + // in the PipelineTask, to perform the mapping. + for (DataFileType dataFileType : outputDataFileTypes) { + datastorePathByDataFileType.put(dataFileType, datastoreWalker() + .pathFromLocationAndRegexpValues(regexpValues, dataFileType.getLocation())); + } + + // Generate the paths of all subtask directories. + Set subtaskDirs = FileUtil.listFiles(taskDirectory(), + Set.of(SubtaskUtils.SUBTASK_DIR_PATTERN), null); + + // Construct a Map from the data file type to the set of output files of that type. + Map> outputFilesByDataFileType = new HashMap<>(); + for (DataFileType dataFileType : outputDataFileTypes) { + Set outputFiles = new HashSet<>(); + for (Path subtaskDir : subtaskDirs) { + outputFiles + .addAll(FileUtil.listFiles(subtaskDir, dataFileType.getFileNameRegexp())); + } + outputFilesByDataFileType.put(dataFileType, outputFiles); + } + + // Copy the files from the subtask directories to the correct datastore location. + Set outputFiles = new HashSet<>(); + for (Map.Entry> outputFilesEntry : outputFilesByDataFileType + .entrySet()) { + Path datastoreLocation = datastorePathByDataFileType.get(outputFilesEntry.getKey()); + try { + Files.createDirectories(datastoreLocation); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + FileUtil.prepareDirectoryTreeForOverwrites(datastoreLocation); + for (Path outputFile : outputFilesEntry.getValue()) { + Path destination = datastoreLocation.resolve(outputFile.getFileName()); + copyOrLink(outputFile, destination); + outputFiles.add(destination); + } + FileUtil.writeProtectDirectoryTree(datastoreLocation); + } + log.info("Copying output files to datastore...done"); + return outputFiles; + } + + /** Returns the number of subtasks for a given task. */ + public int subtaskCount() { + return filesForSubtasks().size(); + } + + /** + * Determines the input files that are associated with outputs (i.e., they are used in a task + * that produced outputs) and the files that are not associated with outputs. Returns an object + * that provides both sets of information for the caller. + */ + public InputFiles inputFilesByOutputStatus() { + + // Identify the subtasks that have, or fail to have, outputs. + Set subtasksWithOutputs = subtaskDirectoriesWithOutputStatus(WITH_OUTPUTS); + Set subtasksWithoutOutputs = subtaskDirectoriesWithOutputStatus(WITHOUT_OUTPUTS); + + // Construct the paths for each kind of subdirectory + Set filesWithOutputs = inputsFilesInSubtaskDirectories(subtasksWithOutputs); + Set filesWithoutOutputs = inputsFilesInSubtaskDirectories(subtasksWithoutOutputs); + + // If a file produced outputs in some subdirectories but not others, we need to count it + // as producing outputs on this task, so remove any entries in filesWithOutputs from + // the set of filesWithoutOutputs. + filesWithoutOutputs.removeAll(filesWithOutputs); + + return new InputFiles(filesWithOutputs, filesWithoutOutputs); + } + + /** + * Returns the {@link Set} of subtask directory {@link Path}s that represent completed subtasks + * with a given outputs status (either with or without outputs). + */ + private Set subtaskDirectoriesWithOutputStatus(Predicate outputsStatus) { + return SubtaskUtils.subtaskDirectories(taskDirectory) + .stream() + .map(Path::toFile) + .filter(AlgorithmStateFiles::isComplete) + .filter(outputsStatus) + .map(File::toPath) + .collect(Collectors.toSet()); + } + + /** Returns the {@link Set} of input file {@link Path}s from a set of subdirectory Paths. */ + private Set inputsFilesInSubtaskDirectories(Set subtaskDirectories) { + Set inputsFiles = new HashSet<>(); + Set inputDataFileTypes = pipelineTask.pipelineDefinitionNode() + .getInputDataFileTypes(); + for (DataFileType fileType : inputDataFileTypes) { + inputsFiles.addAll(filesInSubtaskDirsOfType(fileType, subtaskDirectories)); + } + return inputsFiles; + } + + /** + * Returns the {@link Set} of file {@link Path}s for a given {@link DataFileType}, across a + * collection of subtask directory Paths. + */ + private Set filesInSubtaskDirsOfType(DataFileType dataFileType, + Set subtaskDirectories) { + Set filesInSubtaskDirsOfType = new HashSet<>(); + for (Path subtaskDirectory : subtaskDirectories) { + filesInSubtaskDirsOfType + .addAll(FileUtil.listFiles(subtaskDirectory, dataFileType.getFileNameRegexp())); + } + + // Convert the files back to their datastore names so that we can use this information + // to track the producer-consumer relationships for the files. + Path datastorePath = datastoreWalker().pathFromLocationAndRegexpValues( + DatastoreDirectoryUnitOfWorkGenerator.regexpValues(pipelineTask.uowTaskInstance()), + dataFileType.getLocation()); + return filesInSubtaskDirsOfType.stream() + .map(s -> datastorePath.resolve(s.getFileName())) + .collect(Collectors.toSet()); + } + + boolean singleSubtask() { + return pipelineTask.pipelineDefinitionNode().getSingleSubtask(); + } + + AlertService alertService() { + return alertService; + } + + DatastoreWalker datastoreWalker() { + if (datastoreWalker == null) { + datastoreWalker = DatastoreWalker.newInstance(); + } + return datastoreWalker; + } + + public Path taskDirectory() { + return taskDirectory; + } + + PipelineDefinitionCrud pipelineDefinitionCrud() { + return pipelineDefinitionCrud; + } + + DatastoreProducerConsumerCrud datastoreProducerConsumerCrud() { + return datastoreProducerConsumerCrud; + } + + PipelineTaskCrud pipelineTaskCrud() { + return pipelineTaskCrud; + } + + /** + * Performs a hard-link or a copy of a file. Hard links can generally be created only from a + * target file to another location on the same file system, but Java doesn't appear to give us + * any way to determine the latter. Thus: we try to link, and if an exception occurs we execute + * a copy operation. + */ + @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) + public static void copyOrLink(Path src, Path dest) { + try { + FileUtil.CopyType.LINK.copy(src, dest); + } catch (Exception unableToLinkException) { + FileUtil.CopyType.COPY.copy(src, dest); + } + } + + /** Container class for files that either have outputs associated with them, or not. */ + public static class InputFiles { + private final Set filesWithOutputs; + private final Set filesWithoutOutputs; + + public InputFiles(Set filesWithOutputs, Set filesWithoutOutputs) { + this.filesWithOutputs = filesWithOutputs; + this.filesWithoutOutputs = filesWithoutOutputs; + } + + public Set getFilesWithOutputs() { + return filesWithOutputs; + } + + public Set getFilesWithoutOutputs() { + return filesWithoutOutputs; + } + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreNode.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreNode.java new file mode 100644 index 0000000..f85507b --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreNode.java @@ -0,0 +1,153 @@ +package gov.nasa.ziggy.data.datastore; + +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import jakarta.persistence.ElementCollection; +import jakarta.persistence.Entity; +import jakarta.persistence.FetchType; +import jakarta.persistence.GeneratedValue; +import jakarta.persistence.GenerationType; +import jakarta.persistence.Id; +import jakarta.persistence.JoinTable; +import jakarta.persistence.SequenceGenerator; +import jakarta.persistence.Table; +import jakarta.persistence.Transient; +import jakarta.persistence.UniqueConstraint; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlAttribute; +import jakarta.xml.bind.annotation.XmlElement; +import jakarta.xml.bind.annotation.XmlTransient; + +@XmlAccessorType(XmlAccessType.NONE) +@Entity +@Table(name = "Ziggy_DatastoreNode", + uniqueConstraints = { @UniqueConstraint(columnNames = { "fullPath" }) }) +public class DatastoreNode { + + public static final String CHILD_NODE_NAME_DELIMITER = ","; + + @XmlTransient + @Transient + private static final Logger log = LoggerFactory.getLogger(DatastoreNode.class); + + @Id + @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_DatastoreNode_generator") + @SequenceGenerator(name = "ziggy_DatastoreNode_generator", initialValue = 1, + sequenceName = "ziggy_DatastoreNode_sequence", allocationSize = 1) + private Long id; + + @XmlAttribute(required = true) + private String name; + + // The full path, relative to datastore root, for the node. + @XmlTransient + private String fullPath = ""; + + // Indicates that this node is a representation of a regular expression object. + @XmlAttribute(required = false, name = "isRegexp") + private Boolean regexp = false; + + // The names of the child nodes to this node. XML only. + @Transient + @XmlAttribute(required = false) + private String nodes = ""; + + // Full paths to the child nodes to this node. Database only. + // Note that each node always needs to know its child nodes, + // hence fetch type here is EAGER. + @XmlTransient + @ElementCollection(fetch = FetchType.EAGER) + @JoinTable(name = "ziggy_DatastoreNode_childNodeFullPaths") + private List childNodeFullPaths = new ArrayList<>(); + + // Nodes that are elements of the current node. XML only. + @Transient + @XmlElement(name = "datastoreNode", required = false) + private List xmlNodes = new ArrayList<>(); + + public DatastoreNode() { + } + + /** For testing only. */ + DatastoreNode(String name, boolean regexp) { + this.name = name; + this.regexp = regexp; + } + + public Long getId() { + return id; + } + + public String getName() { + return name; + } + + void setFullPath(String fullPath) { + this.fullPath = fullPath; + } + + public String getFullPath() { + return fullPath; + } + + public Boolean isRegexp() { + return regexp; + } + + /** + * Package scoped because only {@link DatastoreConfigurationImporter} should be able to modify + * this. + */ + void setRegexp(Boolean regexp) { + this.regexp = regexp; + } + + public String getNodes() { + return nodes; + } + + public void setNodes(String nodes) { + this.nodes = nodes; + } + + public List getChildNodeFullPaths() { + return childNodeFullPaths; + } + + public void setChildNodeFullPaths(List childNodeFullPaths) { + this.childNodeFullPaths = childNodeFullPaths; + } + + public List getXmlNodes() { + return xmlNodes; + } + + public void setXmlNodes(List xmlNodes) { + this.xmlNodes = xmlNodes; + } + + // Given that fullPath has to be unique in the database, it's an acceptable field to use + // for hashCode() and equals(). + @Override + public int hashCode() { + return Objects.hash(fullPath); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + DatastoreNode other = (DatastoreNode) obj; + return Objects.equals(fullPath, other.fullPath); + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreNodeCrud.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreNodeCrud.java new file mode 100644 index 0000000..ca0a224 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreNodeCrud.java @@ -0,0 +1,24 @@ +package gov.nasa.ziggy.data.datastore; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import gov.nasa.ziggy.crud.AbstractCrud; + +public class DatastoreNodeCrud extends AbstractCrud { + + public Map retrieveNodesByFullPath() { + Map nodesByFullPath = new HashMap<>(); + List nodes = list(createZiggyQuery(DatastoreNode.class)); + for (DatastoreNode node : nodes) { + nodesByFullPath.put(node.getFullPath(), node); + } + return nodesByFullPath; + } + + @Override + public Class componentClass() { + return DatastoreNode.class; + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreRegexp.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreRegexp.java new file mode 100644 index 0000000..567f2b5 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreRegexp.java @@ -0,0 +1,125 @@ +package gov.nasa.ziggy.data.datastore; + +import java.util.Objects; + +import org.apache.commons.lang3.StringUtils; + +import jakarta.persistence.Entity; +import jakarta.persistence.Id; +import jakarta.persistence.Table; +import jakarta.xml.bind.annotation.XmlAccessType; +import jakarta.xml.bind.annotation.XmlAccessorType; +import jakarta.xml.bind.annotation.XmlAttribute; +import jakarta.xml.bind.annotation.XmlTransient; + +/** + * Models a datastore regular expression ("regexp"). + *

      + * A datastore regexp is an element that can be included in multiple datastore nodes. It provides + * multiple limits on what directory names it will match: + *

        + *
      1. At the top level, the {@link #value} field specifies a "must-meet" regular expression + * criterion. This allows the user to specify that, for example, datastore regexp foo will only + * accept values of "bar" or "baz". This regexp can only be changed by re-importing the datastore + * definitions (i.e., the user can't change it from the console). + *
      2. The user can also set additional include and exclude regexps. These apply additional + * constraints that can be changed as needed. Their purpose is to allow the datastore API to limit + * the directories that are used in a specified processing activity. For example, if the user wanted + * to only process foo directories named "bar", they could either set the include regexp to "bar" or + * the exclude regexp to "baz". + *
      + * + * @author PT + */ +@XmlAccessorType(XmlAccessType.NONE) +@Entity +@Table(name = "Ziggy_DatastoreRegexp") +public class DatastoreRegexp { + + @Id + @XmlAttribute(required = true) + private String name; + + @XmlAttribute(required = true) + private String value; + + @XmlTransient + private String include; + + @XmlTransient + private String exclude; + + public DatastoreRegexp() { + } + + DatastoreRegexp(String name, String value) { + this.name = name; + this.value = value; + } + + public boolean matches(String location) { + boolean matches = location.matches(value); + if (matches && !StringUtils.isBlank(include)) { + matches = matches && location.matches(include); + } + if (matches && !StringUtils.isBlank(exclude)) { + matches = matches && !location.matches(exclude); + } + return matches; + } + + public boolean matchesValue(String location) { + return location.matches(value); + } + + public String getName() { + return name; + } + + public String getValue() { + return value; + } + + /** + * Package scoped because only the {@link DatastoreConfigurationImporter} should be able to + * change this. + */ + void setValue(String value) { + this.value = value; + } + + public String getInclude() { + return include; + } + + public void setInclude(String include) { + this.include = include; + } + + public String getExclude() { + return exclude; + } + + public void setExclude(String exclude) { + this.exclude = exclude; + } + + // The hashCode() and equals() methods use only the name, which must be unique per the + // database uniqueness constraint. + @Override + public int hashCode() { + return Objects.hash(name); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + DatastoreRegexp other = (DatastoreRegexp) obj; + return Objects.equals(name, other.name); + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreRegexpCrud.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreRegexpCrud.java new file mode 100644 index 0000000..7476072 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreRegexpCrud.java @@ -0,0 +1,39 @@ +package gov.nasa.ziggy.data.datastore; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import gov.nasa.ziggy.crud.AbstractCrud; + +public class DatastoreRegexpCrud extends AbstractCrud { + + public DatastoreRegexp retrieve(String name) { + return uniqueResult( + createZiggyQuery(DatastoreRegexp.class).column(DatastoreRegexp_.name).in(name)); + } + + public List retrieveAll() { + return list(createZiggyQuery(DatastoreRegexp.class)); + } + + public Map retrieveRegexpsByName() { + Map regexpByName = new HashMap<>(); + List regexps = list(createZiggyQuery(DatastoreRegexp.class)); + for (DatastoreRegexp regexp : regexps) { + regexpByName.put(regexp.getName(), regexp); + } + return regexpByName; + } + + public List retrieveRegexpNames() { + return list( + createZiggyQuery(DatastoreRegexp.class, String.class).column(DatastoreRegexp_.name) + .select()); + } + + @Override + public Class componentClass() { + return DatastoreRegexp.class; + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java new file mode 100644 index 0000000..20c06da --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/datastore/DatastoreWalker.java @@ -0,0 +1,492 @@ +package gov.nasa.ziggy.data.datastore; + +import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.DirectoryStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.regex.Pattern; + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.services.config.ZiggyConfiguration; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; + +/** + * Provides recursive transiting of the datastore directory tree (as defined by + * {@link DatastoreNode} and {@link DatastoreRegexp} instances) to validate {@link DataFileType} + * locations, determine the locations of all datastore directories required for a task, etc. + * + * @author PT + */ +public class DatastoreWalker { + + private static final Logger log = LoggerFactory.getLogger(DatastoreWalker.class); + + // The nodes in a location are always separated by a slash regardless of what the + // OS uses for the directory separator character. + public static final String NODE_SEPARATOR = "/"; + private static final String REGEXP_VALUE_SEPARATOR = "\\$"; + + // NB This is how SpotBugs suggests to handle the file separator so that + // MacOS, Linux, and Windows all work correctly. + private static final String FILE_SEPARATOR = File.separatorChar == '\\' ? "\\\\" + : File.separator; + + private Map regexpsByName; + private Map datastoreNodesByFullPath; + private Path datastoreRootPath; + + public DatastoreWalker(Map regexpsByName, + Map datastoreNodesByFullPath) { + this.regexpsByName = regexpsByName; + this.datastoreNodesByFullPath = datastoreNodesByFullPath; + datastoreRootPath = Paths.get( + ZiggyConfiguration.getInstance().getString(PropertyName.DATASTORE_ROOT_DIR.property())); + } + + /** + * Creates a {@link DatastoreWalker} objects from the {@link DatastoreRegexp}s and + * {@link DatastoreNode}s in the database. + * + * @see DatastoreRegexpCrud#retrieveRegexpsByName() + * @see DatastoreNodeCrud#retrieveNodesByFullPath() + */ + public static DatastoreWalker newInstance() { + return (DatastoreWalker) DatabaseTransactionFactory.performTransaction(() -> { + Map regexpsByName = new DatastoreRegexpCrud() + .retrieveRegexpsByName(); + Map datastoreNodesByFullPath = new DatastoreNodeCrud() + .retrieveNodesByFullPath(); + return new DatastoreWalker(regexpsByName, datastoreNodesByFullPath); + }); + } + + /** + * Validates a location from a {@link DataFileType} instance. + *

      + * {@link #locationExists(String)} takes a location string of the form used in an instance of + * {@link DataFileType} and performs the following validations on it: + *

        + *
      1. The location exists. + *
      2. Any location element that includes a regexp component (i.e., .../cadenceType$ffi/...) has + * only one such component. + *
      3. Any location element that includes a regexp component is a location that is a reference + * to a {@link DatastoreRegexp} instance. + *
      4. Any location element that includes a regexp component has a valid regexp component. + *
      + */ + public boolean locationExists(String location) { + + String[] locationElements = location.split(NODE_SEPARATOR); + List locationsAndRegexpValues = new ArrayList<>(); + for (String locationElement : locationElements) { + locationsAndRegexpValues.add(new LocationAndRegexpValue(locationElement)); + } + + // If any of the locations had more than one $ in it, that's an instant fail. + List invalidLocations = new ArrayList<>(); + for (int locationIndex = 0; locationIndex < locationsAndRegexpValues + .size(); locationIndex++) { + if (locationsAndRegexpValues.get(locationIndex).getLocation() == null) { + invalidLocations + .add(locationElements[locationIndex].split(REGEXP_VALUE_SEPARATOR)[0]); + } + } + if (!invalidLocations.isEmpty()) { + log.error("Location elements with too many $ characters: {}", + invalidLocations.toString()); + return false; + } + + // Construct the full path of the location + String fullPath = fullPathFromLocations(locationsAndRegexpValues); + + // If the full path does not exist as a datastore node, that's a failure. + if (!datastoreNodesByFullPath.containsKey(fullPath)) { + log.error("Full path {} does not exist as a datastore node", fullPath); + return false; + } + + // Check that any regexp portions of any location elements are valid. + invalidLocations = invalidRegexpLocations(locationsAndRegexpValues); + if (!CollectionUtils.isEmpty(invalidLocations)) { + log.error("Invalid regexp locations and/or definitions: {}", + invalidLocations.toString()); + return false; + } + return true; + } + + private List invalidRegexpLocations( + List locationsAndRegexpValues) { + List invalidLocations = new ArrayList<>(); + StringBuilder incrementalPathBuilder = new StringBuilder(); + for (LocationAndRegexpValue locationAndRegexpValue : locationsAndRegexpValues) { + if (incrementalPathBuilder.length() > 0) { + incrementalPathBuilder.append(NODE_SEPARATOR); + } + String location = locationAndRegexpValue.getLocation(); + incrementalPathBuilder.append(location); + String incrementalFullPath = incrementalPathBuilder.toString(); + if (!datastoreNodesByFullPath.containsKey(incrementalFullPath)) { + invalidLocations.add(location); + continue; + } + if (!datastoreNodesByFullPath.get(incrementalFullPath).isRegexp()) { + continue; + } + DatastoreRegexp regexp = regexpsByName.get(locationAndRegexpValue.getLocation()); + if (regexp == null) { + invalidLocations.add(location); + continue; + } + String locationRegexp = locationAndRegexpValue.getRegexpValue(); + if (!regexp.matchesValue(locationRegexp) && !StringUtils.isEmpty(locationRegexp)) { + invalidLocations.add(location); + } + } + return invalidLocations; + } + + /** + * Determines whether a given location represents a potentially valid directory in the + * datastore. + *

      + * The {@link #locationMatchesDatastore(String)} takes as its argument a definite location in + * the datastore and determines whether, under the datastore layout as defined by the + * {@link DatastoreNode}s, that location would be valid. This is accomplished by breaking the + * location into its component elements; then, for each element, looking to see whether that + * element matches any nodes (an exact match is required for nodes that are not pointers to + * {@link DatastoreRegexp}s; for nodes that point to DatastoreRegexp instances, the directory + * location has to match the {@link Pattern} derived from the regexp's value field). If there is + * a match, the set of nodes that are tested against the next element are the child nodes of the + * matching node. If every element of the location matches a datastore node, + * {@link #locationMatchesDatastore(String)} returns true, otherwise false. + *

      + * Consider an example in which the top node of the datastore is a {@link DatastoreRegexp}, with + * name "sector" and value "sector-[0-9]{4}" (i.e, "sector" followed by a hyphen followed by a 4 + * digit number). The sector node has two child nodes, "mda" and "1sa", each of which is a + * non-regexp {@link DatastoreNode}. The following location arguments will return true: + * + * 

      +     * sector-0002/mda
+     * sector-1024/1sa
+     *
      + * + * The following location arguments will return false: + * + * 
      +     * sector-1/mda        (sector regexp is not matched.)
+     * mda                 (there is no mda datastore node at the top of the tree.)
+     * sector-0002/tbr     (there is no tbr node under the sector node.)
+     * sector-0002/mda/cal (there is no cal node under sector/mda.)
+     *
      + * + * The {@link #locationMatchesDatastore(String)} allows a user to determine whether a specific + * directory would violate the datastore layout, and thus allows the user to ensure that no + * datastore directories are ever created that violate that layout (any such directory would be + * unreachable by the datastore API). + */ + public boolean locationMatchesDatastore(String location) { + String[] locationElements = location.split(NODE_SEPARATOR); + List datastoreNodes = new ArrayList<>(datastoreNodesByFullPath.values()); + for (String locationElement : locationElements) { + + // If there are still location elements but no further nodes down this + // part of the tree, then the location is not a potentially valid one. + if (datastoreNodes.isEmpty()) { + return false; + } + DatastoreNode matchingNode = null; + + // Find a current node which matches this element of the location. + for (DatastoreNode node : datastoreNodes) { + if (node.isRegexp()) { + DatastoreRegexp regexp = regexpsByName.get(node.getName()); + if (regexp.matchesValue(locationElement)) { + matchingNode = node; + continue; + } + } else if (node.getName().equals(locationElement)) { + matchingNode = node; + } + } + + // If we didn't find a match, this is not a valid potential location. + if (matchingNode == null) { + return false; + } + + // Put the child nodes of the current node into the datastoreNodes collection. + datastoreNodes.clear(); + if (!CollectionUtils.isEmpty(matchingNode.getChildNodeFullPaths())) { + for (String fullPath : matchingNode.getChildNodeFullPaths()) { + datastoreNodes.add(datastoreNodesByFullPath.get(fullPath)); + } + } + } + return true; + } + + String fullPathFromLocations(List locationsAndRegexpValues) { + StringBuilder sb = new StringBuilder(); + for (LocationAndRegexpValue locationAndRegexpValue : locationsAndRegexpValues) { + sb.append(locationAndRegexpValue.getLocation()); + sb.append(NODE_SEPARATOR); + } + sb.setLength(sb.length() - NODE_SEPARATOR.length()); + return sb.toString(); + } + + /** + * Returns all the existing datastore directories that exist and that match a given datastore + * location. + *

      + * The {@link #pathsForLocation(String)} takes a datastore path and walks the directories below + * the datastore root directory. It locates all the directories that match the datastore + * location argument and returns them in a list. + */ + public List pathsForLocation(String location) { + if (!locationExists(location)) { + throw new IllegalArgumentException("Datastore location " + location + " not valid"); + } + + String[] locationElements = location.split(NODE_SEPARATOR); + List locationsAndRegexpValues = new ArrayList<>(); + for (String locationElement : locationElements) { + locationsAndRegexpValues.add(new LocationAndRegexpValue(locationElement)); + } + return pathsForLocation(locationsAndRegexpValues, 0, datastoreRootPath); + } + + /** + * Performs the iterative portion of searching for datastore directory paths. The iteration is + * over subdirectory levels. At each step, a check is performed to see if there are additional + * subdirectories below the current directory level. If so, + * {@link #pathsForLocation(List, int, Path)} is called for each of the subdirectories below the + * current level. Otherwise, the method returns, causing all the calls to + * {@link #pathsForLocation(List, int, Path)} to return. + */ + private List pathsForLocation(List locationsAndRegexpValues, + int locationIndex, Path parentPath) { + + // Where are we so far? + String fullPathFromLocations = fullPathFromLocations( + locationsAndRegexpValues.subList(0, locationIndex + 1)); + + DatastoreNode node = datastoreNodesByFullPath.get(fullPathFromLocations); + List pathsThisLevel = List.of(parentPath.resolve(node.getName())); + + // If this is a regexp, find all the directories at this level based on the + // regexp value in the location (if any) and the include / exclude regexps + if (node.isRegexp()) { + DatastoreRegexp regexp = regexpsByName.get(node.getName()); + pathsThisLevel = listPaths(parentPath, + locationsAndRegexpValues.get(locationIndex).getRegexpValue(), regexp); + } + + // If we've gone as far down this location as possible, we can return the paths + // found at this level. + if (locationIndex == locationsAndRegexpValues.size() - 1) { + return pathsThisLevel; + } + + // Otherwise, go down another level + List pathsNextLevel = new ArrayList<>(); + for (Path path : pathsThisLevel) { + pathsNextLevel + .addAll(pathsForLocation(locationsAndRegexpValues, locationIndex + 1, path)); + } + return pathsNextLevel; + } + + /** + * Finds the paths of subdirectories of the current parent path that match a given + * {@link DatastoreRegexp}. This takes into account the value, include, and exclude fields of + * the DatastoreRegexp. + */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private List listPaths(Path parentPath, String regexpValueThisLevel, + DatastoreRegexp regexp) { + List pathsThisLevel = new ArrayList<>(); + DirectoryStream.Filter dirFilter = path -> regexp + .matches(path.getFileName().toString()) + && (StringUtils.isEmpty(regexpValueThisLevel) + || path.getFileName().toString().matches(regexpValueThisLevel)); + try (DirectoryStream dirStream = Files.newDirectoryStream(parentPath, dirFilter)) { + for (Path entry : dirStream) { + pathsThisLevel.add(entry.toAbsolutePath()); + } + } catch (IOException e) { + throw new UncheckedIOException(e); + } + return pathsThisLevel; + } + + /** + * Returns the indices of {@link Path} name elements that vary across a collection of path + * instances. These are used to construct brief states that contain only name elements that + * change from one UOW to the next (i.e., if the paths are "foo/bar/baz" and "foo/baz/bar", we + * only need the "baz" and "bar" elements in the brief state; the "foo" is common to all units + * of work so it doesn't tell us anything). + */ + public List pathElementIndicesForBriefState(List datastorePaths) { + + // Handle the special case of only one Path in the list. + List pathElementIndicesForBriefState = new ArrayList<>(); + if (datastorePaths.size() == 1) { + return pathElementIndicesForBriefState; + } + + // Determine which parts vary across the collection of datastore paths. A set in the "parts" + // list will only have one item if the part is common across all paths. + int nameCount = datastorePaths.get(0).getNameCount(); + List> parts = new ArrayList<>(); + for (int i = 0; i < nameCount; i++) { + parts.add(new HashSet<>()); + } + for (Path path : datastorePaths) { + if (nameCount != path.getNameCount()) { + throw new IllegalArgumentException(path.toString() + " has " + path.getNameCount() + + " elements, but " + nameCount + " was expected"); + } + for (int partIndex = 0; partIndex < path.getNameCount(); partIndex++) { + parts.get(partIndex).add(path.getName(partIndex).getFileName()); + } + } + + // Find the indices of the path parts that have > 1 item (i.e., the ones that vary). + for (int partIndex = 0; partIndex < nameCount; partIndex++) { + if (parts.get(partIndex).size() > 1) { + pathElementIndicesForBriefState.add(partIndex); + } + } + return pathElementIndicesForBriefState; + } + + /** + * Constructs a {@link Map} of regexp values by regexp name. This requires a location string + * (which can be used to determine which parts of the path are regular expressions and which are + * single-valued nodes), and a path (so that the parts of the path that are now known to be + * regexps can be captured and used to populate the Map). + */ + public Map regexpValues(String location, Path path) { + return regexpValues(location, path, true); + } + + /** + * Constructs a {@link Map} of regexp values by regexp name. This requires a location string + * (which can be used to determine which parts of the path are regular expressions and which are + * single-valued nodes), and a path (so that the parts of the path that are now known to be + * regexps can be captured and used to populate the Map). The caller has the option to suppress + * regexp values that are specified in the location string (i.e., "foo$bar"), or populate same. + */ + public Map regexpValues(String location, Path path, + boolean includeValuesFromLocation) { + Map regexpValues = new LinkedHashMap<>(); + Map regexpValuesInLocation = new LinkedHashMap<>(); + String[] pathParts = null; + if (path != null) { + if (path.isAbsolute()) { + path = datastoreRootPath.toAbsolutePath().relativize(path); + } + pathParts = path.toString().split(FILE_SEPARATOR); + } + + String[] locationParts = location.split(NODE_SEPARATOR); + String cumulativePath = ""; + for (int i = 0; i < locationParts.length; i++) { + if (cumulativePath.length() > 0) { + cumulativePath = cumulativePath + NODE_SEPARATOR; + } + String[] locationSubparts = locationParts[i].split(REGEXP_VALUE_SEPARATOR); + String truncatedLocation = locationSubparts[0]; + cumulativePath = cumulativePath + truncatedLocation; + if (datastoreNodesByFullPath.get(cumulativePath).isRegexp()) { + if (locationSubparts.length == 2 && !includeValuesFromLocation) { + continue; + } + if (pathParts != null) { + regexpValues.put(truncatedLocation, pathParts[i]); + } + if (locationSubparts.length == 2 && includeValuesFromLocation) { + regexpValuesInLocation.put(truncatedLocation, locationSubparts[1]); + } + } + } + return path != null ? regexpValues : regexpValuesInLocation; + } + + /** + * Constructs a {@link Path} for a location in which the regular expressions have been replaced + * by specific values. + */ + public Path pathFromLocationAndRegexpValues(Map regexpValues, String location) { + Path path = datastoreRootPath.toAbsolutePath(); + String[] locationParts = location.split(NODE_SEPARATOR); + for (String locationPart : locationParts) { + String[] locationAndValue = locationPart.split(REGEXP_VALUE_SEPARATOR); + String trimmedLocation = locationAndValue[0]; + if (regexpValues.get(trimmedLocation) != null) { + + // NB: if the location has an associated value + // for the current part (i.e., "foo$bar"), + // and there is also a regexp value for that part + // (i.e., regexpValues.get("foo") == "baz"), the value + // from the location takes precedence. + String regexpValue = locationAndValue.length == 1 + ? regexpValues.get(trimmedLocation) + : locationAndValue[1]; + path = path.resolve(regexpValue); + } else { + path = locationAndValue.length == 1 ? path.resolve(trimmedLocation) + : path.resolve(locationAndValue[1]); + } + } + return path; + } + + Map regexpsByName() { + return regexpsByName; + } + + private static class LocationAndRegexpValue { + + private final String location; + private final String regexpValue; + + public LocationAndRegexpValue(String locationWithOptionalRegexp) { + String[] locationComponents = locationWithOptionalRegexp.split("\\$"); + if (locationComponents.length > 2) { + location = null; + regexpValue = null; + return; + } + location = locationComponents[0]; + regexpValue = locationComponents.length == 2 ? locationComponents[1] : ""; + } + + public String getLocation() { + return location; + } + + public String getRegexpValue() { + return regexpValue; + } + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java b/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java index 4f9508f..b6fbbd3 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java +++ b/src/main/java/gov/nasa/ziggy/data/management/Acknowledgement.java @@ -29,6 +29,7 @@ import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.ZiggyShutdownHook; +import gov.nasa.ziggy.util.io.FileUtil; import jakarta.xml.bind.annotation.XmlAccessType; import jakarta.xml.bind.annotation.XmlAccessorType; import jakarta.xml.bind.annotation.XmlAttribute; @@ -50,7 +51,7 @@ public class Acknowledgement implements HasXmlSchemaFilename { private static final String SCHEMA_FILENAME = "manifest-ack.xsd"; - static final String FILENAME_SUFFIX = "-ack"; + static final String FILENAME_SUFFIX = "-ack.xml"; private static final double DEFAULT_MAX_FAILURE_PERCENTAGE = 100; @@ -124,7 +125,7 @@ public static String nameFromManifestName(Manifest manifest) { String manifestFileType = FilenameUtils.getExtension(manifestName); int manifestTypeLength = manifestFileType.length() + 1; String baseName = manifestName.substring(0, manifestName.length() - manifestTypeLength); - return baseName + FILENAME_SUFFIX + "." + manifestFileType; + return baseName + FILENAME_SUFFIX; } /** @@ -335,7 +336,7 @@ public static AcknowledgementEntry of(ManifestEntry manifestEntry, Path dir, // Start by making sure the file exists and is a regular file (or symlink to same) if (Files.exists(file) || Files.isSymbolicLink(file)) { - realFile = DataFileManager.realSourceFile(file); + realFile = FileUtil.realSourceFile(file); ackEntry.setTransferStatus(DataReceiptStatus.PRESENT); } if (ackEntry.getTransferStatus() == DataReceiptStatus.ABSENT) { diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataFileInfo.java b/src/main/java/gov/nasa/ziggy/data/management/DataFileInfo.java deleted file mode 100644 index dc7c6fc..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DataFileInfo.java +++ /dev/null @@ -1,93 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static com.google.common.base.Preconditions.checkArgument; -import static com.google.common.base.Preconditions.checkNotNull; -import static com.google.common.base.Preconditions.checkState; - -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.regex.Pattern; - -/** - * A data file that is transferred between the datastore and task directory. Subclasses must - * implement the {@link #getPattern()} method to identify files in the datastore, as well as the - * constructors, if only to call super(). - *

      - * N.B. This class is intended to replace DatastoreId. If it doesn't, it should be deleted. - * - * @author PT - * @author Bill Wohler - */ -public abstract class DataFileInfo implements Comparable { - - private Path name; - - /** - * Creates an empty DataFileInfo object. The only thing that can be done with this object is to - * call {@link #pathValid(Path)} on it. - */ - protected DataFileInfo() { - } - - /** - * Creates a DataFileInfo object with the given name - * - * @throws NullPointerException if name is null - * @throws IllegalArgumentException if the name doesn't match the pattern defined by this class - */ - protected DataFileInfo(String name) { - this.name = Paths.get(checkNotNull(name, "name")); - checkArgument(pathValid(this.name), - "Data file " + name + " does not match required pattern"); - } - - /** - * Creates a DataFileInfo object with the given name - * - * @throws NullPointerException if name is null - * @throws IllegalArgumentException if the name doesn't match the pattern defined by this class - */ - protected DataFileInfo(Path name) { - checkArgument(pathValid(checkNotNull(name, "name")), - "File " + name.toString() + " does not match required pattern"); - this.name = checkNotNull(name, "name"); - } - - /** - * Returns the pattern that all file names stored within the given DataFileInfo subclass must - * match. - */ - protected abstract Pattern getPattern(); - - /** - * Returns true if the given path matches the pattern defined by this class. - * - * @return false if the given path is null or doesn't match the pattern - */ - public boolean pathValid(Path path) { - return path == null ? false : getPattern().matcher(path.getFileName().toString()).matches(); - } - - /** - * Returns the non-null relative path to the data file. - * - * @throws IllegalStateException if the default constructor was used - */ - public Path getName() { - checkState(name != null, "Default constructor was used"); - return name; - } - - /** - * Implements comparison of DataFileInfo instances by alphabetizing of their names. - */ - @Override - public int compareTo(DataFileInfo other) { - return name.toString().compareTo(other.getName().toString()); - } - - @Override - public String toString() { - return name.toString(); - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataFileManager.java b/src/main/java/gov/nasa/ziggy/data/management/DataFileManager.java deleted file mode 100644 index 90e8f8b..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DataFileManager.java +++ /dev/null @@ -1,1062 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static com.google.common.base.Preconditions.checkArgument; -import static com.google.common.base.Preconditions.checkNotNull; - -import java.io.File; -import java.io.IOException; -import java.io.UncheckedIOException; -import java.lang.reflect.Constructor; -import java.lang.reflect.InvocationTargetException; -import java.nio.file.Files; -import java.nio.file.LinkOption; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.nio.file.StandardCopyOption; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeSet; -import java.util.function.Predicate; -import java.util.regex.Pattern; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -import org.apache.commons.io.FileUtils; -import org.apache.commons.lang3.ArrayUtils; -import org.slf4j.Logger; - -import gov.nasa.ziggy.data.management.DataFileType.RegexType; -import gov.nasa.ziggy.models.ModelImporter; -import gov.nasa.ziggy.module.AlgorithmStateFiles; -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.module.TaskConfigurationManager; -import gov.nasa.ziggy.pipeline.definition.ModelMetadata; -import gov.nasa.ziggy.pipeline.definition.ModelRegistry; -import gov.nasa.ziggy.pipeline.definition.ModelType; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; -import gov.nasa.ziggy.services.config.DirectoryProperties; -import gov.nasa.ziggy.services.config.PropertyName; -import gov.nasa.ziggy.services.config.ZiggyConfiguration; -import gov.nasa.ziggy.uow.TaskConfigurationParameters; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.io.FileUtil; - -/** - * Provides functions that identify data files based on the subclass of DataFileInfo they correspond - * to, and tools to copy such files between the datastore and a task directory; to move files from a - * task directory to the datastore; and to delete unneeded data files from a task directory. - *

      - * The class can be used in one of two ways. - *

      - * The approach that involves less code development is to use DataFileType instances to define the - * names and datastore locations of various types of data file. This allows all of the DataFileType - * information to be specified in XML files that configure the pipeline. The main disadvantage of - * this approach is that it is less flexible in terms of defining the organization of the datastore, - * optionally moving just some files in a given data file type (rather than all of them), etc. - *

      - * In the event that a greater degree of flexibility is desired, the DataFileManager can use - * DataFileInfo classes and a DatastoreFileLocator instance to manage the file names, paths to the - * datastore, etc. This requires additional code in the form of the DataFileInfo classes and the - * DatastoreFileLocator instance. - * - * @author PT - */ -public class DataFileManager { - - private static final Predicate WITH_RESULTS = AlgorithmStateFiles::hasResults; - private static final Predicate WITHOUT_RESULTS = WITH_RESULTS.negate(); - - private DatastorePathLocator datastorePathLocator; - private PipelineTask pipelineTask; - private TaskConfigurationParameters taskConfigurationParameters; - private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud; - private PipelineTaskCrud pipelineTaskCrud; - private Path taskDirectory; - private Path datastoreRoot = DirectoryProperties.datastoreRootDir(); - private DatastoreCopyType taskDirCopyType; - private DatastoreCopyType datastoreCopyType = DatastoreCopyType.MOVE; - - // ========================================================================= - // - // Constructors - // - // ========================================================================= - - public Path getDatastoreRoot() { - return datastoreRoot; - } - - /** - * No-arg constructor. Used in the PipelineInputs and PipelineOutputs classes. - */ - public DataFileManager() { - } - - /** - * Constructor with PipelineTask and DatastorePathLocator arguments. Used in pipeline modules - * that use the DataFileInfo and DatastoreFileLocator classes to identify and manage files that - * need to be moved between the task directory and the datastore. - * - * @param datastorePathLocator instance of a DatastorePathLocator subclass that is sufficient to - * provide datastore paths for all subclasses of DataFileInfo used by the pipeline module that - * instantiates the DataFileManager instance. - * @param pipelineTask PipelineTask supported by this instance. - * @param taskDirectory Path to the task directory. - */ - public DataFileManager(DatastorePathLocator datastorePathLocator, PipelineTask pipelineTask, - Path taskDirectory) { - this.pipelineTask = pipelineTask; - this.datastorePathLocator = datastorePathLocator; - this.taskDirectory = taskDirectory; - if (pipelineTask != null) { - taskConfigurationParameters = pipelineTask - .getParameters(TaskConfigurationParameters.class, false); - } - taskDirCopyType = taskDirCopyType(); - } - - /** - * Constructor with PipelineTask and Paths to the task directory and the datastore root. Used in - * pipeline modules that use the UnitOfWorkGenerator.defaultUnitOfWorkGenerator() and - * DataFileType instances to identify and manage files that need to be moved between the task - * directory and the datastore. - */ - public DataFileManager(Path datastoreRoot, Path taskDirectory, PipelineTask pipelineTask) { - this.pipelineTask = pipelineTask; - this.taskDirectory = taskDirectory; - if (datastoreRoot != null) { - this.datastoreRoot = datastoreRoot; - } - if (pipelineTask != null) { - taskConfigurationParameters = pipelineTask - .getParameters(TaskConfigurationParameters.class, false); - } - datastorePathLocator = null; - taskDirCopyType = taskDirCopyType(); - } - - private DatastoreCopyType taskDirCopyType() { - DatastoreCopyType copyType = null; - boolean useSymlinks = ZiggyConfiguration.getInstance() - .getBoolean(PropertyName.USE_SYMLINKS.property(), false); - if (useSymlinks) { - copyType = DatastoreCopyType.SYMLINK; - } else { - copyType = DatastoreCopyType.COPY; - } - return copyType; - } - - // ========================================================================= - // - // Public methods for use with DataFileType instances - // - // ========================================================================= - - /** - * Obtains a Map from DataFileType instances to files of each type in the task directory. - * - * @param dataFileTypes Set of DataFileType instances to be matched - */ - public Map> taskDirectoryDataFilesMap(Set dataFileTypes) { - - return dataFilesMap(taskDirectory, dataFileTypes, RegexType.TASK_DIR); - } - - /** - * Obtains a Map from DataFileType instances to files of each type in a sub-directory of the - * datastore. - * - * @param datastoreSubDir subdirectory of the datastore to be searched. - * @param dataFileTypes Set of DataFileType instances to be matched. - */ - public Map> datastoreDataFilesMap(Path datastoreSubDir, - Set dataFileTypes) { - return dataFilesMap(datastoreRoot.resolve(datastoreSubDir), dataFileTypes, - RegexType.DATASTORE); - } - - /** - * Determines the number of files of a given type in a given subdirectory of the datastore. Used - * for counting subtasks. - */ - public int countDatastoreFilesOfType(DataFileType type, Path datastoreSubDir) { - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(type); - Map> datastoreFiles = datastoreDataFilesMap(datastoreSubDir, - dataFileTypes); - return datastoreFiles.get(type).size(); - } - - /** - * Copies data files from the datastore to the task directory. - * - * @param datastoreSubDir subdirectory of datastore to use as the file source. - * @param dataFileTypes Set of DataFileType instances to use in the copy. - */ - public Map> copyDataFilesByTypeToTaskDirectory(Path datastoreSubDir, - Set dataFileTypes) { - return copyDataFilesByTypeToTaskDirectory( - datastoreDataFilesMap(datastoreSubDir, dataFileTypes)); - } - - /** - * Copies data files from the datastore to the task directory. - * - * @param datastoreDataFilesMap files to be copied, in the form of a {@link Map} that uses - * {@link DataFileType} as its key and a {@link Set} of data file {@link Path} instances as the - * map values - */ - public Map> copyDataFilesByTypeToTaskDirectory( - Map> datastoreDataFilesMap) { - - Map> datastoreFilesMap = copyDataFilesByTypeToDestination( - datastoreDataFilesMap, RegexType.TASK_DIR, taskDirCopyType); - Set datastoreFiles = new HashSet<>(); - for (Set paths : datastoreFilesMap.values()) { - datastoreFiles.addAll(paths); - } - - // Obtain the originators for all datastore files and replace them as producers to the - // current pipeline task; in the event of a reprocess the correct information is reflected. - pipelineTask - .setProducerTaskIds(datastoreProducerConsumerCrud().retrieveProducers(datastoreFiles)); - - return datastoreFilesMap; - } - - /** - * Identifies the files in the datastore that will be used as inputs for the current task. - * - * @param datastoreSubDir subdirectory of datastore to use as the file source - * @param dataFileTypes set of DataFileType instances to use for the search - * @return non @code{null} set of {@link Path} instances for data files to be used as input - */ - public Set dataFilesForInputs(Path datastoreSubDir, Set dataFileTypes) { - Map> datastoreFilesMap = datastoreDataFilesMap(datastoreSubDir, - dataFileTypes); - Set datastoreFiles = new HashSet<>(); - for (Set paths : datastoreFilesMap.values()) { - datastoreFiles.addAll(paths); - } - return datastoreFiles; - } - - /** - * Copies data files from the working directory to the task directory. This uses the copy type - * that is appropriate for the task directory, so it can either copy files or produce symlinks - * of files. - */ - public void copyDataFilesByTypeFromWorkingDirToTaskDir(Set dataFileTypes) { - Path workingDirectory = DirectoryProperties.workingDir(); - Map> sourceDataFiles = dataFilesMap(workingDirectory, dataFileTypes, - RegexType.TASK_DIR); - for (DataFileType dataFileType : sourceDataFiles.keySet()) { - for (Path sourceFile : sourceDataFiles.get(dataFileType)) { - taskDirCopyType.copy(workingDirectory.resolve(sourceFile), - taskDirectory.resolve(sourceFile)); - } - } - } - - /** - * Determines whether the working directory has any files of the specified data types. - */ - public boolean workingDirHasFilesOfTypes(Set dataFileTypes) { - Path workingDirectory = DirectoryProperties.workingDir(); - Map> sourceDataFiles = dataFilesMap(workingDirectory, dataFileTypes, - RegexType.TASK_DIR); - return sourceDataFiles.values().stream().anyMatch(s -> !s.isEmpty()); - } - - /** - * Copies model files from the datastore to the task directory using the selected copy mode - * (true copies or symlinks). Returns the names of all files copied to the task directory. - */ - public List copyModelFilesToTaskDirectory(ModelRegistry modelRegistry, - Set modelTypes, Logger log) { - List modelFilesCopied = new ArrayList<>(); - Map models = modelRegistry.getModels(); - for (ModelType modelType : modelTypes) { - ModelMetadata modelMetadata = models.get(modelType); - if (modelMetadata == null) { - throw new PipelineException( - "Model " + modelType.getType() + " has no metadata entry"); - } - if (modelMetadata.getDatastoreFileName() == null) { - throw new PipelineException( - "Model " + modelType.getType() + " has no datastore filename"); - } - Path datastoreModelFile = datastoreRoot - .resolve(Paths.get(ModelImporter.DATASTORE_MODELS_SUBDIR_NAME, modelType.getType(), - modelMetadata.getDatastoreFileName())); - Path taskDirectoryModelFile = taskDirectory - .resolve(modelMetadata.getOriginalFileName()); - log.info("Copying file " + datastoreModelFile.getFileName().toString() - + " to task directory"); - taskDirCopyType.copy(datastoreModelFile, taskDirectoryModelFile); - modelFilesCopied.add(taskDirectoryModelFile.getFileName().toString()); - } - return modelFilesCopied; - } - - /** - * Copies files by name from the task directory to the working directory. This uses the copy - * type that is appropriate for the task directory, so it can either copy files or produce - * symlinks of files. - */ - public void copyFilesByNameFromTaskDirToWorkingDir(Collection filenames) { - Path workingDirectory = DirectoryProperties.workingDir(); - for (String filename : filenames) { - taskDirCopyType.copy(taskDirectory.resolve(filename), - workingDirectory.resolve(filename)); - } - } - - /** - * Deletes data files from the task directory given a Set of DataFileType instances. All data - * files that belong to the specified types will be deleted. - * - * @param dataFileTypes Set of DataFileType instances to use in deletion. - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void deleteDataFilesByTypeFromTaskDirectory(Set dataFileTypes) { - - Map> dataFileTypesMap = taskDirectoryDataFilesMap(dataFileTypes); - for (DataFileType dataFileType : dataFileTypesMap.keySet()) { - for (Path dataFilePath : dataFileTypesMap.get(dataFileType)) { - Path fullPath = taskDirectory.resolve(dataFilePath); - try { - if (Files.isSymbolicLink(fullPath)) { - Files.delete(fullPath); - } else if (Files.isRegularFile(fullPath)) { - Files.deleteIfExists(fullPath); - } else { - FileUtils.deleteDirectory(fullPath.toFile()); - } - } catch (IOException e) { - throw new UncheckedIOException("IO Exception for file " + fullPath.toString(), - e); - } - } - } - } - - /** - * Moves data files from the task directory to the datastore given a set of DataFileType - * instances. All data files that belong to the specified types will be moved. - * - * @param dataFileTypes Set of DataFileType instances to use in the move. - */ - public void moveDataFilesByTypeToDatastore(Set dataFileTypes) { - Map> datastoreFilesMap = copyDataFilesByTypeToDestination( - taskDirectoryDataFilesMap(dataFileTypes), RegexType.DATASTORE, datastoreCopyType); - - Set datastoreFiles = new HashSet<>(); - for (Set paths : datastoreFilesMap.values()) { - datastoreFiles.addAll(paths); - } - // Record the originator in the data accountability table in the database - datastoreProducerConsumerCrud().createOrUpdateProducer(pipelineTask, datastoreFiles, - DatastoreProducerConsumer.DataReceiptFileType.DATA); - } - - private Set datastoreFilesInCompletedSubtasks(Set dataFileTypes, - List subtaskDirectories) { - - // Get the data files of the assorted types from all the successful subtask directories - Map> dataFilesMap = dataFilesMap(subtaskDirectories, dataFileTypes, - RegexType.TASK_DIR); - - // Convert the file names from task directory format to datastore format - Set inputFilesDatastoreFormatted = new HashSet<>(); - for (Map.Entry> entry : dataFilesMap.entrySet()) { - DataFileType type = entry.getKey(); - inputFilesDatastoreFormatted.addAll(entry.getValue() - .stream() - .map(s -> s.getFileName().toString()) - .map(s -> type.datastoreFileNameFromTaskDirFileName(s)) - .collect(Collectors.toSet())); - } - return inputFilesDatastoreFormatted; - } - - /** - * Identifies the completed subtasks within a task that produced results, and gets the names of - * all files used by those subtasks based on a set of {@link DataFileType} instances. The file - * names are returned. - */ - public Set datastoreFilesInCompletedSubtasksWithResults( - Set dataFileTypes) { - return datastoreFilesInCompletedSubtasks(dataFileTypes, - completedSubtaskDirectoriesWithResults()); - } - - /** - * Identifies the completed subtasks within a task that failed to produce results, and gets the - * names of all files used by those subtasks based on a set of {@link DataFileType} instances. - * The file names are returned. - */ - public Set datastoreFilesInCompletedSubtasksWithoutResults( - Set dataFileTypes) { - return datastoreFilesInCompletedSubtasks(dataFileTypes, - completedSubtaskDirectoriesWithoutResults()); - } - - // ========================================================================= - // - // Public methods for use with DataFileInfo classes - // - // ========================================================================= - - /** - * Identifies all the files in a given directory that belong to any of a set of DataFileInfo - * subclasses and them as a single Set of DataFileInfo subclass instances. - * - * @param dataFileInfoClasses Set of DataFileInfo classes that are to be matched. - * @param dir Path to directory that will be searched. - * @return Set of DataFileInfo subclass instances that correspond to all the files in the - * specified directory that can be matched by any of the DataFileInfo subclasses. - */ - public Set datastoreFiles(Path dir, - Set> dataFileInfoClasses) { - Set dataFiles = new TreeSet<>(); - Map, Set> dataFilesMap = dataFilesMap( - dir, dataFileInfoClasses); - for (Set s : dataFilesMap.values()) { - dataFiles.addAll(s); - } - return dataFiles; - } - - /** - * Obtains a map from DataFileInfo subclasses to objects in each of the subclasses, where the - * objects are generated from the files in a specified directory. - */ - public Map, Set> dataFilesMap(Path dir, - Set> dataFileInfoClasses) { - return dataFilesMap(new ArrayList<>(Arrays.asList(dir)), dataFileInfoClasses); - } - - /** - * Obtains a map from DataFileInfo subclasses to objects in each of the subclasses, where the - * objects are generated from the files in a specified directory. The search for the objects - * runs through a collection of directories. - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public Map, Set> dataFilesMap( - Collection dirs, Set> dataFileInfoClasses) { - - Map, Set> datastoreMap = new HashMap<>(); - for (Path dir : dirs) { - checkArgument(Files.isDirectory(dir), "File " + dir.toString() + " is not a directory"); - - Set filesSet; - try { - filesSet = Files.list(dir).collect(Collectors.toCollection(TreeSet::new)); - } catch (IOException e) { - throw new UncheckedIOException("Unable to list files in dir " + dir.toString(), e); - } - - for (Class clazz : dataFileInfoClasses) { - Set dataFilesOfClass = dataFilesOfClass(clazz, filesSet); - if (datastoreMap.containsKey(clazz)) { - @SuppressWarnings("unchecked") - Set existingFilesOfClass = (Set) datastoreMap - .get(clazz); - existingFilesOfClass.addAll(dataFilesOfClass); - } else { - datastoreMap.put(clazz, dataFilesOfClass); - } - } - } - return datastoreMap; - } - - /** - * Copies a set of files from the datastore to the task directory. The originators of the files - * are retrieved from the database; the pipeline task's existing set of producers is deleted and - * replaced with the originators for the copied files. - * - * @param dataFiles {@link Set} of instances of {@link DataFileInfo} subclasses representing the - * files to be copied. - */ - public void copyToTaskDirectory(Set dataFiles) { - Map dataFileInfoToPath = findInputFiles(dataFiles); - for (DataFileInfo dataFileInfo : dataFileInfoToPath.keySet()) { - taskDirCopyType.copy(dataFileInfoToPath.get(dataFileInfo), - taskDirectory.resolve(dataFileInfo.getName())); - } - - // Obtain the originators for all datastore files and replace them as producers to the - // current pipeline task; in the event of a reprocess the correct information is reflected. - pipelineTask.setProducerTaskIds(datastoreProducerConsumerCrud() - .retrieveProducers(new HashSet<>(dataFileInfoToPath.values()))); - } - - /** - * Finds the set of files from the datastore that are needed as inputs for this task. - * - * @param dataFiles {@link Set} of instances of {@link DataFileInfo} subclasses representing the - * files to be used as inputs - * @return {@link Set} of {@link Path} instances for the files represented by the - * {@link DataFileInfo} instances. - */ - public Set dataFilesForInputs(Set dataFiles) { - Set datastoreFiles = new HashSet<>(); - Map dataFileInfoToPath = findInputFiles(dataFiles); - datastoreFiles.addAll(dataFileInfoToPath.values()); - return datastoreFiles; - } - - private Map findInputFiles(Set dataFiles) { - Map dataFileInfoToPath = new HashMap<>(); - for (DataFileInfo dataFileInfo : dataFiles) { - dataFileInfoToPath.put(dataFileInfo, datastorePathLocator.datastorePath(dataFileInfo)); - } - return dataFileInfoToPath; - } - - /** - * Deletes a set of files from the task directory. - * - * @param dataFiles Set of instances of DataFileInfo subclasses that represent the files to be - * deleted. - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void deleteFromTaskDirectory(Set dataFiles) { - for (DataFileInfo dataFileInfo : dataFiles) { - Path taskDirLocation = taskDirectory.resolve(dataFileInfo.getName()); - try { - if (Files.isRegularFile(taskDirLocation) || Files.isSymbolicLink(taskDirLocation)) { - Files.deleteIfExists(taskDirLocation); - } else { - FileUtils.deleteDirectory(taskDirLocation.toFile()); - } - } catch (IOException e) { - throw new UncheckedIOException( - "IOException occurred on file " + taskDirLocation.toString(), e); - } - } - } - - /** - * Moves a set of files from the task directory to the datastore. - * - * @param dataFiles Set of instances of DataFileInfo subclasses that represent the files to be - * moved. - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void moveToDatastore(Set dataFiles) { - Set datastoreFiles = new HashSet<>(); - for (DataFileInfo dataFileInfo : dataFiles) { - Path taskDirLocation = taskDirectory.resolve(dataFileInfo.getName()); - Path datastoreLocation = datastorePathLocator.datastorePath(dataFileInfo); - Path datastoreLocationParent = datastoreLocation.getParent(); - if (datastoreLocationParent == null) { - throw new PipelineException("Unable to obtain parent of datastore location"); - } - try { - Files.createDirectories(datastoreLocationParent); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to create directory " + datastoreLocationParent.toString(), e); - } - datastoreCopyType.copy(taskDirLocation, datastoreLocation); - datastoreFiles.add(datastoreRoot.relativize(datastoreLocation)); - } - - // Record the originator in the data accountability table in the database - datastoreProducerConsumerCrud().createOrUpdateProducer(pipelineTask, datastoreFiles, - DatastoreProducerConsumer.DataReceiptFileType.DATA); - } - - private Set filesInCompletedSubtasks(Set> dataFileTypes, - List completedSubtaskDirectories) { - - // Get the data files of the assorted types from all the successful subtask directories - Map, Set> dataFilesMap = dataFilesMap( - completedSubtaskDirectories, dataFileTypes); - - // Convert the file names from task directory format to datastore format - Set inputFiles = new HashSet<>(); - for (Set dataFileInfoSet : dataFilesMap.values()) { - inputFiles.addAll(dataFileInfoSet.stream() - .map(s -> datastorePathLocator.datastorePath(s).toString()) - .collect(Collectors.toSet())); - } - return inputFiles; - } - - /** - * Identifies the completed subtasks within a task that produced results and gets the names of - * all files used by those subtasks based on a set of {@link DataFileInfo} instances. The file - * names are returned. - */ - public Set filesInCompletedSubtasksWithResults( - Set> dataFileTypes) { - return filesInCompletedSubtasks(dataFileTypes, completedSubtaskDirectoriesWithResults()); - } - - /** - * Identifies the completed subtasks within a task that failed to produce results and gets the - * names of all files used by those subtasks based on a set of {@link DataFileInfo} instances. - * The file names are returned. - */ - public Set filesInCompletedSubtasksWithoutResults( - Set> dataFileTypes) { - return filesInCompletedSubtasks(dataFileTypes, completedSubtaskDirectoriesWithoutResults()); - } - - // ========================================================================= - // - // Private and package-private methods that perform services for the public methods - // - // ========================================================================= - - /** - * Returns the DatastoreProducerConsumerCrud instance, constructing if necessary. Public to - * allow mocking in unit tests. - */ - public DatastoreProducerConsumerCrud datastoreProducerConsumerCrud() { - if (datastoreProducerConsumerCrud == null) { - datastoreProducerConsumerCrud = new DatastoreProducerConsumerCrud(); - } - return datastoreProducerConsumerCrud; - } - - /** - * Returns the PipelineTaskCrud instance, constructing if necessary. Public to allow mocking in - * unit tests. - */ - public PipelineTaskCrud pipelineTaskCrud() { - if (pipelineTaskCrud == null) { - pipelineTaskCrud = new PipelineTaskCrud(); - } - return pipelineTaskCrud; - } - - /** - * Helper function that performs the general process of searching a directory for files that - * match DataFileType regular expressions, and returns a Map from the DataFileType instances to - * the identified files. - * - * @param directory Directory to be searched. - * @param dataFileTypes Set of DataFileType instances to search for. - * @param regexType Regex to use (datastore or task dir) - */ - private Map> dataFilesMap(Path directory, - Set dataFileTypes, DataFileType.RegexType regexType) { - return dataFilesMap(new ArrayList<>(Arrays.asList(directory)), dataFileTypes, regexType); - } - - /** - * Helper function that performs the general process of searching a collection of directories - * for files that match DataFileType regular expressions, and returns a Map from the - * DataFileType instances to the identified files. - * - * @param directories Collection of directories to be searched. - * @param dataFileTypes Set of DataFileType instances to search for. - * @param regexType Regex to use (datastore or task dir) - */ - private Map> dataFilesMap(Collection directories, - Set dataFileTypes, DataFileType.RegexType regexType) { - - Map> dataFilesMap = new HashMap<>(); - Set filesSet; - Stream pathStream = null; - Path pathToRelativize = null; - for (Path directory : directories) { - try { - pathStream = regexType.pathStream(directory); - pathToRelativize = regexType.pathToRelativize(directory, datastoreRoot); - final Path finalPathToRelativize = pathToRelativize; - filesSet = pathStream.map(s -> finalPathToRelativize.relativize(s)) - .collect(Collectors.toCollection(TreeSet::new)); - } finally { - if (pathStream != null) { - pathStream.close(); - } - } - - for (DataFileType dataFileType : dataFileTypes) { - Pattern pattern = regexType.getPattern(dataFileType); - Set filesOfType = new TreeSet<>(); - for (Path filePath : filesSet) { - if (pattern.matcher(filePath.toString()).matches()) { - filesOfType.add(filePath); - } - } - filesOfType = filterDataFiles(filesOfType, regexType); - if (dataFilesMap.containsKey(dataFileType)) { - dataFilesMap.get(dataFileType).addAll(filesOfType); - } else { - dataFilesMap.put(dataFileType, filesOfType); - } - } - } - return dataFilesMap; - } - - /** - * Helper method that, for a given subclass of DataFileInfo, finds all the files that correspond - * to that subclass and constructs DataFileInfo instances for them. The DataFileInfo instances - * are returned as a Set. - * - * @param clazz Class object of DataFileInfo subclass - * @param files Set of Path instances for files to be searched - * @return Set of all Paths in files argument that correspond to the specified subclass of - * DataFileInfo. - */ - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - private Set dataFilesOfClass(Class clazz, Set files) { - Set dataFiles = new TreeSet<>(); - Set foundFiles = new HashSet<>(); - T dataFileInfoForPatternCheck; - Constructor stringArgConstructor; - try { - Constructor noArgConstructor = clazz.getDeclaredConstructor(); - stringArgConstructor = clazz.getDeclaredConstructor(String.class); - dataFileInfoForPatternCheck = noArgConstructor.newInstance(); - } catch (NoSuchMethodException | SecurityException | InstantiationException - | IllegalAccessException | IllegalArgumentException | InvocationTargetException e) { - // Can never occur. All DataFileInfo classes can be constructed in the manner - // used above. - throw new AssertionError(e); - } - for (Path file : files) { - - // Check each file against the pattern for this DatastoreId subclass, and if - // validity is indicated, add it to the dataFiles set. - if (dataFileInfoForPatternCheck.pathValid(file)) { - try { - dataFiles.add(stringArgConstructor.newInstance(file.getFileName().toString())); - } catch (InstantiationException | IllegalAccessException | IllegalArgumentException - | InvocationTargetException e) { - // Can never occur. DataFileInfo instances can always be instantiated in - // the manner shown above. - throw new AssertionError(e); - } - foundFiles.add(file); - } - } - - return dataFiles; - } - - /** - * Helper function that performs general copying of the contents of a Map of data file types and - * their data files from one directory to another: either datastore-to-task-dir or - * task-dir-to-datastore. - * - * @param dataFileTypesMap Map from DataFileType instances to data files. - * @param destination RegexType that indicates the destination for the files. - * @param performCopy if true, copy the files, otherwise just find them and return the Map. - * @return Datastore locations of all copied files. For copy from datastore to task dir, the - * returned files are relative to the task directory (i.e., just filenames); for copy from the - * task dir to the datastore, the returned files are relative to the datastore root (i.e., they - * contain the path within the datastore to the directory that contains the file, as well as the - * filename). - */ - private Map> copyDataFilesByTypeToDestination( - Map> dataFileTypesMap, RegexType destination, - DatastoreCopyType copyType) { - - Map> copiedFiles = new HashMap<>(); - for (DataFileType dataFileType : dataFileTypesMap.keySet()) { - Set dataFiles = dataFileTypesMap.get(dataFileType); - if (destination.equals(RegexType.TASK_DIR)) { - copiedFiles.put(dataFileType, dataFiles); - } - Set datastoreFiles = new HashSet<>(); - for (Path dataFile : dataFiles) { - DataFilePaths dataFilePaths = destination.dataFilePaths(datastoreRoot, - taskDirectory, dataFileType, dataFile); - datastoreFiles.add(datastoreRoot.relativize(dataFilePaths.getDatastorePath())); - copyType.copy(dataFilePaths.getSourcePath(), dataFilePaths.getDestinationPath()); - } - if (destination.equals(RegexType.DATASTORE)) { - copiedFiles.put(dataFileType, datastoreFiles); - } - } - return copiedFiles; - } - - private List completedSubtaskDirectoriesWithResults() { - return completedSubtaskDirectories(WITH_RESULTS); - } - - private List completedSubtaskDirectoriesWithoutResults() { - return completedSubtaskDirectories(WITHOUT_RESULTS); - } - - private List completedSubtaskDirectories(Predicate predicate) { - - // Reconstitute the TaskConfigurationManager from the task directory - TaskConfigurationManager configManager = TaskConfigurationManager - .restore(taskDirectory.toFile()); - - // Identify the subtask directories that correspond to successful executions of an algorithm - return configManager.allSubTaskDirectories() - .stream() - .filter(AlgorithmStateFiles::isComplete) - .filter(predicate) - .map(File::toPath) - .collect(Collectors.toList()); - } - - private Set filterDataFiles(Set allDataFiles, RegexType destination) { - - // we never filter files that are in the task directory. - // We also never filter files if the TaskConfigurationParameters isn't defined. - if (destination.equals(RegexType.TASK_DIR) || taskConfigurationParameters == null) { - return allDataFiles; - } - - // if we're doing keep-up reprocessing, that's one case - if (!taskConfigurationParameters.isReprocess()) { - return filterDataFilesForKeepUpProcessing(allDataFiles); - } - - // if there are excluded tasks (i.e., tasks that produced results that should not - // be reprocessed), that's another case - if (taskConfigurationParameters.getReprocessingTasksExclude() != null - && taskConfigurationParameters.getReprocessingTasksExclude().length > 0) { - return filterDataFilesForBugfixProcessing(allDataFiles, - taskConfigurationParameters.getReprocessingTasksExclude()); - } - - // If we got this far then it's just garden variety reprocess-everything, so no - // filtering - return allDataFiles; - } - - /** - * Filters data files for "keep-up" processing. This is processing in which the user only needs - * to process input files that have never been successfully processed by the selected pipeline - * module. All data files that have been successfully processed in the pipeline by a node that - * matches the pipeline task's {@link PipelineDefinitionNode} are filtered out. - */ - private Set filterDataFilesForKeepUpProcessing(Set allDataFiles) { - - // collect all the consumers of the files in question -- note that we want both the - // consumers that produced output with the files AND consumers that failed to produce - // output but which recorded successful processing. The latter are stored with - // negative consumer IDs. This is necessary because for any file that ran successfully - // but produced no output, we don't want to process it again during "keep-up" processing. - List dpcs = datastoreProducerConsumerCrud() - .retrieveByFilename(allDataFiles); - Set allConsumers = new HashSet<>(); - dpcs.stream().forEach(s -> allConsumers.addAll(s.getAllConsumers())); - - // Determine the consumers that have the same pipeline definition node as the - // current task - List consumersWithMatchingNode = pipelineTaskCrud() - .retrieveIdsForPipelineDefinitionNode(allConsumers, - pipelineTask.getPipelineDefinitionNode()); - - // Return the data files that don't have any consumers that have the specified - // pipeline definition node - return dpcs.stream() - .filter(s -> Collections.disjoint(s.getAllConsumers(), consumersWithMatchingNode)) - .map(s -> Paths.get(s.getFilename())) - .collect(Collectors.toSet()); - } - - /** - * Filters data files for "bugfix" reprocessing. This is reprocessing in which the user doesn't - * want to process all files, but just the ones that have failed in a prior processing run. This - * is accomplished by taking the prior run or runs (in the form of pipeline task IDs) and - * excluding all inputs that were successfully processed in any of those prior tasks. - */ - private Set filterDataFilesForBugfixProcessing(Set allDataFiles, - long[] reprocessingTasksExclude) { - - List dpcs = datastoreProducerConsumerCrud() - .retrieveByFilename(allDataFiles); - - // Figure out if any of the excluded tasks use the same pipeline definition node - // as the current one, if not we can walk away - List reprocessingTasksExcludeList = Arrays - .asList(ArrayUtils.toObject(reprocessingTasksExclude)); - List tasksWithMatchingNode = pipelineTaskCrud().retrieveIdsForPipelineDefinitionNode( - reprocessingTasksExcludeList, pipelineTask.getPipelineDefinitionNode()); - if (tasksWithMatchingNode.isEmpty()) { - return allDataFiles; - } - - // Otherwise, any input that DOES NOT have one of the excluded tasks as - // a past consumer should be returned - return dpcs.stream() - .filter(s -> Collections.disjoint(s.getConsumers(), tasksWithMatchingNode)) - .map(s -> Paths.get(s.getFilename())) - .collect(Collectors.toSet()); - } - - /** - * Finds the actual source file for a given source file. If the source file is not a symbolic - * link, then that file is the actual source file. If not, the symbolic link is read to find the - * actual source file. The reading of symbolic links runs iteratively, so it produces the - * correct result even in the case of a link to a link to a link... etc. The process of - * following symbolic links stops at the first such link that is a child of the datastore root - * path. Thus the "actual source" is either a non-symlink file that the src file is a link to, - * or it's a file (symlink or regular file) that lies inside the datastore. - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public static Path realSourceFile(Path src) { - Path datastoreRoot = DirectoryProperties.datastoreRootDir(); - Path trueSrc = src; - if (Files.isSymbolicLink(src) && !src.startsWith(datastoreRoot)) { - try { - trueSrc = realSourceFile(Files.readSymbolicLink(src)); - } catch (IOException e) { - throw new UncheckedIOException("Unable to resolve symbolic link " + src.toString(), - e); - } - } - return trueSrc; - } - - /** - * Enum-with-behavior that supports multiple different copy mechanisms that are specialized for - * use with moving files between the datastore and a working directory. The following options - * are provided: - *

        - *
      1. {@link DatastoreCopyType#COPY} performs a traditional file copy operation. The copy is - * recursive, so directories are supported as well as individual files. - *
      2. {@link DatastoreCopyType#SYMLINK} makes the destination a symbolic link to the true - * source file, as defined by the {@link DataFileManager#realSourceFile(Path)} method. - * Symlinking can be faster than copying and can consume less disk space (assuming the datastore - * and working directories are on the same file system). - *
      3. {@link DatastoreCopyType#MOVE} will move the true source file to the destination; that - * is, it will follow symlinks via the {@link DataFileManager#realSourceFile(Path)} method and - * move the file that is found in this way. In addition, if the source file is a symlink, the - * true source file will be changed to a symlink to the moved file in its new location. In this - * way, the source file symlink remains valid and unchanged, but the file it points to is now - * itself a symlink to the moved file. - *
      - * In addition to all the foregoing, {@link DatastoreCopyType} manages file permissions. After - * execution of any move / copy / symlink operation, the new file's permissions are set to make - * it write-protected and world-readable. If the copy / move / symlink operation is required to - * overwrite the destination file, that file's permissions will be set to allow the overwrite - * prior to execution. - *

      - * IFor copying files from the datastore to the task directory, or from the task directory to - * the subtask directory, {@link DatastoreCopyType#COPY}, and {@link DatastoreCopyType#SYMLINK} - * options are available. For copies from the task directory to the datastore, only one option - * is provided: {@link DatastoreCopyType#MOVE}. - * - * @author PT - */ - private enum DatastoreCopyType { - COPY { - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - protected void copyInternal(Path src, Path dest) { - try { - checkout(src, dest); - if (Files.isRegularFile(src)) { - Files.copy(src, dest, StandardCopyOption.REPLACE_EXISTING); - } else { - FileUtils.copyDirectory(src.toFile(), dest.toFile()); - } - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to copy " + src.toString() + " to " + dest.toString(), e); - } - } - }, - MOVE { - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - protected void copyInternal(Path src, Path dest) { - try { - checkout(src, dest); - Path trueSrc = DataFileManager.realSourceFile(src); - if (Files.exists(dest)) { - FileUtil.prepareDirectoryTreeForOverwrites(dest); - } - Files.move(trueSrc, dest, StandardCopyOption.REPLACE_EXISTING, - StandardCopyOption.ATOMIC_MOVE); - FileUtil.writeProtectDirectoryTree(dest); - if (src != trueSrc) { - Files.delete(src); - Files.createSymbolicLink(trueSrc, dest); - } - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to move or symlink " + src.toString() + " to " + dest.toString(), - e); - } - } - }, - SYMLINK { - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - protected void copyInternal(Path src, Path dest) { - try { - checkout(src, dest); - Path trueSrc = DataFileManager.realSourceFile(src); - if (Files.exists(dest)) { - Files.delete(dest); - } - Files.createSymbolicLink(dest, trueSrc); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to symlink from " + src.toString() + " to " + dest.toString(), e); - } - } - }; - - /** - * Copy operation that allows / forces the caller to manage any {@link IOException} that - * occurs. - */ - protected abstract void copyInternal(Path src, Path dest); - - /** - * Copy operation that manages any resulting {@link IOException}}. In this event, an - * {@link UncheckedIOException} is thrown, which terminates execution of the datastore - * operations. - */ - public void copy(Path src, Path dest) { - copyInternal(src, dest); - } - - private static void checkout(Path src, Path dest) { - checkNotNull(src, "src"); - checkNotNull(dest, "dest"); - checkArgument(Files.exists(src, LinkOption.NOFOLLOW_LINKS), - "Source file " + src + " does not exist"); - } - } - - /** - * Selects a {@link DatastoreCopyType} based on the type of the source file. Source files that - * are symbolic links will use the {@link DatastoreCopyType#SYMLINK} operation, resulting in a - * symbolic link at the destination that links to the true source file and removal of the - * symbolic link that was used as the source file. Sources that are not symbolic links will use - * the {@link DatastoreCopyType#MOVE} operation. - */ - public static void moveOrSymlink(Path src, Path dest) throws IOException { - - if (Files.isSymbolicLink(src)) { - DatastoreCopyType.SYMLINK.copy(src, dest); - Files.delete(src); - } else { - DatastoreCopyType.MOVE.copy(src, dest); - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataFileType.java b/src/main/java/gov/nasa/ziggy/data/management/DataFileType.java deleted file mode 100644 index 697ff46..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DataFileType.java +++ /dev/null @@ -1,446 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import java.io.IOException; -import java.io.UncheckedIOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Objects; -import java.util.Set; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import java.util.stream.Stream; - -import gov.nasa.ziggy.module.io.Persistable; -import gov.nasa.ziggy.module.io.ProxyIgnore; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.RegexBackslashManager; -import gov.nasa.ziggy.util.RegexGroupCounter; -import jakarta.persistence.Entity; -import jakarta.persistence.Id; -import jakarta.persistence.Table; -import jakarta.persistence.Transient; -import jakarta.xml.bind.annotation.XmlAccessType; -import jakarta.xml.bind.annotation.XmlAccessorType; -import jakarta.xml.bind.annotation.XmlAttribute; -import jakarta.xml.bind.annotation.adapters.XmlJavaTypeAdapter; - -/** - * Defines a data file type. A data file type is a named object that has the following properties: - *

        - *
      1. A name convention specification that specifies the name that a data file of this type has - * when located in a task directory. This convention takes the form of a Java regex and is used to - * match files in the task directory (i.e., a file is identified to be of a given type if its - * filename matches the regex). - *
      2. A name convention specification that specifies the name and location that a data file of this - * type has when located in the datastore. The location includes all directories below the datastore - * root. - *
      - *

      - * The class uses regex group numbers to map groups from the task directory specification into the - * datastore specification. In other words, a directory specification of "$1/$3/foo-$2/$4.h5" means - * that the contents of group 1 is the name of the first directory below the datastore root, then - * group 3, then "foo-" plus group 2, then group 4 plus ".h5". - * - * @author PT - */ -@XmlAccessorType(XmlAccessType.NONE) -@Entity -@Table(name = "ziggy_DataFileType") -public class DataFileType implements Persistable { - - public enum RegexType { - TASK_DIR { - @Override - public Pattern getPattern(DataFileType dataFileType) { - return dataFileType.fileNamePatternForTaskDir(); - } - - @Override - public DataFilePaths dataFilePaths(Path datastoreRoot, Path taskDirectory, - DataFileType dataFileType, Path dataFile) { - Path sourcePath = datastoreRoot.resolve(dataFile); - String destinationName = dataFileType - .taskDirFileNameFromDatastoreFileName(dataFile.toString()); - Path destinationPath = taskDirectory.resolve(destinationName); - DataFilePaths paths = new DataFilePaths(sourcePath, destinationPath); - paths.setDatastorePathToSource(); - return paths; - } - - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public Stream pathStream(Path directory) { - try { - return Files.list(directory); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to list files of dir " + directory.toString(), e); - } - } - - @Override - public Path pathToRelativize(Path directory, Path datastoreRoot) { - return directory; - } - }, - DATASTORE { - @Override - public Pattern getPattern(DataFileType dataFileType) { - return dataFileType.fileNamePatternForDatastore(); - } - - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public DataFilePaths dataFilePaths(Path datastoreRoot, Path taskDirectory, - DataFileType dataFileType, Path dataFile) { - Path sourcePath = taskDirectory.resolve(dataFile); - String destinationName = dataFileType - .datastoreFileNameFromTaskDirFileName(dataFile.toString()); - Path destinationPath = datastoreRoot.resolve(destinationName); - try { - Files.createDirectories(destinationPath.getParent()); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to create directory " + destinationPath.getParent().toString(), e); - } - DataFilePaths paths = new DataFilePaths(sourcePath, destinationPath); - paths.setDatastorePathToDestination(); - return paths; - } - - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public Stream pathStream(Path directory) { - try { - return Files.walk(directory); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to walk directory tree " + directory.toString(), e); - } - } - - @Override - public Path pathToRelativize(Path directory, Path datastoreRoot) { - return datastoreRoot; - } - }; - - public abstract Pattern getPattern(DataFileType dataFileType); - - public abstract DataFilePaths dataFilePaths(Path datastoreRoot, Path taskDirectory, - DataFileType dataFileType, Path dataFile); - - public abstract Stream pathStream(Path directory); - - public abstract Path pathToRelativize(Path directory, Path datastoreRoot); - } - - @Transient - // Pattern that will match $ followed by a number, but will not match \$ plus a number - private static final Pattern GROUP_NUMBER_PATTERN = Pattern.compile("(\\$[0-9]+)"); - - @XmlAttribute(required = true) - @Id - private String name; - - @XmlAttribute(required = true) - @XmlJavaTypeAdapter(RegexBackslashManager.XmlRegexAdapter.class) - private String fileNameRegexForTaskDir; - - @XmlAttribute(required = true) - private String fileNameWithSubstitutionsForDatastore; - - @ProxyIgnore - @Transient - private String fileNameRegexForDatastore; - - @ProxyIgnore - @Transient - private Pattern fileNamePatternForTaskDir; - - @ProxyIgnore - @Transient - private Pattern fileNamePatternForDatastore; - - @ProxyIgnore - @Transient - private Map taskDirGroupToDatastoreGroupMap; - - // Used by Hibernate, do not remove. - public DataFileType() { - } - - /** - * Checks that the groups in the task dir regex are consistent with the substitutions in the - * datastore spec, that the datastore spec substitutions are valid, and that the name is not - * blank. - */ - public void validate() { - - // Check that the number of groups equals the number of substitutions, and - // that every group maps to a substitution - int nGroups = countFileNameGroups(); - int nSubs = countAndValidateSubstitutions(); - if (nGroups != nSubs) { - throw new IllegalStateException("Number of task dir groups, " + nGroups - + " does not equal number of datastore substitutions, " + nSubs); - } - - // Check for a name - if (name == null || name.isEmpty()) { - throw new IllegalStateException("No name specified"); - } - } - - /** - * Counts the number of groups in the task dir file name regex. - */ - private int countFileNameGroups() { - return RegexGroupCounter.groupCount(fileNamePatternForTaskDir().pattern()); - } - - /** - * Counts the number of substitution labels in the datastore file name specification. Also - * validates that all substitutions from 1 to some max value are present. - */ - private int countAndValidateSubstitutions() { - Matcher groupNumberMatcher = GROUP_NUMBER_PATTERN - .matcher(getFileNameWithSubstitutionsForDatastore()); - List substitutionList = new ArrayList<>(); - while (groupNumberMatcher.find()) { - String groupNumberString = groupNumberMatcher.group(1); - int groupNumber = Integer.parseInt(groupNumberString.substring(1)); - substitutionList.add(groupNumber); - } - Set substitutionSet = new HashSet<>(substitutionList); - boolean subNumbersOkay = true; - for (int i = 1; i <= substitutionSet.size(); i++) { - if (!substitutionSet.contains(i)) { - subNumbersOkay = false; - } - } - if (!subNumbersOkay) { - throw new IllegalStateException( - "Datastore specification does not contain contiguous substitutions from 1 to " - + substitutionSet.size()); - } - return substitutionSet.size(); - } - - public Pattern fileNamePatternForTaskDir() { - if (fileNamePatternForTaskDir == null) { - fileNamePatternForTaskDir = Pattern.compile(fileNameRegexForTaskDir); - } - return fileNamePatternForTaskDir; - } - - /** - * Generates a new version of the datastore name, with the group numbers replaced by the - * equivalent group regex definitions from the taskDirName - * - * @return - */ - public String fileNameRegexForDatastore() { - if (fileNameRegexForDatastore == null) { - - List groupsAlreadyUsed = new ArrayList<>(); - taskDirGroupToDatastoreGroupMap = new HashMap<>(); - // find all the groups in the taskDirName string - Matcher groupMatcher = RegexGroupCounter.GROUP_PATTERN - .matcher(fileNamePatternForTaskDir().pattern()); - List taskDirGroups = new ArrayList<>(); - while (groupMatcher.find()) { - taskDirGroups.add(groupMatcher.group(1)); - } - - // find and replace $(groupNumber) in datastoreName with the group contents - Matcher groupNumberMatcher = GROUP_NUMBER_PATTERN - .matcher(getFileNameWithSubstitutionsForDatastore()); - StringBuffer datastoreNameStringBuffer = new StringBuffer(); - int datastoreGroupCounter = 0; - while (groupNumberMatcher.find()) { - String groupNumberString = groupNumberMatcher.group(1); - int groupNumber = Integer.parseInt(groupNumberString.substring(1)); - int index = groupsAlreadyUsed.indexOf(groupNumber); - String replacement; - if (index == -1) { - datastoreGroupCounter++; - taskDirGroupToDatastoreGroupMap.put(groupNumber, datastoreGroupCounter); - replacement = "(" + taskDirGroups.get(groupNumber - 1) + ")"; - replacement = RegexBackslashManager.toDoubleBackslash(replacement); - groupsAlreadyUsed.add(groupNumber); - } else { - replacement = "\\\\" + (index + 1); - } - groupNumberMatcher.appendReplacement(datastoreNameStringBuffer, replacement); - } - groupNumberMatcher.appendTail(datastoreNameStringBuffer); - fileNameRegexForDatastore = datastoreNameStringBuffer.toString(); - } - return fileNameRegexForDatastore; - } - - public Pattern fileNamePatternForDatastore() { - if (fileNamePatternForDatastore == null) { - fileNamePatternForDatastore = Pattern.compile(fileNameRegexForDatastore()); - } - return fileNamePatternForDatastore; - } - - /** - * Determines whether a given task directory name matches the task directory name pattern. - */ - public boolean fileNameInTaskDirMatches(String fileNameInTaskDir) { - return fileNamePatternForTaskDir().matcher(fileNameInTaskDir).matches(); - } - - public boolean fileNameInDatastoreMatches(String fileNameInDatastore) { - return fileNamePatternForDatastore().matcher(fileNameInDatastore).matches(); - } - - /** - * Converts a task directory name to the corresponding datastore name for a file. This involves - * extracting the groups using the task dir file name pattern and then substituting them into - * the datastore file name pattern. - * - * @return Converted task dir file name if the name matches the task dir convention, null - * otherwise. - */ - public String datastoreFileNameFromTaskDirFileName(String taskDirName) { - String dName = null; - Matcher taskDirMatcher = fileNamePatternForTaskDir().matcher(taskDirName); - if (taskDirMatcher.matches()) { - Matcher groupNumberMatcher = GROUP_NUMBER_PATTERN - .matcher(getFileNameWithSubstitutionsForDatastore()); - StringBuffer datastoreNameStringBuffer = new StringBuffer(); - while (groupNumberMatcher.find()) { - String groupNumberString = groupNumberMatcher.group(1); - int groupNumber = Integer.parseInt(groupNumberString.substring(1)); - groupNumberMatcher.appendReplacement(datastoreNameStringBuffer, - taskDirMatcher.group(groupNumber)); - } - groupNumberMatcher.appendTail(datastoreNameStringBuffer); - dName = datastoreNameStringBuffer.toString(); - } - return dName; - } - - /** - * Converts a datastore name to the corresponding task directory name for a file. This involves - * extracting the groups from the datastore name pattern and substituting them into the task dir - * name pattern. - * - * @return Converted datastore file name if the name matches the datastore name convention, null - * otherwise. - */ - public String taskDirFileNameFromDatastoreFileName(String datastoreDirName) { - String tName = null; - Matcher datastoreNameMatcher = fileNamePatternForDatastore().matcher(datastoreDirName); - if (datastoreNameMatcher.matches()) { - Matcher groupMatcher = RegexGroupCounter.GROUP_PATTERN.matcher(fileNameRegexForTaskDir); - StringBuffer taskDirStringBuffer = new StringBuffer(); - int taskDirGroup = 0; - while (groupMatcher.find()) { - taskDirGroup++; - int datastoreGroupNumber = taskDirGroupToDatastoreGroupMap.get(taskDirGroup); - groupMatcher.appendReplacement(taskDirStringBuffer, - datastoreNameMatcher.group(datastoreGroupNumber)); - } - groupMatcher.appendTail(taskDirStringBuffer); - tName = taskDirStringBuffer.toString(); - } - return tName; - } - - /** - * Returns a Pattern for a truncated version of the datastore file name. The truncationLevel - * argument determines how many levels below the datastore root to include. For example, if - * truncationLevel == 2, the 2 levels of the Pattern below the datastore root will be included. - */ - public Pattern getDatastorePatternTruncatedToLevel(int truncationLevel) { - String[] splitRegex = splitDatastoreRegex(); - if (truncationLevel < 1 || truncationLevel > splitRegex.length) { - throw new IllegalArgumentException("Unable to truncate regex " - + fileNameRegexForDatastore() + " to level " + truncationLevel); - } - - StringBuilder truncatedRegexBuilder = new StringBuilder(); - for (int i = 0; i < truncationLevel; i++) { - truncatedRegexBuilder.append(splitRegex[i]); - if (i < truncationLevel - 1) { - truncatedRegexBuilder.append("/"); - } - } - return Pattern.compile(truncatedRegexBuilder.toString()); - } - - /** - * Returns a Pattern for the datastore file name in which the lowest directory levels have been - * truncated. The levelsToTruncate argument determines how many levels will be removed. - */ - public Pattern getDatastorePatternWithLowLevelsTruncated(int levelsToTruncate) { - String[] splitRegex = splitDatastoreRegex(); - if (levelsToTruncate < 0 || levelsToTruncate > splitRegex.length) { - throw new IllegalArgumentException("Unable to remove lowest " + levelsToTruncate - + " from regex " + fileNameRegexForDatastore()); - } - int truncationLevel = splitRegex.length - levelsToTruncate; - return getDatastorePatternTruncatedToLevel(truncationLevel); - } - - private String[] splitDatastoreRegex() { - return fileNameRegexForDatastore().split("/"); - } - - // Getters and setters - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getFileNameRegexForTaskDir() { - return fileNameRegexForTaskDir; - } - - public void setFileNameRegexForTaskDir(String fileNameRegexForTaskDir) { - this.fileNameRegexForTaskDir = fileNameRegexForTaskDir; - } - - public String getFileNameWithSubstitutionsForDatastore() { - return fileNameWithSubstitutionsForDatastore; - } - - public void setFileNameWithSubstitutionsForDatastore( - String fileNameWithSubstitutionsForDatastore) { - this.fileNameWithSubstitutionsForDatastore = fileNameWithSubstitutionsForDatastore; - } - - @Override - public int hashCode() { - return Objects.hash(name); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - DataFileType other = (DataFileType) obj; - if (!Objects.equals(name, other.name)) { - return false; - } - return true; - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataFileTypeImporter.java b/src/main/java/gov/nasa/ziggy/data/management/DataFileTypeImporter.java deleted file mode 100644 index d37be69..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DataFileTypeImporter.java +++ /dev/null @@ -1,259 +0,0 @@ -/* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the - * National Aeronautics and Space Administration. All Rights Reserved. - * - * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline - * Management System for Data Analysis Pipelines, under Cooperative Agreement Nos. NNX14AH97A, - * 80NSSC18M0068 & 80NSSC21M0079. - * - * This file is available under the terms of the NASA Open Source Agreement (NOSA). You should have - * received a copy of this agreement with the Ziggy source code; see the file LICENSE.pdf. - * - * Disclaimers - * - * No Warranty: THE SUBJECT SOFTWARE IS PROVIDED "AS IS" WITHOUT ANY WARRANTY OF ANY KIND, EITHER - * EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SUBJECT - * SOFTWARE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A - * PARTICULAR PURPOSE, OR FREEDOM FROM INFRINGEMENT, ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL BE - * ERROR FREE, OR ANY WARRANTY THAT DOCUMENTATION, IF PROVIDED, WILL CONFORM TO THE SUBJECT - * SOFTWARE. THIS AGREEMENT DOES NOT, IN ANY MANNER, CONSTITUTE AN ENDORSEMENT BY GOVERNMENT AGENCY - * OR ANY PRIOR RECIPIENT OF ANY RESULTS, RESULTING DESIGNS, HARDWARE, SOFTWARE PRODUCTS OR ANY - * OTHER APPLICATIONS RESULTING FROM USE OF THE SUBJECT SOFTWARE. FURTHER, GOVERNMENT AGENCY - * DISCLAIMS ALL WARRANTIES AND LIABILITIES REGARDING THIRD-PARTY SOFTWARE, IF PRESENT IN THE - * ORIGINAL SOFTWARE, AND DISTRIBUTES IT "AS IS." - * - * Waiver and Indemnity: RECIPIENT AGREES TO WAIVE ANY AND ALL CLAIMS AGAINST THE UNITED STATES - * GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT. IF RECIPIENT'S - * USE OF THE SUBJECT SOFTWARE RESULTS IN ANY LIABILITIES, DEMANDS, DAMAGES, EXPENSES OR LOSSES - * ARISING FROM SUCH USE, INCLUDING ANY DAMAGES FROM PRODUCTS BASED ON, OR RESULTING FROM, - * RECIPIENT'S USE OF THE SUBJECT SOFTWARE, RECIPIENT SHALL INDEMNIFY AND HOLD HARMLESS THE UNITED - * STATES GOVERNMENT, ITS CONTRACTORS AND SUBCONTRACTORS, AS WELL AS ANY PRIOR RECIPIENT, TO THE - * EXTENT PERMITTED BY LAW. RECIPIENT'S SOLE REMEDY FOR ANY SUCH MATTER SHALL BE THE IMMEDIATE, - * UNILATERAL TERMINATION OF THIS AGREEMENT. - */ - -package gov.nasa.ziggy.data.management; - -import java.io.File; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.HashSet; -import java.util.List; -import java.util.Set; -import java.util.stream.Collectors; - -import org.apache.commons.cli.CommandLine; -import org.apache.commons.cli.CommandLineParser; -import org.apache.commons.cli.DefaultParser; -import org.apache.commons.cli.Options; -import org.apache.commons.cli.ParseException; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.ModelType; -import gov.nasa.ziggy.pipeline.definition.crud.DataFileTypeCrud; -import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; -import gov.nasa.ziggy.pipeline.xml.ValidatingXmlManager; -import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; - -/** - * Performs import of DataFileType and ModelType instances to the database. - * - * @author PT - */ -public class DataFileTypeImporter { - - private static final Logger log = LoggerFactory.getLogger(DataFileTypeImporter.class); - - private List filenames; - private boolean dryrun; - private DataFileTypeCrud dataFileTypeCrud; - private ModelCrud modelCrud; - private int dataFileImportedCount; - private int modelFileImportedCount; - private ValidatingXmlManager xmlManager; - - // The following are instantiated so that unit tests that rely on them don't fail - private static List databaseDataFileTypeNames = new ArrayList<>(); - private static Set databaseModelTypes = new HashSet<>(); - - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - public static void main(String[] args) { - - CommandLineParser parser = new DefaultParser(); - Options options = new Options(); - options.addOption("dryrun", false, - "Parses and creates objects but does not persist to database"); - CommandLine cmdLine = null; - try { - cmdLine = parser.parse(options, args); - } catch (ParseException e) { - System.err.println("Illegal argument: " + e.getMessage()); - } - String[] filenames = cmdLine.getArgs(); - boolean dryrun = cmdLine.hasOption("dryrun"); - DataFileTypeImporter importer = new DataFileTypeImporter(Arrays.asList(filenames), dryrun); - - DatabaseTransactionFactory.performTransaction(() -> { - databaseDataFileTypeNames = new DataFileTypeCrud().retrieveAllNames(); - databaseModelTypes = new ModelCrud().retrieveModelTypeMap().keySet(); - return null; - }); - if (!dryrun) { - DatabaseTransactionFactory.performTransaction(() -> { - importer.importFromFiles(); - return null; - }); - } else { - importer.importFromFiles(); - } - } - - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public DataFileTypeImporter(List filenames, boolean dryrun) { - this.filenames = filenames; - this.dryrun = dryrun; - try { - xmlManager = new ValidatingXmlManager<>(DatastoreConfigurationFile.class); - } catch (IllegalArgumentException | SecurityException e) { - throw new PipelineException( - "Unable to construct ValidatingXmlManager for class DatastoreConfigurationFile", e); - } - } - - /** - * Perform the import from all XML files. The importer will skip any file that fails to validate - * or cannot be parsed, will skip any DataFileType instance that fails internal validation, and - * will skip any DataFileType that has the name of a type that is already in the database; all - * other DataFileTypes will be imported. If any duplicate names are present in the set of - * DataFileType instances to be imported, none will be imported. The import also imports model - * definitions. - */ - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - public void importFromFiles() { - - List dataFileTypes = new ArrayList<>(); - List modelTypes = new ArrayList<>(); - for (String filename : filenames) { - File file = new File(filename); - if (!file.exists() || !file.isFile()) { - log.warn("File " + filename + " is not a regular file"); - continue; - } - - // open and read the XML file - log.info("Reading from " + filename); - DatastoreConfigurationFile configDoc = null; - try { - configDoc = xmlManager.unmarshal(file); - } catch (Exception e) { - log.warn("Unable to parse configuration file " + filename, e); - continue; - } - - log.info("Importing DataFileType definitions from " + filename); - Set dataFileTypesFromFile = configDoc.getDataFileTypes(); - List dataFileTypesNotImported = new ArrayList<>(); - for (DataFileType typeXb : dataFileTypesFromFile) { - try { - typeXb.validate(); - } catch (Exception e) { - log.warn("Unable to validate data file type definition " + typeXb.getName(), e); - dataFileTypesNotImported.add(typeXb); - continue; - } - if (databaseDataFileTypeNames.contains(typeXb.getName())) { - log.warn("Not importing data file type definition \"" + typeXb.getName() - + "\" due to presence of existing type with same name"); - dataFileTypesNotImported.add(typeXb); - continue; - } - } - dataFileTypesFromFile.removeAll(dataFileTypesNotImported); - log.info("Imported " + dataFileTypesFromFile.size() - + " DataFileType definitions from file " + filename); - dataFileTypes.addAll(dataFileTypesFromFile); - - // Now for the models - Set modelTypesFromFile = configDoc.getModelTypes(); - List modelTypesNotImported = new ArrayList<>(); - for (ModelType modelTypeXb : modelTypesFromFile) { - try { - modelTypeXb.validate(); - } catch (Exception e) { - log.warn("Unable to validate model type definition " + modelTypeXb.getType(), - e); - modelTypesNotImported.add(modelTypeXb); - continue; - } - if (databaseModelTypes.contains(modelTypeXb.getType())) { - log.warn("Not importing model type definition \"" + modelTypeXb.getType() - + "\" due to presence of existing type with same name"); - modelTypesNotImported.add(modelTypeXb); - continue; - } - } - - modelTypesFromFile.removeAll(modelTypesNotImported); - log.info("Imported " + modelTypesFromFile.size() + " ModelType definitions from file " - + filename); - modelTypes.addAll(modelTypesFromFile); - } // end loop over files - - List dataFileTypeNames = dataFileTypes.stream() - .map(DataFileType::getName) - .collect(Collectors.toList()); - Set uniqueDataFileTypeNames = new HashSet<>(dataFileTypeNames); - if (dataFileTypeNames.size() != uniqueDataFileTypeNames.size()) { - throw new IllegalStateException( - "Unable to persist data file types due to duplicate names"); - } - dataFileImportedCount = dataFileTypes.size(); - List modelTypeNames = modelTypes.stream() - .map(ModelType::getType) - .collect(Collectors.toList()); - Set uniqueModelTypeNames = new HashSet<>(modelTypeNames); - if (modelTypeNames.size() != uniqueModelTypeNames.size()) { - throw new IllegalStateException("Unable to persist model types due to duplicate names"); - } - modelFileImportedCount = modelTypes.size(); - if (!dryrun) { - log.info( - "Persisting to datastore " + dataFileTypes.size() + " DataFileType definitions"); - dataFileTypeCrud().persist(dataFileTypes); - log.info("Persisting to datastore " + modelTypes.size() + " model definitions"); - modelCrud().persist(modelTypes); - log.info("Persist step complete"); - } else { - log.info("Not persisting because of dryrun option"); - } - } - - // default scope for mocking in unit tests - DataFileTypeCrud dataFileTypeCrud() { - if (dataFileTypeCrud == null) { - dataFileTypeCrud = new DataFileTypeCrud(); - } - return dataFileTypeCrud; - } - - ModelCrud modelCrud() { - if (modelCrud == null) { - modelCrud = new ModelCrud(); - } - return modelCrud; - } - - int getDataFileImportedCount() { - return dataFileImportedCount; - } - - int getModelFileImportedCount() { - return modelFileImportedCount; - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataImporter.java b/src/main/java/gov/nasa/ziggy/data/management/DataImporter.java deleted file mode 100644 index 683d203..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DataImporter.java +++ /dev/null @@ -1,154 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import java.nio.file.Path; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeSet; -import java.util.stream.Collectors; - -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; - -/** - * Parent class for all data importer classes. The {@link DataImporter} implementations are used by - * the {@link DataReceiptPipelineModule} to import mission data files into the datastore. The - * abstract methods in this class perform assorted support functions that are needed by the - * {@link #importFilesToDatastore(List)} method. - * - * @author PT - */ -public abstract class DataImporter { - - protected final PipelineTask pipelineTask; - protected final Path datastoreRoot; - protected final Path dataReceiptPath; - private int invalidFilesCount; - private int failedImportsCount; - private int totalDataFileCount; - private Set successfulImports; - private Set failedImports; - - public DataImporter(PipelineTask pipelineTask, Path dataReceiptPath, Path datastoreRoot) { - this.pipelineTask = pipelineTask; - this.datastoreRoot = datastoreRoot; - this.dataReceiptPath = dataReceiptPath; - } - - /** - * Validates the delivery as a whole. - * - * @return true if delivery is valid, false otherwise. - */ - protected abstract boolean validateDelivery(); - - /** - * Validates an individual data file. - * - * @param dataFile {@link Path} to the file location in the data receipt directory. - * @return true if file is valid, false otherwise. - */ - protected abstract boolean validateDataFile(Path dataFile); - - /** - * {@link Map} from the data receipt directory location of a file to its location in the - * datastore. The Map key is the {@link Path} to the file prior to import, relative to the data - * receipt path; the map value is the {@link Path} to the file after import, relative to the - * datastore root. - */ - protected abstract Map dataFiles(List namesOfValidFiles); - - /** - * Imports files from the data receipt directory to the datastore. - * - * @param dataFiles {@link Map} from data receipt directory file locations to datastore file - * locations. - * @return {@link Set} of locations of files successfully moved to the datastore. - */ - protected abstract Set importFiles(Map dataFiles); - - /** - * Performs the import of a given list of files. - */ - public void importFilesToDatastore(List namesOfFilesToImport) { - // Validate the delivery - if (!validateDelivery()) { - throw new PipelineException("Unable to validate data delivery"); - } - - // Obtain the data file instances and validate them - Map dataFiles = dataFiles(namesOfFilesToImport); - totalDataFileCount = dataFiles.size(); - Set invalidDataFiles = dataFiles.keySet() - .stream() - .filter(s -> !validateDataFile(s)) - .collect(Collectors.toSet()); - Map invalidDataFilesMap = new HashMap<>(); - invalidFilesCount = invalidDataFiles.size(); - if (!invalidDataFiles.isEmpty()) { - for (Path invalidFile : invalidDataFiles) { - invalidDataFilesMap.put(invalidFile, dataFiles.get(invalidFile)); - dataFiles.remove(invalidFile); - } - } - - // Perform the import - Set importedFiles = importFiles(dataFiles); - Set filesNotImported = dataFiles.keySet() - .stream() - .filter(s -> !importedFiles.contains(s)) - .collect(Collectors.toSet()); - failedImportsCount = filesNotImported.size(); - if (!filesNotImported.isEmpty()) { - for (Path fileNotImported : filesNotImported) { - invalidDataFilesMap.put(fileNotImported, dataFiles.get(fileNotImported)); - dataFiles.remove(fileNotImported); - } - } - - // Preserve import records for use by callers. - successfulImports = new TreeSet<>(dataFiles.values()); - failedImports = new TreeSet<>(invalidDataFilesMap.values()); - } - - public PipelineTask getPipelineTask() { - return pipelineTask; - } - - public Path getDatastoreRoot() { - return datastoreRoot; - } - - public Path getDataReceiptPath() { - return dataReceiptPath; - } - - public int getInvalidFilesCount() { - return invalidFilesCount; - } - - public int getFailedImportsCount() { - return failedImportsCount; - } - - public int getTotalDataFileCount() { - return totalDataFileCount; - } - - public Set getSuccessfulImports() { - return successfulImports; - } - - public void setSuccessfulImports(Set successfulImports) { - this.successfulImports = successfulImports; - } - - public Set getFailedImports() { - return failedImports; - } - - public void setFailedImports(Set failedImports) { - this.failedImports = failedImports; - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptDefinition.java b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptDefinition.java new file mode 100644 index 0000000..f6ec281 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptDefinition.java @@ -0,0 +1,71 @@ +package gov.nasa.ziggy.data.management; + +import java.nio.file.Path; +import java.util.List; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +/** + * Defines the data receipt implementation for a given pipeline. A data receipt implementation + * consists of a set of requirements on both the overall delivery and each file in the delivery, and + * a mapping function that maps a filename in the data receipt directory to a location in the data + * storage system (datastore, array store, or other kind of storage). The following implementation + * methods accomplish this: + *

        + *
      1. {@link #setDataImportDirectory(Path)} sets the directory from which the current data receipt + * task is importing files. + *
      2. {@link #isConformingDelivery()} performs checks on the overall delivery to ensure that it + * conforms to the requirements of the {@link DataReceiptDefinition} implementation. This can be + * activities like checking a file manifest, looking for extraneous files, etc. + *
      3. {@link #isConformingFile(Path)} performs checks on a given file in the delivery to ensure + * that it conforms to the requirements of the {@link DataReceiptDefinition} implementation. + *
      4. {@link #successfulImports()} returns the collection of files that were successfully imported. + *
      5. {@link #failedImports()} returns the collection of files that failed to import. + *
      + * Implementations of {@link DataReceiptDefinition} must have a no-argument constructor. + *

      + * The reference implementation of {@link DataReceiptDefinition} is + * {@link DatastoreDirectoryDataReceiptDefinition}. + * + * @author PT + */ +public interface DataReceiptDefinition { + + /** + * Set the path to the files for import. + */ + void setDataImportDirectory(Path dataImportDirectory); + + /** + * Ensures that the delivery as a whole conforms to any requirements of the data receipt system. + * A return value of true indicates to the caller that requirements are met and that it is now + * safe to test the files in the delivery. + */ + boolean isConformingDelivery(); + + /** + * Ensures that each file in the delivery conforms to any requirements of the data receipt + * system. The {@link DataReceiptPipelineModule} will loop over the files in the data receipt + * directory, test each of them, and send a list of nonconforming files to the task log. + */ + boolean isConformingFile(Path dataFile); + + /** Determines the data receipt {@link Path}s for all files that are to be imported. */ + List filesForImport(); + + /** Performs the actual file input. */ + void importFiles(); + + /** + * {@link List} of datastore {@link Path}s for all files successfully imported. + */ + List successfulImports(); + + /** + * {@link List} of datastore {@link Path}s for all files that failed to import. + */ + List failedImports(); + + /** Set the {@link PipelineTask} for the definition instance. */ + void setPipelineTask(PipelineTask pipelineTask); +} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptFile.java b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptFile.java index ba7d540..210dcbb 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptFile.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptFile.java @@ -10,21 +10,18 @@ public class DataReceiptFile { private long taskId; private String name; - private String fileType; private String status; public DataReceiptFile(DatastoreProducerConsumer record) { name = record.getFilename(); taskId = record.getProducer(); status = "Imported"; - fileType = record.getDataReceiptFileType().toString(); } public DataReceiptFile(FailedImport record) { name = record.getFilename(); taskId = record.getDataReceiptTaskId(); status = "Failed"; - fileType = record.getDataReceiptFileType().toString(); } public long getTaskId() { @@ -43,14 +40,6 @@ public void setName(String name) { this.name = name; } - public String getFileType() { - return fileType; - } - - public void setFileType(String fileType) { - this.fileType = fileType; - } - public String getStatus() { return status; } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java index fc23174..3d3f5c6 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModule.java @@ -4,48 +4,39 @@ import java.io.IOException; import java.io.UncheckedIOException; +import java.lang.reflect.Constructor; import java.lang.reflect.InvocationTargetException; import java.nio.file.DirectoryStream; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.nio.file.StandardCopyOption; import java.util.Arrays; import java.util.Collection; -import java.util.Date; import java.util.List; -import java.util.Map; -import java.util.Set; import java.util.stream.Collectors; -import java.util.stream.Stream; import org.apache.commons.configuration2.ImmutableConfiguration; import org.apache.commons.io.FileUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumer.DataReceiptFileType; import gov.nasa.ziggy.models.ModelImporter; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.ModelRegistry; -import gov.nasa.ziggy.pipeline.definition.ModelType; -import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingState; import gov.nasa.ziggy.pipeline.definition.ProcessingStatePipelineModule; -import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.alert.AlertService.Severity; -import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; -import gov.nasa.ziggy.services.database.DatabaseService; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import gov.nasa.ziggy.uow.DataReceiptUnitOfWorkGenerator; import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; import gov.nasa.ziggy.uow.UnitOfWork; +import gov.nasa.ziggy.uow.UnitOfWorkGenerator; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.io.FileUtil; @@ -54,24 +45,16 @@ * Pipeline module that performs data receipt, defined as the process that brings science data and * instrument models into the datastore from the outside world. *

      - * This class requires an instance of an implementation of the {@link DataImporter} interface, which - * provides validation for the overall delivery and for each individual data file. The - * {@link DataImporter} subclass is specified in the properties file. If no such specification is - * provided, the {@link DefaultDataImporter} class will be used, which performs no validations. + * This class requires an instance of an implementation of the {@link DataReceiptDefinition} + * interface, which provides an overall definition of the requirements and conventions of the data + * receipt implementation for a given pipeline. The {@link DataReceiptDefinition} implementing class + * is specified in the properties file. If no such specification is provided, the + * {@link DatastoreDirectoryDataReceiptDefinition} class will be used. *

      - * This class also uses an instance of {@link ModelImporter} to import the models. - *

      - * In order to determine the regular expressions for data and model files for import, one or more - * {@link DataFileType} instances and one or more {@link ModelType} instances must be provided for - * the data receipt node in the pipeline definition. All files that are successfully imported will - * have a database entry that shows which pipeline task was used to perform the import. Any failures - * will be recorded in a separate database table. - *

      - * The importer uses an import directory that is specified in the properties file. Data files in - * this directory must use their task directory name format in order to be located and imported. - * Files that are regular files will be moved to their specified locations in the datastore; files - * that are symlinks will be unlinked, and a new symlink will be created at the specified location - * in the datastore. + * The importer uses an import directory that is specified in the properties file. Files that are + * regular files will be moved to their specified locations in the data storage system; files that + * are symlinks will be unlinked, and a new symlink will be created at the specified location in the + * data storage system. * * @author PT */ @@ -80,40 +63,34 @@ public class DataReceiptPipelineModule extends PipelineModule private static final Logger log = LoggerFactory.getLogger(DataReceiptPipelineModule.class); - private static final String DEFAULT_DATA_RECEIPT_CLASS = "gov.nasa.ziggy.data.management.DefaultDataImporter"; + private static final String DEFAULT_DATA_RECEIPT_CLASS = DatastoreDirectoryDataReceiptDefinition.class + .getCanonicalName(); public static final String DATA_RECEIPT_MODULE_NAME = "data-receipt"; - private DataImporter dataImporter; + private DataReceiptDefinition dataReceiptDefinition; protected ModelImporter modelImporter; - private ManifestCrud manifestCrud; - private List namesOfFilesToImport; private Path dataReceiptTopLevelPath; private Path dataImportPathForTask; - private Manifest manifest; - private Acknowledgement ack; private String dataReceiptDir; private boolean processingComplete = false; - private Path datastoreRoot; - private boolean allFilesImported = true; public DataReceiptPipelineModule(PipelineTask pipelineTask, RunMode runMode) { super(pipelineTask, runMode); - } - - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public boolean processTask() { // Get the top-level DR directory and the datastore root directory ImmutableConfiguration config = ZiggyConfiguration.getInstance(); dataReceiptDir = config.getString(PropertyName.DATA_RECEIPT_DIR.property()); - datastoreRoot = DirectoryProperties.datastoreRootDir(); UnitOfWork uow = pipelineTask.uowTaskInstance(); - dataReceiptTopLevelPath = Paths.get(dataReceiptDir); + dataReceiptTopLevelPath = Paths.get(dataReceiptDir).toAbsolutePath(); dataImportPathForTask = dataReceiptTopLevelPath .resolve(DirectoryUnitOfWorkGenerator.directory(uow)); checkState(Files.isDirectory(dataImportPathForTask), dataImportPathForTask.toString() + " not a directory"); + } + + @Override + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public boolean processTask() { boolean containsNonHiddenFiles = false; Path filePathForException = null; @@ -138,9 +115,6 @@ public boolean processTask() { return true; } - // Read the manifest - readManifest(); - // Get and execute the run mode runMode.run(this); @@ -168,7 +142,7 @@ public List processingStates() { public void processingMainLoop() { while (!processingComplete) { - getProcessingState().taskAction(this); + databaseProcessingState().taskAction(this); } } @@ -178,97 +152,39 @@ public void processingMainLoop() { */ @Override public void initializingTaskAction() { - incrementProcessingState(); + incrementDatabaseProcessingState(); } /** - * Performs the algorithm portion of DR, which is reading the manifest, validating the delivered - * files, and generating an acknowledgement. + * Performs the algorithm portion of DR: validation of the delivery and of the delivered files. */ @Override public void executingTaskAction() { - manifest.setImportTime(new Date()); - manifest.setImportTaskId(pipelineTask.getId()); - - // Check the uniqueness of the dataset ID, unless the ID value is <= 0 - if (manifest.getDatasetId() > 0 - && manifestCrud().datasetIdExists(manifest.getDatasetId())) { - throw new PipelineException( - "Dataset ID " + manifest.getDatasetId() + " has already been used"); - } - - // Generate the acknowledgement object -- note that this also performs the - // transfer validation and size / checksum validation for all files in the - // manifest. If the manifest contains files with problems, the import will - // terminate with an exception. - acknowledgeManifest(); - - // Save the manifest information in the database. - manifest.setAcknowledged(true); - manifestCrud().persist(manifest); - - // Make sure that all the regular files in the directory tree have been validated - // (i.e., there are no files in the directory tree that are absent from the - // manifest). - checkForFilesNotInManifest(); - - incrementProcessingState(); - } - - private void readManifest() { - manifest = Manifest.readManifest(dataImportPathForTask); - if (manifest == null) { - throw new PipelineException( - "No manifest file present in directory " + dataImportPathForTask.toString()); - } - log.info("Read manifest from file " + manifest.getName()); - } - - private void acknowledgeManifest() { - - ack = Acknowledgement.of(manifest, dataImportPathForTask, pipelineTask.getId()); - - // Write the acknowledgement to the directory. - ack.write(dataImportPathForTask); - log.info("Acknowledgement file written: " + ack.getName()); - - // If the acknowledgement has bad status, throw an exception now. - if (ack.getTransferStatus().equals(DataReceiptStatus.INVALID)) { - log.error("Validation of files against the manifest status == INVALID"); - throw new PipelineException( - "Data Receipt terminated due to manifest validation failure"); + // Validate the delivery. + dataReceiptDefinition = dataReceiptDefinition(); + dataReceiptDefinition().setDataImportDirectory(dataImportPathForTask); + dataReceiptDefinition().setPipelineTask(pipelineTask); + boolean deliveryValid = dataReceiptDefinition().isConformingDelivery(); + if (!deliveryValid) { + throw new PipelineException("Delivery validation failed"); } - } - - private void checkForFilesNotInManifest() { - - // Get the names of the files that passed validation (which at this point should - // be the set of all names in the manifest) - List namesOfValidFiles = ack.namesOfValidFiles(); - Map regularFilesInDirTree = FileUtil - .regularFilesInDirTree(dataImportPathForTask); - List filenamesInDirTree = regularFilesInDirTree.keySet() - .stream() + List filesToImport = dataReceiptDefinition().filesForImport(); + List invalidFiles = filesToImport.stream() + .filter(s -> !dataReceiptDefinition().isConformingFile(s)) .map(Path::toString) .collect(Collectors.toList()); - filenamesInDirTree.removeAll(namesOfValidFiles); - filenamesInDirTree.remove(manifest.getName()); - filenamesInDirTree.remove(ack.getName()); - if (filenamesInDirTree.size() != 0) { - log.error("Data receipt directory " + dataImportPathForTask.toString() - + " contains files not listed in manifest "); - for (String filename : filenamesInDirTree) { - log.error("File missing from manifest: " + filename); + if (invalidFiles.size() != 0) { + for (String invalidFile : invalidFiles) { + log.error("File failed data receipt validation: {}", invalidFile); } - ack.write(dataImportPathForTask); - manifest.setAcknowledged(true); - log.info("Acknowledgement file written: " + ack.getName()); - throw new PipelineException("Unable to import files from data receipt directory " - + dataImportPathForTask.toString() - + " due to presence of files not listed in manifest"); + throw new PipelineException("File validation failed, see task log for details"); } + + // If we made it this far, we can proceed to the storing state, which performs the actual + // import. + incrementDatabaseProcessingState(); } /** @@ -281,164 +197,55 @@ private void checkForFilesNotInManifest() { @Override public void storingTaskAction() { - // Make a list of files for import. This is the list of non-hidden files in the - // data receipt directory. Note that in the steps above we have implicitly ensured that - // all regular files that are about to get imported are present in the manifest and are - // valid based on the checksum and the size of the files. Hence, it is now safe to simply - // walk through the contents of the data receipt directory and import everything. - generateFilenamesForImport(); - - importDataFiles(datastoreRoot); + dataReceiptDefinition().importFiles(); + persistProducerConsumerRecords(dataReceiptDefinition().successfulImports(), + dataReceiptDefinition().failedImports()); - importModels(); - - if (!allFilesImported) { + if (dataReceiptDefinition().failedImports().size() > 0) { throw new PipelineException("File import failures detected"); } performDirectoryCleanup(); - incrementProcessingState(); - } - - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_IN_RUNNABLE) - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - private void generateFilenamesForImport() { - try (Stream filestream = Files.list(dataImportPathForTask)) { - namesOfFilesToImport = filestream.filter(t -> { - try { - return !Files.isHidden(t); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to check hidden status of file" + t.toString(), e); - } - }) - .map(s -> dataImportPathForTask.relativize(s)) - .map(Path::toString) - .collect(Collectors.toList()); - } catch (IOException e1) { - throw new UncheckedIOException( - "Unable to list files in directory " + dataImportPathForTask.toString(), e1); - } - } - - /** - * Performs import of mission data files into the datatore. - */ - private void importDataFiles(Path datastoreRootPath) { - - final DataImporter dataImporter = dataImporter(Paths.get(dataReceiptDir), - datastoreRootPath); - - dataImporter.importFilesToDatastore(namesOfFilesToImport); - if (dataImporter.getInvalidFilesCount() > 0) { - allFilesImported = false; - log.warn("Detected " + dataImporter.getInvalidFilesCount() - + " data files that failed validation"); - alertService().generateAndBroadcastAlert("Data Receipt (DR)", pipelineTask.getId(), - AlertService.Severity.WARNING, - "Failed to import " + dataImporter.getInvalidFilesCount() + " data files (out of " - + dataImporter.getTotalDataFileCount() + ")"); - } - - if (dataImporter.getFailedImportsCount() > 0) { - allFilesImported = false; - log.warn("Detected " + dataImporter.getFailedImportsCount() - + " data files that were not imported"); - } - - // Generate data accountability records. - persistProducerConsumerRecords(dataImporter.getSuccessfulImports(), - dataImporter.getFailedImports(), DataReceiptFileType.DATA); + incrementDatabaseProcessingState(); } /** * Create and persist the data accountability records for successful and failed imports. */ protected void persistProducerConsumerRecords(Collection successfulImports, - Collection failedImports, DataReceiptFileType fileType) { + Collection failedImports) { // Persist successful file records to the datastore producer-consumer table if (!successfulImports.isEmpty()) { - new DatastoreProducerConsumerCrud().createOrUpdateProducer(pipelineTask, - successfulImports, fileType); + log.info("Updating {} producer-consumer records ...", successfulImports.size()); + DatabaseTransactionFactory.performTransaction(() -> { + new DatastoreProducerConsumerCrud().createOrUpdateProducer(pipelineTask, + successfulImports); + return null; + }); + log.info("Updating {} producer-consumer records ...done", successfulImports.size()); } // Save the failure cases to the FailedImport database table if (!failedImports.isEmpty()) { - new FailedImportCrud().create(pipelineTask, failedImports, fileType); - } - } - - /** - * Performs import of instrument models to the datastore. This method must be synchronized in - * order to ensure that the model registry is updated by only one task at a time. This ensures - * that imports that run across multiple tasks, with model imports in each task, will not result - * in a corrupted model registry. - */ - private void importModels() { - synchronized (DataReceiptPipelineModule.class) { - - Path dataReceiptPath = Paths.get(dataReceiptDir); - // get the unit of work from the pipeline task - UnitOfWork uow = pipelineTask.uowTaskInstance(); - Path importDirectory = dataReceiptPath - .resolve(DirectoryUnitOfWorkGenerator.directory(uow)); - log.info("Importing models from directory: " + importDirectory.toString()); - - // Obtain the model types from the pipeline task - Set modelTypes = pipelineTask.getPipelineDefinitionNode().getModelTypes(); - - ModelImporter importer = modelImporter(importDirectory, - "Model imports performed at time " + new Date().toString()); - - // Set up and perform imports - importer.setDataReceiptTaskId(pipelineTask.getId()); - importer.setModelTypesToImport(modelTypes); - boolean modelFilesLocated = importer.importModels(namesOfFilesToImport); - - if (modelFilesLocated) { - - // The pipeline instance is supposed to have a model registry with all the current - // models in it. Unfortunately, the instance that contains this importer can't have - // that registry because the models were just imported. Add the registry to the - // instance now. - updateModelRegistryForPipelineInstance(); - - // If there are any failed imports, we need to save that information now - List successfulImports = importer.getSuccessfulImports(); - List failedImports = importer.getFailedImports(); - persistProducerConsumerRecords(successfulImports, failedImports, - DataReceiptFileType.MODEL); - if (!failedImports.isEmpty()) { - allFilesImported = false; - log.warn(failedImports.size() + " out of " - + (successfulImports.size() + failedImports.size()) - + " model files failed to import"); - alertService().generateAndBroadcastAlert("Data Receipt (DR)", - pipelineTask.getId(), AlertService.Severity.WARNING, - "Failed to import " + failedImports.size() + " model files (out of " - + (successfulImports.size() + failedImports.size()) + ")"); - } - // Flush the session so that as soon as the next task starts importing model files - // it already has an up to date registry - flushDatabase(); - } + log.info("Recording {} failed imports...", failedImports.size()); + DatabaseTransactionFactory.performTransaction(() -> { + new FailedImportCrud().create(pipelineTask, failedImports); + return null; + }); + log.info("Recording {} failed imports...done", failedImports.size()); } } /** - * Flushes the database. Broken out to facilitate testing. + * Instantiates a {@link DataReceiptDefinition} instance of the user-specified implementing + * class. */ - protected void flushDatabase() { - DatabaseService.getInstance().getSession().flush(); - } - - // Allows a caller to supply a data receipt instance for test purposes @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - DataImporter dataImporter(Path dataReceiptPath, Path datastoreRootPath) { - if (dataImporter == null) { + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + DataReceiptDefinition dataReceiptDefinition() { + if (dataReceiptDefinition == null) { // Get the data importer implementation ImmutableConfiguration config = ZiggyConfiguration.getInstance(); @@ -450,56 +257,53 @@ DataImporter dataImporter(Path dataReceiptPath, Path datastoreRootPath) { } catch (ClassNotFoundException e) { throw new PipelineException("Class " + classname + " not found", e); } - if (!DataImporter.class.isAssignableFrom(dataReceiptClass)) { + if (!DataReceiptDefinition.class.isAssignableFrom(dataReceiptClass)) { throw new PipelineException( "Class" + classname + " not implementation of DataReceipt interface"); } // Instantiate the appropriate class try { - dataImporter = (DataImporter) dataReceiptClass - .getDeclaredConstructor(PipelineTask.class, Path.class, Path.class) - .newInstance(pipelineTask, dataReceiptPath, datastoreRootPath); + Constructor ctor = dataReceiptClass.getConstructor(); + dataReceiptDefinition = (DataReceiptDefinition) ctor.newInstance(); } catch (InstantiationException | IllegalAccessException | IllegalArgumentException | InvocationTargetException | NoSuchMethodException | SecurityException e) { - // Can never occur. By construction, the data importer class is known and has - // a constructor with an appropriate signature. - throw new AssertionError(e); + throw new PipelineException( + "Class " + dataReceiptClass.getName() + " has no zero-argument constructor"); } } - return dataImporter; + return dataReceiptDefinition; } /** * Performs cleanup on the directory used as the file source for this data receipt unit of work. - * During cleanup the directory is checked to make sure that the only non-hidden files present - * are the manifest and acknowledgement; these are then moved to the master manifest / - * acknowledgement directory. If the UOW used a subdirectory of the main DR directory, that - * directory is deleted. + * Specifically: if the UOW used a subdirectory of the main DR directory, that directory is + * deleted; other directories within the UOW directory are deleted if they are empty, otherwise + * an exception occurs. */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public void performDirectoryCleanup() { + cleanUpSpecifiedDirectory(dataImportPathForTask); + } + /** + * Recursively loops through all directories and subdirectories; if they contain only hidden + * files, they can be deleted. + */ + private void cleanUpSpecifiedDirectory(Path directory) { try { - // Create the manifests directory if it doesn't yet exist - Path manifestDir = DirectoryProperties.manifestsDir(); - Files.createDirectories(manifestDir); - - // Move the manifest and the acknowledgement to the hidden manifests directory. - Files.move(dataImportPathForTask.resolve(manifest.getName()), - manifestDir.resolve(manifest.getName()), StandardCopyOption.REPLACE_EXISTING); - String ackName = Acknowledgement.nameFromManifestName(manifest); - Files.move(dataImportPathForTask.resolve(ackName), manifestDir.resolve(ackName), - StandardCopyOption.REPLACE_EXISTING); - - Path realPath = DataFileManager.realSourceFile(dataImportPathForTask); + Path realPath = FileUtil.realSourceFile(directory); try (DirectoryStream stream = Files.newDirectoryStream(realPath)) { for (Path file : stream) { - // Ignore hidden files + // Ignore hidden files. if (Files.isHidden(file)) { continue; } - // If we got here we have non-hidden, non manifest, non-ack files, so we can't + if (Files.isDirectory(file)) { + cleanUpSpecifiedDirectory(file); + continue; + } + // If we got here we have non-hidden, non-directory files, so we can't // delete this directory throw new PipelineException( "Directory " + dataImportPathForTask.getFileName().toString() @@ -508,8 +312,8 @@ public void performDirectoryCleanup() { } // Delete the directory unless it's the main DR directory - if (!dataImportPathForTask.equals(dataReceiptTopLevelPath)) { - FileUtils.deleteDirectory(dataImportPathForTask.toFile()); + if (!directory.equals(dataReceiptTopLevelPath)) { + FileUtils.deleteDirectory(directory.toFile()); } } catch (IOException e) { throw new UncheckedIOException( @@ -527,40 +331,11 @@ public void processingCompleteTaskAction() { processingComplete = true; } - // Allows a caller to supply an alert service instance for test purposes + // Allows a caller to supply an alert service instance for test purposes. AlertService alertService() { return AlertService.getInstance(); } - // Allows a caller to supply a model importer for test purposes. - ModelImporter modelImporter(Path importDirectory, String description) { - if (modelImporter == null) { - modelImporter = new ModelImporter(importDirectory.toString(), description); - } - return modelImporter; - } - - // Updates the model registry in the current pipeline instance. Package scope - // for testing purposes. - void updateModelRegistryForPipelineInstance() { - ModelCrud modelCrud = new ModelCrud(); - modelCrud.lockCurrentRegistry(); - ModelRegistry modelRegistry = modelCrud.retrieveCurrentRegistry(); - PipelineInstanceCrud pipelineInstanceCrud = new PipelineInstanceCrud(); - PipelineInstance dbInstance = pipelineInstanceCrud - .retrieve(pipelineTask.getPipelineInstance().getId()); - dbInstance.setModelRegistry(modelRegistry); - pipelineInstanceCrud.merge(dbInstance); - pipelineTask.getPipelineInstance().setModelRegistry(modelRegistry); - } - - ManifestCrud manifestCrud() { - if (manifestCrud == null) { - manifestCrud = new ManifestCrud(); - } - return manifestCrud; - } - @Override public String getModuleName() { return "data receipt"; @@ -627,7 +402,7 @@ public void algorithmCompleteTaskAction() { } /** - * Creates the {@link DataReceiptPipelineModule} for import into the database the database. + * Creates the {@link DataReceiptPipelineModule} for import into the database. */ public static PipelineModuleDefinition createDataReceiptPipelineForDb() { @@ -637,6 +412,8 @@ public static PipelineModuleDefinition createDataReceiptPipelineForDb() { ClassWrapper moduleClassWrapper = new ClassWrapper<>( DataReceiptPipelineModule.class); dataReceiptModule.setPipelineModuleClass(moduleClassWrapper); + dataReceiptModule.setUnitOfWorkGenerator( + new ClassWrapper<>(DataReceiptUnitOfWorkGenerator.class)); return dataReceiptModule; } } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java new file mode 100644 index 0000000..75c082d --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinition.java @@ -0,0 +1,445 @@ +package gov.nasa.ziggy.data.management; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.Date; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; + +import org.apache.commons.collections.CollectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.data.datastore.DatastoreNode; +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.data.datastore.DatastoreWalker; +import gov.nasa.ziggy.models.ModelImporter; +import gov.nasa.ziggy.pipeline.definition.ModelRegistry; +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.alert.AlertService.Severity; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.FileUtil; + +/** + * Reference implementation of {@link DataReceiptDefinition}. + *

      + * {@link DatastoreDirectoryDataReceiptDefinition} tests the delivery for conformity by validating + * the delivery {@link Manifest} (all files present, all sizes correct, all checksums correct), and + * ensuring that there are no files in the directory that are not listed in the manifest. Once the + * manifest has been acknowledged, both the manifest and the acknowledgement are copied to the + * pipeline's logs directory. + *

      + * The individual files are tested for conformity by ensuring that the name of each file can be + * translated into a location in the datastore, and that the location conforms to the directory tree + * format of the datastore defined by {@link DatastoreNode} and {@link DatastoreRegexp} instances. + *

      + * Once the validation is complete, the {@link Map} of file locations is generated by applying the + * translation function for the data receipt filename to determine the datastore filename and + * location. + *

      + * To use {@link DatastoreDirectoryDataReceiptDefinition}, it is necessary that the path of each + * file in the delivery directory, relative to the root of that directory, match its ultimate + * location in the datastore, relative to the datastore directory root. In other words, if the + * destination is (datastore-root)/foo/bar/baz, then the location in the data receipt directory must + * be (dr-directory-root)/foo/bar/baz. + * + * @author PT + */ +public class DatastoreDirectoryDataReceiptDefinition implements DataReceiptDefinition { + + private Path dataImportPath; + private Path modelsImportDirectory; + private PipelineTask pipelineTask; + private Manifest manifest; + private ManifestCrud manifestCrud; + private ModelCrud modelCrud; + private PipelineInstanceCrud pipelineInstanceCrud; + private Acknowledgement acknowledgement; + + private List filesForImport; + private List failedImports = new ArrayList<>(); + private List successfulImports = new ArrayList<>(); + private List modelTypes; + private ModelImporter modelImporter; + + private Logger log = LoggerFactory.getLogger(DataReceiptDefinition.class); + + @Override + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public boolean isConformingDelivery() { + + if (!readManifest()) { + log.error("No manifest present in directory {}", dataImportPath.toString()); + return false; + } + + manifest.setImportTime(new Date()); + if (pipelineTask != null) { + manifest.setImportTaskId(pipelineTask.getId()); + } else { + manifest.setImportTaskId(-1L); + } + + if (!checkDatasetId()) { + log.error("Dataset ID {} already used", manifest.getDatasetId()); + return false; + } + + if (!acknowledgeManifest()) { + log.error("Validation of files against the manifest status == INVALID"); + return false; + } + + persistManifest(); + + // Make sure that all the regular files in the directory tree have been validated + // (i.e., there are no files in the directory tree that are absent from the + // manifest). + List filesNotInManifest = filesNotInManifest(); + if (filesNotInManifest.size() != 0) { + log.error("Data receipt directory {} contains files not listed in manifest ", + dataImportPath.toString()); + for (String filename : filesNotInManifest) { + log.error("File missing from manifest: {}", filename); + } + return false; + } + + return true; + } + + /** Reads the manifest and returns true if not null. */ + private boolean readManifest() { + manifest = Manifest.readManifest(dataImportPath); + return manifest != null; + } + + /** Returns true if the dataset ID <= 0 or the dataset ID is not yet present in the database. */ + private boolean checkDatasetId() { + return manifest.getDatasetId() <= 0 + || !manifestCrud().datasetIdExists(manifest.getDatasetId()); + } + + /** Generates acknowledgement and returns true if transfer status is VALID. */ + private boolean acknowledgeManifest() { + acknowledgement = Acknowledgement.of(manifest, dataImportPath, pipelineTask.getId()); + + // Write the acknowledgement to the directory. + acknowledgement.write(dataImportPath); + log.info("Acknowledgement file written: {}", acknowledgement.getName()); + return acknowledgementTransferStatus(); + } + + /** + * Persists the manifest and copies the manifest and acknowledgement to the appropriate + * directory. + */ + private void persistManifest() { + manifest.setAcknowledged(true); + if (manifest.getDatasetId() > 0) { + manifestCrud().persist(manifest); + } + + // Create the manifests directory if it doesn't yet exist + Path manifestDir = DirectoryProperties.manifestsDir(); + try { + Files.createDirectories(manifestDir); + + Files.move(dataImportPath.resolve(manifest.getName()), + manifestDir.resolve(manifest.getName()), StandardCopyOption.REPLACE_EXISTING); + String ackName = Acknowledgement.nameFromManifestName(manifest); + Files.move(dataImportPath.resolve(ackName), manifestDir.resolve(ackName), + StandardCopyOption.REPLACE_EXISTING); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + /** + * Returns the names of any files that are in the data receipt directory that were not in the + * manifest. + */ + private List filesNotInManifest() { + + // Get the names of the files that passed validation (which at this point should + // be the set of all names in the manifest) + List namesOfValidFiles = acknowledgement.namesOfValidFiles(); + + Map regularFilesInDirTree = FileUtil.regularFilesInDirTree(dataImportPath); + List filenamesInDirTree = regularFilesInDirTree.keySet() + .stream() + .map(Path::toString) + .collect(Collectors.toList()); + filenamesInDirTree.removeAll(namesOfValidFiles); + filenamesInDirTree.remove(manifest.getName()); + filenamesInDirTree.remove(acknowledgement.getName()); + return filenamesInDirTree; + } + + /** + * Returns the {@link Path}s for all files to be imported. For + * {@link DatastoreDirectoryDataReceiptDefinition}, this is the collection of regular files in + * the directory tree under the data import directory. + *

      + * The search of the directory tree is only performed on the first call of this method. For all + * subsequent calls, the cached results of the initial call are returned. + */ + @Override + public List filesForImport() { + if (filesForImport == null) { + filesForImport = new ArrayList<>( + FileUtil.regularFilesInDirTree(dataImportPath).values()); + } + return filesForImport; + } + + /** Returns the data files for import. */ + List dataFilesForImport() { + return filesForImport().stream() + .filter(s -> !s.getParent().equals(modelsImportDirectory)) + .collect(Collectors.toList()); + } + + /** Returns the model files for import. */ + List modelFilesForImport() { + return filesForImport().stream() + .filter(s -> s.getParent().equals(modelsImportDirectory)) + .collect(Collectors.toList()); + } + + /** + * Checks an individual file to make sure it conforms to the file name / path standards for the + * data receipt definition. In this case, files in the models subdirectory of the data import + * directory are checked to ensure that their names conform to the naming convention of a known + * model type, while files in other directories are checked to ensure that the path to the given + * file is a legal path according to the datastore definition. + */ + @Override + public boolean isConformingFile(Path file) { + if (!file.isAbsolute()) { + throw new IllegalArgumentException("Path " + file.toString() + " is not absolute"); + } + if (file.getParent().equals(modelsImportDirectory)) { + return isModelType(file); + } + Path location = dataImportPath.relativize(file).getParent(); + if (location == null) { + return false; + } + return datastoreWalker().locationMatchesDatastore(location.toString()); + } + + /** Determines whether a file has the correct name to be a model of a known type. */ + private boolean isModelType(Path file) { + for (ModelType modelType : modelTypes()) { + if (modelType.pattern().matcher(file.getFileName().toString()).matches()) { + return true; + } + } + return false; + } + + /** + * Performs file import. This is accomplished with two separate methods, one of which performs + * data file import and one of which provides model file import. + */ + @Override + public void importFiles() { + importDataFiles(); + importModelFiles(); + } + + /** Imports data files into the datastore. */ + private void importDataFiles() { + + log.info("Importing data files from directory {}...", dataImportPath.toString()); + Path datastoreRoot = DirectoryProperties.datastoreRootDir().toAbsolutePath(); + + List dataFilesForImport = dataFilesForImport(); + + // Generate datastore paths for all data files. + Set datastoreDirectories = new HashSet<>(); + for (Path destPath : dataFilesForImport) { + datastoreDirectories.add(dataImportPath.relativize(destPath).getParent()); + } + for (Path destDir : datastoreDirectories) { + datastoreRoot.resolve(destDir).toFile().mkdirs(); + } + + int exceptionCount = 0; + for (Path file : dataFilesForImport) { + Path destinationFile = datastoreRoot.resolve(dataImportPath.relativize(file)); + try { + move(file, destinationFile); + successfulImports.add(datastoreRoot.relativize(destinationFile)); + } catch (IOException e) { + log.error("Failed to import file " + file.toString(), e); + exceptionCount++; + failedImports.add(datastoreRoot.relativize(destinationFile)); + } + } + log.info("Importing data files from directory {}...done", dataImportPath.toString()); + if (exceptionCount > 0) { + alertService().generateAlert("DefaultDataImporter", Severity.WARNING, + "Data file import encountered " + exceptionCount + " data file import failures, " + + "see log file for details"); + } + } + + /** Datastore file manager moveOrSymlink method broken out to facilitate unit testing. */ + void move(Path source, Path destination) throws IOException { + FileUtil.CopyType.MOVE.copy(source, destination); + } + + /** + * Imports model files. This method needs to be synchronized because there may be more than one + * task attempting to import models, in which case the tasks need to import the model files and + * update the registry one task at a time. + */ + private void importModelFiles() { + synchronized (DatastoreDirectoryDataReceiptDefinition.class) { + + log.info("Importing model files from directory {}...", dataImportPath.toString()); + + ModelImporter importer = modelImporter(); + + // Set up and perform imports + List modelFilesForImport = modelFilesForImport(); + importer.setDataReceiptTaskId(pipelineTask.getId()); + importer.setModelTypesToImport(modelTypes()); + importer.importModels(modelFilesForImport()); + + if (!CollectionUtils.isEmpty(importer.getSuccessfulImports())) { + + // The pipeline instance is supposed to have a model registry with all the current + // models in it. Unfortunately, the instance that contains this importer can't have + // that registry because the models were just imported. Add the registry to the + // instance now. + updateModelRegistryForPipelineInstance(); + successfulImports.addAll(importer.getSuccessfulImports()); + } + + if (!CollectionUtils.isEmpty(importer.getFailedImports())) { + failedImports.addAll(importer.getFailedImports()); + log.warn("{} out of {} model files failed to import", importer.getFailedImports(), + modelFilesForImport.size()); + alertService().generateAndBroadcastAlert("Data Receipt (DR)", pipelineTask.getId(), + AlertService.Severity.WARNING, "Failed to import " + importer.getFailedImports() + + " model files (out of " + modelFilesForImport.size() + ")"); + } + } + } + + @Override + public List successfulImports() { + return successfulImports; + } + + @Override + public List failedImports() { + return failedImports; + } + + // Allows a caller to supply a model importer for test purposes. + ModelImporter modelImporter() { + if (modelImporter == null) { + modelImporter = new ModelImporter(dataImportPath, + "Model imports performed at time " + new Date().toString()); + } + return modelImporter; + } + + // Updates the model registry in the current pipeline instance. Package scope + // for testing purposes. + void updateModelRegistryForPipelineInstance() { + ModelRegistry modelRegistry = (ModelRegistry) DatabaseTransactionFactory + .performTransaction(() -> { + ModelCrud modelCrud = modelCrud(); + modelCrud.lockCurrentRegistry(); + ModelRegistry currentRegistry = modelCrud.retrieveCurrentRegistry(); + PipelineInstanceCrud pipelineInstanceCrud = pipelineInstanceCrud(); + PipelineInstance dbInstance = pipelineInstanceCrud + .retrieve(pipelineTask.getPipelineInstance().getId()); + dbInstance.setModelRegistry(currentRegistry); + pipelineInstanceCrud.merge(dbInstance); + return currentRegistry; + }); + pipelineTask.getPipelineInstance().setModelRegistry(modelRegistry); + } + + ManifestCrud manifestCrud() { + if (manifestCrud == null) { + manifestCrud = new ManifestCrud(); + } + return manifestCrud; + } + + /** + * Sets the data import directory and, as long as we're at it, the models import directory. The + * data import directory must be an absolute path. + */ + @Override + public void setDataImportDirectory(Path dataImportPath) { + if (!dataImportPath.isAbsolute()) { + throw new IllegalArgumentException( + "Data import path " + dataImportPath.toString() + " is not absolute"); + } + this.dataImportPath = dataImportPath; + modelsImportDirectory = dataImportPath.resolve(ModelImporter.DATASTORE_MODELS_SUBDIR_NAME); + } + + @Override + public void setPipelineTask(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; + } + + boolean acknowledgementTransferStatus() { + return acknowledgement.getTransferStatus().equals(DataReceiptStatus.VALID); + } + + DatastoreWalker datastoreWalker() { + return (DatastoreWalker) DatabaseTransactionFactory + .performTransaction(DatastoreWalker::newInstance); + } + + ModelCrud modelCrud() { + if (modelCrud == null) { + modelCrud = new ModelCrud(); + } + return modelCrud; + } + + PipelineInstanceCrud pipelineInstanceCrud() { + if (pipelineInstanceCrud == null) { + pipelineInstanceCrud = new PipelineInstanceCrud(); + } + return pipelineInstanceCrud; + } + + List modelTypes() { + if (modelTypes == null) { + modelTypes = modelCrud().retrieveAllModelTypes(); + } + return modelTypes; + } + + // Allows a caller to supply an alert service instance for test purposes. + AlertService alertService() { + return AlertService.getInstance(); + } +} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastorePathLocator.java b/src/main/java/gov/nasa/ziggy/data/management/DatastorePathLocator.java deleted file mode 100644 index f76d5be..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastorePathLocator.java +++ /dev/null @@ -1,21 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import java.nio.file.Path; - -/** - * Provides logic that determines the path in the datastore to a given file, given the DataStoreFile - * for that file. Each pipeline that uses Ziggy must supply at least one implementation of - * DatastorePathLocator that can be used by pipeline modules and by the DataFileManager. - * - * @author PT - */ -public interface DatastorePathLocator { - - /** - * Determines the Path of a datastore file, given an instance of a DataFileInfo subclass. - * - * @param dataFileInfo non-null, valid instance of DataFileInfo subclass. - * @return Path for corresponding file. - */ - Path datastorePath(DataFileInfo dataFileInfo); -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java index 15b0416..b363649 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumer.java @@ -14,8 +14,6 @@ import jakarta.persistence.Column; import jakarta.persistence.ElementCollection; import jakarta.persistence.Entity; -import jakarta.persistence.EnumType; -import jakarta.persistence.Enumerated; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; @@ -43,10 +41,6 @@ @Table(name = "ziggy_DatastoreProducerConsumer") public class DatastoreProducerConsumer { - public enum DataReceiptFileType { - DATA, MODEL; - } - @Id @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_DatastoreProducerConsumer_generator") @@ -59,10 +53,6 @@ public enum DataReceiptFileType { private long producerId; - @Column - @Enumerated(EnumType.STRING) - private DataReceiptFileType dataReceiptFileType; - @ElementCollection @JoinTable(name = "ziggy_DatastoreProducerConsumer_consumers") private Set consumers = new TreeSet<>(); @@ -71,18 +61,14 @@ public enum DataReceiptFileType { public DatastoreProducerConsumer() { } - public DatastoreProducerConsumer(long producerId, String filename, - DataReceiptFileType dataReceiptFileType) { - checkNotNull(dataReceiptFileType, "dataReceiptFileType"); + public DatastoreProducerConsumer(long producerId, String filename) { checkNotNull(filename, "filename"); - this.dataReceiptFileType = dataReceiptFileType; this.filename = filename; this.producerId = producerId; } - public DatastoreProducerConsumer(PipelineTask pipelineTask, Path datastoreFile, - DataReceiptFileType dataReceiptFileType) { - this(pipelineTask.getId(), datastoreFile.toString(), dataReceiptFileType); + public DatastoreProducerConsumer(PipelineTask pipelineTask, Path datastoreFile) { + this(pipelineTask.getId(), datastoreFile.toString()); } public void setFilename(String filename) { @@ -101,14 +87,6 @@ public void setProducer(long producer) { producerId = producer; } - public DataReceiptFileType getDataReceiptFileType() { - return dataReceiptFileType; - } - - public void setDataReceiptFileType(DataReceiptFileType dataReceiptFileType) { - this.dataReceiptFileType = dataReceiptFileType; - } - public Set getConsumers() { return consumers; } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java index c59e917..576923e 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java +++ b/src/main/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrud.java @@ -1,20 +1,23 @@ package gov.nasa.ziggy.data.management; import java.nio.file.Path; +import java.util.ArrayList; import java.util.Collection; import java.util.HashSet; import java.util.List; import java.util.Set; import java.util.stream.Collectors; +import org.apache.commons.collections.CollectionUtils; + import gov.nasa.ziggy.crud.AbstractCrud; import gov.nasa.ziggy.crud.ZiggyQuery; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumer.DataReceiptFileType; import gov.nasa.ziggy.pipeline.definition.ModelRegistry; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.services.database.DatabaseService; +import jakarta.persistence.criteria.Predicate; /** * CRUD class for {@link DatastoreProducerConsumer}. @@ -41,80 +44,82 @@ public DatastoreProducerConsumerCrud(DatabaseService dbService) { super(dbService); } - /** - * Create or update a producer record for a single file. - * - * @param pipelineTask - * @param datastoreFile - */ - public void createOrUpdateProducer(PipelineTask pipelineTask, Path datastoreFile, - DataReceiptFileType type) { + /** Create or update a producer record for a single file. */ + public void createOrUpdateProducer(PipelineTask pipelineTask, Path datastoreFile) { Set datastoreFileSet = new HashSet<>(); datastoreFileSet.add(datastoreFile); - createOrUpdateProducer(pipelineTask, datastoreFileSet, type); + createOrUpdateProducer(pipelineTask, datastoreFileSet); } - /** - * Create or update a set of files with the their PipelineTask ID as producer. - * - * @param datastoreFiles - * @param pipelineTask - */ - public void createOrUpdateProducer(PipelineTask pipelineTask, Collection datastoreFiles, - DataReceiptFileType type) { + /** Create or update a set of files with the their PipelineTask ID as producer. */ + public void createOrUpdateProducer(PipelineTask pipelineTask, Collection datastoreFiles) { if (datastoreFiles == null || datastoreFiles.isEmpty()) { return; } List datastoreProducerConsumers = retrieveOrCreate(pipelineTask, - datastoreNames(datastoreFiles), type); + datastoreNames(datastoreFiles)); for (DatastoreProducerConsumer datastoreProducerConsumer : datastoreProducerConsumers) { datastoreProducerConsumer.setProducer(pipelineTask.getId()); merge(datastoreProducerConsumer); } } + /** Retrieves / creates {@link DatastoreProducerConsumer}s for a collection of files. */ public List retrieveByFilename(Set datastoreFiles) { - return retrieveOrCreate(null, datastoreNames(datastoreFiles), DataReceiptFileType.DATA); + return retrieveOrCreate(null, datastoreNames(datastoreFiles)); } - /** - * Retrieves the set of names of datastore files consumed by a specified pipeline task. - */ + /** Retrieves the set of names of datastore files consumed by a specified pipeline task. */ public Set retrieveFilesConsumedByTask(long taskId) { + return retrieveFilesConsumedByTasks(Set.of(taskId), null); + } + /** + * Retrieves the set of filenames of datastore files that were consumed by one or more of the + * specified consumer task IDs. If the filenames argument is populated, only files from the + * filenames collection will be included in the return; otherwise, all filenames that have a + * consumer from the collection of consumer IDs will be included. + */ + public Set retrieveFilesConsumedByTasks(Collection consumerIds, + Collection filenames) { ZiggyQuery query = createZiggyQuery( DatastoreProducerConsumer.class, String.class); - query.select(DatastoreProducerConsumer_.filename); - query.where(query.getBuilder() - .isMember(taskId, query.getRoot().get(DatastoreProducerConsumer_.consumers))); - + query.select(DatastoreProducerConsumer_.filename).distinct(true); + if (!CollectionUtils.isEmpty(filenames)) { + query.column(DatastoreProducerConsumer_.filename).chunkedIn(filenames); + } + List predicates = new ArrayList<>(); + for (long consumerId : consumerIds) { + predicates.add(query.getBuilder() + .isMember(consumerId, query.getRoot().get(DatastoreProducerConsumer_.consumers))); + } + Predicate[] predicateArray = new Predicate[predicates.size()]; + for (int predicateIndex = 0; predicateIndex < predicates.size(); predicateIndex++) { + predicateArray[predicateIndex] = predicates.get(predicateIndex); + } + Predicate completePredicate = query.getBuilder().or(predicateArray); + query.where(completePredicate); return new HashSet<>(list(query)); } - /** - * Retrieve producers for a set of files. - */ + /** Retrieve producers for a set of files. */ public Set retrieveProducers(Set datastoreFiles) { if (datastoreFiles == null || datastoreFiles.isEmpty()) { return new HashSet<>(); } Set datastoreNames = datastoreNames(datastoreFiles); - List dpcs = retrieveOrCreate(null, datastoreNames, - DataReceiptFileType.DATA); + List dpcs = retrieveOrCreate(null, datastoreNames); return dpcs.stream() .map(DatastoreProducerConsumer::getProducer) .collect(Collectors.toSet()); } - /** - * Adds a consumer to each of a set of datastore files. - */ + /** Adds a consumer to each of a set of datastore files. */ public void addConsumer(PipelineTask pipelineTask, Set datastoreNames) { if (datastoreNames == null || datastoreNames.isEmpty()) { return; } - List dpcs = retrieveOrCreate(null, datastoreNames, - DataReceiptFileType.DATA); + List dpcs = retrieveOrCreate(null, datastoreNames); dpcs.stream().forEach(s -> addConsumer(s, pipelineTask.getId())); } @@ -128,7 +133,7 @@ public void addNonProducingConsumer(PipelineTask pipelineTask, Set datas return; } List datastoreProducerConsumers = retrieveOrCreate(null, - datastoreNames, DataReceiptFileType.DATA); + datastoreNames); datastoreProducerConsumers.stream().forEach(dpc -> addConsumer(dpc, -pipelineTask.getId())); } @@ -159,7 +164,7 @@ private Set datastoreNames(Collection datastoreFiles) { * versions for files that have database entries and new instances for those that do not. */ protected List retrieveOrCreate(PipelineTask pipelineTask, - Set filenames, DataReceiptFileType type) { + Set filenames) { // Start by finding all the files that already have entries. ZiggyQuery query = createZiggyQuery( @@ -175,8 +180,8 @@ protected List retrieveOrCreate(PipelineTask pipeline long producerId = pipelineTask != null ? pipelineTask.getId() : 0; filenames.removeAll(locatedFilenames); for (String filename : filenames) { - DatastoreProducerConsumer instance = new DatastoreProducerConsumer(producerId, filename, - type); + DatastoreProducerConsumer instance = new DatastoreProducerConsumer(producerId, + filename); persist(instance); datastoreProducerConsumers.add(instance); } @@ -188,9 +193,7 @@ protected List retrieveOrCreate(PipelineTask pipeline return datastoreProducerConsumers; } - /** - * Retrieves all successful imports for a given pipeline instance. - */ + /** Retrieves all successful imports for a given pipeline instance. */ public List retrieveForInstance(long pipelineInstanceId) { // Start with task IDs @@ -205,18 +208,12 @@ public List retrieveForInstance(long pipelineInstance return list(query); } - /** - * Retrieves a count of successful imports for a given pipeline instance. - */ + /** Retrieves a count of successful imports for a given pipeline instance. */ public int retrieveCountForInstance(long pipelineInstanceId) { return retrieveForInstance(pipelineInstanceId).size(); } - /** - * Retrieve all the objects in the database. - * - * @return - */ + /** Retrieve all the objects in the database. */ public List retrieveAll() { return list(createZiggyQuery(DatastoreProducerConsumer.class)); } diff --git a/src/main/java/gov/nasa/ziggy/data/management/DefaultDataImporter.java b/src/main/java/gov/nasa/ziggy/data/management/DefaultDataImporter.java deleted file mode 100644 index 12dcb7b..0000000 --- a/src/main/java/gov/nasa/ziggy/data/management/DefaultDataImporter.java +++ /dev/null @@ -1,139 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import java.io.IOException; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.regex.Pattern; -import java.util.stream.Collectors; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.alert.AlertService.Severity; -import gov.nasa.ziggy.uow.DataReceiptUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.UnitOfWork; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; - -/** - * Default implementation of the {@link DataImporter} interface. This class can be used for data - * receipt subject to the following restrictions: - *

        - *
      1. The unit of work for the data receipt task must be {@link DataReceiptUnitOfWorkGenerator}. - *
      2. The files for receipt must have names that match the task directory regular expression for - * one of the {@link DataFileType} instances passed to the {@link DataImporter} instance. - *
      3. The destination in the datastore for each file must be specified by the datastore file name - * formulation of the relevant {@link DataFileType}. - *
      - * - * @author PT - */ -public class DefaultDataImporter extends DataImporter { - - private final Path dataImportPath; - private AlertService alertService; - - public DefaultDataImporter(PipelineTask pipelineTask, Path dataReceiptPath, - Path datastoreRoot) { - super(pipelineTask, dataReceiptPath, datastoreRoot); - - // Obtain the UOW - UnitOfWork uow = pipelineTask.uowTaskInstance(); - dataImportPath = dataReceiptPath.resolve(DirectoryUnitOfWorkGenerator.directory(uow)); - } - - private Logger log = LoggerFactory.getLogger(DataImporter.class); - - @Override - public boolean validateDelivery() { - return true; - } - - @Override - public boolean validateDataFile(Path dataFile) { - return true; - } - - @Override - public Map dataFiles(List namesOfValidFiles) { - - log.info("Importing data files from directory: " + dataImportPath.toString()); - - // Get the set of input data file types from the pipeline task - Set dataTypes = pipelineTask.getPipelineDefinitionNode() - .getInputDataFileTypes(); - - // Find the files that match one of the data file types, and generate the - // corresponding datastore name - Map dataFiles = new HashMap<>(); - for (DataFileType dataFileType : dataTypes) { - Pattern pattern = dataFileType.fileNamePatternForTaskDir(); - List matchingFilenames = namesOfValidFiles.stream() - .filter(s -> pattern.matcher(s).matches()) - .collect(Collectors.toList()); - log.info("Found " + matchingFilenames.size() + " files that match data type \"" - + dataFileType.getName() + "\""); - for (String filename : matchingFilenames) { - dataFiles.put(Paths.get(filename), - Paths.get(dataFileType.datastoreFileNameFromTaskDirFileName(filename))); - } - } - return dataFiles; - } - - @Override - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - public Set importFiles(Map dataFiles) { - int exceptionCount = 0; - Set importedFiles = new HashSet<>(); - Set datastoreDirectories = new HashSet<>(); - for (Path destPath : dataFiles.values()) { - datastoreDirectories.add(destPath.getParent()); - } - for (Path destDir : datastoreDirectories) { - datastoreRoot.resolve(destDir).toFile().mkdirs(); - } - for (Path sourceFile : dataFiles.keySet()) { - Path fullSourcePath = dataImportPath.resolve(sourceFile); - Path fullDestPath = datastoreRoot.resolve(dataFiles.get(sourceFile)); - try { - moveOrSymlink(fullSourcePath, fullDestPath); - importedFiles.add(sourceFile); - } catch (IOException e) { - log.error("Failed to import file " + sourceFile.toString(), e); - exceptionCount++; - } - } - if (exceptionCount > 0) { - alertService().generateAlert("DefaultDataImporter", Severity.WARNING, - "Data file import encountered " + exceptionCount + " import failures, " - + "see log file for details"); - } - return importedFiles; - } - - // Delegate in order to support testing of the case in which - // an IOException occurs. - void moveOrSymlink(Path fullSourcePath, Path fullDestPath) throws IOException { - DataFileManager.moveOrSymlink(fullSourcePath, fullDestPath); - } - - public Path getDataImportPath() { - return dataImportPath; - } - - AlertService alertService() { - if (alertService == null) { - alertService = AlertService.getInstance(); - } - return alertService; - } -} diff --git a/src/main/java/gov/nasa/ziggy/data/management/FailedImport.java b/src/main/java/gov/nasa/ziggy/data/management/FailedImport.java index 9ff2fe3..d7a1b73 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/FailedImport.java +++ b/src/main/java/gov/nasa/ziggy/data/management/FailedImport.java @@ -2,12 +2,9 @@ import java.nio.file.Path; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumer.DataReceiptFileType; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import jakarta.persistence.Column; import jakarta.persistence.Entity; -import jakarta.persistence.EnumType; -import jakarta.persistence.Enumerated; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; @@ -43,27 +40,15 @@ public class FailedImport { @Column(nullable = false, columnDefinition = "varchar(1000000)", unique = false) private String filename; - @Column - @Enumerated(EnumType.STRING) - private DataReceiptFileType dataReceiptFileType; - // Needed by Hibernate. @SuppressWarnings("unused") private FailedImport() { } - /** - * Public constructor. - * - * @param task {@link PipelineTask} that attempted the import. - * @param filename {@link Path} for the file in datastore format. Note that this path must be - * relative to the datastore root. - * @param dataReceiptFileType Type of file (data or model). - */ - public FailedImport(PipelineTask task, Path filename, DataReceiptFileType dataReceiptFileType) { + /** Public constructor. */ + public FailedImport(PipelineTask task, Path filename) { dataReceiptTaskId = task.getId(); this.filename = filename.toString(); - this.dataReceiptFileType = dataReceiptFileType; } public Long getId() { @@ -89,12 +74,4 @@ public String getFilename() { public void setFilename(String filename) { this.filename = filename; } - - public DataReceiptFileType getDataReceiptFileType() { - return dataReceiptFileType; - } - - public void seDataReceiptFileType(DataReceiptFileType dataReceiptFileType) { - this.dataReceiptFileType = dataReceiptFileType; - } } diff --git a/src/main/java/gov/nasa/ziggy/data/management/FailedImportCrud.java b/src/main/java/gov/nasa/ziggy/data/management/FailedImportCrud.java index 778d0e8..b7bfb69 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/FailedImportCrud.java +++ b/src/main/java/gov/nasa/ziggy/data/management/FailedImportCrud.java @@ -21,11 +21,10 @@ public class FailedImportCrud extends AbstractCrud { /** * Creates a collection of new {@link FailedImport} rows in the database. */ - public void create(PipelineTask pipelineTask, Collection filenames, - DatastoreProducerConsumer.DataReceiptFileType type) { + public void create(PipelineTask pipelineTask, Collection filenames) { for (Path filename : filenames) { - persist(new FailedImport(pipelineTask, filename, type)); + persist(new FailedImport(pipelineTask, filename)); } } diff --git a/src/main/java/gov/nasa/ziggy/data/management/Manifest.java b/src/main/java/gov/nasa/ziggy/data/management/Manifest.java index 8f73d98..d690001 100644 --- a/src/main/java/gov/nasa/ziggy/data/management/Manifest.java +++ b/src/main/java/gov/nasa/ziggy/data/management/Manifest.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -104,7 +104,7 @@ public class Manifest implements HasXmlSchemaFilename { private static final String SCHEMA_FILENAME = "manifest.xsd"; - static final String FILENAME_SUFFIX = "-manifest.xml"; + public static final String FILENAME_SUFFIX = "-manifest.xml"; // Thread pool for checksum calculations static ExecutorService checksumThreadPool = Executors @@ -236,8 +236,9 @@ public void write(Path directory) { public static Manifest readManifest(Path directory) { Manifest manifest = null; ValidatingXmlManager xmlManager = new ValidatingXmlManager<>(Manifest.class); + // TODO : fix this! try (DirectoryStream stream = Files.newDirectoryStream(directory, entry -> { - Path trueFile = DataFileManager.realSourceFile(entry); + Path trueFile = FileUtil.realSourceFile(entry); return Files.isRegularFile(trueFile) && entry.getFileName().toString().endsWith(FILENAME_SUFFIX); })) { @@ -246,7 +247,7 @@ public static Manifest readManifest(Path directory) { throw new IllegalStateException( "Multiple manifest files identified in directory " + directory.toString()); } - manifest = xmlManager.unmarshal(DataFileManager.realSourceFile(entry).toFile()); + manifest = xmlManager.unmarshal(FileUtil.realSourceFile(entry).toFile()); manifest.setName(entry.getFileName().toString()); } } catch (IOException e) { diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/MetricsCli.java b/src/main/java/gov/nasa/ziggy/metrics/report/MetricsCli.java index 35738d5..f3b3207 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/MetricsCli.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/MetricsCli.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java b/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java index 1b01795..358a82b 100644 --- a/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java +++ b/src/main/java/gov/nasa/ziggy/metrics/report/PerformanceReport.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/models/ModelImporter.java b/src/main/java/gov/nasa/ziggy/models/ModelImporter.java index ad22b96..493e12b 100644 --- a/src/main/java/gov/nasa/ziggy/models/ModelImporter.java +++ b/src/main/java/gov/nasa/ziggy/models/ModelImporter.java @@ -1,6 +1,5 @@ package gov.nasa.ziggy.models; -import java.io.File; import java.io.IOException; import java.io.UncheckedIOException; import java.nio.file.Files; @@ -21,14 +20,15 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.data.management.DataFileManager; import gov.nasa.ziggy.pipeline.definition.ModelMetadata; import gov.nasa.ziggy.pipeline.definition.ModelRegistry; import gov.nasa.ziggy.pipeline.definition.ModelType; import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.FileUtil; /** * Imports models of all types from a specified directory. @@ -54,10 +54,10 @@ public class ModelImporter { private static final Logger log = LoggerFactory.getLogger(ModelImporter.class); - private String directory; private Path datastoreRoot; - private ModelCrud modelCrud; - private Path modelsRoot; + private ModelCrud modelCrud = new ModelCrud(); + private Path datastoreModelsRoot; + private Path dataImportPath; String modelDescription; private Set modelTypesToImport = new HashSet<>(); private long dataReceiptTaskId; @@ -66,83 +66,71 @@ public class ModelImporter { public static final String DATASTORE_MODELS_SUBDIR_NAME = "models"; - public ModelImporter(String directory, String modelDescription) { - this.directory = directory; - File directoryFile = new File(directory); - if (!directoryFile.exists() || !directoryFile.isDirectory()) { - throw new IllegalArgumentException( - "Argument " + directory + " is not a directory or does not exist"); - } + public ModelImporter(Path dataImportPath, String modelDescription) { this.modelDescription = modelDescription; - datastoreRoot = DirectoryProperties.datastoreRootDir(); - modelsRoot = datastoreRoot.resolve(Paths.get(ModelImporter.DATASTORE_MODELS_SUBDIR_NAME)); + this.dataImportPath = dataImportPath; + datastoreRoot = DirectoryProperties.datastoreRootDir().toAbsolutePath(); + datastoreModelsRoot = datastoreRoot + .resolve(Paths.get(ModelImporter.DATASTORE_MODELS_SUBDIR_NAME)); } /** * Performs the top level work of the model import process: *
        - *
      1. Identify the files in the directory that are of each model type. + *
      2. Identify the files that are of each model type. *
      3. Add the models to the datastore and the model registry. *
      - * - * @param filenames list of all validated files in the import directory. - * @return true if models were found that required import, false if no models were found to * import. */ - public boolean importModels(List filenames) { + public void importModels(List files) { - log.info("Starting model imports from directory " + directory); - final ModelCrud modelMetadataCrud = modelCrud(); + log.info("Importing models..."); if (modelTypesToImport.isEmpty()) { - modelTypesToImport.addAll(modelMetadataCrud.retrieveAllModelTypes()); + modelTypesToImport.addAll(modelTypes()); log.info("Retrieved " + modelTypesToImport.size() + " model types from database"); } - Map> modelTypeFileNamesMap = new HashMap<>(); + Map> modelFilesByModelType = new HashMap<>(); // build the set of file names for each model type int importFileCount = 0; for (ModelType modelType : modelTypesToImport) { - Map filenamesForModelType = findFilenamesForModelType(filenames, - modelType); + Map filenamesForModelType = findFilenamesForModelType(files, modelType); importFileCount += filenamesForModelType.size(); - modelTypeFileNamesMap.put(modelType, filenamesForModelType); + modelFilesByModelType.put(modelType, filenamesForModelType); } if (importFileCount == 0) { log.info("No models to be imported, exiting"); - return false; + return; } // perform the database portion of the process - ModelRegistry modelRegistry = modelCrud().retrieveUnlockedRegistry(); - for (ModelType modelType : modelTypeFileNamesMap.keySet()) { - addModels(modelRegistry, modelType, modelTypeFileNamesMap.get(modelType)); + ModelRegistry modelRegistry = unlockedRegistry(); + for (ModelType modelType : modelFilesByModelType.keySet()) { + addModels(modelRegistry, modelType, modelFilesByModelType.get(modelType)); } - modelCrud().merge(modelRegistry); + long unlockedModelRegistryId = mergeRegistryAndReturnUnlockedId(modelRegistry); log.info("Update of model registry complete"); - long unlockedModelRegistryId = modelCrud().retrieveUnlockedRegistryId(); + log.info("Importing models...done"); log.info("Current unlocked model registry ID == " + unlockedModelRegistryId); - return true; } /** * Uses the regular expression for a given model type to identify the files that are of that * type. * - * @param filenames List of files in the import directory. - * @param modelType ModelType instance to be used in the search. * @return A Map from the version number of the new files to their names. If the model type in * question does not include a version number in its name, there can be only one file in the * Map. */ - public Map findFilenamesForModelType(List filenames, - ModelType modelType) { + public Map findFilenamesForModelType(List files, ModelType modelType) { // Get all the file names for this model type - Map versionNumberFileNamesMap = new TreeMap<>(); + Map versionNumberFileNamesMap = new TreeMap<>(); Pattern pattern = modelType.pattern(); - for (String filename : filenames) { + for (Path file : files) { + String filename = file.getFileName().toString(); Matcher matcher = pattern.matcher(filename); if (matcher.matches()) { String versionNumber; @@ -151,7 +139,7 @@ public Map findFilenamesForModelType(List filenames, } else { versionNumber = filename; } - versionNumberFileNamesMap.put(versionNumber, filename); + versionNumberFileNamesMap.put(versionNumber, file); } } @@ -173,14 +161,14 @@ public Map findFilenamesForModelType(List filenames, * * @param modelRegistry Current model registry. * @param modelType Type of model to be imported. - * @param versionNumberFileNamesMap Map from version numbers to file names. + * @param modelFilesByVersionId Map from version numbers to file names. */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) private void addModels(ModelRegistry modelRegistry, ModelType modelType, - Map versionNumberFileNamesMap) { + Map modelFilesByVersionId) { // find or make the directory for this type of model - Path modelDir = modelsRoot.resolve(modelType.getType()); + Path modelDir = datastoreModelsRoot.resolve(modelType.getType()).toAbsolutePath(); if (!Files.isDirectory(modelDir)) { try { Files.createDirectories(modelDir); @@ -190,10 +178,10 @@ private void addModels(ModelRegistry modelRegistry, ModelType modelType, } } - Set modelVersions = new TreeSet<>(versionNumberFileNamesMap.keySet()); + Set modelVersions = new TreeSet<>(modelFilesByVersionId.keySet()); for (String version : modelVersions) { - createModel(modelRegistry, modelType, modelDir, versionNumberFileNamesMap.get(version)); - log.info(versionNumberFileNamesMap.size() + " models of type " + modelType.getType() + createModel(modelRegistry, modelType, modelDir, modelFilesByVersionId.get(version)); + log.info(modelFilesByVersionId.size() + " models of type " + modelType.getType() + " added to datastore"); } } @@ -204,56 +192,51 @@ private void addModels(ModelRegistry modelRegistry, ModelType modelType, * model file name does not include a timestamp, one will be added. The model, with these * potential additions to the file name, will then be copied to the correct directory in the * datastore and added to the current model registry. - * - * @param modelRegistry Current version of the registy. - * @param modelType Type of model to add. - * @param modelDir Directory for models of this type in the datastore. - * @param modelName File name for the model in the import directory. */ @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) private void createModel(ModelRegistry modelRegistry, ModelType modelType, Path modelDir, - String modelName) { + Path modelFile) { // The update of the model registry and the move of the file must be done atomically, // and can only be done if the model metadata was successfully created. Thus we do this // in steps. First, create the model metadata, if we can't do so record the failure and // return. ModelMetadata modelMetadata = null; + String modelFilename = modelFile.getFileName().toString(); try { ModelMetadata currentModelRegistryMetadata = modelRegistry .getMetadataForType(modelType); - modelMetadata = modelMetadata(modelType, modelName, modelDescription, + modelMetadata = modelMetadata(modelType, modelFilename, modelDescription, currentModelRegistryMetadata); modelMetadata.setDataReceiptTaskId(dataReceiptTaskId); } catch (Exception e) { - log.error("Unable to create model metadata for file " + modelName); - failedImports.add(Paths.get(modelName)); + log.error("Unable to create model metadata for file " + modelFile); + failedImports.add(dataImportPath.relativize(modelFile)); return; } // Next we move the file, if we can't do so record the failure and return. - Path sourceFile = Paths.get(directory, modelName); Path destinationFile = modelDir.resolve(modelMetadata.getDatastoreFileName()); try { - moveOrSymlink(sourceFile, destinationFile); + move(modelFile, destinationFile); } catch (Exception e) { - log.error("Unable to import file " + modelName + " into datastore"); - failedImports.add(Paths.get(modelName)); + log.error("Unable to import file " + modelFile + " into datastore"); + failedImports.add(dataImportPath.relativize(modelFile)); return; } // If all that worked, then we can update the model registry - modelCrud().persist(modelMetadata); + persistModelMetadata(modelMetadata); modelRegistry.updateModelMetadata(modelMetadata); - log.info("Imported file " + modelName + " to models directory as " + log.info("Imported file " + modelFile + " to models directory as " + modelMetadata.getDatastoreFileName() + " of type " + modelType.getType()); successfulImports.add(datastoreRoot.relativize(destinationFile)); } // The DataFileManager method is broken out in this fashion to facilitate testing. - public void moveOrSymlink(Path src, Path dest) throws IOException { - DataFileManager.moveOrSymlink(src, dest); + public void move(Path src, Path dest) throws IOException { + FileUtil.CopyType.MOVE.copy(src, dest); } // The ModelMetadata constructor is broken out in this fashion to facilitate testing. @@ -287,10 +270,29 @@ public List getSuccessfulImports() { return successfulImports; } - protected ModelCrud modelCrud() { - if (modelCrud == null) { - modelCrud = new ModelCrud(); - } - return modelCrud; + public ModelRegistry unlockedRegistry() { + return (ModelRegistry) DatabaseTransactionFactory + .performTransaction(() -> modelCrud.retrieveUnlockedRegistry()); } + + public void persistModelMetadata(ModelMetadata modelMetadata) { + DatabaseTransactionFactory.performTransaction(() -> { + modelCrud.persist(modelMetadata); + return null; + }); + } + + public long mergeRegistryAndReturnUnlockedId(ModelRegistry modelRegistry) { + return (long) DatabaseTransactionFactory.performTransaction(() -> { + modelCrud.merge(modelRegistry); + return modelCrud.retrieveUnlockedRegistryId(); + }); + } + + @SuppressWarnings("unchecked") + public List modelTypes() { + return (List) DatabaseTransactionFactory + .performTransaction(() -> modelCrud.retrieveAllModelTypes()); + } + } diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java index c4ac353..6009650 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmExecutor.java @@ -10,13 +10,15 @@ import gov.nasa.ziggy.metrics.IntervalMetric; import gov.nasa.ziggy.module.remote.PbsParameters; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.module.remote.SupportedRemoteClusters; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.PipelineTask.ProcessingSummary; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.ProcessingSummaryOperations; import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale;; @@ -37,6 +39,7 @@ public abstract class AlgorithmExecutor { protected final PipelineTask pipelineTask; private ParameterSetCrud parameterSetCrud; + private PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud; private ProcessingSummaryOperations processingSummaryOperations; private StateFile stateFile; @@ -45,7 +48,8 @@ public abstract class AlgorithmExecutor { * Returns a new instance of the appropriate {@link AlgorithmExecutor} subclass. */ public static final AlgorithmExecutor newInstance(PipelineTask pipelineTask) { - return newInstance(pipelineTask, new ParameterSetCrud(), new ProcessingSummaryOperations()); + return newInstance(pipelineTask, new ParameterSetCrud(), new PipelineDefinitionNodeCrud(), + new ProcessingSummaryOperations()); } /** @@ -54,20 +58,21 @@ public static final AlgorithmExecutor newInstance(PipelineTask pipelineTask) { * classes to be mocked for testing. */ static final AlgorithmExecutor newInstance(PipelineTask pipelineTask, - ParameterSetCrud parameterSetCrud, + ParameterSetCrud parameterSetCrud, PipelineDefinitionNodeCrud defNodeCrud, ProcessingSummaryOperations processingSummaryOperations) { if (pipelineTask == null) { log.debug("Pipeline task is null, returning LocalAlgorithmExecutor instance"); return new LocalAlgorithmExecutor(pipelineTask); } - RemoteParameters remoteParams = parameterSetCrud.retrieveRemoteParameters(pipelineTask); + PipelineDefinitionNodeExecutionResources remoteParams = defNodeCrud + .retrieveExecutionResources(pipelineTask.pipelineDefinitionNode()); if (remoteParams == null) { log.debug("Remote parameters null, returning LocalAlgorithmExecutor instance"); return new LocalAlgorithmExecutor(pipelineTask); } - if (!remoteParams.isEnabled()) { + if (!remoteParams.isRemoteExecutionEnabled()) { log.debug("Remote execution not selected, returning LocalAlgorithmExecutor instance"); return new LocalAlgorithmExecutor(pipelineTask); } @@ -112,21 +117,12 @@ protected AlgorithmExecutor(PipelineTask pipelineTask) { /** * Submits the {@link PipelineTask} for execution. This follows a somewhat different code path * depending on whether the submission is the original submission or a resubmission. In the - * event of a resubmission, there is no {@link TaskConfigurationManager} argument required - * because subtask counts can be obtained from the database. - *

      - * In the initial submission, the {@link RemoteParameters} instance that is stored with the - * {@link PipelineTask} is used to generate the parameters for PBS, and the resources requested - * are sufficient to process all subtasks. - *

      - * In a resubmission, the {@link RemoteParameters} instance is retrieved from the database to - * ensure that any changes to parameters made by the user are reflected. In this case, the - * resources requested are scaled back to only what is needed to process the number of remaining - * incomplete subtasks. + * event of a resubmission, there is no {@link TaskConfiguration} argument required because + * subtask counts can be obtained from the database. * * @param inputsHandler Will be null for resubmission. */ - public void submitAlgorithm(TaskConfigurationManager inputsHandler) { + public void submitAlgorithm(TaskConfiguration inputsHandler) { prepareToSubmitAlgorithm(inputsHandler); @@ -136,7 +132,8 @@ public void submitAlgorithm(TaskConfigurationManager inputsHandler) { Files.createDirectories(algorithmLogDir()); Files.createDirectories(DirectoryProperties.stateFilesDir()); Files.createDirectories(taskDataDir()); - SubtaskUtils.clearStaleAlgorithmStates(WorkingDirManager.workingDir(pipelineTask)); + SubtaskUtils.clearStaleAlgorithmStates( + new TaskDirectoryManager(pipelineTask).taskDir().toFile()); log.info("Start remote monitoring (taskId=" + pipelineTask.getId() + ")"); submitForExecution(stateFile); @@ -144,21 +141,21 @@ public void submitAlgorithm(TaskConfigurationManager inputsHandler) { }); } - private void prepareToSubmitAlgorithm(TaskConfigurationManager inputsHandler) { + private void prepareToSubmitAlgorithm(TaskConfiguration inputsHandler) { // execute the external process on a remote host int numSubtasks; PbsParameters pbsParameters = null; + PipelineDefinitionNodeExecutionResources executionResources = (PipelineDefinitionNodeExecutionResources) DatabaseTransactionFactory + .performTransaction(() -> pipelineDefinitionNodeCrud() + .retrieveExecutionResources(pipelineTask.pipelineDefinitionNode())); + // Initial submission: this is indicated by a non-null task configuration manager if (inputsHandler != null) { // indicates initial submission log.info("Processing initial submission of task " + pipelineTask.getId()); - numSubtasks = inputsHandler.numSubTasks(); + numSubtasks = inputsHandler.getSubtaskCount(); - // Generate the state file for the initial submission using the remote parameters - // that are packaged with the pipeline task - RemoteParameters remoteParameters = pipelineTask.getParameters(RemoteParameters.class, - false); - pbsParameters = generatePbsParameters(remoteParameters, numSubtasks); + pbsParameters = generatePbsParameters(executionResources, numSubtasks); // Resubmission: this is indicated by a null task configuration manager, which // means that subtask counts are available in the database @@ -176,9 +173,7 @@ private void prepareToSubmitAlgorithm(TaskConfigurationManager inputsHandler) { / (double) numSubtasks; // Get the current remote parameters - RemoteParameters remoteParameters = parameterSetCrud() - .retrieveRemoteParameters(pipelineTask); - pbsParameters = generatePbsParameters(remoteParameters, + pbsParameters = generatePbsParameters(executionResources, (int) (numSubtasks * subtaskCountScaleFactor)); } @@ -204,8 +199,8 @@ public void resumeMonitoring() { * implementation of {@link AlgorithmExecutor} has specific needs for its PBS command, hence * each needs its own implementation of this method. */ - public abstract PbsParameters generatePbsParameters(RemoteParameters remoteParameters, - int totalSubtaskCount); + public abstract PbsParameters generatePbsParameters( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtaskCount); protected Path algorithmLogDir() { return DirectoryProperties.algorithmLogsDir(); @@ -235,6 +230,13 @@ protected void setParameterSetCrud(ParameterSetCrud parameterSetCrud) { this.parameterSetCrud = parameterSetCrud; } + protected PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud() { + if (pipelineDefinitionNodeCrud == null) { + pipelineDefinitionNodeCrud = new PipelineDefinitionNodeCrud(); + } + return pipelineDefinitionNodeCrud; + } + protected ProcessingSummaryOperations processingSummaryOperations() { if (processingSummaryOperations == null) { processingSummaryOperations = new ProcessingSummaryOperations(); diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycle.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycle.java index e5036e2..2e05195 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycle.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycle.java @@ -16,7 +16,7 @@ public interface AlgorithmLifecycle { * * @param inputs */ - void executeAlgorithm(TaskConfigurationManager inputs); + void executeAlgorithm(TaskConfiguration inputs); /** * Currently generateMemdroneCacheFiles() and doTaskFileCopy(). diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java index 18987b7..1815f47 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmLifecycleManager.java @@ -3,6 +3,7 @@ import java.io.File; import java.io.IOException; import java.io.UncheckedIOException; +import java.nio.file.Path; import org.hibernate.Hibernate; import org.slf4j.Logger; @@ -26,21 +27,22 @@ public class AlgorithmLifecycleManager implements AlgorithmLifecycle { private static final Logger log = LoggerFactory.getLogger(AlgorithmLifecycleManager.class); - private static WorkingDirManager workingDirManager = null; - private File defaultWorkingDir = null; + private TaskDirectoryManager taskDirManager; + private Path taskDir; private PipelineTask pipelineTask; private AlgorithmExecutor executor; public AlgorithmLifecycleManager(PipelineTask pipelineTask) { this.pipelineTask = pipelineTask; + taskDirManager = new TaskDirectoryManager(pipelineTask); // We need an executor at construction time, though it may get replaced later. executor = AlgorithmExecutor.newInstance(pipelineTask); } @Override - public void executeAlgorithm(TaskConfigurationManager inputs) { + public void executeAlgorithm(TaskConfiguration inputs) { // Replace the pipeline task and the executor now, since we have new information // about the task's subtask counts. @@ -63,7 +65,7 @@ public void doPostProcessing() { @Override @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public File getTaskDir(boolean cleanExisting) { - File taskDir = allocateWorkingDir(cleanExisting); + File taskDir = allocateTaskDir(cleanExisting); if (isRemote()) { File stateFileLockFile = new File(taskDir, StateFile.LOCK_FILE_NAME); try { @@ -110,16 +112,6 @@ public AlgorithmExecutor getExecutor() { return executor; } - /** - * Allocate the working directory using the default naming convention: - * INSTANCEID-TASKID-MODULENAME - * - * @return - */ - private File allocateWorkingDir(boolean cleanExisting) { - return allocateWorkingDir(pipelineTask, cleanExisting); - } - /** * Allocate the working directory using the specified prefix. * @@ -127,17 +119,16 @@ private File allocateWorkingDir(boolean cleanExisting) { * @param pipelineTask * @return */ - private File allocateWorkingDir(PipelineTask pipelineTask, boolean cleanExisting) { - synchronized (ExternalProcessPipelineModule.class) { - if (workingDirManager == null) { - workingDirManager = new WorkingDirManager(); - } + private File allocateTaskDir(boolean cleanExisting) { + + if (taskDirManager == null) { + taskDirManager = new TaskDirectoryManager(pipelineTask); } - if (defaultWorkingDir == null) { - defaultWorkingDir = workingDirManager.allocateWorkingDir(pipelineTask, cleanExisting); - log.info("defaultWorkingDir = " + defaultWorkingDir); + if (taskDir == null) { + taskDir = taskDirManager.allocateTaskDir(cleanExisting); + log.info("defaultWorkingDir = " + taskDir); } - return defaultWorkingDir; + return taskDir.toFile(); } } diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java index 1788efb..cb3ff77 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmMonitor.java @@ -22,9 +22,11 @@ import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.PipelineOperations; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingState; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskOperations; import gov.nasa.ziggy.pipeline.definition.crud.ProcessingSummaryOperations; @@ -75,6 +77,17 @@ private enum Disposition { PERSIST { @Override public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) { + StateFile stateFile = new StateFile(pipelineTask.getModuleName(), + pipelineTask.pipelineInstanceId(), pipelineTask.getId()) + .newStateFileFromDiskFile(); + if (stateFile.getNumFailed() != 0) { + log.warn("{} subtasks out of {} failed but task completed", + stateFile.getNumFailed(), stateFile.getNumComplete()); + monitor.alertService() + .generateAndBroadcastAlert("Algorithm Monitor", pipelineTask.getId(), + Severity.WARNING, "Failed subtasks, see logs for details"); + } + log.info("Sending task with id: " + pipelineTask.getId() + " to worker to persist results"); @@ -104,9 +117,8 @@ public void performActions(AlgorithmMonitor monitor, PipelineTask pipelineTask) PipelineTaskCrud pipelineTaskCrud = monitor.pipelineTaskCrud(); PipelineTask dbTask = pipelineTaskCrud.retrieve(pipelineTask.getId()); dbTask.incrementAutoResubmitCount(); - pipelineOperations.setTaskState(pipelineTask, PipelineTask.State.ERROR); - pipelineTaskCrud.merge(dbTask); - return dbTask; + pipelineOperations.setTaskState(dbTask, PipelineTask.State.ERROR); + return pipelineTaskCrud.merge(dbTask); }); // Submit tasks for resubmission at highest priority. @@ -203,7 +215,6 @@ public static Collection remoteTaskStateFiles() { log.info("Starting new monitor for: " + DirectoryProperties.stateFilesDir().toString()); initializeJobMonitor(); - } /** @@ -397,6 +408,10 @@ private void sendTaskToWorker(StateFile remoteState) { Hibernate.initialize(task.getPipelineParameterSets()); Hibernate.initialize(task.getModuleParameterSets()); Hibernate.initialize(task.getPipelineInstance().getId()); + PipelineDefinitionNodeExecutionResources resources = pipelineDefinitionNodeCrud() + .retrieveExecutionResources(task.pipelineDefinitionNode()); + task.setMaxAutoResubmits(resources.getMaxAutoResubmits()); + task.setMaxFailedSubtaskCount(resources.getMaxFailedSubtaskCount()); // Update remote job information pipelineTaskOperations().updateJobs(task); @@ -508,13 +523,14 @@ private Disposition determineDisposition(StateFile state, PipelineTask pipelineT // The total number of bad subtasks includes both the ones that failed and the // ones that never ran / never finished. If there are few enough bad subtasks, // then we can persist results. - if (state.getNumTotal() - state.getNumComplete() <= pipelineTask.maxFailedSubtasks()) { + if (state.getNumTotal() - state.getNumComplete() <= pipelineTask + .getMaxFailedSubtaskCount()) { return Disposition.PERSIST; } // If the task has bad subtasks but the number of automatic resubmits hasn't // been exhausted, then resubmit. - if (pipelineTask.getAutoResubmitCount() < pipelineTask.maxAutoResubmits()) { + if (pipelineTask.getAutoResubmitCount() < pipelineTask.getMaxAutoResubmits()) { return Disposition.RESUBMIT; } @@ -583,6 +599,10 @@ boolean taskIsKilled(long taskId) { return PipelineSupervisor.taskOnKilledTaskList(taskId); } + PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud() { + return new PipelineDefinitionNodeCrud(); + } + /** * Returns the polling interval, in milliseconds. Replace with mocked method for unit testing. */ diff --git a/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java b/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java index 4096c59..c31e99b 100644 --- a/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java +++ b/src/main/java/gov/nasa/ziggy/module/AlgorithmStateFiles.java @@ -20,7 +20,7 @@ public class AlgorithmStateFiles { private static final Logger log = LoggerFactory.getLogger(AlgorithmStateFiles.class); - private static final String HAS_RESULTS = "HAS_RESULTS"; + private static final String HAS_OUTPUTS = "HAS_OUTPUTS"; public enum SubtaskState { // State in which no AlgorithmStateFile is present. Rather than return an actual @@ -56,21 +56,21 @@ public void updateStateCounts(SubtaskStateCounts stateCounts) { private final File processingFlag; private final File completeFlag; private final File failedFlag; - private final File resultsFlag; + private final File outputsFlag; public AlgorithmStateFiles(File workingDir) { processingFlag = new File(workingDir, "." + SubtaskState.PROCESSING.toString()); completeFlag = new File(workingDir, "." + SubtaskState.COMPLETE.toString()); failedFlag = new File(workingDir, "." + SubtaskState.FAILED.toString()); - resultsFlag = new File(workingDir, "." + HAS_RESULTS); + outputsFlag = new File(workingDir, "." + HAS_OUTPUTS); } public static boolean isComplete(File workingDir) { return new AlgorithmStateFiles(workingDir).isComplete(); } - public static boolean hasResults(File workingDir) { - return new AlgorithmStateFiles(workingDir).resultsFlag.exists(); + public static boolean hasOutputs(File workingDir) { + return new AlgorithmStateFiles(workingDir).outputsFlag.exists(); } public void clearState() { @@ -92,7 +92,7 @@ public void clearState() { */ public void clearStaleState() { if (!currentSubtaskState().equals(SubtaskState.COMPLETE)) { - resultsFlag.delete(); + outputsFlag.delete(); } processingFlag.delete(); failedFlag.delete(); @@ -126,11 +126,11 @@ public void updateCurrentState(SubtaskState newState) { } @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void setResultsFlag() { + public void setOutputsFlag() { try { - resultsFlag.createNewFile(); + outputsFlag.createNewFile(); } catch (IOException e) { - throw new UncheckedIOException("Unable to create new file " + resultsFlag.toString(), + throw new UncheckedIOException("Unable to create new file " + outputsFlag.toString(), e); } } diff --git a/src/main/java/gov/nasa/ziggy/module/TaskFileManager.java b/src/main/java/gov/nasa/ziggy/module/BeforeAndAfterAlgorithmExecutor.java similarity index 68% rename from src/main/java/gov/nasa/ziggy/module/TaskFileManager.java rename to src/main/java/gov/nasa/ziggy/module/BeforeAndAfterAlgorithmExecutor.java index 1de4d8e..f3809c4 100644 --- a/src/main/java/gov/nasa/ziggy/module/TaskFileManager.java +++ b/src/main/java/gov/nasa/ziggy/module/BeforeAndAfterAlgorithmExecutor.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -34,27 +34,32 @@ package gov.nasa.ziggy.module; -import org.jfree.util.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; /** - * Manages the movement of data files between the task directory and the subtask directories. + * Performs activities that happen immediately before or immediately after algorithm execution. *

      - * At the start of subtask execution, an instance of a subclass of {@link PipelineInputs} is - * instantiated, and its {@link PipelineInputs#populateSubTaskInputs()} method is called; this puts - * the necessary data and metadata files for execution into the subtask working directory. + * Before algorithm execution, an instance of {@link PipelineInputs} is instantiated and its + * {@link PipelineInputs#beforeAlgorithmExecution()} method executes. After execution, an instance + * of {@link PipelineOutputs} is instantiated, and its + * {@link PipelineOutputs#afterAlgorithmExecution()} method executes. This allows actions that must + * be taken by the {@link SubtaskExecutor}, and which must occur immediately before or after + * algorithm execution, to occur. *

      - * At the end of subtask execution, an instance of a subclass of {@link PipelineOutputs} is - * instantiated, and its {@link PipelineOutputs#populateTaskResults()} and - * {@link PipelineOutputs#setResultsState()} are called. The former method moves any results files - * from the subtask directory to the task directory; the latter determines whether any results were - * produced and sets an appropriate status. + * This class is run by the SubtaskExecutor for a given subtask, once right before SubtaskExecutor + * runs the algorithm and once right after. * * @author PT */ -public final class TaskFileManager { +public final class BeforeAndAfterAlgorithmExecutor { + + private static final Logger log = LoggerFactory + .getLogger(BeforeAndAfterAlgorithmExecutor.class); @AcceptableCatchBlock(rationale = Rationale.CLEANUP_BEFORE_EXIT) public static void main(String[] args) { @@ -64,23 +69,26 @@ public static void main(String[] args) { Class pipelineInputsOutputsClass = Class.forName(fullyQualifiedClassName); if (PipelineInputs.class.isAssignableFrom(pipelineInputsOutputsClass)) { - PipelineInputs p = (PipelineInputs) pipelineInputsOutputsClass + PipelineInputs pipelineInputs = (PipelineInputs) pipelineInputsOutputsClass .getDeclaredConstructor() .newInstance(); - p.populateSubTaskInputs(); + pipelineInputs.beforeAlgorithmExecution(); } else if (PipelineOutputs.class.isAssignableFrom(pipelineInputsOutputsClass)) { PipelineOutputs pipelineOutputs = (PipelineOutputs) pipelineInputsOutputsClass .getDeclaredConstructor() .newInstance(); - pipelineOutputs.populateTaskResults(); - pipelineOutputs.setResultsState(); + pipelineOutputs.afterAlgorithmExecution(); + if (pipelineOutputs.subtaskProducedOutputs()) { + new AlgorithmStateFiles(DirectoryProperties.workingDir().toFile()) + .setOutputsFlag(); + } } else { throw new ModuleFatalProcessingException("Class " + fullyQualifiedClassName + " does not implement PipelineInputsOutputs"); } System.exit(0); } catch (Exception e) { - Log.error("TaskFileManager execution failed", e); + log.error("TaskFileManager execution failed", e); System.exit(1); } } diff --git a/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java b/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java index c8d1b17..aa779a8 100644 --- a/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java +++ b/src/main/java/gov/nasa/ziggy/module/ComputeNodeMaster.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -100,7 +100,7 @@ public class ComputeNodeMaster implements Runnable { private Set subtaskMasters = new HashSet<>(); - private TaskConfigurationManager inputsHandler; + private TaskConfiguration inputsHandler; public ComputeNodeMaster(String workingDir, TaskLog algorithmLog) { this.workingDir = workingDir; @@ -131,7 +131,7 @@ public void initialize() { // It's possible that this node isn't starting until all of the subtasks are // complete! In that case, it should just exit without doing anything else. - monitor = new TaskMonitor(getInputsHandler(), stateFile, taskDir); + monitor = new TaskMonitor(stateFile, taskDir); monitor.updateState(); if (monitor.allSubtasksProcessed()) { log.info("All subtasks processed, ComputeNodeMaster exiting"); @@ -354,9 +354,9 @@ int getStateFileNumTotal() { * Restores the {@link TaskConfigurationHandler} from disk. Package scope so it can be replaced * with a mocked instance. */ - TaskConfigurationManager getInputsHandler() { + TaskConfiguration getTaskConfiguration() { if (inputsHandler == null) { - inputsHandler = TaskConfigurationManager.restore(taskDir); + inputsHandler = TaskConfiguration.deserialize(taskDir); } return inputsHandler; } @@ -392,7 +392,7 @@ boolean allPermitsAvailable() { */ SubtaskServer subtaskServer() { if (subtaskServer == null) { - subtaskServer = new SubtaskServer(coresPerNode, getInputsHandler()); + subtaskServer = new SubtaskServer(coresPerNode, getTaskConfiguration()); } return subtaskServer; } diff --git a/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java new file mode 100644 index 0000000..099ca6c --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputs.java @@ -0,0 +1,250 @@ +package gov.nasa.ziggy.module; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager; +import gov.nasa.ziggy.module.io.ProxyIgnore; +import gov.nasa.ziggy.parameters.ModuleParameters; +import gov.nasa.ziggy.parameters.Parameters; +import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; +import gov.nasa.ziggy.pipeline.definition.ParameterSet; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.uow.UnitOfWork; + +/** + * Reference implementation of the {@link PipelineInputs} interface. + *

      + * {@link DatastoreDirectoryPipelineInputs} provides an inputs class for pipeline modules that use + * the {@link DatastoreDirectoryUnitOfWorkGenerator} to generate units of work. It uses the + * {@link DataFileType} classes that are specified as inputs to the pipeline module to identify the + * directories in the datastore that contain input files for the current module. This is combined + * with information in the task's {@link UnitOfWork} to identify the exact files required for the + * current task. These files are then copied or symlinked to the task directory. The + * {@link DatastoreFileManager} class is also used for many of the low-level file location and file + * copy operations. + *

      + * The class also manages the models required for the pipeline module: the model types that are + * stored with the pipeline definition node are used to copy the current versions of all needed + * models to the task directory. Their names are stored in the modelFilenames member. + *

      + * The class contains an instance of {@link ModuleParameters} that is used to hold the parameter + * sets required for this pipeline module, which in turn are retrieved from the + * {@link PipelineTask}. + * + * @author PT + */ +public class DatastoreDirectoryPipelineInputs implements PipelineInputs { + + @ProxyIgnore + private static final Logger log = LoggerFactory + .getLogger(DatastoreDirectoryPipelineInputs.class); + + private List dataFilenames = new ArrayList<>(); + private List modelFilenames = new ArrayList<>(); + private ModuleParameters moduleParameters = new ModuleParameters(); + + @ProxyIgnore + private PipelineTask pipelineTask; + @ProxyIgnore + private DatastoreFileManager datastoreFileManager; + @ProxyIgnore + private AlertService alertService = new AlertService(); + @ProxyIgnore + private Path taskDirectory; + + public DatastoreDirectoryPipelineInputs() { + } + + /** Locates input files in the datastore using {@link DataFileType} instances. */ + + /** + * Prepares the task directory for processing. Subtasks are generated based on whether the unit + * of work indicates that a single subtask, or multiple subtasks, should be utilized. Data files + * are copied into subtask directories. Module parameters are inserted into the parameterSets + * member. An instance of {@link DatastoreDirectoryPipelineInputs} is serialized to each subtask + * directory, with the input files for the given subtask included in the instance serialized to + * that directory. + */ + @Override + public void copyDatastoreFilesToTaskDirectory(TaskConfiguration taskConfiguration, + Path taskDirectory) { + + log.info("Preparing task directory..."); + + // Determine the files that need to be copied / linked to the task directory. + // The result will be a List of Set instances, one list element for each + // subtask. Later on we'll deal with the possibility that the pipeline definition + // node wants a single subtask. + Map> filesForSubtasks = datastoreFileManager().filesForSubtasks(); + Map modelFilesForTask = datastoreFileManager().modelFilesForTask(); + + // Populate the module parameters + moduleParameters.setModuleParameters(getModuleParameters(getPipelineTask())); + + // Populate the subtasks. + Map> pathsBySubtaskDirectory = datastoreFileManager() + .copyDatastoreFilesToTaskDirectory(new HashSet<>(filesForSubtasks.values()), + modelFilesForTask); + + // Capture the file name regular expressions for output data file types. This will + // be used later to determine whether any given subtask has any outputs. + Set outputDataFileTypes = pipelineTask.pipelineDefinitionNode() + .getOutputDataFileTypes(); + + // Note: for some reason, when I try to use the outputDataFileTypes directly, + // rather than putting them into a new Set, PipelineInputsOutputsUtils + // attempts to serialize the PipelineDefinitionNode. + PipelineInputsOutputsUtils.serializeOutputFileTypesToTaskDirectory( + new HashSet<>(outputDataFileTypes), taskDirectory); + + // Write the inputs to each of the subtask directories, with the correct file names + // in the file names list and the correct model names in the model names list. + for (Map.Entry> entry : pathsBySubtaskDirectory.entrySet()) { + dataFilenames.clear(); + modelFilenames.clear(); + for (Path file : entry.getValue()) { + dataFilenames.add(file.getFileName().toString()); + } + modelFilenames.addAll(modelFilesForTask.values()); + PipelineInputsOutputsUtils.writePipelineInputsToDirectory(this, + getPipelineTask().getModuleName(), entry.getKey()); + } + + taskConfiguration.setSubtaskCount(pathsBySubtaskDirectory.size()); + log.info("Preparing task directory...done"); + } + + /** + * Determines the number of subtasks for a {@link PipelineTask}. This is done by checking to see + * whether the UOW indicates that a single subtask is required, and if not, counting the data + * files of any of the input data file types in the datastore directories that will be used by + * the {@link PipelineTask}. + */ + @Override + public SubtaskInformation subtaskInformation() { + if (singleSubtask()) { + return new SubtaskInformation(getPipelineTask().getModuleName(), + getPipelineTask().uowTaskInstance().briefState(), 1); + } + int subtaskCount = datastoreFileManager().subtaskCount(); + return new SubtaskInformation(getPipelineTask().getModuleName(), + getPipelineTask().uowTaskInstance().briefState(), subtaskCount); + } + + /** + * Returns the module-level and pipeline-level parameter sets. + */ + private List getModuleParameters(PipelineTask pipelineTask) { + + List allParameters = new ArrayList<>(); + log.info("Retrieving module and pipeline parameters"); + allParameters.addAll(getModuleParameters( + getPipelineTask().getPipelineInstance().getPipelineParameterSets())); + allParameters.addAll(getModuleParameters( + getPipelineTask().getPipelineInstanceNode().getModuleParameterSets())); + log.info("Retrieved {} parameter sets", allParameters.size()); + return allParameters; + } + + /** Returns parameter sets from a given {@link Map}. */ + private List getModuleParameters( + Map, ParameterSet> parameterSetMap) { + List parameters = new ArrayList<>(); + + for (ParameterSet parameterSet : parameterSetMap.values()) { + Parameters instance = parameterSet.parametersInstance(); + if (instance instanceof Parameters) { + Parameters defaultInstance = instance; + defaultInstance.setName(parameterSet.getName()); + } + parameters.add(instance); + } + return parameters; + } + + public List getDataFilenames() { + return dataFilenames; + } + + public void setDataFilenames(List filenames) { + dataFilenames = filenames; + } + + public List getModelFilenames() { + return modelFilenames; + } + + public void setModelFilenames(List filenames) { + modelFilenames = filenames; + } + + public void setModuleParameters(ModuleParameters moduleParameters) { + this.moduleParameters = moduleParameters; + } + + public ModuleParameters getModuleParameters() { + return moduleParameters; + } + + AlertService alertService() { + return alertService; + } + + DatastoreFileManager datastoreFileManager() { + if (datastoreFileManager == null) { + datastoreFileManager = new DatastoreFileManager(getPipelineTask(), taskDirectory); + } + return datastoreFileManager; + } + + /** Populates the log stream identifier just prior to algorithm execution. */ + @Override + public void beforeAlgorithmExecution() { + PipelineInputsOutputsUtils.putLogStreamIdentifier(); + } + + @Override + public void writeParameterSetsToTaskDirectory() { + // This isn't actually needed, since the parameter sets are included in the + // DatastoreDirectoryPipelineInputs instance, which is serialized to the + // task directory. + } + + @Override + public void setPipelineTask(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; + } + + @Override + public PipelineTask getPipelineTask() { + return pipelineTask; + } + + boolean singleSubtask() { + return getPipelineTask().getPipelineInstanceNode() + .getPipelineDefinitionNode() + .getSingleSubtask(); + } + + @Override + public void setTaskDirectory(Path taskDirectory) { + this.taskDirectory = taskDirectory; + } + + @Override + public Path getTaskDirectory() { + return taskDirectory; + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java new file mode 100644 index 0000000..0261da5 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputs.java @@ -0,0 +1,112 @@ +package gov.nasa.ziggy.module; + +import java.nio.file.Path; +import java.util.Collection; +import java.util.Set; + +import org.apache.commons.collections.CollectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager; +import gov.nasa.ziggy.module.io.ProxyIgnore; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.util.io.FileUtil; + +/** + * Reference implementation of the {@link PipelineOutputs} interface. + *

      + * {@link DatastoreDirectoryPipelineOutputs} provides an outputs class for pipeline modules that use + * the {@link DatastoreDirectoryUnitOfWorkGenerator} to generate units of work. It makes use of the + * {@link DatastoreFileManager} class and the {@link DataFileType} instances that are used for + * outputs for the current pipeline module. + * + * @author PT + */ +public class DatastoreDirectoryPipelineOutputs implements PipelineOutputs { + + @ProxyIgnore + private static final Logger log = LoggerFactory + .getLogger(DatastoreDirectoryPipelineOutputs.class); + + @ProxyIgnore + private DatastoreFileManager datastoreFileManager; + + @ProxyIgnore + private PipelineTask pipelineTask; + + @ProxyIgnore + private Path taskDirectory; + + public DatastoreDirectoryPipelineOutputs() { + } + + @Override + public Set copyTaskFilesToDatastore() { + + log.info("Moving output files to datastore..."); + Set outputDatastoreFiles = datastoreFileManager().copyTaskDirectoryFilesToDatastore(); + log.info("Moving results files to datastore...done"); + return outputDatastoreFiles; + } + + /** + * Determines whether a given subtask directory contains any output files. This is done by + * loading the collection of output data file types from the task directory and then checking + * the files in the subtask directory for any that match any of the file name regexps for the + * output data file types. + */ + @Override + public boolean subtaskProducedOutputs() { + return subtaskProducedOutputs(PipelineInputsOutputsUtils.taskDir(), + DirectoryProperties.workingDir()); + } + + // Broken out to simplify testing. + boolean subtaskProducedOutputs(Path taskDir, Path workingDir) { + Collection outputDataFileTypes = PipelineInputsOutputsUtils + .deserializedOutputFileTypesFromTaskDirectory(taskDir); + for (DataFileType outputDataFileType : outputDataFileTypes) { + if (!CollectionUtils + .isEmpty(FileUtil.listFiles(workingDir, outputDataFileType.getFileNameRegexp()))) { + return true; + } + } + return false; + } + + @Override + public void afterAlgorithmExecution() { + // In this case we do nothing after algorithm execution. + } + + DatastoreFileManager datastoreFileManager() { + if (datastoreFileManager == null) { + datastoreFileManager = new DatastoreFileManager(getPipelineTask(), taskDirectory); + } + return datastoreFileManager; + } + + @Override + public void setPipelineTask(PipelineTask pipelineTask) { + this.pipelineTask = pipelineTask; + } + + @Override + public PipelineTask getPipelineTask() { + return pipelineTask; + } + + @Override + public void setTaskDirectory(Path taskDirectory) { + this.taskDirectory = taskDirectory; + } + + @Override + public Path getTaskDirectory() { + return taskDirectory; + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/DefaultPipelineInputs.java b/src/main/java/gov/nasa/ziggy/module/DefaultPipelineInputs.java deleted file mode 100644 index 064ca95..0000000 --- a/src/main/java/gov/nasa/ziggy/module/DefaultPipelineInputs.java +++ /dev/null @@ -1,457 +0,0 @@ -package gov.nasa.ziggy.module; - -import java.io.IOException; -import java.io.UncheckedIOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.Collection; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeMap; -import java.util.TreeSet; -import java.util.stream.Collectors; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.data.management.DataFileManager; -import gov.nasa.ziggy.data.management.DataFileType; -import gov.nasa.ziggy.data.management.DatastorePathLocator; -import gov.nasa.ziggy.module.io.ProxyIgnore; -import gov.nasa.ziggy.parameters.ModuleParameters; -import gov.nasa.ziggy.parameters.Parameters; -import gov.nasa.ziggy.parameters.ParametersInterface; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.ModelMetadata; -import gov.nasa.ziggy.pipeline.definition.ModelRegistry; -import gov.nasa.ziggy.pipeline.definition.ModelType; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.services.config.DirectoryProperties; -import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.UnitOfWork; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; - -/** - * Default pipeline inputs class for use by pipeline modules that employ DataFileType instances to - * define their data file needs. The combination of the DataFileType instances and the task unit of - * work make it possible to identify all the files needed by each subtask and to determine the total - * number of subtasks. Class methods can then copy all the files to the task directory and configure - * units of work for each subtask. - *

      - * The class also manages the models required for the pipeline module: the model types that are - * stored with the pipeline definition node are used to copy the current versions of all needed - * models to the task directory. Their names are stored in the modelFilenames member. - *

      - * The DefaultPipelineInputs class can only be used in cases where the pipeline module's unit of - * work is the {@link DatastoreDirectoryUnitOfWorkGenerator} and where the DataFileTypes are used - * for all data files required by the pipeline module; for cases where either a single subtask or - * one subtask per dataset is used; and for cases in which all subtasks can execute in parallel. For - * modules that require more complicated arrangements, users are directed to write their own - * extensions of the PipelineInputs abstract class. - * - * @author PT - */ -public class DefaultPipelineInputs extends PipelineInputs { - - @ProxyIgnore - private static final Logger log = LoggerFactory.getLogger(DefaultPipelineInputs.class); - - private List dataFilenames = new ArrayList<>(); - private List modelFilenames = new ArrayList<>(); - private ModuleParameters moduleParameters = new ModuleParameters(); - private List outputDataFileTypes = new ArrayList<>(); - - @ProxyIgnore - private DataFileManager dataFileManager; - @ProxyIgnore - private AlertService alertService; - - public DefaultPipelineInputs() { - } - - /** - * Constructor for test purposes only. Allows a partially mocked DataFileManager to be inserted. - */ - DefaultPipelineInputs(DataFileManager dataFileManager, AlertService alertService) { - this.dataFileManager = dataFileManager; - this.alertService = alertService; - } - - /** - * This implementation of PipelineInputs does not use a DatastorePathLocator. - */ - @Override - public DatastorePathLocator datastorePathLocator(PipelineTask pipelineTask) { - return null; - } - - @Override - public Set findDatastoreFilesForInputs(PipelineTask pipelineTask) { - - // Obtain the data file types that the module requires - Set dataFileTypes = pipelineTask.getPipelineDefinitionNode() - .getInputDataFileTypes(); - - UnitOfWork uow = pipelineTask.uowTaskInstance(); - - // find the data files for the task - DataFileManager dataFileManager = dataFileManager(DirectoryProperties.datastoreRootDir(), - null, pipelineTask); - - return dataFileManager.dataFilesForInputs( - Paths.get( - uow.getParameter(DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME).getString()), - dataFileTypes); - } - - /** - * Prepares the task directory for processing. All data files are copied to the task directory - * based on the data file types needed for this module and the section of the datastore that the - * unit of work indicates should be used. Subtasks are generated based on whether the unit of - * work indicates that a single subtask, or multiple subtasks, should be utilized. Module - * parameters are inserted into the parameterSets member and serialized to HDF5 in the task - * directory. - */ - @Override - public void copyDatastoreFilesToTaskDirectory(TaskConfigurationManager taskConfigurationManager, - PipelineTask pipelineTask, Path taskDirectory) { - - // Obtain the data file types that the module requires - Set dataFileTypes = pipelineTask.getPipelineDefinitionNode() - .getInputDataFileTypes(); - - // Store the output data file types - outputDataFileTypes - .addAll(pipelineTask.getPipelineDefinitionNode().getOutputDataFileTypes()); - - // Obtain the unit of work - UnitOfWork uow = pipelineTask.uowTaskInstance(); - String directory = DirectoryUnitOfWorkGenerator.directory(uow); - log.info("Unit of work directory: " + directory); - - // populate the module parameters - populateModuleParameters(pipelineTask); - - // Identify the files to be copied from the datastore to the task directory - DataFileManager dataFileManager = dataFileManager(DirectoryProperties.datastoreRootDir(), - taskDirectory, pipelineTask); - Map> dataFilesMap = dataFileManager - .datastoreDataFilesMap(Paths.get(directory), dataFileTypes); - - Set truncatedFilenames = filterDataFilesIfUnequalCounts(dataFilesMap, - pipelineTask.getId()); - - // Copy the data files from the datastore to the task directory - log.info("Copying data files of " + dataFileTypes.size() + " type(s) to working directory " - + taskDirectory.toString()); - dataFileManager.copyDataFilesByTypeToTaskDirectory(dataFilesMap); - log.info("Data file copy completed"); - - // Copy the current models of the required types to the task directory - Set modelTypes = pipelineTask.getPipelineDefinitionNode().getModelTypes(); - ModelRegistry modelRegistry = pipelineTask.getPipelineInstance().getModelRegistry(); - modelFilenames - .addAll(dataFileManager.copyModelFilesToTaskDirectory(modelRegistry, modelTypes, log)); - - // Construct a Map that goes from the truncated file names to a Set of objects - // for each truncated file name - Map> subtaskPathsMap = new TreeMap<>(); - for (String truncatedFileName : truncatedFilenames) { - subtaskPathsMap.put(truncatedFileName, new HashSet<>()); - } - - // Loop over DataFileType instances from the dataFilesMap - for (DataFileType dataFileType : dataFilesMap.keySet()) { - Set datastorePaths = dataFilesMap.get(dataFileType); - for (Path datastorePath : datastorePaths) { - - // For each file, find its truncated name ... - String truncatedFilename = datastorePath.getFileName().toString().split("\\.")[0]; - - // ... and then put the task dir path into that set of paths! - Set subtaskPaths = subtaskPathsMap.get(truncatedFilename); - subtaskPaths.add(Paths.get( - dataFileType.taskDirFileNameFromDatastoreFileName(datastorePath.toString()))); - } - } - - // now we do different things depending on the desired subtask configuration - boolean singleSubtask = DatastoreDirectoryUnitOfWorkGenerator.singleSubtask(uow); - - if (truncatedFilenames.size() != 0) { - if (singleSubtask) { - log.info("Configuring single subtask for task"); - } else { - log.info("Configuring " + truncatedFilenames.size() + " subtasks for task"); - } - } else { - log.info("No files require processing in this task, no subtasks configured"); - } - - Set subtaskFilenamesAllSubtasks = new TreeSet<>(); - for (String truncatedFilename : subtaskPathsMap.keySet()) { - Set subtaskFilenames = new TreeSet<>(); - subtaskPathsMap.get(truncatedFilename) - .stream() - .map(s -> s.getFileName().toString()) - .forEach(s -> subtaskFilenames.add(s)); - subtaskFilenamesAllSubtasks.addAll(subtaskFilenames); - if (!singleSubtask) { - taskConfigurationManager.addFilesForSubtask(subtaskFilenames); - } - } - if (singleSubtask && truncatedFilenames.size() != 0) { - taskConfigurationManager.addFilesForSubtask(subtaskFilenamesAllSubtasks); - } - - // write the contents of this file to HDF5 in the task directory - log.info("Writing parameters to task directory"); - writeToTaskDir(pipelineTask, taskDirectory.toFile()); - log.info("Task directory preparation complete"); - } - - /** - * Handles the case in which the different data file types have different numbers of files - * identified for this UOW. This can happen if, for example, a task combines results from a - * prior task with another source of inputs: in this case, if the user doesn't processes a - * subset of available files in the prior task, the file counts of these two data file types - * will not match. - *

      - * In this case, we assume that the data file type that has the fewest files is the one that - * controls the selection of files in the other types. We also assume that we can use the - * standard approach of matching files from the different data file types: their base names - * should match. Thus we can discard any file that has a base name that is not represented in - * the shortest set of data file paths. - *

      - * - * @param dataFilesMap {@link Map} between the instances of {@link DataFileType}, and the - * {@link Set} of {@link Path} instances found for that type in the datastore. This map is - * altered in place to contain only the files that should be copied to the task directory. - * @return the {@link Set} of truncated file names that are present in this UOW. - */ - private Set filterDataFilesIfUnequalCounts(Map> dataFilesMap, - long pipelineTaskId) { - - List pathSetSizes = new ArrayList<>(); - int minPathSetSize = Integer.MAX_VALUE; - Set shortestSetOfPaths = null; - for (Set paths : dataFilesMap.values()) { - pathSetSizes.add(paths.size()); - if (paths.size() < minPathSetSize) { - shortestSetOfPaths = paths; - minPathSetSize = paths.size(); - } - } - boolean setLengthsMatch = true; - for (int pathSetSize : pathSetSizes) { - setLengthsMatch = setLengthsMatch && pathSetSize == minPathSetSize; - } - - // Now we need to identify the files in each set that match a file in the shortest - // set. First step: construct a set of file base names. - Set baseNames = shortestSetOfPaths.stream() - .map(this::baseName) - .collect(Collectors.toSet()); - - // Here is where we handle the case of mismatched file set lengths. - if (!setLengthsMatch) { - log.warn("Mismatch in data file counts for UOW: " + pathSetSizes.toString()); - alertService().generateAndBroadcastAlert("PI (DefaultPipelineInputs)", pipelineTaskId, - AlertService.Severity.WARNING, - "Mismatch in data file counts for UOW: " + pathSetSizes.toString()); - for (DataFileType dataFileType : dataFilesMap.keySet()) { - Set filteredPaths = dataFilesMap.get(dataFileType) - .stream() - .filter(s -> baseNames.contains(baseName(s))) - .collect(Collectors.toSet()); - - dataFilesMap.put(dataFileType, filteredPaths); - } - } - return new TreeSet<>(baseNames); - } - - private String baseName(Path dataFilePath) { - return dataFilePath.getFileName().toString().split("\\.")[0]; - } - - /** - * Prepares the per-subtask inputs HDF5 file. In the case of the DefaultPipelineInputs, the HDF5 - * file contains only a list of files to be processed by the selected unit of work and all - * parameter sets associated with this processing module. The files are also copied to the - * subtask directory. - */ - @Override - public void populateSubTaskInputs() { - - // Set the subtask information into the thread for logging purposes - PipelineInputsOutputsUtils.putLogStreamIdentifier(); - // Recover the parameter sets from the task directory - readFromTaskDir(); - dataFilenames = new ArrayList<>(); - - Set uowFilenames = filesForSubtask(); - dataFilenames.addAll(uowFilenames); - - Path taskDir = PipelineInputsOutputsUtils.taskDir(); - dataFileManager = new DataFileManager(DirectoryProperties.datastoreRootDir(), taskDir, - null); - log.info(dataFilenames.size() + " filenames added to UOW"); - log.info("Copying inputs files into subtask directory"); - dataFileManager.copyFilesByNameFromTaskDirToWorkingDir(dataFilenames); - - // now copy the models from the task directory to the working directory - if (!modelFilenames.isEmpty()) { - log.info("Copying " + modelFilenames.size() + " model files into subtask directory"); - dataFileManager.copyFilesByNameFromTaskDirToWorkingDir(modelFilenames); - } - log.info("Persisting inputs information to subtask directory"); - writeSubTaskInputs(); - log.info("Persisting inputs completed"); - } - - /** - * Deletes the copies of datastore files used as inputs. This method is run by the - * ExternalProcessPipelineModule after the module processing has completed successfully. - */ - @Override - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void deleteTempInputsFromTaskDirectory(PipelineTask pipelineTask, Path taskDirectory) { - - // Obtain the data file types that the module requires - Set dataFileTypes = pipelineTask.getPipelineDefinitionNode() - .getInputDataFileTypes(); - - // Use the DataFileManager to delete the temporary data files - dataFileManager(null, taskDirectory, pipelineTask) - .deleteDataFilesByTypeFromTaskDirectory(dataFileTypes); - - // Get the model registry and the set of model types - ModelRegistry modelRegistry = pipelineTask.getPipelineInstance().getModelRegistry(); - Set modelTypes = pipelineTask.getPipelineDefinitionNode().getModelTypes(); - - // delete all the model files in the task directory - for (ModelType modelType : modelTypes) { - ModelMetadata modelMetadata = modelRegistry.getModels().get(modelType); - Path modelFile = taskDirectory.resolve(modelMetadata.getOriginalFileName()); - try { - if (Files.isRegularFile(modelFile) || Files.isSymbolicLink(modelFile)) { - Files.delete(modelFile); - } - } catch (IOException e) { - throw new UncheckedIOException("Unable to delete file " + modelFile.toString(), e); - } - } - - } - - /** - * Populates the moduleParameters member with module-level and pipeline-level parameter sets. - */ - protected void populateModuleParameters(PipelineTask pipelineTask) { - - List allParameters = new ArrayList<>(); - log.info("Retrieving module and pipeline parameters"); - allParameters.addAll( - getModuleParameters(pipelineTask.getPipelineInstance().getPipelineParameterSets())); - allParameters.addAll( - getModuleParameters(pipelineTask.getPipelineInstanceNode().getModuleParameterSets())); - log.info("Retrieved " + allParameters.size() + " parameter sets"); - moduleParameters.setModuleParameters(allParameters); - } - - /** - * Determines the number of subtasks for a {@link PipelineTask}. This is done by checking to see - * whether the UOW indicates that a single subtask is required, and if not, counting the data - * files of any of the input data file types in the datastore directory that will be managed by - * the {@link PipelineTask}. - */ - @Override - public SubtaskInformation subtaskInformation(PipelineTask pipelineTask) { - UnitOfWork uow = pipelineTask.uowTaskInstance(); - if (DatastoreDirectoryUnitOfWorkGenerator.singleSubtask(uow)) { - return new SubtaskInformation(pipelineTask.getModuleName(), uow.briefState(), 1, 1); - } - - Set dataFileTypes = pipelineTask.getPipelineDefinitionNode() - .getInputDataFileTypes(); - Path datastoreRoot = DirectoryProperties.datastoreRootDir(); - DataFileManager dataFileManager = dataFileManager(datastoreRoot, null, pipelineTask); - int subtaskCount = dataFileManager.countDatastoreFilesOfType( - dataFileTypes.iterator().next(), - Paths.get(DirectoryUnitOfWorkGenerator.directory(uow))); - return new SubtaskInformation(pipelineTask.getModuleName(), uow.briefState(), subtaskCount, - subtaskCount); - } - - /** - * Inner method for parameter retrieval. - */ - private List getModuleParameters( - Map, ParameterSet> parameterSetMap) { - List parameters = new ArrayList<>(); - - Collection parameterSets = parameterSetMap.values(); - for (ParameterSet parameterSet : parameterSets) { - Parameters instance = parameterSet.parametersInstance(); - if (instance instanceof Parameters) { - Parameters defaultInstance = instance; - defaultInstance.setName(parameterSet.getName()); - } - parameters.add(instance); - } - return parameters; - } - - public void setDataFilenames(List filenames) { - dataFilenames = filenames; - } - - public List getModelFilenames() { - return modelFilenames; - } - - public void setModelFilenames(List filenames) { - modelFilenames = filenames; - } - - public void setModuleParameters(ModuleParameters moduleParameters) { - this.moduleParameters = moduleParameters; - } - - public ModuleParameters getModuleParameters() { - return moduleParameters; - } - - public List getOutputDataFileTypes() { - return outputDataFileTypes; - } - - public void setOutputDataFileTypes(List outputDataFileTypes) { - this.outputDataFileTypes = outputDataFileTypes; - } - - // Package scope so that a partially mocked-out DataFileManager can be supplied. - DataFileManager dataFileManager(Path datastorePath, Path taskDirPath, - PipelineTask pipelineTask) { - if (dataFileManager == null) { - dataFileManager = new DataFileManager(datastorePath, taskDirPath, pipelineTask); - } - return dataFileManager; - } - - private AlertService alertService() { - if (alertService == null) { - alertService = AlertService.getInstance(); - } - return alertService; - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/DefaultPipelineOutputs.java b/src/main/java/gov/nasa/ziggy/module/DefaultPipelineOutputs.java deleted file mode 100644 index 30cb2e6..0000000 --- a/src/main/java/gov/nasa/ziggy/module/DefaultPipelineOutputs.java +++ /dev/null @@ -1,144 +0,0 @@ -package gov.nasa.ziggy.module; - -import static gov.nasa.ziggy.module.PipelineInputsOutputsUtils.moduleName; - -import java.io.File; -import java.nio.file.Path; -import java.util.HashSet; -import java.util.Map; -import java.util.Set; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.data.management.DataFileInfo; -import gov.nasa.ziggy.data.management.DataFileManager; -import gov.nasa.ziggy.data.management.DataFileType; -import gov.nasa.ziggy.data.management.DatastorePathLocator; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; -import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; -import gov.nasa.ziggy.module.io.ProxyIgnore; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; - -/** - * Default pipeline outputs class for pipeline modules that use DataFileType instances to define - * their data file needs. - *

      - * The DefaultPipelineOutputs class can only be used in cases in which the pipeline module produces - * files in the subtask directory that can be copied to the datastore without any reorganization of - * their contents. In cases where some reorganization of the module outputs is required to obtain - * results that can be saved, users are directed to write their own extensions to the - * PipelineOutputs abstract class. - * - * @author PT - */ -public class DefaultPipelineOutputs extends PipelineOutputs { - - @ProxyIgnore - private static final Logger log = LoggerFactory.getLogger(DefaultPipelineOutputs.class); - - @ProxyIgnore - private DataFileManager dataFileManager; - - public DefaultPipelineOutputs() { - } - - /** - * Constructor for test purposes, which allows a modified DataFileManager to be inserted. - */ - DefaultPipelineOutputs(DataFileManager dataFileManager) { - this.dataFileManager = dataFileManager; - } - - /** - * Copies results files from the subtask directory to the task directory. The results files are - * identified by their filenames, which match the regular expressions for outputs data file - * types. The outputs data file types are stored in the DefaultPipelineInputs HDF5 file, which - * must be loaded to obtain the desired information. - */ - @Override - public void populateTaskResults() { - - PipelineInputsOutputsUtils.putLogStreamIdentifier(); - Path taskDir = PipelineInputsOutputsUtils.taskDir(); - log.info("Copying outputs files to task directory..."); - dataFileManager(null, taskDir, null) - .copyDataFilesByTypeFromWorkingDirToTaskDir(outputDataFileTypes()); - log.info("Copying outputs files to task directory...complete"); - } - - private Set outputDataFileTypes() { - Path taskDir = PipelineInputsOutputsUtils.taskDir(); - // Deserialize the DefaultPipelineInputs instance - DefaultPipelineInputs inputs = new DefaultPipelineInputs(); - String filename = ModuleInterfaceUtils.inputsFileName(moduleName()); - hdf5ModuleInterface.readFile(new File(taskDir.toFile(), filename), inputs, true); - return new HashSet<>(inputs.getOutputDataFileTypes()); - } - - /** - * The pipelineResults() method is not used by the DefaultPipelineOutputs workflow. - */ - @Override - public Map pipelineResults() { - return null; - } - - /** - * Moves results files from the task directory to the datastore. - */ - @Override - public void copyTaskDirectoryResultsToDatastore(DatastorePathLocator locator, - PipelineTask pipelineTask, Path taskDir) { - - log.info("Moving results files to datastore..."); - Path datastoreRoot = DirectoryProperties.datastoreRootDir(); - DataFileManager dataFileManager = dataFileManager(datastoreRoot, taskDir, pipelineTask); - Set outputDataFileTypes = pipelineTask.getPipelineDefinitionNode() - .getOutputDataFileTypes(); - dataFileManager.moveDataFilesByTypeToDatastore(outputDataFileTypes); - log.info("Moving results files to datastore...complete"); - } - - /** - * Updates the set of consumers for files that are used as inputs by the pipeline. Only files - * that were used in at least one subtask that completed successfully will be recorded in the - * database. - */ - @Override - public void updateInputFileConsumers(PipelineInputs pipelineInputs, PipelineTask pipelineTask, - Path taskDirectory) { - log.info("Updating input file consumers..."); - Path datastoreRoot = DirectoryProperties.datastoreRootDir(); - DataFileManager dataFileManager = dataFileManager(datastoreRoot, taskDirectory, - pipelineTask); - Set consumedInputFiles = dataFileManager - .datastoreFilesInCompletedSubtasksWithResults( - pipelineTask.getPipelineDefinitionNode().getInputDataFileTypes()); - DatastoreProducerConsumerCrud producerConsumerCrud = new DatastoreProducerConsumerCrud(); - producerConsumerCrud.addConsumer(pipelineTask, consumedInputFiles); - - Set consumedInputFilesWithoutResults = dataFileManager - .datastoreFilesInCompletedSubtasksWithoutResults( - pipelineTask.getPipelineDefinitionNode().getInputDataFileTypes()); - producerConsumerCrud.addNonProducingConsumer(pipelineTask, - consumedInputFilesWithoutResults); - - log.info("Updating input file consumers...complete"); - } - - private DataFileManager dataFileManager(Path datastoreRoot, Path taskDir, - PipelineTask pipelineTask) { - if (dataFileManager == null) { - dataFileManager = new DataFileManager(datastoreRoot, taskDir, pipelineTask); - } - return dataFileManager; - } - - @Override - protected boolean subtaskProducedResults() { - return dataFileManager(null, PipelineInputsOutputsUtils.taskDir(), null) - .workingDirHasFilesOfTypes(outputDataFileTypes()); - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java b/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java index 2a37528..9cb6921 100644 --- a/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/module/ExternalProcessPipelineModule.java @@ -29,12 +29,12 @@ import com.google.common.collect.ImmutableList; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager.InputFiles; import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; import gov.nasa.ziggy.metrics.IntervalMetric; import gov.nasa.ziggy.metrics.Metric; import gov.nasa.ziggy.metrics.ValueMetric; -import gov.nasa.ziggy.module.remote.PbsParameters; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.module.remote.TimestampFile; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineModule; @@ -46,6 +46,7 @@ import gov.nasa.ziggy.pipeline.definition.ProcessingStatePipelineModule; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; @@ -73,40 +74,14 @@ public class ExternalProcessPipelineModule extends PipelineModule // Instance members private AlgorithmLifecycle algorithmManager; private long instanceId; - private TaskConfigurationManager taskConfigurationManager; + private TaskConfiguration taskConfiguration; private String haltStep = "C"; private PipelineInputs pipelineInputs; private PipelineOutputs pipelineOutputs; - protected boolean processingSuccessful; - protected boolean doneLooping; + private boolean processingSuccessful; + private boolean doneLooping; - /** - * Copy datastore files needed as inputs to the specified working directory. - */ - protected void copyDatastoreFilesToTaskDirectory( - TaskConfigurationManager taskConfigurationManager, PipelineTask pipelineTask, - File taskWorkingDirectory) { - pipelineInputs.copyDatastoreFilesToTaskDirectory(taskConfigurationManager, pipelineTask, - taskWorkingDirectory.toPath()); - processingSummaryOperations().updateSubTaskCounts(pipelineTask.getId(), - taskConfigurationManager.getSubtaskCount(), 0, 0); - } - - /** - * Process and store the algorithm result(s), and delete the temporary copies of datastore files - * in the task directory; update input files in the database with new consumers, if necessary. - */ - protected void persistResultsAndDeleteTempFiles(PipelineTask pipelineTask, - ProcessingFailureSummary failureSummary) { - pipelineInputs.deleteTempInputsFromTaskDirectory(pipelineTask, getTaskDir().toPath()); - pipelineOutputs.copyTaskDirectoryResultsToDatastore( - pipelineInputs.datastorePathLocator(pipelineTask), pipelineTask, getTaskDir().toPath()); - pipelineOutputs.updateInputFileConsumers(pipelineInputs, pipelineTask, - getTaskDir().toPath()); - } - - // Constructor public ExternalProcessPipelineModule(PipelineTask pipelineTask, RunMode runMode) { super(pipelineTask, runMode); instanceId = pipelineTask.getPipelineInstance().getId(); @@ -116,18 +91,11 @@ public ExternalProcessPipelineModule(PipelineTask pipelineTask, RunMode runMode) PipelineModuleDefinition pipelineModuleDefinition = pipelineTask.getPipelineInstanceNode() .getPipelineModuleDefinition(); ClassWrapper inputsClass = pipelineModuleDefinition.getInputsClass(); - pipelineInputs = inputsClass.newInstance(); + pipelineInputs = PipelineInputsOutputsUtils.newPipelineInputs(inputsClass, pipelineTask, + taskDirManager().taskDir()); ClassWrapper outputsClass = pipelineModuleDefinition.getOutputsClass(); - pipelineOutputs = outputsClass.newInstance(); - } - - /** - * Indicates that this class should execute {@link processTask} outside of a database - * transaction. - */ - @Override - public boolean processTaskRequiresDatabaseTransaction() { - return false; + pipelineOutputs = PipelineInputsOutputsUtils.newPipelineOutputs(outputsClass, pipelineTask, + taskDirManager().taskDir()); } /** @@ -146,9 +114,7 @@ public List restartModes() { RunMode.RESUME_CURRENT_STEP, RunMode.RESUME_MONITORING); } - // Concrete, non-override methods - - protected File getTaskDir() { + private File getTaskDir() { return algorithmManager().getTaskDir(false); } @@ -159,18 +125,23 @@ protected File getTaskDir() { @Override public void initializingTaskAction() { checkHaltRequest(ProcessingState.INITIALIZING); - incrementProcessingState(); + incrementDatabaseProcessingState(); processingSuccessful = false; } public void checkHaltRequest(ProcessingState state) { String stateShortName = state.shortName(); - if (haltStep.equals(stateShortName)) { + String haltStepShortName = haltStep(); + if (haltStepShortName.equals(stateShortName)) { throw new PipelineException("Halting processing at end of step " + state.toString() - + " due to configuration request for halt after step " + haltStep); + + " due to configuration request for halt after step " + haltStep()); } } + String haltStep() { + return haltStep; + } + /** * Performs inputs marshaling for MARSHALING processing state, also clear all existing producer * task IDs and update the PipelineTask instance after new producer task IDs are set. Updates @@ -190,19 +161,17 @@ public void marshalingTaskAction() { // is also replaced with the updated task. pipelineTask = pipelineTaskCrud().retrieve(taskId()); pipelineTask.clearProducerTaskIds(); - copyDatastoreFilesToTaskDirectory(taskConfigurationManager(), pipelineTask, - taskDir); + copyDatastoreFilesToTaskDirectory(taskConfiguration(), taskDir); }); - taskConfigurationManager().validate(); return null; }); - if (!taskConfigurationManager().isEmpty()) { - taskConfigurationManager().persist(taskDir); + if (taskConfiguration().getSubtaskCount() != 0) { + taskConfiguration().serialize(taskDir); checkHaltRequest(ProcessingState.MARSHALING); // Set the next state, whatever it might be - incrementProcessingState(); + incrementDatabaseProcessingState(); // if there are sub-task inputs, then we can go on to the next step... successful = true; @@ -217,6 +186,17 @@ public void marshalingTaskAction() { processingSuccessful = doneLooping; } + /** + * Copy datastore files needed as inputs to the specified working directory. + */ + void copyDatastoreFilesToTaskDirectory(TaskConfiguration taskConfiguration, + File taskWorkingDirectory) { + pipelineInputs.copyDatastoreFilesToTaskDirectory(taskConfiguration, + taskWorkingDirectory.toPath()); + processingSummaryOperations().updateSubTaskCounts(pipelineTask.getId(), + taskConfiguration.getSubtaskCount(), 0, 0); + } + /** * Perform the necessary processing for state ALGORITHM_SUBMITTING. This is just calling the * algorithm execution method in the algorithm lifecycle object. The processing state is not @@ -227,11 +207,11 @@ public void marshalingTaskAction() { public void submittingTaskAction() { log.info("Processing step: SUBMITTING"); - TaskConfigurationManager tcm = null; + TaskConfiguration taskConfiguration = null; if (runMode.equals(RunMode.STANDARD)) { - tcm = taskConfigurationManager(); + taskConfiguration = taskConfiguration(); } - algorithmManager().executeAlgorithm(tcm); + algorithmManager().executeAlgorithm(taskConfiguration); checkHaltRequest(ProcessingState.ALGORITHM_SUBMITTING); doneLooping = true; processingSuccessful = false; @@ -275,7 +255,7 @@ public void executingTaskAction() { public void algorithmCompleteTaskAction() { checkHaltRequest(ProcessingState.ALGORITHM_COMPLETE); - incrementProcessingState(); + incrementDatabaseProcessingState(); processingSuccessful = false; } @@ -335,7 +315,7 @@ public void storingTaskAction() { performTransaction(() -> { IntervalMetric.measure(STORE_OUTPUTS_METRIC, () -> { // process outputs - persistResultsAndDeleteTempFiles(pipelineTask, failureSummary); + persistResultsAndUpdateConsumers(); return null; }); return null; @@ -343,41 +323,59 @@ public void storingTaskAction() { log.info("Checking for input files that produced no output"); - // Get the names of the input files - Set inputPaths = pipelineInputs().findDatastoreFilesForInputs(pipelineTask); - Set inputFiles = inputPaths.stream() - .map(Path::toString) - .collect(Collectors.toSet()); + // Finally, update status + performTransaction(() -> { + checkHaltRequest(ProcessingState.STORING); + incrementDatabaseProcessingState(); + return null; + }); + doneLooping = true; + processingSuccessful = true; + } + + /** Process and store the algorithm outputs and update producer-consumer database table. */ + void persistResultsAndUpdateConsumers() { + Set outputFiles = pipelineOutputs.copyTaskFilesToDatastore(); - // Get the names of the files successfully consumed by this task - @SuppressWarnings("unchecked") - Set consumedFiles = (Set) performTransaction( - () -> datastoreProducerConsumerCrud().retrieveFilesConsumedByTask(taskId())); + log.info("Creating producer information for output files..."); + datastoreProducerConsumerCrud().createOrUpdateProducer(pipelineTask, + datastorePathsToRelative(outputFiles)); + log.info("Creating producer information for output files...done"); - // Remove the latter set from the former - inputFiles.removeAll(consumedFiles); + log.info("Updating consumer information for input files..."); + InputFiles inputFiles = datastoreFileManager().inputFilesByOutputStatus(); + datastoreProducerConsumerCrud().addConsumer(pipelineTask, + datastorePathsToNames(inputFiles.getFilesWithOutputs())); + datastoreProducerConsumerCrud().addNonProducingConsumer(pipelineTask, + datastorePathsToNames(inputFiles.getFilesWithoutOutputs())); + log.info("Updating consumer information for input files...done"); - // If anything is left, write to the log and generate an alert - if (inputFiles.isEmpty()) { + if (inputFiles.getFilesWithoutOutputs().isEmpty()) { log.info("All input files produced output"); } else { - log.warn(inputFiles.size() + " input files produced no output"); - for (String inputFile : inputFiles) { - log.warn("Input file " + inputFile + " produced no output"); + log.warn("{} input files produced no output", + inputFiles.getFilesWithoutOutputs().size()); + for (Path inputFile : inputFiles.getFilesWithoutOutputs()) { + log.warn("Input file {} produced no output", inputFile.toString()); } AlertService.getInstance() .generateAndBroadcastAlert("Algorithm", taskId(), AlertService.Severity.WARNING, - inputFiles.size() + " input files produced no output, see log for details"); + inputFiles.getFilesWithoutOutputs() + + " input files produced no output, see log for details"); } + } - // Finally, update status - performTransaction(() -> { - checkHaltRequest(ProcessingState.STORING); - incrementProcessingState(); - return null; - }); - doneLooping = true; - processingSuccessful = true; + Set datastorePathsToRelative(Set datastorePaths) { + return datastorePaths.stream() + .map(s -> DirectoryProperties.datastoreRootDir().toAbsolutePath().relativize(s)) + .collect(Collectors.toSet()); + } + + Set datastorePathsToNames(Set datastorePaths) { + return datastorePaths.stream() + .map(s -> DirectoryProperties.datastoreRootDir().toAbsolutePath().relativize(s)) + .map(Path::toString) + .collect(Collectors.toSet()); } @Override @@ -411,7 +409,8 @@ public void processingMainLoop() { // Perform the current action (including advancing to the next // processing state, if appropriate). - getProcessingState().taskAction(this); + ProcessingState processingState = databaseProcessingState(); + processingState.taskAction(this); } } @@ -536,13 +535,6 @@ AlgorithmExecutor executor() { return algorithmManager().getExecutor(); } - StateFile generateStateFile() { - TaskConfigurationManager tcm = taskConfigurationManager(); - PbsParameters pbsParameters = executor().generatePbsParameters( - pipelineTask.getParameters(RemoteParameters.class), tcm.numSubTasks()); - return StateFile.generateStateFile(pipelineTask, pbsParameters, tcm.numSubTasks()); - } - long timestampFileElapsedTimeMillis(TimestampFile.Event startEvent, TimestampFile.Event finishEvent) { return TimestampFile.elapsedTimeMillis(getTaskDir(), startEvent, finishEvent); @@ -567,13 +559,13 @@ public AlgorithmLifecycle algorithmManager() { return algorithmManager; } - TaskConfigurationManager taskConfigurationManager() { - if (taskConfigurationManager == null) { - taskConfigurationManager = new TaskConfigurationManager(getTaskDir()); - taskConfigurationManager.setInputsClass(pipelineInputs.getClass()); - taskConfigurationManager.setOutputsClass(pipelineOutputs.getClass()); + TaskConfiguration taskConfiguration() { + if (taskConfiguration == null) { + taskConfiguration = new TaskConfiguration(getTaskDir()); + taskConfiguration.setInputsClass(pipelineInputs.getClass()); + taskConfiguration.setOutputsClass(pipelineOutputs.getClass()); } - return taskConfigurationManager; + return taskConfiguration; } public long instanceId() { @@ -605,6 +597,14 @@ DatastoreProducerConsumerCrud datastoreProducerConsumerCrud() { return new DatastoreProducerConsumerCrud(); } + DatastoreFileManager datastoreFileManager() { + return new DatastoreFileManager(pipelineTask, getTaskDir().toPath()); + } + + TaskDirectoryManager taskDirManager() { + return new TaskDirectoryManager(pipelineTask); + } + /** For testing only. */ boolean getDoneLooping() { return doneLooping; diff --git a/src/main/java/gov/nasa/ziggy/module/JobMonitor.java b/src/main/java/gov/nasa/ziggy/module/JobMonitor.java index 802e09a..e509bee 100644 --- a/src/main/java/gov/nasa/ziggy/module/JobMonitor.java +++ b/src/main/java/gov/nasa/ziggy/module/JobMonitor.java @@ -66,11 +66,11 @@ default Map exitComment(StateFile stateFile) { } default String getOwner() { - return new String(); + return ""; } default String getServerName() { - return new String(); + return ""; } default QueueCommandManager getQstatCommandManager() { diff --git a/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java b/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java index 380014e..5509c23 100644 --- a/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/LocalAlgorithmExecutor.java @@ -10,7 +10,7 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.remote.PbsParameters; -import gov.nasa.ziggy.module.remote.RemoteParameters; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.services.config.DirectoryProperties; @@ -38,8 +38,8 @@ public LocalAlgorithmExecutor(PipelineTask pipelineTask) { } @Override - public PbsParameters generatePbsParameters(RemoteParameters remoteParameters, - int totalSubtaskCount) { + public PbsParameters generatePbsParameters( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtaskCount) { return null; } diff --git a/src/main/java/gov/nasa/ziggy/module/PipelineInputs.java b/src/main/java/gov/nasa/ziggy/module/PipelineInputs.java index 359ff3c..eebf2a6 100644 --- a/src/main/java/gov/nasa/ziggy/module/PipelineInputs.java +++ b/src/main/java/gov/nasa/ziggy/module/PipelineInputs.java @@ -1,257 +1,56 @@ package gov.nasa.ziggy.module; -import static gov.nasa.ziggy.module.PipelineInputsOutputsUtils.moduleName; -import static gov.nasa.ziggy.module.PipelineInputsOutputsUtils.taskDir; - -import java.io.File; import java.nio.file.Path; -import java.util.ArrayList; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.data.management.DataFileInfo; -import gov.nasa.ziggy.data.management.DataFileManager; -import gov.nasa.ziggy.data.management.DatastorePathLocator; -import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; -import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; import gov.nasa.ziggy.module.io.Persistable; -import gov.nasa.ziggy.module.io.ProxyIgnore; -import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; /** - * Superclass for all pipeline inputs classes. The pipeline inputs class for a given pipeline module - * contains all the information required for that module: data, models, and parameters. This - * information must be assembled from the contents of the pipeline's datastore and relational - * database. The class provides functionality in support of that assembly: + * Defines the capabilities that any pipeline module needs its inputs class to support. The + * functionality required for an inputs class is as follows: *

        - *
      1. Identifies parameter classes that are needed by the pipeline (see - * {@link requiredParameters()}). - *
      2. Identifies datastore files needed to be used as inputs and copies them to the task directory - * (see {@link copyDatastoreFilesToTaskDirectory(TaskConfigurationManager, PipelineTask, Path)}). - *
      3. Supplies an instance of a DatastorePathLocator subclass for use in this task (see - * {@link datastorePathLocator(PipelineTask)}). - *
      4. Serializes parameters to an HDF5 file in the task directory (see - * {@link writeToTaskDir(PipelineTask, File)}). - *
      5. Identifies files in the task directory that contain data and models from the datastore - * required for processing (see {@link resultsFiles()}). Those files were copied to the task - * directory by the pipeline module prior to execution of {@link resultsFiles()}. - *
      6. Reads the parameter file from the task directory (see {@link readFromTaskDir()}). - *
      7. Reads the contents of datastore files that contain data or models required for processing - * (see {@link #readResultsFile(DataFileInfo, PipelineResults)}). - *
      8. Serializes an HDF5 file containing all of the inputs for processing into the sub-task - * directory (see {@link #writeSubTaskInputs()}). + *
      9. Copy or symlink the input files from the datastore to the task directory + * ({@link #copyDatastoreFilesToTaskDirectory(TaskConfiguration, Path)}). + *
      10. Write the pipeline and module parameters to the task directory + * ({@link #writeParameterSetsToTaskDirectory()}). + *
      11. Perform a per-subtask initialization prior to starting the algorithm on a given subtask + * ({@link #beforeAlgorithmExecution()}). + *
      12. Provide information about the task and its subtasks {@link #subtaskInformation()}. *
      + * Direct use of constructors for {@link PipelineInputs} implementations is discouraged. Instead, + * use {@link PipelineInputsOutputsUtils#newPipelineInputs(ClassWrapper, PipelineTask, Path)} to + * ensure that the {@link PipelineTask} and task directory are correctly populated. *

      - * The abstract method {@link populateSubTaskInputs()} performs the steps that read from the task - * directory, populate the members of the inputs class instance, and serialize that instance to the - * sub-task directory using the other methods of the class provided here as tools. - *

      - * The method {@link #deleteTempInputsFromTaskDirectory(PipelineTask, Path)} deletes the files in - * the task directory that were copied to that location by the - * {@link copyDatastoreFilesToTaskDirectory(TaskConfigurationManager, PipelineTask, Path)} method. - * This is executed after the pipeline algorithm has completed, at which time the datastore files - * are superfluous. + * The reference implementation is {@link DatastoreDirectoryPipelineInputs}. * * @author PT */ -public abstract class PipelineInputs implements Persistable { - - private static final Logger log = LoggerFactory.getLogger(PipelineInputs.class); - - @ProxyIgnore - private Hdf5ModuleInterface hdf5ModuleInterface = new Hdf5ModuleInterface(); - - @ProxyIgnore - private Integer subTaskIndex = null; - - /** - * Used to identify the parameter classes that a pipeline requires in order to execute. - * PipelineInputs subclasses should override this method to provide required parameters in cases - * where there are such. - * - * @return List of parameter classes - */ - public List> requiredParameters() { - return new ArrayList<>(); - } - - /** - * Returns an instance of a DatastorePathLocator subclass for use in the pipeline. - * - * @param pipelineTask PipelineTask for the current task - * @return DatastorePathLocator for this task - */ - public abstract DatastorePathLocator datastorePathLocator(PipelineTask pipelineTask); - - /** - * Used by the ExternalProcessPipelineModule, or its subclasses, to identify the files in the - * datastore that are needed in the task directory in order to form the inputs, and copy them to - * that location. - * - * @param taskConfigurationManager TaskConfigurationManager for this task - * @param pipelineTask PipelineTask for this task - * @param taskDirectory task directory for this task - */ - public abstract void copyDatastoreFilesToTaskDirectory( - TaskConfigurationManager taskConfigurationManager, PipelineTask pipelineTask, - Path taskDirectory); - - /** - * Used by {@link ExternalProcessPipelineModule}, or its subclasses, to identify the files in - * the datastore that are provided to the current task as inputs. - * - * @param pipelineTask PipelineTask for this task - * @return a non-null {@link Set} of {@link Path} instances for task inputs - */ - public abstract Set findDatastoreFilesForInputs(PipelineTask pipelineTask); - - /** - * Generates the inputs for a specific sub-task from the contents of the datastore files that - * have been copied to the task directory. - */ - public abstract void populateSubTaskInputs(); - - /** - * Provides an instance of {@link SubtaskInformation} for a given {@link PipelineTask}. This is - * the default implementation, in which there is 1 subtask per task. For inputs classes that - * potentially generate multiple subtasks, this method must be overridden with one that provides - * the correct information. - */ - public SubtaskInformation subtaskInformation(PipelineTask pipelineTask) { - - return new SubtaskInformation(pipelineTask.getModuleName(), - pipelineTask.uowTaskInstance().briefState(), 1, 1); - } +public interface PipelineInputs extends Persistable { /** - * Provides information on whether a given module sets limits on the number of subtasks that can - * be processed in parallel. The default behavior is that modules do not set such limits (i.e., - * all subtasks can potentially be processed in parallel), ergo the default method returns - * false. For pipeline modules that do have such limits, override the default method with one - * that returns true. + * Used by the pipeline module to identify the files in the datastore that are needed in the + * task directory in order to form the inputs, and copy them to that location. */ - public boolean parallelLimits() { - return false; - } + void copyDatastoreFilesToTaskDirectory(TaskConfiguration taskConfiguration, Path taskDirectory); - /** - * Writes a partially-populated input to an HDF5 file in the task directory. This allows the - * pipeline to provide a set of inputs that are common to all sub-tasks in the task directory. - * This is intended to be used by the worker, which has access to the pipeline task instance. - */ - public void writeToTaskDir(PipelineTask pipelineTask, File taskDir) { - String filename = ModuleInterfaceUtils.inputsFileName(pipelineTask.getModuleName()); - log.info("Writing partial inputs to file " + filename + " in task dir"); - File inputInTaskDir = new File(taskDir, filename); - hdf5ModuleInterface.writeFile(inputInTaskDir, this, true); - } + /** Provides information about the task and its subtasks. */ + SubtaskInformation subtaskInformation(); /** - * Reads a partially-populated input from an HDF5 file in the task directory. This allows the - * pipeline to provide a set of inputs that are common to all sub-tasks in the task directory, - * and this can be used as a starting point for populating the sub-task inputs. + * Performs any preparation that has to happen after the supervisor hands off the task to a + * worker, but before a given subtask's algorithm executes. */ - public void readFromTaskDir() { - String filename = ModuleInterfaceUtils.inputsFileName(moduleName()); - log.info("Populating inputs object from file " + filename + " in task dir"); - File inputInTaskDir = taskDir().resolve(filename).toFile(); - hdf5ModuleInterface.readFile(inputInTaskDir, this, true); - } + void beforeAlgorithmExecution(); - /** - * Returns a non-{@code null} set of DataFileInfo subclasses that are needed to populate the - * initial pipeline task inputs. Concrete subclasses of this class should override this with a - * method that returns the needed DatastoreId classes. - */ - public Set> requiredDataFileInfoClasses() { - return Collections.emptySet(); - } + /** Writes the pipeline and module parameters to the task directory. */ + void writeParameterSetsToTaskDirectory(); - /** - * Returns the sub-task index. Assumes that the working directory is the sub-task directory. - */ - public int subtaskIndex() { - if (subTaskIndex == null) { - String regex = "st-(\\d+)"; - Pattern pattern = Pattern.compile(regex); - File userDir = DirectoryProperties.workingDir().toFile(); - String subTaskDirName = userDir.getName(); - Matcher m = pattern.matcher(subTaskDirName); - m.matches(); - subTaskIndex = Integer.valueOf(m.group(1)); - } - return subTaskIndex; - } + void setPipelineTask(PipelineTask pipelineTask); - /** - * Returns the files for the current subtask, based on the contents of a serialized instance of - * {@link TaskConfigurationManager}. - */ - public Set filesForSubtask() { - return TaskConfigurationManager.restoreAndRetrieveFilesForSubtask( - PipelineInputsOutputsUtils.taskDir().toFile(), subtaskIndex()); - } + PipelineTask getPipelineTask(); - /** - * Returns a map from DataFileInfo subclasses to files in the parent directory that can be - * managed by each subclass. - */ - public Map, Set> resultsFiles( - Set> dataFileInfoClasses) { - return new DataFileManager().dataFilesMap(taskDir(), dataFileInfoClasses); - } + void setTaskDirectory(Path taskDirectory); - /** - * Returns a map from DataFileInfo subclasses to files in the parent directory that can be - * managed by each subclass, where the set of subclasses is the set of all DataFileInfo - * subclasses required by a given PipelineInputs class. - */ - public Map, Set> resultsFiles() { - return resultsFiles(requiredDataFileInfoClasses()); - } - - /** - * Loads an HDF5 file into a PipelineResults instance. - */ - public void readResultsFile(S dataFileInfo, - T resultsInstance) { - log.info("Reading data file " + dataFileInfo.getName().toString()); - hdf5ModuleInterface.readFile(taskDir().resolve(dataFileInfo.getName()).toFile(), - resultsInstance, true); - } - - /** - * Saves the object as an HDF5 file in the sub-task directory. - */ - public void writeSubTaskInputs() { - String moduleName = moduleName(); - String filename = ModuleInterfaceUtils.inputsFileName(moduleName); - log.info("Writing file " + filename + " to sub-task directory"); - hdf5ModuleInterface.writeFile(DirectoryProperties.workingDir() - .resolve(ModuleInterfaceUtils.inputsFileName(moduleName)) - .toFile(), this, true); - ModuleInterfaceUtils.writeCompanionXmlFile(this, moduleName); - } - - /** - * Deletes temporary copies of datastore files used as task inputs from the task directory. - * - * @param pipelineTask pipeline task that used the inputs - * @param taskDirectory Directory to be cleared of temporary inputs - */ - public void deleteTempInputsFromTaskDirectory(PipelineTask pipelineTask, Path taskDirectory) { - DataFileManager fileManager = new DataFileManager(null, null, taskDirectory); - Set inputsSet = fileManager.datastoreFiles(taskDirectory, - requiredDataFileInfoClasses()); - fileManager.deleteFromTaskDirectory(inputsSet); - } + Path getTaskDirectory(); } diff --git a/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java b/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java index 38ebe56..2981eaf 100644 --- a/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtils.java @@ -1,10 +1,24 @@ package gov.nasa.ziggy.module; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.UncheckedIOException; import java.nio.file.Path; +import java.util.Collection; +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; +import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; import gov.nasa.ziggy.module.io.Persistable; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; /** * Provides utility functions for PipelineInputs and PipelineOutputs classes. @@ -13,6 +27,8 @@ */ public abstract class PipelineInputsOutputsUtils implements Persistable { + private static final String SERIALIZED_OUTPUTS_TYPE_FILE = ".output-types.ser"; + /** * Returns the task directory. Assumes that the working directory is the sub-task directory. */ @@ -25,7 +41,11 @@ public static Path taskDir() { * directory. */ public static String moduleName() { - String taskDirString = taskDir().getFileName().toString(); + return moduleName(taskDir()); + } + + public static String moduleName(Path taskDir) { + String taskDirString = taskDir.getFileName().toString(); PipelineTask.TaskBaseNameMatcher m = new PipelineTask.TaskBaseNameMatcher(taskDirString); return m.moduleName(); } @@ -38,4 +58,77 @@ public static void putLogStreamIdentifier() { String subtaskName = DirectoryProperties.workingDir().getFileName().toString(); SubtaskUtils.putLogStreamIdentifier(subtaskName); } + + /** Writes an instance of {@link PipelineInputs} to a directory. */ + public static void writePipelineInputsToDirectory(PipelineInputs inputs, String moduleName, + Path directory) { + String filename = ModuleInterfaceUtils.inputsFileName(moduleName); + File inputInTaskDir = new File(directory.toFile(), filename); + new Hdf5ModuleInterface().writeFile(inputInTaskDir, inputs, true); + } + + /** Reads an instance of {@link PipelineInputs} from a directory. */ + public static void readPipelineInputsFromDirectory(PipelineInputs inputs, String moduleName, + Path directory) { + String filename = ModuleInterfaceUtils.inputsFileName(moduleName); + File inputInTaskDir = new File(directory.toFile(), filename); + new Hdf5ModuleInterface().readFile(inputInTaskDir, inputs, true); + } + + /** + * Returns an instance of {@link PipelineInputs} with its {@link PipelineTask} and {@link Path} + * to the task directory initialized. + */ + public static PipelineInputs newPipelineInputs(ClassWrapper inputsClass, + PipelineTask pipelineTask, Path taskDirectory) { + PipelineInputs pipelineInputs = inputsClass.newInstance(); + pipelineInputs.setPipelineTask(pipelineTask); + pipelineInputs.setTaskDirectory(taskDirectory); + return pipelineInputs; + } + + /** + * Returns an instance of {@link PipelineInputs} with its {@link PipelineTask} and {@link Path} + * to the task directory initialized. + */ + public static PipelineOutputs newPipelineOutputs(ClassWrapper outputsClass, + PipelineTask pipelineTask, Path taskDirectory) { + PipelineOutputs pipelineOutputs = outputsClass.newInstance(); + pipelineOutputs.setPipelineTask(pipelineTask); + pipelineOutputs.setTaskDirectory(taskDirectory); + return pipelineOutputs; + } + + /** Serializes the output data file types for a task to the task directory. */ + public static void serializeOutputFileTypesToTaskDirectory( + Collection outputDataFileTypes, Path taskDirectory) { + Path serializationPath = taskDirectory.resolve(SERIALIZED_OUTPUTS_TYPE_FILE); + try (ObjectOutputStream oos = new ObjectOutputStream( + new FileOutputStream(serializationPath.toFile()))) { + oos.writeObject(outputDataFileTypes); + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to persist output data file types to " + taskDirectory.toString(), e); + } + } + + /** Deserializes the output data file types for a task from the task directory. */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) + @SuppressWarnings("unchecked") + public static Collection deserializedOutputFileTypesFromTaskDirectory( + Path taskDirectory) { + Path deserializationPath = taskDirectory.resolve(SERIALIZED_OUTPUTS_TYPE_FILE); + try (ObjectInputStream ois = new ObjectInputStream( + new FileInputStream(deserializationPath.toFile()))) { + return (Collection) ois.readObject(); + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to deserialize output file types from " + taskDirectory.toString(), e); + } catch (ClassNotFoundException e) { + // This should never occur because the DataFileType class is guaranteed to on the + // classpath and Collection is part of Java. + throw new AssertionError(e); + } + } } diff --git a/src/main/java/gov/nasa/ziggy/module/PipelineOutputs.java b/src/main/java/gov/nasa/ziggy/module/PipelineOutputs.java index 4bcd726..c512fb8 100644 --- a/src/main/java/gov/nasa/ziggy/module/PipelineOutputs.java +++ b/src/main/java/gov/nasa/ziggy/module/PipelineOutputs.java @@ -1,104 +1,42 @@ package gov.nasa.ziggy.module; -import static gov.nasa.ziggy.module.PipelineInputsOutputsUtils.moduleName; -import static gov.nasa.ziggy.module.PipelineInputsOutputsUtils.taskDir; - -import java.io.File; -import java.io.FileFilter; import java.nio.file.Path; -import java.util.Collections; -import java.util.Map; import java.util.Set; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.data.management.DataFileInfo; -import gov.nasa.ziggy.data.management.DataFileManager; -import gov.nasa.ziggy.data.management.DatastorePathLocator; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; -import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; -import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; import gov.nasa.ziggy.module.io.Persistable; -import gov.nasa.ziggy.module.io.ProxyIgnore; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; /** - * Superclass for all pipeline outputs classes. The pipeline outputs class for a given pipeline - * module contains all the results of processing a given sub-task for a given module. The class also - * performs the following functions: + * Defines the capabilities that any pipeline module needs its outputs to support. The functionality + * required for an outputs class is as follows: *

        - *
      1. Identifies all the subclasses of DataFileInfo that are produced by the pipeline - * (see @link{reqiredDataFileInfoClasses()}). - *
      2. Identifies all the files in the sub-task directory that are pipeline outputs based on the - * name convention for pipeline outputs files (see @link{outputFiles()}).
        3. - *
      3. Deserializes the contents of a specific sub-task outputs HDF5 file - * (see @link{readSubTaskOutputs(File file)}).
        4. - *
      4. Uses the contents of a sub-task's outputs to construct a map between DatastoreId subclass - * instances and PipelineResults subclass instances - * (see @link{createPipelineResultsByDatastoreIdMap()}).
        5. - *
      5. Detects the task ID by parsing the task directory name so that this can be added to all - * PipelineResults subclass instances (see @link{originator()}).
        6. - *
      6. Serializes the PipelineResults subclass instances as HDF5 in the task directory - * (see @link{saveResultsToTaskDir()}).
        7. + *
      7. Copy outputs files from the task directory back to the datastore + * ({@link #copyTaskFilesToDatastore()}). + *
      8. Determine whether a given subtask produced any outputs {@link #subtaskProducedOutputs()}). + *
      9. Perform any necessary actions after execution of the processing algorithm + * ({@link #afterAlgorithmExecution()}). *
      - * Once the above steps have been performed, the pipeline module can copy the results files from the - * task directory to the datastore. *

      - * The abstract method @link{populateTaskResults()} manages the process of reading outputs, - * redistributing their contents to PipelineResults subclass instances, and writing same to the task - * directory. + * The {@link PipelineOutputs} interface also provides a number of default methods that can be used + * by implementations in the course of their duties. + *

      + * Users are discouraged from calling constructors directly when desirous of instantiating an object + * of a {@link PipelineOutputs} implementation. Instead, use + * {@link PipelineInputsOutputsUtils#newPipelineOutputs(gov.nasa.ziggy.pipeline.definition.ClassWrapper, PipelineTask, Path)}. + * This will ensure that the {@link PipelineTask} and task directory are correctly populated. *

      - * The method @link{copyTaskDirectoryResultsToDatastore(DatastorePathLocator datastorePathLocator, - * PipelineTask pipelineTask, Path taskDirectory, ProcessingFailureSummary failureSummary) provides - * a standard default method that can be used to copy results files from the task directory to the - * datastore, and to delete unneeded datastore file copies from the task directory. + * The reference implementation of {@link PipelineOutputs} is + * {@link DatastoreDirectoryPipelineOutputs}. * * @author PT */ -public abstract class PipelineOutputs implements Persistable { - - private static final Logger log = LoggerFactory.getLogger(PipelineOutputs.class); - - @ProxyIgnore - private Long originator; - - @ProxyIgnore - Hdf5ModuleInterface hdf5ModuleInterface = new Hdf5ModuleInterface(); +public interface PipelineOutputs extends Persistable { /** - * Returns a non-{@code null} set of DataFileInfo subclasses that are produced by the pipeline - * module. Concrete subclasses of this class should override this with a method that returns the - * needed DatastoreId classes. + * Converts the contents of the outputs file in the subtask directory into one or more files in + * the task directory, and returns the datastore paths of the resulting file copies. */ - public Set> requiredDataFileInfoClasses() { - return Collections.emptySet(); - } - - /** - * Converts the contents of the outputs file in the sub-task directory into one or more results - * files in the task directory. - */ - public abstract void populateTaskResults(); - - /** - * Determines whether the subtask that was processed in the current working directory produced - * results, and if so creates a zero-length file in the directory that indicates that results - * were produced. The zero-length file is created by - * {@link AlgorithmStateFiles#setResultsFlag()}. The determination regarding the presence or - * absence of results is performed by the abstract boolean method - * {@link #subtaskProducedResults()}. This allows the persisting code to determine, for each - * subtask, whether or not that subtask produced results. Note that a subtask can run to - * completion but not produce results. - */ - public void setResultsState() { - if (subtaskProducedResults()) { - new AlgorithmStateFiles(DirectoryProperties.workingDir().toFile()).setResultsFlag(); - } - } + Set copyTaskFilesToDatastore(); /** * Determines whether the subtask that ran in the current working directory produced results. A @@ -110,138 +48,21 @@ public void setResultsState() { * determination of whether results were produced, which would allow some results files to be * necessary to the determination but others optional. * - * @return true if all required results files were produced for a given subtask, false - * otherwise. - */ - protected abstract boolean subtaskProducedResults(); - - /** - * Returns an array of files with names that match the pipeline convention for output files for - * a the given CSCI. - * - * @return - */ - public File[] outputFiles() { - File workingDir = DirectoryProperties.workingDir().toFile(); - File[] detectedFiles = workingDir.listFiles((FileFilter) pathname -> { - String filename = pathname.getName(); - Pattern p = ModuleInterfaceUtils.outputsFileNamePattern(moduleName()); - Matcher m = p.matcher(filename); - return m.matches(); - }); - log.info("Number of output files detected: " + detectedFiles.length); - return detectedFiles; - } - - /** - * Populates the outputs instance from an HDF5 file - * - * @param file - */ - public void readSubTaskOutputs(File file) { - log.info("Reading file " + file.getName() + " into memory"); - hdf5ModuleInterface.readFile(file, this, true); - } - - /** - * Generates a map from DataFileInfo to pipeline results instances that are populated from this - * PipelineOutputs instance. - * - * @return + * @return true if required results files were produced for a given subtask, false otherwise. */ - public abstract Map pipelineResults(); - - /** - * Returns the originator for this set of outputs. - * - * @return - */ - public long originator() { - if (originator == null) { - String taskDirName = taskDir().getFileName().toString(); - String regex = "\\d+-(\\d+)-\\w+"; - Pattern pattern = Pattern.compile(regex); - Matcher matcher = pattern.matcher(taskDirName); - matcher.matches(); - originator = Long.valueOf(matcher.group(1)); - } - return originator; - } - - /** - * Saves the results to HDF5 files in the task directory. The results that are saved are all the - * ones produced by the PipelineResults() method. The results instances are populated with the - * originator prior to saving. - */ - public void saveResultsToTaskDir() { - saveResultsToTaskDir(pipelineResults()); - } - - /** - * Saves results to HDF5 files in the task directory. The results that are saved must be in a - * caller-provided Map between DatastoreId instances and PipelineResults instances. The results - * instances are populated with the originator prior to saving. - * - * @param resultsMap - */ - public void saveResultsToTaskDir(Map resultsMap) { - - for (DataFileInfo dataFileInfo : resultsMap.keySet()) { - PipelineResults result = resultsMap.get(dataFileInfo); - result.setOriginator(originator()); - log.info("Writing file " + dataFileInfo.getName().toString() + " to task directory"); - hdf5ModuleInterface.writeFile(taskDir().resolve(dataFileInfo.getName()).toFile(), - result, true); - } - } - - /** - * Performs the final persistence of results from the pipeline and clean-up of datastore files - * in the task directory. NB: if a pipeline requires a more complex final persistence and - * clean-up than is provided here, the outputs class for that pipeline should override this - * method. - * - * @param datastorePathLocator Instance of a DatastorePathLocator subclass for this task. - * @param pipelineTask Pipeline task for this task. - * @param taskDirectory Task directory for this task. - */ - public void copyTaskDirectoryResultsToDatastore(DatastorePathLocator datastorePathLocator, - PipelineTask pipelineTask, Path taskDirectory) { - - DataFileManager fileManager = new DataFileManager(datastorePathLocator, pipelineTask, - taskDirectory); - - // Move the results files from the task directory to the datastore. - Set> dataFileInfoClasses = requiredDataFileInfoClasses(); - Set outputsSet = fileManager.datastoreFiles(taskDirectory, - dataFileInfoClasses); - fileManager.moveToDatastore(outputsSet); - } + boolean subtaskProducedOutputs(); /** - * Updates the set of consumers for files that are used as inputs by the pipeline. Only files - * that were used in at least one subtask that completed successfully will be recorded in the - * database. + * Performs any activities that must be performed in the subtask directory immediately after + * algorithm execution for the subtask completes. */ - public void updateInputFileConsumers(PipelineInputs pipelineInputs, PipelineTask pipelineTask, - Path taskDirectory) { + void afterAlgorithmExecution(); - DatastorePathLocator datastorePathLocator = pipelineInputs - .datastorePathLocator(pipelineTask); - DataFileManager fileManager = new DataFileManager(datastorePathLocator, pipelineTask, - taskDirectory); + void setPipelineTask(PipelineTask pipelineTask); - Set filenames = fileManager - .filesInCompletedSubtasksWithResults(pipelineInputs.requiredDataFileInfoClasses()); - DatastoreProducerConsumerCrud producerConsumerCrud = new DatastoreProducerConsumerCrud(); - producerConsumerCrud.addConsumer(pipelineTask, filenames); + PipelineTask getPipelineTask(); - filenames = fileManager - .filesInCompletedSubtasksWithoutResults(pipelineInputs.requiredDataFileInfoClasses()); - producerConsumerCrud.addNonProducingConsumer(pipelineTask, filenames); - } + void setTaskDirectory(Path taskDirectory); - protected Hdf5ModuleInterface hdf5ModuleInterface() { - return hdf5ModuleInterface; - } + Path getTaskDirectory(); } diff --git a/src/main/java/gov/nasa/ziggy/module/PipelineResults.java b/src/main/java/gov/nasa/ziggy/module/PipelineResults.java deleted file mode 100644 index 28d61ef..0000000 --- a/src/main/java/gov/nasa/ziggy/module/PipelineResults.java +++ /dev/null @@ -1,30 +0,0 @@ -package gov.nasa.ziggy.module; - -import gov.nasa.ziggy.crud.HasOriginator; -import gov.nasa.ziggy.module.io.Persistable; - -/** - * Superclass for all pipeline results. Pipeline results files are files that are stored in the - * datastore. They need to implement Persistable (so that the HDF5 reader and writer can work with - * them), and they need to have an originator (to support data accountability). - * - * @author PT - */ -public abstract class PipelineResults implements Persistable, HasOriginator { - - // Note: it is not necessary to manually set the originator in a PipelineResults - // subclass instance as long as the instance is serialized by the saveResultsToTaskDir() - // method in PipelineOutputs; that method automatically sets the originator before - // serializing. - private long originator; - - @Override - public long getOriginator() { - return originator; - } - - @Override - public void setOriginator(long originator) { - this.originator = originator; - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/StateFile.java b/src/main/java/gov/nasa/ziggy/module/StateFile.java index 3b0efde..6d1a032 100644 --- a/src/main/java/gov/nasa/ziggy/module/StateFile.java +++ b/src/main/java/gov/nasa/ziggy/module/StateFile.java @@ -27,7 +27,6 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.remote.PbsParameters; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -74,7 +73,7 @@ *

      reRunnable
      *
      Whether this task is re-runnable.
      *
      localBinToMatEnabled
      - *
      If true, don't generate .mat files on the remote node. See {@link RemoteParameters}.
      + *
      If true, don't generate .mat files on the remote node.
      *
      requestedWallTime
      *
      Requested wall time for the PBS qsub command.
      *
      symlinksEnabled
      @@ -663,13 +662,13 @@ public void setMinCoresPerNode(int minCoresPerNode) { * Returns the value of the {@value #MIN_GIGS_PER_NODE_PROP_NAME} property, or 0 if not present * or set. */ - public int getMinGigsPerNode() { + public double getMinGigsPerNode() { return props.getProperty(MIN_GIGS_PER_NODE_PROP_NAME) != null - ? props.getInt(MIN_GIGS_PER_NODE_PROP_NAME) + ? props.getDouble(MIN_GIGS_PER_NODE_PROP_NAME) : INVALID_VALUE; } - public void setMinGigsPerNode(int minGigsPerNode) { + public void setMinGigsPerNode(double minGigsPerNode) { props.setProperty(MIN_GIGS_PER_NODE_PROP_NAME, minGigsPerNode); } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java b/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java index 639714f..c863515 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskAllocator.java @@ -8,7 +8,7 @@ /** * Allocates subtasks to clients that execute them in the order specified by an - * {@link TaskConfigurationManager} instance. + * {@link TaskConfiguration} instance. *

      * This class is typically accessed over a socket using {@link SubtaskServer} and * {@link SubtaskClient} @@ -28,9 +28,9 @@ public String toString() { + currentPoolProcessing + "]"; } - public SubtaskAllocator(TaskConfigurationManager inputsHandler) { - if (inputsHandler.numSubTasks() > 0) { - subtaskCompleted = new boolean[inputsHandler.numSubTasks()]; + public SubtaskAllocator(TaskConfiguration taskConfiguration) { + if (taskConfiguration.getSubtaskCount() > 0) { + subtaskCompleted = new boolean[taskConfiguration.getSubtaskCount()]; populateWaitingPool(); } } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java b/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java index 138e7a2..1cffc14 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskExecutor.java @@ -79,7 +79,7 @@ public class SubtaskExecutor { // Constructor is private, use the builder instead. private SubtaskExecutor(File taskDir, int subtaskIndex, String binaryName, int timeoutSecs) { this.taskDir = taskDir; - workingDir = TaskConfigurationManager.subtaskDirectory(taskDir, subtaskIndex); + workingDir = SubtaskUtils.subtaskDirectory(taskDir.toPath(), subtaskIndex).toFile(); this.timeoutSecs = timeoutSecs; this.binaryName = binaryName; } @@ -301,8 +301,8 @@ public int execAlgorithmInternal() { IntervalMetricKey key = IntervalMetric.start(); try { key = IntervalMetric.start(); - TaskConfigurationManager taskConfigurationManager = taskConfigurationManager(); - Class inputsClass = taskConfigurationManager.getInputsClass(); + TaskConfiguration taskConfiguration = taskConfiguration(); + Class inputsClass = taskConfiguration.getInputsClass(); retCode = runInputsOutputsCommand(inputsClass); if (retCode == 0) { inputsProcessingSucceeded = true; @@ -310,8 +310,7 @@ public int execAlgorithmInternal() { } if (retCode == 0) { algorithmProcessingSucceeded = true; - Class outputsClass = taskConfigurationManager - .getOutputsClass(); + Class outputsClass = taskConfiguration.getOutputsClass(); retCode = runInputsOutputsCommand(outputsClass); } } finally { @@ -356,17 +355,19 @@ public int execSimple(List commandLineArgs) { } /** - * Executes the {@link TaskFileManager#main()} method with appropriate arguments for either - * invoking the inputs class process that generates sub-task inputs, or the outputs class - * process that generates results files in the task directory. + * Executes the {@link BeforeAndAfterAlgorithmExecutor#main()} method with appropriate arguments + * such that {@link PipelineInputs#beforeAlgorithmExecution()} executes immediately prior to + * algorithm execution, and {@link PipelineOutputs#afterAlgorithmExecution()} executes + * immediately subsequent to algorithm execution. *

      - * Given that {@link TaskFileManager} is a Java class, and this is a Java class, why is the - * {@link TaskFileManager} invoked using the ziggy program and an external process? By using an - * external process, the inputs and outputs classes can use software libraries that do not - * support concurrency (for example, HDF5). By running each subtask in a separate process, the - * subtask input and output processing can execute in parallel even in cases in which the - * processing uses non-concurrent libraries. Running a bunch of instances of - * {@link TaskFileManager} in separate threads within a common JVM would not permit this. + * Given that {@link BeforeAndAfterAlgorithmExecutor} is a Java class, and this is a Java class, + * why is the {@link BeforeAndAfterAlgorithmExecutor} invoked using the ziggy program and an + * external process? By using an external process, the inputs and outputs classes can use + * software libraries that do not support concurrency (for example, HDF5). By running each + * subtask in a separate process, the subtask input and output processing can execute in + * parallel even in cases in which the processing uses non-concurrent libraries. Running a bunch + * of instances of {@link BeforeAndAfterAlgorithmExecutor} in separate threads within a common + * JVM would not permit this. * * @param inputsOutputsClass Class to be used as argument to TaskFileManager. * @return exit code from the ziggy program @@ -381,7 +382,7 @@ int runInputsOutputsCommand(Class inputsOutputsClass) { String log4jConfig = ExternalProcessUtils.log4jConfigString(); // Construct the class arguments for the ziggy program. - String taskFileManagerClassName = TaskFileManager.class.getCanonicalName(); + String taskFileManagerClassName = BeforeAndAfterAlgorithmExecutor.class.getCanonicalName(); String inputsOutputsClassName = inputsOutputsClass.getCanonicalName(); // Put it all together. @@ -537,8 +538,8 @@ CommandLine commandLine() { return commandLine; } - TaskConfigurationManager taskConfigurationManager() { - return TaskConfigurationManager.restore(workingDir.getParentFile()); + TaskConfiguration taskConfiguration() { + return TaskConfiguration.deserialize(workingDir.getParentFile()); } public static class Builder { diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskInformation.java b/src/main/java/gov/nasa/ziggy/module/SubtaskInformation.java index 01d31ed..1d63f0d 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskInformation.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskInformation.java @@ -10,14 +10,11 @@ public class SubtaskInformation { private final String moduleName; private final String uowBriefState; private final int subtaskCount; - private final int maxParallelSubtasks; - public SubtaskInformation(String moduleName, String uowBriefState, int subtaskCount, - int maxParallelSubtasks) { + public SubtaskInformation(String moduleName, String uowBriefState, int subtaskCount) { this.moduleName = moduleName; this.uowBriefState = uowBriefState; this.subtaskCount = subtaskCount; - this.maxParallelSubtasks = maxParallelSubtasks; } public String getModuleName() { @@ -31,8 +28,4 @@ public String getUowBriefState() { public int getSubtaskCount() { return subtaskCount; } - - public int getMaxParallelSubtasks() { - return maxParallelSubtasks; - } } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskLocator.java b/src/main/java/gov/nasa/ziggy/module/SubtaskLocator.java index 43d34c1..cf2c146 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskLocator.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskLocator.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -35,9 +35,9 @@ package gov.nasa.ziggy.module; import java.io.File; -import java.io.FilenameFilter; +import java.nio.file.Path; +import java.nio.file.Paths; import java.util.List; -import java.util.Set; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.DefaultParser; @@ -45,7 +45,6 @@ import org.apache.commons.cli.Options; import org.apache.commons.cli.ParseException; -import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -55,7 +54,7 @@ *

      * Given a task directory and a filename, the {@link SubtaskLocator} identifies and prints to stdout * all the subtasks that use that filename as any kind of input. Note that the class only works on - * tasks that use {@link DefaultPipelineInputs} to define the inputs. + * tasks that use {@link DatastoreDirectoryPipelineInputs} to define the inputs. * * @author PT */ @@ -84,22 +83,18 @@ public static void main(String[] args) { String taskDir = cmdLine.getOptionValue(directoryOption.getOpt()); String filename = cmdLine.getOptionValue(fileOption.getOpt()); - TaskConfigurationManager taskConfigurationManager = TaskConfigurationManager - .restore(new File(taskDir)); + TaskConfiguration taskConfiguration = TaskConfiguration.deserialize(new File(taskDir)); - Hdf5ModuleInterface hdf5mi = new Hdf5ModuleInterface(); - DefaultPipelineInputs inputs = new DefaultPipelineInputs(); - File[] inputsFiles = new File(taskDir) - .listFiles((FilenameFilter) (dir, name) -> name.endsWith("inputs.h5")); - if (inputsFiles.length != 1) { - throw new PipelineException("Too many inputs in task directory " + taskDir); - } - hdf5mi.readFile(new File(taskDir, inputsFiles[0].getName()), inputs, true); + DatastoreDirectoryPipelineInputs inputs = new DatastoreDirectoryPipelineInputs(); List modelFilenames = inputs.getModelFilenames(); - for (int i = 0; i < taskConfigurationManager.getSubtaskCount(); i++) { + Path taskDirPath = Paths.get(taskDir); + for (int i = 0; i < taskConfiguration.getSubtaskCount(); i++) { - Set filesForSubtask = taskConfigurationManager.filesForSubtask(i); + Path subdirPath = SubtaskUtils.subtaskDirectory(taskDirPath, i); + PipelineInputsOutputsUtils.readPipelineInputsFromDirectory(inputs, + PipelineInputsOutputsUtils.moduleName(subdirPath.getParent()), subdirPath); + List filesForSubtask = inputs.getDataFilenames(); if (filesForSubtask.contains(filename) || modelFilenames.contains(filename)) { System.out.println("Subtask " + i + " contains file " + filename); } diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java b/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java index 615e233..db97b73 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskMaster.java @@ -3,6 +3,7 @@ import java.io.File; import java.io.IOException; import java.io.UncheckedIOException; +import java.nio.file.Paths; import java.util.Objects; import java.util.concurrent.Semaphore; @@ -103,9 +104,9 @@ private void processSubtasks() { log.debug(threadNumber + ": Processing sub-task: " + subtaskIndex); - File subtaskDir = TaskConfigurationManager.subtaskDirectory(new File(taskDir), - subtaskIndex); - File lockFile = new File(subtaskDir, TaskConfigurationManager.LOCK_FILE_NAME); + File subtaskDir = SubtaskUtils.subtaskDirectory(Paths.get(taskDir), subtaskIndex) + .toFile(); + File lockFile = new File(subtaskDir, TaskConfiguration.LOCK_FILE_NAME); try { if (getWriteLockWithoutBlocking(lockFile)) { diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java b/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java index d3d102d..ace6197 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskServer.java @@ -25,12 +25,12 @@ public class SubtaskServer implements Runnable { private SubtaskAllocator subtaskAllocator; private final CountDownLatch serverThreadReady = new CountDownLatch(1); - private TaskConfigurationManager inputsHandler; + private TaskConfiguration inputsHandler; private Thread listenerThread; - public SubtaskServer(int subtaskMasterCount, TaskConfigurationManager inputsHandler) { + public SubtaskServer(int subtaskMasterCount, TaskConfiguration taskConfiguration) { initializeRequestQueue(subtaskMasterCount); - this.inputsHandler = inputsHandler; + inputsHandler = taskConfiguration; } public static void initializeRequestQueue(int subtaskMasterCount) { @@ -39,7 +39,7 @@ public static void initializeRequestQueue(int subtaskMasterCount) { @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public void start() { - log.info("Starting SubtaskServer for inputs: " + inputsHandler); + log.info("Starting SubtaskServer"); try { // NB: if the listener thread constructor and setDaemon() calls are moved diff --git a/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java b/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java index 61f8aaf..0e7076a 100644 --- a/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/SubtaskUtils.java @@ -1,11 +1,24 @@ package gov.nasa.ziggy.module; import java.io.File; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; +import java.util.stream.Stream; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.slf4j.MDC; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; + /** * @author PT * @author Todd Klaus @@ -13,6 +26,18 @@ public class SubtaskUtils { private static final Logger log = LoggerFactory.getLogger(SubtaskUtils.class); + public static final String SUBTASK_DIR_PREFIX = "st-"; + public static final String SUBTASK_DIR_REGEXP = SUBTASK_DIR_PREFIX + "(\\d+)"; + public static final Pattern SUBTASK_DIR_PATTERN = Pattern.compile(SUBTASK_DIR_REGEXP); + + public static Path subtaskDirectory(Path taskWorkingDir, int subtaskIndex) { + return taskWorkingDir.resolve(subtaskDirName(subtaskIndex)); + } + + public static String subtaskDirName(int subtaskIndex) { + return SUBTASK_DIR_PREFIX + subtaskIndex; + } + /** * Sets a thread-specific string that will be included in log messages. Specifically, the * subtask directory, enclosed in parentheses (i.e., "(st-123)") is set as the thread-specific @@ -49,4 +74,40 @@ public static void clearStaleAlgorithmStates(File taskDir) { new AlgorithmStateFiles(subtaskDir).clearStaleState(); } } + + /** + * Returns the subtask index for the current subtask. Assumes that the working directory is the + * subtask directory. + */ + public static int subtaskIndex() { + Matcher m = SUBTASK_DIR_PATTERN + .matcher(DirectoryProperties.workingDir().getFileName().toString()); + if (m.matches()) { + return Integer.parseInt(m.group(1)); + } + throw new PipelineException("Directory " + DirectoryProperties.workingDir().toString() + + " not a subtask directory"); + } + + /** Returns a list of subtask directories in a task dir. */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public static List subtaskDirectories(Path taskDir) { + try (Stream dirStream = Files.list(taskDir)) { + return dirStream.filter(Files::isDirectory) + .filter(s -> SUBTASK_DIR_PATTERN.matcher(s.getFileName().toString()).matches()) + .collect(Collectors.toList()); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + /** Creates a subtask for a given task directory and subtask index. */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public static Path createSubtaskDirectory(Path taskWorkingDir, int subtaskIndex) { + try { + return Files.createDirectories(subtaskDirectory(taskWorkingDir, subtaskIndex)); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } } diff --git a/src/main/java/gov/nasa/ziggy/module/TaskConfiguration.java b/src/main/java/gov/nasa/ziggy/module/TaskConfiguration.java new file mode 100644 index 0000000..7149574 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/TaskConfiguration.java @@ -0,0 +1,148 @@ +package gov.nasa.ziggy.module; + +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.ObjectInputStream; +import java.io.ObjectOutputStream; +import java.io.Serializable; +import java.io.UncheckedIOException; +import java.util.Objects; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.SpotBugsUtils; + +/** + * Serializes and deserializes the subtask count, inputs class, and outputs class for a given + * {@link PipelineTask}. + * + * @author Todd Klaus + * @author PT + */ +public class TaskConfiguration implements Serializable { + private static final Logger log = LoggerFactory.getLogger(TaskConfiguration.class); + private static final long serialVersionUID = 20240112L; + private static final String PERSISTED_FILE_NAME = ".task-configuration.ser"; + public static final String LOCK_FILE_NAME = ".lock"; + + private transient File taskDir = null; + + private Class inputsClass; + private Class outputsClass; + private int subtaskCount; + + public TaskConfiguration() { + } + + public TaskConfiguration(File taskDir) { + this.taskDir = taskDir; + } + + public void serialize() { + serialize(getTaskDir()); + } + + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public void serialize(File dir) { + File dest = serializedFile(dir); + try (ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(dest))) { + log.info("Serializing task configuration to: {}", dest); + oos.writeObject(this); + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to serialize task configuration to " + dir.toString(), e); + } + } + + @SuppressFBWarnings(value = "OBJECT_DESERIALIZATION", + justification = SpotBugsUtils.DESERIALIZATION_JUSTIFICATION) + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) + public static TaskConfiguration deserialize(File taskDir) { + File src = serializedFile(taskDir); + try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(src))) { + log.info("Deserializing task configuration from: {}", src); + + TaskConfiguration s = (TaskConfiguration) ois.readObject(); + s.taskDir = taskDir; + + return s; + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to deserialize configuration manager from " + taskDir.toString(), e); + } catch (ClassNotFoundException e) { + // This can never occur. By construction, the object deserialized here was + // serialized at some prior point by this same class, which means that it is + // guaranteed to be a TaskConfigurationManager instance. + throw new AssertionError(e); + } + } + + public static boolean isSerializedTaskConfigurationPresent(File taskDir) { + return serializedFile(taskDir).exists(); + } + + public static File serializedFile(File taskDir) { + return new File(taskDir, PERSISTED_FILE_NAME); + } + + public File getTaskDir() { + return taskDir; + } + + public void setSubtaskCount(int subtaskCount) { + this.subtaskCount = subtaskCount; + } + + public int getSubtaskCount() { + return subtaskCount; + } + + public void setInputsClass(Class inputsClass) { + this.inputsClass = inputsClass; + } + + public Class getInputsClass() { + return inputsClass; + } + + public void setOutputsClass(Class outputsClass) { + this.outputsClass = outputsClass; + } + + public Class getOutputsClass() { + return outputsClass; + } + + // Note: it was necessary to get the class names for hashCode because you can't hash + // a Class object itself (i.e., hash(DatastoreDirectoryPipelineInputs.class) is not + // defined). + @Override + public int hashCode() { + return Objects.hash(inputsClass.getName(), outputsClass.getName(), subtaskCount); + } + + // Note: it was necessary to get the class names for equals because Class objects + // do not define equals() (i.e., equals(DatastoreDirectoryPipelineInputs.class) is not + // defined). + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + TaskConfiguration other = (TaskConfiguration) obj; + return Objects.equals(inputsClass.getName(), other.inputsClass.getName()) + && Objects.equals(outputsClass.getName(), other.outputsClass.getName()) + && subtaskCount == other.subtaskCount; + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/TaskConfigurationManager.java b/src/main/java/gov/nasa/ziggy/module/TaskConfigurationManager.java deleted file mode 100644 index dcdabb6..0000000 --- a/src/main/java/gov/nasa/ziggy/module/TaskConfigurationManager.java +++ /dev/null @@ -1,301 +0,0 @@ -package gov.nasa.ziggy.module; - -import java.io.File; -import java.io.FileInputStream; -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.ObjectInputStream; -import java.io.ObjectOutputStream; -import java.io.Serializable; -import java.io.UncheckedIOException; -import java.util.ArrayList; -import java.util.LinkedList; -import java.util.List; -import java.util.Objects; -import java.util.Set; - -import org.apache.commons.io.FileUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; -import gov.nasa.ziggy.pipeline.definition.PipelineModule; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.SpotBugsUtils; - -/** - * This class defines execution dependencies for sub-tasks. - *

      - * Each element in the sequence contains one or more sub-tasks. Sub-tasks that are members of the - * same element will run in parallel. - *

      - * For example, consider the sequence [0][1,5][6] Sub-task 0 will run first, then sub-tasks 1, 2, 3, - * 4, and 5 will run in parallel, then sub-task 6 will run. Each element of the sequence starts - * executing only after the previous element is complete. The class also records the unit of work - * instance for each sub-task, as well as the classes that the task uses for inputs and outputs. - *

      - * Important note: the hashCode() and equals() methods required manual modification because a Class - * doesn't implement hashCode() or equals(). Consequently the getCanonicalName() method is used to - * extract something from the Class instance that can be compared via the String hashCode() and - * equals() methods. - * - * @author Todd Klaus - * @author PT - */ -public class TaskConfigurationManager implements Serializable { - private static final Logger log = LoggerFactory.getLogger(TaskConfigurationManager.class); - private static final long serialVersionUID = 20230511L; - private static final String PERSISTED_FILE_NAME = ".task-configuration.ser"; - public static final String LOCK_FILE_NAME = ".lock"; - - private transient File taskDir = null; - - private final List> filesForSubtasks = new ArrayList<>(); - private Class inputsClass; - private Class outputsClass; - - private int subtaskCount = 0; - - public TaskConfigurationManager() { - } - - public TaskConfigurationManager(File taskDir) { - this.taskDir = taskDir; - } - - /** - * Construct the current subtask directory and increment the subtask index. Add the subtask - * files. Put the .lock file into the subtask directory. - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void addFilesForSubtask(Set files) { - - File subTaskDirectory = subtaskDirectory(taskDir, subtaskCount); - filesForSubtasks.add(files); - try { - new File(subTaskDirectory, LOCK_FILE_NAME).createNewFile(); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to create file " + new File(subTaskDirectory, LOCK_FILE_NAME).toString(), - e); - } - subtaskCount++; - } - - /** - * Return the current subtask directory - * - * @return - */ - public File subtaskDirectory() { - return TaskConfigurationManager.subtaskDirectory(taskDir, subtaskCount); - } - - public boolean contains(int subTaskNumber) { - return subtaskCount >= subTaskNumber + 1; - } - - public int numSubTasks() { - return subtaskCount; - } - - public int numInputs() { - int numInputs = subtaskCount; - log.info("numSubTaskInSeq: " + numInputs); - return numInputs; - } - - public boolean isEmpty() { - return subtaskCount == 0; - } - - /** - * Checks to make sure that the number of inputs (based on count of sub-task UOWs) and the - * number of sub-tasks (based on the sub-task pair list) match. - */ - void validate() { - - int numSubTasks = numSubTasks(); - int numInputs = numInputs(); - - // Check that the # of sub-tasks in the list of Pairs equals the number of inputs - // added to the UOW list. Note that this can be true but the Pairs can still be wrong - // in one of two ways: a duplicate combined with an omission in the Pairs (i.e., - // sub-task 0 never processed, sub-task 1 processed in 2 of the Pairs); an offset of the - // pairs (i.e., sub-tasks run from 0 to 20 but the Pairs run from 1 to 21). - if (numInputs != numSubTasks) { - String message = String.format( - "Number of sub-tasks(%d) does not match number of inputs (%d)", numSubTasks, - numInputs); - log.error(message); - throw new PipelineException(message); - } - } - - @Override - public String toString() { - return "SINGLE:[" + 0 + "," + (subtaskCount - 1) + "]"; - } - - public void persist() { - persist(getTaskDir()); - } - - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public void persist(File dir) { - File dest = persistedFile(dir); - try (ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(dest))) { - log.info("Persisting inputs metadata to: " + dest); - oos.writeObject(this); - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to persist task configuration manger to dir " + dir.toString(), e); - } - } - - @SuppressFBWarnings(value = "OBJECT_DESERIALIZATION", - justification = SpotBugsUtils.DESERIALIZATION_JUSTIFICATION) - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - public static TaskConfigurationManager restore(File taskDir) { - File src = persistedFile(taskDir); - try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(src))) { - log.info("Restoring outputs metadata from: " + src); - - TaskConfigurationManager s = (TaskConfigurationManager) ois.readObject(); - s.taskDir = taskDir; - - return s; - } catch (IOException e) { - throw new UncheckedIOException( - "Unable to load task configuration manager from dir " + taskDir.toString(), e); - } catch (ClassNotFoundException e) { - // This can never occur. By construction, the object deserialized here was - // serialized at some prior point by this same class, which means that it is - // guaranteed to be a TaskConfigurationManager instance. - throw new AssertionError(e); - } - } - - public static Set restoreAndRetrieveFilesForSubtask(File taskDir, int subtaskIndex) { - return TaskConfigurationManager.restore(taskDir).filesForSubtask(subtaskIndex); - } - - public static boolean isPersistedInputsHandlerPresent(File taskDir) { - return persistedFile(taskDir).exists(); - } - - public static File persistedFile(File taskDir) { - return new File(taskDir, PERSISTED_FILE_NAME); - } - - /** - * Return a collection of all sub-task directories for this InputsHandler - * - * @return - */ - public List allSubTaskDirectories() { - List subTaskDirs = new LinkedList<>(); - int numSubTasks = numSubTasks(); - for (int subTaskIndex = 0; subTaskIndex < numSubTasks; subTaskIndex++) { - subTaskDirs.add(subtaskDirectory(taskDir, subTaskIndex)); - } - return subTaskDirs; - } - - /** - * For PI use only. {@link PipelineModule} classes should use subTaskDirectory(), above. - *

      - * Create the sub-task working directory (if necessary) and return the path - * - * @param subTaskIndex - * @return - */ - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public static File subtaskDirectory(File taskWorkingDir, int subTaskIndex) { - File subTaskDir = new File(taskWorkingDir, "st-" + subTaskIndex); - try { - - // ensure that the directory exists - if (!subTaskDir.exists()) { - FileUtils.forceMkdir(subTaskDir); - } - - return subTaskDir; - } catch (IOException e) { - throw new UncheckedIOException("Unable to create directory " + subTaskDir.toString(), - e); - } - } - - public Set filesForSubtask(int subtaskNumber) { - return filesForSubtasks.get(subtaskNumber); - } - - @Override - public int hashCode() { - final int prime = 31; - int result = 1; - result = prime * result + subtaskCount; - result = prime * result + (filesForSubtasks == null ? 0 : filesForSubtasks.hashCode()); - result = prime * result - + (inputsClass == null ? 0 : inputsClass.getCanonicalName().hashCode()); - return prime * result - + (outputsClass == null ? 0 : outputsClass.getCanonicalName().hashCode()); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - TaskConfigurationManager other = (TaskConfigurationManager) obj; - if (subtaskCount != other.subtaskCount - || !Objects.equals(filesForSubtasks, other.filesForSubtasks)) { - return false; - } - if (inputsClass == null) { - if (other.inputsClass != null) { - return false; - } - } else if (!inputsClass.getCanonicalName().equals(other.inputsClass.getCanonicalName())) { - return false; - } - if (outputsClass == null) { - if (other.outputsClass != null) { - return false; - } - } else if (!outputsClass.getCanonicalName().equals(other.outputsClass.getCanonicalName())) { - return false; - } - return true; - } - - public File getTaskDir() { - return taskDir; - } - - public int getSubtaskCount() { - return subtaskCount; - } - - public void setInputsClass(Class inputsClass) { - this.inputsClass = inputsClass; - } - - public Class getInputsClass() { - return inputsClass; - } - - public void setOutputsClass(Class outputsClass) { - this.outputsClass = outputsClass; - } - - public Class getOutputsClass() { - return outputsClass; - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java b/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java new file mode 100644 index 0000000..5ea3906 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/module/TaskDirectoryManager.java @@ -0,0 +1,59 @@ +package gov.nasa.ziggy.module; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import gov.nasa.ziggy.util.io.FileUtil; + +/** + * Names and creates the task directories for external process invocation. + * + * @author Todd Klaus + */ +public class TaskDirectoryManager { + private static final Logger log = LoggerFactory.getLogger(TaskDirectoryManager.class); + + private final Path taskDataDir; + private final PipelineTask pipelineTask; + + public TaskDirectoryManager(PipelineTask pipelineTask) { + taskDataDir = DirectoryProperties.taskDataDir(); + this.pipelineTask = pipelineTask; + } + + public Path taskDir() { + return taskDir(pipelineTask.taskBaseName()); + } + + private Path taskDir(String taskBaseName) { + return taskDataDir.resolve(taskBaseName); + } + + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public synchronized Path allocateTaskDir(boolean cleanExisting) { + + if (Files.isDirectory(taskDir()) && cleanExisting) { + log.info( + "Working directory for name=" + pipelineTask.getId() + " already exists, deleting"); + FileUtil.deleteDirectoryTree(taskDir()); + } + + log.info("Creating task working dir: " + taskDir().toString()); + try { + Files.createDirectories(taskDir()); + } catch (IOException e) { + throw new UncheckedIOException("Unable to create dir " + taskDir().toString(), e); + } + + return taskDir(); + } +} diff --git a/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java b/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java index 6b32c12..d4b7888 100644 --- a/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java +++ b/src/main/java/gov/nasa/ziggy/module/TaskMonitor.java @@ -1,6 +1,7 @@ package gov.nasa.ziggy.module; import java.io.File; +import java.nio.file.Path; import java.util.List; import org.slf4j.Logger; @@ -34,10 +35,10 @@ public class TaskMonitor { private final StateFile stateFile; private final File taskDir; private final File lockFile; - private final List subtaskDirectories; + private final List subtaskDirectories; - public TaskMonitor(TaskConfigurationManager inputsHandler, StateFile stateFile, File taskDir) { - subtaskDirectories = inputsHandler.allSubTaskDirectories(); + public TaskMonitor(StateFile stateFile, File taskDir) { + subtaskDirectories = SubtaskUtils.subtaskDirectories(taskDir.toPath()); this.stateFile = stateFile; this.taskDir = taskDir; lockFile = new File(taskDir, StateFile.LOCK_FILE_NAME); @@ -49,8 +50,9 @@ private SubtaskStateCounts countSubtaskStates() { log.warn("No subtask directories found in: " + taskDir); } - for (File subtaskDir : subtaskDirectories) { - AlgorithmStateFiles currentSubtaskStateFile = new AlgorithmStateFiles(subtaskDir); + for (Path subtaskDir : subtaskDirectories) { + AlgorithmStateFiles currentSubtaskStateFile = new AlgorithmStateFiles( + subtaskDir.toFile()); SubtaskState currentSubtaskState = currentSubtaskStateFile.currentSubtaskState(); if (currentSubtaskState == null) { diff --git a/src/main/java/gov/nasa/ziggy/module/WorkingDirManager.java b/src/main/java/gov/nasa/ziggy/module/WorkingDirManager.java deleted file mode 100644 index 7d08b08..0000000 --- a/src/main/java/gov/nasa/ziggy/module/WorkingDirManager.java +++ /dev/null @@ -1,67 +0,0 @@ -package gov.nasa.ziggy.module; - -import java.io.File; -import java.io.IOException; -import java.io.UncheckedIOException; - -import org.apache.commons.io.FileUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; -import gov.nasa.ziggy.util.AcceptableCatchBlock; -import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.io.FileUtil; - -/** - * Names, creates, and deletes the temporary working directories for external process invocation. - * - * @author Todd Klaus - */ -public class WorkingDirManager { - private static final Logger log = LoggerFactory.getLogger(WorkingDirManager.class); - - private final File rootWorkingDir; - - public WorkingDirManager() { - rootWorkingDir = new File(workingDirParent()); - } - - public static String workingDirParent() { - - return DirectoryProperties.taskDataDir().toString(); - } - - public static File workingDirBaseName(PipelineTask pipelineTask) { - return new File(pipelineTask.taskBaseName()); - } - - public static File workingDir(PipelineTask pipelineTask) { - return workingDir(pipelineTask.taskBaseName()); - } - - private static File workingDir(String taskBaseName) { - return new File(workingDirParent(), taskBaseName); - } - - @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public synchronized File allocateWorkingDir(PipelineTask pipelineTask, boolean cleanExisting) { - File workingDir = new File(rootWorkingDir, pipelineTask.taskBaseName()); - - if (workingDir.exists() && cleanExisting) { - log.info( - "Working directory for name=" + pipelineTask.getId() + " already exists, deleting"); - FileUtil.deleteDirectoryTree(workingDir.toPath()); - } - - log.info("Creating task working dir: " + workingDir); - try { - FileUtils.forceMkdir(workingDir); - } catch (IOException e) { - throw new UncheckedIOException("Unable to create dir " + workingDir.toString(), e); - } - - return workingDir; - } -} diff --git a/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java b/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java index 0203df1..bcb3aec 100644 --- a/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java +++ b/src/main/java/gov/nasa/ziggy/module/io/matlab/MatlabUtils.java @@ -3,6 +3,8 @@ import java.io.File; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.util.os.OperatingSystemType; /** @@ -12,9 +14,6 @@ */ public class MatlabUtils { - private static OperatingSystemType osType; - private static String architecture; - // Never try to instantiate this class. private MatlabUtils() { } @@ -47,7 +46,7 @@ public static String mcrPaths(String mcrRoot) { openGlString = ""; break; default: - throw new PipelineException("OS type " + osType.toString() + " not supported"); + throw new PipelineException("OS type " + osType().toString() + " not supported"); } StringBuilder pathBuilder = new StringBuilder(); pathBuilder.append( @@ -64,26 +63,12 @@ public static String mcrPaths(String mcrRoot) { } private static OperatingSystemType osType() { - if (osType == null) { - osType = OperatingSystemType.getInstance(); - } - return osType; + return OperatingSystemType.getInstance(); } private static String architecture() { - if (architecture == null) { - architecture = System.getProperty("os.arch").toLowerCase(); - } - return architecture; - } - - // Use for unit test purposes only. - static void setOsType(OperatingSystemType osTypeArg) { - osType = osTypeArg; - } - - /** For testing only. */ - static void setArchitecture(String arch) { - architecture = arch; + return ZiggyConfiguration.getInstance() + .getString(PropertyName.ARCHITECTURE.property()) + .toLowerCase(); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java b/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java index 81b723a..e57a447 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/PbsParameters.java @@ -15,12 +15,13 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.util.TimeFormatter; /** * Parameters needed to submit a job via the PBS batch system. Parameters are derived from an - * appropriate {@link RemoteParameters} instance. Any optional parameters will be determined by the - * needs of the job as determined from the required parameters. + * appropriate {@link PipelineDefinitionNodeExecutionResources} instance. Any optional parameters + * will be determined by the needs of the job as determined from the required parameters. * * @author PT */ @@ -30,7 +31,7 @@ public class PbsParameters { private String requestedWallTime; private int requestedNodeCount; private String queueName; - private int minGigsPerNode; + private double minGigsPerNode; private int minCoresPerNode; private double gigsPerSubtask; private String remoteGroup; @@ -43,27 +44,27 @@ public class PbsParameters { /** * Populates the architecture property of the {@link PbsParameters} instance. If the * architecture is not specified already, it is selected by use of the architecture optimization - * option in {@link RemoteParameters}. If it is specified, the architecture is nonetheless - * checked to ensure that there is sufficient RAM on the selected architecture to run at least 1 - * subtask per node. + * option in {@link PipelineDefinitionNodeExecutionResources}. If it is specified, the + * architecture is nonetheless checked to ensure that there is sufficient RAM on the selected + * architecture to run at least 1 subtask per node. */ - public void populateArchitecture(RemoteParameters remoteParameters, int totalSubtasks, - SupportedRemoteClusters remoteCluster) { + public void populateArchitecture(PipelineDefinitionNodeExecutionResources executionResources, + int totalSubtasks, SupportedRemoteClusters remoteCluster) { // If the architecture isn't specified, determine it via optimization. if (getArchitecture() == null) { log.info("Selecting infrastructure for " + totalSubtasks + " subtasks using optimizer " - + remoteParameters.getOptimizer()); - selectArchitecture(remoteParameters, totalSubtasks, remoteCluster); + + executionResources.getOptimizer()); + selectArchitecture(executionResources, totalSubtasks, remoteCluster); } // Note that if the architecture IS specified, it may still not be compatible // with the RAM requirements; check that possibility now. - if (remoteParameters.isNodeSharing() - && !nodeHasSufficientRam(getArchitecture(), remoteParameters.getGigsPerSubtask())) { - throw new IllegalStateException("selected node architecture " + if (executionResources.isNodeSharing() + && !nodeHasSufficientRam(getArchitecture(), executionResources.getGigsPerSubtask())) { + throw new IllegalStateException("Selected node architecture " + getArchitecture().toString() + " has insufficient RAM for subtasks that require " - + remoteParameters.getGigsPerSubtask() + " GB"); + + executionResources.getGigsPerSubtask() + " GB"); } } @@ -72,14 +73,13 @@ public void populateArchitecture(RemoteParameters remoteParameters, int totalSub * that corresponds to the {@link SupportedRemoteCluster}, and will be selected from the ones * with sufficient RAM to perform the processing. */ - private void selectArchitecture(RemoteParameters remoteParameters, int totalSubtasks, - SupportedRemoteClusters remoteCluster) { + private void selectArchitecture(PipelineDefinitionNodeExecutionResources executionResources, + int totalSubtasks, SupportedRemoteClusters remoteCluster) { - double gigsPerSubtask = remoteParameters.getGigsPerSubtask(); - RemoteArchitectureOptimizer optimizer = RemoteArchitectureOptimizer - .fromName(remoteParameters.getOptimizer()); + double gigsPerSubtask = executionResources.getGigsPerSubtask(); + RemoteArchitectureOptimizer optimizer = executionResources.getOptimizer(); List acceptableNodes = null; - if (!remoteParameters.isNodeSharing()) { + if (!executionResources.isNodeSharing()) { acceptableNodes = descriptorsSortedByCores(remoteCluster); } else { if (optimizer.equals(RemoteArchitectureOptimizer.COST)) { @@ -91,12 +91,12 @@ private void selectArchitecture(RemoteParameters remoteParameters, int totalSubt } if (acceptableNodes.isEmpty()) { throw new PipelineException( - "All remote architectures have insufficient RAM to support " + gigsPerSubtask - + " GB per subtask requirement"); + "All remote architectures have insufficient RAM for subtasks that require " + + gigsPerSubtask + " GB"); } } - architecture = optimizer.optimalArchitecture(remoteParameters, totalSubtasks, + architecture = optimizer.optimalArchitecture(executionResources, totalSubtasks, acceptableNodes); } @@ -105,17 +105,17 @@ private void selectArchitecture(RemoteParameters remoteParameters, int totalSubt * subtasks per node, the wall time request, the queue, and the remote group to be billed for * the PBS jobs. An estimate of the cost (in SBUs or dollars) is also performed. */ - public void populateResourceParameters(RemoteParameters remoteParameters, - int totalSubtaskCount) { + public void populateResourceParameters( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtaskCount) { - computeActiveCoresPerNode(remoteParameters, totalSubtaskCount); - double subtasksPerCore = subtasksPerCore(remoteParameters, totalSubtaskCount); + computeActiveCoresPerNode(executionResources, totalSubtaskCount); + double subtasksPerCore = subtasksPerCore(executionResources, totalSubtaskCount); // Set the number of nodes and the number of subtasks per node - computeNodeRequest(remoteParameters, totalSubtaskCount, subtasksPerCore); + computeNodeRequest(executionResources, totalSubtaskCount, subtasksPerCore); // Set the wall time and the queue - computeWallTimeAndQueue(remoteParameters, subtasksPerCore); + computeWallTimeAndQueue(executionResources, subtasksPerCore); // Estimate the costs computeEstimatedCost(); @@ -132,8 +132,8 @@ public void populateResourceParameters(RemoteParameters remoteParameters, * smaller than the maximum number of nodes to request (that's what makes it a maximum number * rather than just a number). */ - private void computeNodeRequest(RemoteParameters remoteParameters, int totalSubtaskCount, - double subtasksPerCore) { + private void computeNodeRequest(PipelineDefinitionNodeExecutionResources executionResources, + int totalSubtaskCount, double subtasksPerCore) { // Start by computing the "optimal" number of nodes -- the number of nodes // needed @@ -144,9 +144,8 @@ private void computeNodeRequest(RemoteParameters remoteParameters, int totalSubt // If this number is LARGER than what the user wants, defer to the user; // if it's SMALLER than what the user wants, take the smaller value, since // the larger value would simply result in idle nodes. - if (!StringUtils.isEmpty(remoteParameters.getMaxNodes())) { - requestedNodeCount = Math.min(Integer.parseInt(remoteParameters.getMaxNodes()), - requestedNodeCount); + if (executionResources.getMaxNodes() > 0) { + requestedNodeCount = Math.min(executionResources.getMaxNodes(), requestedNodeCount); } } @@ -156,24 +155,24 @@ private void computeNodeRequest(RemoteParameters remoteParameters, int totalSubt * override will only be used if it will not result in some subtasks not getting processed * during the job. */ - private double subtasksPerCore(RemoteParameters remoteParameters, int totalSubtaskCount) { + private double subtasksPerCore(PipelineDefinitionNodeExecutionResources executionResources, + int totalSubtaskCount) { - double wallTimeRatio = remoteParameters.getSubtaskMaxWallTimeHours() - / remoteParameters.getSubtaskTypicalWallTimeHours(); + double wallTimeRatio = executionResources.getSubtaskMaxWallTimeHours() + / executionResources.getSubtaskTypicalWallTimeHours(); double subtasksPerCore = wallTimeRatio; // If the number of nodes is limited, compute how many subtasks per active core // are needed; if this number is larger, keep it - if (!StringUtils.isEmpty(remoteParameters.getMaxNodes())) { + if (executionResources.getMaxNodes() > 0) { subtasksPerCore = Math.max(subtasksPerCore, (double) totalSubtaskCount - / (activeCoresPerNode * Integer.parseInt(remoteParameters.getMaxNodes()))); + / (activeCoresPerNode * executionResources.getMaxNodes())); } // Finally, if the user has supplied an override for subtasks per core, apply it // unless it conflicts with the value needed to make the job complete - if (!StringUtils.isEmpty(remoteParameters.getSubtasksPerCore())) { - double overrideSubtasksPerCore = Double - .parseDouble(remoteParameters.getSubtasksPerCore()); + if (executionResources.getSubtasksPerCore() > 0) { + double overrideSubtasksPerCore = executionResources.getSubtasksPerCore(); if (overrideSubtasksPerCore >= subtasksPerCore) { subtasksPerCore = overrideSubtasksPerCore; } else { @@ -202,15 +201,15 @@ private void computeRequestedNodeCount(int totalSubtaskCount, double subtasksPer * RAM in the node divided by the number of GB needed per subtask; the number of subtasks in the * task. */ - public void computeActiveCoresPerNode(RemoteParameters remoteParameters, - int totalSubtaskCount) { - if (!remoteParameters.isNodeSharing()) { + public void computeActiveCoresPerNode( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtaskCount) { + if (!executionResources.isNodeSharing()) { activeCoresPerNode = 1; return; } double gigsPerNode = architecture.getGigsPerCore() * minCoresPerNode; activeCoresPerNode = (int) Math - .min(Math.floor(gigsPerNode / remoteParameters.getGigsPerSubtask()), minCoresPerNode); + .min(Math.floor(gigsPerNode / executionResources.getGigsPerSubtask()), minCoresPerNode); activeCoresPerNode = Math.min(activeCoresPerNode, totalSubtaskCount); } @@ -223,19 +222,19 @@ public void computeActiveCoresPerNode(RemoteParameters remoteParameters, * times are scaled inversely to cores per node, this scaling will be applied to estimate the * wall time needed. */ - private void computeWallTimeAndQueue(RemoteParameters remoteParameters, - double subtasksPerCore) { + private void computeWallTimeAndQueue( + PipelineDefinitionNodeExecutionResources executionResources, double subtasksPerCore) { - double typicalWallTimeHours = remoteParameters.getSubtaskTypicalWallTimeHours(); - double bareWallTimeHours = Math.max(remoteParameters.getSubtaskMaxWallTimeHours(), + double typicalWallTimeHours = executionResources.getSubtaskTypicalWallTimeHours(); + double bareWallTimeHours = Math.max(executionResources.getSubtaskMaxWallTimeHours(), subtasksPerCore * typicalWallTimeHours); - if (!remoteParameters.isNodeSharing() && remoteParameters.isWallTimeScaling()) { + if (!executionResources.isNodeSharing() && executionResources.isWallTimeScaling()) { bareWallTimeHours /= minCoresPerNode; } double requestedWallTimeHours = 0.25 * Math.ceil(4 * bareWallTimeHours); requestedWallTime = TimeFormatter.timeInHoursToStringHhMmSs(requestedWallTimeHours); - if (StringUtils.isEmpty(remoteParameters.getQueueName())) { + if (StringUtils.isEmpty(executionResources.getQueueName())) { List queues = descriptorsSortedByMaxTime( architecture.getRemoteCluster()); for (RemoteQueueDescriptor queue : queues) { @@ -245,22 +244,23 @@ private void computeWallTimeAndQueue(RemoteParameters remoteParameters, } } if (queueName == null) { - throw new IllegalStateException( - "No queues can support requested wall time of " + requestedWallTime); + throw new IllegalStateException("No queues can support requested wall time of " + + TimeFormatter.stripSeconds(requestedWallTime)); } } else { RemoteQueueDescriptor descriptor = RemoteQueueDescriptor - .fromQueueName(remoteParameters.getQueueName()); + .fromQueueName(executionResources.getQueueName()); if (descriptor.equals(RemoteQueueDescriptor.UNKNOWN)) { log.warn("Unable to determine max wall time for queue " - + remoteParameters.getQueueName()); - queueName = remoteParameters.getQueueName(); + + executionResources.getQueueName()); + queueName = executionResources.getQueueName(); } else if (descriptor.equals(RemoteQueueDescriptor.RESERVED)) { - queueName = remoteParameters.getQueueName(); + queueName = executionResources.getQueueName(); } else { if (descriptor.getMaxWallTimeHours() < requestedWallTimeHours) { throw new IllegalStateException("Queue " + descriptor.getQueueName() - + " cannot support job with wall time of " + requestedWallTime); + + " cannot support job with wall time of " + + TimeFormatter.stripSeconds(requestedWallTime)); } queueName = descriptor.getQueueName(); } @@ -376,11 +376,11 @@ public void setQueueName(String queueName) { this.queueName = queueName; } - public int getMinGigsPerNode() { + public double getMinGigsPerNode() { return minGigsPerNode; } - public void setMinGigsPerNode(int minGigsPerNode) { + public void setMinGigsPerNode(double minGigsPerNode) { this.minGigsPerNode = minGigsPerNode; } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java b/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java index de4020d..52bd2a2 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/Qsub.java @@ -7,12 +7,14 @@ import java.util.Date; import java.util.List; +import org.apache.commons.configuration2.ImmutableConfiguration; import org.apache.commons.exec.CommandLine; import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.process.ExternalProcess; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -45,7 +47,7 @@ public class Qsub { private String wallTime; private Integer numNodes; private Integer coresPerNode; - private Integer gigsPerNode; + private Double gigsPerNode; private String model; private String groupName; private String datestamp = Iso8601Formatter.dateTimeLocalFormatter().format(new Date()); @@ -72,9 +74,6 @@ public int[] submitMultipleJobsForTask() { private int submit1Job(int jobIndex) { - String propertiesFileName = java.lang.System - .getenv(ZiggyConfiguration.PIPELINE_CONFIG_PATH_ENV); - CommandLine commandLine = new CommandLine("/PBS/bin/qsub"); commandLine.addArgument("-N"); commandLine.addArgument(fullJobName(jobIndex)); @@ -85,11 +84,10 @@ private int submit1Job(int jobIndex) { commandLine.addArgument(resourceOptions(jobIndex < 0)); commandLine.addArgument("-W"); commandLine.addArgument("group_list=" + groupName); - commandLine.addArgument("-v"); - commandLine.addArgument(ZiggyConfiguration.ZIGGY_HOME_ENV); - if (propertiesFileName != null) { + String environment = environment(); + if (!environment.isEmpty()) { commandLine.addArgument("-v"); - commandLine.addArgument(ZiggyConfiguration.PIPELINE_CONFIG_PATH_ENV); + commandLine.addArgument(environment); } commandLine.addArgument("-o"); commandLine.addArgument(pbsLogFile(jobIndex)); @@ -115,6 +113,21 @@ private String fullJobName(int jobIndex) { return pipelineTask.taskBaseName() + "." + jobIndex; } + /** + * Concatenates the internal ziggy.environment and user-defined ziggy.pipeline.environment + * variables for use by the qsub -v option. + */ + private String environment() { + ImmutableConfiguration config = ZiggyConfiguration.getInstance(); + String environment = config.getString(PropertyName.ZIGGY_RUNTIME_ENVIRONMENT.property(), + ""); + String userEnvironment = config.getString(PropertyName.RUNTIME_ENVIRONMENT.property(), ""); + if (!userEnvironment.isEmpty()) { + environment = environment + (!environment.isEmpty() ? "," : "") + userEnvironment; + } + return environment; + } + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) private String pbsLogFile(int jobIndex) { File pbsLogFile = new File(pbsLogDir, "pbs-" + fullJobName(jobIndex) + "-" + datestamp); @@ -230,7 +243,7 @@ private void setCoresPerNode(int coresPerNode) { this.coresPerNode = coresPerNode; } - private void setGigsPerNode(int gigsPerNode) { + private void setGigsPerNode(double gigsPerNode) { this.gigsPerNode = gigsPerNode; } @@ -304,7 +317,7 @@ public Builder coresPerNode(int coresPerNode) { return this; } - public Builder gigsPerNode(int gigsPerNode) { + public Builder gigsPerNode(double gigsPerNode) { qsub.setGigsPerNode(gigsPerNode); return this; } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizer.java b/src/main/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizer.java index 4c06852..9c284d4 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizer.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizer.java @@ -6,8 +6,9 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.remote.nas.NasQueueTimeMetrics; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.util.TimeFormatter; +import gov.nasa.ziggy.util.ZiggyStringUtils; /** * Provides data and code in support of selecting an optimal {@link RemoteNodeDescriptor}. @@ -23,19 +24,20 @@ public enum RemoteArchitectureOptimizer { */ CORES { @Override - public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameters, - int totalSubtasks, List acceptableDescriptors) { - if (!remoteParameters.isNodeSharing()) { + public RemoteNodeDescriptor optimalArchitecture( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks, + List acceptableDescriptors) { + if (!executionResources.isNodeSharing()) { log.info( "COST optimization not supported for one subtask per node, using COST instead"); - return COST.optimalArchitecture(remoteParameters, totalSubtasks, + return COST.optimalArchitecture(executionResources, totalSubtasks, acceptableDescriptors); } double coreRatio = 0; RemoteNodeDescriptor optimalDescriptor = null; for (RemoteNodeDescriptor descriptor : acceptableDescriptors) { double newCoreRatio = Math.min(1, - descriptor.getGigsPerCore() / remoteParameters.getGigsPerSubtask()); + descriptor.getGigsPerCore() / executionResources.getGigsPerSubtask()); if (newCoreRatio > coreRatio) { coreRatio = newCoreRatio; optimalDescriptor = descriptor; @@ -51,13 +53,14 @@ public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameter */ QUEUE_DEPTH { @Override - public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameters, - int totalSubtasks, List acceptableDescriptors) { + public RemoteNodeDescriptor optimalArchitecture( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks, + List acceptableDescriptors) { if (acceptableDescriptors.get(0) .getRemoteCluster() .equals(SupportedRemoteClusters.AWS)) { log.info("QUEUE_DEPTH optimization not supported for AWS, using COST instead"); - return COST.optimalArchitecture(remoteParameters, totalSubtasks, + return COST.optimalArchitecture(executionResources, totalSubtasks, acceptableDescriptors); } RemoteNodeDescriptor optimalDescriptor = null; @@ -79,25 +82,27 @@ public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameter */ QUEUE_TIME { @Override - public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameters, - int totalSubtasks, List acceptableDescriptors) { + public RemoteNodeDescriptor optimalArchitecture( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks, + List acceptableDescriptors) { if (acceptableDescriptors.get(0) .getRemoteCluster() .equals(SupportedRemoteClusters.AWS)) { log.info("QUEUE_DEPTH optimization not supported for AWS, using COST instead"); - return COST.optimalArchitecture(remoteParameters, totalSubtasks, + return COST.optimalArchitecture(executionResources, totalSubtasks, acceptableDescriptors); } - RemoteParameters duplicateParameters = new RemoteParameters(remoteParameters); + PipelineDefinitionNodeExecutionResources duplicateParameters = new PipelineDefinitionNodeExecutionResources( + executionResources); RemoteNodeDescriptor optimalDescriptor = null; double minQueueTime = Double.MAX_VALUE; for (RemoteNodeDescriptor descriptor : acceptableDescriptors) { double queueTimeFactor = NasQueueTimeMetrics.queueTime(descriptor); duplicateParameters.setRemoteNodeArchitecture(descriptor.getNodeName()); - duplicateParameters.setMinCoresPerNode(Integer.toString(descriptor.getMaxCores())); - duplicateParameters.setMinGigsPerNode(Integer.toString(descriptor.getMaxGigs())); + duplicateParameters.setMinCoresPerNode(descriptor.getMaxCores()); + duplicateParameters.setMinGigsPerNode(descriptor.getMaxGigs()); PbsParameters pbsParameters = duplicateParameters.pbsParametersInstance(); - pbsParameters.populateResourceParameters(remoteParameters, totalSubtasks); + pbsParameters.populateResourceParameters(executionResources, totalSubtasks); double totalTime = queueTimeFactor * TimeFormatter .timeStringHhMmSsToTimeInHours(pbsParameters.getRequestedWallTime()); if (totalTime < minQueueTime) { @@ -112,18 +117,20 @@ public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameter /** Selects the architecture that minimizes job cost. */ COST { @Override - public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameters, - int totalSubtasks, List acceptableDescriptors) { + public RemoteNodeDescriptor optimalArchitecture( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks, + List acceptableDescriptors) { - RemoteParameters duplicateParameters = new RemoteParameters(remoteParameters); + PipelineDefinitionNodeExecutionResources duplicateParameters = new PipelineDefinitionNodeExecutionResources( + executionResources); RemoteNodeDescriptor optimalArchitecture = null; double minimumCost = Double.MAX_VALUE; for (RemoteNodeDescriptor descriptor : acceptableDescriptors) { duplicateParameters.setRemoteNodeArchitecture(descriptor.getNodeName()); - duplicateParameters.setMinCoresPerNode(Integer.toString(descriptor.getMaxCores())); - duplicateParameters.setMinGigsPerNode(Integer.toString(descriptor.getMaxGigs())); + duplicateParameters.setMinCoresPerNode(descriptor.getMaxCores()); + duplicateParameters.setMinGigsPerNode(descriptor.getMaxGigs()); PbsParameters pbsParameters = duplicateParameters.pbsParametersInstance(); - pbsParameters.populateResourceParameters(remoteParameters, totalSubtasks); + pbsParameters.populateResourceParameters(executionResources, totalSubtasks); if (pbsParameters.getEstimatedCost() < minimumCost) { minimumCost = pbsParameters.getEstimatedCost(); optimalArchitecture = descriptor; @@ -135,8 +142,9 @@ public RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameter private static final Logger log = LoggerFactory.getLogger(RemoteArchitectureOptimizer.class); - public abstract RemoteNodeDescriptor optimalArchitecture(RemoteParameters remoteParameters, - int totalSubtasks, List acceptableDescriptors); + public abstract RemoteNodeDescriptor optimalArchitecture( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks, + List acceptableDescriptors); /** * Finds an optimizer option for a given String. The matching between the String and the @@ -166,6 +174,6 @@ public static String options() { @Override public String toString() { - return StringUtils.constantToSentenceWithSpaces(super.toString()); + return ZiggyStringUtils.constantToSentenceWithSpaces(super.toString()); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptor.java b/src/main/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptor.java index bb93bb1..7d06a27 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptor.java @@ -4,60 +4,76 @@ import java.util.List; import java.util.stream.Collectors; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; /** * Defines the architectures of remote nodes that can be used for Ziggy batch jobs. */ public enum RemoteNodeDescriptor { - // Cost factors for the NAS systems are the SBU2 factors + // Cost factors for the NAS systems are the SBU2 factors. + // Costs effective 2018-10-01 are listed at: + // https://www.nas.nasa.gov/hecc/support/kb/news/new-standard-billing-unit-(sbu)-rates-effective-october-1_443.html + // Check for updates in the news at: + // https://www.nas.nasa.gov/hecc/support/kb/search/?s=1&sb=&q=+Standard+Billing+Unit++Rates&in=news&by=all&period=all&pv=u&date_from=20240111&is_from=1&date_to=20240111&is_to=1 + /** Specifies that any architecture may be used. */ ANY, /** * The Sandy Bridge architecture. See Preparing to Run on Pleiades Sandy Bridge Nodes. + * "https://www.nas.nasa.gov/hecc/support/kb/preparing-to-run-on-pleiades-sandy-bridge-nodes_322.html"> + * Preparing to Run on Pleiades Sandy Bridge Nodes. */ SANDY_BRIDGE("san", 16, 16, 2, 0.47, SupportedRemoteClusters.NAS), /** * The Ivy Bridge architecture. See Preparing to Run on Pleiades Ivy Bridge Nodes. + * "https://www.nas.nasa.gov/hecc/support/kb/preparing-to-run-on-pleiades-ivy-bridge-nodes_446.html"> + * Preparing to Run on Pleiades Ivy Bridge Nodes. */ IVY_BRIDGE("ivy", 20, 20, 3.2, 0.66, SupportedRemoteClusters.NAS), /** * The Broadwell architecture. See Preparing to Run on Pleiades Broadwell nodes. + * "https://www.nas.nasa.gov/hecc/support/kb/preparing-to-run-on-pleiades-broadwell-nodes_530.html"> + * Preparing to Run on Pleiades Broadwell nodes. */ BROADWELL("bro", 28, 28, 4.57, 1.00, SupportedRemoteClusters.NAS), /** * The Haswell architecture. See Preparing to Run on Pleiades Haswell Nodes. + * "https://www.nas.nasa.gov/hecc/support/kb/preparing-to-run-on-pleiades-haswell-nodes_491.html"> + * Preparing to Run on Pleiades Haswell Nodes. */ HASWELL("has", 24, 24, 5.33, 0.80, SupportedRemoteClusters.NAS), /** * The Skylake architecture. See Preparing to Run on Electra Skylake nodes. + * "https://www.nas.nasa.gov/hecc/support/kb/preparing-to-run-on-electra-skylake-nodes_551.html"> + * Preparing to Run on Electra Skylake nodes. */ SKYLAKE("sky_ele", 40, 40, 4.8, 1.59, SupportedRemoteClusters.NAS), - CASCADE_LAKE("cas_ait", 40, 40, 4.0, 1.64, SupportedRemoteClusters.NAS), + /** + * The Cascade architecture. See + * Preparing to Run on Aitken Cascade Lake Nodes. + */ + CASCADE_LAKE("cas_ait", 40, 40, 4.8, 1.64, SupportedRemoteClusters.NAS), + /** + * The Rome architecture. See + * Preparing to Run on Aitken Rome Nodes. + */ ROME("rom_ait", 128, 128, 4.0, 4.06, SupportedRemoteClusters.NAS), // Cost factors for AWS architectures is based on the actual cost in $/hour for the // least expensive node in each family (i.e., the one with the number of cores shown // in the min cores for that architecture), EBS only storage, and up to 10 gigabit // network bandwidth + /** * AWS C5 architecture. This is the one with the smallest ratio of RAM per core. */ @@ -304,6 +320,6 @@ public static RemoteNodeDescriptor[] allDescriptors() { @Override public String toString() { - return StringUtils.constantToCamelWithSpaces(super.toString()); + return ZiggyStringUtils.constantToCamelWithSpaces(super.toString()); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/RemoteQueueDescriptor.java b/src/main/java/gov/nasa/ziggy/module/remote/RemoteQueueDescriptor.java index ce198ec..13763e9 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/RemoteQueueDescriptor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/RemoteQueueDescriptor.java @@ -12,29 +12,32 @@ /** * Describes the available remote queues and their properties. * + * @see https://www.nas.nasa.gov/hecc/support/kb/pbs-job-queue-structure_187.html * @author PT */ public enum RemoteQueueDescriptor implements Comparator { ANY, - LOW("low", 4.0, SupportedRemoteClusters.NAS, true), - NORMAL("normal", 12.0, SupportedRemoteClusters.NAS, true), - LONG("long", 120.0, SupportedRemoteClusters.NAS, true), - DEVEL("devel", 2.0, SupportedRemoteClusters.NAS, false), - DEBUG("debug", 2.0, SupportedRemoteClusters.NAS, false), - RESERVED("reserved", Double.MAX_VALUE, SupportedRemoteClusters.NAS, false), - CLOUD("cloud", Double.MAX_VALUE, SupportedRemoteClusters.AWS, true), - UNKNOWN("", Double.MAX_VALUE, SupportedRemoteClusters.NAS, false); + LOW("low", 4.0, Integer.MAX_VALUE, SupportedRemoteClusters.NAS, true), + NORMAL("normal", 8.0, Integer.MAX_VALUE, SupportedRemoteClusters.NAS, true), + LONG("long", 120.0, Integer.MAX_VALUE, SupportedRemoteClusters.NAS, true), + DEVEL("devel", 2.0, 1, SupportedRemoteClusters.NAS, false), + DEBUG("debug", 2.0, 2, SupportedRemoteClusters.NAS, false), + RESERVED("reserved", Double.MAX_VALUE, Integer.MAX_VALUE, SupportedRemoteClusters.NAS, false), + CLOUD("cloud", Double.MAX_VALUE, Integer.MAX_VALUE, SupportedRemoteClusters.AWS, true), + UNKNOWN("", Double.MAX_VALUE, Integer.MAX_VALUE, SupportedRemoteClusters.NAS, false); private String queueName; private double maxWallTimeHours; + private int maxNodes; private SupportedRemoteClusters remoteCluster; private boolean autoSelectable; - RemoteQueueDescriptor(String queueName, double maxWallTimeHours, + RemoteQueueDescriptor(String queueName, double maxWallTimeHours, int maxNodes, SupportedRemoteClusters supportedCluster, boolean autoSelectable) { this.queueName = queueName; this.maxWallTimeHours = maxWallTimeHours; + this.maxNodes = maxNodes; remoteCluster = supportedCluster; this.autoSelectable = autoSelectable; } @@ -50,6 +53,10 @@ public double getMaxWallTimeHours() { return maxWallTimeHours; } + public int getMaxNodes() { + return maxNodes; + } + public SupportedRemoteClusters getRemoteCluster() { return remoteCluster; } @@ -130,6 +137,6 @@ public static RemoteQueueDescriptor[] allDescriptors() { @Override public String toString() { - return gov.nasa.ziggy.util.StringUtils.constantToSentenceWithSpaces(super.toString()); + return gov.nasa.ziggy.util.ZiggyStringUtils.constantToSentenceWithSpaces(super.toString()); } } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java b/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java index e8fc5c8..1ccde3f 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/aws/AwsExecutor.java @@ -3,8 +3,8 @@ import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.module.remote.RemoteExecutor; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.module.remote.SupportedRemoteClusters; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** @@ -26,19 +26,19 @@ protected AwsExecutor(PipelineTask pipelineTask) { * parameters can be determined. */ @Override - public PbsParameters generatePbsParameters(RemoteParameters remoteParameters, - int totalSubtasks) { + public PbsParameters generatePbsParameters( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks) { - PbsParameters pbsParameters = remoteParameters.pbsParametersInstance(); - pbsParameters.populateArchitecture(remoteParameters, totalSubtasks, + PbsParameters pbsParameters = executionResources.pbsParametersInstance(); + pbsParameters.populateArchitecture(executionResources, totalSubtasks, SupportedRemoteClusters.AWS); // We need to make sure that we request a minimum of cores and gigs that is sufficient // to run at least 1 subtask per node - double gigsPerSubtask = remoteParameters.getGigsPerSubtask(); + double gigsPerSubtask = executionResources.getGigsPerSubtask(); int minCoresPerNode = (int) Math .ceil(gigsPerSubtask / pbsParameters.getArchitecture().getGigsPerCore()); - int minGigsPerNode = (int) Math.ceil(remoteParameters.getGigsPerSubtask()); + int minGigsPerNode = (int) Math.ceil(executionResources.getGigsPerSubtask()); // We also need to make sure we don't ask for less than the minimum available for // the specified architecture @@ -67,7 +67,7 @@ public PbsParameters generatePbsParameters(RemoteParameters remoteParameters, pbsParameters.setMinGigsPerNode(requestedMinGigsPerNode); } - pbsParameters.populateResourceParameters(remoteParameters, totalSubtasks); + pbsParameters.populateResourceParameters(executionResources, totalSubtasks); return pbsParameters; } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java index 85a8a81..1d913ee 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasExecutor.java @@ -3,8 +3,8 @@ import gov.nasa.ziggy.module.StateFile; import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.module.remote.RemoteExecutor; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.module.remote.SupportedRemoteClusters; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** @@ -23,11 +23,11 @@ public NasExecutor(PipelineTask pipelineTask) { * must be selected and the resource parameters can then be determined without any further ado. */ @Override - public PbsParameters generatePbsParameters(RemoteParameters remoteParameters, - int totalSubtasks) { + public PbsParameters generatePbsParameters( + PipelineDefinitionNodeExecutionResources executionResources, int totalSubtasks) { - PbsParameters pbsParameters = remoteParameters.pbsParametersInstance(); - pbsParameters.populateArchitecture(remoteParameters, totalSubtasks, + PbsParameters pbsParameters = executionResources.pbsParametersInstance(); + pbsParameters.populateArchitecture(executionResources, totalSubtasks, SupportedRemoteClusters.NAS); // Pleiades doesn't actually make use of the cores or gigs per node specifications, @@ -35,8 +35,7 @@ public PbsParameters generatePbsParameters(RemoteParameters remoteParameters, // values for the architecture pbsParameters.setMinCoresPerNode(pbsParameters.getArchitecture().getMaxCores()); pbsParameters.setMinGigsPerNode(pbsParameters.getArchitecture().getMaxGigs()); - - pbsParameters.populateResourceParameters(remoteParameters, totalSubtasks); + pbsParameters.populateResourceParameters(executionResources, totalSubtasks); return pbsParameters; } diff --git a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java index 2d96e4d..b96d266 100644 --- a/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java +++ b/src/main/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetrics.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/parameters/InternalParameters.java b/src/main/java/gov/nasa/ziggy/parameters/InternalParameters.java deleted file mode 100644 index 42af5bd..0000000 --- a/src/main/java/gov/nasa/ziggy/parameters/InternalParameters.java +++ /dev/null @@ -1,12 +0,0 @@ -package gov.nasa.ziggy.parameters; - -/** - * This class indicates a parameter class that is intended for internal use only. Classes that - * implement {@link InternalParameters} are not displayed on the GUI and hence cannot be edited via - * the GUI. - * - * @author PT - */ -public class InternalParameters extends Parameters { - -} diff --git a/src/main/java/gov/nasa/ziggy/parameters/ParameterLibraryImportExportCli.java b/src/main/java/gov/nasa/ziggy/parameters/ParameterLibraryImportExportCli.java index cc4f1c6..b43fe40 100644 --- a/src/main/java/gov/nasa/ziggy/parameters/ParameterLibraryImportExportCli.java +++ b/src/main/java/gov/nasa/ziggy/parameters/ParameterLibraryImportExportCli.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/parameters/Parameters.java b/src/main/java/gov/nasa/ziggy/parameters/Parameters.java index 68ce599..f55f790 100644 --- a/src/main/java/gov/nasa/ziggy/parameters/Parameters.java +++ b/src/main/java/gov/nasa/ziggy/parameters/Parameters.java @@ -26,7 +26,7 @@ * @author Bill Wohler */ -public class Parameters extends TypedParameterCollection implements ParametersInterface { +public class Parameters extends TypedParameterCollection { public static final String NAME_FIELD = "name"; @@ -41,22 +41,11 @@ public void setName(String name) { this.name = name; } - @Override - public void validate() { - // Do nothing, by default. - } - @Override public int hashCode() { return Objects.hash(name); } - @Override - public void updateParameter(String name, String value) { - getParametersByName().get(name).setString(value); - populate(getParameters()); - } - // Note: hashCode() and equals() use only the name so that there cannot be any duplicate copies // of the parameter set in a Set. @Override diff --git a/src/main/java/gov/nasa/ziggy/parameters/ParametersInterface.java b/src/main/java/gov/nasa/ziggy/parameters/ParametersInterface.java index df9b4da..b66ce14 100644 --- a/src/main/java/gov/nasa/ziggy/parameters/ParametersInterface.java +++ b/src/main/java/gov/nasa/ziggy/parameters/ParametersInterface.java @@ -15,7 +15,7 @@ public interface ParametersInterface extends Persistable { @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) - default void populate(Set typedParameters) { + default void populate(Collection typedParameters) { setParameters(typedParameters); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineConfigurator.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineConfigurator.java index a3bdfe7..d5567c4 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineConfigurator.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineConfigurator.java @@ -268,7 +268,7 @@ public PipelineDefinitionNode addNode(PipelineModuleDefinition moduleDef, moduleParameterSetNamesMap.put(node, paramSetNamesMap); if (taskGenerator != null) { - node.setUnitOfWorkGenerator(new ClassWrapper<>(taskGenerator)); + moduleDef.setUnitOfWorkGenerator(new ClassWrapper<>(taskGenerator)); } if (currentNode != null) { diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java index 7055835..db45f74 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineExecutor.java @@ -7,10 +7,14 @@ import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.Set; +import java.util.stream.Collectors; +import org.hibernate.Hibernate; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; @@ -18,6 +22,7 @@ import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; @@ -27,18 +32,21 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask.ProcessingSummary; import gov.nasa.ziggy.pipeline.definition.ProcessingState; import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.TypedParameter; import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.pipeline.definition.crud.ProcessingSummaryOperations; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.alert.AlertService.Severity; import gov.nasa.ziggy.services.database.DatabaseService; import gov.nasa.ziggy.services.database.DatabaseTransaction; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; import gov.nasa.ziggy.services.events.ZiggyEventHandler; -import gov.nasa.ziggy.services.events.ZiggyEventLabels; import gov.nasa.ziggy.services.messages.RemoveTaskFromKilledTasksMessage; import gov.nasa.ziggy.services.messages.TaskRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; @@ -48,13 +56,13 @@ /*** * Encapsulates the launch and transition logic for pipelines. *

      - * Note that the methods - * {@link #launch(PipelineDefinition, String, PipelineDefinitionNode, PipelineDefinitionNode, String)} - * and {@link #transitionToNextInstanceNode(PipelineInstance, PipelineTask, TaskCounts)} must not be - * run in the context of a transaction; these methods provide their own transactions in order to - * ensure that the transactions are completed before any task requests can be sent. Other methods, - * including {@link #restartFailedTasks(Collection, boolean, RunMode)}, can (or in some cases must) - * be run in a transaction context. + * Note that the methods {@link #launch(PipelineDefinition, String, PipelineDefinitionNode, + * PipelineDefinitionNode, Set)} and + * {@link #transitionToNextInstanceNode(PipelineInstanceNode, TaskCounts)} must not be run in the + * context of a transaction; these methods provide their own transactions in order to ensure that + * the transactions are completed before any task requests can be sent. Other methods, including + * {@link #restartFailedTasks(Collection, boolean, RunMode)}, can (or in some cases must) be run in + * a transaction context. * * @author Todd Klaus * @author PT @@ -62,22 +70,16 @@ public class PipelineExecutor { private static final Logger log = LoggerFactory.getLogger(PipelineExecutor.class); - private PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud; + /** Map from pipeline instance ID to event labels. */ + private static Map> instanceEventLabels = new HashMap<>(); + + private PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud = new PipelineModuleDefinitionCrud(); private ParameterSetCrud parameterSetCrud; private PipelineInstanceCrud pipelineInstanceCrud; private PipelineInstanceNodeCrud pipelineInstanceNodeCrud; private PipelineTaskCrud pipelineTaskCrud; private PipelineOperations pipelineOperations; - - public PipelineExecutor() { - - pipelineModuleDefinitionCrud = new PipelineModuleDefinitionCrud(); - parameterSetCrud = new ParameterSetCrud(); - pipelineInstanceCrud = new PipelineInstanceCrud(); - pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); - pipelineTaskCrud = new PipelineTaskCrud(); - pipelineOperations = new PipelineOperations(); - } + private PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud = new PipelineDefinitionNodeCrud(); /** * Launch a new {@link PipelineInstance} for this {@link PipelineDefinition} with optional @@ -86,8 +88,7 @@ public PipelineExecutor() { * transfer information to the pipeline that it needs at runtime. */ public PipelineInstance launch(PipelineDefinition pipeline, String instanceName, - PipelineDefinitionNode startNode, PipelineDefinitionNode endNode, - String eventHandlerParamSetName) { + PipelineDefinitionNode startNode, PipelineDefinitionNode endNode, Set eventLabels) { List nodesForLaunch = new ArrayList<>(); @@ -121,10 +122,6 @@ public PipelineInstance launch(PipelineDefinition pipeline, String instanceName, */ Map, String> triggerParamNames = pipeline .getPipelineParameterSetNames(); - if (eventHandlerParamSetName != null) { - triggerParamNames.put(new ClassWrapper<>(ZiggyEventLabels.class), - eventHandlerParamSetName); - } Map, ParameterSet> instanceParams = instance .getPipelineParameterSets(); @@ -132,7 +129,7 @@ public PipelineInstance launch(PipelineDefinition pipeline, String instanceName, instance.setPipelineParameterSets(instanceParams); - pipelineInstanceCrud.persist(instance); + pipelineInstanceCrud().persist(instance); if (startNode == null) { // start at the root @@ -163,6 +160,9 @@ public PipelineInstance launch(PipelineDefinition pipeline, String instanceName, // make sure the new PipelineInstanceNodes are in the db for // launchNode, below DatabaseService.getInstance().flush(); + if (eventLabels != null) { + instanceEventLabels.put(instance.getId(), eventLabels); + } return instance; }); @@ -173,7 +173,7 @@ public PipelineInstance launch(PipelineDefinition pipeline, String instanceName, // Get the current state of the pipeline instance from the database and return same. return (PipelineInstance) DatabaseTransactionFactory - .performTransaction(() -> pipelineInstanceCrud.retrieve(pipelineInstance.getId())); + .performTransaction(() -> pipelineInstanceCrud().retrieve(pipelineInstance.getId())); } /** @@ -207,11 +207,17 @@ public void transitionToNextInstanceNode(PipelineInstanceNode instanceNode, // Retrieve instance nodes for the remaining definition nodes (there might not // be any if we're at the end of the pipeline instance). for (PipelineDefinitionNode node : nextPipelineDefinitionNodesNewUowTransition) { - PipelineInstanceNode nextInstanceNode = pipelineInstanceNodeCrud + PipelineInstanceNode nextInstanceNode = pipelineInstanceNodeCrud() .retrieve(instanceNode.getPipelineInstance(), node); if (nextInstanceNode != null) { log.info("Launching node {} with a new UOW", node.getModuleName()); nodesForLaunch.add(nextInstanceNode); + Hibernate.initialize(nextInstanceNode.getPipelineInstance()); + Hibernate.initialize(nextInstanceNode.getPipelineDefinitionNode()); + Hibernate + .initialize(nextInstanceNode.getPipelineDefinitionNode().getNextNodes()); + Hibernate.initialize( + nextInstanceNode.getPipelineDefinitionNode().getInputDataFileTypes()); } } return null; @@ -228,7 +234,7 @@ public void transitionToNextInstanceNode(PipelineInstanceNode instanceNode, * Called after the transition logic runs. */ public void logUpdatedInstanceState(PipelineInstance pipelineInstance) { - TaskCounts state = pipelineOperations.taskCounts(pipelineInstance); + TaskCounts state = pipelineOperations().taskCounts(pipelineInstance); log.info(""" updateInstanceState: all nodes:\s\ @@ -283,7 +289,7 @@ private void restartFailedTasksForNode( log.info("Restarting {} tasks for instance node {} ({})", tasks.size(), node.getId(), node.getPipelineDefinitionNode().getModuleName()); - pipelineOperations.taskCounts(node); + pipelineOperations().taskCounts(node); logInstanceNodeCounts(node, "initial"); // Loop over tasks and prepare for restart, including sending the task request message. @@ -294,13 +300,13 @@ private void restartFailedTasksForNode( // Update and log the instance state. PipelineInstance instance = node.getPipelineInstance(); logUpdatedInstanceState(instance); - pipelineInstanceCrud.merge(instance); + pipelineInstanceCrud().merge(instance); logInstanceNodeCounts(node, "final"); } private void logInstanceNodeCounts(PipelineInstanceNode node, String initialOrFinal) { - TaskCounts instanceNodeCounts = pipelineOperations.taskCounts(node); + TaskCounts instanceNodeCounts = pipelineOperations().taskCounts(node); log.info(""" node {} state:\s\ numTasks/numSubmittedTasks/numCompletedTasks/numFailedTasks =\s\s\ @@ -336,12 +342,12 @@ private void restartFailedTask(PipelineTask task, boolean doTransitionOnly, // Retrieve the task so that it can be modified in the database using the Hibernate // infrastructure - PipelineTask databaseTask = pipelineTaskCrud.retrieve(task.getId()); + PipelineTask databaseTask = pipelineTaskCrud().retrieve(task.getId()); log.info("Restarting task id={}, oldState : {}", databaseTask.getId(), oldState); databaseTask.setRetry(true); - pipelineOperations.setTaskState(databaseTask, PipelineTask.State.SUBMITTED); - removeTaskFromKilledTaskList(pipelineTaskCrud.merge(databaseTask).getId()); + pipelineOperations().setTaskState(databaseTask, PipelineTask.State.SUBMITTED); + removeTaskFromKilledTaskList(pipelineTaskCrud().merge(databaseTask).getId()); // Send the task message to the supervisor. sendTaskRequestMessage(databaseTask, Priority.HIGHEST, doTransitionOnly, restartMode); @@ -421,11 +427,19 @@ private void launchNode(PipelineInstanceNode instanceNode, log.debug("Generating tasks"); List tasks = new ArrayList<>(); - tasks = unitOfWorkGenerator.newInstance().generateUnitsOfWork(uowParams); + tasks = generateUnitsOfWork(unitOfWorkGenerator.newInstance(), instanceNode, instance); log.info("Generated " + tasks.size() + " tasks for pipeline definition node " + instanceNode.getPipelineDefinitionNode().getModuleName()); if (tasks.isEmpty()) { + AlertService.getInstance() + .generateAndBroadcastAlert("PI", AlertService.DEFAULT_TASK_ID, Severity.ERROR, + "No tasks generated for " + + instanceNode.getPipelineDefinitionNode().getModuleName()); + DatabaseTransactionFactory.performTransaction(() -> { + pipelineOperations().setInstanceToErrorsStalledState(instance); + return null; + }); throw new PipelineException("Task generation did not generate any tasks! UOW class: " + unitOfWorkGenerator.getClassName()); } @@ -433,7 +447,7 @@ private void launchNode(PipelineInstanceNode instanceNode, } public ClassWrapper unitOfWorkGenerator(PipelineDefinitionNode node) { - return UnitOfWorkGenerator.unitOfWorkGenerator(node); + return pipelineModuleDefinitionCrud.retrieveUnitOfWorkGenerator(node.getModuleName()); } /** @@ -455,7 +469,7 @@ private PipelineInstanceNode createInstanceNodes(PipelineInstance instance, PipelineInstanceNode instanceNode = new PipelineInstanceNode(instance, node, moduleDefinition); - pipelineInstanceNodeCrud.persist(instanceNode); + pipelineInstanceNodeCrud().persist(instanceNode); Map, String> pipelineNodeParameters = node .getModuleParameterSetNames(); @@ -487,7 +501,7 @@ private void bindParameters(Map, String> param Map, ParameterSet> params) { for (ClassWrapper paramClass : paramNames.keySet()) { String pipelineParamName = paramNames.get(paramClass); - ParameterSet paramSet = parameterSetCrud + ParameterSet paramSet = parameterSetCrud() .retrieveLatestVersionForName(pipelineParamName); params.put(paramClass, paramSet); paramSet.lock(); @@ -500,17 +514,21 @@ private void bindParameters(Map, String> param private void launchTasks(PipelineInstanceNode instanceNode, PipelineInstance instance, List tasks) { List pipelineTasks = new ArrayList<>(); + PipelineDefinitionNodeExecutionResources resources = pipelineDefinitionNodeCrud + .retrieveExecutionResources(instanceNode.getPipelineDefinitionNode()); for (UnitOfWork task : tasks) { PipelineTask pipelineTask = new PipelineTask(instance, instanceNode); pipelineTask.setState(PipelineTask.State.SUBMITTED); pipelineTask.setUowTaskParameters(task.getParameters()); + pipelineTask.setMaxFailedSubtaskCount(resources.getMaxFailedSubtaskCount()); + pipelineTask.setMaxAutoResubmits(resources.getMaxAutoResubmits()); pipelineTasks.add(pipelineTask); } DatabaseTransactionFactory.performTransaction(new DatabaseTransaction() { @Override public Void transaction() { - pipelineTaskCrud.persist(pipelineTasks); + pipelineTaskCrud().persist(pipelineTasks); return null; } @@ -541,45 +559,75 @@ private void sendTaskRequestMessage(PipelineTask task, Priority priority, + task.getModuleName()); TaskRequest taskRequest = new TaskRequest(task.pipelineInstanceId(), - task.pipelineInstanceNodeId(), task.getPipelineDefinitionNode().getId(), task.getId(), + task.pipelineInstanceNodeId(), task.pipelineDefinitionNode().getId(), task.getId(), priority, doTransitionOnly, runMode); ZiggyMessenger.publish(taskRequest); } - /** - * For mocking purposes only - * - * @param pipelineInstanceCrud the pipelineInstanceCrud to set - */ - public void setPipelineInstanceCrud(PipelineInstanceCrud pipelineInstanceCrud) { - this.pipelineInstanceCrud = pipelineInstanceCrud; + public static List generateUnitsOfWork(UnitOfWorkGenerator uowGenerator, + PipelineInstanceNode pipelineInstanceNode) { + return generateUnitsOfWork(uowGenerator, pipelineInstanceNode, null); } /** - * For mocking purposes only - * - * @param pipelineTaskCrud the pipelineTaskCrud to set + * Generates the set of UOWs using the + * {@link UnitOfWorkGenerator#generateUnitsOfWork(PipelineInstanceNode)} and method of a given + * {@link UnitOfWorkGenerator} implementation. The resulting {@link UnitOfWork} instance will + * also contain a property that specifies the class name of the generator. */ - public void setPipelineTaskCrud(PipelineTaskCrud pipelineTaskCrud) { - this.pipelineTaskCrud = pipelineTaskCrud; + public static List generateUnitsOfWork(UnitOfWorkGenerator uowGenerator, + PipelineInstanceNode pipelineInstanceNode, PipelineInstance instance) { + + // Produce the tasks. + Set eventLabels = instance != null ? instanceEventLabels.get(instance.getId()) + : null; + List uows = uowGenerator.unitsOfWork(pipelineInstanceNode, eventLabels); + + // Add some metadata parameters to all the instances. + for (UnitOfWork uow : uows) { + uow.addParameter(new TypedParameter(UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, + uowGenerator.getClass().getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); + } + + // Now that the UOWs have their brief states properly assigned, sort them by brief state + // and return. + return uows.stream().sorted().collect(Collectors.toList()); + } + + public PipelineInstanceCrud pipelineInstanceCrud() { + if (pipelineInstanceCrud == null) { + pipelineInstanceCrud = new PipelineInstanceCrud(); + } + return pipelineInstanceCrud; } - public void setPipelineInstanceNodeCrud(PipelineInstanceNodeCrud pipelineInstanceNodeCrud) { - this.pipelineInstanceNodeCrud = pipelineInstanceNodeCrud; + public PipelineTaskCrud pipelineTaskCrud() { + if (pipelineTaskCrud == null) { + pipelineTaskCrud = new PipelineTaskCrud(); + } + return pipelineTaskCrud; } - void setPipelineModuleDefinitionCrud( - PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud) { - this.pipelineModuleDefinitionCrud = pipelineModuleDefinitionCrud; + public PipelineInstanceNodeCrud pipelineInstanceNodeCrud() { + if (pipelineInstanceNodeCrud == null) { + pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); + } + return pipelineInstanceNodeCrud; } - void setPipelineModuleParameterSetCrud(ParameterSetCrud parameterSetCrud) { - this.parameterSetCrud = parameterSetCrud; + public ParameterSetCrud parameterSetCrud() { + if (parameterSetCrud == null) { + parameterSetCrud = new ParameterSetCrud(); + } + return parameterSetCrud; } - public void setPipelineOperations(PipelineOperations pipelineOperations) { - this.pipelineOperations = pipelineOperations; + public PipelineOperations pipelineOperations() { + if (pipelineOperations == null) { + pipelineOperations = new PipelineOperations(); + } + return pipelineOperations; } public boolean taskRequestEnabled() { diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineOperations.java index ed0749e..287fca4 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineOperations.java @@ -4,7 +4,6 @@ import java.io.IOException; import java.io.UncheckedIOException; import java.util.HashMap; -import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Map.Entry; @@ -35,7 +34,6 @@ import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; -import gov.nasa.ziggy.uow.UnitOfWorkGenerator; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -64,32 +62,6 @@ public ParameterSet retrieveLatestParameterSet(String parameterSetName) { return crud.retrieveLatestVersionForName(parameterSetName); } - /** - * Returns a {@link Set} containing all {@link Parameters} classes required by the specified - * {@link PipelineDefinitionNode}. This is a union of the Parameters classes required by the - * PipelineModule itself and the Parameters classes required by the UnitOfWorkTaskGenerator - * associated with the node. - */ - public Set> retrieveRequiredParameterClassesForNode( - PipelineDefinitionNode pipelineNode) { - PipelineModuleDefinitionCrud modDefCrud = new PipelineModuleDefinitionCrud(); - PipelineModuleDefinition modDef = modDefCrud - .retrieveLatestVersionForName(pipelineNode.getModuleName()); - - Set> allRequiredParams = new HashSet<>(); - - List> uowParams = UnitOfWorkGenerator - .unitOfWorkGenerator(pipelineNode) - .newInstance() - .requiredParameterClasses(); - for (Class uowParam : uowParams) { - allRequiredParams.add(new ClassWrapper<>(uowParam)); - } - allRequiredParams.addAll(modDef.getRequiredParameterClasses()); - - return allRequiredParams; - } - /** * Update the specified {@link ParameterSet} with the specified {@link Parameters}. *

      @@ -184,8 +156,7 @@ public boolean compareParameters(ParametersInterface currentParameters, * driven by an event handler. */ public PipelineInstance fireTrigger(PipelineDefinition pipelineDefinition, String instanceName, - PipelineDefinitionNode startNode, PipelineDefinitionNode endNode, - String eventHandlerParamSetName) { + PipelineDefinitionNode startNode, PipelineDefinitionNode endNode, Set eventLabels) { TriggerValidationResults validationResults = validateTrigger(pipelineDefinition); if (validationResults.hasErrors()) { throw new PipelineException( @@ -193,7 +164,7 @@ public PipelineInstance fireTrigger(PipelineDefinition pipelineDefinition, Strin } return pipelineExecutor().launch(pipelineDefinition, instanceName, startNode, endNode, - eventHandlerParamSetName); + eventLabels); } /** @@ -208,45 +179,9 @@ public TriggerValidationResults validateTrigger(PipelineDefinition pipelineDefin TriggerValidationResults validationResults = new TriggerValidationResults(); pipelineDefinition.buildPaths(); - - validateTriggerParameters(pipelineDefinition, validationResults); - return validationResults; } - /** - * Validate that the trigger {@link ParameterSetName}s are all set and match the parameter - * classes specified in the {@link PipelineDefinition} - */ - private void validateTriggerParameters(PipelineDefinition pipelineDefinition, - TriggerValidationResults validationResults) { - validateParameterClassExists(pipelineDefinition.getPipelineParameterSetNames(), - "Pipeline parameters", validationResults); - - for (PipelineDefinitionNode rootNode : pipelineDefinition.getRootNodes()) { - validateTriggerParametersForNode(pipelineDefinition, rootNode, validationResults); - } - } - - private void validateTriggerParametersForNode(PipelineDefinition pipelineDefinition, - PipelineDefinitionNode pipelineDefinitionNode, TriggerValidationResults validationResults) { - String errorLabel = "module: " + pipelineDefinitionNode.getModuleName(); - - Set> requiredParameterClasses = retrieveRequiredParameterClassesForNode( - pipelineDefinitionNode); - - validateParameterClassExists(pipelineDefinitionNode.getModuleParameterSetNames(), - errorLabel, validationResults); - - validateTriggerParameters(requiredParameterClasses, - pipelineDefinition.getPipelineParameterSetNames(), - pipelineDefinitionNode.getModuleParameterSetNames(), errorLabel, validationResults); - - for (PipelineDefinitionNode childNode : pipelineDefinitionNode.getNextNodes()) { - validateTriggerParametersForNode(pipelineDefinition, childNode, validationResults); - } - } - /** * Validate that the trigger {@link ParameterSetName}s are all set and match the parameter * classes specified in the {@link PipelineDefinition} for a given trigger node (module) @@ -412,6 +347,18 @@ public void updateInstanceState(PipelineTask pipelineTask, TaskCounts instanceNo instance.setState(state); } + /** + * Forces a {@link PipelineInstance} into the ERRORS_STALLED state. This should only be used in + * the peculiar circumstance of an instance that has failed without any {@link PipelineTask}s + * associated with the instance failing. This is the case when UOW generation for a given + * {@link PipelineInstanceNode} has failed in some way. + */ + public void setInstanceToErrorsStalledState(PipelineInstance pipelineInstance) { + PipelineInstance.State.ERRORS_STALLED.setExecutionClockState(pipelineInstance); + pipelineInstance.setState(PipelineInstance.State.ERRORS_STALLED); + new PipelineInstanceCrud().merge(pipelineInstance); + } + /** * Returns a {@link TaskCounts} instance for a given {@link PipelineInstance}. */ @@ -480,7 +427,7 @@ public String generatePedigreeReport(PipelineInstance instance) { } report.append(nl); - report.append("Modules" + nl); + report.append("Nodes" + nl); List pipelineNodes = pipelineInstanceNodeCrud.retrieveAll(instance); @@ -550,7 +497,7 @@ public String generatePipelineReport(PipelineDefinition pipelineDefinition) { } report.append(nl); - report.append("Modules" + nl); + report.append("Nodes" + nl); List nodes = pipelineDefinition.getNodes(); for (PipelineDefinitionNode node : nodes) { @@ -582,8 +529,6 @@ private void appendModule(String nl, StringBuilder report, PipelineModuleDefinit " Module definition: " + module.getName() + ", version=" + module.getVersion() + nl); report.append(" Java classname: " + module.getPipelineModuleClass().getClazz().getSimpleName() + nl); - report.append(" exe timeout seconds: " + module.getExeTimeoutSecs() + nl); - report.append(" min memory MB: " + module.getMinMemoryMegaBytes() + nl); } @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskDebugger.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskDebugger.java index 4fd920d..f102692 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskDebugger.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskDebugger.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java b/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java index 4d74f3f..d1111ab 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/PipelineTaskInformation.java @@ -8,8 +8,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.module.PipelineInputs; +import gov.nasa.ziggy.module.PipelineInputsOutputsUtils; import gov.nasa.ziggy.module.SubtaskInformation; import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.parameters.ParametersInterface; @@ -100,18 +100,6 @@ static void setInstance(PipelineTaskInformation newInstance) { */ private static Map> subtaskInformationMap = new HashMap<>(); - /** - * Cache of {@link ParameterSetName}s for {@link RemoteParameters} instances, organized by - * {@link PipelineDefinitionNode}. - */ - private static Map remoteParametersMap = new HashMap<>(); - - /** - * Cache that stores information on whether a given module has limits on the number of subtasks - * that can be processed in parallel. - */ - private static Map modulesWithParallelLimitsMap = new HashMap<>(); - /** * Deletes the cached information for a given {@link PipelineDefinitionNode}. Used when the user * is aware of changes that should force recalculation. @@ -120,12 +108,6 @@ public synchronized static void reset(PipelineDefinitionNode triggerDefinitionNo if (subtaskInformationMap.containsKey(triggerDefinitionNode)) { subtaskInformationMap.put(triggerDefinitionNode, null); } - if (remoteParametersMap.containsKey(triggerDefinitionNode)) { - remoteParametersMap.remove(triggerDefinitionNode); - } - if (modulesWithParallelLimitsMap.containsKey(triggerDefinitionNode)) { - remoteParametersMap.remove(triggerDefinitionNode); - } } public synchronized static boolean hasPipelineDefinitionNode(PipelineDefinitionNode node) { @@ -146,30 +128,6 @@ public static synchronized List subtaskInformation( return subtaskInformationMap.get(node); } - /** - * Returns the name of the {@link ParameterSet} for a specified node's {@link RemoteParameters} - * instance. If the module has no such parameter set, null is returned. - */ - public static synchronized String remoteParameters(PipelineDefinitionNode node) { - if (!hasPipelineDefinitionNode(node)) { - generateSubtaskInformation(node); - } - return remoteParametersMap.get(node); - } - - /** - * Determines whether a given {@link PipelineDefinitionNode} corresponds to a module that limits - * the maximum number of subtasks that can be processed in parallel (this is usually the case - * for a module that is forced to perform its processing in multiple steps, where each step - * processes a unique set of subtasks). - */ - public static synchronized boolean parallelLimits(PipelineDefinitionNode node) { - if (!hasPipelineDefinitionNode(node)) { - generateSubtaskInformation(node); - } - return modulesWithParallelLimitsMap.get(node); - } - /** * Calculation engine that generates the {@link List} of {@link SubtaskInformation} instances * for a given {@link PipelineDefinitionNode}. The calculation first generates the @@ -211,42 +169,9 @@ private static synchronized void generateSubtaskInformation(PipelineDefinitionNo ClassWrapper unitOfWorkGenerator = instance.unitOfWorkGenerator(node); - // Produce a combined map from Parameter classes to Parameter instances - Map, ParameterSet> compositeParameterSets = new HashMap<>( - pipelineInstance.getPipelineParameterSets()); - - for (ClassWrapper moduleParameterClass : instanceNode - .getModuleParameterSets() - .keySet()) { - if (compositeParameterSets.containsKey(moduleParameterClass)) { - throw new PipelineException( - "Configuration Error: Module parameter and pipeline parameter Maps both contain a value for parameter class: " - + moduleParameterClass); - } - compositeParameterSets.put(moduleParameterClass, - instanceNode.getModuleParameterSets().get(moduleParameterClass)); - } - - // Set the flag that indicates whether this module limits the number of subtasks that - // can run in parallel - - modulesWithParallelLimitsMap.put(node, instance.parallelLimits(moduleDefinition)); - Map, ParametersInterface> uowParams = new HashMap<>(); - - for (ClassWrapper parametersClass : compositeParameterSets.keySet()) { - ParameterSet parameterSet = compositeParameterSets.get(parametersClass); - Class clazz = parametersClass.getClazz(); - if (clazz.equals(RemoteParameters.class)) { - remoteParametersMap.put(node, parameterSet.getName()); - } - uowParams.put(clazz, parameterSet.parametersInstance()); - } - if (!remoteParametersMap.containsKey(node)) { - remoteParametersMap.put(node, null); - } - // Generate the units of work. - List tasks = instance.unitsOfWork(unitOfWorkGenerator, uowParams); + List tasks = instance.unitsOfWork(unitOfWorkGenerator, instanceNode, + pipelineInstance); // Generate the subtask information for all tasks List subtaskInformationList = new LinkedList<>(); @@ -265,9 +190,10 @@ private static synchronized void generateSubtaskInformation(PipelineDefinitionNo * unit tests. */ List unitsOfWork(ClassWrapper wrappedUowGenerator, - Map, ParametersInterface> uowParams) { + PipelineInstanceNode pipelineInstanceNode, PipelineInstance pipelineInstance) { UnitOfWorkGenerator taskGenerator = wrappedUowGenerator.newInstance(); - return taskGenerator.generateUnitsOfWork(uowParams); + return PipelineExecutor.generateUnitsOfWork(taskGenerator, pipelineInstanceNode, + pipelineInstance); } /** @@ -275,7 +201,7 @@ List unitsOfWork(ClassWrapper wrappedUowGenerat * support of unit tests. */ ClassWrapper unitOfWorkGenerator(PipelineDefinitionNode node) { - return UnitOfWorkGenerator.unitOfWorkGenerator(node); + return pipelineModuleDefinitionCrud().retrieveUnitOfWorkGenerator(node.getModuleName()); } /** @@ -288,19 +214,15 @@ PipelineTask pipelineTask(PipelineInstance instance, PipelineInstanceNode instan return pipelineTask; } - boolean parallelLimits(PipelineModuleDefinition moduleDefinition) { - PipelineInputs pipelineInputs = moduleDefinition.getInputsClass().newInstance(); - return pipelineInputs.parallelLimits(); - } - /** * Generates the {@link SubtaskInformation} instance for a single {@link PipelineTask}. * Implemented as an instance method in support of unit tests. */ SubtaskInformation subtaskInformation(PipelineModuleDefinition moduleDefinition, PipelineTask pipelineTask) { - PipelineInputs pipelineInputs = moduleDefinition.getInputsClass().newInstance(); - return pipelineInputs.subtaskInformation(pipelineTask); + PipelineInputs pipelineInputs = PipelineInputsOutputsUtils + .newPipelineInputs(moduleDefinition.getInputsClass(), pipelineTask, null); + return pipelineInputs.subtaskInformation(); } private static void populateParameters( diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/AuditInfo.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/AuditInfo.java index 7dc4ea6..6b95799 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/AuditInfo.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/AuditInfo.java @@ -5,34 +5,33 @@ import org.apache.commons.lang3.builder.ReflectionToStringBuilder; -import gov.nasa.ziggy.services.security.User; -import jakarta.persistence.Column; import jakarta.persistence.Embeddable; import jakarta.persistence.Entity; -import jakarta.persistence.JoinColumn; -import jakarta.persistence.ManyToOne; /** * This {@link Embeddable} class is used by {@link Entity} classes that can be modified through the * UI. It tracks who made the last modification to the object and and when those changes were made, * for audit trail purposes. + *

      + * Instances of {@link AuditInfo} are immutable. When an instance is created, it is created with the + * current date and current user. When a class that uses {@link AuditInfo} is updated, the existing + * instance is replaced with a new one that contains the user and date. * * @author Todd Klaus + * @author PT */ @Embeddable public class AuditInfo { - @ManyToOne - @JoinColumn(name = "lastChangedUser") - private User lastChangedUser = null; - @Column(name = "lastChangedTime") - private Date lastChangedTime = null; + private final String lastChangedUser; + private final Date lastChangedTime; public AuditInfo() { lastChangedTime = new Date(); + lastChangedUser = ProcessHandle.current().info().user().get(); } - public AuditInfo(User lastChangedUser, Date lastChangedTime) { - this.lastChangedUser = lastChangedUser; + public AuditInfo(Date lastChangedTime) { + lastChangedUser = ProcessHandle.current().info().user().get(); this.lastChangedTime = lastChangedTime; } @@ -43,27 +42,13 @@ public Date getLastChangedTime() { return lastChangedTime; } - /** - * @param lastChangedTime the lastChangedTime to set - */ - public void setLastChangedTime(Date lastChangedTime) { - this.lastChangedTime = lastChangedTime; - } - /** * @return the lastChangedUser */ - public User getLastChangedUser() { + public String getLastChangedUser() { return lastChangedUser; } - /** - * @param lastChangedUser the lastChangedUser to set - */ - public void setLastChangedUser(User lastChangedUser) { - this.lastChangedUser = lastChangedUser; - } - @Override public int hashCode() { return Objects.hash(lastChangedTime, lastChangedUser); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java index 7165ccc..83e0c31 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ClassWrapper.java @@ -186,7 +186,7 @@ public static class ClassWrapperAdapter extends XmlAdapter unmarshal(String v) { if (v == null) { return null; @@ -196,9 +196,7 @@ public ClassWrapper unmarshal(String v) { clazz = (Class) Class.forName(v); return new ClassWrapper<>(clazz); } catch (ClassNotFoundException e) { - // This can never occur. The caller provides the string from the name - // of a known existing class. - throw new AssertionError(e); + throw new PipelineException("Class " + v + " is not on classpath", e); } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java index 449911f..846f31a 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/Group.java @@ -1,15 +1,24 @@ package gov.nasa.ziggy.pipeline.definition; +import java.util.HashSet; import java.util.Objects; +import java.util.Set; +import jakarta.persistence.ElementCollection; import jakarta.persistence.Entity; +import jakarta.persistence.GeneratedValue; +import jakarta.persistence.GenerationType; import jakarta.persistence.Id; +import jakarta.persistence.JoinTable; import jakarta.persistence.ManyToOne; +import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; +import jakarta.persistence.Transient; +import jakarta.persistence.UniqueConstraint; /** - * Group identifier for {@link PipelineDefinition}s, {@link PipelineModuleDefinition}s, - * {@link ParameterSet}s, and {@link PipelineInstance}s. + * Group identifier for {@link PipelineDefinition}s, {@link PipelineModuleDefinition}s, and + * {@link ParameterSet}s. *

      * Groups are used in the console to organize these entities into folders since their numbers can * grow large over the course of the mission. @@ -17,11 +26,17 @@ * @author Todd Klaus */ @Entity -@Table(name = "ziggy_Group") +@Table(name = "ziggy_Group", uniqueConstraints = { @UniqueConstraint(columnNames = { "name" }) }) + public class Group { public static final Group DEFAULT = new Group(); @Id + @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_Group_generator") + @SequenceGenerator(name = "ziggy_Group_generator", initialValue = 1, + sequenceName = "ziggy_Group_sequence", allocationSize = 1) + private Long id; + private String name; @ManyToOne @@ -29,6 +44,22 @@ public class Group { private String description; + @ElementCollection + @JoinTable(name = "ziggy_Group_pipelineDefinitionNames") + private Set pipelineDefinitionNames = new HashSet<>(); + + @ElementCollection + @JoinTable(name = "ziggy_Group_pipelineModuleNames") + private Set pipelineModuleNames = new HashSet<>(); + + @ElementCollection + @JoinTable(name = "ziggy_Group_parameterSetNames") + private Set parameterSetNames = new HashSet<>(); + + /** Contains whichever of the above is correct for the current use-case. */ + @Transient + private Set memberNames; + Group() { } @@ -74,6 +105,38 @@ public void setParentGroup(Group parentGroup) { this.parentGroup = parentGroup; } + public Set getPipelineDefinitionNames() { + return pipelineDefinitionNames; + } + + public void setPipelineDefinitionNames(Set pipelineDefinitionNames) { + this.pipelineDefinitionNames = pipelineDefinitionNames; + } + + public Set getPipelineModuleNames() { + return pipelineModuleNames; + } + + public void setPipelineModuleNames(Set pipelineModuleNames) { + this.pipelineModuleNames = pipelineModuleNames; + } + + public Set getParameterSetNames() { + return parameterSetNames; + } + + public void setParameterSetNames(Set parameterSetNames) { + this.parameterSetNames = parameterSetNames; + } + + public Set getMemberNames() { + return memberNames; + } + + public void setMemberNames(Set memberNames) { + this.memberNames = memberNames; + } + @Override public int hashCode() { return Objects.hash(name); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/Groupable.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/Groupable.java new file mode 100644 index 0000000..d4b43f9 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/Groupable.java @@ -0,0 +1,15 @@ +package gov.nasa.ziggy.pipeline.definition; + +/** + * A database entity that can be assigned to a {@link Group}. + * + * @author PT + */ +public interface Groupable { + + /** + * Name of the object in the class that implements {@link Groupable}. Not to be confused with + * the name of the group itself. + */ + String getName(); +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/HasGroup.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/HasGroup.java deleted file mode 100644 index f5209a6..0000000 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/HasGroup.java +++ /dev/null @@ -1,17 +0,0 @@ -package gov.nasa.ziggy.pipeline.definition; - -/** - * A database entity that has a {@link Group}. - * - * @author PT - */ -public interface HasGroup { - - Group group(); - - default String groupName() { - return group() != null ? group().getName() : "default"; - } - - void setGroup(Group group); -} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelMetadata.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelMetadata.java index 52b7100..677cb92 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelMetadata.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelMetadata.java @@ -1,5 +1,6 @@ package gov.nasa.ziggy.pipeline.definition; +import java.nio.file.Path; import java.util.Date; import java.util.GregorianCalendar; import java.util.regex.Matcher; @@ -8,7 +9,9 @@ import javax.xml.datatype.DatatypeFactory; import javax.xml.datatype.XMLGregorianCalendar; +import gov.nasa.ziggy.models.ModelImporter; import gov.nasa.ziggy.models.SemanticVersionNumber; +import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.Iso8601Formatter; @@ -280,6 +283,14 @@ Date currentDate() { return new Date(); } + public Path datastoreModelPath() { + return DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve(ModelImporter.DATASTORE_MODELS_SUBDIR_NAME) + .resolve(getModelType().getType()) + .resolve(getDatastoreFileName()); + } + private static class DateAdapter extends XmlAdapter { @Override diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelRegistry.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelRegistry.java index 9cdc1cb..e93ec51 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelRegistry.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ModelRegistry.java @@ -53,7 +53,7 @@ public class ModelRegistry implements HasXmlSchemaFilename { @XmlElement(name = "modelMetadata") @Transient - private List modelMetadata = new ArrayList<>(); + private List modelsMetadata = new ArrayList<>(); private boolean locked = false; private Date lockTime; @@ -93,8 +93,8 @@ public void updateModelMetadata(ModelMetadata modelMetadata) { } public void populateXmlFields() { - modelMetadata.clear(); - modelMetadata.addAll(models.values()); + modelsMetadata.clear(); + modelsMetadata.addAll(models.values()); } public void lock() { @@ -145,12 +145,12 @@ public ModelType getModelType(String modelType) { return modelTypesMap.get(modelType); } - public List getModelMetadata() { - return modelMetadata; + public List getModelsMetadata() { + return modelsMetadata; } - public void setModelMetadata(List modelMetadata) { - this.modelMetadata = modelMetadata; + public void setModelsMetadata(List modelsMetadata) { + this.modelsMetadata = modelsMetadata; } @Override diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java index db71416..de147f5 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ParameterSet.java @@ -7,20 +7,17 @@ import java.util.TreeSet; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.parameters.InternalParameters; import gov.nasa.ziggy.parameters.Parameters; import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import jakarta.persistence.ElementCollection; -import jakarta.persistence.Embedded; import jakarta.persistence.Entity; import jakarta.persistence.FetchType; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; import jakarta.persistence.JoinTable; -import jakarta.persistence.ManyToOne; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; import jakarta.persistence.Transient; @@ -41,20 +38,13 @@ @Table(name = "ziggy_ParameterSet", uniqueConstraints = { @UniqueConstraint(columnNames = { "name", "version" }) }) public class ParameterSet extends UniqueNameVersionPipelineComponent - implements HasGroup { + implements Groupable { @Id @GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "ziggy_ParameterSet_generator") @SequenceGenerator(name = "ziggy_ParameterSet_generator", initialValue = 1, sequenceName = "ziggy_ParameterSet_sequence", allocationSize = 1) private Long id; - @Embedded - // init with empty placeholder, to be filled in by console - private AuditInfo auditInfo = new AuditInfo(); - - @ManyToOne - private Group group = null; - private String description = null; @ElementCollection(fetch = FetchType.EAGER) @@ -77,11 +67,6 @@ public ParameterSet(String name) { setName(name); } - public ParameterSet(AuditInfo auditInfo, String name) { - this.auditInfo = auditInfo; - setName(name); - } - // Populates the XML fields (classname and xmlParameters) from the database fields public void populateXmlFields() { for (TypedParameter typedProperty : typedParameters) { @@ -147,14 +132,6 @@ public Class clazz() { } } - /** - * Determines whether the parameter set contains an instance of {@link Parameters}, or one of - * {@link InternalParameters}. - */ - public boolean visibleParameterSet() { - return !(parametersInstance() instanceof InternalParameters); - } - @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) public boolean parametersClassDeleted() { boolean deleted = false; @@ -207,24 +184,6 @@ public void setTypedParameters(Set typedParameters) { populateXmlFields(); } - public AuditInfo getAuditInfo() { - return auditInfo; - } - - public void setAuditInfo(AuditInfo auditInfo) { - this.auditInfo = auditInfo; - } - - @Override - public Group group() { - return group; - } - - @Override - public void setGroup(Group group) { - this.group = group; - } - public Long getId() { return id; } @@ -256,10 +215,9 @@ public boolean totalEquals(Object obj) { return false; } ParameterSet other = (ParameterSet) obj; - return Objects.equals(auditInfo, other.auditInfo) - && Objects.equals(classname, other.classname) - && Objects.equals(description, other.description) && Objects.equals(group, other.group) - && Objects.equals(id, other.id) && new TypedParameterCollection(typedParameters) + return Objects.equals(classname, other.classname) + && Objects.equals(description, other.description) && Objects.equals(id, other.id) + && new TypedParameterCollection(typedParameters) .totalEquals(new TypedParameterCollection(other.typedParameters)); } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java index 1837fa1..f15f0b8 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinition.java @@ -10,6 +10,7 @@ import java.util.Stack; import java.util.stream.Collectors; +import org.apache.commons.collections.CollectionUtils; import org.hibernate.annotations.Cascade; import org.hibernate.annotations.CascadeType; import org.slf4j.Logger; @@ -21,7 +22,6 @@ import gov.nasa.ziggy.pipeline.xml.XmlReference.ParameterSetReference; import gov.nasa.ziggy.util.CollectionFilters; import jakarta.persistence.ElementCollection; -import jakarta.persistence.Embedded; import jakarta.persistence.Entity; import jakarta.persistence.EnumType; import jakarta.persistence.Enumerated; @@ -31,7 +31,6 @@ import jakarta.persistence.JoinColumn; import jakarta.persistence.JoinTable; import jakarta.persistence.ManyToMany; -import jakarta.persistence.ManyToOne; import jakarta.persistence.OrderColumn; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; @@ -55,7 +54,7 @@ @Table(name = "ziggy_PipelineDefinition", uniqueConstraints = { @UniqueConstraint(columnNames = { "name", "version" }) }) public class PipelineDefinition extends UniqueNameVersionPipelineComponent - implements HasGroup { + implements Groupable { @SuppressWarnings("unused") private static final Logger log = LoggerFactory.getLogger(PipelineDefinition.class); @@ -66,16 +65,9 @@ public class PipelineDefinition extends UniqueNameVersionPipelineComponent(); for (Entry, String> pipelineParameterSetName : pipelineParameterSetNames @@ -253,14 +233,6 @@ public void populateXmlFields() { } } - public AuditInfo getAuditInfo() { - return auditInfo; - } - - public void setAuditInfo(AuditInfo auditInfo) { - this.auditInfo = auditInfo; - } - public String getDescription() { return description; } @@ -269,28 +241,10 @@ public void setDescription(String description) { this.description = description; } - @Override - public Group group() { - return group; - } - - @Override - public void setGroup(Group group) { - this.group = group; - } - public Long getId() { return id; } - public String getGroupName() { - Group group = group(); - if (group == null) { - return Group.DEFAULT.getName(); - } - return group.getName(); - } - public Priority getInstancePriority() { return instancePriority; } @@ -379,10 +333,89 @@ public void setRootNodeNames(String rootNodeNames) { this.rootNodeNames = rootNodeNames; } - // TODO: Define what totalEquals() should do in the case of a pipeline definition. + /** + * Defines {@link UniqueNameVersionPipelineComponent#totalEquals(Object)} for + * {@link PipelineDefinition} instances. What this means in practice is the following: + *

        + *
      1. the {@link AuditInfo} fields for the two instances are identical. + *
      2. The {@link Priority} fields for the two instances are identical. + *
      3. The {@link ParameterSet} names for the two instances are identical (note that the + * instances themselves do not have to be identical, because we separately track changes to + * parameter sets). + *
      4. The {@link PipelineDefinitionNode} graphs are identical for the two instances. Here we + * mean that the node IDs are identical. The nodes are not name-version controlled because some + * of the information in them is tracked separately (i.e., module parameter set name-and-version + * are tracked for each pipeline task, so a change in the module parameters doesn't need to be + * tracked in the node definition), and the rest is not relevant to data accountability (i.e., + * nobody cares that the user changed from remote to local execution). + *
      + */ @Override public boolean totalEquals(Object other) { - return false; + PipelineDefinition otherDef = (PipelineDefinition) other; + + if (!java.util.Objects.equals(description, otherDef.description) + || !java.util.Objects.equals(instancePriority, otherDef.instancePriority)) { + return false; + } + Map, String> otherParameters = otherDef + .getPipelineParameterSetNames(); + if (pipelineParameterSetNames.size() != otherParameters.size() + || !pipelineParameterSetNames.values().containsAll(otherParameters.values()) + || !nodeTreesIdentical(rootNodes, otherDef.rootNodes)) { + return false; + } + return true; + } + + /** + * Recursive comparison of the node trees of two {@link PipelineDefinition}s. + */ + private boolean nodeTreesIdentical(List nodes, + List otherNodes) { + + // Check for empty-or-null condition so we know in subsequent steps that + // neither list is null. + if (CollectionUtils.isEmpty(nodes) && CollectionUtils.isEmpty(otherNodes)) { + return true; + } + + // Different sizes means false. + if (nodes.size() != otherNodes.size()) { + return false; + } + + // Different nodes in the two lists (by ID number) means false. + Map nodeMap = pipelineDefinitionNodeIds(nodes); + Map otherNodeMap = pipelineDefinitionNodeIds(otherNodes); + if (!nodeMap.keySet().containsAll(otherNodeMap.keySet())) { + return false; + } + + // At this point we know that the two instances contain the same nodes based on ID numbers. + // Now we loop over this list's nodes, retrieve the matching node from the other list, + // and compare their child nodes. This is where it gets recursive. + boolean nextNodesIdentical = true; + for (long id : nodeMap.keySet()) { + List nextNodes = nodeMap.get(id).getNextNodes(); + List otherNextNodes = otherNodeMap.get(id).getNextNodes(); + if (CollectionUtils.isEmpty(nextNodes) && !CollectionUtils.isEmpty(otherNextNodes) + || !CollectionUtils.isEmpty(nextNodes) && CollectionUtils.isEmpty(otherNextNodes)) { + return false; + } + nextNodesIdentical = nextNodesIdentical + && nodeTreesIdentical(nextNodes, otherNextNodes); + } + return nextNodesIdentical; + } + + Map pipelineDefinitionNodeIds( + List nodes) { + Map nodeMap = new HashMap<>(); + for (PipelineDefinitionNode node : nodes) { + nodeMap.put(node.getId(), node); + } + return nodeMap; } @Override diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionCli.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionCli.java index 2aa3b8e..e8b1435 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionCli.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionCli.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNode.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNode.java index 36099f1..398e502 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNode.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNode.java @@ -13,7 +13,7 @@ import org.hibernate.annotations.Cascade; import org.hibernate.annotations.CascadeType; -import gov.nasa.ziggy.data.management.DataFileType; +import gov.nasa.ziggy.data.datastore.DataFileType; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.parameters.Parameters; import gov.nasa.ziggy.parameters.ParametersInterface; @@ -22,14 +22,9 @@ import gov.nasa.ziggy.pipeline.xml.XmlReference.ModelTypeReference; import gov.nasa.ziggy.pipeline.xml.XmlReference.OutputTypeReference; import gov.nasa.ziggy.pipeline.xml.XmlReference.ParameterSetReference; -import gov.nasa.ziggy.services.messages.WorkerResources; -import gov.nasa.ziggy.uow.UnitOfWorkGenerator; import gov.nasa.ziggy.util.CollectionFilters; -import jakarta.persistence.AttributeOverride; -import jakarta.persistence.AttributeOverrides; import jakarta.persistence.Column; import jakarta.persistence.ElementCollection; -import jakarta.persistence.Embedded; import jakarta.persistence.Entity; import jakarta.persistence.FetchType; import jakarta.persistence.GeneratedValue; @@ -48,7 +43,6 @@ import jakarta.xml.bind.annotation.XmlAttribute; import jakarta.xml.bind.annotation.XmlElement; import jakarta.xml.bind.annotation.XmlElements; -import jakarta.xml.bind.annotation.adapters.XmlJavaTypeAdapter; /** * This class models a single node in a pipeline definition. Each node maps to a @@ -72,37 +66,31 @@ public class PipelineDefinitionNode { /** * Indicates the maximum number of worker processes that should be spun up for this node. If - * zero, the pipeline will default to the number of workers specified for the pipeline as a - * whole, either in the properties file or the command line arguments used when the cluster was - * started. + * left out of the node definition, the default maximum worker process count will be used. + *

      + * This field is used for XML import only. The value is then stored in an instance of + * {@link PipelineDefinitionNodeExecutionResources). *

      * Optional XML attributes cannot be primitives. */ + @Transient @XmlAttribute(required = false, name = "maxWorkers") - private Integer maxWorkerCount = 0; + @Column(name = "maxWorkerCount", nullable = true) + private Integer maxWorkerCount; /** - * Indicates the maximum total Java heap size, in MB, for worker processes spun up for this - * node. If zero, the pipeline will default to the heap size specified for the pipeline as a - * whole, either in the properties file or the command line arguments used when the cluster was - * started. + * Indicates the maximum Java heap that should be allocated up for this node. If left out of the + * node definition, the default maximum worker process count will be used. + *

      + * This field is used for XML import only. The value is then stored in an instance of + * {@link PipelineDefinitionNodeExecutionResources). *

      * Optional XML attributes cannot be primitives. */ + @Transient @XmlAttribute(required = false, name = "heapSizeMb") - private Integer heapSizeMb = 0; - - /** - * If non-null, this UOW generator definition is used to generate the tasks for this node. May - * only be null if the module for this node does not have a defined unit of work generator, in - * which case the generator will be determined at runtime. - */ - @XmlAttribute(required = false, name = "uowGenerator") - @XmlJavaTypeAdapter(ClassWrapper.ClassWrapperAdapter.class) - @Embedded - @AttributeOverrides({ - @AttributeOverride(name = "clazz", column = @Column(name = "unitOfWorkGenerator")) }) - private ClassWrapper unitOfWorkGenerator; + @Column(name = "heapSizeMb", nullable = true) + private Integer heapSizeMb; @XmlAttribute(required = true, name = "moduleName") private String moduleName; @@ -152,6 +140,9 @@ public class PipelineDefinitionNode { // Name of the PipelineDefinition instance for this object. private String pipelineName; + @XmlAttribute(name = "singleSubtask", required = false) + private Boolean singleSubtask = false; + /* * Not stored in the database, but can be set for all nodes in a pipeline by calling * PipelineDefinition.buildPaths() @@ -175,10 +166,6 @@ public PipelineDefinitionNode(PipelineDefinitionNode other) { maxWorkerCount = other.maxWorkerCount; heapSizeMb = other.heapSizeMb; - if (other.unitOfWorkGenerator != null) { - unitOfWorkGenerator = new ClassWrapper<>(other.unitOfWorkGenerator); - } - moduleName = other.moduleName; for (PipelineDefinitionNode otherNode : other.nextNodes) { @@ -191,7 +178,7 @@ public PipelineDefinitionNode(PipelineDefinitionNode other) { moduleParameterSetName.getValue()); } - inputDataFileTypes.addAll(other.inputDataFileTypes); + inputDataFileTypes = new HashSet<>(other.inputDataFileTypes); outputDataFileTypes.addAll(other.outputDataFileTypes); modelTypes.addAll(other.modelTypes); pipelineName = other.pipelineName; @@ -227,6 +214,10 @@ public void populateXmlFields() { .map(ParameterSetReference::new) .collect(Collectors.toSet())); + // Use the setters to fill in the optional XML values. + setMaxWorkerCount(getMaxWorkerCount()); + setHeapSizeMb(getHeapSizeMb()); + // We don't want to touch the childNodeNames String unless the nextNodes List is populated if (!nextNodes.isEmpty()) { setChildNodeNames(); @@ -247,30 +238,6 @@ private void setChildNodeNames() { childNodeNames = sb.toString(); } - /** - * Returns the worker resources for the current node. If resources are not specified, the - * resources object will have nulls, in which case default values will be retrieved from the - * {@link WorkerResources} singleton when the object is queried. - */ - public WorkerResources workerResources() { - Integer workerCount = maxWorkerCount == null || maxWorkerCount <= 0 ? null : maxWorkerCount; - Integer heapSize = heapSizeMb == null || heapSizeMb <= 0 ? null : heapSizeMb; - return new WorkerResources(workerCount, heapSize); - } - - /** - * Applies the worker resources values to a pipeline instance node. If the values are the - * default ones, the node's values will be set to zero rather than the values returned by the - * resources object. - */ - public void applyWorkerResources(WorkerResources resources) { - maxWorkerCount = resources.maxWorkerCountIsDefault() ? 0 : resources.getMaxWorkerCount(); - heapSizeMb = resources.heapSizeIsDefault() ? 0 : resources.getHeapSizeMb(); - } - - /** - * @return the id - */ public Long getId() { return id; } @@ -304,34 +271,18 @@ public Integer getMaxWorkerCount() { return maxWorkerCount; } - public void setMaxWorkerCount(int workers) { + public void setMaxWorkerCount(Integer workers) { maxWorkerCount = workers; } - public boolean useDefaultWorkerCount() { - return maxWorkerCount == null || maxWorkerCount.intValue() == 0; - } - public Integer getHeapSizeMb() { return heapSizeMb; } - public void setHeapSizeMb(int heapSizeMb) { + public void setHeapSizeMb(Integer heapSizeMb) { this.heapSizeMb = heapSizeMb; } - public boolean useDefaultHeapSize() { - return heapSizeMb == null || heapSizeMb.intValue() == 0; - } - - public ClassWrapper getUnitOfWorkGenerator() { - return unitOfWorkGenerator; - } - - public void setUnitOfWorkGenerator(ClassWrapper unitOfWorkGenerator) { - this.unitOfWorkGenerator = unitOfWorkGenerator; - } - public String getModuleName() { return moduleName; } @@ -340,6 +291,14 @@ public void setModuleName(String moduleName) { this.moduleName = moduleName; } + public Boolean getSingleSubtask() { + return singleSubtask != null ? singleSubtask : false; + } + + public void setSingleSubtask(Boolean singleSubtask) { + this.singleSubtask = singleSubtask; + } + public void setPipelineModuleDefinition(PipelineModuleDefinition moduleDefinition) { moduleName = moduleDefinition.getName(); } @@ -405,17 +364,11 @@ public void addInputDataFileType(DataFileType dataFileType) { populateXmlFields(); } - public void addAllInputDataFileTypes(Set dataFileTypes) { + public void addAllInputDataFileTypes(Collection dataFileTypes) { inputDataFileTypes.addAll(dataFileTypes); populateXmlFields(); } - public void setInputDataFileTypes(Set dataFileTypes) { - inputDataFileTypes = dataFileTypes; - CollectionFilters.removeTypeFromCollection(xmlReferences, InputTypeReference.class); - populateXmlFields(); - } - public Set getInputDataFileTypes() { return inputDataFileTypes; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeExecutionResources.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeExecutionResources.java new file mode 100644 index 0000000..f6af93a --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeExecutionResources.java @@ -0,0 +1,358 @@ +package gov.nasa.ziggy.pipeline.definition; + +import java.util.Objects; + +import org.apache.commons.lang3.StringUtils; + +import gov.nasa.ziggy.module.remote.PbsParameters; +import gov.nasa.ziggy.module.remote.RemoteArchitectureOptimizer; +import gov.nasa.ziggy.module.remote.RemoteNodeDescriptor; +import gov.nasa.ziggy.worker.WorkerResources; +import jakarta.persistence.Entity; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.GeneratedValue; +import jakarta.persistence.GenerationType; +import jakarta.persistence.Id; +import jakarta.persistence.SequenceGenerator; +import jakarta.persistence.Table; +import jakarta.persistence.UniqueConstraint; + +/** + * Parameters relevant for configuring execution of a {@link PipelineDefinitionNode}. The + * configuration is related to a specific {@link PipelineDefinitionNode} via fields that contain the + * node's module name and pipeline name. This ensures that a given + * {@link PipelineDefinitionNodeExecutionResources} is associated with any and all versions of its + * definition node and that none of these parameters are involved in determining whether a node + * definition is up to date (which would be the case if the class was embedded). + * + * @author PT + */ + +@Entity +@Table(name = "ziggy_PipelineDefinitionNode_executionResources", uniqueConstraints = { + @UniqueConstraint(columnNames = { "pipelineName", "pipelineModuleName" }) }) +public class PipelineDefinitionNodeExecutionResources { + + @Id + @GeneratedValue(strategy = GenerationType.SEQUENCE, + generator = "ziggy_PipelineDefinitionNode_executionResources_generator") + @SequenceGenerator(name = "ziggy_PipelineDefinitionNode_executionResources_generator", + initialValue = 1, sequenceName = "ziggy_PipelineDefinitionNode_executionResources_sequence", + allocationSize = 1) + private Long id; + + // Fields that provide mapping to a specific pipeline definition. + private final String pipelineName; + private final String pipelineModuleName; + + // Fields that control worker-side execution resource options. + private int maxWorkerCount = 0; + private int heapSizeMb = 0; + private int maxFailedSubtaskCount = 0; + private int maxAutoResubmits = 0; + + // Fields that control remote execution and are mandatory. + private boolean remoteExecutionEnabled = false; + private double subtaskMaxWallTimeHours = 0; + private double subtaskTypicalWallTimeHours = 0; + private double gigsPerSubtask = 0; + @Enumerated(EnumType.STRING) + private RemoteArchitectureOptimizer optimizer = RemoteArchitectureOptimizer.COST; + private boolean nodeSharing = true; + private boolean wallTimeScaling = true; + + // Optional parameters for remote execution. The user can specify these, or can + // allow the remote execution calculator decide the values. For the Strings, an + // empty string indicates that the user has not set a value. For the int and double + // primitives, this is indicated by a zero; the exception is minSubtasksForRemoteExecution, + // in which -1 indicates that the user has not entered a value. + private String remoteNodeArchitecture = ""; + private String queueName = ""; + private int maxNodes; + private double subtasksPerCore; + private int minCoresPerNode; + private double minGigsPerNode; + private int minSubtasksForRemoteExecution = -1; + + // "The JPA specification requires that all persistent classes have a no-arg constructor. This + // constructor may be public or protected." + protected PipelineDefinitionNodeExecutionResources() { + this("", ""); + } + + public PipelineDefinitionNodeExecutionResources(String pipelineName, + String pipelineModuleName) { + this.pipelineName = pipelineName; + this.pipelineModuleName = pipelineModuleName; + } + + /** Copy constructor. */ + public PipelineDefinitionNodeExecutionResources( + PipelineDefinitionNodeExecutionResources original) { + this(original.pipelineName, original.pipelineModuleName); + populateFrom(original); + } + + /** + * Populates one instance with the values of another. This is useful for quickly putting values + * from a copied instance back into the original. This allows the original to be merged back + * into the database. + */ + public void populateFrom(PipelineDefinitionNodeExecutionResources other) { + heapSizeMb = other.heapSizeMb; + maxWorkerCount = other.maxWorkerCount; + maxFailedSubtaskCount = other.maxFailedSubtaskCount; + maxAutoResubmits = other.maxAutoResubmits; + remoteExecutionEnabled = other.remoteExecutionEnabled; + subtaskMaxWallTimeHours = other.subtaskMaxWallTimeHours; + subtaskTypicalWallTimeHours = other.subtaskTypicalWallTimeHours; + gigsPerSubtask = other.gigsPerSubtask; + minSubtasksForRemoteExecution = other.minSubtasksForRemoteExecution; + optimizer = other.optimizer; + nodeSharing = other.nodeSharing; + wallTimeScaling = other.wallTimeScaling; + + remoteNodeArchitecture = other.remoteNodeArchitecture; + queueName = other.queueName; + maxNodes = other.maxNodes; + subtasksPerCore = other.subtasksPerCore; + minCoresPerNode = other.minCoresPerNode; + minGigsPerNode = other.minGigsPerNode; + } + + public PbsParameters pbsParametersInstance() { + PbsParameters pbsParameters = new PbsParameters(); + pbsParameters.setEnabled(remoteExecutionEnabled); + pbsParameters.setArchitecture(RemoteNodeDescriptor.fromName(remoteNodeArchitecture)); + pbsParameters.setGigsPerSubtask(gigsPerSubtask); + if (!StringUtils.isEmpty(queueName)) { + pbsParameters.setQueueName(queueName); + } + if (minCoresPerNode > 0) { + pbsParameters.setMinCoresPerNode(minCoresPerNode); + } + if (minGigsPerNode > 0) { + pbsParameters.setMinGigsPerNode(minGigsPerNode); + } + if (maxNodes > 0) { + pbsParameters.setRequestedNodeCount(maxNodes); + } + return pbsParameters; + } + + /** + * Returns the worker resources for the current node. If resources are not specified, the + * resources object will have nulls, in which case default values will be retrieved from the + * {@link WorkerResources} singleton when the object is queried. + */ + public WorkerResources workerResources() { + Integer workerCount = maxWorkerCount <= 0 ? null : maxWorkerCount; + Integer heapSize = heapSizeMb <= 0 ? null : heapSizeMb; + return new WorkerResources(workerCount, heapSize); + } + + /** + * Applies the worker resources values to a pipeline instance node. If the values are the + * default ones, the node's values will be set to zero rather than the values returned by the + * resources object. + */ + public void applyWorkerResources(WorkerResources resources) { + maxWorkerCount = resources.getMaxWorkerCount() == null ? 0 : resources.getMaxWorkerCount(); + heapSizeMb = resources.getHeapSizeMb() == null ? 0 : resources.getHeapSizeMb(); + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getPipelineName() { + return pipelineName; + } + + public String getPipelineModuleName() { + return pipelineModuleName; + } + + public int getMaxWorkerCount() { + return maxWorkerCount; + } + + public void setMaxWorkerCount(int maxWorkerCount) { + this.maxWorkerCount = maxWorkerCount; + } + + public int getHeapSizeMb() { + return heapSizeMb; + } + + public void setHeapSizeMb(int heapSizeMb) { + this.heapSizeMb = heapSizeMb; + } + + public int getMaxFailedSubtaskCount() { + return maxFailedSubtaskCount; + } + + public void setMaxFailedSubtaskCount(int maxFailedSubtaskCount) { + this.maxFailedSubtaskCount = maxFailedSubtaskCount; + } + + public int getMaxAutoResubmits() { + return maxAutoResubmits; + } + + public void setMaxAutoResubmits(int maxAutoResubmits) { + this.maxAutoResubmits = maxAutoResubmits; + } + + public boolean isRemoteExecutionEnabled() { + return remoteExecutionEnabled; + } + + public void setRemoteExecutionEnabled(boolean remoteExecutionEnabled) { + this.remoteExecutionEnabled = remoteExecutionEnabled; + } + + public double getSubtaskMaxWallTimeHours() { + return subtaskMaxWallTimeHours; + } + + public void setSubtaskMaxWallTimeHours(double subtaskMaxWallTimeHours) { + this.subtaskMaxWallTimeHours = subtaskMaxWallTimeHours; + } + + public double getSubtaskTypicalWallTimeHours() { + return subtaskTypicalWallTimeHours; + } + + public void setSubtaskTypicalWallTimeHours(double subtaskTypicalWallTimeHours) { + this.subtaskTypicalWallTimeHours = subtaskTypicalWallTimeHours; + } + + public double getGigsPerSubtask() { + return gigsPerSubtask; + } + + public void setGigsPerSubtask(double gigsPerSubtask) { + this.gigsPerSubtask = gigsPerSubtask; + } + + public int getMinSubtasksForRemoteExecution() { + return minSubtasksForRemoteExecution; + } + + public void setMinSubtasksForRemoteExecution(int minSubtasksForRemoteExecution) { + this.minSubtasksForRemoteExecution = minSubtasksForRemoteExecution; + } + + public RemoteArchitectureOptimizer getOptimizer() { + return optimizer; + } + + public void setOptimizer(RemoteArchitectureOptimizer optimizer) { + this.optimizer = optimizer; + } + + public boolean isNodeSharing() { + return nodeSharing; + } + + public void setNodeSharing(boolean nodeSharing) { + this.nodeSharing = nodeSharing; + } + + public boolean isWallTimeScaling() { + return wallTimeScaling; + } + + public void setWallTimeScaling(boolean wallTimeScaling) { + this.wallTimeScaling = wallTimeScaling; + } + + public String getRemoteNodeArchitecture() { + return remoteNodeArchitecture; + } + + public void setRemoteNodeArchitecture(String remoteNodeArchitecture) { + this.remoteNodeArchitecture = remoteNodeArchitecture; + } + + public String getQueueName() { + return queueName; + } + + public void setQueueName(String queueName) { + this.queueName = queueName; + } + + public int getMaxNodes() { + return maxNodes; + } + + public void setMaxNodes(int maxNodes) { + this.maxNodes = maxNodes; + } + + public double getSubtasksPerCore() { + return subtasksPerCore; + } + + public void setSubtasksPerCore(double subtasksPerCore) { + this.subtasksPerCore = subtasksPerCore; + } + + public int getMinCoresPerNode() { + return minCoresPerNode; + } + + public void setMinCoresPerNode(int minCoresPerNode) { + this.minCoresPerNode = minCoresPerNode; + } + + public double getMinGigsPerNode() { + return minGigsPerNode; + } + + public void setMinGigsPerNode(double minGigsPerNode) { + this.minGigsPerNode = minGigsPerNode; + } + + @Override + public int hashCode() { + return Objects.hash(gigsPerSubtask, maxNodes, minCoresPerNode, minGigsPerNode, + minSubtasksForRemoteExecution, nodeSharing, optimizer, queueName, + remoteExecutionEnabled, remoteNodeArchitecture, subtaskMaxWallTimeHours, + subtaskTypicalWallTimeHours, subtasksPerCore, wallTimeScaling); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null || getClass() != obj.getClass()) { + return false; + } + PipelineDefinitionNodeExecutionResources other = (PipelineDefinitionNodeExecutionResources) obj; + return Double.doubleToLongBits(gigsPerSubtask) == Double + .doubleToLongBits(other.gigsPerSubtask) && Objects.equals(maxNodes, other.maxNodes) + && Objects.equals(minCoresPerNode, other.minCoresPerNode) + && Objects.equals(minGigsPerNode, other.minGigsPerNode) + && minSubtasksForRemoteExecution == other.minSubtasksForRemoteExecution + && nodeSharing == other.nodeSharing && optimizer == other.optimizer + && Objects.equals(queueName, other.queueName) + && remoteExecutionEnabled == other.remoteExecutionEnabled + && Objects.equals(remoteNodeArchitecture, other.remoteNodeArchitecture) + && Double.doubleToLongBits(subtaskMaxWallTimeHours) == Double + .doubleToLongBits(other.subtaskMaxWallTimeHours) + && Double.doubleToLongBits(subtaskTypicalWallTimeHours) == Double + .doubleToLongBits(other.subtaskTypicalWallTimeHours) + && Objects.equals(subtasksPerCore, other.subtasksPerCore) + && wallTimeScaling == other.wallTimeScaling; + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionOperations.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionOperations.java index 862f3f0..7756033 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionOperations.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionOperations.java @@ -15,7 +15,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.data.management.DataFileType; +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.module.ExternalProcessPipelineModule; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.parameters.ParametersOperations; @@ -24,9 +25,11 @@ import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; import gov.nasa.ziggy.pipeline.xml.ValidatingXmlManager; import gov.nasa.ziggy.pipeline.xml.XmlReference; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; /** * Contains methods for importing and exporting pipeline configurations. @@ -51,6 +54,7 @@ public class PipelineDefinitionOperations { private PipelineDefinitionCrud pipelineCrud; private ParameterSetCrud parameterSetCrud; private PipelineOperations pipelineOperations; + private PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud; private DataFileTypeCrud dataFileTypeCrud; private ModelCrud modelCrud; private ValidatingXmlManager xmlManager; @@ -148,6 +152,29 @@ public void importPipelineConfiguration(Collection files) { private void importModules(List newModules) { PipelineModuleDefinitionCrud moduleCrud = pipelineModuleDefinitionCrud(); for (PipelineModuleDefinition newModule : newModules) { + PipelineModuleExecutionResources executionResources = moduleCrud + .retrieveExecutionResources(newModule); + executionResources.setExeTimeoutSeconds(newModule.getExeTimeoutSecs()); + executionResources.setMinMemoryMegabytes(newModule.getMinMemoryMegabytes()); + + // Additional validation: + // ExternalProcessPipelineModule must not have a UOW generator in its XML. + // All other pipeline module classes must have a UOW generator in their XMLs. + if (newModule.getPipelineModuleClass() + .getClazz() + .equals(ExternalProcessPipelineModule.class)) { + if (newModule.getUnitOfWorkGenerator() != null) { + throw new PipelineException("Module " + newModule.getName() + + " uses ExternalProcessPipelineModule, specified UOW " + + newModule.getUnitOfWorkGenerator().getClazz().toString() + " is invalid"); + } + newModule.setUnitOfWorkGenerator( + new ClassWrapper<>(DatastoreDirectoryUnitOfWorkGenerator.class)); + } else if (newModule.getUnitOfWorkGenerator() == null) { + throw new PipelineException( + "Module " + newModule.getName() + " must specify a unit of work generator"); + } + moduleCrud.merge(executionResources); moduleCrud.merge(newModule); } } @@ -158,7 +185,6 @@ private void importPipelines(List newPipelineDefinitions) { for (PipelineDefinition newPipelineDefinition : newPipelineDefinitions) { String pipelineName = newPipelineDefinition.getName(); - newPipelineDefinition.setAuditInfo(new AuditInfo()); Set nodes = newPipelineDefinition.getNodesFromXml(); Map nodesByName = new HashMap<>(); @@ -191,7 +217,6 @@ private void importPipelines(List newPipelineDefinitions) { newPipelineDefinition.getRootNodeNames()); addNodes(newPipelineDefinition.getName(), rootNodeNames, newPipelineDefinition.getRootNodes(), nodesByName); - pipelineCrud.merge(newPipelineDefinition); } } @@ -274,6 +299,18 @@ private void addNodes(String pipelineName, List rootNodeNames, xmlNode.addAllModelTypes(modelTypes); pipelineRootNodes.add(xmlNode); + // Execution Resources: Store in an instance of + // PipelineDefinitionNodeExecutionResources. + PipelineDefinitionNodeExecutionResources executionResources = pipelineDefinitionNodeCrud() + .retrieveExecutionResources(xmlNode); + if (xmlNode.getHeapSizeMb() != null) { + executionResources.setHeapSizeMb(xmlNode.getHeapSizeMb()); + } + if (xmlNode.getMaxWorkerCount() != null) { + executionResources.setMaxWorkerCount(xmlNode.getMaxWorkerCount()); + } + pipelineDefinitionNodeCrud.merge(executionResources); + // Child nodes String childNodeIds = xmlNode.getChildNodeNames(); @@ -351,4 +388,11 @@ ModelCrud modelCrud() { } return modelCrud; } + + PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud() { + if (pipelineDefinitionNodeCrud == null) { + pipelineDefinitionNodeCrud = new PipelineDefinitionNodeCrud(); + } + return pipelineDefinitionNodeCrud; + } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionProcessingOptions.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionProcessingOptions.java new file mode 100644 index 0000000..fd4cbdb --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionProcessingOptions.java @@ -0,0 +1,66 @@ +package gov.nasa.ziggy.pipeline.definition; + +import jakarta.persistence.Column; +import jakarta.persistence.Entity; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.Id; +import jakarta.persistence.Table; + +/** + * Stores processing options for a given pipeline. + *

      + * The ProcessingMode enumeration specifies whether to process all data, including data that has + * already been processed, or to process only new data that has never been processed before. + * + * @author PT + */ +@Entity +@Table(name = "ziggy_PipelineDefinition_processingOptions") +public class PipelineDefinitionProcessingOptions { + + public enum ProcessingMode { + PROCESS_NEW("new"), PROCESS_ALL("all"); + + private String displayString; + + ProcessingMode(String displayString) { + this.displayString = displayString; + } + + @Override + public String toString() { + return displayString; + } + } + + @Id + private String pipelineName; + + @Enumerated(EnumType.STRING) + @Column(name = "processingMode") + private ProcessingMode processingMode = ProcessingMode.PROCESS_ALL; + + public PipelineDefinitionProcessingOptions() { + } + + public PipelineDefinitionProcessingOptions(String pipelineName) { + this.pipelineName = pipelineName; + } + + public String getPipelineName() { + return pipelineName; + } + + public void setPipelineName(String pipelineName) { + this.pipelineName = pipelineName; + } + + public ProcessingMode getProcessingMode() { + return processingMode; + } + + public void setProcessingMode(ProcessingMode processingMode) { + this.processingMode = processingMode; + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java index 6002d3e..eb5c2bf 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstance.java @@ -45,8 +45,6 @@ @Entity @Table(name = "ziggy_PipelineInstance") public class PipelineInstance implements PipelineExecutionTime { - private static final long serialVersionUID = 20230712L; - private static final Logger log = LoggerFactory.getLogger(PipelineInstance.class); public enum State { @@ -115,13 +113,10 @@ public Priority unmarshal(String priority) throws Exception { /** * Descriptive name specified by the user at launch-time. Used when displaying the instance in - * the console. Does not have to be unique + * the console. Does not have to be unique. */ private String name; - @ManyToOne - private Group group = null; - /** Timestamp that processing started on this pipeline instance */ private Date startProcessingTime = new Date(0); @@ -318,66 +313,34 @@ public ParameterSet putParameterSet(ClassWrapper key, Param return pipelineParameterSets.put(key, value); } - /** - * @return the name - */ public String getName() { return name; } - /** - * @param name the name to set - */ public void setName(String name) { this.name = name; } - /** - * @return the startNode - */ public PipelineInstanceNode getStartNode() { return startNode; } - /** - * @param startNode the startNode to set - */ public void setStartNode(PipelineInstanceNode startNode) { this.startNode = startNode; } - /** - * @return the endNode - */ public PipelineInstanceNode getEndNode() { return endNode; } - /** - * @param endNode the endNode to set - */ public void setEndNode(PipelineInstanceNode endNode) { this.endNode = endNode; } - public Group getGroup() { - return group; - } - - public void setGroup(Group group) { - this.group = group; - } - - /** - * @return the modelRegistry - */ public ModelRegistry getModelRegistry() { return modelRegistry; } - /** - * @param modelRegistry the modelRegistry to set - */ public void setModelRegistry(ModelRegistry modelRegistry) { this.modelRegistry = modelRegistry; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceNode.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceNode.java index 9d17e1a..30fcd90 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceNode.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineInstanceNode.java @@ -41,7 +41,7 @@ public class PipelineInstanceNode { private Long id; /** Timestamp this was created (either by launcher or transition logic) */ - private Date created = new Date(System.currentTimeMillis()); + private Date created = new Date(); @ManyToOne private PipelineInstance pipelineInstance; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java index dbe7e02..a338164 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModule.java @@ -22,18 +22,10 @@ * It defines the entry point called by the pipeline infrastructure when a task arrives for this * module (processTask()). *

      - * Important note related to task deletion: - *

      - * Ziggy provides the capability to halt task execution, both prior to the start of execution (when - * a task is submitted but not yet processing) and during execution. However: because Java's model - * of thread interruption is "cooperative," {@link PipelineModule} subclasses need to provide - * support for task deletion if they are expected to allow a task to be deleted once - * {@link #processTask()} has been called. In particular, the module must check for thread - * interruption using {@link Thread#isInterrupted()} on the current thread and returning from - * {@link #processTask()} if an interruption is detected. * * @author Todd Klaus * @author Sean McCauliff + * @author PT */ public abstract class PipelineModule { @@ -64,14 +56,6 @@ public final long taskId() { return pipelineTask.getId(); } - /** - * Indicates whether the {@link processTask} method of a given subclass must be executed within - * a database transaction. Override to set to false if this is not the case. - */ - public boolean processTaskRequiresDatabaseTransaction() { - return true; - } - /** * Modules should subclass this or in some cases generateInputs and processOutputs(). This is * how they perform the work for a pipeline task. diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinition.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinition.java index f409d6f..686f9b4 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinition.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinition.java @@ -1,16 +1,14 @@ package gov.nasa.ziggy.pipeline.definition; -import java.util.HashSet; -import java.util.List; import java.util.Objects; -import java.util.Set; -import gov.nasa.ziggy.module.DefaultPipelineInputs; -import gov.nasa.ziggy.module.DefaultPipelineOutputs; +import gov.nasa.ziggy.module.DatastoreDirectoryPipelineInputs; +import gov.nasa.ziggy.module.DatastoreDirectoryPipelineOutputs; import gov.nasa.ziggy.module.ExternalProcessPipelineModule; import gov.nasa.ziggy.module.PipelineInputs; import gov.nasa.ziggy.module.PipelineOutputs; -import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.uow.UnitOfWorkGenerator; import jakarta.persistence.AttributeOverride; import jakarta.persistence.AttributeOverrides; import jakarta.persistence.Column; @@ -19,9 +17,9 @@ import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; -import jakarta.persistence.ManyToOne; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; +import jakarta.persistence.Transient; import jakarta.persistence.UniqueConstraint; import jakarta.xml.bind.annotation.XmlAccessType; import jakarta.xml.bind.annotation.XmlAccessorType; @@ -31,6 +29,12 @@ /** * This class models a pipeline module, which consists of an algorithm and the parameters that * control the behavior of that algorithm. + *

      + * By default, pipeline module definitions will use{@link ExternalProcessPipelineModule} for their + * execution module, {@link DatastoreDirectoryUnitOfWorkGenerator} to generate units of work, + * {@link DatastoreDirectoryPipelineInputs} and {@link DatastoreDirectoryPipelineOutputs}, + * respectively, for the inputs and outputs class. In the case where the user wishes to accept these + * defaults, there is no need to specify any of them in the module definition. * * @author Todd Klaus */ @@ -48,16 +52,16 @@ public class PipelineModuleDefinition sequenceName = "ziggy_PipelineModuleDefinition_sequence", allocationSize = 1) private Long id; - @ManyToOne - private Group group = null; - - @Embedded - // init with empty placeholder, to be filled in by console - private AuditInfo auditInfo = new AuditInfo(); - @XmlAttribute private String description = "description"; + @XmlAttribute(required = false, name = "uowGenerator") + @XmlJavaTypeAdapter(ClassWrapper.ClassWrapperAdapter.class) + @Embedded + @AttributeOverrides({ + @AttributeOverride(name = "clazz", column = @Column(name = "unitOfWorkGenerator")) }) + private ClassWrapper unitOfWorkGenerator; + @XmlAttribute(required = false) @XmlJavaTypeAdapter(ClassWrapper.ClassWrapperAdapter.class) @Embedded @@ -72,7 +76,7 @@ public class PipelineModuleDefinition @AttributeOverrides({ @AttributeOverride(name = "clazz", column = @Column(name = "inputsClass")) }) private ClassWrapper inputsClass = new ClassWrapper<>( - DefaultPipelineInputs.class); + DatastoreDirectoryPipelineInputs.class); @Embedded @XmlAttribute(required = false) @@ -80,17 +84,21 @@ public class PipelineModuleDefinition @AttributeOverrides({ @AttributeOverride(name = "clazz", column = @Column(name = "outputsClass")) }) private ClassWrapper outputsClass = new ClassWrapper<>( - DefaultPipelineOutputs.class); + DatastoreDirectoryPipelineOutputs.class); // Using the Integer class rather than int here because XML won't allow optional - // attributes that are primitive types + // attributes that are primitive types. Transient so that modules can be imported + // with the value set, but the value can then get put into the database in an + // instance of PipelineModuleExecutionResources. + @Transient @XmlAttribute(required = false) - private Integer exeTimeoutSecs = 60 * 60 * 50; // 50 hours + private Integer exeTimeoutSecs = PipelineModuleExecutionResources.DEFAULT_TIMEOUT_SECONDS; // Using the Integer class rather than int here because XML won't allow optional // attributes that are primitive types + @Transient @XmlAttribute(required = false) - private Integer minMemoryMegaBytes = 0; // zero means memory usage is not constrained + private Integer minMemoryMegabytes = PipelineModuleExecutionResources.DEFAULT_MEMORY_MEGABYTES; // for hibernate use only public PipelineModuleDefinition() { @@ -100,11 +108,6 @@ public PipelineModuleDefinition(String name) { setName(name); } - public PipelineModuleDefinition(AuditInfo auditInfo, String name) { - this.auditInfo = auditInfo; - setName(name); - } - /** * @return Returns the id. */ @@ -112,14 +115,6 @@ public Long getId() { return id; } - public AuditInfo getAuditInfo() { - return auditInfo; - } - - public void setAuditInfo(AuditInfo auditInfo) { - this.auditInfo = auditInfo; - } - public String getDescription() { return description; } @@ -160,47 +155,22 @@ public void setOutputsClass(ClassWrapper outputsClass) { this.outputsClass = outputsClass; } - /** - * @return the minMemoryBytes - */ - public int getMinMemoryMegaBytes() { - return minMemoryMegaBytes; - } - - /** - * @param minMemoryBytes the minMemoryBytes to set - */ - public void setMinMemoryMegaBytes(int minMemoryBytes) { - minMemoryMegaBytes = minMemoryBytes; - } - - public Set> getRequiredParameterClasses() { - PipelineInputs pipelineInputs = inputsClass.newInstance(); - - Set> requiredParameters = new HashSet<>(); - List> moduleParameters = pipelineInputs - .requiredParameters(); - - for (Class clazz : moduleParameters) { - requiredParameters.add(new ClassWrapper<>(clazz)); - } - - return requiredParameters; + public int getMinMemoryMegabytes() { + return minMemoryMegabytes; } - public Group getGroup() { - return group; + public ClassWrapper getUnitOfWorkGenerator() { + return unitOfWorkGenerator; } - public void setGroup(Group group) { - this.group = group; + public void setUnitOfWorkGenerator(ClassWrapper unitOfWorkGenerator) { + this.unitOfWorkGenerator = unitOfWorkGenerator; } @Override public int hashCode() { - return Objects.hash(description, getOptimisticLockValue(), exeTimeoutSecs, group, id, - inputsClass, isLocked(), minMemoryMegaBytes, getName(), outputsClass, - pipelineModuleClass, getVersion()); + return Objects.hash(description, getOptimisticLockValue(), id, inputsClass, isLocked(), + getName(), outputsClass, pipelineModuleClass, getVersion()); } @Override @@ -214,13 +184,9 @@ public boolean equals(Object obj) { PipelineModuleDefinition other = (PipelineModuleDefinition) obj; boolean equalModule = Objects.equals(description, other.description); equalModule = equalModule && getOptimisticLockValue() == other.getOptimisticLockValue(); - equalModule = equalModule && exeTimeoutSecs.intValue() == other.exeTimeoutSecs.intValue(); - equalModule = equalModule && Objects.equals(group, other.group); equalModule = equalModule && Objects.equals(id, other.id); equalModule = equalModule && Objects.equals(inputsClass, other.inputsClass); equalModule = equalModule && isLocked() == other.isLocked(); - equalModule = equalModule - && minMemoryMegaBytes.intValue() == other.minMemoryMegaBytes.intValue(); equalModule = equalModule && Objects.equals(getName(), other.getName()); equalModule = equalModule && Objects.equals(outputsClass, other.outputsClass); equalModule = equalModule && Objects.equals(pipelineModuleClass, other.pipelineModuleClass); diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleExecutionResources.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleExecutionResources.java new file mode 100644 index 0000000..8c852a6 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleExecutionResources.java @@ -0,0 +1,65 @@ +package gov.nasa.ziggy.pipeline.definition; + +import jakarta.persistence.Entity; +import jakarta.persistence.GeneratedValue; +import jakarta.persistence.GenerationType; +import jakarta.persistence.Id; +import jakarta.persistence.SequenceGenerator; +import jakarta.persistence.Table; +import jakarta.persistence.Transient; + +/** + * Execution resources for {@link PipelineModuleDefinition} instances. The execution resources table + * is not linked to the module definition by a foreign key constraint. Rather, the name of the + * module is stored along with its parameters. This ensures that a single instance of + * {@link PipelineModuleExecutionResources} is associated with all versions of the + * {@link PipelineModuleDefinition} in the database, and, conversely, changing the resource + * parameters does not cause the module definition to update to a new version. + * + * @author PT + */ +@Entity +@Table(name = "ziggy_PipelineModuleExecutionResources") +public class PipelineModuleExecutionResources { + + @Transient + public static final int DEFAULT_TIMEOUT_SECONDS = 60 * 60 * 50; // 50 hours. + + @Transient + public static final int DEFAULT_MEMORY_MEGABYTES = 0; // 0 means memory usage not constrained. + + @Id + @GeneratedValue(strategy = GenerationType.SEQUENCE, + generator = "ziggy_PipelineModuleExecutionResources_generator") + @SequenceGenerator(name = "ziggy_PipelineModuleExecutionResources_generator", initialValue = 1, + sequenceName = "ziggy_PipelineModuleExecutionResources_sequence", allocationSize = 1) + private Long id; + + private String pipelineModuleName; + private int exeTimeoutSeconds; + private int minMemoryMegabytes; + + public String getPipelineModuleName() { + return pipelineModuleName; + } + + public void setPipelineModuleName(String pipelineModuleName) { + this.pipelineModuleName = pipelineModuleName; + } + + public int getExeTimeoutSeconds() { + return exeTimeoutSeconds; + } + + public void setExeTimeoutSeconds(int exeTimeoutSeconds) { + this.exeTimeoutSeconds = exeTimeoutSeconds; + } + + public int getMinMemoryMegabytes() { + return minMemoryMegabytes; + } + + public void setMinMemoryMegabytes(int minMemoryMegabytes) { + this.minMemoryMegabytes = minMemoryMegabytes; + } +} diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java index b70d92f..aad4b79 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/PipelineTask.java @@ -25,7 +25,6 @@ import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.PipelineOperations; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; -import gov.nasa.ziggy.uow.TaskConfigurationParameters; import gov.nasa.ziggy.uow.UnitOfWork; import gov.nasa.ziggy.uow.UnitOfWorkGenerator; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -44,6 +43,7 @@ import jakarta.persistence.OrderColumn; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; +import jakarta.persistence.Transient; /** * Represents a single pipeline unit of work Associated with a{@link PipelineInstance}, a @@ -147,7 +147,7 @@ public String toHtmlString() { private boolean retry = false; /** Timestamp this task was created (either by launcher or transition logic) */ - private Date created = new Date(0); // Date(System.currentTimeMillis()); + private Date created = new Date(); /** hostname of the worker that processed (or is processing) this task */ private String workerHost; @@ -207,6 +207,12 @@ public String toHtmlString() { private long currentExecutionStartTimeMillis; + @Transient + private int maxFailedSubtaskCount; + + @Transient + private int maxAutoResubmits; + /** * Required by Hibernate */ @@ -303,18 +309,6 @@ public ParameterSet getParameterSet(Class parametersCl return pipelineParamSet; } - public int maxFailedSubtasks() { - TaskConfigurationParameters configParams = getParameters(TaskConfigurationParameters.class, - false); - return configParams != null ? configParams.getMaxFailedSubtaskCount() : 0; - } - - public int maxAutoResubmits() { - TaskConfigurationParameters configParams = getParameters(TaskConfigurationParameters.class, - false); - return configParams != null ? configParams.getMaxAutoResubmits() : 0; - } - /** * Conveninence method for getting the externalId for a model for this pipeline task. * @@ -391,7 +385,7 @@ public PipelineModule getModuleImplementation(RunMode runMode) { } } - public PipelineDefinitionNode getPipelineDefinitionNode() { + public PipelineDefinitionNode pipelineDefinitionNode() { return pipelineInstanceNode.getPipelineDefinitionNode(); } @@ -415,6 +409,10 @@ public long pipelineInstanceId() { return pipelineInstance.getId(); } + public PipelineDefinition pipelineDefinition() { + return pipelineInstance.getPipelineDefinition(); + } + public Long getId() { return id; } @@ -785,6 +783,22 @@ public long getCurrentExecutionStartTimeMillis() { return currentExecutionStartTimeMillis; } + public int getMaxFailedSubtaskCount() { + return maxFailedSubtaskCount; + } + + public void setMaxFailedSubtaskCount(int maxFailedSubtaskCount) { + this.maxFailedSubtaskCount = maxFailedSubtaskCount; + } + + public int getMaxAutoResubmits() { + return maxAutoResubmits; + } + + public void setMaxAutoResubmits(int maxAutoResubmits) { + this.maxAutoResubmits = maxAutoResubmits; + } + public Set getRemoteJobs() { return remoteJobs; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStatePipelineModule.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStatePipelineModule.java index 36f2109..d6a9896 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStatePipelineModule.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/ProcessingStatePipelineModule.java @@ -50,9 +50,9 @@ default ProcessingState nextProcessingState(ProcessingState currentProcessingSta * Increments the processing state of a {@link PipelineTask} in the database from its current * state in the database. */ - default void incrementProcessingState() { - processingSummaryOperations().updateProcessingState(pipelineTaskId(), - nextProcessingState(getProcessingState())); + default void incrementDatabaseProcessingState() { + ProcessingState nextState = nextProcessingState(databaseProcessingState()); + processingSummaryOperations().updateProcessingState(pipelineTaskId(), nextState); } default ProcessingSummaryOperations processingSummaryOperations() { @@ -64,8 +64,8 @@ default ProcessingSummaryOperations processingSummaryOperations() { * * @return current processing state. */ - default ProcessingState getProcessingState() { - return new ProcessingSummaryOperations().processingSummary(pipelineTaskId()) + default ProcessingState databaseProcessingState() { + return processingSummaryOperations().processingSummary(pipelineTaskId()) .getProcessingState(); } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskExecutionLog.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskExecutionLog.java index 8b87fab..585882b 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskExecutionLog.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/TaskExecutionLog.java @@ -4,7 +4,7 @@ import java.util.Date; import gov.nasa.ziggy.pipeline.definition.PipelineTask.State; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; import jakarta.persistence.Embeddable; import jakarta.persistence.EnumType; import jakarta.persistence.Enumerated; @@ -117,7 +117,7 @@ public String toString() { return "TaskExecutionLog [wh=" + workerHost + ", wt=" + workerThread + ", start=" + start + ", end=" + end + ", elapsed=" - + StringUtils.elapsedTime(startProcessingTime, endProcessingTime) + ", Si=" + + ZiggyStringUtils.elapsedTime(startProcessingTime, endProcessingTime) + ", Si=" + initialState + ", Sf=" + finalState + ", PSi=" + initialProcessingState + ", PSf=" + finalProcessingState + "]"; } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameter.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameter.java index 876010d..dc70a22 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameter.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameter.java @@ -9,7 +9,7 @@ import gov.nasa.ziggy.collections.ZiggyArrayUtils; import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.parameters.Parameters; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; import jakarta.persistence.Column; import jakarta.persistence.Embeddable; import jakarta.persistence.EnumType; @@ -132,7 +132,7 @@ public TypedParameter(TypedParameter original) { * all values. */ private String trimWhitespace(String value) { - return StringUtils.trimListWhitespace(value); + return ZiggyStringUtils.trimListWhitespace(value); } /** diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameterCollection.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameterCollection.java index f0f5fab..a43f753 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameterCollection.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/TypedParameterCollection.java @@ -8,6 +8,7 @@ import gov.nasa.ziggy.module.io.ProxyIgnore; import gov.nasa.ziggy.parameters.Parameters; +import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.uow.UnitOfWork; /** @@ -16,7 +17,7 @@ * * @author PT */ -public class TypedParameterCollection { +public class TypedParameterCollection implements ParametersInterface { @ProxyIgnore private Map parametersByName = new HashMap<>(); @@ -29,6 +30,7 @@ public TypedParameterCollection(Collection parameters) { } /** Returns the parameters in this collection as a sorted set. */ + @Override public Set getParameters() { return new TreeSet<>(parametersByName.values()); } @@ -42,6 +44,7 @@ public Set getParametersCopy() { return parameters; } + @Override public void setParameters(Collection parameters) { parametersByName = new HashMap<>(); for (TypedParameter parameter : parameters) { @@ -53,12 +56,19 @@ public void addParameter(TypedParameter parameter) { parametersByName.put(parameter.getName(), parameter); } + @Override + public void updateParameter(String name, String value) { + parametersByName.get(name).setString(value); + populate(parametersByName.values()); + } + /** Returns the given parameter. */ public TypedParameter getParameter(String name) { return parametersByName.get(name); } /** Returns the original map. */ + @Override public Map getParametersByName() { return parametersByName; } @@ -79,4 +89,9 @@ public boolean totalEquals(TypedParameterCollection other) { } return true; } + + @Override + public void validate() { + // Do nothing, by default. + } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/UniqueNameVersionPipelineComponent.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/UniqueNameVersionPipelineComponent.java index cecfb28..02f7461 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/UniqueNameVersionPipelineComponent.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/UniqueNameVersionPipelineComponent.java @@ -8,6 +8,7 @@ import gov.nasa.ziggy.pipeline.definition.crud.UniqueNameVersionPipelineComponentCrud; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; +import jakarta.persistence.Embedded; import jakarta.persistence.MappedSuperclass; import jakarta.persistence.Version; import jakarta.xml.bind.annotation.XmlAccessType; @@ -45,6 +46,9 @@ public abstract class UniqueNameVersionPipelineComponent retrieveAll() { - return list(createZiggyQuery(Group.class).column(Group_.name).ascendingOrder()); + List groups = list( + createZiggyQuery(Group.class).column(Group_.name).ascendingOrder()); + for (Group group : groups) { + Hibernate.initialize(group.getParameterSetNames()); + Hibernate.initialize(group.getPipelineDefinitionNames()); + Hibernate.initialize(group.getPipelineModuleNames()); + } + return groups; + } + + public List retrieveAll(Class clazz) { + List groups = retrieveAll(); + for (Group group : groups) { + setGroupMemberNames(group, clazz); + } + return groups; + } + + public Group retrieveGroupByName(String name, Class clazz) { + if (StringUtils.isEmpty(name)) { + return Group.DEFAULT; + } + Group group = uniqueResult(createZiggyQuery(Group.class).column(Group_.name).in(name)); + setGroupMemberNames(group, clazz); + return group; + } + + private void setGroupMemberNames(Group group, Class clazz) { + if (clazz.equals(PipelineDefinition.class)) { + Hibernate.initialize(group.getPipelineDefinitionNames()); + group.setMemberNames(group.getPipelineDefinitionNames()); + } + if (clazz.equals(ParameterSet.class)) { + Hibernate.initialize(group.getParameterSetNames()); + group.setMemberNames(group.getParameterSetNames()); + } } @Override diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ModelCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ModelCrud.java index d7ffe2f..324934c 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ModelCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ModelCrud.java @@ -30,21 +30,17 @@ public class ModelCrud extends AbstractCrud { */ public ModelRegistry retrieveCurrentRegistry() { - // I don't know how to do this in 1 query so I'll use 2. - // TODO: reformat as subquery. - ZiggyQuery idQuery = createZiggyQuery(ModelRegistry.class, Long.class); - idQuery.column(ModelRegistry_.id).max(); - Long maxId = uniqueResult(idQuery); - if (maxId == null) { - ModelRegistry modelRegistry = new ModelRegistry(); - persist(modelRegistry); - return modelRegistry; - } - ZiggyQuery query = createZiggyQuery(ModelRegistry.class); - query.column(ModelRegistry_.id).in(maxId); - - return uniqueResult(query); + ZiggyQuery idSubquery = query.ziggySubquery(ModelRegistry.class, + Long.class); + idSubquery.column(ModelRegistry_.id).max(); + query.column(ModelRegistry_.id).in(idSubquery); + ModelRegistry currentRegistry = uniqueResult(query); + if (currentRegistry == null) { + currentRegistry = new ModelRegistry(); + persist(currentRegistry); + } + return currentRegistry; } /** diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ParameterSetCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ParameterSetCrud.java index ce9192f..f27b6d7 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ParameterSetCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/ParameterSetCrud.java @@ -1,23 +1,15 @@ package gov.nasa.ziggy.pipeline.definition.crud; -import static com.google.common.base.Preconditions.checkNotNull; - import java.util.Collection; import java.util.List; import java.util.stream.Collectors; import gov.nasa.ziggy.crud.ZiggyQuery; -import gov.nasa.ziggy.module.remote.RemoteParameters; -import gov.nasa.ziggy.parameters.InternalParameters; import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.ParameterSet_; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** - * Provides CRUD methods for {@link ParameterSet}. Note: methods that return all parameter sets will - * not return those that contain instances of {@link InternalParameters}, since these are supposed - * to be "invisible" to users under normal circumstances (and in all events they aren't supposed to - * be edited by users). + * Provides CRUD methods for {@link ParameterSet}. * * @author Todd Klaus * @author PT @@ -34,9 +26,7 @@ public List retrieveAll() { } public List visibleParameterSets(List allParameterSets) { - return allParameterSets.stream() - .filter(ParameterSet::visibleParameterSet) - .collect(Collectors.toList()); + return allParameterSets.stream().collect(Collectors.toList()); } public ParameterSet retrieve(long id) { @@ -47,48 +37,6 @@ public ParameterSet retrieve(long id) { return result; } - /** - * Retrieves the latest {@link RemoteParameters} from the database. These may be later than - * those associated with the task to allow restarting the task with modified parameters. - * - * @param pipelineTask the non-{@code null} pipeline task to base the retrieval upon - * @return the RemoteParameters to use - */ - public RemoteParameters retrieveRemoteParameters(PipelineTask pipelineTask) { - checkNotNull(pipelineTask, "pipelineTask"); - - ParameterSet parameterSet = pipelineTask.getParameterSet(RemoteParameters.class, false); - if (parameterSet == null) { - return null; - } - String name = parameterSet.getName(); - ParameterSet latestParameterSet = retrieveLatestVersionForName(name); - - return latestParameterSet.parametersInstance(); - } - - /** - * Retrieves the version number of the latest {@link RemoteParameters} from the database. This - * may be later than the value associated with the task to allow restarting the task with - * modified parameters. - * - * @param pipelineTask the non-{@code null} pipeline task to base the retrieval upon - * @return the version number of the current {@link RemoteParameters} instance for the selected - * task. - */ - public int retrieveRemoteParameterVersionNumber(PipelineTask pipelineTask) { - checkNotNull(pipelineTask, "pipelineTask"); - - ParameterSet parameterSet = pipelineTask.getParameterSet(RemoteParameters.class); - if (parameterSet == null) { - return -1; - } - String name = parameterSet.getName(); - ParameterSet latestParameterSet = retrieveLatestVersionForName(name); - - return latestParameterSet.getVersion(); - } - /** * Populates the XML fields for a {@link ParameterSet} instance. This ensures that the instance * has its XML and database fields consistent with each other upon retrieval from the database. diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrud.java index e5617d4..6cf1dc4 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrud.java @@ -12,6 +12,9 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions_; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition_; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; @@ -99,6 +102,38 @@ private void deleteNodes(List nodes) { } } + public boolean processingModeExistsInDatabase(String pipelineName) { + return uniqueResult(createZiggyQuery(PipelineDefinitionProcessingOptions.class) + .column(PipelineDefinitionProcessingOptions_.pipelineName) + .in(pipelineName)) != null; + } + + public ProcessingMode retrieveProcessingMode(String pipelineName) { + return uniqueResult( + createZiggyQuery(PipelineDefinitionProcessingOptions.class, ProcessingMode.class) + .column(PipelineDefinitionProcessingOptions_.pipelineName) + .in(pipelineName) + .column(PipelineDefinitionProcessingOptions_.processingMode) + .select()); + } + + public PipelineDefinitionProcessingOptions updateProcessingMode(String pipelineName, + ProcessingMode processingMode) { + PipelineDefinitionProcessingOptions pipelineDefinitionProcessingOptions = uniqueResult( + createZiggyQuery(PipelineDefinitionProcessingOptions.class) + .column(PipelineDefinitionProcessingOptions_.pipelineName) + .in(pipelineName)); + pipelineDefinitionProcessingOptions.setProcessingMode(processingMode); + return super.merge(pipelineDefinitionProcessingOptions); + } + + public PipelineDefinition merge(PipelineDefinition pipelineDefinition) { + if (!processingModeExistsInDatabase(pipelineDefinition.getName())) { + persist(new PipelineDefinitionProcessingOptions(pipelineDefinition.getName())); + } + return super.merge(pipelineDefinition); + } + @Override public String componentNameForExceptionMessages() { return "pipeline definition"; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionNodeCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionNodeCrud.java index dfedd88..1a6fed8 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionNodeCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionNodeCrud.java @@ -1,14 +1,13 @@ package gov.nasa.ziggy.pipeline.definition.crud; import gov.nasa.ziggy.crud.AbstractCrud; +import gov.nasa.ziggy.crud.ZiggyQuery; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources_; /** * CRUD class for {@link PipelineDefinitionNode}. - *

      - * The PipelineDefinitionNode requires only the generic CRUD features like - * {@link AbstractCrud#persist(Object)} and {@link AbstractCrud#merge(Object)}. Thus no other - * class-specific methods are defined here. * * @author PT */ @@ -18,4 +17,27 @@ public class PipelineDefinitionNodeCrud extends AbstractCrud componentClass() { return PipelineDefinitionNode.class; } + + /** + * Retrieves the {@link PipelineDefinitionNodeExecutionResources} for a given + * {@link PipelineDefinitionNode}. If none exists, one is created and persisted (and then + * returned, of course). + */ + public PipelineDefinitionNodeExecutionResources retrieveExecutionResources( + PipelineDefinitionNode node) { + + ZiggyQuery query = createZiggyQuery( + PipelineDefinitionNodeExecutionResources.class); + query.column(PipelineDefinitionNodeExecutionResources_.pipelineName) + .in(node.getPipelineName()); + query.column(PipelineDefinitionNodeExecutionResources_.pipelineModuleName) + .in(node.getModuleName()); + PipelineDefinitionNodeExecutionResources executionResources = uniqueResult(query); + if (executionResources == null) { + executionResources = new PipelineDefinitionNodeExecutionResources( + node.getPipelineName(), node.getModuleName()); + persist(executionResources); + } + return executionResources; + } } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrud.java index f4e4def..3c7fc35 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrud.java @@ -3,8 +3,13 @@ import java.util.Collection; import java.util.List; +import gov.nasa.ziggy.crud.ZiggyQuery; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition_; +import gov.nasa.ziggy.pipeline.definition.PipelineModuleExecutionResources; +import gov.nasa.ziggy.pipeline.definition.PipelineModuleExecutionResources_; +import gov.nasa.ziggy.uow.UnitOfWorkGenerator; /** * Provides CRUD methods for {@link PipelineModuleDefinition} @@ -23,6 +28,24 @@ public List retrieveAll() { .ascendingOrder()); } + public PipelineModuleExecutionResources retrieveExecutionResources( + PipelineModuleDefinition module) { + ZiggyQuery query = createZiggyQuery( + PipelineModuleExecutionResources.class); + query.column(PipelineModuleExecutionResources_.pipelineModuleName).in(module.getName()); + PipelineModuleExecutionResources resources = uniqueResult(query); + if (resources == null) { + resources = new PipelineModuleExecutionResources(); + resources.setPipelineModuleName(module.getName()); + resources = merge(resources); + } + return resources; + } + + public ClassWrapper retrieveUnitOfWorkGenerator(String moduleName) { + return retrieveLatestVersionForName(moduleName).getUnitOfWorkGenerator(); + } + @Override protected void populateXmlFields(Collection objects) { } diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrud.java index 590ad6b..766d3b8 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrud.java @@ -10,12 +10,13 @@ import java.util.Map; import java.util.Set; +import org.apache.commons.collections.CollectionUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.crud.AbstractCrud; import gov.nasa.ziggy.crud.ZiggyQuery; -import gov.nasa.ziggy.module.remote.RemoteParameters; +import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; import gov.nasa.ziggy.pipeline.PipelineOperations; import gov.nasa.ziggy.pipeline.PipelineOperations.TaskStateSummary; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; @@ -153,21 +154,26 @@ public List retrieveAll(PipelineInstanceNode pipelineInstanceNode) * module name as the {@link PipelineDefinitionNode} argument, and also have the same pipeline * name. This ensures that if the node has been duplicated, both the original and the copy will * count as having processed the task in question. + *

      + * If the taskIds argument is null or empty, the method will return all the task IDs in the + * database that correspond to the specified pipeline definition node. */ - public List retrieveIdsForPipelineDefinitionNode(Collection taskIds, - PipelineDefinitionNode pipelineDefinitionNode) { + public List retrieveIdsForPipelineDefinitionNode( + PipelineDefinitionNode pipelineDefinitionNode, Collection taskIds) { String pipelineDefinitionNodeName = pipelineDefinitionNode.getModuleName(); String pipelineDefinitionName = pipelineDefinitionNode.getPipelineName(); ZiggyQuery query = createZiggyQuery(PipelineTask.class, Long.class); - query.column(PipelineTask_.id).select(); + query.column(PipelineTask_.id).select().distinct(true); query.where(query.in(query.get(PipelineTask_.pipelineInstanceNode) .get(PipelineInstanceNode_.pipelineDefinitionNode) .get(PipelineDefinitionNode_.moduleName), pipelineDefinitionNodeName)); query.where(query.in(query.get(PipelineTask_.pipelineInstanceNode) .get(PipelineInstanceNode_.pipelineDefinitionNode) .get(PipelineDefinitionNode_.pipelineName), pipelineDefinitionName)); - query.column(PipelineTask_.id).chunkedIn(taskIds); + if (!CollectionUtils.isEmpty(taskIds)) { + query.column(PipelineTask_.id).chunkedIn(taskIds); + } return list(query); } @@ -274,8 +280,8 @@ public ClearStaleStateResults clearStaleState() { // If the task was executing remotely, and it was queued or executing, we can try // to resume monitoring on it -- the jobs may have continued to run while the // supervisor was down. - RemoteParameters remoteParams = new ParameterSetCrud().retrieveRemoteParameters(task); - if (remoteParams != null && remoteParams.isEnabled()) { + if (task.getProcessingMode() != null + && task.getProcessingMode().equals(AlgorithmType.REMOTE)) { ProcessingState state = new ProcessingSummaryOperations() .processingSummary(task.getId()) .getProcessingState(); @@ -283,7 +289,7 @@ public ClearStaleStateResults clearStaleState() { || state == ProcessingState.ALGORITHM_EXECUTING) { log.info("Resuming monitoring for task " + task.getId()); TaskRequest taskRequest = new TaskRequest(instanceId, instanceNodeId, - task.getPipelineDefinitionNode().getId(), task.getId(), Priority.HIGHEST, + task.pipelineDefinitionNode().getId(), task.getId(), Priority.HIGHEST, false, PipelineModule.RunMode.RESUME_MONITORING); ZiggyMessenger.publish(taskRequest); continue; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/UniqueNameVersionPipelineComponentCrud.java b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/UniqueNameVersionPipelineComponentCrud.java index 43b53f7..37ce5c8 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/UniqueNameVersionPipelineComponentCrud.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/definition/crud/UniqueNameVersionPipelineComponentCrud.java @@ -39,21 +39,16 @@ public abstract class UniqueNameVersionPipelineComponentCrud versionQuery = createZiggyQuery(componentClass(), Integer.class); + ZiggyQuery query = createZiggyQuery(componentClass()); + ZiggyQuery versionQuery = query.ziggySubquery(componentClass(), Integer.class); versionQuery.column(UniqueNameVersionPipelineComponent_.NAME).in(name); versionQuery.column(UniqueNameVersionPipelineComponent_.VERSION).max(); - Integer maxVersionForName = uniqueResult(versionQuery); - - if (maxVersionForName == null) { - return null; - } - - ZiggyQuery query = createZiggyQuery(componentClass()); query.column(UniqueNameVersionPipelineComponent_.NAME).in(name); - query.column(UniqueNameVersionPipelineComponent_.VERSION).in(maxVersionForName); + query.column(UniqueNameVersionPipelineComponent_.VERSION).in(versionQuery); U result = uniqueResult(query); + if (result == null) { + return null; + } populateXmlFields(List.of(result)); return result; @@ -123,9 +118,17 @@ public U rename(U pipelineComponent, String newName) { * * @see #merge(Object) */ + // TODO If this method calls merge, it must return the merged object! + // Note that this note was added in a commit where this call was made and the parameter o was + // later used in a merge() call. The merge() call created a second object and subsequently + // caused exceptions when uniqueResult() was called. @SuppressWarnings("unchecked") @Override public void persist(Object o) { + if (!(o instanceof UniqueNameVersionPipelineComponent)) { + super.merge(o); + return; + } persistOrMerge((U) o); } @@ -151,6 +154,9 @@ public void persist(Object o) { @SuppressWarnings("unchecked") @Override public T merge(T o) { + if (!(o instanceof UniqueNameVersionPipelineComponent)) { + return super.merge(o); + } return (T) persistOrMerge((U) o); } @@ -160,6 +166,7 @@ private U persistOrMerge(U pipelineComponent) { // If there's nothing at all in the database, we persist. if (latestVersion == null) { + pipelineComponent.updateAuditInfo(); super.persist(pipelineComponent); return pipelineComponent; } @@ -173,9 +180,15 @@ private U persistOrMerge(U pipelineComponent) { // If there's an instance in the database, we take the one that needs to go // to the database and set its version as needed; then merge it. U unlockedVersion = pipelineComponent.unlockedVersion(); + unlockedVersion.updateAuditInfo(); return super.merge(unlockedVersion); } + /** Uses the {@link AbstractCrud} method to persist an object. */ + public void persistPojo(V pojo) { + super.persist(pojo); + } + /** * String that can be used in exception messages so that the messages are properly customized to * the component class and CRUD class that are causing the exception. diff --git a/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlReference.java b/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlReference.java index 687885c..c0ea9da 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlReference.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlReference.java @@ -2,7 +2,7 @@ import java.util.Objects; -import gov.nasa.ziggy.data.management.DataFileType; +import gov.nasa.ziggy.data.datastore.DataFileType; import gov.nasa.ziggy.pipeline.definition.ModelType; import jakarta.xml.bind.annotation.XmlAccessType; import jakarta.xml.bind.annotation.XmlAccessorType; diff --git a/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java b/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java index 7e85396..e068320 100644 --- a/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java +++ b/src/main/java/gov/nasa/ziggy/pipeline/xml/XmlSchemaExporter.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -48,8 +48,8 @@ import com.google.common.collect.ImmutableSet; +import gov.nasa.ziggy.data.datastore.DatastoreConfigurationFile; import gov.nasa.ziggy.data.management.Acknowledgement; -import gov.nasa.ziggy.data.management.DatastoreConfigurationFile; import gov.nasa.ziggy.data.management.Manifest; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.parameters.ParameterLibrary; diff --git a/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java b/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java index 108e701..3ff3859 100644 --- a/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java +++ b/src/main/java/gov/nasa/ziggy/services/alert/AlertService.java @@ -34,7 +34,7 @@ public String toString() { } private static final Logger log = LoggerFactory.getLogger(AlertService.class); - + public static final int DEFAULT_TASK_ID = -1; public static final boolean BROADCAST_ALERTS_ENABLED_DEFAULT = false; public boolean broadcastEnabled = false; @@ -59,12 +59,12 @@ public AlertService() { } public void generateAlert(String sourceComponent, String message) { - generateAlert(sourceComponent, -1, message); + generateAlert(sourceComponent, DEFAULT_TASK_ID, message); } public void generateAlert(String sourceComponent, AlertService.Severity severity, String message) { - generateAlert(sourceComponent, -1, severity, message); + generateAlert(sourceComponent, DEFAULT_TASK_ID, severity, message); } /** diff --git a/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java b/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java index 1ed8b93..a302eb3 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java +++ b/src/main/java/gov/nasa/ziggy/services/config/ConfigMerge.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java b/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java index ff80eda..b52bf2e 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java +++ b/src/main/java/gov/nasa/ziggy/services/config/DirectoryProperties.java @@ -221,4 +221,9 @@ public static Path datastoreRootDir() { return Paths.get( ZiggyConfiguration.getInstance().getString(PropertyName.DATASTORE_ROOT_DIR.property())); } + + public static Path dataReceiptDir() { + return Paths.get( + ZiggyConfiguration.getInstance().getString(PropertyName.DATA_RECEIPT_DIR.property())); + } } diff --git a/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java b/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java index 46d753a..06cac51 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java +++ b/src/main/java/gov/nasa/ziggy/services/config/PropertyName.java @@ -70,18 +70,17 @@ public enum PropertyName { * The log4j2 configuration property. As it is not user-modifiable, this property should not be * documented in the manual. */ - LOG4J2_CONFIGURATION_FILE("log4j2.configurationFile"), - /** - * Property in the config service that points to the log4j.xml file used by Java code called - * from MATLAB - */ - MATLAB_LOG4J_CONFIG_FILE("matlab.log4j.config"), - /** Set to true to initialize log4j when starting MATLAB. */ MATLAB_LOG4J_CONFIG_INITIALIZE("matlab.log4j.initialize"), + /** + * Architecture of the operating system. As it is not user-modifiable, this property should not + * be documented in the manual. + */ + ARCHITECTURE("os.arch"), + /** * Name for the operating system. As it is not user-modifiable, this property should not be * documented in the manual. @@ -145,9 +144,18 @@ public enum PropertyName { /** Name of relational database software package. */ DATABASE_SOFTWARE("ziggy.database.software.name"), + /** + * Environment definition used by Ziggy. This property should not be documented in the manual to + * prevent the user from breaking Ziggy. The user should use ziggy.pipeline.environment instead. + */ + ZIGGY_RUNTIME_ENVIRONMENT("ziggy.environment"), + /** Ziggy home directory (the build directory in the top-level Ziggy directory). */ ZIGGY_HOME_DIR("ziggy.home.dir"), + /** Location of the current log file. */ + ZIGGY_LOG_FILE("ziggy.logFile"), + /** Location and name of the logo file for the pipeline (not the Ziggy logo). */ PIPELINE_LOGO_FILE("ziggy.logoFile"), @@ -255,15 +263,6 @@ public enum PropertyName { */ SUPERVISOR_PORT("ziggy.supervisor.port"), - /** - * Allows Ziggy unit test classes to inform the configuration system that it is, in fact, a test - * environment, hence to not load the pipeline properties. Usually this is accomplished - * automatically, but there are some corner cases where it's necessary to manually inform the - * configuration system. In this case, it's necessary to set this as a system property so that - * it survives resets of the configuration. - */ - TEST_ENVIRONMENT("ziggy.test.environment"), - /** * Property used by tests to ensure that ziggy.properties can be read. Its value is expected to * be {@code from.default.location}. This property should not be documented in the manual. @@ -299,7 +298,7 @@ public String property() { } @Override - /** The {@link #property} method is favored. */ + /** The {@link #property} method is favored unless this method is used implicitly. */ public String toString() { return property(); } diff --git a/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java b/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java index 6bea3ba..acf983a 100644 --- a/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java +++ b/src/main/java/gov/nasa/ziggy/services/config/ZiggyConfiguration.java @@ -58,7 +58,7 @@ public class ZiggyConfiguration { public static final String PIPELINE_CONFIG_DEFAULT_FILE = "ziggy.properties"; private static ImmutableConfiguration instance; - private static Configuration mutableInstance; + private static CompositeConfiguration mutableInstance; private static ConfigurationInterpolator interpolator; /** @@ -96,20 +96,12 @@ public static synchronized ImmutableConfiguration getInstance() { private static Configuration getConfiguration() { CompositeConfiguration config = new CompositeConfiguration(); config.setThrowExceptionOnMissing(true); - + if (mutableInstance != null) { + config.addConfiguration(mutableInstance); + } loadSystemConfiguration(config); - - // We know that we are in a test environment if there's a mutable instance. - // However, in rare cases, the mutable instance is null even though we are - // in a test environment. In those cases, the test can define the - // TEST_ENVIRONMENT as a system property to accomplish the same thing. - boolean testConfiguration = mutableInstance != null - || config.getString(PropertyName.TEST_ENVIRONMENT.toString(), null) != null; - - if (!testConfiguration) { + if (mutableInstance == null) { loadPipelineConfiguration(config); - } else if (mutableInstance != null) { - config.addConfiguration(mutableInstance); } loadZiggyConfiguration(config); loadBuildConfiguration(config); @@ -249,20 +241,25 @@ public static void logJvmProperties() { } /** - * Returns a mutable configuration object as described in {@link #getInstance()}. This method - * resets the immutable configuration in case a prior test forgot to call reset, so that - * subsequent calls to {@link #getInstance()} will use this immutable instance rather than the - * prior instance. For testing only. Production code should call {@link #getInstance()}. + * Returns a mutable configuration object as described in {@link #getInstance()}. For testing + * only. Production code should call {@link #getInstance()}. */ - public static synchronized Configuration getMutableInstance() { - if (mutableInstance == null) { - mutableInstance = getConfiguration(); - mutableInstance.setSynchronizer(new ReadWriteSynchronizer()); - instance = null; - } + public static synchronized CompositeConfiguration getMutableInstance() { return mutableInstance; } + /** + * Sets the mutable configuration object as described in {@link #getInstance()}. Resets the + * production instance in case a prior test neglected to call {@link #reset()} so that the next + * call to {@link #getInstance()} uses this mutable instance. For testing only. Production code + * should call {@link #getInstance()}. + */ + public static synchronized void setMutableInstance(CompositeConfiguration mutableInstance) { + ZiggyConfiguration.mutableInstance = mutableInstance; + ZiggyConfiguration.mutableInstance.setSynchronizer(new ReadWriteSynchronizer()); + instance = null; + } + /** Clear the immutable and mutable configuration instances. */ public static synchronized void reset() { instance = null; diff --git a/src/main/java/gov/nasa/ziggy/services/database/DatabaseTransactionFactory.java b/src/main/java/gov/nasa/ziggy/services/database/DatabaseTransactionFactory.java index 4849680..7ac0fc0 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/DatabaseTransactionFactory.java +++ b/src/main/java/gov/nasa/ziggy/services/database/DatabaseTransactionFactory.java @@ -50,7 +50,7 @@ public class DatabaseTransactionFactory { /** * Specifies the transaction context. Specifically, specifies whether the call to - * {@link DatabaseTransactionFactory#performTransaction(DatabaseTransaction) occurs in the + * {@link DatabaseTransactionFactory#performTransaction(DatabaseTransaction)} occurs in the * context of an existing transaction. If the context is an existing transaction, then * performTransaction should not begin, commit, or roll back the transaction, or close the * session at the end of the operation; if the context is outside of an existing transaction, @@ -58,7 +58,7 @@ public class DatabaseTransactionFactory { * * @author PT */ - private enum TransactionContext { + public enum TransactionContext { /** * Existing transaction, so don't do anything to break the transaction that is outside of diff --git a/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java b/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java index 5aa87b3..29cf738 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java +++ b/src/main/java/gov/nasa/ziggy/services/database/HsqldbController.java @@ -3,7 +3,9 @@ import static com.google.common.base.Preconditions.checkNotNull; import static gov.nasa.ziggy.services.config.PropertyName.HIBERNATE_URL; import static gov.nasa.ziggy.util.WrapperUtils.WRAPPER_CLASSPATH_PROP_NAME_PREFIX; +import static gov.nasa.ziggy.util.WrapperUtils.WRAPPER_JAVA_ADDITIONAL_PROP_NAME_PREFIX; import static gov.nasa.ziggy.util.WrapperUtils.WRAPPER_LIBRARY_PATH_PROP_NAME_PREFIX; +import static gov.nasa.ziggy.util.WrapperUtils.WRAPPER_LOG_FILE_PROP_NAME; import static gov.nasa.ziggy.util.WrapperUtils.wrapperParameter; import java.io.File; @@ -27,6 +29,7 @@ import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.process.ExternalProcess; +import gov.nasa.ziggy.services.process.ExternalProcessUtils; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.WrapperUtils.WrapperCommand; @@ -54,7 +57,6 @@ public class HsqldbController extends DatabaseController { private static final String SCHEMA_CREATE_FILE = "ddl.hsqldb-create.sql"; private static final String SCHEMA_DROP_FILE = "ddl.hsqldb-drop.sql"; private static final String DRIVER_CLASS_NAME = "org.hsqldb.jdbc.JDBCDriver"; - private static final String LOG_SUFFIX = ".log"; private static final String INIT_TABLE_NAME = "HSQLDB_CONTROLLER_CREATED"; private static final String TABLE_COUNT = "select count(*) from INFORMATION_SCHEMA.tables where TABLE_SCHEMA = 'PUBLIC' and table_name != '%s'"; @@ -65,6 +67,7 @@ public class HsqldbController extends DatabaseController { private static final String INSERT_INIT_TABLE_SQL = "insert into %s values('This database schema was automatically created on %s.')"; private static final String HSQLDB_BIN_NAME = "hsqldb"; + private static final String LOG_FILE = "hsqldb.log"; private static final int DATABASE_SETTLE_MILLIS = 1000; /** @@ -82,18 +85,14 @@ public Path dataDir() { return databaseDir; } - /** Not used by Ziggy. */ @Override public Path logDir() { - return dataDir(); + return DirectoryProperties.databaseLogDir(); } - /** Not used by Ziggy. */ @Override public Path logFile() { - return logDir().resolve( - ZiggyConfiguration.getInstance().getString(PropertyName.DATABASE_NAME.property()) - + LOG_SUFFIX); + return logDir().resolve(LOG_FILE); } @Override @@ -372,10 +371,15 @@ private CommandLine hsqldbCommand(WrapperCommand cmd) { String ziggyLibDir = DirectoryProperties.ziggyLibDir().toString(); commandLine + .addArgument(wrapperParameter(WRAPPER_LOG_FILE_PROP_NAME, logFile().toString())) .addArgument(wrapperParameter(WRAPPER_CLASSPATH_PROP_NAME_PREFIX, 1, DirectoryProperties.ziggyHomeDir().resolve("libs").resolve("*.jar").toString())) .addArgument( - wrapperParameter(WRAPPER_LIBRARY_PATH_PROP_NAME_PREFIX, 1, ziggyLibDir)); + wrapperParameter(WRAPPER_LIBRARY_PATH_PROP_NAME_PREFIX, 1, ziggyLibDir)) + .addArgument(wrapperParameter(WRAPPER_JAVA_ADDITIONAL_PROP_NAME_PREFIX, 3, + ExternalProcessUtils.log4jConfigString())) + .addArgument(wrapperParameter(WRAPPER_JAVA_ADDITIONAL_PROP_NAME_PREFIX, 4, + ExternalProcessUtils.ziggyLog(logFile().toString()))); } return commandLine.addArgument(cmd.toString()); diff --git a/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java b/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java index b26c128..9e1ccce 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java +++ b/src/main/java/gov/nasa/ziggy/services/database/MatlabJavaInitialization.java @@ -18,6 +18,7 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -30,14 +31,16 @@ * @author Todd Klaus */ public class MatlabJavaInitialization { + private static final Logger log = LoggerFactory.getLogger(MatlabJavaInitialization.class); - public static final String LOG4J_LOGFILE_PREFIX = "log4j.logfile.prefix"; public static final String MATLAB_PIDS_FILENAME = ".matlab.pids"; - - private static final String DEFAULT_LOG4J_LOGFILE_PREFIX = "${ziggy.config.dir}/../logs/matlab"; public static final String PID_FILE_CHARSET = "ISO-8859-1"; + private static final String MATLAB_LOG_FILE = DirectoryProperties.cliLogDir() + .resolve("matlab.log") + .toString(); + private static boolean initialized = false; /** @@ -70,9 +73,9 @@ public static synchronized void initialize() { if (config.getBoolean(PropertyName.MATLAB_LOG4J_CONFIG_INITIALIZE.property(), false)) { String log4jConfigFile = config - .getString(PropertyName.MATLAB_LOG4J_CONFIG_FILE.property()); + .getString(PropertyName.LOG4J2_CONFIGURATION_FILE.property()); - log.info(PropertyName.MATLAB_LOG4J_CONFIG_FILE + " = " + log4jConfigFile); + log.info(PropertyName.LOG4J2_CONFIGURATION_FILE + " = " + log4jConfigFile); if (log4jConfigFile != null) { log.info("Log4j initialized with DOMConfigurator from: " + log4jConfigFile); @@ -81,7 +84,7 @@ public static synchronized void initialize() { // statement will have no effect. Consider rearchitecting so that this property // is already set before the MATLAB binary is started, presuming this property // is even used. - System.setProperty(LOG4J_LOGFILE_PREFIX, DEFAULT_LOG4J_LOGFILE_PREFIX); + System.setProperty(PropertyName.ZIGGY_LOG_FILE.property(), MATLAB_LOG_FILE); ConfigurationFactory.setConfigurationFactory(new XmlConfigurationFactory()); try { Configurator.reconfigure(new URI(log4jConfigFile)); diff --git a/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java b/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java index cbacde1..10ad52d 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java +++ b/src/main/java/gov/nasa/ziggy/services/database/PostgresqlController.java @@ -16,7 +16,6 @@ import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.process.ExternalProcess; -import gov.nasa.ziggy.ui.ClusterController; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.io.FileUtil; @@ -222,7 +221,7 @@ public int start() { .execute(); // Postgres will shut down and exit if it is pinged too soon. - ClusterController.waitForProcessToSettle(DATABASE_SETTLE_MILLIS); + waitForProcessToSettle(DATABASE_SETTLE_MILLIS); return status; } @@ -301,4 +300,16 @@ private String commandStringWithPath(String command) { Path databaseBinDir = DirectoryProperties.databaseBinDir(); return databaseBinDir != null ? databaseBinDir.resolve(command).toString() : command; } + + /** + * Waits the given number of milliseconds for a process to settle. + */ + public static void waitForProcessToSettle(long millis) { + try { + log.debug("Waiting for process to settle"); + Thread.sleep(millis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } } diff --git a/src/main/java/gov/nasa/ziggy/services/database/ZiggySchemaExport.java b/src/main/java/gov/nasa/ziggy/services/database/ZiggySchemaExport.java index 7da0c77..5fa0afc 100644 --- a/src/main/java/gov/nasa/ziggy/services/database/ZiggySchemaExport.java +++ b/src/main/java/gov/nasa/ziggy/services/database/ZiggySchemaExport.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java index deef7c3..977ac1a 100644 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java +++ b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEvent.java @@ -2,11 +2,14 @@ import java.util.Date; import java.util.Objects; +import java.util.Set; +import jakarta.persistence.ElementCollection; import jakarta.persistence.Entity; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; +import jakarta.persistence.JoinTable; import jakarta.persistence.SequenceGenerator; import jakarta.persistence.Table; @@ -31,13 +34,19 @@ public class ZiggyEvent { private Date eventTime; private long pipelineInstanceId; + @ElementCollection + @JoinTable(name = "ziggy_Event_eventLabels") + private Set eventLabels; + @SuppressWarnings("unused") private ZiggyEvent() { } - public ZiggyEvent(String eventHandlerName, String pipelineName, long pipelineInstance) { + public ZiggyEvent(String eventHandlerName, String pipelineName, long pipelineInstance, + Set eventLabels) { this.eventHandlerName = eventHandlerName; this.pipelineName = pipelineName; + this.eventLabels = eventLabels; eventTime = new Date(); pipelineInstanceId = pipelineInstance; } @@ -82,6 +91,14 @@ public void setPipelineInstanceId(long pipelineInstance) { pipelineInstanceId = pipelineInstance; } + public Set getEventLabels() { + return eventLabels; + } + + public void setEventLabels(Set eventLabels) { + this.eventLabels = eventLabels; + } + @Override public int hashCode() { return Objects.hash(id); diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java index 97839d7..20f3fbd 100644 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java +++ b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandler.java @@ -8,7 +8,6 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.util.Date; import java.util.HashMap; import java.util.HashSet; import java.util.Map; @@ -25,10 +24,8 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.PipelineOperations; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionCrud; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.alert.AlertService.Severity; @@ -39,7 +36,6 @@ import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.Iso8601Formatter; import gov.nasa.ziggy.util.ZiggyShutdownHook; import jakarta.persistence.Entity; import jakarta.persistence.Id; @@ -232,43 +228,16 @@ private void startPipeline(ReadyFile readyFile) { log.debug("Event handler labels: " + readyFile.labels()); log.info("Event handler " + name + " starting pipeline " + pipelineName + "..."); - // Start by saving the event labels as a parameter set. - String paramSetName = (String) DatabaseTransactionFactory.performTransaction(() -> { - - String parameterSetName = name + " " + readyFile.getName(); - String parameterSetDescription = "Created by event handler " + name + " @ " - + new Date(); - ZiggyEventLabels eventLabels = null; - ParameterSet paramSet = new ParameterSetCrud() - .retrieveLatestVersionForName(parameterSetName); - if (paramSet != null) { - eventLabels = (ZiggyEventLabels) paramSet.parametersInstance(); - eventLabels.setEventName(readyFile.getName()); - eventLabels.setEventLabels(readyFile.labelsArray()); - pipelineOperations().updateParameterSet(paramSet, eventLabels, - parameterSetDescription, true); - } else { - paramSet = new ParameterSet(parameterSetName); - paramSet.setDescription(parameterSetDescription); - eventLabels = new ZiggyEventLabels(); - eventLabels.setEventHandlerName(name); - eventLabels.setEventName(readyFile.getName()); - eventLabels.setEventLabels(readyFile.labelsArray()); - paramSet.populateFromParametersInstance(eventLabels); - new ParameterSetCrud().persist(paramSet); - } - return parameterSetName; - }); - - // Create a new pipeline instance that includes the event handler labels parameter set. + // Create a new pipeline instance that includes the event handler labels. PipelineDefinition pipelineDefinition = (PipelineDefinition) DatabaseTransactionFactory .performTransaction( () -> new PipelineDefinitionCrud().retrieveLatestVersionForName(pipelineName)); PipelineInstance pipelineInstance = pipelineOperations().fireTrigger(pipelineDefinition, - instanceName(), null, null, paramSetName); + null, null, null, readyFile.getLabels()); ZiggyMessenger.publish(new InvalidateConsoleModelsMessage()); DatabaseTransactionFactory.performTransaction(() -> { - final ZiggyEvent event = new ZiggyEvent(name, pipelineName, pipelineInstance.getId()); + final ZiggyEvent event = new ZiggyEvent(name, pipelineName, pipelineInstance.getId(), + readyFile.getLabels()); new ZiggyEventCrud().persist(event); return null; }); @@ -346,15 +315,6 @@ long readyFileCheckIntervalMillis() { return READY_FILE_CHECK_INTERVAL_MILLIS; } - /** - * Returns an instance name that combines the name of the {@link ZiggyEventHandler} with a - * timestamp. The instance name is provided by a method which allows a fixed name to be - * specified for test purposes. Package scope for tests. - */ - String instanceName() { - return name + "-" + Iso8601Formatter.dateTimeLocalFormatter().format(new Date()); - } - private Path interpolatedDirectory() { return Paths.get((String) ZiggyConfiguration.interpolate(directory)); } @@ -492,8 +452,8 @@ public String labels() { return labels.toString(); } - public String[] labelsArray() { - return labels.toArray(new String[0]); + public Set getLabels() { + return labels; } @Override diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java index 170befb..fa916e0 100644 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java +++ b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerDefinitionImporter.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline diff --git a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventLabels.java b/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventLabels.java deleted file mode 100644 index a3faca0..0000000 --- a/src/main/java/gov/nasa/ziggy/services/events/ZiggyEventLabels.java +++ /dev/null @@ -1,49 +0,0 @@ -package gov.nasa.ziggy.services.events; - -import gov.nasa.ziggy.collections.ZiggyArrayUtils; -import gov.nasa.ziggy.collections.ZiggyDataType; -import gov.nasa.ziggy.parameters.InternalParameters; -import gov.nasa.ziggy.pipeline.definition.TypedParameter; - -/** - * Contains the event labels associated with a particular event that has been managed by the - * {@link ZiggyEventHandler}. - * - * @author PT - */ -public class ZiggyEventLabels extends InternalParameters { - - // If a field is renamed, update the parameter string in its setter. - - private String eventHandlerName; - private String eventName; - private String[] eventLabels; - - public String getEventHandlerName() { - return eventHandlerName; - } - - public void setEventHandlerName(String eventHandlerName) { - this.eventHandlerName = eventHandlerName; - addParameter(new TypedParameter("eventHandlerName", this.eventHandlerName)); - } - - public String getEventName() { - return eventName; - } - - public void setEventName(String eventName) { - this.eventName = eventName; - addParameter(new TypedParameter("eventName", this.eventName)); - } - - public String[] getEventLabels() { - return eventLabels; - } - - public void setEventLabels(String[] eventLabels) { - this.eventLabels = eventLabels; - addParameter(new TypedParameter("eventLabels", - ZiggyArrayUtils.arrayToString(this.eventLabels), ZiggyDataType.ZIGGY_STRING, false)); - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/DefaultWorkerResourcesRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/DefaultWorkerResourcesRequest.java deleted file mode 100644 index c865a76..0000000 --- a/src/main/java/gov/nasa/ziggy/services/messages/DefaultWorkerResourcesRequest.java +++ /dev/null @@ -1,11 +0,0 @@ -package gov.nasa.ziggy.services.messages; - -/** - * Request that the recipient return the default {@link WorkerResources} instance. - * - * @author PT - */ -public class DefaultWorkerResourcesRequest extends PipelineMessage { - - private static final long serialVersionUID = 20230714L; -} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java index a64cdd9..7e71779 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/EventHandlerToggleStateRequest.java @@ -5,8 +5,6 @@ import gov.nasa.ziggy.services.events.ZiggyEventHandler; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; /** * Sends a request to the supervisor process to toggle the state of a single @@ -27,7 +25,6 @@ public class EventHandlerToggleStateRequest extends PipelineMessage { * private, which in turn forces the user to verify privileges before the request is sent. */ public static void requestEventHandlerToggle(String handlerName) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); log.debug("Sending toggle request for event handler \"" + handlerName + "\""); ZiggyMessenger.publish(new EventHandlerToggleStateRequest(handlerName)); } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/HeartbeatCheckMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/HeartbeatCheckMessage.java new file mode 100644 index 0000000..c5e3284 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/HeartbeatCheckMessage.java @@ -0,0 +1,16 @@ +package gov.nasa.ziggy.services.messages; + +public class HeartbeatCheckMessage extends PipelineMessage { + + private static final long serialVersionUID = 20231126L; + + private final long heartbeatTime; + + public HeartbeatCheckMessage(long heartbeatTime) { + this.heartbeatTime = heartbeatTime; + } + + public long getHeartbeatTime() { + return heartbeatTime; + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/SingleTaskLogRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/SingleTaskLogRequest.java index 78f16a5..96e2cdd 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/SingleTaskLogRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/SingleTaskLogRequest.java @@ -2,8 +2,6 @@ import gov.nasa.ziggy.services.logging.TaskLogInformation; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; import gov.nasa.ziggy.util.Requestor; /** @@ -24,7 +22,6 @@ private SingleTaskLogRequest(Requestor sender, TaskLogInformation taskLogInforma public static void requestSingleTaskLog(Requestor sender, TaskLogInformation taskLogInformation) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); ZiggyMessenger.publish(new SingleTaskLogRequest(sender, taskLogInformation)); } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java index fd4329c..218ea28 100644 --- a/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java +++ b/src/main/java/gov/nasa/ziggy/services/messages/TaskLogInformationRequest.java @@ -5,8 +5,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.logging.TaskLogInformation; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; import gov.nasa.ziggy.util.Requestor; /** @@ -36,7 +34,6 @@ private TaskLogInformationRequest(Requestor sender, long instanceId, long taskId * method. */ public static void requestTaskLogInformation(Requestor sender, PipelineTask task) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); ZiggyMessenger.publish(new TaskLogInformationRequest(sender, task.getPipelineInstance().getId(), task.getId())); } diff --git a/src/main/java/gov/nasa/ziggy/services/messages/WorkerResources.java b/src/main/java/gov/nasa/ziggy/services/messages/WorkerResources.java deleted file mode 100644 index fc0b6ef..0000000 --- a/src/main/java/gov/nasa/ziggy/services/messages/WorkerResources.java +++ /dev/null @@ -1,66 +0,0 @@ -package gov.nasa.ziggy.services.messages; - -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.supervisor.PipelineSupervisor; -import gov.nasa.ziggy.ui.ZiggyGuiConsole; -import gov.nasa.ziggy.ui.util.HumanReadableHeapSize; - -/** - * Notifies the {@link ZiggyGuiConsole} of the default settings for worker count and Java heap size - * that are stored in the {@link PipelineSupervisor}. This information is also used internally to - * determine the correct worker count and heap size for a given {@link PipelineDefinitionNode}, - * based on whether the node has any non-default values set. - *

      - * The class contains a singleton instance of {@link WorkerResources} that holds the default values - * for the pipeline. If a given pipeline definition node has no values set for one or both - * parameters, the default values will be returned by the class getter methods. There are also - * boolean methods that indicate whether the default or per-node values are being returned. - * - * @author PT - */ -public class WorkerResources extends PipelineMessage { - - private static final long serialVersionUID = 20230714L; - - /** - * Singleton instance that specifies the default values for the resources. - */ - private static WorkerResources defaultResources; - - // Values are boxed so they can be null. - private Integer maxWorkerCount; - private Integer heapSizeMb; - - public WorkerResources(Integer maxWorkerCount, Integer heapSizeMb) { - this.maxWorkerCount = maxWorkerCount; - this.heapSizeMb = heapSizeMb; - } - - public int getMaxWorkerCount() { - return !maxWorkerCountIsDefault() ? maxWorkerCount : defaultResources.getMaxWorkerCount(); - } - - public int getHeapSizeMb() { - return !heapSizeIsDefault() ? heapSizeMb : defaultResources.getHeapSizeMb(); - } - - public HumanReadableHeapSize humanReadableHeapSize() { - return new HumanReadableHeapSize(heapSizeMb); - } - - public boolean heapSizeIsDefault() { - return heapSizeMb == null; - } - - public boolean maxWorkerCountIsDefault() { - return maxWorkerCount == null; - } - - public static void setDefaultResources(WorkerResources resources) { - defaultResources = resources; - } - - public static WorkerResources getDefaultResources() { - return defaultResources; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/WorkerResourcesMessage.java b/src/main/java/gov/nasa/ziggy/services/messages/WorkerResourcesMessage.java new file mode 100644 index 0000000..aaabbae --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/WorkerResourcesMessage.java @@ -0,0 +1,60 @@ +package gov.nasa.ziggy.services.messages; + +import gov.nasa.ziggy.supervisor.PipelineSupervisor; +import gov.nasa.ziggy.worker.WorkerResources; + +/** + * Sends information about worker resources in response to a {@link WorkerResourcesRequest}. + *

      + * Each instance of {@link WorkerResourcesMessage} contains two instances of + * {@link WorkerResources}: an instance for default values, and an instance for non-default values. + * When the {@link PipelineSupervisor} publishes this message, only the default resources are + * populated. All other publishers will populate only the non-default resources. Subscribers will + * typically want to use one or the other, and can determine whether a given message is "for them" + * by checking which {@link WorkerResources} instance is populated. + *

      + * Note that the {@link WorkerResourcesMessage} cannot transport a non-default + * {@link WorkerResources} instance if any of its resource values are null. + */ +public class WorkerResourcesMessage extends PipelineMessage { + + private static final long serialVersionUID = 20231204L; + + private final WorkerResources defaultResources; + private final WorkerResources resources; + + public WorkerResourcesMessage(WorkerResources defaultResources, WorkerResources resources) { + this.defaultResources = defaultResources; + this.resources = resources; + validate(); + } + + public void validate() { + if (defaultResources != null && resources != null) { + throw new IllegalArgumentException( + "default resources and resources cannot both be non-null"); + } + if (defaultResources == null && resources == null) { + throw new IllegalArgumentException( + "default resources and resources cannot both be null"); + } + if (defaultResources != null && (defaultResources.getHeapSizeMb() == null + || defaultResources.getHeapSizeMb() == 0 || defaultResources.getMaxWorkerCount() == null + || defaultResources.getMaxWorkerCount() == 0)) { + throw new IllegalArgumentException( + "Default resources must not contain any zero values or null values"); + } + if (resources != null + && (resources.getHeapSizeMb() == null || resources.getMaxWorkerCount() == null)) { + throw new IllegalArgumentException("Resources must not include null values"); + } + } + + public WorkerResources getDefaultResources() { + return defaultResources; + } + + public WorkerResources getResources() { + return resources; + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messages/WorkerResourcesRequest.java b/src/main/java/gov/nasa/ziggy/services/messages/WorkerResourcesRequest.java new file mode 100644 index 0000000..a3b8b52 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/services/messages/WorkerResourcesRequest.java @@ -0,0 +1,14 @@ +package gov.nasa.ziggy.services.messages; + +/** + * Request that the recipient return a {@link WorkerResourcesMessage} instance. + * + * @author PT + */ +public class WorkerResourcesRequest extends PipelineMessage { + + private static final long serialVersionUID = 20231204L; + + public WorkerResourcesRequest() { + } +} diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/ProcessHeartbeatManager.java b/src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java similarity index 50% rename from src/main/java/gov/nasa/ziggy/services/messaging/ProcessHeartbeatManager.java rename to src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java index 37b7cba..b6a99ce 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/ProcessHeartbeatManager.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/HeartbeatManager.java @@ -7,11 +7,8 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import gov.nasa.ziggy.services.messages.HeartbeatCheckMessage; import gov.nasa.ziggy.services.messages.HeartbeatMessage; -import gov.nasa.ziggy.ui.ClusterController; -import gov.nasa.ziggy.ui.status.Indicator; -import gov.nasa.ziggy.ui.status.ProcessesStatusPanel; -import gov.nasa.ziggy.ui.status.StatusPanel; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.SystemProxy; @@ -23,18 +20,16 @@ * to indicate that it has not crashed. *

      * At startup, a console puts the Processes summary into the "Gray" (undefined) state. When the - * {@link ZiggyRmiClient} is created, an instance of {@link ProcessHeartbeatManager} is also - * created; it waits for 1 heartbeat interval to hear from the supervisor. If the supervisor is - * heard from in that interval, the state of the Processes summary goes to "Green;" if not heard - * from, "Red." + * {@link ZiggyRmiClient} is created, an instance of {@link HeartbeatManager} is also created; it + * waits for 1 heartbeat interval to hear from the supervisor. If the supervisor is heard from in + * that interval, the state of the Processes summary goes to "Green;" if not heard from, "Red." *

      * Once the Processes summary is "Green," a {@link ScheduledThreadPoolExecutor} is started that * checks the timestamp of the latest heartbeat at an interval that is 2 * the heartbeat interval. - * If there have been no new heartbeats since the last one recorded by the - * {@link ProcessHeartbeatManager}, the Processes summary goes "Yellow," the {@link ZiggyRmiClient} - * instance is deleted, and a new instance is created; this is necessary because, if the supervisor - * has restarted, it needs new {@link ZiggyRmiClient} services from each process that uses - * {@link ZiggyRmiClient}. + * If there have been no new heartbeats since the last one recorded by the {@link HeartbeatManager}, + * the Processes summary goes "Yellow," the {@link ZiggyRmiClient} instance is deleted, and a new + * instance is created; this is necessary because, if the supervisor has restarted, it needs new + * {@link ZiggyRmiClient} services from each process that uses {@link ZiggyRmiClient}. *

      * If, after deleting and reconstructing the {@link ZiggyRmiClient}, the supervisor is still not * heard from, the Processes summary goes to "Red," and the heartbeat monitoring is terminated. At @@ -43,64 +38,52 @@ * * @author PT */ -public class ProcessHeartbeatManager { +public class HeartbeatManager { - private static final Logger log = LoggerFactory.getLogger(ProcessHeartbeatManager.class); + private static final Logger log = LoggerFactory.getLogger(HeartbeatManager.class); - private static ProcessHeartbeatManager instance; + private static HeartbeatManager instance; private static boolean initializeInThread = true; private long heartbeatIntervalMillis; private ScheduledThreadPoolExecutor heartbeatListener; private long priorHeartbeatTime; private long heartbeatTime; - protected HeartbeatManagerAssistant heartbeatManagerAssistant; - private boolean isInitialized = false; - private ClusterController clusterController; + private boolean started; private boolean reinitializeOnMissedHeartbeat = true; private CountDownLatch heartbeatCountdownLatch; - public ProcessHeartbeatManager(HeartbeatManagerAssistant heartbeatManagerAssistant) { - this(heartbeatManagerAssistant, new ClusterController(100, 1)); - } - - /** - * Constructor for test purposes. This allows the caller to supply a mocked instance of the - * class that performs all "external" methods (where in this case an external method is one we - * want to mock and/or detect the use of). - */ - protected ProcessHeartbeatManager(HeartbeatManagerAssistant heartbeatManagerAssistant, - ClusterController clusterController) { - this.heartbeatManagerAssistant = heartbeatManagerAssistant; + /** For testing only. Use static method startInstance() to start singleton. */ + HeartbeatManager() { heartbeatIntervalMillis = HeartbeatMessage.heartbeatIntervalMillis(); - this.clusterController = clusterController; ZiggyMessenger.subscribe(HeartbeatMessage.class, message -> { heartbeatTime = message.getHeartbeatTimeMillis(); if (heartbeatCountdownLatch != null) { heartbeatCountdownLatch.countDown(); } }); + heartbeatTime = -1L; } - public static void initializeInstance(HeartbeatManagerAssistant heartbeatManagerAssistant) { - if (isInitialized()) { + public static synchronized void startInstance() { + if (isInstanceStarted()) { log.info("ProcessHeartbeatManager instance already available, skipping instantiation"); } - instance = new ProcessHeartbeatManager(heartbeatManagerAssistant); - instance.initializeHeartbeatManager(); + instance = new HeartbeatManager(); + instance.start(); } - public static boolean isInitialized() { - return instance != null; + private static boolean isInstanceStarted() { + return instance != null && instance.started; } /** * Start checking for heartbeats. If they are detected, start the automated at-intervals * checking. */ - public void initializeHeartbeatManager() { + void start() { - if (isInitialized) { + if (started) { return; } // wait for one heartbeat interval to see if we get a heartbeat message. We @@ -138,44 +121,27 @@ private void initializeHeartbeatManagerInternal() { } finally { heartbeatCountdownLatch = null; } + ZiggyMessenger.publish(new HeartbeatCheckMessage(heartbeatTime), false); if (heartbeatTime <= 0) { - log.debug("Setting RMI state to error"); - setRmiIndicator(Indicator.State.ERROR); if (heartbeatListener != null) { heartbeatListener.shutdownNow(); } log.error("Unable to detect supervisor heartbeat messages"); return; } - log.debug("Setting RMI state to normal"); - setRmiIndicator(Indicator.State.NORMAL); priorHeartbeatTime = heartbeatTime; if (heartbeatListener == null) { startHeartbeatListener(); } - isInitialized = true; + started = true; log.debug("initializeHeartbeatManagerInternal: done"); } - /** - * Updates the state of the Processes idiot light. - */ - protected void setRmiIndicator(Indicator.State state) { - heartbeatManagerAssistant.setRmiIndicator(state); - } - - protected void setSupervisorIndicator(Indicator.State state) { - heartbeatManagerAssistant.setSupervisorIndicator(state); - } - - protected void setDatabaseIndicator(Indicator.State state) { - heartbeatManagerAssistant.setDatabaseIndicator(state); - } - /** * Start at-intervals checking for heartbeats, with a check interval that is 2 * the interval at * which the supervisor emits heartbeats. */ + @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) protected void startHeartbeatListener() { log.info("Starting heartbeat listener thread"); heartbeatListener = new ScheduledThreadPoolExecutor(1); @@ -191,6 +157,7 @@ protected void startHeartbeatListener() { // NB: I'm pretty certain this block could never be executed because if there's // a NoHeartbeatException thrown in checkForHeartbeat(), this thread will // immediately be shut down. + throw new AssertionError(e); } }, 2 * heartbeatIntervalMillis, 2 * heartbeatIntervalMillis, TimeUnit.MILLISECONDS); ZiggyShutdownHook.addShutdownHook(() -> { @@ -208,44 +175,28 @@ protected void checkForHeartbeat() throws NoHeartbeatException { if (heartbeatTime > priorHeartbeatTime) { priorHeartbeatTime = heartbeatTime; } else { - setRmiIndicator(Indicator.State.WARNING); priorHeartbeatTime = 0L; heartbeatTime = 0L; - restartClientCommunicator(); - isInitialized = false; + restartZiggyRmiClient(); + started = false; if (reinitializeOnMissedHeartbeat) { - initializeHeartbeatManager(); + start(); } } - if (clusterController.isDatabaseAvailable()) { - setDatabaseIndicator(Indicator.State.NORMAL); - } else { - setDatabaseIndicator(Indicator.State.ERROR); - } - if (clusterController.isSupervisorRunning()) { - setSupervisorIndicator(Indicator.State.NORMAL); - } else { - setSupervisorIndicator(Indicator.State.ERROR); - } + ZiggyMessenger.publish(new HeartbeatCheckMessage(heartbeatTime), false); } - public static synchronized void stopHeartbeatListener() { - if (isInitialized() && instance.getHeartbeatListener() != null) { - instance.getHeartbeatListener().shutdownNow(); - } - instance = null; + /** Broken out for unit tests. */ + void restartZiggyRmiClient() { + ZiggyRmiClient.restart(); } public static synchronized void resetHeartbeatTime() { - if (isInitialized()) { + if (isInstanceStarted()) { instance.heartbeatTime = 0L; } } - protected void restartClientCommunicator() { - heartbeatManagerAssistant.restartClientCommunicator(); - } - public ScheduledThreadPoolExecutor getHeartbeatListener() { return heartbeatListener; } @@ -258,11 +209,6 @@ public long getHeartbeatTime() { return heartbeatTime; } - /** For testing use only. */ - protected void setClusterController(ClusterController clusterController) { - this.clusterController = clusterController; - } - /** For testing use only. */ void setReinitializeOnMissedHeartbeat(boolean reinitialize) { reinitializeOnMissedHeartbeat = reinitialize; @@ -278,26 +224,6 @@ static void setInitializeInThread(boolean initInThread) { initializeInThread = initInThread; } - /** - * Defines methods that are used by the {@link ProcessHeartbeatManager} to control the console - * stoplights based on the state of heartbeat detection. These are only used by the console; for - * the worker, the implementation of this interface is all no-op methods. - * - * @author PT - */ - public interface HeartbeatManagerAssistant { - - void setRmiIndicator(Indicator.State state); - - void setSupervisorIndicator(Indicator.State state); - - void setDatabaseIndicator(Indicator.State state); - - default void restartClientCommunicator() { - ZiggyRmiClient.restart(); - } - } - public static class NoHeartbeatException extends RuntimeException { private static final long serialVersionUID = 20210310L; @@ -306,71 +232,4 @@ public NoHeartbeatException(String string) { super(string); } } - - public static class WorkerHeartbeatManagerAssistant implements HeartbeatManagerAssistant { - @Override - public void setRmiIndicator(Indicator.State state) { - } - - @Override - public void setSupervisorIndicator(Indicator.State state) { - } - - @Override - public void setDatabaseIndicator(Indicator.State state) { - } - } - - public static class ConsoleHeartbeatManagerAssistant implements HeartbeatManagerAssistant { - - static final String RMI_ERROR_MESSAGE = "Unable to establish communication with supervisor"; - static final String RMI_WARNING_MESSAGE = "Attempting to establish communication with supervisor"; - static final String SUPERVISOR_ERROR_MESSAGE = "Supervisor process has failed"; - static final String DATABASE_ERROR_MESSAGE = "Database process has failed"; - - @Override - public void setRmiIndicator(Indicator.State state) { - setIndicator(ProcessesStatusPanel.messagingIndicator(), state, RMI_WARNING_MESSAGE, - RMI_ERROR_MESSAGE); - } - - @Override - public void setSupervisorIndicator(Indicator.State state) { - setIndicator(ProcessesStatusPanel.supervisorIndicator(), state, null, - SUPERVISOR_ERROR_MESSAGE); - } - - @Override - public void setDatabaseIndicator(Indicator.State state) { - setIndicator(ProcessesStatusPanel.databaseIndicator(), state, null, - DATABASE_ERROR_MESSAGE); - } - - public void updateProcessesIndicator() { - StatusPanel.ContentItem.PROCESSES.menuItem() - .setState( - Indicator.summaryState(ProcessesStatusPanel.messagingIndicator(), - ProcessesStatusPanel.supervisorIndicator(), - ProcessesStatusPanel.databaseIndicator()), - Indicator.summaryToolTipText(ProcessesStatusPanel.messagingIndicator(), - ProcessesStatusPanel.supervisorIndicator(), - ProcessesStatusPanel.databaseIndicator())); - } - - private void setIndicator(Indicator indicator, Indicator.State state, String warningTooltip, - String errorTooltip) { - if (indicator == null) { - return; - } - String tooltip = null; - if (state == Indicator.State.WARNING) { - tooltip = warningTooltip; - } - if (state == Indicator.State.ERROR) { - tooltip = errorTooltip; - } - indicator.setState(state, tooltip); - updateProcessesIndicator(); - } - } } diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java index bfc7b13..f85eaed 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyMessenger.java @@ -53,7 +53,7 @@ public class ZiggyMessenger { * Blocking queue for outgoing messages. Messages wait here until the singleton instance is free * to deal with them, at which time they get sent from the RMI client to the RMI server. */ - private static LinkedBlockingQueue outgoingMessageQueue = new LinkedBlockingQueue<>(); + private static LinkedBlockingQueue outgoingMessageQueue = new LinkedBlockingQueue<>(); /** For testing only. */ static List messagesFromOutgoingQueue = new ArrayList<>(); @@ -97,9 +97,9 @@ private ZiggyMessenger() { outgoingMessageThread = new Thread(() -> { while (!Thread.currentThread().isInterrupted()) { try { - PipelineMessage message = outgoingMessageQueue.take(); + Message message = outgoingMessageQueue.take(); if (storeMessages) { - messagesFromOutgoingQueue.add(message); + messagesFromOutgoingQueue.add(message.getPipelineMessage()); } publishMessage(message); } catch (InterruptedException e) { @@ -111,17 +111,17 @@ private ZiggyMessenger() { outgoingMessageThread.start(); } - private void publishMessage(PipelineMessage message) { - CountDownLatch latch = messageCountdownLatches.get(message); - messageCountdownLatches.remove(message); - if (ZiggyRmiClient.isInitialized()) { - log.debug("Sending message of " + message.getClass().toString()); - ZiggyRmiClient.send(message, latch); + private void publishMessage(Message message) { + CountDownLatch latch = messageCountdownLatches.get(message.getPipelineMessage()); + messageCountdownLatches.remove(message.getPipelineMessage()); + if (ZiggyRmiClient.isInitialized() && message.isBroadcastOverRmi()) { + log.debug("Sending message {}", message); + ZiggyRmiClient.send(message.getPipelineMessage(), latch); } else { - takeAction(message); - if (latch != null) { - latch.countDown(); - } + takeAction(message.getPipelineMessage()); + } + if (latch != null) { + latch.countDown(); } } @@ -132,12 +132,12 @@ private void takeAction(T message) { return; } for (MessageAction action : actions) { - log.debug("Applying action for message " + message.getClass().toString()); + log.debug("Applying action for message {}", message); ((MessageAction) action).action(message); } } - /** For testing only. */ + /** For internal use and testing only. */ static void initializeInstance() { if (!isInitialized()) { log.info("Initializing ZiggyMessenger singleton"); @@ -191,15 +191,25 @@ private void removeSubscription(Class messageClas * Publishes a message via the {@link ZiggyMessenger} singleton. */ public static void publish(PipelineMessage message) { - publish(message, null); + publish(message, true, null); + } + + public static void publish(PipelineMessage message, boolean broadcastOverRmi) { + publish(message, broadcastOverRmi, null); + } + + public static void publish(PipelineMessage message, CountDownLatch latch) { + publish(message, true, latch); } /** * Publishes a message via the {@link ZiggyMessenger} singleton, and holds onto a - * {@link CountDownLatch} for the message. + * {@link CountDownLatch} for the message. The latch is quietly ignored if null. */ @AcceptableCatchBlock(rationale = Rationale.CLEANUP_BEFORE_EXIT) - public static void publish(PipelineMessage message, CountDownLatch latch) { + public static void publish(PipelineMessage message, boolean broadcastOverRmi, + CountDownLatch latch) { + if (!isInitialized()) { initializeInstance(); } @@ -207,7 +217,7 @@ public static void publish(PipelineMessage message, CountDownLatch latch) { if (latch != null) { instance.messageCountdownLatches.put(message, latch); } - outgoingMessageQueue.put(message); + outgoingMessageQueue.put(new Message(message, broadcastOverRmi)); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } @@ -223,7 +233,7 @@ static void actOnMessage(PipelineMessage message) { throw new PipelineException("Unable to act on message of " + message.getClass().toString() + " due to absence of ZiggyMessenger instance"); } - log.debug("Taking action on message of " + message.getClass().toString()); + log.debug("Taking action on message of {}", message); instance.takeAction(message); } @@ -240,7 +250,7 @@ static Map, List>> getSubscrip } /** For testing only. */ - static LinkedBlockingQueue getOutgoingMessageQueue() { + static LinkedBlockingQueue getOutgoingMessageQueue() { return ZiggyMessenger.outgoingMessageQueue; } @@ -248,4 +258,28 @@ static LinkedBlockingQueue getOutgoingMessageQueue() { static void setStoreMessages(boolean storeMessages) { instance.storeMessages = storeMessages; } + + private static class Message { + + private final PipelineMessage pipelineMessage; + private final boolean broadcastOverRmi; + + public Message(PipelineMessage pipelineMessage, boolean broadcastOverRmi) { + this.pipelineMessage = pipelineMessage; + this.broadcastOverRmi = broadcastOverRmi; + } + + public PipelineMessage getPipelineMessage() { + return pipelineMessage; + } + + public boolean isBroadcastOverRmi() { + return broadcastOverRmi; + } + + @Override + public String toString() { + return pipelineMessage.toString(); + } + } } diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java index 446fb3e..531c545 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiClient.java @@ -17,7 +17,7 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.services.messages.PipelineMessage; -import gov.nasa.ziggy.services.messaging.ProcessHeartbeatManager.NoHeartbeatException; +import gov.nasa.ziggy.services.messaging.HeartbeatManager.NoHeartbeatException; import gov.nasa.ziggy.supervisor.PipelineSupervisor; import gov.nasa.ziggy.ui.ZiggyConsole; import gov.nasa.ziggy.util.AcceptableCatchBlock; @@ -31,11 +31,11 @@ * messages to be sent to the server and arranges for the server to have broadcast access to the * clients. *

      - * A {@link ZiggyRmiClient} singleton is created is created via the {@link initializeInstance} - * method. The {@link PipelineSupervisor} has a client, as does every {@link PipelineWorker} and - * {@link ZiggyConsole}. The singleton will be destroyed and re-created via {@link restart} if + * A {@link ZiggyRmiClient} singleton is created via the {@link #start(String)} method. The + * {@link PipelineSupervisor} has a client, as does every {@link PipelineWorker} and + * {@link ZiggyConsole}. The singleton will be destroyed and re-created via {@link #restart()} if * contact is lost with the server. When the creating instance exits, the {@link ZiggyRmiClient} - * singleton will be destroyed via the {@link reset} method. + * singleton will be destroyed via the {@link #reset()} method. *

      * The {@link ZiggyRmiClient} is used by Ziggy console, supervisor, and worker processes. Each * instance is created with an appropriate collection of {@link MessageAction} instances that can @@ -49,6 +49,9 @@ public class ZiggyRmiClient implements ZiggyRmiClientService { private static final String RMI_REGISTRY_HOST = "localhost"; + private static final int REGISTRY_LOOKUP_EFFORTS = 25; + private static final int REGISTRY_LOOKUP_PAUSE_MILLIS = 200; + /** * Singleton instance of {@link ZiggyRmiClient} class. All threads in the UI process can access * this instance via the static methods. @@ -66,22 +69,37 @@ public class ZiggyRmiClient implements ZiggyRmiClientService { private final ZiggyRmiServerService ziggyRmiServerService; private final Registry registry; - private final int rmiPort; // Stores the type of client (worker, supervisor, console). private final String clientType; - private ZiggyRmiClient(int rmiPort, String clientType) - throws RemoteException, NotBoundException { + private ZiggyRmiClient(String clientType) throws RemoteException, NotBoundException { this.clientType = clientType; - this.rmiPort = rmiPort; log.info("Retrieving registry on {}", RMI_REGISTRY_HOST); - registry = LocateRegistry.getRegistry(RMI_REGISTRY_HOST, rmiPort); - - // get the stub that the server provided. - log.info("Looking up services in registry"); - ziggyRmiServerService = (ZiggyRmiServerService) registry - .lookup(ZiggyRmiServerService.SERVICE_NAME); + registry = LocateRegistry.getRegistry(RMI_REGISTRY_HOST, ZiggyRmiServer.rmiPort()); + + // Get the stub that the server provided. In case the server just started, try every 200 ms + // for five seconds. + ZiggyRmiServerService service = null; + for (int i = 0; i < REGISTRY_LOOKUP_EFFORTS; i++) { + log.info("Looking up services in registry (take {}/{})", i + 1, + REGISTRY_LOOKUP_EFFORTS); + try { + service = (ZiggyRmiServerService) registry + .lookup(ZiggyRmiServerService.SERVICE_NAME); + break; + } catch (RemoteException | NotBoundException e) { + if (i == REGISTRY_LOOKUP_EFFORTS - 1) { + throw e; + } + try { + Thread.sleep(REGISTRY_LOOKUP_PAUSE_MILLIS); + } catch (InterruptedException interrupt) { + Thread.currentThread().interrupt(); + } + } + } + ziggyRmiServerService = service; } /** @@ -94,16 +112,18 @@ private ZiggyRmiClient(int rmiPort, String clientType) */ @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public static synchronized void initializeInstance(int rmiPort, String clientType) { + public static synchronized void start(String clientType) { if (isInitialized()) { log.info("ZiggyRmiClient instance already available, skipping instantiation"); return; } + log.info("Starting ZiggyRmiClient instance with registry on port {}...", + ZiggyRmiServer.rmiPort()); try { log.info("Starting new ZiggyRmiClient instance"); - instance = new ZiggyRmiClient(rmiPort, clientType); + instance = new ZiggyRmiClient(clientType); // Construct a stub of this instance and add that to the server's // list of same. Note that we first need to unexport it if it was previously @@ -119,10 +139,10 @@ public static synchronized void initializeInstance(int rmiPort, String clientTyp .exportObject(instance, 0); log.info("Adding client service stub to server instance"); instance.ziggyRmiServerService.addClientStub(exportedClient); - log.info("ZiggyRmiClient construction complete"); + log.info("Starting ZiggyRmiClient instance with registry on port {}...done", + ZiggyRmiServer.rmiPort()); } catch (RemoteException | NotBoundException | NoHeartbeatException e) { - throw new PipelineException( - "Exception occurred when attempting to initialize UiCommunicator", e); + throw new PipelineException("Could not start RMI client", e); } } @@ -139,6 +159,10 @@ public static boolean isInitialized() { */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) static synchronized void send(PipelineMessage message, CountDownLatch latch) { + if (!isInitialized()) { + throw new IllegalStateException("ZiggyRmiClient isn't running"); + } + log.debug("Sending {}", message.getClass().getName()); try { instance.ziggyRmiServerService.transmitToServer(message); @@ -182,14 +206,18 @@ public String clientName() throws RemoteException { } public static Registry getRegistry() { - return instance.registry; - } + if (!isInitialized()) { + throw new IllegalStateException("ZiggyRmiClient isn't running"); + } - public static int getRmiPort() { - return instance.rmiPort; + return instance.registry; } public static String getClientType() { + if (!isInitialized()) { + throw new IllegalStateException("ZiggyRmiClient isn't running"); + } + return instance.clientType; } @@ -201,19 +229,23 @@ public static String getClientType() { */ public static synchronized void restart() { log.info("Attempting restart of ZiggyRmiClient instance"); - ProcessHeartbeatManager.resetHeartbeatTime(); - int rmiPort = ZiggyRmiClient.getRmiPort(); + HeartbeatManager.resetHeartbeatTime(); String clientType = ZiggyRmiClient.getClientType(); instance = null; - initializeInstance(rmiPort, clientType); + start(clientType); } public static synchronized void reset() { + log.debug("Resetting"); instance = null; } /** For testing only. */ static ZiggyRmiServerService ziggyRmiServerService() { + if (!isInitialized()) { + throw new IllegalStateException("ZiggyRmiClient isn't running"); + } + return instance.ziggyRmiServerService; } @@ -229,6 +261,10 @@ static void clearDetectedMessages() { /** For testing only. */ static void setUseMessenger(boolean useMessenger) { + if (!isInitialized()) { + throw new IllegalStateException("ZiggyRmiClient isn't running"); + } + instance.useMessenger = useMessenger; } } diff --git a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiServer.java b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiServer.java index 3ba8c2f..7cca536 100644 --- a/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiServer.java +++ b/src/main/java/gov/nasa/ziggy/services/messaging/ZiggyRmiServer.java @@ -20,6 +20,8 @@ import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.PipelineMessage; import gov.nasa.ziggy.supervisor.PipelineSupervisor; @@ -84,7 +86,7 @@ public class ZiggyRmiServer implements ZiggyRmiServerService { private LinkedBlockingQueue messageQueue = new LinkedBlockingQueue<>(); @AcceptableCatchBlock(rationale = Rationale.MUST_NOT_CRASH) - private ZiggyRmiServer(int rmiPort) throws RemoteException { + private ZiggyRmiServer() throws RemoteException { // Try to create the registry. If that fails due to ExportException, then this // is a new SupervisorCommunicator started on a system where the supervisor crashed and @@ -95,13 +97,13 @@ private ZiggyRmiServer(int rmiPort) throws RemoteException { Registry createdOrRetrievedRegistry = null; try { log.info("Creating RMI registry"); - createdOrRetrievedRegistry = LocateRegistry.createRegistry(rmiPort); + createdOrRetrievedRegistry = LocateRegistry.createRegistry(rmiPort()); constructorCreatedRegistry = true; } catch (ExportException ignored) { // This just means that the registry already exists, but there's no // way to know that other than trying to create it and failing. log.info("Retrieving registry on localhost"); - createdOrRetrievedRegistry = LocateRegistry.getRegistry("localhost", rmiPort); + createdOrRetrievedRegistry = LocateRegistry.getRegistry("localhost", rmiPort()); } registry = createdOrRetrievedRegistry; registryCreated = constructorCreatedRegistry; @@ -135,15 +137,18 @@ private ZiggyRmiServer(int rmiPort) throws RemoteException { * of {@link ZiggyRmiServerService}, to the RMI registry. */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) - public static void initializeInstance(int rmiPort) { + public static void start() { if (instance != null) { log.info("ZiggyRmiServer instance already available, skipping instantiation"); return; } + + log.info("Starting RMI communications server with registry on port {}", rmiPort()); + try { log.info("Starting new ZiggyRmiServer instance"); - ZiggyRmiServer serverInstance = new ZiggyRmiServer(rmiPort); + ZiggyRmiServer serverInstance = new ZiggyRmiServer(); log.info("Exporting and binding objects into registry"); ZiggyRmiServerService commStub = (ZiggyRmiServerService) UnicastRemoteObject @@ -156,7 +161,8 @@ public static void initializeInstance(int rmiPort) { log.info("SHUTDOWN: ZiggyRmiServer...done"); }); instance = serverInstance; - log.info("ZiggyRmiServer construction complete"); + log.info("Starting RMI communications server with registry on port {}...done", + rmiPort()); } catch (RemoteException e) { throw new PipelineException( "Exception occurred when attempting to initialize ZiggyRmiServer", e); @@ -393,6 +399,11 @@ static boolean isAllMessagingComplete() { return true; } + static int rmiPort() { + return ZiggyConfiguration.getInstance() + .getInt(PropertyName.SUPERVISOR_PORT.property(), ZiggyRmiServer.RMI_PORT_DEFAULT); + } + /** * Provides a separate {@link Thread} for each client to use for outgoing messages. This ensures * that if one client freezes or is waiting to time out, it will not block messages that go out diff --git a/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java b/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java index fa251fe..5bfd34c 100644 --- a/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java +++ b/src/main/java/gov/nasa/ziggy/services/process/ExternalProcess.java @@ -28,8 +28,8 @@ import gov.nasa.ziggy.services.logging.WriterLogOutputStream; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.StringUtils; import gov.nasa.ziggy.util.ZiggyShutdownHook; +import gov.nasa.ziggy.util.ZiggyStringUtils; import gov.nasa.ziggy.util.os.ProcessUtils; /** @@ -333,7 +333,7 @@ private LogOutputStream logOutputStream(boolean doLogging, boolean doWriting, Wr */ public List stdout() { if (outputLog != null) { - return StringUtils.breakStringAtLineTerminations(outputLog.toString()); + return ZiggyStringUtils.breakStringAtLineTerminations(outputLog.toString()); } return null; } @@ -345,8 +345,9 @@ public List stdout() { */ public List stdout(String... targetStrings) { if (outputLog != null) { - return StringUtils.stringsContainingTargets( - StringUtils.breakStringAtLineTerminations(outputLog.toString()), targetStrings); + return ZiggyStringUtils.stringsContainingTargets( + ZiggyStringUtils.breakStringAtLineTerminations(outputLog.toString()), + targetStrings); } return null; } @@ -356,7 +357,7 @@ public List stdout(String... targetStrings) { */ public List stderr() { if (errorLog != null) { - return StringUtils.breakStringAtLineTerminations(errorLog.toString()); + return ZiggyStringUtils.breakStringAtLineTerminations(errorLog.toString()); } return null; } @@ -368,8 +369,8 @@ public List stderr() { */ public List stderr(String... targetStrings) { if (errorLog != null) { - return StringUtils.stringsContainingTargets( - StringUtils.breakStringAtLineTerminations(errorLog.toString()), targetStrings); + return ZiggyStringUtils.stringsContainingTargets( + ZiggyStringUtils.breakStringAtLineTerminations(errorLog.toString()), targetStrings); } return null; } diff --git a/src/main/java/gov/nasa/ziggy/services/process/ExternalProcessUtils.java b/src/main/java/gov/nasa/ziggy/services/process/ExternalProcessUtils.java index 10e28b7..abc7162 100644 --- a/src/main/java/gov/nasa/ziggy/services/process/ExternalProcessUtils.java +++ b/src/main/java/gov/nasa/ziggy/services/process/ExternalProcessUtils.java @@ -13,8 +13,6 @@ */ public class ExternalProcessUtils { - private static final String SUPERVISOR_LOG_FILE_NAME = "supervisor.log"; - /** * The Log4j configuration file as a JVM argument. */ @@ -23,6 +21,15 @@ public static String log4jConfigString() { + DirectoryProperties.ziggyHomeDir() + "/etc/log4j2.xml"; } + /** + * The log file as a JVM argument. + * + * @see "etc/log4j2.xml" + */ + public static String ziggyLog(String logFile) { + return "-D" + PropertyName.ZIGGY_LOG_FILE + "=" + logFile; + } + /** * The Java library path as a JVM argument. */ @@ -45,20 +52,4 @@ private static String bareJavaLibraryPath() { return StringUtils.isEmpty(pipelineLibPath) ? DirectoryProperties.ziggyLibDir().toString() : pipelineLibPath + ":" + DirectoryProperties.ziggyLibDir().toString(); } - - /** - * The prefix to be used for the supervisor log file (i.e., it goes into logs/supervisor). - */ - public static String supervisorLogPrefix() { - return "-Dlog4j.logfile.prefix=" + DirectoryProperties.supervisorLogDir().toString(); - } - - /** - * The log file name used by the supervisor (hence also by the workers). - * - * @return - */ - public static String supervisorLogFilename() { - return DirectoryProperties.supervisorLogDir().resolve(SUPERVISOR_LOG_FILE_NAME).toString(); - } } diff --git a/src/main/java/gov/nasa/ziggy/services/security/Privilege.java b/src/main/java/gov/nasa/ziggy/services/security/Privilege.java deleted file mode 100644 index e3df39e..0000000 --- a/src/main/java/gov/nasa/ziggy/services/security/Privilege.java +++ /dev/null @@ -1,87 +0,0 @@ -package gov.nasa.ziggy.services.security; - -/** - * List of privileges. - * - * @author Todd Klaus - */ -public enum Privilege { - // PI privileges - - /** Launch and manage pipeline instances */ - PIPELINE_OPERATIONS("Pipeline Ops"), - - /** View pipeline instance/task data, parameters, cluster status */ - PIPELINE_MONITOR("Pipeline Mon"), - - /** - * Create and modify Pipeline configurations (pipelines, modules, parameter sets, triggers, - * etc.) - */ - PIPELINE_CONFIG("Pipeline Config"), - - /** Create and modify users, reset passwords */ - USER_ADMIN("User Admin"), - - // TODO Delete MR privileges? - - // MR privileges - - // Role: reports - MR_PERM_REPORT_ALERTS("mr.alerts"), - MR_PERM_REPORT_BAD_PIXELS("mr.bad-pixels"), - MR_PERM_REPORT_CONFIG_MAP("mr.config-map"), - MR_PERM_REPORT_DATA_COMPRESSION("mr.data-compression"), - MR_PERM_REPORT_DATA_GAP("mr.data-gap"), - MR_PERM_REPORT_DR_SUMMARY("mr.dr-summary"), - MR_PERM_REPORT_FC("mr.fc"), - MR_PERM_REPORT_GENERIC_REPORT("mr.generic-report"), - MR_PERM_REPORT_HUFFMAN_TABLES("mr.huffman-tables"), - MR_PERM_REPORT_PI_PROCESSING("mr.pipeline-processing"), - MR_PERM_REPORT_PI_INSTANCE_DETAIL("mr.pipeline-instance-detail"), - MR_PERM_REPORT_REQUANT_TABLES("mr.requantization-tables"), - MR_PERM_REPORT_TAD_CCD_MODULE_OUTPUT("mr.tad-ccd-module-output"), - MR_PERM_REPORT_TAD_SUMMARY("mr.tad-summary"), - - /* OpenEdit standard permissions. */ - - // Role: editors - MR_BLOG("oe.blog"), - MR_EDIT("oe.edit"), - MR_EDIT_FTPUPLOAD("oe.edit.ftpUpload"), - MR_EDIT_UPLOAD("oe.edit.upload"), - - // Role: intranet - MR_FILEMANAGER("oe.filemanager"), - MR_INTRANET("oe.intranet"), - - // Role: administrators - MR_ADMINISTRATION("oe.administration"), - MR_EDIT_APPROVES("oe.edit.approves"), - MR_EDIT_APPROVE_LEVEL1("oe.edit.approve.level1"), - MR_EDIT_APPROVE_LEVEL2("oe.edit.approve.level2"), - MR_EDIT_APPROVE_LEVEL3("oe.edit.approve.level3"), - MR_EDIT_DIRECTEDIT("oe.edit.directedit"), - MR_EDIT_DRAFTMODE("oe.edit.draftmode"), - MR_EDIT_EDITOR_ADVANCED("oe.edit.editor.advanced"), - MR_EDIT_EDITSLANGUAGES("oe.edit.editslanguages"), - MR_EDIT_LINKS("oe.edit.links"), - MR_EDIT_MANAGENOTIFICATIONS("oe.edit.managenotifications"), - MR_EDIT_NOTIFY("oe.edit.notify"), - MR_EDIT_RECENTEDITS("oe.edit.recentedits"), - MR_EDIT_SETTINGS_ADVANCED("oe.edit.settings.advanced"), - MR_EDIT_UPDATE("oe.edit.update"), - MR_ERROR_NOTIFY("oe.error.notify"), - MR_USERMANAGER("oe.usermanager"); - - private String displayName; - - Privilege(String displayName) { - this.displayName = displayName; - } - - @Override - public String toString() { - return displayName; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/security/Role.java b/src/main/java/gov/nasa/ziggy/services/security/Role.java deleted file mode 100644 index a9d6b5f..0000000 --- a/src/main/java/gov/nasa/ziggy/services/security/Role.java +++ /dev/null @@ -1,135 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import java.util.Date; -import java.util.LinkedList; -import java.util.List; -import java.util.Objects; - -import jakarta.persistence.ElementCollection; -import jakarta.persistence.Entity; -import jakarta.persistence.Id; -import jakarta.persistence.JoinTable; -import jakarta.persistence.ManyToOne; -import jakarta.persistence.Table; -import jakarta.persistence.Version; - -/** - * @author Todd Klaus - */ -@Entity -@Table(name = "ziggy_Role") -public class Role { - @Id - private String name; - private Date created; - - @ManyToOne - private User createdBy = null; - - @ElementCollection - @JoinTable(name = "ziggy_Role_privileges") - private List privileges = new LinkedList<>(); - - /** - * used by Hibernate to implement optimistic locking. Should prevent 2 different console users - * from clobbering each others changes - */ - @Version - private final int dirty = 0; - - /** - * Used only by the persistence layer - */ - Role() { - } - - public Role(String name) { - this(name, null); - } - - public Role(String name, User createdBy) { - this.name = name; - this.createdBy = createdBy; - created = new Date(System.currentTimeMillis()); - } - - public List getPrivileges() { - return privileges; - } - - public void setPrivileges(List privileges) { - this.privileges = privileges; - } - - public void addPrivilege(String privilege) { - if (!hasPrivilege(privilege)) { - privileges.add(privilege); - } - } - - public void addPrivileges(Role role) { - List privileges = role.getPrivileges(); - if (privileges != null) { - for (String privilege : privileges) { - addPrivilege(privilege); - } - } - } - - public boolean hasPrivilege(String privilege) { - return privileges.contains(privilege); - } - - public String getName() { - return name; - } - - public void setName(String roleName) { - name = roleName; - } - - public Date getCreated() { - return created; - } - - public void setCreated(Date created) { - this.created = created; - } - - public User getCreatedBy() { - return createdBy; - } - - public void setCreatedBy(User createdBy) { - this.createdBy = createdBy; - } - - public int getDirty() { - return dirty; - } - - @Override - public int hashCode() { - return name.hashCode(); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - final Role other = (Role) obj; - if (!Objects.equals(name, other.name)) { - return false; - } - return true; - } - - @Override - public String toString() { - return getName(); - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/security/SecurityOperations.java b/src/main/java/gov/nasa/ziggy/services/security/SecurityOperations.java deleted file mode 100644 index 68236c7..0000000 --- a/src/main/java/gov/nasa/ziggy/services/security/SecurityOperations.java +++ /dev/null @@ -1,39 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import gov.nasa.ziggy.services.database.DatabaseService; - -/** - * @author Todd Klaus - */ -public class SecurityOperations { - private UserCrud userCrud = null; - private User user; - - public SecurityOperations() { - userCrud = new UserCrud(); - } - - public SecurityOperations(DatabaseService databaseService) { - userCrud = new UserCrud(databaseService); - } - - public boolean validateLogin(String loginName) { - String name = loginName != null ? loginName : ""; - user = userCrud.retrieveUser(name); - return user != null; - } - - public boolean hasPrivilege(User user, String privilege) { - String privilegeName = privilege != null ? privilege : ""; - return user.hasPrivilege(privilegeName); - } - - /** - * Returns the user that was last validated with {@link #validateLogin}. - * - * @return the user object, or {@code null} if a user has not yet been validated - */ - public User getCurrentUser() { - return user; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/security/User.java b/src/main/java/gov/nasa/ziggy/services/security/User.java deleted file mode 100644 index 804bd1d..0000000 --- a/src/main/java/gov/nasa/ziggy/services/security/User.java +++ /dev/null @@ -1,169 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import java.util.ArrayList; -import java.util.Date; -import java.util.List; -import java.util.Objects; - -import jakarta.persistence.ElementCollection; -import jakarta.persistence.Entity; -import jakarta.persistence.Id; -import jakarta.persistence.JoinTable; -import jakarta.persistence.ManyToMany; -import jakarta.persistence.Table; -import jakarta.persistence.Version; - -/** - * A user object. - * - * @author Bill Wohler - * @author Todd Klaus - */ -@Entity -@Table(name = "ziggy_User") -public class User { - - @Id - private String loginName; - private String displayName; - private String email; - private String phone; - private Date created; - - @ManyToMany - @JoinTable(name = "ziggy_User_roles") - private List roles = new ArrayList<>(); - - @ElementCollection - @JoinTable(name = "ziggy_User_privileges") - private List privileges = new ArrayList<>(); - - /** - * used by Hibernate to implement optimistic locking. Should prevent 2 different console users - * from clobbering each others changes - */ - @Version - private final int dirty = 0; - - public User() { - this(null, null, null, null); - } - - public User(String loginName, String displayName, String email, String phone) { - this.loginName = loginName; - this.displayName = displayName; - this.email = email; - this.phone = phone; - created = new Date(System.currentTimeMillis()); - } - - public String getLoginName() { - return loginName; - } - - public void setLoginName(String loginName) { - this.loginName = loginName; - } - - public String getDisplayName() { - return displayName; - } - - public void setDisplayName(String displayName) { - this.displayName = displayName; - } - - public String getEmail() { - return email; - } - - public void setEmail(String email) { - this.email = email; - } - - public String getPhone() { - return phone; - } - - public void setPhone(String phone) { - this.phone = phone; - } - - public List getRoles() { - return roles; - } - - public void setRoles(List roles) { - this.roles = roles; - } - - public void addRole(Role role) { - roles.add(role); - } - - public List getPrivileges() { - return privileges; - } - - public void setPrivileges(List privileges) { - this.privileges = privileges; - } - - public void addPrivilege(String privilege) { - privileges.add(privilege); - } - - public boolean hasPrivilege(String privilege) { - // First check for user-level override. - if (privileges.contains(privilege)) { - return true; - } - - // Next check the user's roles. - for (Role role : roles) { - if (role.hasPrivilege(privilege)) { - return true; - } - } - - // No matches. - return false; - } - - public Date getCreated() { - return created; - } - - public void setCreated(Date created) { - this.created = created; - } - - public int getDirty() { - return dirty; - } - - @Override - public int hashCode() { - return loginName.hashCode(); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (obj == null || getClass() != obj.getClass()) { - return false; - } - final User other = (User) obj; - if (!Objects.equals(loginName, other.loginName)) { - return false; - } - return true; - } - - @Override - public String toString() { - return displayName; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/security/UserCrud.java b/src/main/java/gov/nasa/ziggy/services/security/UserCrud.java deleted file mode 100644 index a64e39a..0000000 --- a/src/main/java/gov/nasa/ziggy/services/security/UserCrud.java +++ /dev/null @@ -1,75 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import java.util.List; - -import org.hibernate.Hibernate; - -import gov.nasa.ziggy.crud.AbstractCrud; -import gov.nasa.ziggy.crud.ZiggyQuery; -import gov.nasa.ziggy.services.database.DatabaseService; - -/** - * This class provides CRUD methods for the security entities ({@link User} and {@link Role}). - * - * @author Todd Klaus - */ -public class UserCrud extends AbstractCrud { - public UserCrud() { - } - - public UserCrud(DatabaseService databaseService) { - super(databaseService); - } - - public void createUser(User user) { - persist(user); - } - - public List retrieveAllUsers() { - - List results = list(createZiggyQuery(User.class)); - for (User user : results) { - Hibernate.initialize(user.getPrivileges()); - Hibernate.initialize(user.getRoles()); - } - return results; - } - - public User retrieveUser(String loginName) { - ZiggyQuery query = createZiggyQuery(User.class); - query.column(User_.loginName).in(loginName); - User user = uniqueResult(query); - if (user != null) { - Hibernate.initialize(user.getPrivileges()); - Hibernate.initialize(user.getRoles()); - } - return user; - } - - public void deleteUser(User user) { - remove(user); - } - - public void createRole(Role role) { - persist(role); - } - - public List retrieveAllRoles() { - return list(createZiggyQuery(Role.class)); - } - - public Role retrieveRole(String roleName) { - ZiggyQuery query = createZiggyQuery(Role.class); - query.column(Role_.name).in(roleName); - return uniqueResult(query); - } - - public void deleteRole(Role role) { - remove(role); - } - - @Override - public Class componentClass() { - return User.class; - } -} diff --git a/src/main/java/gov/nasa/ziggy/services/security/package.html b/src/main/java/gov/nasa/ziggy/services/security/package.html deleted file mode 100644 index e86221a..0000000 --- a/src/main/java/gov/nasa/ziggy/services/security/package.html +++ /dev/null @@ -1,28 +0,0 @@ - - -

      - The first sentence is used as the brief description of the - package and appears at the top of the package documentation. - Additional sentences will appear at the bottom in the - Description section. -

      -
      -
      Author
      -
      Bill Wohler
      -
      PT
      -
      - -

      Headings

      - -

      - Additional headings should use h2. -

      - -

      Sub-headings

      - -

      - Sub-headings should use h3 and so on. -

      - - - diff --git a/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java b/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java index 662c23b..820434a 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/PipelineInstanceManager.java @@ -2,8 +2,6 @@ import static gov.nasa.ziggy.services.database.DatabaseTransactionFactory.performTransaction; -import java.util.ArrayList; - import org.hibernate.Hibernate; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -62,9 +60,14 @@ public class PipelineInstanceManager { /** * Provided with package scope to facilitate testing. */ + // TODO Unused? Delete? PipelineInstanceManager() { } + public PipelineInstanceManager(FireTriggerRequest triggerRequest) { + initialize(triggerRequest); + } + /** * Provided with package scope to facilitate testing. */ @@ -106,46 +109,50 @@ void initialize(FireTriggerRequest triggerRequest) { }); } - public PipelineInstanceManager(FireTriggerRequest triggerRequest) { - initialize(triggerRequest); - } - /** * Fires the trigger for the given pipeline. If repetitions are requested, they are also managed * by this method. */ @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public void fireTrigger() { - // loop over repeats + while (repeats < maxRepeats) { - StringBuilder currentInstanceName = new StringBuilder().append(instanceName); + + // Append count/total if repeats in use. Use "-" to represent "forever" more briefly + // than 2147483647. + StringBuilder currentInstanceName = new StringBuilder( + instanceName != null ? instanceName : ""); if (maxRepeats > 1) { - currentInstanceName.append(":-").append(Integer.toString(repeats)); + if (!currentInstanceName.isEmpty()) { + currentInstanceName.append(" "); + } + currentInstanceName.append(repeats + 1).append("/"); + if (maxRepeats == Integer.MAX_VALUE) { + currentInstanceName.append("-"); + } else { + currentInstanceName.append(maxRepeats); + } } - final String finalCurrentInstanceName = currentInstanceName.toString(); - new ArrayList<>(); + PipelineInstance pipelineInstance = pipelineOperations().fireTrigger(pipeline, - finalCurrentInstanceName, startNode, endNode, null); + currentInstanceName.toString(), startNode, endNode, null); currentInstanceId = pipelineInstance.getId(); - // if we're not on the last repeat, we need to start the waiting - if (repeats < maxRepeats - 1) { + // If we're not on the last repeat, we need to wait. + // While the counter is 0-based, messages to the user are 1-based. + if (repeats++ < maxRepeats - 1) { try { if (!waitAndCheckStatus()) { throw new ModuleFatalProcessingException( - "Unable to start pipeline repeat " + (repeats + 1) - + " due to errored status of pipeline repeat " + repeats); + "Unable to start pipeline repeat " + repeats + + " due to errored status of pipeline repeat " + (repeats - 1)); } - repeats++; } catch (InterruptedException e) { throw new ModuleFatalProcessingException( - "Unable to start pipeline repeat " + (repeats + 1) - + " due to InterruptedException during pipeline repeat" + repeats, + "Unable to start pipeline repeat " + repeats + + " due to InterruptedException during pipeline repeat" + (repeats - 1), e); } - // If we're on the last repeat, we can exit the loop by incrementing repeats. - } else { - repeats++; } } } diff --git a/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java b/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java index 8f9fa59..1fd5d4c 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/PipelineSupervisor.java @@ -22,7 +22,6 @@ import java.util.stream.Collectors; import org.apache.commons.collections4.CollectionUtils; -import org.apache.commons.configuration2.ImmutableConfiguration; import org.apache.commons.exec.CommandLine; import org.apache.commons.io.FileUtils; import org.slf4j.Logger; @@ -49,7 +48,6 @@ import gov.nasa.ziggy.services.events.ZiggyEventHandler; import gov.nasa.ziggy.services.events.ZiggyEventHandler.ZiggyEventHandlerInfoForDisplay; import gov.nasa.ziggy.services.logging.TaskLog; -import gov.nasa.ziggy.services.messages.DefaultWorkerResourcesRequest; import gov.nasa.ziggy.services.messages.EventHandlerRequest; import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.KillTasksRequest; @@ -60,7 +58,8 @@ import gov.nasa.ziggy.services.messages.StartMemdroneRequest; import gov.nasa.ziggy.services.messages.TaskLogInformationMessage; import gov.nasa.ziggy.services.messages.TaskLogInformationRequest; -import gov.nasa.ziggy.services.messages.WorkerResources; +import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; +import gov.nasa.ziggy.services.messages.WorkerResourcesRequest; import gov.nasa.ziggy.services.messages.ZiggyEventHandlerInfoMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; @@ -71,6 +70,7 @@ import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; import gov.nasa.ziggy.util.WrapperUtils.WrapperCommand; import gov.nasa.ziggy.util.ZiggyShutdownHook; +import gov.nasa.ziggy.worker.WorkerResources; import hdf.hdf5lib.H5; /** @@ -87,6 +87,7 @@ public class PipelineSupervisor extends AbstractPipelineProcess { public static final String NAME = "Supervisor"; private static final String SUPERVISOR_BIN_NAME = "supervisor"; + private static final String SUPERVISOR_LOG_FILE_NAME = "supervisor.log"; public static final int WORKER_STATUS_REPORT_INTERVAL_MILLIS_DEFAULT = 15000; private static final Set ziggyEventHandlers = ConcurrentHashMap.newKeySet(); @@ -95,6 +96,8 @@ public class PipelineSupervisor extends AbstractPipelineProcess { // a PipelineWorker, or the delete command of a remote system's batch scheduler. private static final Set killedTaskIds = ConcurrentHashMap.newKeySet(); + private static WorkerResources defaultResources; + private ScheduledExecutorService heartbeatExecutor; private QueueCommandManager queueCommandManager; @@ -102,7 +105,7 @@ public PipelineSupervisor(int workerCount, int workerHeapSize) { super(NAME); checkArgument(workerCount > 0, "Worker count must be positive"); checkArgument(workerHeapSize > 0, "Worker heap size must be positive"); - WorkerResources.setDefaultResources(new WorkerResources(workerCount, workerHeapSize)); + defaultResources = new WorkerResources(workerCount, workerHeapSize); log.debug("Starting pipeline supervisor with " + workerCount + " workers and " + workerHeapSize + " MB max heap"); } @@ -116,9 +119,6 @@ public PipelineSupervisor(boolean messaging, boolean database) { public void initialize() { try { super.initialize(); - - ImmutableConfiguration config = ZiggyConfiguration.getInstance(); - clearStaleTaskStates(); // if HDF5 is to be used as the default binfile format (or indeed at all), @@ -137,19 +137,14 @@ public void initialize() { TriggerRequestManager.start(); log.info("Subscribing to messages...done"); - int rmiPort = config.getInt(PropertyName.SUPERVISOR_PORT.property(), - ZiggyRmiServer.RMI_PORT_DEFAULT); - log.info("Starting RMI communications server with registry on port " + rmiPort); - ZiggyRmiServer.initializeInstance(rmiPort); + ZiggyRmiServer.start(); - log.info("Starting ZiggyRmiClient instance with registry on port " + rmiPort + "..."); - ZiggyRmiClient.initializeInstance(rmiPort, NAME); + ZiggyRmiClient.start(NAME); ZiggyShutdownHook.addShutdownHook(() -> { log.info("Sending shutdown notification"); ZiggyRmiServer.shutdown(); ZiggyRmiClient.reset(); }); - log.info("Starting ZiggyRmiClient instance ... done"); // Start the heartbeat messages log.info("Starting supervisor-client heartbeat generator"); @@ -249,8 +244,8 @@ private void subscribe() { ZiggyMessenger.subscribe(SingleTaskLogRequest.class, message -> { ZiggyMessenger.publish(new SingleTaskLogMessage(message, taskLogContents(message))); }); - ZiggyMessenger.subscribe(DefaultWorkerResourcesRequest.class, message -> { - ZiggyMessenger.publish(WorkerResources.getDefaultResources()); + ZiggyMessenger.subscribe(WorkerResourcesRequest.class, message -> { + ZiggyMessenger.publish(new WorkerResourcesMessage(defaultResources, null)); }); ZiggyMessenger.subscribe(KillTasksRequest.class, message -> { killRemoteTasks(message); @@ -341,16 +336,18 @@ public static CommandLine supervisorCommand(WrapperCommand cmd, int workerCount, if (cmd == WrapperCommand.START) { // Refer to supervisor.wrapper.conf for appropriate indices for the parameters specified // here. - String ziggyLibDir = DirectoryProperties.ziggyLibDir().toString(); commandLine - .addArgument(wrapperParameter(WRAPPER_LOG_FILE_PROP_NAME, - ExternalProcessUtils.supervisorLogFilename())) + .addArgument(wrapperParameter(WRAPPER_LOG_FILE_PROP_NAME, supervisorLogFilename())) .addArgument(wrapperParameter(WRAPPER_CLASSPATH_PROP_NAME_PREFIX, 1, DirectoryProperties.ziggyHomeDir().resolve("libs").resolve("*.jar").toString())) - .addArgument( - wrapperParameter(WRAPPER_LIBRARY_PATH_PROP_NAME_PREFIX, 1, ziggyLibDir)) + .addArgument(wrapperParameter(WRAPPER_LIBRARY_PATH_PROP_NAME_PREFIX, 1, + DirectoryProperties.ziggyLibDir().toString())) + .addArgument(wrapperParameter(WRAPPER_JAVA_ADDITIONAL_PROP_NAME_PREFIX, 3, + ExternalProcessUtils.log4jConfigString())) + .addArgument(wrapperParameter(WRAPPER_JAVA_ADDITIONAL_PROP_NAME_PREFIX, 4, + ExternalProcessUtils.ziggyLog(supervisorLogFilename()))) .addArgument(wrapperParameter(WRAPPER_JAVA_ADDITIONAL_PROP_NAME_PREFIX, 5, - "-Djna.library.path=" + ziggyLibDir)) + ExternalProcessUtils.jnaLibraryPath())) .addArgument(wrapperParameter(WRAPPER_APP_PARAMETER_PROP_NAME_PREFIX, 2, Integer.toString(workerCount))) .addArgument(wrapperParameter(WRAPPER_APP_PARAMETER_PROP_NAME_PREFIX, 3, @@ -373,6 +370,13 @@ public static CommandLine supervisorCommand(WrapperCommand cmd, int workerCount, return commandLine; } + /** + * The log file name used by the supervisor. + */ + public static String supervisorLogFilename() { + return DirectoryProperties.supervisorLogDir().resolve(SUPERVISOR_LOG_FILE_NAME).toString(); + } + public static Set ziggyEventHandlers() { return ziggyEventHandlers; } @@ -391,6 +395,10 @@ public static void removeTaskFromKilledTaskList(long taskId) { killedTaskIds.remove(taskId); } + public static WorkerResources defaultResources() { + return defaultResources; + } + // Package scoped for testing purposes. Collection remoteTaskStateFiles() { return AlgorithmMonitor.remoteTaskStateFiles(); diff --git a/src/main/java/gov/nasa/ziggy/supervisor/TaskFileCopy.java b/src/main/java/gov/nasa/ziggy/supervisor/TaskFileCopy.java index 4200742..c59a1ea 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/TaskFileCopy.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/TaskFileCopy.java @@ -17,7 +17,7 @@ import gov.nasa.ziggy.metrics.ValueMetric; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.module.PipelineMetrics; -import gov.nasa.ziggy.module.WorkingDirManager; +import gov.nasa.ziggy.module.TaskDirectoryManager; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -42,7 +42,7 @@ public TaskFileCopy(PipelineTask pipelineTask, TaskFileCopyParameters copyParams @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) public void copyTaskFiles() { - File srcTaskDir = WorkingDirManager.workingDir(pipelineTask); + File srcTaskDir = new TaskDirectoryManager(pipelineTask).taskDir().toFile(); try { if (copyParams.isDeleteWithoutCopy()) { log.warn("*** TEST USE ONLY ***: deleting source directory without copying"); diff --git a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java index afaa9df..3b6f40a 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandler.java @@ -190,7 +190,6 @@ private void processTaskRequest(TaskRequest taskRequest) { }); transitionToNextInstanceNode(taskRequest.getInstanceNodeId()); - } /** diff --git a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java index 0d69295..f41fe0a 100644 --- a/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java +++ b/src/main/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManager.java @@ -15,15 +15,18 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.PipelineOperations; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; import gov.nasa.ziggy.services.messages.KillTasksRequest; import gov.nasa.ziggy.services.messages.KilledTaskMessage; import gov.nasa.ziggy.services.messages.TaskRequest; -import gov.nasa.ziggy.services.messages.WorkerResources; +import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; +import gov.nasa.ziggy.services.messages.WorkerResourcesRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.util.ZiggyShutdownHook; +import gov.nasa.ziggy.worker.WorkerResources; /** * Manages instances of {@link TaskRequestHandler} throughout their lifecycle. @@ -51,11 +54,15 @@ public class TaskRequestHandlerLifecycleManager extends Thread { private CountDownLatch taskRequestThreadCountdownLatch; private final PriorityBlockingQueue taskRequestQueue = new PriorityBlockingQueue<>(); - // Use a boxed instance so it can be null. + // Use a wrapper class so it can be null. private Long pipelineDefinitionNodeId; private ExecutorService taskRequestThreadPool; + // The current actual resources available to the current node, including defaults if + // appropriate. + private WorkerResources workerResources = new WorkerResources(0, 0); + // For testing purposes, the TaskRequestDispatcher can optionally store all of the // task request handlers it creates, organized by thread pool instance. private List> taskRequestHandlers = new ArrayList<>(); @@ -77,6 +84,13 @@ protected TaskRequestHandlerLifecycleManager(boolean storeTaskRequestHandlers) { ZiggyMessenger.subscribe(KillTasksRequest.class, message -> { killQueuedTasksAction(message); }); + + // Subscribe to requests for the current worker resources. This allows the + // console to find out what's currently running in the event that the console + // starts up when a pipeline is already executing. + ZiggyMessenger.subscribe(WorkerResourcesRequest.class, message -> { + ZiggyMessenger.publish(new WorkerResourcesMessage(null, workerResources)); + }); } /** @@ -189,10 +203,12 @@ private void startLifecycle() { initialRequest.getPipelineDefinitionNodeId()); pipelineDefinitionNodeId = initialRequest.getPipelineDefinitionNodeId(); - WorkerResources workerResources = workerResources(initialRequest); + workerResources = workerResources(initialRequest); int workerCount = workerResources.getMaxWorkerCount(); int heapSizeMb = workerResources.getHeapSizeMb(); + ZiggyMessenger.publish(new WorkerResourcesMessage(null, workerResources)); + // Put the task back onto the queue. taskRequestQueue.put(initialRequest); @@ -201,7 +217,8 @@ private void startLifecycle() { if (workerCount < 1) { throw new PipelineException("worker count < 1"); } - log.info("Starting {} workers", workerCount); + log.info("Starting {} workers with total heap size {}", workerCount, + workerResources.humanReadableHeapSize().toString()); taskRequestThreadPool = Executors.newFixedThreadPool(workerCount); List handlers = new ArrayList<>(); for (int i = 0; i < workerCount; i++) { @@ -238,11 +255,23 @@ public void shutdown() { taskRequestThreadPool = null; } + /** + * Gets the actual resources for the current pipeline definition node, including default values + * as appropriate. + */ WorkerResources workerResources(TaskRequest taskRequest) { - return (WorkerResources) DatabaseTransactionFactory - .performTransaction(() -> new PipelineTaskCrud().retrieve(taskRequest.getTaskId()) - .getPipelineDefinitionNode() + WorkerResources databaseResources = (WorkerResources) DatabaseTransactionFactory + .performTransaction(() -> new PipelineDefinitionNodeCrud() + .retrieveExecutionResources(new PipelineTaskCrud().retrieve(taskRequest.getTaskId()) + .pipelineDefinitionNode()) .workerResources()); + Integer compositeWorkerCount = databaseResources.getMaxWorkerCount() != null + ? databaseResources.getMaxWorkerCount() + : PipelineSupervisor.defaultResources().getMaxWorkerCount(); + Integer compositeHeapSizeMb = databaseResources.getHeapSizeMb() != null + ? databaseResources.getHeapSizeMb() + : PipelineSupervisor.defaultResources().getHeapSizeMb(); + return new WorkerResources(compositeWorkerCount, compositeHeapSizeMb); } /** For testing only. */ diff --git a/src/main/java/gov/nasa/ziggy/ui/ClusterController.java b/src/main/java/gov/nasa/ziggy/ui/ClusterController.java index 5d76ecd..128d167 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ClusterController.java +++ b/src/main/java/gov/nasa/ziggy/ui/ClusterController.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -58,7 +58,7 @@ import com.google.common.collect.ImmutableList; -import gov.nasa.ziggy.data.management.DataFileTypeImporter; +import gov.nasa.ziggy.data.datastore.DatastoreConfigurationImporter; import gov.nasa.ziggy.data.management.DataReceiptPipelineModule; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.parameters.ParameterLibraryImportExportCli.ParamIoMode; @@ -76,7 +76,6 @@ import gov.nasa.ziggy.services.messages.ShutdownMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; -import gov.nasa.ziggy.services.messaging.ZiggyRmiServer; import gov.nasa.ziggy.services.process.ExternalProcess; import gov.nasa.ziggy.util.WrapperUtils.WrapperCommand; import gov.nasa.ziggy.util.io.FileUtil; @@ -192,13 +191,16 @@ public ClusterController(int workerHeapSize, int workerCount) { private int cpuCount() { int availableProcessors = Runtime.getRuntime().availableProcessors(); - log.info("Setting number of worker threads to number of available processors (" - + availableProcessors + ")"); + log.info("Setting number of worker threads to number of available processors ({})", + availableProcessors); return availableProcessors; } public static void main(String[] args) { + log.info("Ziggy cluster (version {})", + ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + // Define all the command options. Options options = new Options() .addOption(Option.builder("f") @@ -296,7 +298,7 @@ public static void main(String[] args) { } if (commands.contains(VERSION_COMMAND)) { - log.info( + System.out.println( ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); } } @@ -370,8 +372,8 @@ public Void transaction() throws Exception { + pipelineDefsDir.toString() + " does not exist"); } - log.info( - "Importing parameter libraries from directory " + pipelineDefsDir.toString()); + log.info("Importing parameter libraries from directory {}", + pipelineDefsDir.toString()); File[] parameterFiles = pipelineDefsDir.toFile() .listFiles( (FilenameFilter) (dir, name) -> (name.startsWith(PARAM_LIBRARY_PREFIX) @@ -379,24 +381,27 @@ public Void transaction() throws Exception { Arrays.sort(parameterFiles, Comparator.comparing(File::getName)); ParametersOperations paramsOps = new ParametersOperations(); for (File parameterFile : parameterFiles) { - log.info("Importing library " + parameterFile.getName()); + log.info("Importing library {}", parameterFile.getName()); paramsOps.importParameterLibrary(parameterFile, null, ParamIoMode.STANDARD); } - log.info("Importing data file types from directory " + pipelineDefsDir.toString()); + log.info("Importing datastore configuration from directory " + + pipelineDefsDir.toString()); File[] dataTypeFiles = pipelineDefsDir.toFile() .listFiles((FilenameFilter) (dir, name) -> (name.startsWith(TYPE_FILE_PREFIX) && name.endsWith(XML_SUFFIX))); Arrays.sort(dataTypeFiles, Comparator.comparing(File::getName)); List dataTypeFileNames = new ArrayList<>(dataTypeFiles.length); for (File dataTypeFile : dataTypeFiles) { - log.info("Adding " + dataTypeFile.getName() + " to imports list"); + log.info("Adding {} to imports list", dataTypeFile.getName()); dataTypeFileNames.add(dataTypeFile.getAbsolutePath()); } - new DataFileTypeImporter(dataTypeFileNames, false).importFromFiles(); + DatastoreConfigurationImporter importer = new DatastoreConfigurationImporter( + dataTypeFileNames, false); + importer.importConfiguration(); - log.info( - "Importing pipeline definitions from directory " + pipelineDefsDir.toString()); + log.info("Importing pipeline definitions from directory {}", + pipelineDefsDir.toString()); File[] pipelineDefinitionFiles = pipelineDefsDir.toFile() .listFiles( (FilenameFilter) (dir, name) -> (name.startsWith(PIPELINE_DEF_FILE_PREFIX) @@ -404,13 +409,13 @@ public Void transaction() throws Exception { Arrays.sort(pipelineDefinitionFiles, Comparator.comparing(File::getName)); List pipelineDefFileList = new ArrayList<>(); for (File pipelineDefinitionFile : pipelineDefinitionFiles) { - log.info("Adding " + pipelineDefinitionFile.getName() + " to imports list"); + log.info("Adding {} to imports list", pipelineDefinitionFile.getName()); pipelineDefFileList.add(pipelineDefinitionFile); } new PipelineDefinitionOperations().importPipelineConfiguration(pipelineDefFileList); - log.info( - "Importing event definitions from directory " + pipelineDefsDir.toString()); + log.info("Importing event definitions from directory {}", + pipelineDefsDir.toString()); File[] handlerDefinitionFiles = pipelineDefsDir.toFile() .listFiles((FilenameFilter) (dir, name) -> (name.startsWith(EVENT_HANDLER_DEF_FILE_PREFIX) @@ -429,6 +434,7 @@ public void finallyBlock() { } }); log.info("Database initialization and creation complete"); + System.out.println("Cluster initialized"); } /** @@ -475,7 +481,7 @@ public int databaseStatus() { public boolean isSupervisorRunning() { CommandLine supervisorStatusCommand = supervisorCommand(WrapperCommand.STATUS, workerCount, workerHeapSize); - log.debug("Command line: " + supervisorStatusCommand); + log.debug("Command line: {}", supervisorStatusCommand); return ExternalProcess.simpleExternalProcess(supervisorStatusCommand).execute() == 0; } @@ -488,7 +494,7 @@ private void startCluster(boolean force) { if (!force) { throw new PipelineException("Cannot start cluster; cluster not initialized"); } - log.warn("Attempting to start uninitialized cluster"); + log.error("Attempting to start uninitialized cluster"); } try { @@ -514,17 +520,18 @@ private void startCluster(boolean force) { log.info("Supervisor already running"); } else { log.info("Starting supervisor"); - log.debug("Creating directory " + DirectoryProperties.supervisorLogDir()); + log.debug("Creating directory {}", DirectoryProperties.supervisorLogDir()); Files.createDirectories(DirectoryProperties.supervisorLogDir()); CommandLine supervisorStartCommand = supervisorCommand(WrapperCommand.START, workerCount, workerHeapSize); - log.debug("Command line: " + supervisorStartCommand.toString()); + log.debug("Command line: {}", supervisorStartCommand.toString()); ExternalProcess.simpleExternalProcess(supervisorStartCommand) .exceptionOnFailure() .execute(); log.info("Supervisor started"); } log.info("Cluster started"); + System.out.println("Cluster started"); } catch (Throwable t) { log.error("Caught exception when trying to start cluster, shutting down", t); stopCluster(); @@ -534,12 +541,8 @@ private void startCluster(boolean force) { private void stopCluster() { // Start RMI in order to publish the shutdown message. - int rmiPort = ZiggyConfiguration.getInstance() - .getInt(PropertyName.SUPERVISOR_PORT.property(), ZiggyRmiServer.RMI_PORT_DEFAULT); - log.info("Starting ZiggyRmiClient instance with registry on port " + rmiPort); try { - ZiggyRmiClient.initializeInstance(rmiPort, NAME); - log.info("Starting ZiggyRmiClient instance...done"); + ZiggyRmiClient.start(NAME); ZiggyMessenger.publish(new ShutdownMessage()); } catch (PipelineException e) { log.info("Starting ZiggyRmiClient instance...(no server to talk to)"); @@ -548,7 +551,7 @@ private void stopCluster() { log.info("Supervisor stopping"); CommandLine supervisorStopCommand = supervisorCommand(WrapperCommand.STOP, workerCount, workerHeapSize); - log.debug("Command line: " + supervisorStopCommand.toString()); + log.debug("Command line: {}", supervisorStopCommand.toString()); ExternalProcess.simpleExternalProcess(supervisorStopCommand).execute(true); if (!isSupervisorRunning()) { log.info("Supervisor stopped"); @@ -570,6 +573,7 @@ private void stopCluster() { } ZiggyRmiClient.reset(); log.info("Cluster stopped"); + System.out.println("Cluster stopped"); // Force exit due to the RMI client. System.exit(0); @@ -577,13 +581,13 @@ private void stopCluster() { private void status() { int databaseStatus = databaseStatus(); - log.info("Cluster is " + (isInitialized() ? "initialized" : "NOT initialized")); - log.info("Supervisor is " + (isSupervisorRunning() ? "running" : "NOT running")); - log.info("Database " + System.out.println("Cluster is " + (isInitialized() ? "initialized" : "NOT initialized")); + System.out.println("Supervisor is " + (isSupervisorRunning() ? "running" : "NOT running")); + System.out.println("Database " + (databaseStatus == 0 ? "is" : databaseStatus == DatabaseController.NOT_SUPPORTED ? "should be" : "is NOT") + " available"); - log.info("Cluster is " + System.out.println("Cluster is " + (isInitialized() && isDatabaseAvailable() && isSupervisorRunning() ? "running" : "NOT running")); } @@ -596,22 +600,10 @@ private void startPipelineConsole() { String consoleCommand = DirectoryProperties.ziggyBinDir() .resolve(ZIGGY_CONSOLE_COMMAND) .toString(); - log.debug("Command line: " + consoleCommand); + log.debug("Command line: {}", consoleCommand); ExternalProcess.simpleExternalProcess(consoleCommand).execute(false); } - /** - * Waits the given number of milliseconds for a process to settle. - */ - public static void waitForProcessToSettle(long millis) { - try { - log.debug("Waiting for process to settle"); - Thread.sleep(millis); - } catch (InterruptedException e) { - Thread.currentThread().interrupt(); - } - } - private static void usageAndExit(Options options, String message) { usageAndExit(options, message, null); } diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java index 4d21cb3..a3c51c4 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsole.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -36,11 +36,12 @@ import java.util.ArrayList; import java.util.Collection; -import java.util.Date; import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.DefaultParser; @@ -76,11 +77,9 @@ import gov.nasa.ziggy.services.messages.FireTriggerRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; -import gov.nasa.ziggy.services.messaging.ZiggyRmiServer; import gov.nasa.ziggy.ui.util.proxy.PipelineExecutorProxy; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; -import gov.nasa.ziggy.util.Iso8601Formatter; import gov.nasa.ziggy.util.TasksStates; import gov.nasa.ziggy.util.ZiggyShutdownHook; import gov.nasa.ziggy.util.dispmod.AlertLogDisplayModel; @@ -159,6 +158,9 @@ version Display the version (as a Git tag) private static final int HELP_WIDTH = 100; + // Other constants. + private static final long MESSAGE_SENT_WAIT_MILLIS = 500; + @AcceptableCatchBlock(rationale = Rationale.USAGE) @AcceptableCatchBlock(rationale = Rationale.SYSTEM_EXIT) public static void main(String[] args) throws Exception { @@ -252,7 +254,7 @@ private static void usageAndExit(Options options, String message, Throwable e) { if (message != null) { System.err.println(message); } - new HelpFormatter().printHelp(HELP_WIDTH, "ZiggyConsole [options] command...", + new HelpFormatter().printHelp(HELP_WIDTH, "ZiggyConsole command [options]", COMMAND_HELP, options, null); } else if (e != null) { log.error(message, e); @@ -283,29 +285,61 @@ private static Command checkForAmbiguousCommand(String userCommand) { return commands.iterator().next(); } - private static void startZiggyClient() { - int rmiPort = ZiggyConfiguration.getInstance() - .getInt(PropertyName.SUPERVISOR_PORT.property(), ZiggyRmiServer.RMI_PORT_DEFAULT); - ZiggyRmiClient.initializeInstance(rmiPort, NAME); + /** + * Starts a {@link ZiggyRmiClient}. This method ensures that the RMI client is no longer needed + * before allowing the system to shut down. The caller notifies this code that it is done with + * the client by decrementing the latch that this method returns. + * + * @return a countdown latch that should be decremented after the caller no longer needs the + * client + */ + private CountDownLatch startZiggyClient() { + ZiggyRmiClient.start(NAME); + + final CountDownLatch clientStillNeededLatch = new CountDownLatch(1); ZiggyShutdownHook.addShutdownHook(() -> { + + // Wait for any messages to be sent before we reset the ZiggyRmiClient. + try { + clientStillNeededLatch.await(MESSAGE_SENT_WAIT_MILLIS, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + ZiggyRmiClient.reset(); }); + + return clientStillNeededLatch; } - private void runCommand(Command command, CommandLine cmdLine, List commands) - throws Exception { + @AcceptableCatchBlock(rationale = Rationale.USAGE) + private void runCommand(Command command, CommandLine cmdLine, List commands) { + + Throwable exception = (Throwable) DatabaseTransactionFactory.performTransaction(() -> { + try { + switch (command) { + case CANCEL -> cancel(); + case CONFIG -> config(cmdLine); + case DISPLAY -> display(cmdLine); + case HELP -> throw new IllegalArgumentException(""); + case LOG -> log(); + case RESET -> reset(cmdLine); + case RESTART -> restart(cmdLine); + case START -> start(commands); + case VERSION -> System.out.println(ZiggyConfiguration.getInstance() + .getString(PropertyName.ZIGGY_VERSION.property())); + } + } catch (Throwable e) { + return e; + } + return null; + }); - switch (command) { - case CANCEL -> cancel(); - case CONFIG -> config(cmdLine); - case DISPLAY -> display(cmdLine); - case HELP -> throw new IllegalArgumentException(""); - case LOG -> log(); - case RESET -> reset(cmdLine); - case RESTART -> restart(cmdLine); - case START -> start(commands); - case VERSION -> System.out.println( - ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION.property())); + if (exception instanceof RuntimeException) { + throw (RuntimeException) exception; + } + if (exception != null) { + throw new PipelineException(exception); } } @@ -635,17 +669,13 @@ private ResetType parseResetType(CommandLine cmdLine) { return resetType; } + /** + * Sets the pipeline task state to ERROR for any tasks assigned to this worker that are in the + * PROCESSING state. This condition indicates that the previous instance of the worker process + * on this host died abnormally. + */ private void resetPipelineInstance(PipelineInstance instance, boolean allStalledTasks) { - - /* - * Set the pipeline task state to ERROR for any tasks assigned to this worker that are in - * the PROCESSING state. This condition indicates that the previous instance of the worker - * process on this host died abnormally. - */ - DatabaseTransactionFactory.performTransaction(() -> { - new PipelineTaskCrud().resetTaskStates(instance.getId(), allStalledTasks); - return null; - }); + new PipelineTaskCrud().resetTaskStates(instance.getId(), allStalledTasks); } private void restart(CommandLine cmdLine) { @@ -653,7 +683,7 @@ private void restart(CommandLine cmdLine) { throw new IllegalArgumentException("One or more tasks are not specified"); } - startZiggyClient(); + CountDownLatch messageSentLatch = startZiggyClient(); Collection taskIds = new ArrayList<>(); for (String taskId : cmdLine.getOptionValues(TASK_OPTION)) { @@ -688,7 +718,8 @@ public Void transaction() throws Exception { return null; } - new PipelineExecutorProxy().restartTasks(tasks, RunMode.RESTART_FROM_BEGINNING); + new PipelineExecutorProxy().restartTasks(tasks, RunMode.RESTART_FROM_BEGINNING, + messageSentLatch); return null; } @@ -700,22 +731,19 @@ private void start(List commands) { throw new IllegalArgumentException("A pipeline name is not specified"); } - startZiggyClient(); + CountDownLatch messageSentLatch = startZiggyClient(); String pipelineName = commands.get(0); String instanceName = commands.size() > 1 ? commands.get(1) : null; String startNodeName = commands.size() > 2 ? commands.get(2) : null; String stopNodeName = commands.size() > 3 ? commands.get(3) : null; - if (instanceName == null) { - instanceName = Iso8601Formatter.dateTimeLocalFormatter().format(new Date()); - } - System.out.println(String.format("Launching %s: name=%s, start=%s, stop=%s...", pipelineName, instanceName, startNodeName != null ? startNodeName : "", stopNodeName != null ? stopNodeName : "")); ZiggyMessenger.publish( - new FireTriggerRequest(pipelineName, instanceName, startNodeName, stopNodeName, 1, 0)); + new FireTriggerRequest(pipelineName, instanceName, startNodeName, stopNodeName, 1, 0), + messageSentLatch); System.out.println(String.format("Launching %s: name=%s, start=%s, stop=%s...done", pipelineName, instanceName, startNodeName != null ? startNodeName : "", stopNodeName != null ? stopNodeName : "")); diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java index e15f69b..7dc4169 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyConsolePanel.java @@ -25,6 +25,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import gov.nasa.ziggy.ui.datastore.ViewEditDatastorePanel; import gov.nasa.ziggy.ui.dr.DataReceiptPanel; import gov.nasa.ziggy.ui.events.ZiggyEventHandlerPanel; import gov.nasa.ziggy.ui.instances.InstancesTasksPanel; @@ -32,8 +33,6 @@ import gov.nasa.ziggy.ui.module.ViewEditModuleLibraryPanel; import gov.nasa.ziggy.ui.parameters.ViewEditParameterSetsPanel; import gov.nasa.ziggy.ui.pipeline.ViewEditPipelinesPanel; -import gov.nasa.ziggy.ui.security.ViewEditRolesPanel; -import gov.nasa.ziggy.ui.security.ViewEditUsersPanel; import gov.nasa.ziggy.ui.status.StatusPanel; import gov.nasa.ziggy.ui.util.ViewEditKeyValuePairPanel; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; @@ -59,6 +58,7 @@ public class ZiggyConsolePanel extends JSplitPane { private enum ContentItem { LOGO("Logo", false, true, ContentPanel::createLogoCard), PARAMETER_LIBRARY("Parameter Library", ViewEditParameterSetsPanel::newInstance), + DATASTORE("Datastore", ViewEditDatastorePanel::new), PIPELINES("Pipelines", ViewEditPipelinesPanel::newInstance), INSTANCES("Instances", InstancesTasksPanel::new), STATUS("Status", StatusPanel::new), @@ -66,8 +66,6 @@ private enum ContentItem { EVENT_HANDLERS("Event Definitions", ZiggyEventHandlerPanel::new), MODULE_LIBRARY("Module Library", ViewEditModuleLibraryPanel::new), METRILYZER("Metrilyzer", false, true, MetrilyzerPanel::new), - USERS("Users", false, true, ViewEditUsersPanel::new), - ROLES("Roles", false, true, ViewEditRolesPanel::new), GENERAL("General", false, true, ViewEditKeyValuePairPanel::new); private String label; diff --git a/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java b/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java index 315acf6..af14dfa 100644 --- a/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java +++ b/src/main/java/gov/nasa/ziggy/ui/ZiggyGuiConsole.java @@ -29,29 +29,26 @@ import javax.swing.LayoutStyle.ComponentPlacement; import javax.swing.SwingConstants; -import org.apache.commons.configuration2.ImmutableConfiguration; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; -import gov.nasa.ziggy.services.messages.DefaultWorkerResourcesRequest; import gov.nasa.ziggy.services.messages.InvalidateConsoleModelsMessage; import gov.nasa.ziggy.services.messages.ShutdownMessage; -import gov.nasa.ziggy.services.messages.WorkerResources; -import gov.nasa.ziggy.services.messaging.ProcessHeartbeatManager; +import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; +import gov.nasa.ziggy.services.messages.WorkerResourcesRequest; +import gov.nasa.ziggy.services.messaging.HeartbeatManager; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; import gov.nasa.ziggy.services.messaging.ZiggyRmiServer; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.services.security.UserCrud; import gov.nasa.ziggy.ui.status.StatusSummaryPanel; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.models.DatabaseModelRegistry; -import gov.nasa.ziggy.ui.util.proxy.UserCrudProxy; import gov.nasa.ziggy.util.Requestor; import gov.nasa.ziggy.util.ZiggyShutdownHook; +import gov.nasa.ziggy.worker.WorkerResources; /** * The console GUI. @@ -70,11 +67,11 @@ public class ZiggyGuiConsole extends javax.swing.JFrame implements Requestor { private static final String ZIGGY_LOGO_FILE_NAME = "ziggy-small-clear.png"; private static final String ZIGGY_LOGO_DIR = "/images/"; - public static User currentUser; - private static Image pipelineImage; private static Image ziggyImage; + private static WorkerResources defaultResources; + private final UUID uuid = UUID.randomUUID(); { @@ -84,8 +81,7 @@ public class ZiggyGuiConsole extends javax.swing.JFrame implements Requestor { private ZiggyGuiConsole() { // Initialize the ProcessHeartbeatManager for this process. log.info("Initializing ProcessHeartbeatManager"); - ProcessHeartbeatManager - .initializeInstance(new ProcessHeartbeatManager.ConsoleHeartbeatManagerAssistant()); + HeartbeatManager.startInstance(); log.info("Initializing ProcessHeartbeatManager...done"); ZiggyMessenger.subscribe(ShutdownMessage.class, message -> { @@ -93,8 +89,10 @@ private ZiggyGuiConsole() { shutdown(); }); - ZiggyMessenger.subscribe(WorkerResources.class, message -> { - WorkerResources.setDefaultResources(message); + ZiggyMessenger.subscribe(WorkerResourcesMessage.class, message -> { + if (message.getDefaultResources() != null && defaultResources == null) { + defaultResources = message.getDefaultResources(); + } }); ZiggyMessenger.subscribe(InvalidateConsoleModelsMessage.class, message -> { @@ -104,7 +102,7 @@ private ZiggyGuiConsole() { int rmiPort = ZiggyConfiguration.getInstance() .getInt(PropertyName.SUPERVISOR_PORT.property(), ZiggyRmiServer.RMI_PORT_DEFAULT); log.info("Starting ZiggyRmiClient instance with registry on port {}", rmiPort); - ZiggyRmiClient.initializeInstance(rmiPort, NAME); + ZiggyRmiClient.start(NAME); ZiggyShutdownHook.addShutdownHook(() -> { ZiggyRmiClient.reset(); }); @@ -112,7 +110,7 @@ private ZiggyGuiConsole() { buildComponent(); - ZiggyMessenger.publish(new DefaultWorkerResourcesRequest()); + ZiggyMessenger.publish(new WorkerResourcesRequest()); } public static void launch() { @@ -122,8 +120,6 @@ public static void launch() { ZiggyConfiguration.logJvmProperties(); - login(); - ZiggyGuiConsole instance = new ZiggyGuiConsole(); instance.setLocationByPlatform(true); instance.setVisible(true); @@ -135,36 +131,6 @@ public static void launch() { log.debug("Ziggy Console initialization complete"); } - private static void login() { - ImmutableConfiguration config = ZiggyConfiguration.getInstance(); - boolean devModeRequireLogin = config - .getBoolean(PropertyName.REQUIRE_LOGIN_OVERRIDE.property(), false); - - // TODO Resurrect ZiggyVersion.isRelease() or delete commented-out code - // In the unlikely event this is needed, I'd suggest resurrecting ZiggyVersion.version() as - // well and replace code that current says - // ZiggyConfiguration.getInstance().getString(PropertyName.ZIGGY_VERSION) with - // ZiggyVersion.version(). - boolean requireLogin = devModeRequireLogin /* || ZiggyVersion.isRelease() */; - - // Don't require login if there are no configured users. - if (new UserCrud().retrieveAllUsers().isEmpty()) { - requireLogin = false; - } - - UserCrudProxy userCrud = new UserCrudProxy(); - if (requireLogin) { - - currentUser = userCrud - .retrieveUser(config.getString(PropertyName.USER_NAME.property())); - - if (currentUser == null) { - log.error("Exceeded max login attempts"); - System.exit(-1); - } - } - } - private void buildComponent() { setTitle(NAME); setSize(ZiggyGuiConstants.MAIN_WINDOW_WIDTH, ZiggyGuiConstants.MAIN_WINDOW_HEIGHT); @@ -326,6 +292,10 @@ private static Image getImage(URL url) { return image; } + public static WorkerResources defaultResources() { + return defaultResources; + } + @Override public UUID requestorIdentifier() { return uuid; diff --git a/src/main/java/gov/nasa/ziggy/ui/datastore/EditDatastoreRegexpDialog.java b/src/main/java/gov/nasa/ziggy/ui/datastore/EditDatastoreRegexpDialog.java new file mode 100644 index 0000000..6de2a7d --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/datastore/EditDatastoreRegexpDialog.java @@ -0,0 +1,140 @@ +package gov.nasa.ziggy.ui.datastore; + +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.SAVE; +import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.boldLabel; +import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; +import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButtonPanel; + +import java.awt.BorderLayout; +import java.awt.Window; +import java.awt.event.ActionEvent; + +import javax.swing.GroupLayout; +import javax.swing.JLabel; +import javax.swing.JPanel; +import javax.swing.JTextField; +import javax.swing.LayoutStyle.ComponentPlacement; + +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.data.datastore.DatastoreRegexpCrud; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import gov.nasa.ziggy.ui.util.MessageUtil; + +/** + * Panel for editing the include and exclude regular expressions within a {@link DatastoreRegexp}. + * + * @author PT + * @author Bill Wohler + */ +public class EditDatastoreRegexpDialog extends javax.swing.JDialog { + + private static final long serialVersionUID = 20240208L; + + private static final String TITLE = "Edit datastore regular expressions"; + + private DatastoreRegexp datastoreRegexp; + private boolean cancelled; + private JTextField includeTextField; + private JTextField excludeTextField; + + public EditDatastoreRegexpDialog(Window owner, DatastoreRegexp datastoreRegexp) { + super(owner, DEFAULT_MODALITY_TYPE); + this.datastoreRegexp = datastoreRegexp; + buildComponent(); + setLocationRelativeTo(owner); + } + + private void buildComponent() { + setTitle(TITLE); + getContentPane().add(createDataPanel(), BorderLayout.CENTER); + getContentPane().add( + createButtonPanel(createButton(SAVE, this::save), createButton(CANCEL, this::cancel)), + BorderLayout.SOUTH); + + pack(); + } + + private JPanel createDataPanel() { + JLabel name = boldLabel("Name"); + JLabel nameText = new JLabel(datastoreRegexp.getName()); + JLabel value = boldLabel("Value"); + JLabel valueText = new JLabel(datastoreRegexp.getValue()); + JLabel include = boldLabel("Include"); + includeTextField = new JTextField(datastoreRegexp.getInclude()); + includeTextField.setColumns(minDialogWidth()); + JLabel exclude = boldLabel("Exclude"); + excludeTextField = new JTextField(datastoreRegexp.getExclude()); + excludeTextField.setColumns(minDialogWidth()); + + JPanel panel = new JPanel(); + GroupLayout dataPanelLayout = new GroupLayout(panel); + dataPanelLayout.setAutoCreateContainerGaps(true); + panel.setLayout(dataPanelLayout); + + dataPanelLayout.setHorizontalGroup(dataPanelLayout.createParallelGroup() + .addComponent(name) + .addComponent(nameText) + .addComponent(value) + .addComponent(valueText) + .addComponent(include) + .addComponent(includeTextField) + .addComponent(exclude) + .addComponent(excludeTextField)); + + dataPanelLayout.setVerticalGroup(dataPanelLayout.createSequentialGroup() + .addComponent(name) + .addComponent(nameText) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(value) + .addComponent(valueText) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(include) + .addComponent(includeTextField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, + GroupLayout.PREFERRED_SIZE) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(exclude) + .addComponent(excludeTextField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, + GroupLayout.PREFERRED_SIZE)); + + return panel; + } + + /** + * Returns the minimum dialog width to avoid truncating the title. + * + * @return the number of characters that a full-width field should use + */ + private int minDialogWidth() { + return TITLE.length(); + } + + private void save(ActionEvent evt) { + try { + // Trim the values. It's exceedingly unlikely that the data files have leading or + // trailing spaces, but it's more likely that a user might accidently enter a space here + // and then regexp would fail to match. If leading and trailing spaces are required, + // then allow single or double quotes (' text ') to protect the space. Strip the quotes + // before saving to the database, and add them if leading or trailing spaces are + // detected when reading from the database. + datastoreRegexp.setInclude(includeTextField.getText().trim()); + datastoreRegexp.setExclude(excludeTextField.getText().trim()); + DatabaseTransactionFactory.performTransaction(() -> { + new DatastoreRegexpCrud().merge(datastoreRegexp); + return null; + }); + setVisible(false); + } catch (Exception e) { + MessageUtil.showError(this, e); + } + } + + private void cancel(ActionEvent evt) { + cancelled = true; + setVisible(false); + } + + public boolean isCancelled() { + return cancelled; + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java b/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java new file mode 100644 index 0000000..8927c17 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/datastore/ViewEditDatastorePanel.java @@ -0,0 +1,132 @@ +package gov.nasa.ziggy.ui.datastore; + +import static com.google.common.base.Preconditions.checkArgument; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import javax.swing.SwingUtilities; + +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.ui.util.MessageUtil; +import gov.nasa.ziggy.ui.util.models.AbstractDatabaseModel; +import gov.nasa.ziggy.ui.util.proxy.DatastoreRegexpCrudProxy; +import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; + +/** + * Panel for viewing and editing datastore configurations. + * + * @author PT + * @author Bill Wohler + */ +public class ViewEditDatastorePanel extends AbstractViewEditPanel { + + private static final long serialVersionUID = 20240208L; + + public ViewEditDatastorePanel() { + super(new RegexpTableModel()); + buildComponent(); + + // An explicit refresh to show the data shouldn't be necessary, but it is. + refresh(); + } + + @Override + protected void refresh() { + try { + ziggyTable.loadFromDatabase(); + } catch (Throwable e) { + MessageUtil.showError(this, e); + } + } + + @Override + protected void create() { + throw new UnsupportedOperationException("Create not supported"); + } + + @Override + protected void edit(int row) { + DatastoreRegexp regexp = ziggyTable.getContentAtViewRow(row); + + if (regexp != null) { + EditDatastoreRegexpDialog dialog = new EditDatastoreRegexpDialog( + SwingUtilities.getWindowAncestor(this), regexp); + dialog.setVisible(true); + if (!dialog.isCancelled()) { + ziggyTable.loadFromDatabase(); + } + } + } + + @Override + protected void delete(int row) { + throw new UnsupportedOperationException("Delete not supported"); + } + + @Override + protected Set optionalViewEditFunctions() { + return new HashSet<>(); + } + + public static class RegexpTableModel extends AbstractDatabaseModel { + + private static final long serialVersionUID = 20240124L; + + private static final String[] COLUMN_NAMES = { "Name", "Value", "Include", "Exclude" }; + + private List datastoreRegexps = new ArrayList<>(); + + @Override + public int getRowCount() { + return datastoreRegexps.size(); + } + + @Override + public int getColumnCount() { + return COLUMN_NAMES.length; + } + + @Override + public String getColumnName(int column) { + checkColumnArgument(column); + return COLUMN_NAMES[column]; + } + + private void checkColumnArgument(int columnIndex) { + checkArgument(columnIndex < COLUMN_NAMES.length, "Column value of " + columnIndex + + " outside of expected range from 0 to " + COLUMN_NAMES.length); + } + + @Override + public Object getValueAt(int rowIndex, int columnIndex) { + checkColumnArgument(columnIndex); + DatastoreRegexp regexp = getContentAtRow(rowIndex); + return switch (columnIndex) { + case 0 -> regexp.getName(); + case 1 -> regexp.getValue(); + case 2 -> regexp.getInclude(); + case 3 -> regexp.getExclude(); + default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); + }; + } + + @Override + public void loadFromDatabase() { + datastoreRegexps = new DatastoreRegexpCrudProxy().retrieveAll(); + fireTableDataChanged(); + } + + @Override + public DatastoreRegexp getContentAtRow(int row) { + return datastoreRegexps.get(row); + } + + @Override + public Class tableModelContentClass() { + return DatastoreRegexp.class; + } + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java b/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java index da9bd0d..c6b008d 100644 --- a/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/dr/DataReceiptInstanceDialog.java @@ -126,8 +126,8 @@ private static class DataReceiptInstanceTableModel private static final long serialVersionUID = 20230823L; - private static final String[] COLUMN_NAMES = { "Task ID", "Name", "Type", "Status" }; - private static final int[] COLUMN_WIDTHS = { 100, 500, 100, 100 }; + private static final String[] COLUMN_NAMES = { "Task ID", "Name", "Status" }; + private static final int[] COLUMN_WIDTHS = { 100, 500, 100 }; private List dataReceiptFiles = new ArrayList<>(); private DataReceiptInstance dataReceiptInstance; @@ -158,8 +158,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { return switch (columnIndex) { case 0 -> dataReceiptFile.getTaskId(); case 1 -> dataReceiptFile.getName(); - case 2 -> dataReceiptFile.getFileType(); - case 3 -> dataReceiptFile.getStatus(); + case 2 -> dataReceiptFile.getStatus(); default -> throw new IllegalArgumentException( "Invalid column index: " + columnIndex); }; diff --git a/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java b/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java index 958bc42..7876051 100644 --- a/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/events/ZiggyEventHandlerPanel.java @@ -36,9 +36,9 @@ import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.ButtonPanelContext; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.table.ZiggyTable; import gov.nasa.ziggy.util.Requestor; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * Panel to display the collection of {@link ZiggyEventHandler} instances and their states. @@ -119,7 +119,7 @@ public void update() { } private static class EventHandlerTableModel extends AbstractTableModel - implements TableModelContentClass, Requestor { + implements ModelContentClass, Requestor { private static final long serialVersionUID = 20230824L; diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/AlertLogDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/AlertLogDialog.java index d7469de..5070d92 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/AlertLogDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/AlertLogDialog.java @@ -17,10 +17,10 @@ import gov.nasa.ziggy.services.alert.AlertLog; import gov.nasa.ziggy.ui.ConsoleSecurityException; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.proxy.AlertLogCrudProxy; import gov.nasa.ziggy.ui.util.table.ZiggyTable; import gov.nasa.ziggy.util.dispmod.AlertLogDisplayModel; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * @author Bill Wohler @@ -70,7 +70,7 @@ private void close(ActionEvent evt) { } private static class AlertLogTableModel extends AbstractTableModel - implements TableModelContentClass { + implements ModelContentClass { private final AlertLogCrudProxy alertLogCrud; private List alerts = new ArrayList<>(); private final long pipelineInstanceId; diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java index 626064a..25b0cfb 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialog.java @@ -8,6 +8,7 @@ import java.awt.BorderLayout; import java.awt.Window; import java.awt.event.ActionEvent; +import java.text.DecimalFormat; import java.util.List; import javax.swing.GroupLayout; @@ -24,9 +25,9 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.LabelType; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.proxy.PipelineTaskOperationsProxy; import gov.nasa.ziggy.ui.util.table.ZiggyTable; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * Displays cost estimates for a pipeline instance and the tasks within that instance. Costs will be @@ -126,16 +127,31 @@ private void close(ActionEvent evt) { setVisible(false); } - private String instanceCost(List pipelineTasksInInstance) { + static String instanceCost(List pipelineTasksInInstance) { double totalCost = 0; for (PipelineTask task : pipelineTasksInInstance) { totalCost += task.costEstimate(); } - return Long.toString(Math.round(totalCost)); + return formatCost(totalCost); + } + + private static String formatCost(double cost) { + String format; + if (cost < 1) { + format = "#.####"; + } else if (cost < 10) { + format = "#.###"; + } else if (cost < 100) { + format = "#.##"; + } else { + format = "#.#"; + } + + return new DecimalFormat(format).format(cost); } private static class TaskCostEstimateTableModel extends AbstractTableModel - implements TableModelContentClass { + implements ModelContentClass { private static final long serialVersionUID = 20230817L; @@ -171,7 +187,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { case 1 -> task.getModuleName(); case 2 -> task.uowTaskInstance().briefState(); case 3 -> task.getState().toString(); - case 4 -> Long.toString(Math.round(task.costEstimate())); + case 4 -> formatCost(task.costEstimate()); default -> throw new IllegalArgumentException( "Illegal column number: " + columnIndex); }; diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java index c09ca3a..ea1aee1 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstanceStatsDialog.java @@ -21,9 +21,9 @@ import gov.nasa.ziggy.ui.ConsoleSecurityException; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.LabelType; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.proxy.PipelineTaskCrudProxy; import gov.nasa.ziggy.ui.util.table.ZiggyTable; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; import gov.nasa.ziggy.util.dispmod.PipelineStatsDisplayModel; import gov.nasa.ziggy.util.dispmod.PipelineStatsDisplayModel.ProcessingStatistics; import gov.nasa.ziggy.util.dispmod.TaskMetricsDisplayModel.ModuleTaskMetrics; @@ -133,7 +133,7 @@ private void close(ActionEvent evt) { } private static class PipelineStatsTableModel extends AbstractTableModel - implements TableModelContentClass { + implements ModelContentClass { private PipelineStatsDisplayModel pipelineStatsDisplayModel; diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java index 90a062b..99ecd52 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTable.java @@ -18,6 +18,7 @@ import javax.swing.SwingUtilities; import javax.swing.SwingWorker; +import org.apache.commons.lang3.StringUtils; import org.netbeans.swing.etable.ETable; import org.netbeans.swing.etable.ETableColumnModel; import org.slf4j.Logger; @@ -400,8 +401,9 @@ public Object getValueAt(int rowIndex, int columnIndex) { return switch (columnIndex) { case 0 -> pipelineInstance.getId(); - case 1 -> pipelineInstance.getPipelineDefinition().getName() + ": " - + pipelineInstance.getName(); + case 1 -> pipelineInstance.getPipelineDefinition().getName() + + (StringUtils.isEmpty(pipelineInstance.getName()) ? "" + : ": " + pipelineInstance.getName()); case 2 -> ziggyEvent != null ? ziggyEvent.getEventHandlerName() : "-"; case 3 -> ziggyEvent != null ? ziggyEvent.getEventTime() : pipelineInstance.getStartProcessingTime(); diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java index 67b06d6..327a083 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/InstancesTasksPanelAutoRefresh.java @@ -68,8 +68,8 @@ public void run() { public void updatePanel() { instancesTable.loadFromDatabase(); tasksTableModel.loadFromDatabase(); + setInstancesStatusLight(instancesTable.getStateOfInstanceWithMaxid()); SwingUtilities.invokeLater(() -> { - setInstancesStatusLight(instancesTable.getStateOfInstanceWithMaxid()); taskStatusSummaryPanel.update(tasksTableModel); }); } diff --git a/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java b/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java index d009d00..9d42f55 100644 --- a/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java +++ b/src/main/java/gov/nasa/ziggy/ui/instances/TaskMetricsTableModel.java @@ -6,7 +6,7 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.ui.ConsoleSecurityException; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; import gov.nasa.ziggy.util.dispmod.TaskMetricsDisplayModel; import gov.nasa.ziggy.util.dispmod.TaskMetricsDisplayModel.ModuleTaskMetrics; @@ -15,7 +15,7 @@ */ @SuppressWarnings("serial") public class TaskMetricsTableModel extends AbstractTableModel - implements TableModelContentClass { + implements ModelContentClass { private TaskMetricsDisplayModel taskMetricsDisplayModel; private boolean completedTasksOnly; diff --git a/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java b/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java index 23ce583..5c2a188 100644 --- a/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/module/EditModuleDialog.java @@ -30,6 +30,7 @@ import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineModule; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; +import gov.nasa.ziggy.pipeline.definition.PipelineModuleExecutionResources; import gov.nasa.ziggy.ui.util.ClasspathUtils; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.proxy.PipelineModuleDefinitionCrudProxy; @@ -43,7 +44,8 @@ public class EditModuleDialog extends javax.swing.JDialog { private static final Logger log = LoggerFactory.getLogger(EditModuleDialog.class); private static final long serialVersionUID = 20230824L; - private final PipelineModuleDefinition module; + private PipelineModuleDefinition module; + private PipelineModuleExecutionResources executionResources; private final PipelineModuleDefinitionCrudProxy pipelineModuleDefinitionCrud = new PipelineModuleDefinitionCrudProxy(); private JTextArea descText; @@ -56,7 +58,8 @@ public class EditModuleDialog extends javax.swing.JDialog { public EditModuleDialog(Window owner, PipelineModuleDefinition module) { super(owner, DEFAULT_MODALITY_TYPE); this.module = module; - + executionResources = pipelineModuleDefinitionCrud + .retrievePipelineModuleExecutionResources(module); buildComponent(); setLocationRelativeTo(owner); } @@ -88,11 +91,13 @@ private JPanel createDataPanel() { implementingClassComboBox = createImplementingClassComboBox(); JLabel exeTimeout = boldLabel("Executable timeout (seconds)"); - exeTimeoutText = new JTextField(Integer.toString(module.getExeTimeoutSecs())); + exeTimeoutText = new JTextField( + Integer.toString(executionResources.getExeTimeoutSeconds())); exeTimeoutText.setColumns(15); JLabel minMemory = boldLabel("Minimum memory (MB)"); - minMemoryText = new JTextField(Integer.toString(module.getMinMemoryMegaBytes())); + minMemoryText = new JTextField( + Integer.toString(executionResources.getMinMemoryMegabytes())); minMemoryText.setColumns(15); JPanel dataPanel = new JPanel(); @@ -196,10 +201,12 @@ private void save(ActionEvent evt) { .getSelectedItem(); module.setPipelineModuleClass(selectedImplementingClass); - module.setExeTimeoutSecs(toInt(exeTimeoutText.getText(), 0)); - module.setMinMemoryMegaBytes(toInt(minMemoryText.getText(), 0)); + executionResources.setExeTimeoutSeconds(toInt(exeTimeoutText.getText(), 0)); + executionResources.setMinMemoryMegabytes(toInt(minMemoryText.getText(), 0)); - pipelineModuleDefinitionCrud.createOrUpdate(module); + module = pipelineModuleDefinitionCrud.merge(module); + executionResources = pipelineModuleDefinitionCrud + .mergeExecutionResources(executionResources); setVisible(false); } catch (Exception e) { diff --git a/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java b/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java index f284c36..d4c2441 100644 --- a/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/module/ViewEditModuleLibraryPanel.java @@ -11,12 +11,9 @@ import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.User; import gov.nasa.ziggy.ui.ConsoleSecurityException; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.models.AbstractDatabaseModel; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; import gov.nasa.ziggy.ui.util.proxy.PipelineModuleDefinitionCrudProxy; import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; @@ -35,19 +32,13 @@ public ViewEditModuleLibraryPanel() { } @Override - protected Set optionalViewEditFunctions() { - return Set.of(OptionalViewEditFunctions.DELETE, OptionalViewEditFunctions.NEW, - OptionalViewEditFunctions.RENAME); + protected Set optionalViewEditFunctions() { + return Set.of(OptionalViewEditFunction.DELETE, OptionalViewEditFunction.NEW, + OptionalViewEditFunction.RENAME); } @Override protected void create() { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } String newModuleName = JOptionPane.showInputDialog(SwingUtilities.getWindowAncestor(this), "Enter the name for the new Module Definition", "New Pipeline Module Definition", @@ -69,12 +60,6 @@ protected void create() { @Override protected void rename(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } PipelineModuleDefinition selectedModule = ziggyTable.getContentAtViewRow(row); @@ -101,12 +86,6 @@ protected void rename(int row) { @Override protected void edit(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } setCursor(Cursor.getPredefinedCursor(Cursor.WAIT_CURSOR)); showEditDialog(ziggyTable.getContentAtViewRow(row)); @@ -116,13 +95,6 @@ protected void edit(int row) { @Override protected void delete(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - PipelineModuleDefinition module = ziggyTable.getContentAtViewRow(row); if (!module.isLocked()) { @@ -207,7 +179,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { AuditInfo auditInfo = module.getAuditInfo(); - User lastChangedUser = null; + String lastChangedUser = null; Date lastChangedTime = null; if (auditInfo != null) { @@ -220,7 +192,7 @@ public Object getValueAt(int rowIndex, int columnIndex) { case 1 -> module.getName(); case 2 -> module.getVersion(); case 3 -> module.isLocked(); - case 4 -> lastChangedUser != null ? lastChangedUser.getLoginName() : "---"; + case 4 -> lastChangedUser != null ? lastChangedUser : "---"; case 5 -> lastChangedTime != null ? lastChangedTime : "---"; default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); }; diff --git a/src/main/java/gov/nasa/ziggy/ui/parameters/ImportParamLibDialog.java b/src/main/java/gov/nasa/ziggy/ui/parameters/ImportParamLibDialog.java index f8c6e76..53e555a 100644 --- a/src/main/java/gov/nasa/ziggy/ui/parameters/ImportParamLibDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/parameters/ImportParamLibDialog.java @@ -33,7 +33,7 @@ import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; import gov.nasa.ziggy.ui.util.table.ZiggyTable; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; /** * This dialog is used to import a parameter library from disk into the parameter library in the @@ -164,7 +164,7 @@ private void appendToReport(StringBuilder report, List d } for (ParameterSetDescriptor desc : descs) { - report.append(StringUtils.pad(desc.getName(), maxNameLength + 5) + "[" + report.append(ZiggyStringUtils.pad(desc.getName(), maxNameLength + 5) + "[" + desc.shortClassName() + "]\n"); } } diff --git a/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java b/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java index 01622b2..fbb5632 100644 --- a/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/parameters/ViewEditParameterSetsPanel.java @@ -8,10 +8,12 @@ import java.awt.Cursor; import java.awt.event.ActionEvent; import java.io.File; +import java.util.ArrayList; import java.util.Date; import java.util.List; import java.util.Set; +import javax.swing.JButton; import javax.swing.JFileChooser; import javax.swing.JOptionPane; import javax.swing.SwingUtilities; @@ -24,20 +26,16 @@ import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.pipeline.definition.Group; import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.ui.ConsoleSecurityException; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.TextualReportDialog; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; import gov.nasa.ziggy.ui.util.proxy.ParameterSetCrudProxy; import gov.nasa.ziggy.ui.util.proxy.ParametersOperationsProxy; import gov.nasa.ziggy.ui.util.proxy.PipelineOperationsProxy; import gov.nasa.ziggy.ui.util.proxy.RetrieveLatestVersionsCrudProxy; -import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; +import gov.nasa.ziggy.ui.util.table.AbstractViewEditGroupPanel; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * View / Edit panel for {@link ParameterSet} instances. The user can also use this panel to move @@ -46,46 +44,45 @@ * @author PT * @author Bill Wohler */ -public class ViewEditParameterSetsPanel extends AbstractViewEditPanel { +public class ViewEditParameterSetsPanel extends AbstractViewEditGroupPanel { private static final long serialVersionUID = 20230810L; private ParameterSetCrudProxy parameterSetCrud = new ParameterSetCrudProxy(); private String defaultParamLibImportExportPath; + public ViewEditParameterSetsPanel(RowModel rowModel, ZiggyTreeModel treeModel) { + super(rowModel, treeModel, "Name"); + buildComponent(); + + for (int column = 0; column < ParameterSetsRowModel.COLUMN_WIDTHS.length; column++) { + ziggyTable.setPreferredColumnWidth(column, ParameterSetsRowModel.COLUMN_WIDTHS[column]); + } + } + /** * Convenience method that can be used instead of the constructor. Helpful because the row model * for parameter sets needs the tree model in its constructor. */ public static ViewEditParameterSetsPanel newInstance() { - ZiggyTreeModel treeModel = new ZiggyTreeModel<>(new ParameterSetCrudProxy()); + ZiggyTreeModel treeModel = new ZiggyTreeModel<>(new ParameterSetCrudProxy(), + ParameterSet.class); ParameterSetsRowModel rowModel = new ParameterSetsRowModel(treeModel); return new ViewEditParameterSetsPanel(rowModel, treeModel); } - public ViewEditParameterSetsPanel(RowModel rowModel, ZiggyTreeModel treeModel) { - super(rowModel, treeModel, "Name"); - buildComponent(); - - ZiggySwingUtils.addButtonsToPanel(getButtonPanel(), - ZiggySwingUtils.createButton(REPORT, this::report), - ZiggySwingUtils.createButton(IMPORT, this::importParameterLibrary), - ZiggySwingUtils.createButton(EXPORT, this::exportParameterLibrary)); - - for (int column = 0; column < ParameterSetsRowModel.COLUMN_WIDTHS.length; column++) { - ziggyTable.setPreferredColumnWidth(column, ParameterSetsRowModel.COLUMN_WIDTHS[column]); - } + @Override + protected List buttons() { + List buttons = new ArrayList<>( + List.of(ZiggySwingUtils.createButton(REPORT, this::report), + ZiggySwingUtils.createButton(IMPORT, this::importParameterLibrary), + ZiggySwingUtils.createButton(EXPORT, this::exportParameterLibrary))); + buttons.addAll(super.buttons()); + return buttons; } private void report(ActionEvent evt) { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - Object[] options = { "Formatted", "Colon-delimited" }; int n = JOptionPane.showOptionDialog(SwingUtilities.getWindowAncestor(this), "Specify report type", "Report type", JOptionPane.YES_NO_CANCEL_OPTION, @@ -152,10 +149,9 @@ private void exportParameterLibrary(ActionEvent evt) { } @Override - protected Set optionalViewEditFunctions() { - return Set.of(OptionalViewEditFunctions.DELETE, OptionalViewEditFunctions.NEW, - OptionalViewEditFunctions.COPY, OptionalViewEditFunctions.RENAME, - OptionalViewEditFunctions.GROUP); + protected Set optionalViewEditFunctions() { + return Set.of(OptionalViewEditFunction.DELETE, OptionalViewEditFunction.NEW, + OptionalViewEditFunction.COPY, OptionalViewEditFunction.RENAME); } @Override @@ -165,7 +161,6 @@ protected RetrieveLatestVersionsCrudProxy getCrudProxy() { @Override protected void copy(int row) { - checkPrivileges(); ParameterSet selectedParameterSet = ziggyTable.getContentAtViewRow(row); @@ -205,7 +200,6 @@ private void showEditDialog(ParameterSet module, boolean isNew) { @Override protected void rename(int row) { - checkPrivileges(); ParameterSet selectedParameterSet = ziggyTable.getContentAtViewRow(row); @@ -229,7 +223,6 @@ protected void rename(int row) { @Override protected void edit(int row) { - checkPrivileges(); ParameterSet selectedParameterSet = ziggyTable.getContentAtViewRow(row); if (selectedParameterSet != null) { @@ -239,7 +232,6 @@ protected void edit(int row) { @Override protected void delete(int row) { - checkPrivileges(); ParameterSet selectedParameterSet = ziggyTable.getContentAtViewRow(row); if (selectedParameterSet.isLocked()) { @@ -264,7 +256,6 @@ protected void delete(int row) { @Override protected void create() { - checkPrivileges(); setCursor(Cursor.getPredefinedCursor(Cursor.WAIT_CURSOR)); ParameterSet newParameterSet = NewParameterSetDialog.createParameterSet(this); @@ -293,7 +284,7 @@ public void refresh() { * @author PT */ private static class ParameterSetsRowModel - implements RowModel, TableModelContentClass { + implements RowModel, ModelContentClass { private ZiggyTreeModel treeModel; @@ -325,7 +316,7 @@ public String getColumnName(int column) { } private void checkColumnArgument(int columnIndex) { - checkArgument(columnIndex >= 0 && columnIndex < COLUMN_NAMES.length, "column value of " + checkArgument(columnIndex >= 0 && columnIndex < COLUMN_NAMES.length, "Column value of " + columnIndex + " outside of expected range from 0 to " + COLUMN_NAMES.length); } @@ -343,7 +334,7 @@ public Object getValueFor(Object treeNode, int columnIndex) { AuditInfo auditInfo = parameterSet.getAuditInfo(); - User lastChangedUser = null; + String lastChangedUser = null; Date lastChangedTime = null; if (auditInfo != null) { diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineDialog.java index b2b6872..aaf8f1f 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineDialog.java @@ -12,10 +12,9 @@ import java.awt.event.ActionEvent; import java.io.File; import java.util.HashMap; -import java.util.HashSet; import java.util.Map; -import java.util.Set; +import javax.swing.ButtonGroup; import javax.swing.GroupLayout; import javax.swing.JCheckBox; import javax.swing.JFileChooser; @@ -23,19 +22,25 @@ import javax.swing.JList; import javax.swing.JOptionPane; import javax.swing.JPanel; +import javax.swing.JRadioButton; import javax.swing.JScrollPane; import javax.swing.JSpinner; import javax.swing.LayoutStyle.ComponentPlacement; import javax.swing.SpinnerListModel; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import gov.nasa.ziggy.metrics.report.ReportFilePaths; import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.PipelineTaskInformation; import gov.nasa.ziggy.pipeline.TriggerValidationResults; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; import gov.nasa.ziggy.pipeline.definition.PipelineInstance.Priority; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.TextualReportDialog; @@ -44,6 +49,7 @@ import gov.nasa.ziggy.ui.util.ZiggySwingUtils.LabelType; import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; import gov.nasa.ziggy.ui.util.proxy.PipelineDefinitionCrudProxy; +import gov.nasa.ziggy.ui.util.proxy.PipelineDefinitionNodeCrudProxy; import gov.nasa.ziggy.ui.util.proxy.PipelineOperationsProxy; /** @@ -51,25 +57,27 @@ */ @SuppressWarnings("serial") public class EditPipelineDialog extends javax.swing.JDialog { + + private static final Logger log = LoggerFactory.getLogger(EditPipelineDialog.class); + private JSpinner prioritySpinner; private JLabel pipelineNameTextField; private JCheckBox validCheckBox; private JList modulesList; + private JRadioButton reprocessButton; + private PipelineDefinition pipeline; private String pipelineName; - private PipelineModulesListModel pipelineModulesListModel; private ZiggyTreeModel pipelineModel; - // Contains all parameter sets (except remote parameters) that have been edited since this + // Contains all parameter sets that have been edited since this // window was opened. private Map editedParameterSets = new HashMap<>(); - // Contains all remote parameter sets that have been edited since this window was opened. - // This has to be done separately from the general edited parameter sets because the remote - // execution dialog needs some of the parameter set infrastructure. - private Map editedRemoteParameterSets = new HashMap<>(); + // Contains all pipeline definition nodes with updated execution resources. + private Map updatedExecutionResources = new HashMap<>(); public EditPipelineDialog(Window owner, String pipelineName, ZiggyTreeModel pipelineModel) { @@ -85,8 +93,9 @@ public EditPipelineDialog(Window owner, String pipelineName, PipelineDefinition ZiggyTreeModel pipelineModel) { super(owner, DEFAULT_MODALITY_TYPE); - this.pipelineName = pipelineName; this.pipeline = pipeline; + this.pipelineName = !StringUtils.isEmpty(pipelineName) ? pipelineName + : this.pipeline.getName(); this.pipelineModel = pipelineModel; buildComponent(); @@ -131,12 +140,25 @@ private JPanel createDataPanel() { validCheckBox = new JCheckBox(); validCheckBox.setEnabled(false); + JLabel processingMode = boldLabel("Processing mode"); + ButtonGroup processConfigButtonGroup = new ButtonGroup(); + reprocessButton = new JRadioButton("Process all data"); + JRadioButton forwardProcessButton = new JRadioButton("Process new data"); + processConfigButtonGroup.add(reprocessButton); + processConfigButtonGroup.add(forwardProcessButton); + + if (new PipelineDefinitionCrudProxy() + .retrieveProcessingMode(this.pipelineName) == ProcessingMode.PROCESS_ALL) { + reprocessButton.setSelected(true); + } else { + forwardProcessButton.setSelected(true); + } + JLabel pipelineParameterSetsGroup = boldLabel("Pipeline parameter sets", LabelType.HEADING1); ParameterSetMapEditorPanel pipelineParameterSetMapEditorPanel = new ParameterSetMapEditorPanel( - pipeline.getPipelineParameterSetNames(), new HashSet<>(), new HashMap<>(), - editedParameterSets); + pipeline.getPipelineParameterSetNames(), new HashMap<>(), editedParameterSets); pipelineParameterSetMapEditorPanel .setMapListener(source -> pipeline.setPipelineParameterSetNames( pipelineParameterSetMapEditorPanel.getParameterSetsMap())); @@ -171,8 +193,12 @@ private JPanel createDataPanel() { .addComponent(prioritySpinner, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addGroup(dataPanelLayout.createSequentialGroup() - .addComponent(valid) - .addComponent(validCheckBox)))) + .addGroup(dataPanelLayout.createSequentialGroup() + .addComponent(valid) + .addComponent(validCheckBox))) + .addComponent(processingMode) + .addComponent(reprocessButton) + .addComponent(forwardProcessButton))) .addComponent(pipelineParameterSetsGroup) .addComponent(pipelineParameterSetMapEditorPanel) .addComponent(modulesGroup) @@ -196,6 +222,10 @@ private JPanel createDataPanel() { .addGroup(dataPanelLayout.createParallelGroup() .addComponent(valid) .addComponent(validCheckBox)) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(processingMode) + .addComponent(reprocessButton) + .addComponent(forwardProcessButton) .addGap(ZiggySwingUtils.GROUP_GAP) .addComponent(pipelineParameterSetsGroup) .addPreferredGap(ComponentPlacement.RELATED) @@ -269,10 +299,6 @@ private void editModuleParameters(ActionEvent evt) { final PipelineDefinitionNode pipelineNode = pipelineModulesListModel .getPipelineNodeAt(selectedRow); - PipelineOperationsProxy pipelineOps = new PipelineOperationsProxy(); - Set> allRequiredParams = pipelineOps - .retrieveRequiredParameterClassesForNode(pipelineNode); - Map, String> currentModuleParams = pipelineNode .getModuleParameterSetNames(); Map, String> currentPipelineParams = pipeline @@ -280,8 +306,7 @@ private void editModuleParameters(ActionEvent evt) { try { final ModuleParameterSetMapEditorDialog dialog = new ModuleParameterSetMapEditorDialog( - this, currentModuleParams, allRequiredParams, currentPipelineParams, - editedParameterSets); + this, currentModuleParams, currentPipelineParams, editedParameterSets); dialog.setMapListener(source -> pipelineNode .setModuleParameterSetNames(dialog.getParameterSetsMap())); @@ -300,7 +325,7 @@ private void save(ActionEvent evt) { try { String newName = pipelineNameTextField.getText(); - PipelineDefinition existingPipeline = pipelineModel.pipelineByName(newName); + PipelineDefinition existingPipeline = pipelineModel.objectByName(newName); if (existingPipeline != null && !newName.equals(pipeline.getName())) { // Operator changed pipeline name & it conflicts with an existing @@ -322,16 +347,24 @@ private void save(ActionEvent evt) { pipelineOperationsProxy.updateParameterSet(mapEntry.getKey(), mapEntry.getValue()); } - // Save any remote parameter sets that have been touched since the dialog box + // Save any pipeline definition nodes that have been touched since the dialog box // was opened. - for (Map.Entry mapEntry : editedRemoteParameterSets.entrySet()) { - pipelineOperationsProxy.updateParameterSet(mapEntry.getKey(), - mapEntry.getValue().parametersInstance()); + PipelineDefinitionNodeCrudProxy nodeProxy = new PipelineDefinitionNodeCrudProxy(); + for (PipelineDefinitionNodeExecutionResources executionResources : updatedExecutionResources + .values()) { + updatedExecutionResources.put(executionResources.getId(), + nodeProxy.merge(executionResources)); } + // Update the reprocess selection. + ProcessingMode processingMode = reprocessButton.isSelected() + ? ProcessingMode.PROCESS_ALL + : ProcessingMode.PROCESS_NEW; + new PipelineDefinitionCrudProxy().updateProcessingMode(newName, processingMode); + setVisible(false); } catch (Throwable e) { - MessageUtil.showError(this, "Error Saving Pipeline", e.getMessage(), e); + MessageUtil.showError(this, "Error saving pipeline", e.getMessage(), e); } } @@ -352,40 +385,56 @@ private void displayTaskInformation(ActionEvent evt) { } } - private void configureResources(ActionEvent evt) { - try { - new WorkerResourcesDialog(this, pipeline).setVisible(true); - } catch (Throwable e) { - MessageUtil.showError(this, e); + private void configureRemoteExecution(ActionEvent evt) { + PipelineDefinitionNode pipelineNode = prepNodeForResourcesUpdate(); + if (pipelineNode == null) { + return; } + + PipelineDefinitionNodeExecutionResources executionResources = updatedExecutionResources + .get(pipelineNode.getId()); + RemoteExecutionDialog remoteExecutionDialog = new RemoteExecutionDialog(this, + executionResources, pipelineNode, + PipelineTaskInformation.subtaskInformation(pipelineNode)); + remoteExecutionDialog.setVisible(true); + log.debug("original == current? {}", + executionResources.equals(remoteExecutionDialog.getCurrentConfiguration())); + executionResources.populateFrom(remoteExecutionDialog.getCurrentConfiguration()); } - private void configureRemoteExecution(ActionEvent evt) { + /** + * Helper method for locating a {@link PipelineDefinitionNode} instance from the dialog and + * retrieving (or creating) its {@link PipelineDefinitionNodeExecutionResources} instance. The + * pair are added to the map used to track nodes with updated resources. + */ + private PipelineDefinitionNode prepNodeForResourcesUpdate() { int selectedRow = modulesList.getSelectedIndex(); - if (selectedRow == -1) { MessageUtil.showError(this, "No module selected"); - } else { - final PipelineDefinitionNode pipelineNode = pipelineModulesListModel - .getPipelineNodeAt(selectedRow); - String remoteParametersName = PipelineTaskInformation.remoteParameters(pipelineNode); - if (remoteParametersName == null) { - JOptionPane.showMessageDialog(this, - "Selected node has no RemoteParameters instance"); - return; - } + return null; + } - // Retrieve the remote parameter set if it's not already in the edited parameters map. - if (!editedRemoteParameterSets.containsKey(remoteParametersName)) { - ParameterSet remoteParameterSet = new PipelineOperationsProxy() - .retrieveLatestParameterSet(remoteParametersName); - editedRemoteParameterSets.put(remoteParametersName, remoteParameterSet); - } - ParameterSet remoteParameters = editedRemoteParameterSets.get(remoteParametersName); + final PipelineDefinitionNode pipelineNode = pipelineModulesListModel + .getPipelineNodeAt(selectedRow); - new RemoteExecutionDialog(this, remoteParameters, pipelineNode, - PipelineTaskInformation.subtaskInformation(pipelineNode)).setVisible(true); + // Make sure the pipeline node is in the set of nodes with edited remote parameters. + if (!updatedExecutionResources.containsKey(pipelineNode.getId())) { + updatedExecutionResources.put(pipelineNode.getId(), + new PipelineDefinitionNodeCrudProxy() + .retrieveRemoteExecutionConfiguration(pipelineNode)); + } + return pipelineNode; + } + + private void configureResources(ActionEvent evt) { + PipelineDefinitionNode pipelineNode = prepNodeForResourcesUpdate(); + if (pipelineNode == null) { + return; } + PipelineDefinitionNodeExecutionResources executionResources = updatedExecutionResources + .get(pipelineNode.getId()); + new PipelineDefinitionNodeResourcesDialog(this, pipelineName, pipelineNode, + executionResources).setVisible(true); } /** diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java index 85253cc..9f0b698 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/EditPipelineNodeDialog.java @@ -38,6 +38,7 @@ */ @SuppressWarnings("serial") public class EditPipelineNodeDialog extends javax.swing.JDialog { + @SuppressWarnings("unused") private static final Logger log = LoggerFactory.getLogger(EditPipelineNodeDialog.class); private JLabel moduleLabel; @@ -73,7 +74,6 @@ public EditPipelineNodeDialog(Window owner, PipelineDefinition pipeline, private void save(ActionEvent evt) { try { - PipelineModuleDefinition selectedModule = (PipelineModuleDefinition) moduleComboBox .getSelectedItem(); pipelineNode.setPipelineModuleDefinition(selectedModule); @@ -244,7 +244,8 @@ private JPanel getUowPanel() { private JLabel getUowTypeLabel() { if (uowTypeLabel == null) { uowTypeLabel = new JLabel(); - ClassWrapper uowWrapper = pipelineNode.getUnitOfWorkGenerator(); + ClassWrapper uowWrapper = new PipelineModuleDefinitionCrudProxy() + .retrieveUnitOfWorkGenerator(pipelineNode.getModuleName()); String uowName = uowWrapper.getClassName(); String uowLabel = "Unit of Work Class: " + uowName; uowTypeLabel.setText(uowLabel); @@ -258,6 +259,7 @@ private JLabel getUowTypeLabel() { * * @param message */ + @SuppressWarnings("unused") private void setError(String message) { if (saveButton != null) { saveButton.setEnabled(message.isEmpty()); diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ModuleParameterSetMapEditorDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ModuleParameterSetMapEditorDialog.java index 70bbbfc..ea61198 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ModuleParameterSetMapEditorDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/ModuleParameterSetMapEditorDialog.java @@ -9,7 +9,6 @@ import java.awt.Window; import java.awt.event.ActionEvent; import java.util.Map; -import java.util.Set; import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; @@ -29,27 +28,24 @@ public class ModuleParameterSetMapEditorDialog extends javax.swing.JDialog public ModuleParameterSetMapEditorDialog(Window owner, Map, String> currentModuleParameters, - Set> requiredParameters, Map, String> currentPipelineParameters, Map editedParameterSets) { super(owner, DEFAULT_MODALITY_TYPE); - buildComponent(currentModuleParameters, requiredParameters, currentPipelineParameters, - editedParameterSets); + buildComponent(currentModuleParameters, currentPipelineParameters, editedParameterSets); setLocationRelativeTo(owner); } private void buildComponent( Map, String> currentModuleParameters, - Set> requiredParameters, Map, String> currentPipelineParameters, Map editedParameterSets) { setTitle("Edit parameter sets"); - getContentPane().add(createDataPanel(currentModuleParameters, requiredParameters, - currentPipelineParameters, editedParameterSets), BorderLayout.CENTER); + getContentPane().add(createDataPanel(currentModuleParameters, currentPipelineParameters, + editedParameterSets), BorderLayout.CENTER); getContentPane().add(createButtonPanel(createButton(CLOSE, this::close)), BorderLayout.SOUTH); @@ -59,12 +55,11 @@ private void buildComponent( private ParameterSetMapEditorPanel createDataPanel( Map, String> currentModuleParameters, - Set> requiredParameters, Map, String> currentPipelineParameters, Map editedParameterSets) { parameterSetMapEditorPanel = new ParameterSetMapEditorPanel(currentModuleParameters, - requiredParameters, currentPipelineParameters, editedParameterSets); + currentPipelineParameters, editedParameterSets); parameterSetMapEditorPanel.setMapListener(this); return parameterSetMapEditorPanel; diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java index eb56aa2..5802322 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/ParameterSetMapEditorPanel.java @@ -34,10 +34,10 @@ import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.ButtonPanelContext; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.proxy.ParameterSetCrudProxy; import gov.nasa.ziggy.ui.util.proxy.PipelineOperationsProxy; import gov.nasa.ziggy.ui.util.table.ZiggyTable; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * Edit/view all of the {@link ParameterSet}s for a pipeline or node. This panel is the one that @@ -56,7 +56,6 @@ public class ParameterSetMapEditorPanel extends javax.swing.JPanel { private int selectedModelIndex = -1; private Map, String> currentParameters; - private Set> requiredParameters; private Map, String> currentPipelineParameters; private Map editedParameterSets; @@ -64,11 +63,9 @@ public class ParameterSetMapEditorPanel extends javax.swing.JPanel { public ParameterSetMapEditorPanel( Map, String> currentParameters, - Set> requiredParameters, Map, String> currentPipelineParameters, Map editedParameterSets) { this.currentParameters = currentParameters; - this.requiredParameters = requiredParameters; this.currentPipelineParameters = currentPipelineParameters; this.editedParameterSets = editedParameterSets; @@ -88,7 +85,7 @@ private void buildComponent() { this::autoAssign)); paramSetMapTableModel = new ParameterSetNamesTableModel(currentParameters, - requiredParameters, currentPipelineParameters); + currentPipelineParameters); ziggyTable = new ZiggyTable<>(paramSetMapTableModel); JScrollPane parameterSets = new JScrollPane(ziggyTable.getTable()); parameterSets.setPreferredSize(new Dimension(0, 100)); @@ -133,8 +130,7 @@ private void add(ActionEvent evt) { mapListener.notifyMapChanged(this); } - paramSetMapTableModel.update(currentParameters, requiredParameters, - currentPipelineParameters); + paramSetMapTableModel.update(currentParameters, currentPipelineParameters); } } } @@ -185,8 +181,7 @@ private void select(int modelIndex) { mapListener.notifyMapChanged(this); } - paramSetMapTableModel.update(currentParameters, requiredParameters, - currentPipelineParameters); + paramSetMapTableModel.update(currentParameters, currentPipelineParameters); } } } else { @@ -277,8 +272,7 @@ private void autoAssign(ActionEvent evt) { mapListener.notifyMapChanged(this); } - paramSetMapTableModel.update(currentParameters, requiredParameters, - currentPipelineParameters); + paramSetMapTableModel.update(currentParameters, currentPipelineParameters); } } @@ -339,8 +333,7 @@ private void removeSelected(ActionEvent evt) { mapListener.notifyMapChanged(this); } - paramSetMapTableModel.update(currentParameters, requiredParameters, - currentPipelineParameters); + paramSetMapTableModel.update(currentParameters, currentPipelineParameters); } public ParameterSetMapEditorListener getMapListener() { @@ -357,7 +350,7 @@ public Map, String> getParameterSetsMap() { private class ParameterSetNamesTableModel extends AbstractZiggyTableModel - implements TableModelContentClass { + implements ModelContentClass { private static final String[] COLUMN_NAMES = { "Type", "Name" }; @@ -365,9 +358,8 @@ private class ParameterSetNamesTableModel public ParameterSetNamesTableModel( Map, String> currentParameters, - Set> requiredParameters, Map, String> currentPipelineParameters) { - update(currentParameters, requiredParameters, currentPipelineParameters); + update(currentParameters, currentPipelineParameters); } /** @@ -376,43 +368,11 @@ public ParameterSetNamesTableModel( * '(pipeline)' if there are any left in current params (not reqd), add those */ public void update(Map, String> currentParameters, - Set> requiredParameters, Map, String> currentPipelineParameters) { paramSetAssignments.clear(); Set> types = new HashSet<>(); - // for each required param type, create a ParameterSetAssignment - for (ClassWrapper requiredType : requiredParameters) { - ParameterSetAssignment param = new ParameterSetAssignment(requiredType); - - // if required param type exists in current params, use that - // ParameterSetName - String currentAssignment = currentParameters.get(requiredType); - if (currentAssignment != null) { - param.setAssignedName(currentAssignment); - } - - // if required param type exists in current *pipeline* params, - // display that (read-only) - if (currentPipelineParameters.containsKey(requiredType)) { - param.setAssignedName(currentPipelineParameters.get(requiredType)); - param.setAssignedAtPipelineLevel(true); - - if (currentAssignment != null) { - param.setAssignedAtBothLevels(true); - } - } - - if (param.isAssignedAtPipelineLevel() || param.isAssignedAtBothLevels()) { - paramSetAssignments.addFirst(param); - } else { - paramSetAssignments.add(param); - } - - types.add(requiredType); - } - // If there are any param types left over in current params (not required), add those. // This also covers the case where empty lists are passed in for required params and // current pipeline params (when using this model to edit pipeline params on the diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java index b0117f1..37fec49 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineDefinitionNodeResourcesDialog.java @@ -28,11 +28,13 @@ import javax.swing.text.NumberFormatter; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.services.messages.WorkerResources; -import gov.nasa.ziggy.ui.util.HumanReadableHeapSize; -import gov.nasa.ziggy.ui.util.HumanReadableHeapSize.HeapSizeUnit; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; +import gov.nasa.ziggy.ui.ZiggyGuiConsole; import gov.nasa.ziggy.ui.util.ValidityTestingFormattedTextField; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; +import gov.nasa.ziggy.util.HumanReadableHeapSize; +import gov.nasa.ziggy.util.HumanReadableHeapSize.HeapSizeUnit; +import gov.nasa.ziggy.worker.WorkerResources; /** * Dialog box for editing the worker count and heap size for a {@link PipelineDefinitionNode}. @@ -58,9 +60,12 @@ */ public class PipelineDefinitionNodeResourcesDialog extends JDialog { - private static final long serialVersionUID = 20230810L; + private static final long serialVersionUID = 20231212L; + + private static final int COLUMNS = 10; private final PipelineDefinitionNode node; + private final PipelineDefinitionNodeExecutionResources executionResources; private final String pipelineDefinitionName; private final WorkerResources initialResources; private JCheckBox workerDefaultCheckBox; @@ -70,9 +75,11 @@ public class PipelineDefinitionNodeResourcesDialog extends JDialog { private JRadioButton tbUnitsButton; private ValidityTestingFormattedTextField workerCountTextArea; private ValidityTestingFormattedTextField heapSizeTextArea; + private ValidityTestingFormattedTextField maxFailedSubtaskTextArea; + private ValidityTestingFormattedTextField maxAutoResubmitsTextArea; private ButtonGroup unitButtonGroup; - private int workerCountCurrentUserValue; - private int heapSizeMbCurrentUserValue; + private Integer workerCountCurrentUserValue; + private Integer heapSizeMbCurrentUserValue; private JButton closeButton; private JButton cancelButton; @@ -82,12 +89,13 @@ public class PipelineDefinitionNodeResourcesDialog extends JDialog { private Consumer validityCheck = valid -> setCloseButtonState(); public PipelineDefinitionNodeResourcesDialog(Window owner, String pipelineDefinitionName, - PipelineDefinitionNode node) { + PipelineDefinitionNode node, PipelineDefinitionNodeExecutionResources executionResources) { super(owner, DEFAULT_MODALITY_TYPE); this.pipelineDefinitionName = pipelineDefinitionName; this.node = node; + this.executionResources = executionResources; - initialResources = node.workerResources(); + initialResources = executionResources.workerResources(); workerCountCurrentUserValue = initialResources.getMaxWorkerCount(); heapSizeMbCurrentUserValue = initialResources.getHeapSizeMb(); @@ -123,6 +131,12 @@ private JPanel createDataPanel() { heapSizeTextArea = createHeapSizeTextArea(); heapSizeDefaultCheckBox = createHeapSizeDefaultCheckBox(); + JLabel maxFailedSubtasks = boldLabel("Maximum failed subtasks"); + maxFailedSubtaskTextArea = createMaxFailedSubtaskTextArea(); + + JLabel maxAutoResubmits = boldLabel("Maximum automatic resubmits"); + maxAutoResubmitsTextArea = createMaxAutoResubmitsTextArea(); + unitButtonGroup = new ButtonGroup(); mbUnitsButton = new JRadioButton("MB"); mbUnitsButton.setToolTipText("Set heap size units to megabytes."); @@ -158,20 +172,26 @@ private JPanel createDataPanel() { .addComponent(gbUnitsButton) .addPreferredGap(ComponentPlacement.RELATED) .addComponent(tbUnitsButton)) - .addComponent(heapSizeDefaultCheckBox)); + .addComponent(heapSizeDefaultCheckBox) + .addComponent(maxFailedSubtasks) + .addComponent(maxFailedSubtaskTextArea, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + .addComponent(maxAutoResubmits) + .addComponent(maxAutoResubmitsTextArea, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)); dataPanelLayout.setVerticalGroup(dataPanelLayout.createSequentialGroup() .addComponent(pipeline) .addComponent(pipelineText) - .addPreferredGap(ComponentPlacement.UNRELATED) + .addPreferredGap(ComponentPlacement.RELATED) .addComponent(module) .addComponent(moduleText) - .addPreferredGap(ComponentPlacement.UNRELATED) + .addPreferredGap(ComponentPlacement.RELATED) .addComponent(maxWorkers) .addComponent(workerCountTextArea, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addComponent(workerDefaultCheckBox) - .addPreferredGap(ComponentPlacement.UNRELATED) + .addPreferredGap(ComponentPlacement.RELATED) .addComponent(maxHeapSize) .addGroup(dataPanelLayout.createParallelGroup(GroupLayout.Alignment.CENTER) .addComponent(heapSizeTextArea, GroupLayout.PREFERRED_SIZE, @@ -179,7 +199,13 @@ private JPanel createDataPanel() { .addComponent(mbUnitsButton) .addComponent(gbUnitsButton) .addComponent(tbUnitsButton)) - .addComponent(heapSizeDefaultCheckBox)); + .addComponent(heapSizeDefaultCheckBox) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(maxFailedSubtasks) + .addComponent(maxFailedSubtaskTextArea) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(maxAutoResubmits) + .addComponent(maxAutoResubmitsTextArea)); return dataPanel; } @@ -217,7 +243,9 @@ public void focusGained(FocusEvent e) { /** * Capture the current state of worker resource parameters in the relevant - * {@link PipelineDefinitionNode} instance and close the dialog box. + * {@link PipelineDefinitionNode} instance and close the dialog box. Resources are only captured + * in cases where the relevant "use default" dialog box is not checked (i.e., places where the + * user wants default values are set to null). */ private void updateWorkerResources(AWTEvent evt) { Integer finalWorkerCount = null; @@ -228,7 +256,10 @@ private void updateWorkerResources(AWTEvent evt) { if (!heapSizeDefaultCheckBox.isSelected()) { finalHeapSizeMb = heapSizeMbFromTextField(); } - node.applyWorkerResources(new WorkerResources(finalWorkerCount, finalHeapSizeMb)); + executionResources + .applyWorkerResources(new WorkerResources(finalWorkerCount, finalHeapSizeMb)); + executionResources.setMaxFailedSubtaskCount((Integer) maxFailedSubtaskTextArea.getValue()); + executionResources.setMaxAutoResubmits((Integer) maxAutoResubmitsTextArea.getValue()); dispose(); } @@ -276,7 +307,7 @@ private JButton createCancelButton() { * Reverts the worker resources to their initial values and exits. */ private void revertWorkerResources(ActionEvent evt) { - node.applyWorkerResources(initialResources); + executionResources.applyWorkerResources(initialResources); dispose(); } @@ -293,7 +324,7 @@ private ValidityTestingFormattedTextField createWorkerCountTextArea() { ValidityTestingFormattedTextField workerCountTextArea = new ValidityTestingFormattedTextField( formatter); workerCountTextArea.setEmptyIsValid(false); - workerCountTextArea.setColumns(10); + workerCountTextArea.setColumns(COLUMNS); workerCountTextArea .setToolTipText(htmlBuilder("Set the maximum number of worker processes.").appendBreak() .append("Must be between 1 and number of cores on your system (") @@ -313,8 +344,8 @@ private ValidityTestingFormattedTextField createWorkerCountTextArea() { */ private JCheckBox createWorkerDefaultCheckBox() { JCheckBox workerDefaultCheckBox = new JCheckBox( - "Default (" + WorkerResources.getDefaultResources().getMaxWorkerCount() + ")"); - workerDefaultCheckBox.setSelected(initialResources.maxWorkerCountIsDefault()); + "Default (" + ZiggyGuiConsole.defaultResources().getMaxWorkerCount() + ")"); + workerDefaultCheckBox.setSelected(workerCountCurrentUserValue == null); workerDefaultCheckBox.setToolTipText("Use the pipeline default worker count."); workerDefaultCheckBox.addItemListener(this::workerCheckBoxChanged); return workerDefaultCheckBox; @@ -334,9 +365,14 @@ private void workerCheckBoxChanged(ItemEvent evt) { workerCountTextArea.setValue(workerCountCurrentUserValue); } else { - // Switching from the user-set value to the default. - if (workerCountTextArea.isValidState()) { - workerCountCurrentUserValue = (int) workerCountTextArea.getValue(); + // Switching from the user-set value to the default. Update the current + // user value from the text area so that if the user changes their mind and + // deselects the default, the dialog box will "remember" what was in the + // worker count text box and put it back there. Note that we need to check both + // the valid state of the text box and whether it's null because a null value in + // a disabled text area is a valid state. + if (workerCountTextArea.isValidState() && workerCountTextArea.getValue() != null) { + workerCountCurrentUserValue = (Integer) workerCountTextArea.getValue(); } else { // Force the text field into a valid state. This is a kind of kludgey way // to do it, but I haven't been able to figure out any way for Java to @@ -361,7 +397,7 @@ private ValidityTestingFormattedTextField createHeapSizeTextArea() { ValidityTestingFormattedTextField heapSizeTextArea = new ValidityTestingFormattedTextField( formatter); heapSizeTextArea.setEmptyIsValid(false); - heapSizeTextArea.setColumns(10); + heapSizeTextArea.setColumns(COLUMNS); heapSizeTextArea.setToolTipText( htmlBuilder("Set the maximum Java heap size shared by all workers.").appendBreak() .append("Must be between 1 and 1000, inclusive.") @@ -377,8 +413,8 @@ private ValidityTestingFormattedTextField createHeapSizeTextArea() { */ private JCheckBox createHeapSizeDefaultCheckBox() { JCheckBox heapSizeDefaultCheckBox = new JCheckBox( - "Default (" + WorkerResources.getDefaultResources().humanReadableHeapSize() + ")"); - heapSizeDefaultCheckBox.setSelected(initialResources.heapSizeIsDefault()); + "Default (" + ZiggyGuiConsole.defaultResources().humanReadableHeapSize() + ")"); + heapSizeDefaultCheckBox.setSelected(heapSizeMbCurrentUserValue == null); heapSizeDefaultCheckBox.setToolTipText("Use the pipeline default heap size."); heapSizeDefaultCheckBox.addItemListener(this::heapSizeCheckBoxChanged); return heapSizeDefaultCheckBox; @@ -403,8 +439,10 @@ private void heapSizeCheckBoxChanged(ItemEvent evt) { setHeapSizeTextFromCurrentUserValue(); } else { - // Switching from the user-set heap size to the default value. - if (heapSizeTextArea.isValidState()) { + // Switching from the user-set heap size to the default value. Note that we + // have to check whether the text area is null because when the text area is + // both disabled and null, that constitutes a valid state. + if (heapSizeTextArea.isValidState() && heapSizeTextArea.getValue() != null) { heapSizeMbCurrentUserValue = heapSizeMbFromTextField(); } else { // Force the text field into a valid state. This is a kind of kludgey way @@ -426,7 +464,13 @@ private void heapSizeCheckBoxChanged(ItemEvent evt) { */ private void setHeapSizeTextFromCurrentUserValue() { HumanReadableHeapSize humanReadableHeapSize = new HumanReadableHeapSize( - heapSizeMbCurrentUserValue); + ZiggyGuiConsole.defaultResources().getHeapSizeMb()); + if (heapSizeMbCurrentUserValue == null) { + heapSizeTextArea.setValue(null); + } else { + humanReadableHeapSize = new HumanReadableHeapSize(heapSizeMbCurrentUserValue); + heapSizeTextArea.setValue((double) humanReadableHeapSize.getHumanReadableHeapSize()); + } switch (humanReadableHeapSize.getHeapSizeUnit()) { case MB: mbUnitsButton.setSelected(true); @@ -438,7 +482,45 @@ private void setHeapSizeTextFromCurrentUserValue() { tbUnitsButton.setSelected(true); break; } - heapSizeTextArea.setValue((double) humanReadableHeapSize.getHumanReadableHeapSize()); + } + + private ValidityTestingFormattedTextField createMaxFailedSubtaskTextArea() { + NumberFormatter formatter = new NumberFormatter(NumberFormat.getInstance()); + formatter.setValueClass(Integer.class); + formatter.setMinimum(0); + ValidityTestingFormattedTextField maxFailedSubtaskTextArea = new ValidityTestingFormattedTextField( + formatter); + maxFailedSubtaskTextArea.setEmptyIsValid(false); + maxFailedSubtaskTextArea.setColumns(COLUMNS); + maxFailedSubtaskTextArea + .setToolTipText(htmlBuilder("Set the maximum number of failed subtasks.").appendBreak() + .append( + "This allows tasks to report successful completion if some subtasks failed.") + .toString()); + maxFailedSubtaskTextArea + .setText(Integer.toString(executionResources.getMaxFailedSubtaskCount())); + maxFailedSubtaskTextArea.setExecuteOnValidityCheck(validityCheck); + return maxFailedSubtaskTextArea; + } + + private ValidityTestingFormattedTextField createMaxAutoResubmitsTextArea() { + NumberFormatter formatter = new NumberFormatter(NumberFormat.getInstance()); + formatter.setValueClass(Integer.class); + formatter.setMinimum(0); + ValidityTestingFormattedTextField maxFailedSubtaskTextArea = new ValidityTestingFormattedTextField( + formatter); + maxFailedSubtaskTextArea.setEmptyIsValid(false); + maxFailedSubtaskTextArea.setColumns(COLUMNS); + maxFailedSubtaskTextArea.setToolTipText( + htmlBuilder("Set the maximum number of automatic resubmits of failed tasks.") + .appendBreak() + .append( + "This allows tasks that have failed to automatically resubmit execution of failed or missed subtasks.") + .toString()); + maxFailedSubtaskTextArea + .setText(Integer.toString(executionResources.getMaxAutoResubmits())); + maxFailedSubtaskTextArea.setExecuteOnValidityCheck(validityCheck); + return maxFailedSubtaskTextArea; } /** diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java index 80a8c01..aca304f 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/PipelineNodeWidget.java @@ -12,6 +12,7 @@ import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; +import gov.nasa.ziggy.ui.util.proxy.PipelineModuleDefinitionCrudProxy; /** * @author Todd Klaus @@ -81,7 +82,10 @@ private JLabel getLabel() { } else { String uowtgShortName = "-"; try { - uowtgShortName = pipelineNode.getUnitOfWorkGenerator().newInstance().toString(); + uowtgShortName = new PipelineModuleDefinitionCrudProxy() + .retrieveUnitOfWorkGenerator(pipelineNode.getModuleName()) + .newInstance() + .toString(); } catch (Exception e) { } label.setText(pipelineNode.getModuleName() + " (" + uowtgShortName + ")"); diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java index bbcabe7..bfcae11 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/RemoteExecutionDialog.java @@ -10,10 +10,11 @@ import java.awt.BorderLayout; import java.awt.Window; import java.awt.event.ActionEvent; -import java.awt.event.FocusAdapter; -import java.awt.event.FocusEvent; +import java.awt.event.HierarchyEvent; import java.awt.event.ItemEvent; +import java.text.MessageFormat; import java.text.NumberFormat; +import java.text.ParseException; import java.util.HashSet; import java.util.List; import java.util.Set; @@ -27,7 +28,10 @@ import javax.swing.JLabel; import javax.swing.JOptionPane; import javax.swing.JPanel; +import javax.swing.JTextArea; import javax.swing.LayoutStyle.ComponentPlacement; +import javax.swing.SwingUtilities; +import javax.swing.text.DefaultFormatter; import javax.swing.text.NumberFormatter; import org.apache.commons.lang3.StringUtils; @@ -43,12 +47,15 @@ import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.module.remote.RemoteQueueDescriptor; import gov.nasa.ziggy.pipeline.PipelineTaskInformation; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; import gov.nasa.ziggy.ui.util.ValidityTestingFormattedTextField; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.ButtonPanelContext; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.LabelType; +import gov.nasa.ziggy.ui.util.proxy.PipelineDefinitionCrudProxy; +import gov.nasa.ziggy.util.TimeFormatter; /** * Dialog box that allows the user to set values in an instance of {@link RemoteParameters} and @@ -59,13 +66,14 @@ */ public class RemoteExecutionDialog extends JDialog { - @SuppressWarnings("unused") - private static Logger log = LoggerFactory.getLogger(RemoteExecutionDialog.class); + private static final Logger log = LoggerFactory.getLogger(RemoteExecutionDialog.class); - /** Minimum width to ensure {@code pack()} allows for the initially empty fields. */ - private static final int PBS_PARAMETERS_MINIMUM_WIDTH = 100; + private static final long serialVersionUID = 20240111L; + + private static final int COLUMNS = 6; - private static final long serialVersionUID = 20230927L; + /** Minimum width to ensure {@code pack()} allows for the initially empty fields. */ + private static final int PBS_PARAMETERS_MINIMUM_WIDTH = 120; // Reserved queue name: this name is a static variable so that it's "sticky," // i.e., once the user sets a reserved queue name it sticks around until the user @@ -74,29 +82,37 @@ public class RemoteExecutionDialog extends JDialog { private static String reservedQueueName = ""; // Data model - private RemoteParameters originalParameters; - private RemoteParameters currentParameters; + private PipelineDefinitionNodeExecutionResources originalConfiguration; + private PipelineDefinitionNodeExecutionResources currentConfiguration; private PipelineDefinitionNode node; + private int taskCount; + private int originalTaskCount; private int subtaskCount; - private int maxParallelSubtaskCount; - private ParameterSet parameterSet; private int originalSubtaskCount; - private int originalMaxParallelSubtaskCount; private List tasksInformation; // Dialog box elements + private ValidityTestingFormattedTextField tasksField; private ValidityTestingFormattedTextField subtasksField; private ValidityTestingFormattedTextField gigsPerSubtaskField; private ValidityTestingFormattedTextField maxWallTimeField; - private ValidityTestingFormattedTextField wallTimeRatioField; + private ValidityTestingFormattedTextField typicalWallTimeField; private JComboBox optimizerComboBox; private JComboBox architectureComboBox; + private RemoteNodeDescriptor lastArchitectureComboBoxSelection; + private JLabel architectureLimits; private JComboBox queueComboBox; + private RemoteQueueDescriptor lastQueueComboBoxSelection; + private JLabel queueName; + private ValidityTestingFormattedTextField queueNameField; + private JLabel queueLimits; private ValidityTestingFormattedTextField maxNodesField; private ValidityTestingFormattedTextField subtasksPerCoreField; - private JButton calculateButton; - private JCheckBox nodeSharingCheckBox; + private ValidityTestingFormattedTextField minSubtasksRemoteExecutionField; + private JCheckBox oneSubtaskCheckBox; private JCheckBox wallTimeScalingCheckBox; + private JCheckBox remoteExecutionEnabledCheckBox; + private JButton closeButton; private JLabel pbsArch; private JLabel pbsQueue; @@ -104,41 +120,53 @@ public class RemoteExecutionDialog extends JDialog { private JLabel pbsNodeCount; private JLabel pbsActiveCoresPerNode; private JLabel pbsCost; + private JTextArea pbsLimits; private Set validityTestingFormattedTextFields = new HashSet<>(); - private Consumer checkFieldsAndEnableButtons = valid -> setButtonState(); + private Consumer checkFieldsAndRecalculate = this::checkFieldsAndRecalculate; + private boolean skipCheck; + private boolean pbsParametersValid = true; - public RemoteExecutionDialog(Window owner, ParameterSet originalParameterSet, - PipelineDefinitionNode node, List tasksInformation) { + public RemoteExecutionDialog(Window owner, + PipelineDefinitionNodeExecutionResources originalConfiguration, PipelineDefinitionNode node, + List tasksInformation) { super(owner, DEFAULT_MODALITY_TYPE); - parameterSet = originalParameterSet; - originalParameters = originalParameterSet.parametersInstance(); - currentParameters = new RemoteParameters(originalParameters); - this.node = node; + + // Note that the current configuration is a copy of the original. If the user elects + // to reset the configuration, the current configuration is repopulated from the + // original; if the user elects to close the dialog box, the original configuration is + // repopulated from the current one. This ensures that the configuration that is eventually + // saved from the Edit Pipeline dialog box is the one retrieved from the database, so it + // can be merged safely. + this.originalConfiguration = originalConfiguration; + currentConfiguration = new PipelineDefinitionNodeExecutionResources(originalConfiguration); this.tasksInformation = tasksInformation; + this.node = node; + + taskCount = tasksInformation.size(); + originalTaskCount = taskCount; for (SubtaskInformation taskInformation : tasksInformation) { subtaskCount += taskInformation.getSubtaskCount(); - maxParallelSubtaskCount += taskInformation.getMaxParallelSubtasks(); } originalSubtaskCount = subtaskCount; - originalMaxParallelSubtaskCount = maxParallelSubtaskCount; buildComponent(); setLocationRelativeTo(owner); + addHierarchyListener(this::hierarchyChanged); } private void buildComponent() { setTitle("Edit remote execution parameters"); getContentPane().add(createDataPanel(), BorderLayout.CENTER); - getContentPane().add(createButtonPanel( - createButton(CLOSE, htmlBuilder("Close this dialog box.").appendBreak() - .append( - "Your remote parameter changes won't be saved until you hit Save on the edit pipeline dialog.") - .toString(), this::updateRemoteParameters), - createButton(CANCEL, "Clear all changes made in this dialog box and close it.", - this::cancel)), + closeButton = createButton(CLOSE, htmlBuilder("Close this dialog box.").appendBreak() + .append( + "Your remote parameter changes won't be saved until you hit Save on the edit pipeline dialog.") + .toString(), this::close); + getContentPane().add( + createButtonPanel(closeButton, createButton(CANCEL, + "Clear all changes made in this dialog box and close it.", this::cancel)), BorderLayout.SOUTH); populateTextFieldsAndComboBoxes(); @@ -146,46 +174,59 @@ private void buildComponent() { } private JPanel createDataPanel() { - JPanel remoteParametersToolBar = createButtonPanel(ButtonPanelContext.TOOL_BAR, + JPanel executionResourcesToolBar = createButtonPanel(ButtonPanelContext.TOOL_BAR, createButton("Reset", "Reset RemoteParameters values to the values at the start of this dialog box.", this::resetAction), createButton("Display task info", "Display task and subtask information for this pipeline node.", this::displayTaskInformation)); - remoteParametersToolBar + executionResourcesToolBar .setToolTipText("Controls user-settable parameters for remote execution"); JLabel moduleGroup = boldLabel("Module", LabelType.HEADING1); JLabel pipeline = boldLabel("Pipeline"); - JLabel pipelineText = new JLabel(node.getPipelineName()); + ProcessingMode processingMode = new PipelineDefinitionCrudProxy() + .retrieveProcessingMode(originalConfiguration.getPipelineName()); + JLabel pipelineText = new JLabel(MessageFormat.format("{0} (processing {1} data)", + originalConfiguration.getPipelineName(), processingMode.toString())); JLabel module = boldLabel("Module"); - JLabel moduleText = new JLabel(node.getModuleName()); - - JLabel nodeSharingGroup = boldLabel("Node sharing", LabelType.HEADING1); - nodeSharingGroup.setToolTipText("Controls parallel processing of subtasks on each node."); - nodeSharingCheckBox = new JCheckBox("Node sharing"); - nodeSharingCheckBox - .setToolTipText("Enables concurrent processing of multiple subtasks on each node."); - nodeSharingCheckBox.addItemListener(this::nodeSharingCheckBoxEvent); - wallTimeScalingCheckBox = new JCheckBox("Wall time scaling"); + JLabel moduleText = new JLabel(originalConfiguration.getPipelineModuleName()); + + JLabel requiredRemoteParametersGroup = boldLabel("Required parameters", LabelType.HEADING1); + requiredRemoteParametersGroup + .setToolTipText("Parameters needed for PBS parameter calculation."); + + remoteExecutionEnabledCheckBox = new JCheckBox("Enable remote execution"); + remoteExecutionEnabledCheckBox.addItemListener(this::itemStateChanged); + remoteExecutionEnabledCheckBox.setToolTipText("Enables or disables remote execution."); + + oneSubtaskCheckBox = new JCheckBox("Run one subtask per node"); + oneSubtaskCheckBox + .setToolTipText("Disables concurrent processing of multiple subtasks on each node."); + oneSubtaskCheckBox.addItemListener(this::nodeSharingCheckBoxEvent); + + wallTimeScalingCheckBox = new JCheckBox("Scale wall time by number of cores"); wallTimeScalingCheckBox.setToolTipText( htmlBuilder("Scales subtask wall times inversely to the number of cores per node.") .appendBreak() - .append("Only enabled when node sharing is disabled.") + .append("Only enabled when running one subtask per node.") .toString()); - JLabel requiredRemoteParametersGroup = boldLabel("Required", LabelType.HEADING1); - requiredRemoteParametersGroup.setToolTipText("Parameters that must be set."); + JLabel tasks = boldLabel("Total tasks"); + tasksField = createIntegerField( + "Set the total number of tasks for the selected module (must be >= 1)."); + validityTestingFormattedTextFields.add(tasksField); JLabel subtasks = boldLabel("Total subtasks"); - subtasksField = createSubtasksField(); + subtasksField = createIntegerField( + "Set the total number of subtasks for the selected module (must be >= 1)."); validityTestingFormattedTextFields.add(subtasksField); JLabel gigsPerSubtask = boldLabel("Gigs per subtask"); - gigsPerSubtaskField = createGigsPerSubtaskField(); + gigsPerSubtaskField = createDoubleField(""); validityTestingFormattedTextFields.add(gigsPerSubtaskField); JLabel maxWallTime = boldLabel("Max subtask wall time"); @@ -193,13 +234,15 @@ private JPanel createDataPanel() { validityTestingFormattedTextFields.add(maxWallTimeField); JLabel typicalWallTime = boldLabel("Typical subtask wall time"); - wallTimeRatioField = createTypicalWallTimeField(); - validityTestingFormattedTextFields.add(wallTimeRatioField); + typicalWallTimeField = createDoubleField(Double.MIN_VALUE, + currentConfiguration.getSubtaskMaxWallTimeHours(), + "Enter the TYPICAL wall time needed by subtasks, in hours (>0, <= max wall time)"); + validityTestingFormattedTextFields.add(typicalWallTimeField); JLabel optimizer = boldLabel("Optimizer"); optimizerComboBox = createOptimizerComboBox(); - JLabel optionalRemoteParametersGroup = boldLabel("Optional", LabelType.HEADING1); + JLabel optionalRemoteParametersGroup = boldLabel("Optional parameters", LabelType.HEADING1); optionalRemoteParametersGroup.setToolTipText( htmlBuilder("Parameters that can be calculated if not set.").appendBreak() .append("Values set by users will be included when calculating PBS parameters.") @@ -208,26 +251,45 @@ private JPanel createDataPanel() { JLabel architecture = boldLabel("Architecture"); architectureComboBox = createArchitectureComboBox(); + architectureLimits = new JLabel(); + JLabel queue = boldLabel("Queue"); queueComboBox = createQueueComboBox(); + queueName = boldLabel("Reserved queue name"); + queueNameField = createQueueNameField(); + validityTestingFormattedTextFields.add(queueNameField); + + queueLimits = new JLabel(); + JLabel maxNodesPerTask = boldLabel("Max nodes per task"); - maxNodesField = createMaxNodesField(); + maxNodesField = createIntegerField(true, + htmlBuilder("Enter the maximum number of nodes to request for each task (>=1)") + .appendBreak() + .append("or leave empty to let the algorithm determine the number.") + .toString()); validityTestingFormattedTextFields.add(maxNodesField); JLabel subtasksPerCore = boldLabel("Subtasks per core"); - subtasksPerCoreField = createSubtasksPerCoreField(); + subtasksPerCoreField = createDoubleField(1.0, Double.MAX_VALUE, true, + htmlBuilder("Enter the number of subtasks per active core (>=1)").appendBreak() + .append("or leave empty to let the algorithm decide the number") + .toString()); validityTestingFormattedTextFields.add(subtasksPerCoreField); + JLabel minSubtasksRemoteExecution = boldLabel("Minimum subtasks for remote execution"); + minSubtasksRemoteExecutionField = createIntegerField(0, Integer.MAX_VALUE, true, + htmlBuilder( + "Enter the minimum number of subtasks that are required to use remote execution;") + .appendBreak() + .append( + "otherwise, the subtasks will be processed locally, even if Enable remote execution is checked.") + .toString()); + validityTestingFormattedTextFields.add(minSubtasksRemoteExecutionField); + JLabel pbsParametersGroup = boldLabel("PBS parameters", LabelType.HEADING1); pbsParametersGroup.setToolTipText("Displays parameters that will be sent to PBS."); - calculateButton = createButton("Calculate", - "Generate PBS parameters from the RemoteParameters values.", - this::calculatePbsParameters); - JPanel pbsParametersToolBar = createButtonPanel(ButtonPanelContext.TOOL_BAR, - calculateButton); - JLabel pbsArchLabel = boldLabel("Architecture:"); pbsArch = new JLabel(); @@ -246,13 +308,18 @@ private JPanel createDataPanel() { JLabel pbsCostLabel = boldLabel("Cost (SBUs):"); pbsCost = new JLabel(); + pbsLimits = new JTextArea(); + pbsLimits.setEditable(false); + pbsLimits.setLineWrap(true); + pbsLimits.setWrapStyleWord(true); + JPanel dataPanel = new JPanel(); GroupLayout dataPanelLayout = new GroupLayout(dataPanel); dataPanelLayout.setAutoCreateContainerGaps(true); dataPanel.setLayout(dataPanelLayout); dataPanelLayout.setHorizontalGroup(dataPanelLayout.createParallelGroup() - .addComponent(remoteParametersToolBar) + .addComponent(executionResourcesToolBar) .addGroup(dataPanelLayout.createSequentialGroup() .addGroup(dataPanelLayout.createParallelGroup() .addComponent(moduleGroup) @@ -263,16 +330,16 @@ private JPanel createDataPanel() { .addComponent(pipelineText) .addComponent(module) .addComponent(moduleText))) - .addComponent(nodeSharingGroup) - .addGroup(dataPanelLayout.createSequentialGroup() - .addGap(ZiggySwingUtils.INDENT) - .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(nodeSharingCheckBox) - .addComponent(wallTimeScalingCheckBox))) .addComponent(requiredRemoteParametersGroup) .addGroup(dataPanelLayout.createSequentialGroup() .addGap(ZiggySwingUtils.INDENT) .addGroup(dataPanelLayout.createParallelGroup() + .addComponent(oneSubtaskCheckBox) + .addComponent(wallTimeScalingCheckBox) + .addComponent(remoteExecutionEnabledCheckBox) + .addComponent(tasks) + .addComponent(tasksField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addComponent(subtasks) .addComponent(subtasksField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) @@ -283,57 +350,75 @@ private JPanel createDataPanel() { .addComponent(maxWallTimeField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addComponent(typicalWallTime) - .addComponent(wallTimeRatioField, GroupLayout.PREFERRED_SIZE, + .addComponent(typicalWallTimeField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addComponent(optimizer) .addComponent(optimizerComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE))) - .addComponent(optionalRemoteParametersGroup) + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)))) + .addGroup(dataPanelLayout.createParallelGroup() + .addComponent(pbsParametersGroup) .addGroup(dataPanelLayout.createSequentialGroup() .addGap(ZiggySwingUtils.INDENT) + .addGroup(dataPanelLayout.createParallelGroup() + .addGroup(dataPanelLayout.createSequentialGroup() + .addGroup(dataPanelLayout.createParallelGroup() + .addComponent(pbsArchLabel) + .addComponent(pbsQueueLabel) + .addComponent(pbsWallTimeLabel) + .addComponent(pbsNodeCountLabel) + .addComponent(pbsActiveCoresPerNodeLabel) + .addComponent(pbsCostLabel)) + .addPreferredGap(ComponentPlacement.RELATED) + .addGroup(dataPanelLayout.createParallelGroup() + .addComponent(pbsArch, PBS_PARAMETERS_MINIMUM_WIDTH, + GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) + .addComponent(pbsQueue, PBS_PARAMETERS_MINIMUM_WIDTH, + GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) + .addComponent(pbsWallTime, PBS_PARAMETERS_MINIMUM_WIDTH, + GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) + .addComponent(pbsNodeCount, PBS_PARAMETERS_MINIMUM_WIDTH, + GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) + .addComponent(pbsActiveCoresPerNode, + PBS_PARAMETERS_MINIMUM_WIDTH, GroupLayout.DEFAULT_SIZE, + GroupLayout.DEFAULT_SIZE) + .addComponent(pbsCost, PBS_PARAMETERS_MINIMUM_WIDTH, + GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE))) + .addComponent(pbsLimits))))) + .addComponent(optionalRemoteParametersGroup) + .addGroup(dataPanelLayout.createSequentialGroup() + .addGap(ZiggySwingUtils.INDENT) + .addGroup(dataPanelLayout.createParallelGroup() + .addGroup(dataPanelLayout.createSequentialGroup() .addGroup(dataPanelLayout.createParallelGroup() .addComponent(architecture) .addComponent(architectureComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(architectureLimits)) + .addGroup(dataPanelLayout.createSequentialGroup() + .addGroup(dataPanelLayout.createParallelGroup() .addComponent(queue) .addComponent(queueComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addComponent(maxNodesPerTask) - .addComponent(maxNodesField, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addComponent(subtasksPerCore) - .addComponent(subtasksPerCoreField, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)))) - .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(pbsParametersGroup) - .addGroup(dataPanelLayout.createSequentialGroup() - .addGap(ZiggySwingUtils.INDENT) - .addGroup(dataPanelLayout.createSequentialGroup() - .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(pbsParametersToolBar) - .addComponent(pbsArchLabel) - .addComponent(pbsQueueLabel) - .addComponent(pbsWallTimeLabel) - .addComponent(pbsNodeCountLabel) - .addComponent(pbsActiveCoresPerNodeLabel) - .addComponent(pbsCostLabel)) - .addPreferredGap(ComponentPlacement.RELATED) - .addGroup(dataPanelLayout.createParallelGroup() - .addComponent(pbsArch, PBS_PARAMETERS_MINIMUM_WIDTH, - GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) - .addComponent(pbsQueue, PBS_PARAMETERS_MINIMUM_WIDTH, - GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) - .addComponent(pbsWallTime, PBS_PARAMETERS_MINIMUM_WIDTH, - GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) - .addComponent(pbsNodeCount, PBS_PARAMETERS_MINIMUM_WIDTH, - GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) - .addComponent(pbsActiveCoresPerNode, PBS_PARAMETERS_MINIMUM_WIDTH, - GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE) - .addComponent(pbsCost, PBS_PARAMETERS_MINIMUM_WIDTH, - GroupLayout.DEFAULT_SIZE, GroupLayout.DEFAULT_SIZE))))))); + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) + .addPreferredGap(ComponentPlacement.RELATED) + .addGroup(dataPanelLayout.createParallelGroup() + .addComponent(queueName) + .addComponent(queueNameField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(queueLimits)) + .addComponent(maxNodesPerTask) + .addComponent(maxNodesField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + .addComponent(subtasksPerCore) + .addComponent(subtasksPerCoreField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + .addComponent(minSubtasksRemoteExecution) + .addComponent(minSubtasksRemoteExecutionField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)))); dataPanelLayout.setVerticalGroup(dataPanelLayout.createSequentialGroup() - .addComponent(remoteParametersToolBar, GroupLayout.PREFERRED_SIZE, + .addComponent(executionResourcesToolBar, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addPreferredGap(ComponentPlacement.RELATED) .addGroup(dataPanelLayout.createParallelGroup() @@ -346,12 +431,15 @@ private JPanel createDataPanel() { .addComponent(module) .addComponent(moduleText) .addGap(ZiggySwingUtils.GROUP_GAP) - .addComponent(nodeSharingGroup) + .addComponent(requiredRemoteParametersGroup) .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(nodeSharingCheckBox) + .addComponent(remoteExecutionEnabledCheckBox) + .addComponent(oneSubtaskCheckBox) .addComponent(wallTimeScalingCheckBox) - .addGap(ZiggySwingUtils.GROUP_GAP) - .addComponent(requiredRemoteParametersGroup) + .addPreferredGap(ComponentPlacement.UNRELATED) + .addComponent(tasks) + .addComponent(tasksField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, + GroupLayout.PREFERRED_SIZE) .addPreferredGap(ComponentPlacement.RELATED) .addComponent(subtasks) .addComponent(subtasksField, GroupLayout.PREFERRED_SIZE, @@ -366,36 +454,15 @@ private JPanel createDataPanel() { GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addPreferredGap(ComponentPlacement.RELATED) .addComponent(typicalWallTime) - .addComponent(wallTimeRatioField, GroupLayout.PREFERRED_SIZE, + .addComponent(typicalWallTimeField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) .addPreferredGap(ComponentPlacement.RELATED) .addComponent(optimizer) .addComponent(optimizerComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addGap(ZiggySwingUtils.GROUP_GAP) - .addComponent(optionalRemoteParametersGroup) - .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(architecture) - .addComponent(architectureComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(queue) - .addComponent(queueComboBox, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(maxNodesPerTask) - .addComponent(maxNodesField, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(subtasksPerCore) - .addComponent(subtasksPerCoreField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) .addGroup(dataPanelLayout.createSequentialGroup() .addComponent(pbsParametersGroup) .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(pbsParametersToolBar, GroupLayout.PREFERRED_SIZE, - GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) - .addPreferredGap(ComponentPlacement.RELATED) .addGroup(dataPanelLayout.createParallelGroup() .addGroup(dataPanelLayout.createSequentialGroup() .addComponent(pbsArchLabel) @@ -420,107 +487,188 @@ private JPanel createDataPanel() { .addPreferredGap(ComponentPlacement.RELATED) .addComponent(pbsActiveCoresPerNode) .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(pbsCost)))))); + .addComponent(pbsCost))) + .addPreferredGap(ComponentPlacement.UNRELATED) + .addComponent(pbsLimits))) + .addGap(ZiggySwingUtils.GROUP_GAP) + .addComponent(optionalRemoteParametersGroup) + .addPreferredGap(ComponentPlacement.RELATED) + .addGroup(dataPanelLayout.createParallelGroup() + .addGroup(dataPanelLayout.createSequentialGroup() + .addComponent(architecture) + .addComponent(architectureComboBox, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) + .addGroup(dataPanelLayout.createSequentialGroup() + .addPreferredGap(ComponentPlacement.RELATED, GroupLayout.PREFERRED_SIZE, + Short.MAX_VALUE) + .addComponent(architectureLimits))) + .addPreferredGap(ComponentPlacement.RELATED) + .addGroup(dataPanelLayout.createParallelGroup() + .addGroup(dataPanelLayout.createSequentialGroup() + .addComponent(queue) + .addComponent(queueComboBox, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) + .addGroup(dataPanelLayout.createSequentialGroup() + .addComponent(queueName) + .addComponent(queueNameField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)) + .addGroup(dataPanelLayout.createSequentialGroup() + .addPreferredGap(ComponentPlacement.RELATED, GroupLayout.PREFERRED_SIZE, + Short.MAX_VALUE) + .addComponent(queueLimits))) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(maxNodesPerTask) + .addComponent(maxNodesField, GroupLayout.PREFERRED_SIZE, GroupLayout.DEFAULT_SIZE, + GroupLayout.PREFERRED_SIZE) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(subtasksPerCore) + .addComponent(subtasksPerCoreField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE) + .addPreferredGap(ComponentPlacement.RELATED) + .addComponent(minSubtasksRemoteExecution) + .addComponent(minSubtasksRemoteExecutionField, GroupLayout.PREFERRED_SIZE, + GroupLayout.DEFAULT_SIZE, GroupLayout.PREFERRED_SIZE)); return dataPanel; } - private void setButtonState() { + private void itemStateChanged(ItemEvent evt) { + if (evt.getStateChange() == ItemEvent.SELECTED + || evt.getSource() == remoteExecutionEnabledCheckBox) { + checkFieldsAndRecalculate(null); + } + } + + // Validation is postponed until the dialog is visible. + private void hierarchyChanged(HierarchyEvent evt) { + if ((HierarchyEvent.SHOWING_CHANGED & evt.getChangeFlags()) != 0 && isVisible()) { + checkFieldsAndRecalculate(null); + } + } + + private void checkFieldsAndRecalculate(Boolean valid) { + log.debug("valid={}, visible={}, skipCheck={}", valid, isVisible(), skipCheck); + if (!isVisible() || skipCheck) { + return; + } + + if (allFieldsRequiredForCalculationValid()) { + pbsParametersValid = calculatePbsParameters(); + } else { + pbsParametersValid = false; + displayPbsValues(null); + } + closeButton.setEnabled(!remoteExecutionEnabledCheckBox.isSelected() + || allFieldsRequiredForCloseValid() && pbsParametersValid); + + log.debug( + "remoteExecutionEnabled={}, allFieldsRequiredForCalculationValid()={}, pbsParametersValid={}, allFieldsRequiredForCloseValid()={}", + remoteExecutionEnabledCheckBox.isSelected(), allFieldsRequiredForCloseValid(), + pbsParametersValid); + } + + /** + * Determines whether all the validity-checking fields are valid. Used to determine whether to + * calculate the PBS parameters. + */ + private boolean allFieldsRequiredForCalculationValid() { + return allFieldsRequiredForCloseValid() && tasksField.isValidState() + && subtasksField.isValidState(); + } + + /** + * Determines whether all the validity-checking fields are valid except for the subtask count. + * This is used to determine whether to enable the Close button, which is used to update the + * remote execution configuration when the dialog box closes. + */ + private boolean allFieldsRequiredForCloseValid() { boolean allFieldsValid = true; for (ValidityTestingFormattedTextField field : validityTestingFormattedTextFields) { - allFieldsValid = allFieldsValid && field.isValidState(); + if (!field.equals(tasksField) && !field.equals(subtasksField)) { + allFieldsValid = allFieldsValid && field.isValidState(); + } } - calculateButton.setEnabled(allFieldsValid); + return allFieldsValid; } - private ValidityTestingFormattedTextField createSubtasksField() { - NumberFormat numberFormat = NumberFormat.getInstance(); - NumberFormatter formatter = new NumberFormatter(numberFormat); + private ValidityTestingFormattedTextField createIntegerField(String toolTipText) { + return createIntegerField(false, toolTipText); + } + + private ValidityTestingFormattedTextField createIntegerField(boolean emptyIsValid, + String toolTipText) { + return createIntegerField(1, Integer.MAX_VALUE, emptyIsValid, toolTipText); + } + + private ValidityTestingFormattedTextField createIntegerField(int min, int max, + boolean emptyIsValid, String toolTipText) { + NumberFormatter formatter = new NumberFormatter(NumberFormat.getInstance()); formatter.setValueClass(Integer.class); - formatter.setMinimum(1); - formatter.setMaximum(Integer.MAX_VALUE); - ValidityTestingFormattedTextField subtasksField = new ValidityTestingFormattedTextField( + formatter.setMinimum(min); + formatter.setMaximum(max); + ValidityTestingFormattedTextField integerField = new ValidityTestingFormattedTextField( formatter); - subtasksField.setColumns(6); - subtasksField.setExecuteOnValidityCheck(checkFieldsAndEnableButtons); - subtasksField.setToolTipText( - "Set the total number of subtasks for the selected module (must be >= 1)."); + integerField.setColumns(COLUMNS); + integerField.setEmptyIsValid(emptyIsValid); + integerField.setToolTipText(toolTipText); + integerField.setExecuteOnValidityCheck(checkFieldsAndRecalculate); - return subtasksField; + return integerField; } - private void nodeSharingCheckBoxEvent(ItemEvent evt) { - wallTimeScalingCheckBox.setEnabled(!nodeSharingCheckBox.isSelected()); - gigsPerSubtaskField.setEnabled(nodeSharingCheckBox.isSelected()); - updateGigsPerSubtaskToolTip(); + private ValidityTestingFormattedTextField createDoubleField(String toolTipText) { + return createDoubleField(Double.MIN_VALUE, Double.MAX_VALUE, toolTipText); } - private ValidityTestingFormattedTextField createGigsPerSubtaskField() { - NumberFormat numberFormat = NumberFormat.getInstance(); - NumberFormatter formatter = new NumberFormatter(numberFormat); + private ValidityTestingFormattedTextField createDoubleField(double min, double max, + String toolTipText) { + return createDoubleField(min, max, false, toolTipText); + } + + private ValidityTestingFormattedTextField createDoubleField(double min, double max, + boolean emptyIsValid, String toolTipText) { + NumberFormatter formatter = new NumberFormatter(NumberFormat.getInstance()); formatter.setValueClass(Double.class); - formatter.setMinimum(Double.MIN_VALUE); - formatter.setMaximum(Double.MAX_VALUE); - ValidityTestingFormattedTextField gigsPerSubtaskField = new ValidityTestingFormattedTextField( + formatter.setMinimum(min); + formatter.setMaximum(max); + ValidityTestingFormattedTextField doubleField = new ValidityTestingFormattedTextField( formatter); - gigsPerSubtaskField.setColumns(6); - gigsPerSubtaskField.setExecuteOnValidityCheck(checkFieldsAndEnableButtons); - return gigsPerSubtaskField; + doubleField.setColumns(COLUMNS); + doubleField.setEmptyIsValid(emptyIsValid); + doubleField.setToolTipText(toolTipText); + doubleField.setExecuteOnValidityCheck(checkFieldsAndRecalculate); + + return doubleField; + } + + private void nodeSharingCheckBoxEvent(ItemEvent evt) { + wallTimeScalingCheckBox.setEnabled(oneSubtaskCheckBox.isSelected()); + gigsPerSubtaskField.setEnabled(!oneSubtaskCheckBox.isSelected()); + gigsPerSubtaskField.setToolTipText(gigsPerSubtaskToolTip()); } - private void updateGigsPerSubtaskToolTip() { + private String gigsPerSubtaskToolTip() { String enabledTip = "Enter the number of GB needed for each subtask (>0)."; - String disabledTip = "Gigs per subtask is disabled when node sharing is disabled."; - String tip = gigsPerSubtaskField.isEnabled() ? enabledTip : disabledTip; - gigsPerSubtaskField.setToolTipText(tip); + String disabledTip = "Gigs per subtask is disabled when running one subtask per node."; + return gigsPerSubtaskField.isEnabled() ? enabledTip : disabledTip; } private ValidityTestingFormattedTextField createMaxWallTimeField() { - NumberFormat numberFormat = NumberFormat.getInstance(); - NumberFormatter formatter = new NumberFormatter(numberFormat); - formatter.setValueClass(Double.class); - formatter.setMinimum(Double.MIN_VALUE); - formatter.setMaximum(Double.MAX_VALUE); - ValidityTestingFormattedTextField maxWallTimeField = new ValidityTestingFormattedTextField( - formatter); - maxWallTimeField.setColumns(6); - maxWallTimeField - .setToolTipText("Enter the MAXIMUM wall time needed by any subtask, in hours (>0)."); - maxWallTimeField.setExecuteOnValidityCheck(checkFieldsAndEnableButtons); - maxWallTimeField.addFocusListener(new FocusAdapter() { - - // Here is where we modify the typical field when the max field has been - // updated. - @Override - public void focusLost(FocusEvent e) { - if (maxWallTimeField.isValidState()) { - ValidityTestingFormattedTextField typicalField = wallTimeRatioField; - double maxValue = (double) maxWallTimeField.getValue(); - double typicalValue = (double) typicalField.getValue(); - NumberFormatter formatter = (NumberFormatter) typicalField.getFormatter(); - formatter.setMaximum(maxValue); - if (typicalValue > maxValue) { - typicalField.setBorder(ValidityTestingFormattedTextField.INVALID_BORDER); - } - } + maxWallTimeField = createDoubleField(Double.MIN_VALUE, Double.MAX_VALUE, + "Enter the MAXIMUM wall time needed by any subtask, in hours (>0)."); + + // Update the typical field when the max field has been updated. + maxWallTimeField.addPropertyChangeListener(evt -> { + if (!maxWallTimeField.isValidState()) { + return; } + double maxValue = (double) maxWallTimeField.getValue(); + double typicalValue = (double) typicalWallTimeField.getValue(); + ((NumberFormatter) typicalWallTimeField.getFormatter()).setMaximum(maxValue); + typicalWallTimeField.updateBorder(typicalValue <= maxValue); }); - return maxWallTimeField; - } - private ValidityTestingFormattedTextField createTypicalWallTimeField() { - NumberFormat numberFormat = NumberFormat.getInstance(); - NumberFormatter formatter = new NumberFormatter(numberFormat); - formatter.setValueClass(Double.class); - formatter.setMinimum(Double.MIN_VALUE); - formatter.setMaximum(currentParameters.getSubtaskMaxWallTimeHours()); - ValidityTestingFormattedTextField wallTimeRatioField = new ValidityTestingFormattedTextField( - formatter); - wallTimeRatioField.setColumns(6); - wallTimeRatioField.setToolTipText( - "Enter the TYPICAL wall time needed by subtasks, in hours (>0, <= max wall time)"); - wallTimeRatioField.setExecuteOnValidityCheck(checkFieldsAndEnableButtons); - return wallTimeRatioField; + return maxWallTimeField; } private JComboBox createOptimizerComboBox() { @@ -537,6 +685,7 @@ private JComboBox createOptimizerComboBox() { .appendBreak() .append(" Cost minimizes the number of SBUs.") .toString()); + optimizerComboBox.addItemListener(this::itemStateChanged); return optimizerComboBox; } @@ -544,129 +693,193 @@ private JComboBox createArchitectureComboBox() { JComboBox architectureComboBox = new JComboBox<>( RemoteNodeDescriptor.allDescriptors()); architectureComboBox.setToolTipText("Select remote node architecture."); + architectureComboBox.addItemListener(this::validateArchitectureComboBox); return architectureComboBox; } + private void validateArchitectureComboBox(ItemEvent evt) { + if (evt.getStateChange() == ItemEvent.DESELECTED || !isVisible() || skipCheck) { + return; + } + + pbsParametersValid = calculatePbsParameters(); + + // Reset combo box if the chosen value is invalid. Note that this method is also called when + // the dialog is first displayed, so if the configuration is invalid, simply set the last + // selection to the current one and let the warning dialog shown by + // handlePbsParametersException() guide the user. + if (pbsParametersValid || lastArchitectureComboBoxSelection == null) { + lastArchitectureComboBoxSelection = (RemoteNodeDescriptor) architectureComboBox + .getSelectedItem(); + } else { + architectureComboBox.setSelectedItem(lastArchitectureComboBoxSelection); + } + + if (lastArchitectureComboBoxSelection != RemoteNodeDescriptor.ANY) { + architectureLimits + .setText(MessageFormat.format("{0} cores, {1} GB/core, {2} fractional SBUs", + lastArchitectureComboBoxSelection.getMaxCores(), + lastArchitectureComboBoxSelection.getGigsPerCore(), + lastArchitectureComboBoxSelection.getCostFactor())); + } else { + architectureLimits.setText(""); + } + } + private JComboBox createQueueComboBox() { JComboBox queueComboBox = new JComboBox<>( RemoteQueueDescriptor.allDescriptors()); queueComboBox.setToolTipText("Select batch queue for use with these jobs."); - queueComboBox.addActionListener(this::validateQueueComboBox); + queueComboBox.addItemListener(this::validateQueueComboBox); return queueComboBox; } /** - * Prompts the user to enter a reserved queue name when setting the queue combo box to - * "reserved". Note that the input dialog won't allow the user to return to messing around with - * the rest of the remote execution dialog until a valid value is entered. + * Enables the queue name fields if RESERVED is selected, shows the queue limits, and populates + * the max nodes fields if DEBUG or DEVEL are selected. */ - private void validateQueueComboBox(ActionEvent evt) { - if (queueComboBox.getSelectedItem() == null - || queueComboBox.getSelectedItem() != RemoteQueueDescriptor.RESERVED) { + private void validateQueueComboBox(ItemEvent evt) { + if (evt.getStateChange() == ItemEvent.DESELECTED || !isVisible() || skipCheck) { return; } - String userReservedQueueName = ""; - while (!userReservedQueueName.startsWith("R")) { - userReservedQueueName = JOptionPane.showInputDialog( - "Enter name of Reserved Queue (must start with R)", reservedQueueName); + + boolean reservedQueue = queueComboBox.getSelectedItem() != null + && queueComboBox.getSelectedItem() == RemoteQueueDescriptor.RESERVED; + if (queueName.isEnabled() != reservedQueue) { + queueName.setEnabled(reservedQueue); + queueNameField.setEnabled(reservedQueue); } - reservedQueueName = userReservedQueueName; - } - private ValidityTestingFormattedTextField createMaxNodesField() { - NumberFormat numberFormat = NumberFormat.getInstance(); - NumberFormatter formatter = new NumberFormatter(numberFormat); - formatter.setValueClass(Integer.class); - formatter.setMinimum(1); - formatter.setMaximum(Integer.MAX_VALUE); - ValidityTestingFormattedTextField maxNodesField = new ValidityTestingFormattedTextField( - formatter); - maxNodesField.setColumns(6); - maxNodesField.setToolTipText( - htmlBuilder("Enter the maximum number of nodes to request for each task (>=1)") - .appendBreak() - .append("or leave empty to let the algorithm determine the number.") - .toString()); - maxNodesField.setEmptyIsValid(true); - maxNodesField.setExecuteOnValidityCheck(checkFieldsAndEnableButtons); - return maxNodesField; + pbsParametersValid = calculatePbsParameters(); + + // Reset combo box if the chosen value is invalid, but allow the user to enter a reserved + // queue name. + if (pbsParametersValid || reservedQueue) { + lastQueueComboBoxSelection = (RemoteQueueDescriptor) queueComboBox.getSelectedItem(); + } else { + queueComboBox.setSelectedItem(lastQueueComboBoxSelection); + } + + // Show queue limits. + if (lastQueueComboBoxSelection.getMaxWallTimeHours() > 0 + && lastQueueComboBoxSelection.getMaxWallTimeHours() < Double.MAX_VALUE) { + queueLimits.setText(MessageFormat.format("{0} hrs max wall time", + lastQueueComboBoxSelection.getMaxWallTimeHours())); + } else { + queueLimits.setText(""); + } + + // Populate maxNodes field if applicable. + int maxNodes = ((RemoteQueueDescriptor) queueComboBox.getSelectedItem()).getMaxNodes(); + if (maxNodes > 0 && maxNodes < Integer.MAX_VALUE) { + maxNodesField.setValue(maxNodes); + } else if (originalConfiguration.getMaxNodes() > 0) { + maxNodesField.setValue(originalConfiguration.getMaxNodes()); + } else { + maxNodesField.setValue(null); + } } - private ValidityTestingFormattedTextField createSubtasksPerCoreField() { - NumberFormat numberFormat = NumberFormat.getInstance(); - NumberFormatter formatter = new NumberFormatter(numberFormat); - formatter.setValueClass(Double.class); - formatter.setMinimum(1.0); - formatter.setMaximum(Double.MAX_VALUE); - ValidityTestingFormattedTextField subtasksPerCoreField = new ValidityTestingFormattedTextField( - formatter); - subtasksPerCoreField.setColumns(6); - subtasksPerCoreField.setToolTipText( - htmlBuilder("Enter the number of subtasks per active core (>=1)").appendBreak() - .append("or leave empty to let the algorithm decide the number") - .toString()); - subtasksPerCoreField.setEmptyIsValid(true); - subtasksPerCoreField.setExecuteOnValidityCheck(checkFieldsAndEnableButtons); - return subtasksPerCoreField; + private ValidityTestingFormattedTextField createQueueNameField() { + @SuppressWarnings("serial") + ValidityTestingFormattedTextField queueNameField = new ValidityTestingFormattedTextField( + new DefaultFormatter() { + @Override + public Object stringToValue(String s) throws ParseException { + if (!s.matches("^\\s*R\\d+\\s*$")) { + throw new ParseException( + "Queue is named with the letter R followed by a number", 1); + } + return super.stringToValue(s); + } + }); + queueNameField.setColumns(10); + queueNameField.setName("Queue name"); + queueNameField.setToolTipText( + "The reservation queue is named with the letter R followed by a number."); + queueNameField.setExecuteOnValidityCheck(checkFieldsAndRecalculate); + + return queueNameField; } private void populateTextFieldsAndComboBoxes() { // Subtask counts. + tasksField.setValue(taskCount); subtasksField.setValue(subtaskCount); // Required parameters. - if (!StringUtils.isEmpty(currentParameters.getOptimizer()) - && RemoteArchitectureOptimizer.fromName(currentParameters.getOptimizer()) != null) { - optimizerComboBox.setSelectedItem( - RemoteArchitectureOptimizer.fromName(currentParameters.getOptimizer())); - } - wallTimeRatioField.setValue(currentParameters.getSubtaskTypicalWallTimeHours()); - maxWallTimeField.setValue(currentParameters.getSubtaskMaxWallTimeHours()); - gigsPerSubtaskField.setValue(currentParameters.getGigsPerSubtask()); - nodeSharingCheckBox.setSelected(currentParameters.isNodeSharing()); - wallTimeScalingCheckBox.setSelected(currentParameters.isWallTimeScaling()); - wallTimeScalingCheckBox.setEnabled(!currentParameters.isNodeSharing()); - gigsPerSubtaskField.setEnabled(currentParameters.isNodeSharing()); - updateGigsPerSubtaskToolTip(); + optimizerComboBox.setSelectedItem(currentConfiguration.getOptimizer()); + + remoteExecutionEnabledCheckBox.setSelected(currentConfiguration.isRemoteExecutionEnabled()); + typicalWallTimeField.setValue(currentConfiguration.getSubtaskTypicalWallTimeHours()); + maxWallTimeField.setValue(currentConfiguration.getSubtaskMaxWallTimeHours()); + gigsPerSubtaskField.setValue(currentConfiguration.getGigsPerSubtask()); + oneSubtaskCheckBox.setSelected(!currentConfiguration.isNodeSharing()); + wallTimeScalingCheckBox.setSelected(currentConfiguration.isWallTimeScaling()); + wallTimeScalingCheckBox.setEnabled(!currentConfiguration.isNodeSharing()); + gigsPerSubtaskField.setEnabled(currentConfiguration.isNodeSharing()); + gigsPerSubtaskField.setToolTipText(gigsPerSubtaskToolTip()); // Optional parameters. - if (!StringUtils.isEmpty(currentParameters.getRemoteNodeArchitecture()) - && RemoteNodeDescriptor - .fromName(currentParameters.getRemoteNodeArchitecture()) != null) { + if (!StringUtils.isEmpty(currentConfiguration.getRemoteNodeArchitecture())) { architectureComboBox.setSelectedItem( - RemoteNodeDescriptor.fromName(currentParameters.getRemoteNodeArchitecture())); + RemoteNodeDescriptor.fromName(currentConfiguration.getRemoteNodeArchitecture())); } else { architectureComboBox.setSelectedItem(RemoteNodeDescriptor.ANY); } + lastArchitectureComboBoxSelection = (RemoteNodeDescriptor) architectureComboBox + .getSelectedItem(); - if (!StringUtils.isEmpty(currentParameters.getQueueName()) - && RemoteQueueDescriptor.fromQueueName(currentParameters.getQueueName()) != null) { + if (!StringUtils.isEmpty(currentConfiguration.getQueueName())) { queueComboBox.setSelectedItem( - RemoteQueueDescriptor.fromQueueName(currentParameters.getQueueName())); + RemoteQueueDescriptor.fromQueueName(currentConfiguration.getQueueName())); if (queueComboBox.getSelectedItem() == RemoteQueueDescriptor.RESERVED) { - reservedQueueName = currentParameters.getQueueName(); + reservedQueueName = currentConfiguration.getQueueName(); } + queueName.setEnabled(queueComboBox.getSelectedItem() == RemoteQueueDescriptor.RESERVED); + queueNameField + .setEnabled(queueComboBox.getSelectedItem() == RemoteQueueDescriptor.RESERVED); } else { queueComboBox.setSelectedItem(RemoteQueueDescriptor.ANY); + queueName.setEnabled(false); + queueNameField.setEnabled(false); } - - if (!StringUtils.isEmpty(currentParameters.getMaxNodes())) { - maxNodesField.setValue(Integer.parseInt(currentParameters.getMaxNodes())); - } - if (!StringUtils.isEmpty(currentParameters.getSubtasksPerCore())) { - subtasksPerCoreField - .setValue(Double.parseDouble(currentParameters.getSubtasksPerCore())); - } + lastQueueComboBoxSelection = (RemoteQueueDescriptor) queueComboBox.getSelectedItem(); + queueNameField.setText(reservedQueueName); + + Integer maxNodes = currentConfiguration.getMaxNodes() > 0 + ? currentConfiguration.getMaxNodes() + : null; + maxNodesField.setValue(maxNodes); + + Double subtasksPerCore = currentConfiguration.getSubtasksPerCore() > 0 + ? currentConfiguration.getSubtasksPerCore() + : null; + subtasksPerCoreField.setValue(subtasksPerCore); + + Integer minSubtasksRemoteExecution = currentConfiguration + .getMinSubtasksForRemoteExecution() >= 0 + ? currentConfiguration.getMinSubtasksForRemoteExecution() + : null; + minSubtasksRemoteExecutionField.setValue(minSubtasksRemoteExecution); } private void resetAction(ActionEvent evt) { + reset(true); + } + + private void reset(boolean checkFields) { + skipCheck = true; PipelineTaskInformation.reset(node); - currentParameters = new RemoteParameters(originalParameters); + currentConfiguration = new PipelineDefinitionNodeExecutionResources(originalConfiguration); + taskCount = originalTaskCount; subtaskCount = originalSubtaskCount; - maxParallelSubtaskCount = originalMaxParallelSubtaskCount; populateTextFieldsAndComboBoxes(); - displayPbsValues(null); + skipCheck = false; + if (checkFields) { + checkFieldsAndRecalculate(null); + } } private void displayTaskInformation(ActionEvent evt) { @@ -674,160 +887,194 @@ private void displayTaskInformation(ActionEvent evt) { infoTable.setVisible(true); } - private void calculatePbsParameters(ActionEvent evt) { + /** + * Calculates the PBS parameters from the user-defined parameters. + * + * @return Returns true if the parameters that contribute to the PBS parameters are valid; + * otherwise, false + */ + private boolean calculatePbsParameters() { try { populateCurrentParameters(); - String currentOptimizer = currentParameters.getOptimizer(); - if (!currentParameters.isNodeSharing() - && currentOptimizer.equals(RemoteArchitectureOptimizer.CORES.toString())) { + RemoteArchitectureOptimizer currentOptimizer = currentConfiguration.getOptimizer(); + if (!currentConfiguration.isNodeSharing() + && currentOptimizer == RemoteArchitectureOptimizer.CORES) { JOptionPane.showMessageDialog(this, - "Cores optimization disabled when node sharing disabled.\n" + "Cores optimization disabled when running one subtask per node.\n" + "Cost optimization will be used instead."); - currentParameters.setOptimizer(RemoteArchitectureOptimizer.COST.toString()); + currentConfiguration.setOptimizer(RemoteArchitectureOptimizer.COST); } - // If the user has not changed the subtask counts parameter, use the original - // subtask counts that were generated for each task. + // If the user has changed the task count, use it in lieu of the calculated task count. + // Otherwise, if the user has not changed the subtask counts parameter, use the original + // subtask counts that were generated for each task. Otherwise, use the total subtasks + // given. AlgorithmExecutor executor = AlgorithmExecutor.newRemoteInstance(null); PbsParameters pbsParameters = null; - if (subtaskCount == originalSubtaskCount) { + if (taskCount != originalTaskCount) { + Set perTaskPbsParameters = new HashSet<>(); + for (int i = 0; i < taskCount; i++) { + perTaskPbsParameters.add(executor.generatePbsParameters(currentConfiguration, + subtaskCount / taskCount)); + } + pbsParameters = PbsParameters.aggregatePbsParameters(perTaskPbsParameters); + } else if (subtaskCount == originalSubtaskCount) { Set perTaskPbsParameters = new HashSet<>(); for (SubtaskInformation taskInformation : tasksInformation) { - perTaskPbsParameters.add(executor.generatePbsParameters(currentParameters, + perTaskPbsParameters.add(executor.generatePbsParameters(currentConfiguration, taskInformation.getSubtaskCount())); } pbsParameters = PbsParameters.aggregatePbsParameters(perTaskPbsParameters); } else { - pbsParameters = executor.generatePbsParameters(currentParameters, subtaskCount); + pbsParameters = executor.generatePbsParameters(currentConfiguration, subtaskCount); } displayPbsValues(pbsParameters); - currentParameters.setOptimizer(currentOptimizer); - } catch (Exception f) { - boolean handled = false; - if (f instanceof IllegalStateException || f instanceof PipelineException) { - handled = handlePbsParametersException(f.getStackTrace()); - } - if (!handled) { - throw f; + currentConfiguration.setOptimizer(currentOptimizer); + } catch (Exception e) { + if ((e instanceof IllegalArgumentException || e instanceof IllegalStateException + || e instanceof PipelineException) && handlePbsParametersException(e)) { + return false; } + throw e; } + return true; } private void populateCurrentParameters() { - // Subtask counts. - subtaskCount = (int) subtasksField.getValue(); + // Task counts. + taskCount = textToInt(tasksField); + subtaskCount = textToInt(subtasksField); // Required parameters. + currentConfiguration.setRemoteExecutionEnabled(remoteExecutionEnabledCheckBox.isSelected()); RemoteArchitectureOptimizer optimizerSelection = (RemoteArchitectureOptimizer) optimizerComboBox .getSelectedItem(); - currentParameters.setOptimizer(optimizerSelection.toString()); - currentParameters.setSubtaskTypicalWallTimeHours((double) wallTimeRatioField.getValue()); - currentParameters.setSubtaskMaxWallTimeHours((double) maxWallTimeField.getValue()); - currentParameters.setGigsPerSubtask((double) gigsPerSubtaskField.getValue()); - currentParameters.setNodeSharing(nodeSharingCheckBox.isSelected()); - currentParameters.setWallTimeScaling(wallTimeScalingCheckBox.isSelected()); + currentConfiguration.setOptimizer(optimizerSelection); + currentConfiguration.setSubtaskTypicalWallTimeHours(textToDouble(typicalWallTimeField)); + currentConfiguration.setSubtaskMaxWallTimeHours(textToDouble(maxWallTimeField)); + currentConfiguration.setGigsPerSubtask(textToDouble(gigsPerSubtaskField)); + currentConfiguration.setNodeSharing(!oneSubtaskCheckBox.isSelected()); + currentConfiguration.setWallTimeScaling(wallTimeScalingCheckBox.isSelected()); // Optional parameters. if (architectureComboBox.getSelectedItem() == null || architectureComboBox.getSelectedItem() == RemoteNodeDescriptor.ANY) { - currentParameters.setRemoteNodeArchitecture(""); + currentConfiguration.setRemoteNodeArchitecture(""); } else { - currentParameters.setRemoteNodeArchitecture( + currentConfiguration.setRemoteNodeArchitecture( ((RemoteNodeDescriptor) architectureComboBox.getSelectedItem()).getNodeName()); } if (queueComboBox.getSelectedItem() == null || queueComboBox.getSelectedItem() == RemoteQueueDescriptor.ANY) { - currentParameters.setQueueName(""); + currentConfiguration.setQueueName(""); } else { RemoteQueueDescriptor queue = (RemoteQueueDescriptor) queueComboBox.getSelectedItem(); if (queue == RemoteQueueDescriptor.RESERVED) { - currentParameters.setQueueName(reservedQueueName); + reservedQueueName = queueNameField.getText().trim(); + currentConfiguration.setQueueName(reservedQueueName); } else { - currentParameters.setQueueName(queue.getQueueName()); + currentConfiguration.setQueueName(queue.getQueueName()); } } - Object maxNodesValue = maxNodesField.getValue(); - if (maxNodesValue != null) { - currentParameters.setMaxNodes(Integer.toString((int) maxNodesValue)); - } else { - currentParameters.setMaxNodes(""); + currentConfiguration.setMaxNodes(textToInt(maxNodesField)); + currentConfiguration.setSubtasksPerCore(textToDouble(subtasksPerCoreField)); + currentConfiguration.setMinSubtasksForRemoteExecution( + minSubtasksRemoteExecutionField.getText().isBlank() ? -1 + : textToInt(minSubtasksRemoteExecutionField)); + + log.debug("Updated currentConfiguration"); + } + + private int textToInt(ValidityTestingFormattedTextField field) { + try { + return Integer.parseInt(field.getText()); + } catch (NumberFormatException e) { + return 0; } + } - Object subtasksPerCoreValue = subtasksPerCoreField.getValue(); - if (subtasksPerCoreValue != null) { - currentParameters.setSubtasksPerCore(Double.toString((double) subtasksPerCoreValue)); - } else { - currentParameters.setSubtasksPerCore(""); + private double textToDouble(ValidityTestingFormattedTextField field) { + try { + return Double.parseDouble(field.getText()); + } catch (NumberFormatException e) { + return 0.0; } } private void displayPbsValues(PbsParameters pbsParameters) { if (pbsParameters == null) { - pbsArch.setText(" "); - pbsQueue.setText(" "); - pbsWallTime.setText(" "); - pbsNodeCount.setText(" "); - pbsActiveCoresPerNode.setText(" "); - pbsCost.setText(" "); + pbsArch.setText(""); + pbsQueue.setText(""); + pbsWallTime.setText(""); + pbsNodeCount.setText(""); + pbsActiveCoresPerNode.setText(""); + pbsCost.setText(""); + pbsLimits.setText(""); } else { pbsArch.setText(pbsParameters.getArchitecture().toString()); pbsQueue.setText(pbsParameters.getQueueName()); - pbsWallTime.setText(pbsParameters.getRequestedWallTime()); + pbsWallTime.setText(TimeFormatter.stripSeconds(pbsParameters.getRequestedWallTime())); pbsNodeCount.setText(Integer.toString(pbsParameters.getRequestedNodeCount())); pbsActiveCoresPerNode.setText(Integer.toString(pbsParameters.getActiveCoresPerNode())); pbsCost.setText(String.format("%.2f", pbsParameters.getEstimatedCost())); + + double maxWallTimeHours = RemoteQueueDescriptor + .fromQueueName(pbsParameters.getQueueName()) + .getMaxWallTimeHours(); + pbsLimits.setText(MessageFormat.format( + "The {0} architecture has {1} cores, {2} GB/core, and a cost factor of {3} SBUs, " + + "and for the {4} queue, the limit is {5} hrs max wall time.", + pbsParameters.getArchitecture(), pbsParameters.getArchitecture().getMaxCores(), + pbsParameters.getArchitecture().getGigsPerCore(), + pbsParameters.getArchitecture().getCostFactor(), pbsParameters.getQueueName(), + maxWallTimeHours == Double.MAX_VALUE ? "infinite" : maxWallTimeHours)); } } - private boolean handlePbsParametersException(StackTraceElement[] stackTrace) { - boolean handled = false; - String message = null; - for (StackTraceElement element : stackTrace) { - if (element.getClassName().equals(PbsParameters.class.getName())) { - if (element.getMethodName().equals("populateArchitecture")) { - message = "Selected architecture has insufficient RAM"; - break; - } - if (element.getMethodName().equals("selectArchitecture")) { - message = "All architectures lack sufficient RAM"; - break; - } - if (element.getMethodName().equals("computeWallTimeAndQueue")) { - message = "No queue exists with sufficiently high time limit"; - break; - } + private boolean handlePbsParametersException(Exception e) { + for (StackTraceElement element : e.getStackTrace()) { + if (element.getClassName().equals(PbsParameters.class.getName()) + && (element.getMethodName().equals("populateArchitecture") + || element.getMethodName().equals("selectArchitecture") + || element.getMethodName().equals("computeWallTimeAndQueue"))) { + SwingUtilities + .invokeLater(() -> JOptionPane.showMessageDialog(this, e.getMessage())); + return true; } } - if (message != null) { - handled = true; - JOptionPane.showMessageDialog(this, message); - } - return handled; + return false; } /** - * Updates remote parameters. This means that any parameter changes the user made in this dialog - * box are returned to the edit pipeline dialog box. When the Save action for that dialog box - * happens, the parameters will be saved to the database; conversely, if the user chooses Cancel - * at that point, any changes made here are discarded. + * Close the dialog. Any parameter changes the user made in this dialog box are returned to the + * edit pipeline dialog box via {@link #getCurrentConfiguration()}, which is updated as the user + * makes changes. When the Save action for that dialog box happens, the parameters will be saved + * to the database; conversely, if the user chooses Cancel at that point, any changes made here + * are discarded. + *

      + * The user can generally make any changes they want. The exception is that if they want to set + * remote execution to enabled, then the other parameters have to be valid. For example, you + * can't turn on remote execution if things like the typical and max time per subtask are 0. If + * this is the case, the Close button should be disabled so that this method can't be called. */ - private void updateRemoteParameters(ActionEvent evt) { - - // If the user has made parameter changes that cause the remote parameters instance to be - // invalid, don't save them to the edit pipeline dialog box. - if (calculateButton.isEnabled()) { - parameterSet.setTypedParameters(currentParameters.getParameters()); - } + private void close(ActionEvent evt) { + // Because currentConfiguration isn't updated if a field is invalid, explicitly update + // configuration so that the parameters are preserved upon re-entry. + populateCurrentParameters(); dispose(); } private void cancel(ActionEvent evt) { - resetAction(null); + reset(false); dispose(); } + + public PipelineDefinitionNodeExecutionResources getCurrentConfiguration() { + return currentConfiguration; + } } diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java index 60c9939..de5121c 100644 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/pipeline/ViewEditPipelinesPanel.java @@ -12,9 +12,12 @@ import java.awt.Dialog; import java.awt.event.ActionEvent; import java.util.Date; +import java.util.List; import java.util.Set; +import javax.swing.JButton; import javax.swing.JDialog; +import javax.swing.JMenuItem; import javax.swing.JOptionPane; import javax.swing.SwingUtilities; import javax.swing.tree.DefaultMutableTreeNode; @@ -24,14 +27,13 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; -import gov.nasa.ziggy.services.security.User; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; import gov.nasa.ziggy.ui.util.proxy.PipelineDefinitionCrudProxy; import gov.nasa.ziggy.ui.util.proxy.RetrieveLatestVersionsCrudProxy; -import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; +import gov.nasa.ziggy.ui.util.table.AbstractViewEditGroupPanel; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * Panel for viewing and editing pipelines. @@ -39,9 +41,9 @@ * @author PT * @author Bill Wohler */ -public class ViewEditPipelinesPanel extends AbstractViewEditPanel { +public class ViewEditPipelinesPanel extends AbstractViewEditGroupPanel { - private static final long serialVersionUID = 20230810L; + private static final long serialVersionUID = 20231112L; private PipelineDefinitionCrudProxy crudProxy = new PipelineDefinitionCrudProxy(); private ZiggyTreeModel treeModel; @@ -59,22 +61,27 @@ public ViewEditPipelinesPanel(PipelineRowModel rowModel, */ public static ViewEditPipelinesPanel newInstance() { ZiggyTreeModel treeModel = new ZiggyTreeModel<>( - new PipelineDefinitionCrudProxy()); + new PipelineDefinitionCrudProxy(), PipelineDefinition.class); PipelineRowModel rowModel = new PipelineRowModel(treeModel); return new ViewEditPipelinesPanel(rowModel, treeModel); } @Override - protected void buildComponent() { - super.buildComponent(); - ZiggySwingUtils.addButtonsToPanel(getButtonPanel(), createButton(START, this::start)); - getPopupMenu().add( - createMenuItem("New version of selected pipeline (unlock)" + DIALOG, this::newVersion), - 0); + protected List buttons() { + List buttons = super.buttons(); + buttons.add(createButton(START, this::start)); + return buttons; + } + + @Override + protected List menuItems() { + List menuItems = super.menuItems(); + menuItems.add( + createMenuItem("New version of selected pipeline (unlock)" + DIALOG, this::newVersion)); + return menuItems; } private void start(ActionEvent evt) { - checkPrivileges(); int tableRow = ziggyTable.getSelectedRow(); selectedModelRow = ziggyTable.convertRowIndexToModel(tableRow); @@ -90,7 +97,6 @@ private void start(ActionEvent evt) { } private void newVersion(ActionEvent evt) { - checkPrivileges(); PipelineDefinition selectedPipeline = ziggyTable.getContentAtViewRow(selectedModelRow); @@ -112,11 +118,11 @@ protected RetrieveLatestVersionsCrudProxy getCrudProxy() { } @Override - protected Set optionalViewEditFunctions() { + protected Set optionalViewEditFunctions() { return Set.of( - /* TODO Implement OptionalViewEditFunctions.NEW, per ZIGGY-284 */ OptionalViewEditFunctions.VIEW, - OptionalViewEditFunctions.GROUP, OptionalViewEditFunctions.COPY, - OptionalViewEditFunctions.RENAME, OptionalViewEditFunctions.DELETE); + /* TODO Implement OptionalViewEditFunctions.NEW, per ZIGGY-284 */ OptionalViewEditFunction.VIEW, + OptionalViewEditFunction.COPY, OptionalViewEditFunction.RENAME, + OptionalViewEditFunction.DELETE); } @Override @@ -130,7 +136,6 @@ protected void refresh() { @Override protected void create() { - checkPrivileges(); NewPipelineDialog newPipelineDialog = new NewPipelineDialog( SwingUtilities.getWindowAncestor(this)); @@ -153,7 +158,6 @@ protected void create() { @Override protected void view(int row) { - checkPrivileges(); PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(row); JDialog dialog = new JDialog(SwingUtilities.getWindowAncestor(this), pipeline.getName(), @@ -175,7 +179,6 @@ private void close(ActionEvent evt) { @Override protected void edit(int row) { - checkPrivileges(); PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(row); if (pipeline == null) { @@ -194,7 +197,6 @@ protected void edit(int row) { @Override protected void copy(int row) { - checkPrivileges(); PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(row); if (pipeline == null) { @@ -212,7 +214,6 @@ protected void copy(int row) { @Override protected void rename(int row) { - checkPrivileges(); PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(row); if (pipeline == null) { @@ -242,7 +243,6 @@ protected void rename(int row) { @Override protected void delete(int row) { - checkPrivileges(); PipelineDefinition pipeline = ziggyTable.getContentAtViewRow(row); if (pipeline == null) { @@ -271,7 +271,7 @@ protected void delete(int row) { } private static class PipelineRowModel - implements RowModel, TableModelContentClass { + implements RowModel, ModelContentClass { private static final String[] COLUMN_NAMES = { "Version", "Locked", "User", "Modified", "Node count" }; @@ -317,7 +317,7 @@ public Object getValueFor(Object treeNode, int columnIndex) { AuditInfo auditInfo = pipeline.getAuditInfo(); - User lastChangedUser = null; + String lastChangedUser = null; Date lastChangedTime = null; if (auditInfo != null) { @@ -328,7 +328,7 @@ public Object getValueFor(Object treeNode, int columnIndex) { return switch (columnIndex) { case 0 -> pipeline.getVersion(); case 1 -> pipeline.isLocked(); - case 2 -> lastChangedUser != null ? lastChangedUser.getLoginName() : "---"; + case 2 -> lastChangedUser != null ? lastChangedUser : "---"; case 3 -> lastChangedTime != null ? lastChangedTime : "---"; case 4 -> pipeline.getNodes().size(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); diff --git a/src/main/java/gov/nasa/ziggy/ui/pipeline/WorkerResourcesDialog.java b/src/main/java/gov/nasa/ziggy/ui/pipeline/WorkerResourcesDialog.java deleted file mode 100644 index 04b3f1d..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/pipeline/WorkerResourcesDialog.java +++ /dev/null @@ -1,236 +0,0 @@ -package gov.nasa.ziggy.ui.pipeline; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CLOSE; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DIALOG; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EDIT; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.boldLabel; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButtonPanel; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createMenuItem; -import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createPopupMenu; - -import java.awt.BorderLayout; -import java.awt.Window; -import java.awt.event.ActionEvent; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -import javax.swing.GroupLayout; -import javax.swing.JDialog; -import javax.swing.JLabel; -import javax.swing.JPanel; -import javax.swing.JScrollPane; -import javax.swing.LayoutStyle.ComponentPlacement; -import javax.swing.ListSelectionModel; - -import org.netbeans.swing.etable.ETable; - -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.services.messages.WorkerResources; -import gov.nasa.ziggy.ui.util.MessageUtil; -import gov.nasa.ziggy.ui.util.ZiggySwingUtils; -import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; -import gov.nasa.ziggy.ui.util.table.TableMouseListener; -import gov.nasa.ziggy.ui.util.table.ZiggyTable; - -/** - * Displays the max workers and Java heap size values for each node, and allows the user to edit the - * values. The value editing is actually performed by the - * {@link PipelineDefinitionNodeResourcesDialog}, which is launched from this dialog box. - * - * @author PT - * @author Bill Wohler - */ -public class WorkerResourcesDialog extends JDialog implements TableMouseListener { - - private static final long serialVersionUID = 20230810L; - - private ZiggyTable nodeResourcesTable; - private int selectedRow; - private String pipelineDefinitionName; - - // Initial values in case the user decides to cancel the edits. - private Map initialResources = new HashMap<>(); - - public WorkerResourcesDialog(Window owner, PipelineDefinition pipelineDefinition) { - super(owner, DEFAULT_MODALITY_TYPE); - pipelineDefinitionName = pipelineDefinition.getName(); - for (PipelineDefinitionNode node : pipelineDefinition.getNodes()) { - initialResources.put(node, node.workerResources()); - } - - nodeResourcesTable = new ZiggyTable<>(new WorkerResourcesTableModel(pipelineDefinition)); - buildComponent(); - setLocationRelativeTo(owner); - } - - private void buildComponent() { - setTitle("Worker resources"); - - getContentPane().add(createDataPanel(), BorderLayout.CENTER); - getContentPane() - .add(createButtonPanel(createButton(CLOSE, "Close this dialog box.", this::close), - createButton(CANCEL, "Cancel any changes made here and close dialog box.", - this::cancel)), - BorderLayout.SOUTH); - - setMinimumSize(ZiggySwingUtils.MIN_DIALOG_SIZE); - pack(); - } - - private JPanel createDataPanel() { - WorkerResources resources = WorkerResources.getDefaultResources(); - - JLabel pipeline = boldLabel("Pipeline"); - JLabel pipelineText = new JLabel(pipelineDefinitionName); - - JLabel defaultWorkerCount = boldLabel("Default worker count"); - JLabel defaultWorkerCountText = new JLabel(Integer.toString(resources.getMaxWorkerCount())); - - JLabel defaultHeapSize = boldLabel("Default worker heap size"); - JLabel defaultHeapSizeText = new JLabel(resources.humanReadableHeapSize().toString()); - - ETable table = nodeResourcesTable.getTable(); - table.setSelectionMode(ListSelectionModel.SINGLE_SELECTION); - ZiggySwingUtils.addTableMouseListener(table, - createPopupMenu(createMenuItem(EDIT + DIALOG, WorkerResourcesDialog.this::edit)), this); - JScrollPane nodeResourcesTableScrollPane = new JScrollPane(table); - - JPanel dataPanel = new JPanel(); - GroupLayout dataPanelLayout = new GroupLayout(dataPanel); - dataPanelLayout.setAutoCreateContainerGaps(true); - dataPanel.setLayout(dataPanelLayout); - - dataPanelLayout.setHorizontalGroup(dataPanelLayout.createParallelGroup() - .addComponent(pipeline) - .addComponent(pipelineText) - .addComponent(defaultWorkerCount) - .addComponent(defaultWorkerCountText) - .addComponent(defaultHeapSize) - .addComponent(defaultHeapSizeText) - .addComponent(nodeResourcesTableScrollPane)); - - dataPanelLayout.setVerticalGroup(dataPanelLayout.createSequentialGroup() - .addComponent(pipeline) - .addComponent(pipelineText) - .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(defaultWorkerCount) - .addComponent(defaultWorkerCountText) - .addPreferredGap(ComponentPlacement.RELATED) - .addComponent(defaultHeapSize) - .addComponent(defaultHeapSizeText) - .addPreferredGap(ComponentPlacement.UNRELATED) - .addComponent(nodeResourcesTableScrollPane)); - - return dataPanel; - } - - @Override - public void rowSelected(int row) { - selectedRow = row; - } - - @Override - public void rowDoubleClicked(int row) { - edit(row); - } - - private void edit(ActionEvent evt) { - edit(selectedRow); - } - - private void edit(int row) { - int modelRow = nodeResourcesTable.getTable().convertRowIndexToModel(row); - try { - new PipelineDefinitionNodeResourcesDialog(this, pipelineDefinitionName, - nodeResourcesTable.getContentAtViewRow(modelRow)).setVisible(true); - nodeResourcesTable.fireTableDataChanged(); - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - - /** - * Closes the dialog box. Changes are kept, but can still be discarded in the edit pipelines - * dialog box by selecting cancel. - */ - private void close(ActionEvent evt) { - setVisible(false); - } - - /** - * Resets all the pipeline definition node resource settings to their initial values on entry to - * this dialog box and closes the dialog box. - */ - private void cancel(ActionEvent evt) { - for (Map.Entry entry : initialResources - .entrySet()) { - entry.getKey().applyWorkerResources(entry.getValue()); - } - setVisible(false); - } - - /** - * Table model for the worker resources dialog box. - * - * @author PT - */ - private static class WorkerResourcesTableModel - extends AbstractZiggyTableModel { - - private static final long serialVersionUID = 20230810L; - - private static final String[] COLUMN_NAMES = { "Name", "Max workers", "Heap size" }; - - private List pipelineDefinitionNodes = new ArrayList<>(); - - public WorkerResourcesTableModel(PipelineDefinition pipelineDefinition) { - pipelineDefinitionNodes = pipelineDefinition.getNodes(); - fireTableDataChanged(); - } - - @Override - public Class tableModelContentClass() { - return PipelineDefinitionNode.class; - } - - @Override - public int getRowCount() { - return pipelineDefinitionNodes.size(); - } - - @Override - public int getColumnCount() { - return COLUMN_NAMES.length; - } - - @Override - public String getColumnName(int columnIndex) { - return COLUMN_NAMES[columnIndex]; - } - - @Override - public Object getValueAt(int rowIndex, int columnIndex) { - WorkerResources resources = getContentAtRow(rowIndex).workerResources(); - return switch (columnIndex) { - case 0 -> getContentAtRow(rowIndex).getModuleName(); - case 1 -> resources.maxWorkerCountIsDefault() ? "Default" - : Integer.toString(resources.getMaxWorkerCount()); - case 2 -> resources.heapSizeIsDefault() ? "Default" - : resources.humanReadableHeapSize().toString(); - default -> throw new PipelineException( - "Column index " + columnIndex + " not supported"); - }; - } - - @Override - public PipelineDefinitionNode getContentAtRow(int row) { - return pipelineDefinitionNodes.get(row); - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/security/EditUserPanel.java b/src/main/java/gov/nasa/ziggy/ui/security/EditUserPanel.java deleted file mode 100644 index 0ea6fec..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/security/EditUserPanel.java +++ /dev/null @@ -1,417 +0,0 @@ -package gov.nasa.ziggy.ui.security; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EDIT; - -import java.awt.Dimension; -import java.awt.FlowLayout; -import java.awt.GridBagConstraints; -import java.awt.GridBagLayout; -import java.awt.Insets; -import java.util.LinkedList; -import java.util.List; - -import javax.swing.DefaultListModel; -import javax.swing.JButton; -import javax.swing.JLabel; -import javax.swing.JList; -import javax.swing.JPanel; -import javax.swing.JScrollPane; -import javax.swing.JSeparator; -import javax.swing.JTextField; -import javax.swing.SwingUtilities; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.Role; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.ui.util.DoubleListDialog; -import gov.nasa.ziggy.ui.util.GenericListModel; -import gov.nasa.ziggy.ui.util.MessageUtil; -import gov.nasa.ziggy.ui.util.ZiggySwingUtils; -import gov.nasa.ziggy.ui.util.proxy.UserCrudProxy; - -/** - * @author Todd Klaus - */ -@SuppressWarnings("serial") -public class EditUserPanel extends javax.swing.JPanel { - @SuppressWarnings("unused") - private static final Logger log = LoggerFactory.getLogger(EditUserPanel.class); - - private JLabel loginLabel; - private JTextField loginTextField; - private JTextField nameText; - private JLabel userLabel; - private JSeparator rolesSep; - private JButton privsButton; - private JButton rolesButton; - private JLabel metaLabel; - private JList privsList; - private JScrollPane privsScollPane; - private JList rolesList; - private JScrollPane rolesScrollPane; - private JPanel actionButtonPanel; - private JSeparator privsSep; - private JLabel privsLabel; - private JLabel rolesLabel; - private JTextField phoneText; - private JLabel phoneLabel; - private JTextField emailText; - private JLabel emailLabel; - private JSeparator userSep; - private JLabel nameLabel; - private final User user; - - private final UserCrudProxy userCrud; - - public EditUserPanel(User user) { - this.user = user; - userCrud = new UserCrudProxy(); - buildComponent(); - } - - public void updateUser() { - user.setLoginName(loginTextField.getText()); - user.setDisplayName(nameText.getText()); - user.setEmail(emailText.getText()); - user.setPhone(phoneText.getText()); - } - - private void rolesButtonActionPerformed() { - try { - List currentRoles = user.getRoles(); - List allRoles = userCrud.retrieveAllRoles(); - List availableRoles = new LinkedList<>(); - for (Role role : allRoles) { - if (!currentRoles.contains(role)) { - availableRoles.add(role); - } - } - - DoubleListDialog roleSelectionDialog = new DoubleListDialog<>( - SwingUtilities.getWindowAncestor(this), "Roles for " + user.getDisplayName(), - "Available Roles", availableRoles, "Selected Roles", currentRoles); - roleSelectionDialog.setVisible(true); - - if (roleSelectionDialog.wasSavePressed()) { - List selectedRoles = roleSelectionDialog.getSelectedListContents(); - user.setRoles(selectedRoles); - rolesList.setModel(new GenericListModel<>(selectedRoles)); - } - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - - private void privsButtonActionPerformed() { - try { - List currentPrivs = user.getPrivileges(); - List availablePrivs = new LinkedList<>(); - for (Privilege priv : Privilege.values()) { - if (!currentPrivs.contains(priv.toString())) { - availablePrivs.add(priv.toString()); - } - } - - DoubleListDialog privSelectionDialog = new DoubleListDialog<>( - SwingUtilities.getWindowAncestor(this), "Privileges for " + user.getDisplayName(), - "Available Privileges", availablePrivs, "Selected Privileges", currentPrivs); - privSelectionDialog.setVisible(true); - - if (privSelectionDialog.wasSavePressed()) { - List selectedPrivs = privSelectionDialog.getSelectedListContents(); - user.setPrivileges(selectedPrivs); - privsList.setModel(new GenericListModel<>(selectedPrivs)); - } - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - - private void buildComponent() { - - GridBagLayout layout = new GridBagLayout(); // rows - layout.columnWeights = new double[] { 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1 }; - layout.columnWidths = new int[] { 7, 7, 7, 7, 7, 7, 7, 7, 7 }; - layout.rowWeights = new double[] { 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1 }; - layout.rowHeights = new int[] { 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 }; - setLayout(layout); - setPreferredSize(new Dimension(600, 400)); - add(getLoginLabel(), new GridBagConstraints(0, 1, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getLoginTextField(), - new GridBagConstraints(1, 1, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - add(getNameLabel(), new GridBagConstraints(4, 1, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getNameText(), new GridBagConstraints(5, 1, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - add(getUserLabel(), new GridBagConstraints(0, 0, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getUserSep(), new GridBagConstraints(1, 0, 7, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - - add(getEmailLabel(), new GridBagConstraints(0, 2, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getEmailText(), new GridBagConstraints(1, 2, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - add(getPhoneLabel(), new GridBagConstraints(4, 2, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getPhoneText(), new GridBagConstraints(5, 2, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - add(getRolesLabel(), new GridBagConstraints(0, 3, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getRolesSep(), new GridBagConstraints(1, 3, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - add(getPrivsLabel(), new GridBagConstraints(4, 3, 1, 1, 0.0, 0.0, - GridBagConstraints.LINE_END, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getPrivsSep(), new GridBagConstraints(5, 3, 3, 1, 0.0, 0.0, GridBagConstraints.CENTER, - GridBagConstraints.HORIZONTAL, new Insets(2, 2, 2, 2), 0, 0)); - add(getRolesScrollPane(), new GridBagConstraints(1, 4, 3, 4, 0.0, 0.0, - GridBagConstraints.CENTER, GridBagConstraints.BOTH, new Insets(2, 2, 2, 2), 0, 0)); - add(getPrivsScollPane(), new GridBagConstraints(5, 4, 3, 4, 0.0, 0.0, - GridBagConstraints.CENTER, GridBagConstraints.BOTH, new Insets(2, 2, 2, 2), 0, 0)); - add(getMetaLabel(), new GridBagConstraints(0, 9, 9, 1, 0.0, 0.0, - GridBagConstraints.LINE_START, GridBagConstraints.NONE, new Insets(0, 0, 0, 0), 0, 0)); - add(getRolesButton(), new GridBagConstraints(2, 8, 1, 1, 0.0, 0.0, - GridBagConstraints.CENTER, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getPrivsButton(), new GridBagConstraints(6, 8, 1, 1, 0.0, 0.0, - GridBagConstraints.CENTER, GridBagConstraints.NONE, new Insets(2, 2, 2, 2), 0, 0)); - add(getActionButtonPanel(), new GridBagConstraints(2, 8, 5, 1, 0.0, 0.0, - GridBagConstraints.CENTER, GridBagConstraints.NONE, new Insets(0, 0, 0, 0), 0, 0)); - } - - private JLabel getLoginLabel() { - if (loginLabel == null) { - loginLabel = new JLabel(); - loginLabel.setText("Login"); - } - - return loginLabel; - } - - private JTextField getLoginTextField() { - if (loginTextField == null) { - loginTextField = new JTextField(); - loginTextField.setText(user.getLoginName()); - } - - return loginTextField; - } - - private JLabel getNameLabel() { - if (nameLabel == null) { - nameLabel = new JLabel(); - nameLabel.setText("Full Name"); - } - - return nameLabel; - } - - private JTextField getNameText() { - if (nameText == null) { - nameText = new JTextField(); - nameText.setText(user.getDisplayName()); - } - - return nameText; - } - - private JLabel getUserLabel() { - if (userLabel == null) { - userLabel = new JLabel(); - userLabel.setText("User"); - userLabel.setFont(new java.awt.Font("Dialog", 1, 14)); - } - - return userLabel; - } - - private JSeparator getUserSep() { - if (userSep == null) { - userSep = new JSeparator(); - } - - return userSep; - } - - private JLabel getEmailLabel() { - if (emailLabel == null) { - emailLabel = new JLabel(); - emailLabel.setText("Email"); - } - - return emailLabel; - } - - private JTextField getEmailText() { - if (emailText == null) { - emailText = new JTextField(); - emailText.setText(user.getEmail()); - } - - return emailText; - } - - private JLabel getPhoneLabel() { - if (phoneLabel == null) { - phoneLabel = new JLabel(); - phoneLabel.setText("Phone"); - } - - return phoneLabel; - } - - private JTextField getPhoneText() { - if (phoneText == null) { - phoneText = new JTextField(); - phoneText.setText(user.getPhone()); - } - - return phoneText; - } - - private JLabel getRolesLabel() { - if (rolesLabel == null) { - rolesLabel = new JLabel(); - rolesLabel.setText("Roles"); - rolesLabel.setFont(new java.awt.Font("Dialog", 1, 14)); - } - - return rolesLabel; - } - - private JSeparator getRolesSep() { - if (rolesSep == null) { - rolesSep = new JSeparator(); - } - - return rolesSep; - } - - private JLabel getPrivsLabel() { - if (privsLabel == null) { - privsLabel = new JLabel(); - privsLabel.setText("Privileges"); - privsLabel.setFont(new java.awt.Font("Dialog", 1, 14)); - } - - return privsLabel; - } - - private JSeparator getPrivsSep() { - if (privsSep == null) { - privsSep = new JSeparator(); - } - - return privsSep; - } - - private JScrollPane getRolesScrollPane() { - if (rolesScrollPane == null) { - rolesScrollPane = new JScrollPane(); - rolesScrollPane.setViewportView(getRolesList()); - } - - return rolesScrollPane; - } - - private JList getRolesList() { - if (rolesList == null) { - DefaultListModel rolesListModel = new DefaultListModel<>(); - for (Role role : user.getRoles()) { - rolesListModel.addElement(role); - } - rolesList = new JList<>(); - rolesList.setModel(rolesListModel); - rolesList.setVisibleRowCount(3); - } - - return rolesList; - } - - private JScrollPane getPrivsScollPane() { - if (privsScollPane == null) { - privsScollPane = new JScrollPane(); - privsScollPane.setViewportView(getPrivsList()); - } - - return privsScollPane; - } - - private JList getPrivsList() { - if (privsList == null) { - DefaultListModel privsListModel = new DefaultListModel<>(); - for (String privilege : user.getPrivileges()) { - privsListModel.addElement(privilege); - } - privsList = new JList<>(); - privsList.setModel(privsListModel); - privsList.setVisibleRowCount(3); - } - - return privsList; - } - - private JLabel getMetaLabel() { - if (metaLabel == null) { - metaLabel = new JLabel(); - metaLabel.setText("Modified: " + user.getCreated() + " by admin"); - // metaLabel.setText("Modified: 7/1/05 17:55:00 by admin"); - metaLabel.setFont(new java.awt.Font("Dialog", 2, 12)); - } - - return metaLabel; - } - - private JButton getRolesButton() { - if (rolesButton == null) { - rolesButton = new JButton(); - rolesButton.setText(EDIT); - rolesButton.addActionListener(evt -> { - rolesButtonActionPerformed(); - }); - } - - return rolesButton; - } - - private JButton getPrivsButton() { - if (privsButton == null) { - privsButton = new JButton(); - privsButton.setText(EDIT); - privsButton.addActionListener(evt -> { - privsButtonActionPerformed(); - }); - } - - return privsButton; - } - - private JPanel getActionButtonPanel() { - if (actionButtonPanel == null) { - actionButtonPanel = new JPanel(); - FlowLayout actionButtonPanelLayout = new FlowLayout(); - actionButtonPanelLayout.setHgap(35); - actionButtonPanel.setLayout(actionButtonPanelLayout); - } - - return actionButtonPanel; - } - - public static void main(String[] args) { - User newUser = new User("user1", "User One", "user1@example.com", "555-0100"); - Role r1 = new Role("role1"); - r1.addPrivilege(Privilege.PIPELINE_OPERATIONS.toString()); - r1.addPrivilege(Privilege.PIPELINE_MONITOR.toString()); - Role r2 = new Role("role2"); - r2.addPrivilege(Privilege.PIPELINE_OPERATIONS.toString()); - r2.addPrivilege(Privilege.PIPELINE_MONITOR.toString()); - newUser.addRole(r1); - newUser.addRole(r2); - - ZiggySwingUtils.displayTestDialog(new EditUserPanel(newUser)); - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/security/GroupListModel.java b/src/main/java/gov/nasa/ziggy/ui/security/GroupListModel.java deleted file mode 100644 index 3889aeb..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/security/GroupListModel.java +++ /dev/null @@ -1,59 +0,0 @@ -package gov.nasa.ziggy.ui.security; - -import java.util.ArrayList; -import java.util.List; - -import javax.swing.ComboBoxModel; - -import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.ui.util.models.AbstractDatabaseListModel; -import gov.nasa.ziggy.ui.util.proxy.GroupCrudProxy; - -/** - * @author Todd Klaus - */ -@SuppressWarnings("serial") -public class GroupListModel extends AbstractDatabaseListModel - implements ComboBoxModel { - private List groups = new ArrayList<>(); - private Group selectedGroup = null; - GroupCrudProxy groupCrud = new GroupCrudProxy(); - - public GroupListModel() { - } - - @Override - public void loadFromDatabase() { - groups = groupCrud.retrieveAll(); - - if (groups.size() > 0) { - selectedGroup = groups.get(0); - } - - fireContentsChanged(this, 0, groups.size() - 1); - } - - @Override - public Group getElementAt(int index) { - validityCheck(); - return groups.get(index); - } - - @Override - public int getSize() { - validityCheck(); - return groups.size(); - } - - @Override - public Object getSelectedItem() { - validityCheck(); - return selectedGroup; - } - - @Override - public void setSelectedItem(Object anItem) { - validityCheck(); - selectedGroup = (Group) anItem; - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/security/UserEditDialog.java b/src/main/java/gov/nasa/ziggy/ui/security/UserEditDialog.java deleted file mode 100644 index 02ad464..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/security/UserEditDialog.java +++ /dev/null @@ -1,123 +0,0 @@ -package gov.nasa.ziggy.ui.security; - -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.CANCEL; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.SAVE; - -import java.awt.BorderLayout; -import java.awt.FlowLayout; -import java.awt.Window; - -import javax.swing.JButton; -import javax.swing.JPanel; -import javax.swing.WindowConstants; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.ui.util.MessageUtil; -import gov.nasa.ziggy.ui.util.ZiggySwingUtils; -import gov.nasa.ziggy.ui.util.proxy.UserCrudProxy; - -/** - * @author Todd Klaus - */ -@SuppressWarnings("serial") -public class UserEditDialog extends javax.swing.JDialog { - @SuppressWarnings("unused") - private static final Logger log = LoggerFactory.getLogger(UserEditDialog.class); - - private EditUserPanel userPanel; - private final User user; - private JButton cancelButton; - private JButton saveButton; - private JPanel buttonPanel; - - private final UserCrudProxy userCrud; - - public UserEditDialog(Window owner, User user) { - super(owner, DEFAULT_MODALITY_TYPE); - this.user = user; - userCrud = new UserCrudProxy(); - buildComponent(); - setLocationRelativeTo(owner); - } - - private void buildComponent() { - setTitle("Edit User " + user.getDisplayName()); - getContentPane().setLayout(new BorderLayout()); - setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE); - getContentPane().add(getUserPanel(), BorderLayout.CENTER); - getContentPane().add(getButtonPanel(), BorderLayout.SOUTH); - setSize(700, 483); - } - - private void saveButtonActionPerformed() { - try { - userPanel.updateUser(); - userCrud.saveUser(user); - setVisible(false); - } catch (Exception e) { - MessageUtil.showError(this, "Error Saving User", e.getMessage(), e); - } - } - - private void cancelButtonActionPerformed() { - setVisible(false); - } - - private EditUserPanel getUserPanel() { - - if (userPanel == null) { - userPanel = new EditUserPanel(user); - } - - return userPanel; - } - - private JPanel getButtonPanel() { - - if (buttonPanel == null) { - buttonPanel = new JPanel(); - FlowLayout buttonPanelLayout = new FlowLayout(); - buttonPanelLayout.setHgap(40); - buttonPanel.setLayout(buttonPanelLayout); - buttonPanel.add(getSaveButton()); - buttonPanel.add(getCancelButton()); - } - - return buttonPanel; - } - - private JButton getSaveButton() { - - if (saveButton == null) { - saveButton = new JButton(); - saveButton.setText(SAVE); - saveButton.addActionListener(evt -> { - - saveButtonActionPerformed(); - }); - } - - return saveButton; - } - - private JButton getCancelButton() { - - if (cancelButton == null) { - cancelButton = new JButton(); - cancelButton.setText(CANCEL); - cancelButton.addActionListener(evt -> { - - cancelButtonActionPerformed(); - }); - } - - return cancelButton; - } - - public static void main(String[] args) { - ZiggySwingUtils.displayTestDialog(new UserEditDialog(null, new User())); - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/security/ViewEditRolesPanel.java b/src/main/java/gov/nasa/ziggy/ui/security/ViewEditRolesPanel.java deleted file mode 100644 index 886066a..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/security/ViewEditRolesPanel.java +++ /dev/null @@ -1,211 +0,0 @@ -package gov.nasa.ziggy.ui.security; - -import java.util.LinkedList; -import java.util.List; - -import javax.swing.JOptionPane; -import javax.swing.SwingUtilities; - -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.Role; -import gov.nasa.ziggy.ui.ConsoleSecurityException; -import gov.nasa.ziggy.ui.util.DoubleListDialog; -import gov.nasa.ziggy.ui.util.MessageUtil; -import gov.nasa.ziggy.ui.util.models.AbstractDatabaseModel; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; -import gov.nasa.ziggy.ui.util.proxy.UserCrudProxy; -import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; - -@SuppressWarnings("serial") -public class ViewEditRolesPanel extends AbstractViewEditPanel { - - private final UserCrudProxy userCrud = new UserCrudProxy(); - - public ViewEditRolesPanel() { - super(new RolesTableModel()); - - buildComponent(); - } - - @Override - protected void create() { - try { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - - String roleName = (String) JOptionPane.showInputDialog( - SwingUtilities.getWindowAncestor(this), "Enter a name for the new Role", "New Role", - JOptionPane.QUESTION_MESSAGE, null, null, ""); - - if (roleName != null && !roleName.isEmpty()) { - showEditDialog(new Role(roleName)); - } - } - - @Override - protected void edit(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - - showEditDialog(ziggyTable.getContentAtViewRow(row)); - } - - @Override - protected void delete(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - - Role role = ziggyTable.getContentAtViewRow(row); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete role '" + role.getName() + "'?"); - - if (choice == JOptionPane.YES_OPTION) { - try { - try { - userCrud.deleteRole(role); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - } - - ziggyTable.loadFromDatabase(); - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - } - - @Override - protected void refresh() { - try { - ziggyTable.loadFromDatabase(); - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - - private void showEditDialog(Role role) { - try { - List currentPrivs = role.getPrivileges(); - List availablePrivs = new LinkedList<>(); - for (Privilege priv : Privilege.values()) { - if (!currentPrivs.contains(priv.toString())) { - availablePrivs.add(priv.toString()); - } - } - - DoubleListDialog privSelectionDialog = new DoubleListDialog<>( - SwingUtilities.getWindowAncestor(this), "Privileges for Role " + role.getName(), - "Available Privileges", availablePrivs, "Selected Privileges", currentPrivs); - privSelectionDialog.setVisible(true); - - if (privSelectionDialog.wasSavePressed()) { - List selectedPrivs = privSelectionDialog.getSelectedListContents(); - role.setPrivileges(selectedPrivs); - try { - userCrud.saveRole(role); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - } - ziggyTable.loadFromDatabase(); - } - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - - /** - * @author Todd Klaus - */ - private static class RolesTableModel extends AbstractDatabaseModel { - - private static final String[] COLUMN_NAMES = { "Role", "Privileges" }; - - private List roles = new LinkedList<>(); - private final UserCrudProxy userCrud; - - public RolesTableModel() { - userCrud = new UserCrudProxy(); - } - - @Override - public void loadFromDatabase() { - try { - roles = userCrud.retrieveAllRoles(); - } catch (ConsoleSecurityException ignore) { - } - - fireTableDataChanged(); - } - - @Override - public int getRowCount() { - validityCheck(); - return roles.size(); - } - - @Override - public int getColumnCount() { - return COLUMN_NAMES.length; - } - - @Override - public Object getValueAt(int rowIndex, int columnIndex) { - validityCheck(); - - Role role = roles.get(rowIndex); - - return switch (columnIndex) { - case 0 -> role.getName(); - case 1 -> getPrivilegeList(role); - default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); - }; - } - - private String getPrivilegeList(Role role) { - StringBuilder privList = new StringBuilder(); - boolean first = true; - - for (String privilege : role.getPrivileges()) { - if (!first) { - privList.append(", "); - } - first = false; - privList.append(privilege); - } - return privList.toString(); - } - - @Override - public Class getColumnClass(int columnIndex) { - return String.class; - } - - @Override - public String getColumnName(int column) { - return COLUMN_NAMES[column]; - } - - @Override - public Role getContentAtRow(int row) { - validityCheck(); - return roles.get(row); - } - - @Override - public Class tableModelContentClass() { - return Role.class; - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/security/ViewEditUsersPanel.java b/src/main/java/gov/nasa/ziggy/ui/security/ViewEditUsersPanel.java deleted file mode 100644 index 29b0c86..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/security/ViewEditUsersPanel.java +++ /dev/null @@ -1,173 +0,0 @@ -package gov.nasa.ziggy.ui.security; - -import java.util.LinkedList; -import java.util.List; - -import javax.swing.JOptionPane; -import javax.swing.SwingUtilities; - -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.ui.ConsoleSecurityException; -import gov.nasa.ziggy.ui.util.MessageUtil; -import gov.nasa.ziggy.ui.util.models.AbstractDatabaseModel; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; -import gov.nasa.ziggy.ui.util.proxy.UserCrudProxy; -import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; - -@SuppressWarnings("serial") -public class ViewEditUsersPanel extends AbstractViewEditPanel { - - private final UserCrudProxy userCrud; - - public ViewEditUsersPanel() { - super(new UsersTableModel()); - userCrud = new UserCrudProxy(); - - buildComponent(); - } - - @Override - protected void create() { - try { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - - showEditDialog(new User()); - } - - @Override - protected void edit(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - - showEditDialog(ziggyTable.getContentAtViewRow(row)); - } - - @Override - protected void delete(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - - User user = ziggyTable.getContentAtViewRow(row); - - int choice = JOptionPane.showConfirmDialog(this, - "Are you sure you want to delete user '" + user.getLoginName() + "'?"); - - if (choice == JOptionPane.YES_OPTION) { - try { - try { - userCrud.deleteUser(user); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - } - ziggyTable.loadFromDatabase(); - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - } - - @Override - protected void refresh() { - try { - ziggyTable.loadFromDatabase(); - } catch (Throwable e) { - MessageUtil.showError(this, e); - } - } - - private void showEditDialog(User user) { - new UserEditDialog(SwingUtilities.getWindowAncestor(this), user).setVisible(true); - try { - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtil.showError(this, e); - } - } - - private static class UsersTableModel extends AbstractDatabaseModel { - - private static final String[] COLUMN_NAMES = { "Login", "Name" }; - - private List users = new LinkedList<>(); - private final UserCrudProxy userCrud; - - public UsersTableModel() { - userCrud = new UserCrudProxy(); - } - - @Override - public void loadFromDatabase() { - try { - users = userCrud.retrieveAllUsers(); - } catch (ConsoleSecurityException ignore) { - } - - fireTableDataChanged(); - } - - // TODO Find a use for getUserAtRow or delete - @SuppressWarnings("unused") - public User getUserAtRow(int rowIndex) { - validityCheck(); - return users.get(rowIndex); - } - - @Override - public int getRowCount() { - validityCheck(); - return users.size(); - } - - @Override - public int getColumnCount() { - return COLUMN_NAMES.length; - } - - @Override - public Object getValueAt(int rowIndex, int columnIndex) { - validityCheck(); - - User user = users.get(rowIndex); - - return switch (columnIndex) { - case 0 -> user.getLoginName(); - case 1 -> user.getDisplayName(); - default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); - }; - } - - @Override - public Class getColumnClass(int columnIndex) { - return String.class; - } - - @Override - public String getColumnName(int column) { - return COLUMN_NAMES[column]; - } - - @Override - public User getContentAtRow(int row) { - validityCheck(); - return users.get(row); - } - - @Override - public Class tableModelContentClass() { - return User.class; - } - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/security/package.html b/src/main/java/gov/nasa/ziggy/ui/security/package.html deleted file mode 100644 index e86221a..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/security/package.html +++ /dev/null @@ -1,28 +0,0 @@ - - -

      - The first sentence is used as the brief description of the - package and appears at the top of the package documentation. - Additional sentences will appear at the bottom in the - Description section. -

      -
      -
      Author
      -
      Bill Wohler
      -
      PT
      -
      - -

      Headings

      - -

      - Additional headings should use h2. -

      - -

      Sub-headings

      - -

      - Sub-headings should use h3 and so on. -

      - - - diff --git a/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java b/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java index 72302e5..eb3465b 100644 --- a/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/status/AlertsStatusPanel.java @@ -17,8 +17,8 @@ import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.ButtonPanelContext; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.table.ZiggyTable; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * Displays a table of recent alerts. @@ -67,7 +67,7 @@ private void acknowledge(ActionEvent evt) { } private static class AlertsTableModel extends AbstractTableModel - implements TableModelContentClass { + implements ModelContentClass { private static final long serialVersionUID = 20230822L; diff --git a/src/main/java/gov/nasa/ziggy/ui/status/Indicator.java b/src/main/java/gov/nasa/ziggy/ui/status/Indicator.java index 807175d..a145155 100644 --- a/src/main/java/gov/nasa/ziggy/ui/status/Indicator.java +++ b/src/main/java/gov/nasa/ziggy/ui/status/Indicator.java @@ -250,6 +250,10 @@ public void addDataComponent(Component component) { infoPanel.add(component); } + public void removeDataComponent(Component component) { + infoPanel.remove(component); + } + public IdiotLight getIdiotLight() { return idiotLight; } diff --git a/src/main/java/gov/nasa/ziggy/ui/status/ProcessesStatusPanel.java b/src/main/java/gov/nasa/ziggy/ui/status/ProcessesStatusPanel.java index d1f9047..824a568 100644 --- a/src/main/java/gov/nasa/ziggy/ui/status/ProcessesStatusPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/status/ProcessesStatusPanel.java @@ -2,6 +2,7 @@ import javax.swing.GroupLayout; import javax.swing.JPanel; +import javax.swing.SwingUtilities; import org.apache.commons.configuration2.ImmutableConfiguration; import org.slf4j.Logger; @@ -9,9 +10,12 @@ import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; -import gov.nasa.ziggy.services.messages.WorkerResources; +import gov.nasa.ziggy.services.messages.HeartbeatCheckMessage; +import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiServer; +import gov.nasa.ziggy.ui.ClusterController; +import gov.nasa.ziggy.worker.WorkerResources; /** * Displays the status of the Ziggy processes. @@ -21,19 +25,29 @@ */ public class ProcessesStatusPanel extends JPanel { - private static final long serialVersionUID = 20230822L; + private static final long serialVersionUID = 20231126L; private static Logger log = LoggerFactory.getLogger(ProcessesStatusPanel.class); + public static final String RMI_ERROR_MESSAGE = "Unable to establish communication with supervisor"; + public static final String RMI_WARNING_MESSAGE = "Attempting to establish communication with supervisor"; + public static final String SUPERVISOR_ERROR_MESSAGE = "Supervisor process has failed"; + public static final String DATABASE_ERROR_MESSAGE = "Database process has failed"; + private static ProcessesStatusPanel instance; private Indicator supervisorIndicator; private Indicator databaseIndicator; private Indicator messagingIndicator; + private ClusterController clusterController = new ClusterController(100, 1); + + private LabelValue workerLabel; + private LabelValue heapSizeLabel; public ProcessesStatusPanel() { buildComponent(); - ZiggyMessenger.subscribe(WorkerResources.class, this::addWorkerDataComponents); + ZiggyMessenger.subscribe(WorkerResourcesMessage.class, this::addWorkerDataComponents); + ZiggyMessenger.subscribe(HeartbeatCheckMessage.class, this::performHeartbeatChecks); } private void buildComponent() { @@ -92,18 +106,57 @@ private Indicator createIndicator(String name, String normalStateToolTipText, return indicator; } + private void performHeartbeatChecks(HeartbeatCheckMessage message) { + + if (clusterController.isDatabaseAvailable()) { + databaseIndicator.setState(Indicator.State.NORMAL); + } else { + databaseIndicator.setState(Indicator.State.ERROR, DATABASE_ERROR_MESSAGE); + } + if (clusterController.isSupervisorRunning()) { + supervisorIndicator.setState(Indicator.State.NORMAL); + } else { + supervisorIndicator.setState(Indicator.State.ERROR, SUPERVISOR_ERROR_MESSAGE); + } + + if (message.getHeartbeatTime() > 0) { + log.debug("Setting RMI state to normal"); + messagingIndicator.setState(Indicator.State.NORMAL); + } else if (message.getHeartbeatTime() == 0) { + log.warn("Missed supervisor heartbeat message, setting RMI state to warning"); + messagingIndicator.setState(Indicator.State.WARNING, RMI_WARNING_MESSAGE); + } else { + log.error("Unable to detect supervisor heartbeat messages"); + messagingIndicator.setState(Indicator.State.ERROR, RMI_ERROR_MESSAGE); + } + } + private static boolean monitoringDatabase() { return ZiggyConfiguration.getInstance() .getString(PropertyName.DATABASE_SOFTWARE.property(), null) != null; } - public void addWorkerDataComponents(WorkerResources workerResources) { - log.info("Resource values returned: threads {}, heap size {} MB", + public void addWorkerDataComponents(WorkerResourcesMessage workerResourcesMessage) { + if (workerResourcesMessage.getResources() == null) { + return; + } + WorkerResources workerResources = workerResourcesMessage.getResources(); + log.debug("Resource values returned: threads {}, heap size {} MB", workerResources.getMaxWorkerCount(), workerResources.getHeapSizeMb()); - supervisorIndicator().addDataComponent( - new LabelValue("Workers", Integer.toString(workerResources.getMaxWorkerCount()))); - supervisorIndicator().addDataComponent( - new LabelValue("Worker Heap Size", workerResources.humanReadableHeapSize().toString())); + SwingUtilities.invokeLater(() -> { + if (workerLabel != null) { + supervisorIndicator.removeDataComponent(workerLabel); + } + if (heapSizeLabel != null) { + supervisorIndicator.removeDataComponent(heapSizeLabel); + } + workerLabel = new LabelValue("Workers", + Integer.toString(workerResources.getMaxWorkerCount())); + heapSizeLabel = new LabelValue("Worker Heap Size", + workerResources.humanReadableHeapSize().toString()); + supervisorIndicator().addDataComponent(workerLabel); + supervisorIndicator().addDataComponent(heapSizeLabel); + }); } public static Indicator supervisorIndicator() { diff --git a/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java b/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java index 2ab9fc1..db82963 100644 --- a/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/status/WorkerStatusPanel.java @@ -16,14 +16,16 @@ import org.netbeans.swing.outline.Outline; import gov.nasa.ziggy.services.messages.HeartbeatMessage; -import gov.nasa.ziggy.services.messages.WorkerResources; +import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; +import gov.nasa.ziggy.services.messages.WorkerResourcesRequest; import gov.nasa.ziggy.services.messages.WorkerStatusMessage; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.process.StatusMessage; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; import gov.nasa.ziggy.ui.util.table.ZiggyTable; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; +import gov.nasa.ziggy.worker.WorkerResources; /** * A status panel for worker processes. Status information is displayed using {@link Outline}. @@ -40,6 +42,7 @@ public class WorkerStatusPanel extends JPanel { private ZiggyTable table = new ZiggyTable<>(model); private JLabel countTextField; private JLabel heapTextField; + private boolean waitingForWorkers; public WorkerStatusPanel() { buildComponent(); @@ -50,7 +53,9 @@ public WorkerStatusPanel() { ZiggyMessenger.subscribe(WorkerStatusMessage.class, this::update); - ZiggyMessenger.subscribe(WorkerResources.class, this::updateWorkerResources); + ZiggyMessenger.subscribe(WorkerResourcesMessage.class, this::updateWorkerResources); + + ZiggyMessenger.publish(new WorkerResourcesRequest()); } private void buildComponent() { @@ -96,11 +101,30 @@ public void update(StatusMessage statusMessage) { : Indicator.State.NORMAL; StatusPanel.ContentItem.WORKERS.menuItem().setState(workerState); }); + + // If there are no workers operating, tell the world that the current resources are + // (0, 0), and note that we're waiting for workers to resume working. + if (model.getRowCount() == 0) { + ZiggyMessenger.publish(new WorkerResourcesMessage(null, new WorkerResources(0, 0))); + waitingForWorkers = true; + } else if (waitingForWorkers) { + + // Get the resources from the TaskRequestHandlerLifecycleManager and + // stop waiting for workers. + ZiggyMessenger.publish(new WorkerResourcesRequest()); + waitingForWorkers = false; + } } - public void updateWorkerResources(WorkerResources resources) { - countTextField.setText(Integer.toString(resources.getMaxWorkerCount())); - heapTextField.setText(resources.humanReadableHeapSize().toString()); + public void updateWorkerResources(WorkerResourcesMessage resourcesMessage) { + if (resourcesMessage.getResources() == null) { + return; + } + WorkerResources resources = resourcesMessage.getResources(); + SwingUtilities.invokeLater(() -> { + countTextField.setText(Integer.toString(resources.getMaxWorkerCount())); + heapTextField.setText(resources.humanReadableHeapSize().toString()); + }); } public static void main(String[] args) { @@ -213,7 +237,7 @@ public synchronized Object getValueAt(int rowIndex, int columnIndex) { return switch (columnIndex) { case 0 -> message.getSourceProcess().getKey(); case 1 -> message.getState(); - case 2 -> StringUtils.elapsedTime(message.getProcessingStartTime(), + case 2 -> ZiggyStringUtils.elapsedTime(message.getProcessingStartTime(), System.currentTimeMillis()); case 3 -> message.getInstanceId(); case 4 -> message.getTaskId(); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java b/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java new file mode 100644 index 0000000..eace7d7 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/util/GroupInformation.java @@ -0,0 +1,89 @@ +package gov.nasa.ziggy.ui.util; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +import com.google.common.collect.Sets; + +import gov.nasa.ziggy.pipeline.definition.Group; +import gov.nasa.ziggy.pipeline.definition.Groupable; +import gov.nasa.ziggy.ui.util.proxy.GroupCrudProxy; + +public class GroupInformation { + + private List defaultGroup = new LinkedList<>(); + private Map> groups = new HashMap<>(); + private Map objectsByName = new HashMap<>(); + private Map objectGroups = new HashMap<>(); + + private final GroupCrudProxy groupCrudProxy = new GroupCrudProxy(); + + private final Class clazz; + + public GroupInformation(Class clazz, List allObjects) { + this.clazz = clazz; + initialize(allObjects); + } + + private void initialize(List allObjects) { + List allGroups = groupCrudProxy.retrieveAll(clazz); + + defaultGroup = new LinkedList<>(); + groups = new HashMap<>(); + objectsByName = new HashMap<>(); + + for (T object : allObjects) { + objectsByName.put(object.getName(), object); + } + + List groupList = new ArrayList<>(); + for (Group group : allGroups) { + + // Does this group contain any of the objects we're interested in today? + Sets.SetView objectsThisGroup = Sets.intersection(group.getMemberNames(), + objectsByName.keySet()); + if (!objectsThisGroup.isEmpty()) { + List objects = new ArrayList<>(); + for (String objectName : objectsThisGroup) { + objects.add(objectsByName.get(objectName)); + } + groups.put(group, objects); + groupList.addAll(objects); + } + } + + // Now populate the default group. + List objectsWithNoGroup = new ArrayList<>(objectsByName.values()); + objectsWithNoGroup.removeAll(groupList); + defaultGroup.addAll(objectsWithNoGroup); + + // Populate the inverse map. + for (Map.Entry> entry : groups.entrySet()) { + for (T object : entry.getValue()) { + objectGroups.put(object, entry.getKey()); + } + } + for (T object : objectsWithNoGroup) { + objectGroups.put(object, Group.DEFAULT); + } + } + + public Map> getGroups() { + return groups; + } + + public List getDefaultGroup() { + return defaultGroup; + } + + public Map getObjectGroups() { + return objectGroups; + } + + public Map getObjectsByName() { + return objectsByName; + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/ValidityTestingFormattedTextField.java b/src/main/java/gov/nasa/ziggy/ui/util/ValidityTestingFormattedTextField.java index 186f1d9..2e774e5 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/ValidityTestingFormattedTextField.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/ValidityTestingFormattedTextField.java @@ -1,13 +1,14 @@ package gov.nasa.ziggy.ui.util; import java.awt.Color; +import java.awt.event.FocusAdapter; import java.awt.event.FocusEvent; -import java.awt.event.FocusListener; import java.text.Format; import java.text.ParseException; import java.util.function.Consumer; import javax.swing.JFormattedTextField; +import javax.swing.SwingUtilities; import javax.swing.UIManager; import javax.swing.border.Border; import javax.swing.border.LineBorder; @@ -15,35 +16,45 @@ import javax.swing.event.DocumentListener; import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * Subclass of {@link JFormattedTextField} which provides the following additional functionality: *
        *
      1. Instances check on every keystroke whether they remain valid (i.e., whether the contents of * the text field can be parsed by the instance's formatter). - *
      2. The value is updated on every keystroke, provided that the current text field is valid. - *
      3. A public method, {@link isValid}, allows the user to determine at any time whether any - * instance is currently valid. - *
      4. For invalid instances, a red border appears inside the text field when the instance loses - * focus, and disappears when the instance regains the focus. + *
      5. The value is updated on every keystroke, provided that the current text field is valid; + * otherwise, the value is undefined. The method {@link #getText()} will always return the content + * of the field. + *
      6. A public method, {@link ValidityTestingFormattedTextField#isValidState()}, allows the user to + * determine at any time whether any instance is currently valid. + *
      7. For invalid instances, a red border appears inside the text field. *
      8. Each instance can be provided with an instance of the {@link Consumer}{@code } * functional interface, which can perform actions as part of the validity check depending on * whether the check indicates that the instance is currently valid or invalid. *
      9. Disabled instances are automatically cleared of any values and are treated as valid. *
      - * The user is also able to select whether an empty text box constitutes a valid or invalid state. + * Use {@link #setEmptyIsValid(boolean)} to choose whether an empty text box constitutes a valid or + * invalid state. * * @author PT + * @author Bill Wohler */ -public class ValidityTestingFormattedTextField extends JFormattedTextField { +public class ValidityTestingFormattedTextField extends JFormattedTextField + implements DocumentListener { - private static final long serialVersionUID = 20230511L; + private static final long serialVersionUID = 20240111L; + private static final Logger log = LoggerFactory + .getLogger(ValidityTestingFormattedTextField.class); - public static final Border INVALID_BORDER = new LineBorder(Color.RED, 2); + private static final Border INVALID_BORDER = new LineBorder(Color.RED, 2); private boolean validState; private Consumer executeOnValidityCheck; private boolean emptyIsValid; + private boolean priorValidState; + private String priorText; public ValidityTestingFormattedTextField() { buildComponent(); @@ -78,83 +89,110 @@ public ValidityTestingFormattedTextField(Object value) { private void buildComponent() { setFocusLostBehavior(JFormattedTextField.PERSIST); validState = false; - getDocument().addDocumentListener(new DocumentListener() { + addFocusListener(new FocusAdapter() { @Override - public void insertUpdate(DocumentEvent e) { - checkForValidState(); + public void focusGained(FocusEvent evt) { + // The selection needs to be called from invokeLater to take effect. + // The selection triggers remoteUpdate() and insertUpdate() calls and possibly an + // avalanche of error dialogs if one of the fields are invalid. Therefore, wait + // until the selection is done before adding the document listener. + SwingUtilities.invokeLater(() -> { + selectAll(); + getDocument().addDocumentListener(ValidityTestingFormattedTextField.this); + }); } @Override - public void removeUpdate(DocumentEvent e) { - checkForValidState(); - } - - @Override - public void changedUpdate(DocumentEvent e) { - checkForValidState(); + public void focusLost(FocusEvent evt) { + getDocument().removeDocumentListener(ValidityTestingFormattedTextField.this); } }); - addFocusListener(new FocusListener() { + } - @Override - public void focusGained(FocusEvent e) { - setBorder(UIManager.getLookAndFeel().getDefaults().getBorder("TextField.border")); - } + @Override + public void insertUpdate(DocumentEvent evt) { + checkForValidState(); + } - @Override - public void focusLost(FocusEvent e) { - if (!isEnabled()) { - validState = true; - setValue(null); - } - if (!isValidState()) { - setBorder(INVALID_BORDER); - } - } - }); + @Override + public void removeUpdate(DocumentEvent evt) { + checkForValidState(); + } + + @Override + public void changedUpdate(DocumentEvent evt) { + checkForValidState(); + } + + @Override + public void setEnabled(boolean enable) { + super.setEnabled(enable); + checkForValidState(); + } + + @Override + public void setText(String text) { + super.setText(text); + checkForValidState(); + } + + @Override + public void setValue(Object value) { + super.setValue(value); + checkForValidState(); } private void checkForValidState() { try { - commitEdit(); validState = true; - if (StringUtils.isEmpty(getText()) && !emptyIsValid && isEnabled()) { - validState = false; - } + commitEdit(); } catch (ParseException e) { - if (StringUtils.isEmpty(getText()) && emptyIsValid || !isEnabled()) { - validState = true; - setValue(null); - } else { - validState = false; - } + validState = StringUtils.isEmpty(getText()) && emptyIsValid || !isEnabled(); } finally { - if (validState) { - setBorder(UIManager.getLookAndFeel().getDefaults().getBorder("TextField.border")); - } - if (executeOnValidityCheck != null) { + updateBorder(validState); + + // Skip check if neither field nor state has changed. + log.debug("priorText={}, text={}, value={}, priorValidState={}, validState={}", + priorText, getText(), getValue(), priorValidState, validState); + if (executeOnValidityCheck != null + && (!getText().equals(priorText) || validState != priorValidState)) { executeOnValidityCheck.accept(validState); } + + priorText = getText(); + priorValidState = validState; } } + public void updateBorder(boolean validState) { + setBorder( + validState ? UIManager.getLookAndFeel().getDefaults().getBorder("TextField.border") + : INVALID_BORDER); + } + public boolean isValidState() { return validState; } + /** + * Sets the function to be called when the field is updated. The parameter is true if the field + * is valid; otherwise, it is false. + */ public void setExecuteOnValidityCheck(Consumer executeOnValidityCheck) { this.executeOnValidityCheck = executeOnValidityCheck; } + /** + * Sets whether an empty field is valid. The default is false. This method access the content + * and enabled state of the field, so call this method after those operations. + */ public void setEmptyIsValid(boolean emptyIsValid) { this.emptyIsValid = emptyIsValid; - if (emptyIsValid && getValue() == null) { - validState = true; - } - if (!emptyIsValid && getValue() == null) { - validState = false; + if (StringUtils.isEmpty(getText()) && isEnabled()) { + validState = emptyIsValid; } + updateBorder(validState); if (executeOnValidityCheck != null) { executeOnValidityCheck.accept(validState); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java b/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java index b903cd6..c716da7 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/ViewEditKeyValuePairPanel.java @@ -8,10 +8,8 @@ import gov.nasa.ziggy.services.config.KeyValuePair; import gov.nasa.ziggy.services.config.KeyValuePairCrud; -import gov.nasa.ziggy.services.security.Privilege; import gov.nasa.ziggy.ui.ConsoleSecurityException; import gov.nasa.ziggy.ui.util.models.AbstractDatabaseModel; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; import gov.nasa.ziggy.ui.util.proxy.KeyValuePairCrudProxy; import gov.nasa.ziggy.ui.util.table.AbstractViewEditPanel; @@ -28,25 +26,11 @@ public ViewEditKeyValuePairPanel() { @Override protected void create() { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - showEditDialog(new KeyValuePair()); } @Override protected void edit(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - showEditDialog(ziggyTable.getContentAtViewRow(row)); } @@ -66,13 +50,6 @@ private void showEditDialog(KeyValuePair keyValuePair) { @Override protected void delete(int row) { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(this, e); - return; - } - KeyValuePair keyValuePair = ziggyTable.getContentAtViewRow(row); int choice = JOptionPane.showConfirmDialog(this, diff --git a/src/main/java/gov/nasa/ziggy/ui/util/models/AbstractZiggyTableModel.java b/src/main/java/gov/nasa/ziggy/ui/util/models/AbstractZiggyTableModel.java index 1bcd6bc..3548f38 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/models/AbstractZiggyTableModel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/models/AbstractZiggyTableModel.java @@ -2,20 +2,22 @@ import javax.swing.table.AbstractTableModel; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; + /** * Extension of the {@link AbstractTableModel} for Ziggy. This class adds the following features to * its superclass: *
        *
      1. A method, {@link #getContentAtRow(int)}, that returns the object at a given row in the table. - *
      2. The {@link TableModelContentClass} interface, which returns the class of objects managed by - * the table model (i.e., the actual value of parameter T). + *
      3. The {@link ModelContentClass} interface, which returns the class of objects managed by the + * table model (i.e., the actual value of parameter T). *
      * * @author PT * @param Class of objects managed by the model. */ public abstract class AbstractZiggyTableModel extends AbstractTableModel - implements TableModelContentClass { + implements ModelContentClass { private static final long serialVersionUID = 20230511L; diff --git a/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java b/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java index 71338c9..569f2c3 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/models/ZiggyTreeModel.java @@ -7,7 +7,6 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; -import java.util.Set; import javax.swing.tree.DefaultMutableTreeNode; import javax.swing.tree.DefaultTreeModel; @@ -18,8 +17,8 @@ import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.HasGroup; -import gov.nasa.ziggy.ui.ConsoleSecurityException; +import gov.nasa.ziggy.pipeline.definition.Groupable; +import gov.nasa.ziggy.ui.util.GroupInformation; import gov.nasa.ziggy.ui.util.proxy.RetrieveLatestVersionsCrudProxy; /** @@ -34,7 +33,7 @@ * @author Todd Klaus * @author PT */ -public class ZiggyTreeModel extends DefaultTreeModel +public class ZiggyTreeModel extends DefaultTreeModel implements ConsoleDatabaseModel { private static final long serialVersionUID = 20230511L; @@ -43,7 +42,7 @@ public class ZiggyTreeModel extends DefaultTreeModel private List defaultGroup = new LinkedList<>(); private Map> groups = new HashMap<>(); - private Map objectsByGroupName = new HashMap<>(); + private Map objectsByName = new HashMap<>(); private final RetrieveLatestVersionsCrudProxy crudProxy; @@ -51,75 +50,53 @@ public class ZiggyTreeModel extends DefaultTreeModel private DefaultMutableTreeNode defaultGroupNode; private Map groupNodes; + private Class modelClass; + private boolean modelValid = false; - public ZiggyTreeModel(RetrieveLatestVersionsCrudProxy crudProxy) { + public ZiggyTreeModel(RetrieveLatestVersionsCrudProxy crudProxy, Class modelClass) { super(new DefaultMutableTreeNode("")); rootNode = (DefaultMutableTreeNode) getRoot(); this.crudProxy = crudProxy; + this.modelClass = modelClass; DatabaseModelRegistry.registerModel(this); } public void loadFromDatabase() throws PipelineException { - List allObjects = null; - - try { - if (groups != null) { - log.debug("Clearing the Hibernate cache of all loaded pipelines"); - for (List objects : groups.values()) { - crudProxy.evictAll(objects); // clear the cache - } - } - - if (defaultGroup != null) { - crudProxy.evictAll(defaultGroup); // clear the cache + if (groups != null) { + log.debug("Clearing the Hibernate cache of all loaded pipelines"); + for (List objects : groups.values()) { + crudProxy.evictAll(objects); // clear the cache } - - defaultGroup = new LinkedList<>(); - groups = new HashMap<>(); - objectsByGroupName = new HashMap<>(); - groupNodes = new HashMap<>(); - - allObjects = crudProxy.retrieveLatestVersions(); - } catch (ConsoleSecurityException ignore) { - return; } - for (T object : allObjects) { - objectsByGroupName.put(object.groupName(), object); - - Group group = object.group(); - - if (group == null) { - // default group - defaultGroup.add(object); - } else { - List groupList = groups.get(group); - - if (groupList == null) { - groupList = new LinkedList<>(); - groups.put(group, groupList); - } - - groupList.add(object); - } + if (defaultGroup != null) { + crudProxy.evictAll(defaultGroup); // clear the cache } - // create the tree + // Obtain information on the groups for this component class. + GroupInformation groupInformation = new GroupInformation<>(modelClass, + crudProxy.retrieveLatestVersions()); + objectsByName = groupInformation.getObjectsByName(); + + // Add the default group. rootNode.removeAllChildren(); defaultGroupNode = new DefaultMutableTreeNode(""); insertNodeInto(defaultGroupNode, rootNode, rootNode.getChildCount()); + defaultGroup = groupInformation.getDefaultGroup(); + Collections.sort(defaultGroup, Comparator.comparing(Object::toString)); + for (T object : defaultGroup) { DefaultMutableTreeNode pipelineNode = new DefaultMutableTreeNode(object); insertNodeInto(pipelineNode, defaultGroupNode, defaultGroupNode.getChildCount()); } - // sort groups alphabetically - - Set groupsSet = groups.keySet(); - List groupsList = new ArrayList<>(groupsSet); + // Add the rest of the groups alphabetically. + groups = groupInformation.getGroups(); + List groupsList = new ArrayList<>(groups.keySet()); Collections.sort(groupsList, Comparator.comparing(Group::getName)); + groupNodes = new HashMap<>(); for (Group group : groupsList) { DefaultMutableTreeNode groupNode = new DefaultMutableTreeNode(group.getName()); @@ -127,10 +104,11 @@ public void loadFromDatabase() throws PipelineException { groupNodes.put(group.getName(), groupNode); List objects = groups.get(group); + Collections.sort(objects, Comparator.comparing(Object::toString)); for (T object : objects) { - DefaultMutableTreeNode pipelineNode = new DefaultMutableTreeNode(object); - insertNodeInto(pipelineNode, groupNode, groupNode.getChildCount()); + DefaultMutableTreeNode treeNode = new DefaultMutableTreeNode(object); + insertNodeInto(treeNode, groupNode, groupNode.getChildCount()); } } @@ -147,15 +125,9 @@ public Map getGroupNodes() { return groupNodes; } - /** - * Returns true if an object already exists with the specified name. checked when the operator - * changes the pipeline name so we can warn them before we get a database constraint violation. - * - * @param name - * @return - */ - public T pipelineByName(String name) { - return objectsByGroupName.get(name); + /** Returns an object based on its name, or null if no object exists with that name. */ + public T objectByName(String name) { + return objectsByName.get(name); } @Override diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/AlertLogCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/AlertLogCrudProxy.java index 6b9eb22..3719184 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/AlertLogCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/AlertLogCrudProxy.java @@ -4,7 +4,6 @@ import gov.nasa.ziggy.services.alert.AlertLog; import gov.nasa.ziggy.services.alert.AlertLogCrud; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -15,7 +14,6 @@ public AlertLogCrudProxy() { } public List retrieveForPipelineInstance(final long pipelineInstanceId) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { AlertLogCrud crud = new AlertLogCrud(); return crud.retrieveForPipelineInstance(pipelineInstanceId); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/CrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/CrudProxy.java index a0fa1ab..cce93c4 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/CrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/CrudProxy.java @@ -1,18 +1,12 @@ package gov.nasa.ziggy.ui.util.proxy; import java.util.Collection; -import java.util.Date; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.services.database.DatabaseService; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.ui.ConsoleSecurityException; -import gov.nasa.ziggy.ui.ZiggyGuiConsole; /** * Base class for all console CrudProxy classes. @@ -25,20 +19,6 @@ public abstract class CrudProxy { public CrudProxy() { } - /** - * Verify that the currently-logged in User has the proper Privilege to perform the requested - * operation. Always returns true if there is no logged in user (dev mode) - * - * @param requestedOperation - */ - public static void verifyPrivileges(Privilege requestedOperation) { - User user = ZiggyGuiConsole.currentUser; - - if (user != null && !user.hasPrivilege(requestedOperation.toString())) { - throw new ConsoleSecurityException("You do not have permission to perform this action"); - } - } - /** * Proxy method for DatabaseService.evictAll() Uses {@link CrudProxyExecutor} to invoke the * {@link DatabaseService} method from the dedicated database thread @@ -62,24 +42,4 @@ public void evictAll(final Collection collection) throws PipelineException { public T update(T entity) { throw new UnsupportedOperationException("update method not supported"); } - - /** - * Update the specified AuditInfo object with the currently logged in user and the current time. - * Should be called by subclasses when creating/updating entities that have AuditInfo. - * - * @param auditInfo - */ - protected void updateAuditInfo(AuditInfo auditInfo) { - if (auditInfo == null) { - log.warn("AuditInfo is null, not updating"); - return; - } - - User user = ZiggyGuiConsole.currentUser; - - if (user != null) { - auditInfo.setLastChangedUser(user); - auditInfo.setLastChangedTime(new Date()); - } - } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/DataReceiptOperationsProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/DataReceiptOperationsProxy.java index 0758982..9d79cca 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/DataReceiptOperationsProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/DataReceiptOperationsProxy.java @@ -5,7 +5,6 @@ import gov.nasa.ziggy.data.management.DataReceiptFile; import gov.nasa.ziggy.data.management.DataReceiptInstance; import gov.nasa.ziggy.data.management.DataReceiptOperations; -import gov.nasa.ziggy.services.security.Privilege; /** * Proxy class for {@link DataReceiptOperations}, used to perform operations of same in the context @@ -16,13 +15,11 @@ public class DataReceiptOperationsProxy { public List dataReceiptInstances() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction( () -> new DataReceiptOperations().dataReceiptInstances()); } public List dataReceiptFilesForInstance(long instanceId) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction( () -> new DataReceiptOperations().dataReceiptFilesForInstance(instanceId)); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/DatastoreRegexpCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/DatastoreRegexpCrudProxy.java new file mode 100644 index 0000000..0f6a312 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/DatastoreRegexpCrudProxy.java @@ -0,0 +1,20 @@ +package gov.nasa.ziggy.ui.util.proxy; + +import java.util.List; + +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.data.datastore.DatastoreRegexpCrud; + +/** + * Proxy class for {@link DatastoreRegexpCrud}, used to perform operations of same in the context of + * the pipeline console. + * + * @author Bill Wohler + */ +public class DatastoreRegexpCrudProxy { + + public List retrieveAll() { + return CrudProxyExecutor + .executeSynchronousDatabaseTransaction(() -> new DatastoreRegexpCrud().retrieveAll()); + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/GroupCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/GroupCrudProxy.java index 2d2bdfe..7806ff7 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/GroupCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/GroupCrudProxy.java @@ -1,17 +1,18 @@ package gov.nasa.ziggy.ui.util.proxy; +import java.util.HashSet; import java.util.List; +import java.util.Set; import gov.nasa.ziggy.pipeline.definition.Group; +import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.pipeline.definition.crud.GroupCrud; -import gov.nasa.ziggy.services.security.Privilege; public class GroupCrudProxy { public GroupCrudProxy() { } public void save(final Group group) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { GroupCrud crud = new GroupCrud(); crud.persist(group); @@ -20,7 +21,6 @@ public void save(final Group group) { } public void delete(final Group group) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { GroupCrud crud = new GroupCrud(); crud.remove(group); @@ -29,11 +29,41 @@ public void delete(final Group group) { } public List retrieveAll() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { GroupCrud crud = new GroupCrud(); - return crud.retrieveAll(); }); } + + public List retrieveAll(Class clazz) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + GroupCrud crud = new GroupCrud(); + return crud.retrieveAll(clazz); + }); + } + + public Group retrieveGroupByName(String name, Class clazz) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + GroupCrud crud = new GroupCrud(); + return crud.retrieveGroupByName(name, clazz); + }); + } + + public Set merge(Set groups) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + Set mergedGroups = new HashSet<>(); + GroupCrud crud = new GroupCrud(); + for (Group group : groups) { + if (group != Group.DEFAULT) { + mergedGroups.add(crud.merge(group)); + } + } + return mergedGroups; + }); + } + + public Group merge(Group group) { + return CrudProxyExecutor + .executeSynchronousDatabaseTransaction(() -> new GroupCrud().merge(group)); + } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/KeyValuePairCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/KeyValuePairCrudProxy.java index ab69366..fb1d2ac 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/KeyValuePairCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/KeyValuePairCrudProxy.java @@ -4,7 +4,6 @@ import gov.nasa.ziggy.services.config.KeyValuePair; import gov.nasa.ziggy.services.config.KeyValuePairCrud; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -15,7 +14,6 @@ public KeyValuePairCrudProxy() { } public void save(final KeyValuePair keyValuePair) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { KeyValuePairCrud crud = new KeyValuePairCrud(); crud.create(keyValuePair); @@ -24,7 +22,6 @@ public void save(final KeyValuePair keyValuePair) { } public void delete(final KeyValuePair keyValuePair) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { KeyValuePairCrud crud = new KeyValuePairCrud(); crud.delete(keyValuePair); @@ -33,7 +30,6 @@ public void delete(final KeyValuePair keyValuePair) { } public KeyValuePair retrieve(final String key) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { KeyValuePairCrud crud = new KeyValuePairCrud(); return crud.retrieve(key); @@ -41,7 +37,6 @@ public KeyValuePair retrieve(final String key) { } public List retrieveAll() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { KeyValuePairCrud crud = new KeyValuePairCrud(); return crud.retrieveAll(); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/MetricsLogCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/MetricsLogCrudProxy.java index af94fba..d250a85 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/MetricsLogCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/MetricsLogCrudProxy.java @@ -6,7 +6,6 @@ import gov.nasa.ziggy.metrics.MetricType; import gov.nasa.ziggy.metrics.MetricValue; import gov.nasa.ziggy.metrics.MetricsCrud; -import gov.nasa.ziggy.services.security.Privilege; import gov.nasa.ziggy.util.TimeRange; /** @@ -17,7 +16,6 @@ public MetricsLogCrudProxy() { } public List retrieveAllMetricTypes() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { MetricsCrud crud = new MetricsCrud(); return crud.retrieveAllMetricTypes(); @@ -26,7 +24,6 @@ public List retrieveAllMetricTypes() { public List retrieveAllMetricValuesForType(final MetricType metricType, final Date start, final Date end) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { MetricsCrud crud = new MetricsCrud(); return crud.retrieveAllMetricValuesForType(metricType, start, end); @@ -34,7 +31,6 @@ public List retrieveAllMetricValuesForType(final MetricType metricT } public TimeRange getTimestampRange(final MetricType metricType) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { MetricsCrud crud = new MetricsCrud(); return crud.getTimestampRange(metricType); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParameterSetCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParameterSetCrudProxy.java index c244b0c..abebaac 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParameterSetCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParameterSetCrudProxy.java @@ -6,7 +6,6 @@ import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -17,26 +16,21 @@ public ParameterSetCrudProxy() { } public void save(final ParameterSet moduleParameterSet) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); - updateAuditInfo(moduleParameterSet.getAuditInfo()); crud.persist(moduleParameterSet); return null; }); } public ParameterSet rename(final ParameterSet parameterSet, final String newName) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); - updateAuditInfo(parameterSet.getAuditInfo()); return crud.rename(parameterSet, newName); }); } public List retrieveAll() { - verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); return crud.retrieveAll(); @@ -44,7 +38,6 @@ public List retrieveAll() { } public List retrieveAllVersionsForName(final String name) { - verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); return crud.retrieveAllVersionsForName(name); @@ -52,7 +45,6 @@ public List retrieveAllVersionsForName(final String name) { } public ParameterSet retrieveLatestVersionForName(final String name) { - verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); return crud.retrieveLatestVersionForName(name); @@ -61,7 +53,6 @@ public ParameterSet retrieveLatestVersionForName(final String name) { @Override public List retrieveLatestVersions() { - verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); return crud.retrieveLatestVersions(); @@ -69,7 +60,6 @@ public List retrieveLatestVersions() { } public void delete(final ParameterSet moduleParameterSet) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); crud.remove(moduleParameterSet); @@ -80,7 +70,6 @@ public void delete(final ParameterSet moduleParameterSet) { @Override public ParameterSet update(ParameterSet entity) { checkArgument(entity instanceof ParameterSet, "entity must be ParameterSet"); - verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParameterSetCrud crud = new ParameterSetCrud(); return crud.merge(entity); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParametersOperationsProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParametersOperationsProxy.java index 32e10b4..c531056 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParametersOperationsProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ParametersOperationsProxy.java @@ -5,7 +5,6 @@ import gov.nasa.ziggy.parameters.ParameterLibraryImportExportCli.ParamIoMode; import gov.nasa.ziggy.parameters.ParameterSetDescriptor; import gov.nasa.ziggy.parameters.ParametersOperations; -import gov.nasa.ziggy.services.security.Privilege; /** * GUI Proxy class for {@link ParametersOperations} @@ -19,7 +18,6 @@ public ParametersOperationsProxy() { public List importParameterLibrary(final String sourcePath, final List excludeList, final boolean dryRun) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParametersOperations paramsOps = new ParametersOperations(); return paramsOps.importParameterLibrary(sourcePath, excludeList, @@ -29,7 +27,6 @@ public List importParameterLibrary(final String sourcePa public void exportParameterLibrary(final String destinationPath, final List excludeList, final boolean dryRun) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { ParametersOperations paramsOps = new ParametersOperations(); paramsOps.exportParameterLibrary(destinationPath, excludeList, diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionCrudProxy.java index 4377dab..4f564a7 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionCrudProxy.java @@ -6,8 +6,8 @@ import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionCrud; -import gov.nasa.ziggy.services.security.Privilege; public class PipelineDefinitionCrudProxy extends RetrieveLatestVersionsCrudProxy { @@ -16,16 +16,13 @@ public PipelineDefinitionCrudProxy() { } public PipelineDefinition rename(final PipelineDefinition pipeline, final String newName) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); - updateAuditInfo(pipeline.getAuditInfo()); return crud.rename(pipeline, newName); }); } public void deletePipeline(final PipelineDefinition pipeline) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); crud.deletePipeline(pipeline); @@ -34,7 +31,6 @@ public void deletePipeline(final PipelineDefinition pipeline) { } public void deletePipelineNode(final PipelineDefinitionNode pipelineNode) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); crud.remove(pipelineNode); @@ -43,7 +39,6 @@ public void deletePipelineNode(final PipelineDefinitionNode pipelineNode) { } public PipelineDefinition retrieveLatestVersionForName(final String name) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); PipelineDefinition result1 = crud.retrieveLatestVersionForName(name); @@ -55,7 +50,6 @@ public PipelineDefinition retrieveLatestVersionForName(final String name) { @Override public List retrieveLatestVersions() { - verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); @@ -71,7 +65,6 @@ public List retrieveLatestVersions() { } public List retrieveAll() { - verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); @@ -92,7 +85,6 @@ public PipelineDefinition update(PipelineDefinition entity) { } public PipelineDefinition createOrUpdate(PipelineDefinition entity) { - verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); return crud.merge(entity); @@ -107,4 +99,19 @@ private void initializePipelineDefinitionNodes(PipelineDefinition pipelineDefini Hibernate.initialize(node.getModelTypes()); } } + + public ProcessingMode retrieveProcessingMode(String pipelineName) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); + return crud.retrieveProcessingMode(pipelineName); + }); + } + + public void updateProcessingMode(String pipelineName, ProcessingMode processingMode) { + CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + PipelineDefinitionCrud crud = new PipelineDefinitionCrud(); + crud.updateProcessingMode(pipelineName, processingMode); + return null; + }); + } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionNodeCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionNodeCrudProxy.java new file mode 100644 index 0000000..b281061 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineDefinitionNodeCrudProxy.java @@ -0,0 +1,20 @@ +package gov.nasa.ziggy.ui.util.proxy; + +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; + +public class PipelineDefinitionNodeCrudProxy { + + public PipelineDefinitionNodeExecutionResources merge( + PipelineDefinitionNodeExecutionResources executionResources) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction( + () -> new PipelineDefinitionNodeCrud().merge(executionResources)); + } + + public PipelineDefinitionNodeExecutionResources retrieveRemoteExecutionConfiguration( + PipelineDefinitionNode node) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction( + () -> new PipelineDefinitionNodeCrud().retrieveExecutionResources(node)); + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineExecutorProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineExecutorProxy.java index 5ff026d..aebe9a5 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineExecutorProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineExecutorProxy.java @@ -5,6 +5,7 @@ import java.util.ArrayList; import java.util.List; +import java.util.concurrent.CountDownLatch; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; @@ -13,7 +14,6 @@ import gov.nasa.ziggy.services.database.DatabaseService; import gov.nasa.ziggy.services.messages.StartMemdroneRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; -import gov.nasa.ziggy.services.security.Privilege; import gov.nasa.ziggy.ui.util.models.DatabaseModelRegistry; /** @@ -26,35 +26,36 @@ public PipelineExecutorProxy() { /** * Wrapper for the PipelineExecutor.restartTask() method that also handles messaging and - * database service transactions. - * - * @param task - * @throws Exception + * database service transactions. Note that restart task requests are always sent with maximum + * priority. */ public void restartTask(final PipelineTask task, RunMode restartMode) throws Exception { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); - List taskList = new ArrayList<>(); - taskList.add(task); - - restartTasks(taskList, restartMode); + restartTasks(List.of(task), restartMode, null); } /** * Wrapper for the PipelineExecutor.restartFailedTask() method that also handles messaging and * database service transactions. Note that restart task requests are always sent with maximum * priority. - * - * @param failedTasks - * @throws Exception */ public void restartTasks(final List failedTasks, final RunMode restartMode) throws Exception { + restartTasks(failedTasks, restartMode, null); + } + + /** + * Wrapper for the PipelineExecutor.restartFailedTask() method that also handles messaging and + * database service transactions. Note that restart task requests are always sent with maximum + * priority. If a latch is provided, it is decremented once after a related message is + * published. + */ + public void restartTasks(final List failedTasks, final RunMode restartMode, + CountDownLatch messageSentLatch) throws Exception { + checkNotNull(failedTasks, "failedTasks"); checkArgument(failedTasks.size() > 0, "failedTasks must not be empty"); - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - List databaseTasks = new ArrayList<>(); PipelineTaskCrud pipelineTaskCrud = new PipelineTaskCrud(); for (PipelineTask failedTask : failedTasks) { @@ -65,9 +66,10 @@ public void restartTasks(final List failedTasks, final RunMode res new PipelineExecutor().restartFailedTasks(databaseTasks, false, restartMode); DatabaseService.getInstance().flush(); ZiggyMessenger.publish(new StartMemdroneRequest(failedTasks.get(0).getModuleName(), - failedTasks.get(0).getPipelineInstance().getId())); + failedTasks.get(0).getPipelineInstance().getId()), messageSentLatch); return null; }); + // invalidate the models since restarting a task changes other states. DatabaseModelRegistry.invalidateModels(); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceCrudProxy.java index 9f2919c..57c998c 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceCrudProxy.java @@ -5,7 +5,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceFilter; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -16,7 +15,6 @@ public PipelineInstanceCrudProxy() { } public void save(final PipelineInstance instance) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); crud.persist(instance); @@ -33,7 +31,6 @@ public void save(final PipelineInstance instance) { * @param newName */ public void updateName(final long id, final String newName) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); crud.updateName(id, newName); @@ -42,7 +39,6 @@ public void updateName(final long id, final String newName) { } public void delete(final PipelineInstance instance) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); crud.remove(instance); @@ -51,7 +47,6 @@ public void delete(final PipelineInstance instance) { } public PipelineInstance retrieve(final long id) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); return crud.retrieve(id); @@ -59,7 +54,6 @@ public PipelineInstance retrieve(final long id) { } public List retrieve() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); return crud.retrieveAll(); @@ -67,7 +61,6 @@ public List retrieve() { } public List retrieve(final PipelineInstanceFilter filter) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); return crud.retrieve(filter); @@ -75,7 +68,6 @@ public List retrieve(final PipelineInstanceFilter filter) { } public List retrieveAllActive() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceCrud crud = new PipelineInstanceCrud(); return crud.retrieveAllActive(); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceNodeCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceNodeCrudProxy.java index 8d9fa5b..a58752e 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceNodeCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineInstanceNodeCrudProxy.java @@ -5,7 +5,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceNodeCrud; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -16,7 +15,6 @@ public PipelineInstanceNodeCrudProxy() { } public PipelineInstanceNode retrieve(final long id) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceNodeCrud crud = new PipelineInstanceNodeCrud(); return crud.retrieve(id); @@ -24,7 +22,6 @@ public PipelineInstanceNode retrieve(final long id) { } public List retrieveAll(final PipelineInstance pipelineInstance) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineInstanceNodeCrud crud = new PipelineInstanceNodeCrud(); return crud.retrieveAll(pipelineInstance); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineModuleDefinitionCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineModuleDefinitionCrudProxy.java index be90917..8aecf7e 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineModuleDefinitionCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineModuleDefinitionCrudProxy.java @@ -2,9 +2,11 @@ import java.util.List; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; +import gov.nasa.ziggy.pipeline.definition.PipelineModuleExecutionResources; import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; -import gov.nasa.ziggy.services.security.Privilege; +import gov.nasa.ziggy.uow.UnitOfWorkGenerator; /** * @author Todd Klaus @@ -14,7 +16,6 @@ public PipelineModuleDefinitionCrudProxy() { } public void delete(final PipelineModuleDefinition module) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); crud.remove(module); @@ -25,7 +26,6 @@ public void delete(final PipelineModuleDefinition module) { public PipelineModuleDefinition rename(final PipelineModuleDefinition moduleDef, final String newName) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); return crud.rename(moduleDef, newName); @@ -33,7 +33,6 @@ public PipelineModuleDefinition rename(final PipelineModuleDefinition moduleDef, } public List retrieveAll() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); return crud.retrieveAll(); @@ -41,7 +40,6 @@ public List retrieveAll() { } public PipelineModuleDefinition retrieveLatestVersionForName(final String name) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); return crud.retrieveLatestVersionForName(name); @@ -49,19 +47,39 @@ public PipelineModuleDefinition retrieveLatestVersionForName(final String name) } public List retrieveLatestVersions() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); return crud.retrieveLatestVersions(); }); } - public void createOrUpdate(PipelineModuleDefinition module) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); - CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + public PipelineModuleDefinition merge(PipelineModuleDefinition module) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); - crud.merge(module); - return null; + return crud.merge(module); + }); + } + + public PipelineModuleExecutionResources retrievePipelineModuleExecutionResources( + PipelineModuleDefinition module) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); + return crud.retrieveExecutionResources(module); + }); + } + + public PipelineModuleExecutionResources mergeExecutionResources( + PipelineModuleExecutionResources executionResources) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); + return crud.merge(executionResources); + }); + } + + public ClassWrapper retrieveUnitOfWorkGenerator(String moduleName) { + return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { + PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); + return crud.retrieveUnitOfWorkGenerator(moduleName); }); } } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineOperationsProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineOperationsProxy.java index ce345e0..e6d3d61 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineOperationsProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineOperationsProxy.java @@ -1,7 +1,6 @@ package gov.nasa.ziggy.ui.util.proxy; import java.io.File; -import java.util.Set; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -10,10 +9,8 @@ import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.PipelineOperations; import gov.nasa.ziggy.pipeline.TriggerValidationResults; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.TaskCounts; @@ -21,7 +18,6 @@ import gov.nasa.ziggy.services.messages.InvalidateConsoleModelsMessage; import gov.nasa.ziggy.services.messages.RunningPipelinesCheckRequest; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -33,36 +29,17 @@ public PipelineOperationsProxy() { } public ParameterSet retrieveLatestParameterSet(final String parameterSetName) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineOperations pipelineOps = new PipelineOperations(); return pipelineOps.retrieveLatestParameterSet(parameterSetName); }); } - /** - * Returns a {@link Set} containing all {@link Parameters} classes required by the specified - * node. This is a union of the Parameters classes required by the PipelineModule itself and the - * Parameters classes required by the UnitOfWorkTaskGenerator associated with the node. - * - * @param pipelineNode - * @return - */ - public Set> retrieveRequiredParameterClassesForNode( - final PipelineDefinitionNode pipelineNode) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); - return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - PipelineOperations pipelineOps = new PipelineOperations(); - return pipelineOps.retrieveRequiredParameterClassesForNode(pipelineNode); - }); - } - /** * @param instance * @return */ public String generatePedigreeReport(final PipelineInstance instance) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineOperations pipelineOps = new PipelineOperations(); return pipelineOps.generatePedigreeReport(instance); @@ -75,7 +52,6 @@ public String generatePedigreeReport(final PipelineInstance instance) { */ public void exportPipelineParams(final PipelineDefinition pipelineDefinition, final File destinationDirectory) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineOperations pipelineOps = new PipelineOperations(); pipelineOps.exportPipelineParams(pipelineDefinition, destinationDirectory); @@ -89,7 +65,6 @@ public void exportPipelineParams(final PipelineDefinition pipelineDefinition, * @return */ public String generatePipelineReport(final PipelineDefinition pipelineDefinition) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineOperations pipelineOps = new PipelineOperations(); return pipelineOps.generatePipelineReport(pipelineDefinition); @@ -104,7 +79,6 @@ public String generatePipelineReport(final PipelineDefinition pipelineDefinition * @return */ public String generateParameterLibraryReport(final boolean csvMode) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineOperations pipelineOps = new PipelineOperations(); return pipelineOps.generateParameterLibraryReport(csvMode); @@ -113,7 +87,6 @@ public String generateParameterLibraryReport(final boolean csvMode) { public ParameterSet updateParameterSet(final ParameterSet parameterSet, final Parameters newParameters, final String newDescription, final boolean forceSave) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineOperations pipelineOps = new PipelineOperations(); return pipelineOps.updateParameterSet(parameterSet, newParameters, newDescription, @@ -131,7 +104,6 @@ public ParameterSet updateParameterSet(final ParameterSet parameterSet, // Hence, retrieve the parameter set in one transaction and merge in another. public ParameterSet updateParameterSet(String parameterSetName, ParametersInterface newParameters) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); ParameterSet databaseParameterSet = CrudProxyExecutor.executeSynchronousDatabaseTransaction( () -> new PipelineOperations().retrieveLatestParameterSet(parameterSetName)); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { @@ -144,7 +116,6 @@ public ParameterSet updateParameterSet(String parameterSetName, * Sends a start pipeline request message to the supervisor. */ public void sendPipelineMessage(FireTriggerRequest pipelineRequest) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); ZiggyMessenger.publish(pipelineRequest); // invalidate the models since starting a pipeline can change the locked state of versioned // database objects @@ -156,7 +127,6 @@ public void sendPipelineMessage(FireTriggerRequest pipelineRequest) { * or queued. */ public void sendRunningPipelinesCheckRequestMessage() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); log.info("Sending message to request status of running instances"); ZiggyMessenger.publish(new RunningPipelinesCheckRequest()); } @@ -167,13 +137,11 @@ public void sendRunningPipelinesCheckRequestMessage() { * {@link ParameterSet}s are set. */ public TriggerValidationResults validatePipeline(final PipelineDefinition pipelineDefinition) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor .executeSynchronous(() -> new PipelineOperations().validateTrigger(pipelineDefinition)); } public TaskCounts taskCounts(PipelineInstanceNode node) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor .executeSynchronousDatabaseTransaction(() -> new PipelineOperations().taskCounts(node)); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskCrudProxy.java index 5c8c715..3b2b111 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskCrudProxy.java @@ -8,7 +8,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -19,7 +18,6 @@ public PipelineTaskCrudProxy() { } public void save(final PipelineTask task) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_OPERATIONS); CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineTaskCrud crud = new PipelineTaskCrud(); crud.persist(task); @@ -28,7 +26,6 @@ public void save(final PipelineTask task) { } public PipelineTask retrieve(final long id) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineTaskCrud crud = new PipelineTaskCrud(); return crud.retrieve(id); @@ -36,7 +33,6 @@ public PipelineTask retrieve(final long id) { } public List retrieveAll(final PipelineInstance instance) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineTaskCrud crud = new PipelineTaskCrud(); List r = crud.retrieveTasksForInstance(instance); @@ -51,7 +47,6 @@ public List retrieveAll(final PipelineInstance instance) { public List retrieveAll(final PipelineInstance instance, final PipelineTask.State state) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineTaskCrud crud = new PipelineTaskCrud(); return crud.retrieveAll(instance, state); @@ -59,7 +54,6 @@ public List retrieveAll(final PipelineInstance instance, } public List retrieveAll(final Collection taskIds) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { PipelineTaskCrud crud = new PipelineTaskCrud(); return crud.retrieveAll(taskIds); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskOperationsProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskOperationsProxy.java index 32f3af4..99e58c4 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskOperationsProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/PipelineTaskOperationsProxy.java @@ -5,12 +5,10 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskOperations; -import gov.nasa.ziggy.services.security.Privilege; public class PipelineTaskOperationsProxy { public List updateAndRetrieveTasks(PipelineInstance pipelineInstance) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction( () -> new PipelineTaskOperations().updateJobs(pipelineInstance)); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ProcessingSummaryOpsProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ProcessingSummaryOpsProxy.java index eb46676..51019d0 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ProcessingSummaryOpsProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ProcessingSummaryOpsProxy.java @@ -4,7 +4,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineTask.ProcessingSummary; import gov.nasa.ziggy.pipeline.definition.crud.ProcessingSummaryOperations; -import gov.nasa.ziggy.services.security.Privilege; /** * @author Todd Klaus @@ -14,13 +13,11 @@ public ProcessingSummaryOpsProxy() { } public ProcessingSummary retrieveByTaskId(final long taskId) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor.executeSynchronousDatabaseTransaction( () -> new ProcessingSummaryOperations().processingSummaryInternal(taskId)); } public Map retrieveByInstanceId(final long instanceId) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor .executeSynchronousDatabaseTransaction(() -> new ProcessingSummaryOperations() .processingSummariesForInstanceInternal(instanceId)); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/UserCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/UserCrudProxy.java deleted file mode 100644 index c1558cd..0000000 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/UserCrudProxy.java +++ /dev/null @@ -1,99 +0,0 @@ -package gov.nasa.ziggy.ui.util.proxy; - -import java.util.List; - -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.services.security.Role; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.services.security.UserCrud; - -/** - * This proxy class provides wrappers for the CRUD methods in {@link UserCrud} to support 'off-line' - * conversations (modifications to persisted objects without immediate db updates) The pattern is - * similar for all CRUD operations: - * - * 
      - *
- * 1- start a transaction
- * 2- invoke real CRUD method
- * 3- call Session.flush()
- * 4- commit the transaction
- *
      - * - * This class assumes that auto-flushing has been turned off for the current session by the - * application before calling this class. - * - * @author Todd Klaus - */ -public class UserCrudProxy { - public UserCrudProxy() { - } - - public void saveRole(final Role role) { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - crud.createRole(role); - return null; - }); - } - - public Role retrieveRole(final String roleName) { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - return crud.retrieveRole(roleName); - }); - } - - public List retrieveAllRoles() { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - return crud.retrieveAllRoles(); - }); - } - - public void deleteRole(final Role role) { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - crud.deleteRole(role); - return null; - }); - } - - public void saveUser(final User user) { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - crud.createUser(user); - return null; - }); - } - - public User retrieveUser(final String loginName) { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - return crud.retrieveUser(loginName); - }); - } - - public List retrieveAllUsers() { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - return CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - return crud.retrieveAllUsers(); - }); - } - - public void deleteUser(final User user) { - CrudProxy.verifyPrivileges(Privilege.USER_ADMIN); - CrudProxyExecutor.executeSynchronousDatabaseTransaction(() -> { - UserCrud crud = new UserCrud(); - crud.deleteUser(user); - return null; - }); - } -} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ZiggyEventCrudProxy.java b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ZiggyEventCrudProxy.java index 61bcd3d..8e382a7 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/proxy/ZiggyEventCrudProxy.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/proxy/ZiggyEventCrudProxy.java @@ -6,18 +6,15 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.services.events.ZiggyEvent; import gov.nasa.ziggy.services.events.ZiggyEventCrud; -import gov.nasa.ziggy.services.security.Privilege; public class ZiggyEventCrudProxy { public List retrieveAllEvents() { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor .executeSynchronousDatabaseTransaction(() -> new ZiggyEventCrud().retrieveAllEvents()); } public List retrieve(Collection instances) { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_MONITOR); return CrudProxyExecutor .executeSynchronousDatabaseTransaction(() -> new ZiggyEventCrud().retrieve(instances)); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java new file mode 100644 index 0000000..c15d7de --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditGroupPanel.java @@ -0,0 +1,135 @@ +package gov.nasa.ziggy.ui.util.table; + +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.ASSIGN_GROUP; +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.COLLAPSE_ALL; +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DIALOG; +import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EXPAND_ALL; +import static gov.nasa.ziggy.ui.util.ZiggySwingUtils.createButton; + +import java.awt.event.ActionEvent; +import java.util.List; +import java.util.Set; + +import javax.swing.JButton; +import javax.swing.JMenuItem; +import javax.swing.SwingUtilities; + +import org.apache.commons.collections.CollectionUtils; +import org.netbeans.swing.outline.RowModel; + +import gov.nasa.ziggy.pipeline.definition.Group; +import gov.nasa.ziggy.pipeline.definition.Groupable; +import gov.nasa.ziggy.ui.util.GroupInformation; +import gov.nasa.ziggy.ui.util.GroupsDialog; +import gov.nasa.ziggy.ui.util.MessageUtil; +import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; +import gov.nasa.ziggy.ui.util.proxy.GroupCrudProxy; + +/** + * Extension of {@link AbstractViewEditPanel} for classes of objects that extend {@link Groupable}, + * and hence need some mechanism by which their groups can be configured. + *

      + * The addition of this class was necessary because I couldn't figure out a way to explain to the + * parent class that when we're using the grouping methods in {@link Groupable}; but otherwise, it + * might or might not. + * + * @author PT + */ +public abstract class AbstractViewEditGroupPanel + extends AbstractViewEditPanel { + + private static final long serialVersionUID = 20231112L; + + public AbstractViewEditGroupPanel(RowModel rowModel, ZiggyTreeModel treeModel, + String nodesColumnLabel) { + super(rowModel, treeModel, nodesColumnLabel); + } + + @Override + protected List buttons() { + List buttons = super.buttons(); + buttons.addAll(List.of(createButton(EXPAND_ALL, this::expandAll), + createButton(COLLAPSE_ALL, this::collapseAll))); + return buttons; + } + + private void expandAll(ActionEvent evt) { + ziggyTable.expandAll(); + } + + private void collapseAll(ActionEvent evt) { + ziggyTable.collapseAll(); + } + + @Override + protected List menuItems() { + List menuItems = super.menuItems(); + menuItems.addAll(List.of(getGroupMenuItem())); + return menuItems; + } + + @SuppressWarnings("serial") + private JMenuItem getGroupMenuItem() { + return new JMenuItem(new ViewEditPanelAction(ASSIGN_GROUP + DIALOG, null) { + @Override + public void actionPerformed(ActionEvent evt) { + try { + group(); + } catch (Exception e) { + MessageUtil.showError(SwingUtilities.getWindowAncestor(panel), e); + } + } + }); + } + + /** + * Assign objects in the table to a selected {@link Group}. + */ + protected void group() { + try { + Group group = GroupsDialog.selectGroup(this); + if (group == null) { + return; + } + List selectedObjects = ziggyTable.getContentAtSelectedRows(); + if (selectedObjects.isEmpty()) { + throw new UnsupportedOperationException("Grouping not permitted"); + } + updateGroups(selectedObjects, group); + ziggyTable.loadFromDatabase(); + } catch (Exception e) { + MessageUtil.showError(this, e); + } + } + + /** + * Updates information in the new group and the groups that formerly held the objects in + * question. + */ + @SuppressWarnings("unchecked") + private void updateGroups(List objects, Group group) { + if (CollectionUtils.isEmpty(objects)) { + return; + } + GroupCrudProxy crudProxy = new GroupCrudProxy(); + Group databaseGroup = crudProxy.retrieveGroupByName(group.getName(), + objects.get(0).getClass()); + GroupInformation groupInformation = new GroupInformation<>( + (Class) objects.get(0).getClass(), objects); + for (T object : objects) { + Set memberNames = groupInformation.getObjectGroups() + .get(object) + .getMemberNames(); + if (!CollectionUtils.isEmpty(memberNames)) { + memberNames.remove(object.getName()); + } + if (databaseGroup != Group.DEFAULT) { + databaseGroup.getMemberNames().add(object.getName()); + } + } + if (databaseGroup != Group.DEFAULT) { + crudProxy.merge(databaseGroup); + } + crudProxy.merge(groupInformation.getGroups().keySet()); + } +} diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java index 0db6f04..e969ba3 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/AbstractViewEditPanel.java @@ -1,12 +1,9 @@ package gov.nasa.ziggy.ui.util.table; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.ASSIGN_GROUP; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.COLLAPSE_ALL; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.COPY; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DELETE; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.DIALOG; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EDIT; -import static gov.nasa.ziggy.ui.ZiggyGuiConstants.EXPAND_ALL; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.NEW; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.REFRESH; import static gov.nasa.ziggy.ui.ZiggyGuiConstants.RENAME; @@ -19,11 +16,13 @@ import java.awt.event.ActionEvent; import java.awt.event.MouseAdapter; import java.awt.event.MouseEvent; +import java.util.ArrayList; import java.util.List; import java.util.Set; import javax.swing.AbstractAction; import javax.swing.Icon; +import javax.swing.JButton; import javax.swing.JMenuItem; import javax.swing.JPanel; import javax.swing.JPopupMenu; @@ -34,16 +33,10 @@ import org.netbeans.swing.etable.ETable; import org.netbeans.swing.outline.RowModel; -import gov.nasa.ziggy.pipeline.definition.Group; -import gov.nasa.ziggy.pipeline.definition.HasGroup; -import gov.nasa.ziggy.services.security.Privilege; -import gov.nasa.ziggy.ui.ConsoleSecurityException; -import gov.nasa.ziggy.ui.util.GroupsDialog; import gov.nasa.ziggy.ui.util.MessageUtil; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.ZiggySwingUtils.ButtonPanelContext; import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; -import gov.nasa.ziggy.ui.util.proxy.CrudProxy; import gov.nasa.ziggy.ui.util.proxy.RetrieveLatestVersionsCrudProxy; /** @@ -57,20 +50,15 @@ * that the panel can provide: copying an object, renaming an object, and assigning an object to a * group. The selection of optional functions is controlled by the * {@link #optionalViewEditFunctions()} method, which returns a {@link Set} of instances of - * {@link OptionalViewEditFunctions}: + * {@link OptionalViewEditFunction}: *

        *
      1. A subclass will support copying table objects if the {@link #optionalViewEditFunctions()} - * returns a {@link Set} that includes {@link OptionalViewEditFunctions#COPY}. In addition, the + * returns a {@link Set} that includes {@link OptionalViewEditFunction#COPY}. In addition, the * {@link #copy(int)} method must be overridden. A copy option will be added to the context menu. *
      2. A subclass will support renaming table objects if the {@link #optionalViewEditFunctions()} - * returns a {@link Set} that includes {@link OptionalViewEditFunctions#RENAME}. In addition, the + * returns a {@link Set} that includes {@link OptionalViewEditFunction#RENAME}. In addition, the * {@link #rename(int)} method must be overridden. A rename option will be added to the context * menu. - *
      3. A subclass will support assigning table objects to groups if the - * {@link #optionalViewEditFunctions()} returns a {@link Set} that includes - * {@link OptionalViewEditFunctions#GROUP}. A group option will be added to the context menu, and - * expand all and collapse all buttons will be added to the button panel. Finally, the class of - * objects that are handled by the table must implement the {@link HasGroup} interface. *
      * * @author Todd Klaus @@ -79,15 +67,13 @@ @SuppressWarnings("serial") public abstract class AbstractViewEditPanel extends JPanel { - public enum OptionalViewEditFunctions { - NEW, VIEW, GROUP, COPY, RENAME, DELETE; + public enum OptionalViewEditFunction { + NEW, VIEW, COPY, RENAME, DELETE; } protected ZiggyTable ziggyTable; private ETable table; protected int selectedModelRow = -1; - private JScrollPane scrollPane; - private JPopupMenu popupMenu; private JPanel buttonPanel; public AbstractViewEditPanel(TableModel tableModel) { @@ -108,23 +94,34 @@ protected void buildComponent() { add(getScrollPane(), BorderLayout.CENTER); } - protected JPanel getButtonPanel() { + private JPanel getButtonPanel() { if (buttonPanel == null) { buttonPanel = ZiggySwingUtils.createButtonPanel(ButtonPanelContext.TOOL_BAR, null, createButton(REFRESH, this::refresh), - optionalViewEditFunctions().contains(OptionalViewEditFunctions.NEW) + optionalViewEditFunctions().contains(OptionalViewEditFunction.NEW) ? createButton(NEW, this::newItem) - : null, - optionalViewEditFunctions().contains(OptionalViewEditFunctions.GROUP) - ? createButton(EXPAND_ALL, this::expandAll) - : null, - optionalViewEditFunctions().contains(OptionalViewEditFunctions.GROUP) - ? createButton(COLLAPSE_ALL, this::collapseAll) : null); } + for (JButton button : buttons()) { + ZiggySwingUtils.addButtonsToPanel(buttonPanel, button); + } return buttonPanel; } + /** + * Additional buttons that must be added to the button panel. + *

      + * This method is provided so that subclasses of {@link AbstractViewEditPanel} can supply + * additional buttons that they need on the button panel. Classes that require buttons should + * override this method. + *

      + * Because a superclass may have also added buttons, the subclass should prepend or append their + * buttons to {@code super.buttons()} as appropriate. + */ + protected List buttons() { + return new ArrayList<>(); + } + private void newItem(ActionEvent evt) { try { create(); @@ -137,25 +134,15 @@ private void refresh(ActionEvent evt) { refresh(); } - private void expandAll(ActionEvent evt) { - ziggyTable.expandAll(); - } - - private void collapseAll(ActionEvent evt) { - ziggyTable.collapseAll(); - } - protected JScrollPane getScrollPane() { - if (scrollPane == null) { - scrollPane = new JScrollPane(table); - table.addMouseListener(new MouseAdapter() { - @Override - public void mouseClicked(MouseEvent evt) { - tableMouseClicked(evt); - } - }); - setComponentPopupMenu(table, getPopupMenu()); - } + JScrollPane scrollPane = new JScrollPane(table); + table.addMouseListener(new MouseAdapter() { + @Override + public void mouseClicked(MouseEvent evt) { + tableMouseClicked(evt); + } + }); + setComponentPopupMenu(table, getPopupMenu()); return scrollPane; } @@ -199,30 +186,42 @@ public void mouseReleased(java.awt.event.MouseEvent e) { }); } - protected JPopupMenu getPopupMenu() { - if (popupMenu == null) { - popupMenu = new JPopupMenu(); - addMenuItem(popupMenu, OptionalViewEditFunctions.NEW, getNewMenuItem()); - addMenuItem(popupMenu, OptionalViewEditFunctions.VIEW, getViewMenuItem()); - popupMenu.add(getEditMenuItem()); - addMenuItem(popupMenu, OptionalViewEditFunctions.GROUP, getGroupMenuItem()); - addMenuItem(popupMenu, OptionalViewEditFunctions.COPY, getCopyMenuItem()); - addMenuItem(popupMenu, OptionalViewEditFunctions.RENAME, getRenameMenuItem()); - addMenuItem(popupMenu, OptionalViewEditFunctions.DELETE, getDeleteMenuItem()); - } + private JPopupMenu getPopupMenu() { + JPopupMenu popupMenu = new JPopupMenu(); + addOptionalMenuItem(popupMenu, OptionalViewEditFunction.NEW, getNewMenuItem()); + addOptionalMenuItem(popupMenu, OptionalViewEditFunction.VIEW, getViewMenuItem()); + popupMenu.add(getEditMenuItem()); + addOptionalMenuItem(popupMenu, OptionalViewEditFunction.COPY, getCopyMenuItem()); + addOptionalMenuItem(popupMenu, OptionalViewEditFunction.RENAME, getRenameMenuItem()); + addOptionalMenuItem(popupMenu, OptionalViewEditFunction.DELETE, getDeleteMenuItem()); + + for (JMenuItem menuItem : menuItems()) { + popupMenu.add(menuItem); + } return popupMenu; } - private void addMenuItem(JPopupMenu popupMenu, OptionalViewEditFunctions function, + /** + * Adds additional, optional menu items to the context menu. Subclasses that need such menu + * items should override this method. + *

      + * Because a superclass may have also added menu items, the subclass should prepend or append + * their menu items to {@code super.menuItems()} as appropriate. + */ + protected List menuItems() { + return new ArrayList<>(); + } + + private void addOptionalMenuItem(JPopupMenu popupMenu, OptionalViewEditFunction function, JMenuItem menuItem) { if (optionalViewEditFunctions().contains(function)) { popupMenu.add(menuItem); } } - protected Set optionalViewEditFunctions() { - return Set.of(OptionalViewEditFunctions.DELETE, OptionalViewEditFunctions.NEW); + protected Set optionalViewEditFunctions() { + return Set.of(OptionalViewEditFunction.DELETE, OptionalViewEditFunction.NEW); } private JMenuItem getNewMenuItem() { @@ -264,19 +263,6 @@ public void actionPerformed(ActionEvent evt) { }); } - private JMenuItem getGroupMenuItem() { - return new JMenuItem(new ViewEditPanelAction(ASSIGN_GROUP + DIALOG, null) { - @Override - public void actionPerformed(ActionEvent evt) { - try { - group(); - } catch (Exception e) { - MessageUtil.showError(SwingUtilities.getWindowAncestor(panel), e); - } - } - }); - } - private JMenuItem getCopyMenuItem() { return new JMenuItem(new ViewEditPanelAction(COPY + DIALOG, null) { @Override @@ -325,35 +311,6 @@ protected void view(int row) { protected abstract void edit(int row); - /** - * Assign objects in the table to a selected {@link Group}. - */ - protected void group() { - checkPrivileges(); - try { - Group group = GroupsDialog.selectGroup(this); - if (group == null) { - return; - } - List selectedObjects = ziggyTable.getContentAtSelectedRows(); - if (!selectedObjects.isEmpty() && !(selectedObjects.get(0) instanceof HasGroup)) { - throw new UnsupportedOperationException("Grouping not permitted"); - } - for (T object : selectedObjects) { - HasGroup groupableObject = (HasGroup) object; - if (group == Group.DEFAULT) { - groupableObject.setGroup(null); - } else { - groupableObject.setGroup(group); - } - getCrudProxy().update(object); - } - ziggyTable.loadFromDatabase(); - } catch (Exception e) { - MessageUtil.showError(this, e); - } - } - protected void copy(int row) { } @@ -366,22 +323,13 @@ protected RetrieveLatestVersionsCrudProxy getCrudProxy() { return null; } - protected void checkPrivileges() { - try { - CrudProxy.verifyPrivileges(Privilege.PIPELINE_CONFIG); - } catch (ConsoleSecurityException e) { - MessageUtil.showError(SwingUtilities.getWindowAncestor(this), e); - return; - } - } - /** * Extension of {@link AbstractAction} that allows a reference to the parent panel to be passed * to the {@link AbstractAction#actionPerformed(ActionEvent)} method. * * @author PT */ - private abstract class ViewEditPanelAction extends AbstractAction { + abstract class ViewEditPanelAction extends AbstractAction { protected Component panel; diff --git a/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java b/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java index ee39270..1070c0b 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java +++ b/src/main/java/gov/nasa/ziggy/ui/util/table/ZiggyTable.java @@ -49,15 +49,15 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import gov.nasa.ziggy.pipeline.definition.HasGroup; +import gov.nasa.ziggy.pipeline.definition.Groupable; import gov.nasa.ziggy.ui.util.ZiggySwingUtils; import gov.nasa.ziggy.ui.util.models.AbstractDatabaseModel; import gov.nasa.ziggy.ui.util.models.AbstractZiggyTableModel; import gov.nasa.ziggy.ui.util.models.ConsoleDatabaseModel; import gov.nasa.ziggy.ui.util.models.DatabaseModelRegistry; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.ui.util.models.ZiggyTreeModel; import gov.nasa.ziggy.util.Iso8601Formatter; +import gov.nasa.ziggy.util.dispmod.ModelContentClass; /** * The {@link ZiggyTable} provides a combination of a two-dimensional table of cells and the data @@ -127,9 +127,9 @@ public class ZiggyTable { */ @SuppressWarnings("unchecked") public ZiggyTable(TableModel tableModel) { - checkArgument(tableModel instanceof TableModelContentClass, + checkArgument(tableModel instanceof ModelContentClass, "ZiggyTable model must implement TableModelContentClass"); - modelContentsClass = ((TableModelContentClass) tableModel).tableModelContentClass(); + modelContentsClass = ((ModelContentClass) tableModel).tableModelContentClass(); this.tableModel = tableModel; table = new ZiggyETable(); table.setModel(tableModel); @@ -143,11 +143,11 @@ public ZiggyTable(TableModel tableModel) { */ @SuppressWarnings("unchecked") public ZiggyTable(RowModel rowModel, ZiggyTreeModel treeModel, String nodesColumnLabel) { - checkArgument(rowModel instanceof TableModelContentClass, - "ZiggyTable rowModel must implement TableModelContentClass"); - modelContentsClass = ((TableModelContentClass) rowModel).tableModelContentClass(); - checkArgument(HasGroup.class.isAssignableFrom(modelContentsClass), - "ZiggyTable model content class must implement HasGroup"); + checkArgument(rowModel instanceof ModelContentClass, + "ZiggyTable rowModel must implement ModelContentClass"); + modelContentsClass = ((ModelContentClass) rowModel).tableModelContentClass(); + checkArgument(Groupable.class.isAssignableFrom(modelContentsClass), + "ZiggyTable model content class must extend Groupable"); this.treeModel = treeModel; table = new ZiggyOutline(); outlineModel = DefaultOutlineModel.createOutlineModel(treeModel, rowModel, false, diff --git a/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java index d3775c4..90686e5 100644 --- a/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGenerator.java @@ -1,18 +1,25 @@ package gov.nasa.ziggy.uow; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.DirectoryStream; +import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.HashSet; import java.util.List; -import java.util.Map; import java.util.Set; import java.util.stream.Collectors; -import com.google.common.collect.Sets; - -import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.collections.ZiggyDataType; +import gov.nasa.ziggy.data.management.Manifest; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.TypedParameter; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; -import gov.nasa.ziggy.services.events.ZiggyEventLabels; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; /** * Subclass of {@link DirectoryUnitOfWorkGenerator} that selects units of work based on the @@ -20,40 +27,107 @@ * * @author PT */ -public class DataReceiptUnitOfWorkGenerator extends DatastoreDirectoryUnitOfWorkGenerator { +public class DataReceiptUnitOfWorkGenerator extends DirectoryUnitOfWorkGenerator { @Override protected Path rootDirectory() { - return Paths.get( - ZiggyConfiguration.getInstance().getString(PropertyName.DATA_RECEIPT_DIR.property())); + return Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .toAbsolutePath(); + } + + @Override + public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode) { + return generateUnitsOfWork(pipelineInstanceNode, null); } /** - * Extends the superclass generateTasks() method by filtering out any DR unit of work that has a - * directory of ".manifests". + * Generates units of work by looking for a manifest in the data receipt directory (in which + * case the data receipt directory is the only UOW), if none is found then searching the + * top-level subdirectories of the data receipt directory for manifests (each directory that has + * one becomes a UOW). The resulting UOWs are filtered by the event labels argument so that, if + * there are any event labels, only units of work that match the event labels will be processed. */ @Override - public List generateTasks( - Map, ParametersInterface> parameters) { - List unitsOfWork = super.generateTasks(parameters); - - // If the pipeline that's going to execute data receipt was launched by an event handler, - // we need to restrict the UOWs to the ones that are specified by the event handler. - ZiggyEventLabels eventLabels = (ZiggyEventLabels) parameters.get(ZiggyEventLabels.class); - if (eventLabels != null) { - - // Handle the special case in which the user wants to trigger data receipt to operate - // at the top-level DR directory (i.e., not any subdirectories of same). - if (eventLabels.getEventLabels().length == 0 && unitsOfWork.size() == 1 - && unitsOfWork.get(0).getParameter(DIRECTORY_PROPERTY_NAME).getString().isEmpty()) { - return unitsOfWork; + public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode, + Set eventLabels) { + List unitsOfWork = new ArrayList<>(); + + // If the root directory for this UOW generator contains a manifest, then the root directory + // will be the only unit of work. + if (directoryContainsManifest(rootDirectory())) { + UnitOfWork uow = new UnitOfWork(); + uow.addParameter(new TypedParameter(DIRECTORY_PARAMETER_NAME, + rootDirectory().toString(), ZiggyDataType.ZIGGY_STRING)); + unitsOfWork.add(uow); + } else { + + // Check for subdirectories that have manifests + Set subdirs = subdirsWithManifests(); + for (Path subdir : subdirs) { + UnitOfWork uow = new UnitOfWork(); + uow.addParameter(new TypedParameter(DIRECTORY_PARAMETER_NAME, subdir.toString(), + ZiggyDataType.ZIGGY_STRING)); + unitsOfWork.add(uow); + } + } + + // Handle two special cases: the event labels Set is null (DR not triggered by an + // event); the event labels Set is non-null but empty (DR triggered by an event but + // the event wants DR to run in the main DR directory). In either case, there can + // be only one UOW. + if (eventLabels == null || eventLabels.size() == 0 && unitsOfWork.size() == 1 + && unitsOfWork.get(0) + .getParameter(DIRECTORY_PARAMETER_NAME) + .getString() + .equals(rootDirectory().toString())) { + return unitsOfWork; + } + + // Otherwise, filter against the event labels. + return unitsOfWork.stream() + .filter(s -> eventLabels.contains(uowDirectoryFileName(s))) + .collect(Collectors.toList()); + } + + private String uowDirectoryFileName(UnitOfWork uow) { + Path uowDirectory = Paths.get(uow.getParameter(DIRECTORY_PARAMETER_NAME).getString()); + return uowDirectory.getFileName().toString(); + } + + @Override + public void setBriefState(UnitOfWork uow, PipelineInstanceNode pipelineInstanceNode) { + Path directory = Paths.get(uow.getParameter(DIRECTORY_PARAMETER_NAME).getString()); + uow.setBriefState(directory.getFileName().toString()); + } + + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private boolean directoryContainsManifest(Path directory) { + try (DirectoryStream dirStream = Files.newDirectoryStream(directory, + file -> file.getFileName().toString().endsWith(Manifest.FILENAME_SUFFIX))) { + for (@SuppressWarnings("unused") + Path manifestFile : dirStream) { + return true; + } + return false; + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + private Set subdirsWithManifests() { + Set subdirsWithManifests = new HashSet<>(); + try (DirectoryStream dirStream = Files.newDirectoryStream(rootDirectory(), + Files::isDirectory)) { + for (Path subdirPath : dirStream) { + if (directoryContainsManifest(subdirPath)) { + subdirsWithManifests.add(subdirPath); + } } - Set eventLabelSet = Sets.newHashSet(eventLabels.getEventLabels()); - unitsOfWork = unitsOfWork.stream() - .filter(s -> eventLabelSet - .contains(s.getParameter(DIRECTORY_PROPERTY_NAME).getString())) - .collect(Collectors.toList()); + return subdirsWithManifests; + } catch (IOException e) { + throw new UncheckedIOException(e); } - return unitsOfWork; } } diff --git a/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java index 64206a1..aee508c 100644 --- a/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkGenerator.java @@ -1,86 +1,365 @@ package gov.nasa.ziggy.uow; import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Set; + +import org.apache.commons.collections.CollectionUtils; import gov.nasa.ziggy.collections.ZiggyDataType; +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.data.datastore.DatastoreWalker; import gov.nasa.ziggy.module.ExternalProcessPipelineModule; -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.TypedParameter; import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; /** * Subclass of {@link DirectoryUnitOfWorkGenerator} that selects units of work based on the * directory tree configuration of the datastore. This is the default unit of work for the * {@link ExternalProcessPipelineModule} class. + *

      + * Each unit of work has a brief state generated by a {@link DatastoreWalker} instance. This implies + * that, if the UOW has multiple input data file types, all the paths for a given UOW must + * correspond to the same brief state. + *

      + * The unit of work also captures the values of all {@link DatastoreRegexp} instances that were used + * to construct the UOW. * * @author PT */ public class DatastoreDirectoryUnitOfWorkGenerator extends DirectoryUnitOfWorkGenerator { - private boolean singleSubtask; - public static final String SINGLE_SUBTASK_PROPERTY_NAME = "singleSubtask"; + private static final String BRIEF_STATE_PART_SEPARATOR = ";"; + private static final String BRIEF_STATE_OPEN_STRING = "["; + private static final String BRIEF_STATE_CLOSE_STRING = "]"; + private static final String SINGLE_UOW_BRIEF_STATE = "Single"; + + private DatastoreWalker datastoreWalker; + + @Override + protected Path rootDirectory() { + return DirectoryProperties.datastoreRootDir(); + } + + @Override + public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode) { + Set inputDataFileTypes = pipelineInstanceNode.getPipelineDefinitionNode() + .getInputDataFileTypes(); + if (CollectionUtils.isEmpty(inputDataFileTypes)) { + throw new IllegalArgumentException("Pipeline definition has no input data file types"); + } + + // Construct a List of DataFileTypeInformation instances. + List dataFileTypesInformation = dataFileTypesInformation( + inputDataFileTypes); + + // If we didn't find any datastore paths that can be made into UOWs, exit now. + if (dataFileTypesInformation.isEmpty()) { + return new ArrayList<>(); + } + + // Rearrange the paths and brief states to a List of UowPathInformation instances. + // This will give us one UowPathInformation instance per UOW. + Collection unitsOfWorkPathInformation = pathsForUnitOfWork( + dataFileTypesInformation); + + // Generate the UOWs from the BriefStatePathInformation instances. + List unitsOfWork = new ArrayList<>(); + for (UowPathInformation unitOfWorkPathInformation : unitsOfWorkPathInformation) { + UnitOfWork uow = new UnitOfWork(); + + // Populate the UOW parameters for the datastore paths used by this UOW. + populateDirectoryTypedParameters(uow, unitOfWorkPathInformation); + + // Populate the regular expression parameters used by this UOW. + Map uowRegexpValuesByName = uowRegexpValuesByName( + unitOfWorkPathInformation); + for (Map.Entry regexpEntry : uowRegexpValuesByName.entrySet()) { + uow.addParameter(new TypedParameter(regexpEntry.getKey(), regexpEntry.getValue(), + ZiggyDataType.ZIGGY_STRING)); + } + uow.setBriefState(unitOfWorkPathInformation.getBriefState()); + unitsOfWork.add(uow); + } + return unitsOfWork; + } /** - * Convenience method that returns the value of the single subtask property. + * Constructs a {@link List} of {@link DataFileTypeInformation} instances. Each instance + * contains one of the {@link DataFileType}s used for inputs and the brief state and + * {@link Path} that correspond to that DataFileType. The returned List therefore contains all + * of the Paths that will need to be used by the current pipeline module to find inputs, along + * with the DataFileType and brief state that go with that path. + *

      + * Note that, at this point, no attempt is made to organize the information by brief state. + * Thus, the returned List will contain multiple entries for each brief state; specifically, one + * entry per input data file type per brief state. This will later be rearranged into a Map that + * is organized by brief state. */ - public static boolean singleSubtask(UnitOfWork uow) { - String clazz = uow.getParameter(UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME) - .getString(); - try { - Class cls = Class.forName(clazz); - if (DatastoreDirectoryUnitOfWorkGenerator.class.isAssignableFrom(cls)) { - return (Boolean) uow.getParameter(SINGLE_SUBTASK_PROPERTY_NAME).getValue(); + private List dataFileTypesInformation( + Set inputDataFileTypes) { + List dataFileTypesInformation = new ArrayList<>(); + for (DataFileType inputType : inputDataFileTypes) { + List uowPaths = datastoreWalker().pathsForLocation(inputType.getLocation()); + if (CollectionUtils.isEmpty(uowPaths)) { + continue; + } + + List pathElementIndicesForBriefState = datastoreWalker() + .pathElementIndicesForBriefState(uowPaths); + + // If there is only 1 UOW, it stands to reason that we can't identify any path + // elements that vary from one UOW to another! In this special case, we use + // all regexp values EXCEPT the ones specified in the location (i.e., + // if the location is "foo/bar$baz/blah", and all 3 elements are regexps, we + // take only "foo" and "blah"). + if (CollectionUtils.isEmpty(pathElementIndicesForBriefState)) { + dataFileTypesInformation.add(new DataFileTypeInformation( + inputType, briefStateFromAllRegexps(datastoreWalker() + .regexpValues(inputType.getLocation(), uowPaths.get(0), false)), + uowPaths.get(0))); + } else { + for (Path uowPath : uowPaths) { + DatastoreDirectoryBriefStateBuilder briefStateBuilder = new DatastoreDirectoryBriefStateBuilder(); + for (int elementIndex : pathElementIndicesForBriefState) { + briefStateBuilder.addUowPart(uowPath.getName(elementIndex)); + } + dataFileTypesInformation.add( + new DataFileTypeInformation(inputType, briefStateBuilder.build(), uowPath)); + } } - throw new PipelineException( - "Class " + clazz + " not a subclass of DatastoreDirectoryUnitOfWorkGenerator"); - } catch (ClassNotFoundException e) { - throw new PipelineException("Generator class " + clazz + " not found", e); } + return dataFileTypesInformation; } - @Override - protected Path rootDirectory() { - return DirectoryProperties.datastoreRootDir(); + /** Generates a brief state string from the values of all regular expressions. */ + private String briefStateFromAllRegexps(Map regexpValues) { + + // A special case within a special case: we have no regular expressions at all! + // In this case, the only thing we can do is issue a default brief state string. + if (regexpValues.isEmpty()) { + return singlePartBriefState(); + } + DatastoreDirectoryBriefStateBuilder briefStateBuilder = new DatastoreDirectoryBriefStateBuilder(); + for (Map.Entry regexpEntry : regexpValues.entrySet()) { + briefStateBuilder.addUowPart(regexpEntry.getValue()); + } + return briefStateBuilder.build(); } /** - * Extends the {@link DirectoryUnitOfWorkGenerator#generateTasks(Map)} method. Specifically, the - * superclass method is used for the initial generation of the UOW instances, following which - * the value of the singleSubtask field is set according to the value of - * {@link TaskConfigurationParameters#isSingleSubtask()}. + * Generate {@link UowPathInformation}. Each {@link UowPathInformation} instance has a brief + * state string and, for that brief state, a {@link Map} of paths by data file type. Thus, each + * entry in the returned {@link List} represents a single unit of work and contains all the + * datastore paths that will be needed to populate the inputs of the task for that unit of work. */ - @Override - public List generateTasks( - Map, ParametersInterface> parameters) { - List tasks = super.generateTasks(parameters); - TaskConfigurationParameters taskConfigurationParameters = (TaskConfigurationParameters) parameters - .get(TaskConfigurationParameters.class); - boolean singleSubtask = taskConfigurationParameters.isSingleSubtask(); - for (UnitOfWork uow : tasks) { - uow.addParameter(new TypedParameter(SINGLE_SUBTASK_PROPERTY_NAME, - Boolean.toString(singleSubtask), ZiggyDataType.ZIGGY_BOOLEAN)); - } - return tasks; + private Collection pathsForUnitOfWork( + List dataFileTypesInformation) { + + Map pathInfoByBriefState = new HashMap<>(); + + // Construct the Map from all of the brief states. + for (DataFileTypeInformation info : dataFileTypesInformation) { + String briefState = info.getBriefState(); + if (pathInfoByBriefState.get(briefState) == null) { + pathInfoByBriefState.put(briefState, new UowPathInformation(briefState)); + } + pathInfoByBriefState.get(info.getBriefState()) + .addDataFileTypeAndPath(info.getDataFileType(), info.getDataFilePath()); + } + + return pathInfoByBriefState.values(); + } + + /** Generates a {@link Map} of regular expression values for a given unit of work. */ + private Map uowRegexpValuesByName(UowPathInformation uowPathInformation) { + + Map uowRegexpValuesByName = new HashMap<>(); + Set regexpNamesToExclude = new HashSet<>(); + for (Map.Entry entry : uowPathInformation.getPathByDataFileType() + .entrySet()) { + + Map datastoreWalkerRegexpValues = datastoreWalker() + .regexpValues(entry.getKey().getLocation(), entry.getValue()); + for (Map.Entry regexpEntry : datastoreWalkerRegexpValues.entrySet()) { + if (regexpNamesToExclude.contains(regexpEntry.getKey())) { + continue; + } + + // Capture any regexp values we don't already have. + if (uowRegexpValuesByName.get(regexpEntry.getKey()) == null) { + uowRegexpValuesByName.put(regexpEntry.getKey(), regexpEntry.getValue()); + } + + // If we have multiple input data file types, there may be regular expressions + // that differ from one data file type to another. For example, if one data file + // type has a location of "foo/bar$bar/blah" and another is "foo/bar$baz/blah", then + // the two will agree on the "foo" and the "blah" but not the "bar/baz" in the + // middle. Any regular expression that falls in + if (!uowRegexpValuesByName.get(regexpEntry.getKey()) + .equals(regexpEntry.getValue())) { + regexpNamesToExclude.add(regexpEntry.getKey()); + uowRegexpValuesByName.remove(regexpEntry.getKey()); + } + } + } + return uowRegexpValuesByName; + } + + /** Add to a {@link UnitOfWork} instance the datastore directory paths for that UOW. */ + private void populateDirectoryTypedParameters(UnitOfWork uow, + UowPathInformation uowPathInformation) { + for (Map.Entry entry : uowPathInformation.getPathByDataFileType() + .entrySet()) { + uow.addParameter(new TypedParameter( + DIRECTORY_PARAMETER_NAME + DirectoryUnitOfWorkGenerator.DIRECTORY_NAME_SEPARATOR + + entry.getKey().getName(), + entry.getValue().toString(), ZiggyDataType.ZIGGY_STRING)); + } + } + + /** Extracts datastore regexp values from typed parameters and returns as a Map. */ + @AcceptableCatchBlock(rationale = Rationale.CAN_NEVER_OCCUR) + public static Map regexpValues(UnitOfWork uow) { + String uowGenerator = uow.getParameter(GENERATOR_CLASS_PARAMETER_NAME).getString(); + Class uowGeneratorClass; + try { + uowGeneratorClass = Class.forName(uowGenerator); + } catch (ClassNotFoundException e) { + // This can never occur, since Ziggy used the generator in question to construct + // the UOW instance in the first place. + throw new AssertionError(e); + } + if (!DatastoreDirectoryUnitOfWorkGenerator.class.isAssignableFrom(uowGeneratorClass)) { + throw new IllegalArgumentException("Unit of work generator " + uowGenerator + + " is not DatastoreDirectoryUnitOfWorkGenerator or a subclass of same"); + } + Map regexpValues = new HashMap<>(); + for (TypedParameter typedParameter : uow.getParameters()) { + if (typedParameter.getName() + .startsWith(DirectoryUnitOfWorkGenerator.DIRECTORY_PARAMETER_NAME) + || typedParameter.getName().equals(UnitOfWork.BRIEF_STATE_PARAMETER_NAME) + || typedParameter.getName().equals(GENERATOR_CLASS_PARAMETER_NAME)) { + continue; + } + regexpValues.put(typedParameter.getName(), typedParameter.getString()); + } + return regexpValues; } /** - * Determines whether processing of the data in this UOW is performed in a single subtask or in - * multiple subtasks. Multiple subtasks are appropriate for the situation in which the - * processing is "embarrassingly parallel" (i.e., there are numerous chunks of data and there - * are no dependencies between the chunks, thus each chunk can be processed independently of any - * others). Single subtask processing is appropriate for situations in which all the data must - * be processed together due to dependencies between the data (a simple example: averaging all - * the chunks of data together requires them to be processed together, hence single subtask - * would be used in such a case). + * The {@link setBriefState} method is empty because we set the brief state in the UOW + * generator, above. */ - public boolean isSingleSubtask() { - return singleSubtask; + @Override + public void setBriefState(UnitOfWork uow, PipelineInstanceNode pipelineInstanceNode) { + } + + public DatastoreWalker datastoreWalker() { + if (datastoreWalker == null) { + datastoreWalker = DatastoreWalker.newInstance(); + } + return datastoreWalker; + } + + /** Uses a fluent pattern to assemble a UOW brief state from Strings. */ + public static class DatastoreDirectoryBriefStateBuilder { + + private final List uowParts = new ArrayList<>(); + + public DatastoreDirectoryBriefStateBuilder() { + } + + public DatastoreDirectoryBriefStateBuilder addUowPart(String uowPart) { + uowParts.add(uowPart); + return this; + } + + public DatastoreDirectoryBriefStateBuilder addUowPart(Path path) { + uowParts.add(path.getFileName().toString()); + return this; + } + + public String build() { + if (CollectionUtils.isEmpty(uowParts)) { + return new String(); + } + StringBuilder sb = new StringBuilder(BRIEF_STATE_OPEN_STRING); + for (String uowPart : uowParts) { + sb.append(uowPart); + sb.append(BRIEF_STATE_PART_SEPARATOR); + } + sb.setLength(sb.length() - 1); + sb.append(BRIEF_STATE_CLOSE_STRING); + return sb.toString(); + } } - public void setSingleSubtask(boolean singleSubtask) { - this.singleSubtask = singleSubtask; + public static String singlePartBriefState() { + return new DatastoreDirectoryBriefStateBuilder().addUowPart(SINGLE_UOW_BRIEF_STATE).build(); + } + + /** Container for holding a data file type, a brief state, and a data path. */ + private static class DataFileTypeInformation { + + private final DataFileType dataFileType; + private final String briefState; + private final Path dataFilePath; + + public DataFileTypeInformation(DataFileType dataFileType, String briefState, + Path dataFilePath) { + this.dataFileType = dataFileType; + this.briefState = briefState; + this.dataFilePath = dataFilePath; + } + + public DataFileType getDataFileType() { + return dataFileType; + } + + public String getBriefState() { + return briefState; + } + + private Path getDataFilePath() { + return dataFilePath; + } + } + + /** + * Container that holds the brief state and the {@link Map} of datastore paths by data file type + * for a single unit of work. + * + * @author PT + */ + private static class UowPathInformation { + + private final String briefState; + private final Map pathByDataFileType = new HashMap<>(); + + public UowPathInformation(String briefState) { + this.briefState = briefState; + } + + public String getBriefState() { + return briefState; + } + + public Map getPathByDataFileType() { + return pathByDataFileType; + } + + public void addDataFileTypeAndPath(DataFileType dataFileType, Path path) { + pathByDataFileType.put(dataFileType, path); + } } } diff --git a/src/main/java/gov/nasa/ziggy/uow/DefaultUnitOfWorkIdentifier.java b/src/main/java/gov/nasa/ziggy/uow/DefaultUnitOfWorkIdentifier.java deleted file mode 100644 index 7cf58e0..0000000 --- a/src/main/java/gov/nasa/ziggy/uow/DefaultUnitOfWorkIdentifier.java +++ /dev/null @@ -1,26 +0,0 @@ -package gov.nasa.ziggy.uow; - -import gov.nasa.ziggy.pipeline.definition.PipelineModule; -import gov.nasa.ziggy.services.config.PropertyName; - -/** - * Abstract superclass for classes that map {@link PipelineModule} subclasses to - * {@link UnitOfWorkGenerator} subclasses. This allows different pipeline modules to use different - * concrete classes to generate their own units of work. When implementing this class, provide a - * no-argument constructor. Specify the fully-qualified name of your subclass in the property - * {@link PropertyName#PIPELINE_DEFAULT_UOW_IDENTIFIER_CLASS}. - *

      - * The concrete class that supports this functionality for Ziggy pipeline modules does not yet - * exist. In the meantime, use the method {@link UnitOfWorkGenerator#ziggyDefaultUowGenerators()}. - * - * @author PT - */ -public abstract class DefaultUnitOfWorkIdentifier { - - /** - * Determines the default {@link UnitOfWorkGenerator} subclass for a given subclass of - * {@link PipelineModule}. - */ - public abstract Class defaultUnitOfWorkGeneratorForClass( - Class module); -} diff --git a/src/main/java/gov/nasa/ziggy/uow/DirectoryUnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/DirectoryUnitOfWorkGenerator.java index 5b14c8d..92f1653 100644 --- a/src/main/java/gov/nasa/ziggy/uow/DirectoryUnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/DirectoryUnitOfWorkGenerator.java @@ -1,170 +1,89 @@ package gov.nasa.ziggy.uow; -import java.io.IOException; -import java.nio.file.Files; import java.nio.file.Path; -import java.util.ArrayList; +import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.regex.Matcher; -import java.util.regex.Pattern; import java.util.stream.Collectors; -import java.util.stream.Stream; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.definition.TypedParameter; -import gov.nasa.ziggy.util.RegexBackslashManager; +import gov.nasa.ziggy.util.AcceptableCatchBlock; +import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; /** * Defines a UOW generator in which the units of work are based on the subdirectories of a parent - * directory. The class uses an instance of {@link TaskConfigurationParameters} to specify a regular - * expression that is used to identify the units of work. This allows the user to specify that some - * but not all subdirectories are to be included, or that the subdirectories should be further than - * 1 level below the parent directory. + * directory. * * @author PT */ public abstract class DirectoryUnitOfWorkGenerator implements UnitOfWorkGenerator { - private static final Logger log = LoggerFactory.getLogger(DirectoryUnitOfWorkGenerator.class); - - public static final String DIRECTORY_PROPERTY_NAME = "directory"; + public static final String DIRECTORY_PARAMETER_NAME = "directory"; public static final String REGEX_PROPERTY_NAME = "taskDirectoryRegex"; - - @Override - public List> requiredParameterClasses() { - List> requiredParameterClasses = new ArrayList<>(); - requiredParameterClasses.add(TaskConfigurationParameters.class); - return requiredParameterClasses; - } + public static final String DIRECTORY_NAME_SEPARATOR = ":"; /** * Convenience method that extracts the directory from a UOW instance constructed by a subclass - * of {@link DirectoryUnitOfWorkGenerator}. - * - * @param uow - * @return + * of {@link DirectoryUnitOfWorkGenerator}. If the UOW has more than one directory typed + * parameter, the values are returned in the form produced by the toString() method of Java + * {@link List}s. */ public static String directory(UnitOfWork uow) { - String clazz = uow.getParameter(UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME) - .getString(); - try { - Class cls = Class.forName(clazz); - if (DirectoryUnitOfWorkGenerator.class.isAssignableFrom(cls)) { - return uow.getParameter(DIRECTORY_PROPERTY_NAME).getString(); - } - throw new PipelineException( - "Class " + clazz + " not a subclass of DirectoryUnitOfWorkGenerator"); - } catch (ClassNotFoundException e) { - throw new PipelineException("Generator class " + clazz + " not found", e); + checkUowClass(uow); + List directories = directories(uow); + if (directories.size() == 0) { + return ""; } + if (directories.size() == 1) { + return directories.get(0); + } + return directories.toString(); } - /** - * Returns the directory to be used as the top-level directory for generation of UOW instances, - * as a {@link Path}. - */ - protected abstract Path rootDirectory(); - - @Override - public List generateTasks( - Map, ParametersInterface> parameters) { - String taskDirectoryRegex = taskDirectoryRegex(parameters); - List unitsOfWork = new ArrayList<>(); + /** Returns a {@link List} of directories for the given UOW. */ + public static List directories(UnitOfWork uow) { + checkUowClass(uow); + return uow.getParameters() + .stream() + .filter(s -> s.getName().startsWith(DIRECTORY_PARAMETER_NAME)) + .map(TypedParameter::getString) + .collect(Collectors.toList()); + } - // If there's no taskDirectoryRegex, then return a single task with no directory field. - // This will signal to the pipeline module that the parent directory itself should be - // used for the unit of work. - if (taskDirectoryRegex == null || taskDirectoryRegex.isEmpty()) { - UnitOfWork uow = new UnitOfWork(); - uow.addParameter( - new TypedParameter(DIRECTORY_PROPERTY_NAME, "", ZiggyDataType.ZIGGY_STRING)); - uow.addParameter( - new TypedParameter(REGEX_PROPERTY_NAME, "", ZiggyDataType.ZIGGY_STRING)); - unitsOfWork.add(uow); - return unitsOfWork; + /** Returns a {@link Map} from data file type names to directories in a UOW. */ + public static Map directoriesByDataFileType(UnitOfWork uow) { + checkUowClass(uow); + Map directoriesByDataFileType = new HashMap<>(); + for (TypedParameter parameter : uow.getParameters()) { + if (parameter.getName().startsWith(DIRECTORY_PARAMETER_NAME)) { + String[] splitParameterName = parameter.getName().split(DIRECTORY_NAME_SEPARATOR); + directoriesByDataFileType.put(splitParameterName[1], parameter.getString()); + } } + return directoriesByDataFileType; + } - // build a Pattern from the task dir regex - Pattern taskDirPattern = Pattern.compile(taskDirectoryRegex); - - // determine the number of directory levels below the datastore root - int dirLevelsCount = taskDirectoryRegex.split("/").length; - - // Get all directories below datastore root down to the specified depth - log.info("Searching for UOW directories in parent directory " + rootDirectory().toString()); - try (Stream allDirs = Files.walk(rootDirectory(), dirLevelsCount)) { - List taskDirs = allDirs.filter(Files::isDirectory) - .map(s -> rootDirectory().relativize(s)) - .filter(s -> taskDirPattern.matcher(s.toString()).matches()) - .collect(Collectors.toList()); - log.info("Located " + taskDirs.size() + " subdirectories to parent directory"); - for (Path taskDir : taskDirs) { - log.info("Processing directory " + taskDir.toString()); - UnitOfWork uow = new UnitOfWork(); - uow.addParameter(new TypedParameter(DIRECTORY_PROPERTY_NAME, taskDir.toString(), - ZiggyDataType.ZIGGY_STRING)); - uow.addParameter(new TypedParameter(REGEX_PROPERTY_NAME, taskDirectoryRegex, - ZiggyDataType.ZIGGY_STRING)); - - unitsOfWork.add(uow); + /** Ensures that the UOW was generated by a DirectoryUnitOfWorkGenerator subclass. */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + private static void checkUowClass(UnitOfWork uow) { + String generatorClassName = uow + .getParameter(UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME) + .getString(); + try { + Class clazz = Class.forName(generatorClassName); + if (!DirectoryUnitOfWorkGenerator.class.isAssignableFrom(clazz)) { + throw new PipelineException("Class " + generatorClassName + + " not a subclass of DirectoryUnitOfWorkGenerator"); } - } catch (IOException e) { - throw new PipelineException("IO Exception occurred when attempting to construct UOWs", - e); + } catch (ClassNotFoundException e) { + throw new PipelineException("Generator class " + generatorClassName + " not found", e); } - return unitsOfWork; } /** - * Returns the regular expression that defines the directory names that are allowed to become - * units of work. Broken out as a separate method to allow overriding. + * Returns the directory to be used as the top-level directory for generation of UOW instances, + * as a {@link Path}. */ - protected String taskDirectoryRegex( - Map, ParametersInterface> parameters) { - TaskConfigurationParameters taskConfigurationParameters = (TaskConfigurationParameters) parameters - .get(TaskConfigurationParameters.class); - if (taskConfigurationParameters == null) { - return new String(); - } - String bareRegex = taskConfigurationParameters.getTaskDirectoryRegex(); - if (bareRegex == null || bareRegex.isEmpty()) { - return new String(); - } - return RegexBackslashManager.toSingleBackslash(bareRegex); - } - - @Override - public String briefState(UnitOfWork uow) { - - String directory = uow.getParameter(DIRECTORY_PROPERTY_NAME).getString(); - String taskDirectoryRegex = uow.getParameter(REGEX_PROPERTY_NAME).getString(); - if (directory.isEmpty()) { - return rootDirectory().toString(); - } - if (taskDirectoryRegex.isEmpty()) { - return directory; - } - // If the regex has captured groups, they become the brief state of - // the UOW. Otherwise, the full dir is the brief state - Matcher matcher = Pattern.compile(taskDirectoryRegex).matcher(directory); - matcher.matches(); - StringBuilder briefStateBuilder = new StringBuilder(); - if (matcher.groupCount() > 0) { - for (int i = 1; i <= matcher.groupCount(); i++) { - briefStateBuilder.append(matcher.group(i)); - if (i < matcher.groupCount()) { - briefStateBuilder.append(","); - } - } - } else { - briefStateBuilder.append(matcher.group(0)); - } - return briefStateBuilder.toString(); - } + protected abstract Path rootDirectory(); } diff --git a/src/main/java/gov/nasa/ziggy/uow/SingleUnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/SingleUnitOfWorkGenerator.java index 848e586..3def133 100644 --- a/src/main/java/gov/nasa/ziggy/uow/SingleUnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/SingleUnitOfWorkGenerator.java @@ -1,13 +1,11 @@ package gov.nasa.ziggy.uow; -import java.util.Collections; import java.util.LinkedList; import java.util.List; -import java.util.Map; import org.apache.commons.lang3.builder.ReflectionToStringBuilder; -import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; public class SingleUnitOfWorkGenerator implements UnitOfWorkGenerator { @@ -20,22 +18,16 @@ public String toString() { } @Override - public String briefState(UnitOfWork uow) { - return "single"; - } - - @Override - public List> requiredParameterClasses() { - return Collections.emptyList(); - } - - @Override - public List generateTasks( - Map, ParametersInterface> parameters) { + public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode) { List tasks = new LinkedList<>(); UnitOfWork prototypeTask = new UnitOfWork(); tasks.add(prototypeTask); return tasks; } + + @Override + public void setBriefState(UnitOfWork uow, PipelineInstanceNode pipelineInstanceNode) { + uow.setBriefState("single"); + } } diff --git a/src/main/java/gov/nasa/ziggy/uow/TaskConfigurationParameters.java b/src/main/java/gov/nasa/ziggy/uow/TaskConfigurationParameters.java deleted file mode 100644 index 5fe5e13..0000000 --- a/src/main/java/gov/nasa/ziggy/uow/TaskConfigurationParameters.java +++ /dev/null @@ -1,76 +0,0 @@ -package gov.nasa.ziggy.uow; - -import gov.nasa.ziggy.module.io.ProxyIgnore; -import gov.nasa.ziggy.parameters.Parameters; - -/** - * Defines the task and subtask generation for Ziggy unit of work generators. - *

      - * The taskDirectoryRegex parameter defines a regular expression for the directories below the - * datastore root that are to be made into units of work. For example, "sector-([0-9]{4})/cal" would - * make a unit of work for directories under the datastore root such as sector-0001/cal, - * sector-0002/cal, etc. The singleSubtask parameter indicates whether each task should have a - * single subtask rather than generating 1 subtask per file based on the pipeline module's - * DataFileType instances. - * - * @author PT - */ -public class TaskConfigurationParameters extends Parameters { - - private String taskDirectoryRegex; - private boolean singleSubtask; - private int maxFailedSubtaskCount; - private boolean reprocess; - private int maxAutoResubmits; - - @ProxyIgnore - private long[] reprocessingTasksExclude = {}; - - public String getTaskDirectoryRegex() { - return taskDirectoryRegex; - } - - public void setTaskDirectoryRegex(String taskDirectoryRegex) { - this.taskDirectoryRegex = taskDirectoryRegex; - } - - public boolean isSingleSubtask() { - return singleSubtask; - } - - public void setSingleSubtask(boolean singleSubtask) { - this.singleSubtask = singleSubtask; - } - - public int getMaxFailedSubtaskCount() { - return maxFailedSubtaskCount; - } - - public void setMaxFailedSubtaskCount(int maxFailedSubtaskCount) { - this.maxFailedSubtaskCount = maxFailedSubtaskCount; - } - - public boolean isReprocess() { - return reprocess; - } - - public void setReprocess(boolean reprocess) { - this.reprocess = reprocess; - } - - public long[] getReprocessingTasksExclude() { - return reprocessingTasksExclude; - } - - public void setReprocessingTasksExclude(long[] reprocessingTasksExclude) { - this.reprocessingTasksExclude = reprocessingTasksExclude; - } - - public int getMaxAutoResubmits() { - return maxAutoResubmits; - } - - public void setMaxAutoResubmits(int maxAutoResubmits) { - this.maxAutoResubmits = maxAutoResubmits; - } -} diff --git a/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java b/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java index 5b247b2..86352bd 100644 --- a/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java +++ b/src/main/java/gov/nasa/ziggy/uow/UnitOfWork.java @@ -2,6 +2,7 @@ import java.io.Serializable; +import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.pipeline.definition.TypedParameter; import gov.nasa.ziggy.pipeline.definition.TypedParameterCollection; @@ -20,7 +21,8 @@ *

      * All instances of {@link UnitOfWork} must have a {@link String} property, "briefState," which is * used to identify the UOW of each pipeline task when displayed on the pipeline console. The UOW - * generators must populate this property. + * generators must populate this property. The {@link #setBriefState(String)} method allows the + * brief state value to be set for the unit of work. * * @author PT */ @@ -35,6 +37,11 @@ public String briefState() { return getParameter(BRIEF_STATE_PARAMETER_NAME).getString(); } + public void setBriefState(String briefState) { + addParameter( + new TypedParameter(BRIEF_STATE_PARAMETER_NAME, briefState, ZiggyDataType.ZIGGY_STRING)); + } + /** * Allow the UOWs to sort by brief state. */ diff --git a/src/main/java/gov/nasa/ziggy/uow/UnitOfWorkGenerator.java b/src/main/java/gov/nasa/ziggy/uow/UnitOfWorkGenerator.java index 0c49b57..4fe5766 100644 --- a/src/main/java/gov/nasa/ziggy/uow/UnitOfWorkGenerator.java +++ b/src/main/java/gov/nasa/ziggy/uow/UnitOfWorkGenerator.java @@ -1,46 +1,20 @@ package gov.nasa.ziggy.uow; -import java.lang.reflect.InvocationTargetException; import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; +import java.util.Set; -import com.google.common.collect.ImmutableMap; - -import gov.nasa.ziggy.collections.ZiggyDataType; -import gov.nasa.ziggy.data.management.DataReceiptPipelineModule; -import gov.nasa.ziggy.module.ExternalProcessPipelineModule; -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.parameters.Parameters; -import gov.nasa.ziggy.parameters.ParametersInterface; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineModule; -import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; -import gov.nasa.ziggy.pipeline.definition.TypedParameter; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; -import gov.nasa.ziggy.services.config.PropertyName; -import gov.nasa.ziggy.services.config.ZiggyConfiguration; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; /** * Interface for all unit of work generators. A unit of work generator constructs instances of the * {@link UnitOfWork} class that can be used by pipeline modules to determine which set of data the * corresponding pipeline task should process (for example: a time range). *

      - * Implementations of {@link UnitOfWorkGenerator} are required to provide the following methods: - *

        - *
      1. {@link #requiredParameterClasses()}, which specifies the implementations of - * {@link Parameters} that the generator requires to generate units of work. During UOW generation, - * {@link UnitOfWorkGenerator} will ensure that instances of each required parameter class have been - * supplied as arguments to {@link #generateTasks(Map)}. - *
      2. {@link #generateTasks(Map)}, which returns a {@link List} of {@link UnitOfWork} instances. - * The method can make use of the {@link Parameters} instances passed to it in the form of a - * {@link Map}. - *
      3. {@link #briefState(UnitOfWork)}, which generates a brief state {@link String} for an instance - * based on its properties, and adds same to the properties collection in the {@link UnitOfWork} - * instance. This is used for display purposes, and so should be informative about what the UOW - * represents. - *
      + * Implementations of {@link UnitOfWorkGenerator} are required to provide the + * {@link #generateUnitsOfWork(PipelineInstanceNode)}, method, which generates the units of work for + * a given {@link PipelineInstanceNode}. They may also optionally override the + * {@link #generateUnitsOfWork(PipelineInstanceNode, Set)} method, which allows a UOW generator to + * make use of event labels from a Ziggy event generator. * * @author Todd Klaus * @author PT @@ -49,126 +23,54 @@ public interface UnitOfWorkGenerator { String GENERATOR_CLASS_PARAMETER_NAME = "uowGenerator"; - static Map, Class> ziggyDefaultUowGenerators() { - return ImmutableMap.of(ExternalProcessPipelineModule.class, - DatastoreDirectoryUnitOfWorkGenerator.class, DataReceiptPipelineModule.class, - DataReceiptUnitOfWorkGenerator.class); + /** Constructs completely-populated units of work, including brief state. */ + default List unitsOfWork(PipelineInstanceNode pipelineInstanceNode) { + List unitsOfWork = generateUnitsOfWork(pipelineInstanceNode); + setBriefStates(unitsOfWork, pipelineInstanceNode); + return unitsOfWork; } /** - * Should return the {@link Parameters} classes required by this task generator, or and empty - * list if no {@link Parameters} classes are required. - *

      - * Used by the console to prevent misconfigurations. - */ - List> requiredParameterClasses(); - - /** - * Generate the task objects for this unit of work. This method must be supplied by the - * implementing class. It is used in conjunction with {@link #briefState(UnitOfWork)} to - * generate all UOWs for all tasks. - */ - List generateTasks( - Map, ParametersInterface> parameters); - - /** - * Generates the content of the briefState property based on the other properties in a UOW. It - * is used in conjunction with {@link #generateTasks(Map)} to generate all UOWs for all tasks. + * Constructs completely populated units of work, including brief state and making use of event + * labels. */ - String briefState(UnitOfWork uow); - - /** - * Generates the set of UOWs using the {@link #generateTasks(Map)} and - * {@link #briefState(UnitOfWork)} methods of a given implementation. The resulting - * {@link UnitOfWork} instance will also contain a property that specifies the class name of the - * generator. - */ - default List generateUnitsOfWork( - Map, ParametersInterface> parameters) { - - // Produce the tasks and sort by brief state - List uows = generateTasks(parameters); + default List unitsOfWork(PipelineInstanceNode pipelineInstanceNode, + Set eventLabels) { + List unitsOfWork = generateUnitsOfWork(pipelineInstanceNode, eventLabels); + setBriefStates(unitsOfWork, pipelineInstanceNode); + return unitsOfWork; + } - // Add some metadata parameters to all the instances. - for (UnitOfWork uow : uows) { - uow.addParameter(new TypedParameter(UnitOfWork.BRIEF_STATE_PARAMETER_NAME, - briefState(uow), ZiggyDataType.ZIGGY_STRING)); - uow.addParameter(new TypedParameter(UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, - getClass().getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); + /** Assigns brief states to a {@Link List} of {@link UnitOfWork} instances. */ + default void setBriefStates(List unitsOfWork, + PipelineInstanceNode pipelineInstanceNode) { + for (UnitOfWork uow : unitsOfWork) { + setBriefState(uow, pipelineInstanceNode); } - - // Now that the UOWs have their brief states properly assigned, sort them by brief state - // and return. - return uows.stream().sorted().collect(Collectors.toList()); } /** - * Obtains the UOW generator for a pipeline node. The generator is the one in the database for - * that node. If none is present, the default unit of work generator must be looked up and - * returned. + * Generate the units of work for the current module in the current pipeline run. */ - static ClassWrapper unitOfWorkGenerator(PipelineDefinitionNode node) { + List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode); - ClassWrapper unitOfWorkGenerator = node.getUnitOfWorkGenerator(); - if (unitOfWorkGenerator == null) { - - // Get the current definition of the pipeline module - PipelineModuleDefinition module = new PipelineModuleDefinitionCrud() - .retrieveLatestVersionForName(node.getModuleName()); - Class moduleClass = module.getPipelineModuleClass() - .getClazz(); - Class uowClass = defaultUnitOfWorkGenerator(moduleClass); - unitOfWorkGenerator = new ClassWrapper<>(uowClass); - } - return unitOfWorkGenerator; + /** + * Generate units of work, taking into account labels from an event. By default, the event + * labels are ignored. Override this method to make a UOW generator capable of using the event + * labels. + */ + default List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode, + Set eventLabels) { + return generateUnitsOfWork(pipelineInstanceNode); } /** - * Determines the {@link UnitOfWorkGenerator} class that serves as the default UOW for a given - * subclass of {@link PipelineModule}. The user-specified implementation of - * {@link DefaultUnitOfWorkIdentifier}, if any, is used first. If the UOW class is not found in - * that implementation, or if that implementation does not exist, the - * {@link #ziggyDefaultUowGenerators()} is tried as a fallback. + * Determines the brief state string for a given {@link UnitOfWork} instance. This is used by + * the {@link #unitsOfWork(PipelineInstanceNode)} methods to ensure that the brief state is set + * before returning the UOWs to a caller. + * + * @param uow + * @param pipelineInstanceNode */ - static Class defaultUnitOfWorkGenerator( - Class module) { - - Class defaultUnitOfWorkGenerator = null; - - // Start by using the pipeline-side identifier for default UOWs, if any is specified - String pipelineUowIdentifierClassName = ZiggyConfiguration.getInstance() - .getString(PropertyName.PIPELINE_DEFAULT_UOW_IDENTIFIER_CLASS.property(), null); - if (pipelineUowIdentifierClassName != null) { - - // Try to instantiate the Pipeline-side default UOW generator, and throw an exception - // if unable to do so. - try { - DefaultUnitOfWorkIdentifier pipelineUowIdentifier = (DefaultUnitOfWorkIdentifier) Class - .forName(pipelineUowIdentifierClassName) - .getConstructor() - .newInstance(); - defaultUnitOfWorkGenerator = pipelineUowIdentifier - .defaultUnitOfWorkGeneratorForClass(module); - } catch (InstantiationException | IllegalAccessException | ClassNotFoundException - | IllegalArgumentException | InvocationTargetException | NoSuchMethodException - | SecurityException e) { - throw new PipelineException("Pipeline default unit of work generator class " - + pipelineUowIdentifierClassName + " could not be instantiated", e); - } - } - - if (defaultUnitOfWorkGenerator == null) { // try the Ziggy version - defaultUnitOfWorkGenerator = ziggyDefaultUowGenerators().get(module); - } - - // If we still don't have a default UOW generator, throw an exception, since the only case - // in which this method is called is when the user specified that the default generator for - // a module was supposed to be used, and if the module doesn't have one then we have to fail - // out. - if (defaultUnitOfWorkGenerator == null) { - throw new PipelineException( - "Unable to locate default UOW generator for " + module.getName()); - } - return defaultUnitOfWorkGenerator; - } + void setBriefState(UnitOfWork uow, PipelineInstanceNode pipelineInstanceNode); } diff --git a/src/main/java/gov/nasa/ziggy/ui/util/HumanReadableHeapSize.java b/src/main/java/gov/nasa/ziggy/util/HumanReadableHeapSize.java similarity index 98% rename from src/main/java/gov/nasa/ziggy/ui/util/HumanReadableHeapSize.java rename to src/main/java/gov/nasa/ziggy/util/HumanReadableHeapSize.java index 830f6ee..befab57 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/HumanReadableHeapSize.java +++ b/src/main/java/gov/nasa/ziggy/util/HumanReadableHeapSize.java @@ -1,4 +1,4 @@ -package gov.nasa.ziggy.ui.util; +package gov.nasa.ziggy.util; /** * Support for human-readable heap sizes, and conversions between human-readable and heap sizes in diff --git a/src/main/java/gov/nasa/ziggy/util/Iso8601Formatter.java b/src/main/java/gov/nasa/ziggy/util/Iso8601Formatter.java index e7a82f1..372fe7a 100644 --- a/src/main/java/gov/nasa/ziggy/util/Iso8601Formatter.java +++ b/src/main/java/gov/nasa/ziggy/util/Iso8601Formatter.java @@ -2,6 +2,7 @@ import java.text.DateFormat; import java.text.SimpleDateFormat; +import java.util.Date; import java.util.HashMap; import java.util.Map; import java.util.TimeZone; diff --git a/src/main/java/gov/nasa/ziggy/util/LogSectionBreak.java b/src/main/java/gov/nasa/ziggy/util/LogSectionBreak.java deleted file mode 100644 index f01a7f2..0000000 --- a/src/main/java/gov/nasa/ziggy/util/LogSectionBreak.java +++ /dev/null @@ -1,28 +0,0 @@ -package gov.nasa.ziggy.util; - -import org.slf4j.Logger; - -public class LogSectionBreak { - - private LogSectionBreak() { - } - - /** - * Writes a visible section break into a log file - * - * @param log the log file which needs the section break - * @param message a message, if any, which is to be written to the middle of the break - */ - public static void writeSectionBreak(Logger log, String message) { - String breakString = "//========================================================="; - log.info(""); - log.info(breakString); - log.info("//"); - log.info("//"); - log.info("// " + message); - log.info("//"); - log.info("//"); - log.info(breakString); - log.info(""); - } -} diff --git a/src/main/java/gov/nasa/ziggy/util/RegexGroupCounter.java b/src/main/java/gov/nasa/ziggy/util/RegexGroupCounter.java index a4a9781..7234f60 100644 --- a/src/main/java/gov/nasa/ziggy/util/RegexGroupCounter.java +++ b/src/main/java/gov/nasa/ziggy/util/RegexGroupCounter.java @@ -12,12 +12,17 @@ public class RegexGroupCounter { public static final Pattern GROUP_PATTERN = Pattern.compile("\\(([^)]+)\\)"); + /** + * Provides a count of the number of groups in a regular expression. + *

      + * Warning: Nested groups are not counted properly. + */ public static int groupCount(String regex) { Matcher groupMatcher = GROUP_PATTERN.matcher(regex); - int iGroup = 0; + int groupCount = 0; while (groupMatcher.find()) { - iGroup++; + groupCount++; } - return iGroup; + return groupCount; } } diff --git a/src/main/java/gov/nasa/ziggy/util/SystemProxy.java b/src/main/java/gov/nasa/ziggy/util/SystemProxy.java index be9820d..d0b3290 100644 --- a/src/main/java/gov/nasa/ziggy/util/SystemProxy.java +++ b/src/main/java/gov/nasa/ziggy/util/SystemProxy.java @@ -73,7 +73,7 @@ public static void setUserTime(long userTime) { * Note that {@link #disableExit()} is a one-time use method, in that the first use of * {@link #exit(int)} after the call to {@link #disableExit()} will re-enable calls to the * system exit method. Hence the recommended use of this method is to call it from a unit test's - * {@link Before} method. + * {@code Before} method. */ public static void disableExit() { exitEnabled = false; diff --git a/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java b/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java index 11d04fd..26bf2ae 100644 --- a/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java +++ b/src/main/java/gov/nasa/ziggy/util/TaskProcessingTimeStats.java @@ -102,38 +102,19 @@ public int getCount() { return count; } - /** - * @return the minStart - */ public Date getMinStart() { return minStart; } - /** - * @return the maxEnd - */ public Date getMaxEnd() { return maxEnd; } - /** - * @return the totalElapsed - */ public double getTotalElapsed() { return totalElapsed; } - /** - * @return the sum - */ public double getSum() { return sum; } - - /** - * @param sum the sum to set - */ - public void setSum(double sum) { - this.sum = sum; - } } diff --git a/src/main/java/gov/nasa/ziggy/util/TasksStates.java b/src/main/java/gov/nasa/ziggy/util/TasksStates.java index 9f0e394..e372fe9 100644 --- a/src/main/java/gov/nasa/ziggy/util/TasksStates.java +++ b/src/main/java/gov/nasa/ziggy/util/TasksStates.java @@ -15,13 +15,13 @@ */ public class TasksStates { public class Summary { - private int submittedCount = 0; - private int processingCount = 0; - private int errorCount = 0; - private int completedCount = 0; - private int subTaskTotalCount = 0; - private int subTaskCompleteCount = 0; - private int subTaskFailedCount = 0; + private int submittedCount; + private int processingCount; + private int errorCount; + private int completedCount; + private int subTaskTotalCount; + private int subTaskCompleteCount; + private int subTaskFailedCount; public int getSubmittedCount() { return submittedCount; @@ -52,13 +52,13 @@ public int getSubTaskFailedCount() { } } - private int totalSubmittedCount = 0; - private int totalProcessingCount = 0; - private int totalErrorCount = 0; - private int totalCompletedCount = 0; - private int totalSubTaskTotalCount = 0; - private int totalSubTaskCompleteCount = 0; - private int totalSubTaskFailedCount = 0; + private int totalSubmittedCount; + private int totalProcessingCount; + private int totalErrorCount; + private int totalCompletedCount; + private int totalSubTaskTotalCount; + private int totalSubTaskCompleteCount; + private int totalSubTaskFailedCount; private final List moduleNames = new LinkedList<>(); private final Map moduleStates = new HashMap<>(); diff --git a/src/main/java/gov/nasa/ziggy/util/TimeFormatter.java b/src/main/java/gov/nasa/ziggy/util/TimeFormatter.java index 58806ce..ccb281b 100644 --- a/src/main/java/gov/nasa/ziggy/util/TimeFormatter.java +++ b/src/main/java/gov/nasa/ziggy/util/TimeFormatter.java @@ -1,5 +1,11 @@ package gov.nasa.ziggy.util; +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + +import java.util.regex.Matcher; +import java.util.regex.Pattern; + /** * Utilities for converting time from one format to another. * @@ -7,10 +13,15 @@ */ public class TimeFormatter { + private static final Pattern TIME_REGEXP = Pattern.compile("(\\d+:\\d+)(:\\d+)?"); + /** * Convert a string in HH:mm:SS format to a double-precision number of hours. */ public static double timeStringHhMmSsToTimeInHours(String timeString) { + checkNotNull(timeString, "timeString"); + checkArgument(!timeString.isEmpty(), "timeString can't be empty"); + String[] wallTimeChunks = timeString.split(":"); return Double.parseDouble(wallTimeChunks[0]) + Double.parseDouble(wallTimeChunks[1]) / 60 + Double.parseDouble(wallTimeChunks[2]) / 3600; @@ -20,6 +31,9 @@ public static double timeStringHhMmSsToTimeInHours(String timeString) { * Convert a string in HH:mm:SS format to a double-precision number of seconds. */ public static double timeStringHhMmSsToTimeInSeconds(String timeString) { + checkNotNull(timeString, "timeString"); + checkArgument(!timeString.isEmpty(), "timeString can't be empty"); + String[] wallTimeChunks = timeString.split(":"); return Double.parseDouble(wallTimeChunks[0]) * 3600 + Double.parseDouble(wallTimeChunks[1]) * 60 + Double.parseDouble(wallTimeChunks[2]); @@ -29,6 +43,8 @@ public static double timeStringHhMmSsToTimeInSeconds(String timeString) { * Convert a double-precision number of hours to a string in HH:mm:SS format. */ public static String timeInHoursToStringHhMmSs(double timeHours) { + checkArgument(timeHours >= 0, "timeHours can't be negative"); + StringBuilder sb = new StringBuilder(); double wallTimeHours = Math.floor(timeHours); sb.append((int) wallTimeHours); @@ -43,6 +59,8 @@ public static String timeInHoursToStringHhMmSs(double timeHours) { } public static String timeInSecondsToStringHhMmSs(int timeSeconds) { + checkArgument(timeSeconds >= 0, "timeSeconds can't be negative"); + double timeHours = (double) timeSeconds / 3600; return timeInHoursToStringHhMmSs(timeHours); } @@ -50,4 +68,22 @@ public static String timeInSecondsToStringHhMmSs(int timeSeconds) { private static String twoDigitString(double value) { return String.format("%02d", (int) value); } + + /** + * Given a time string such as 1:23:45, strip off the seconds so you're left with 1:23. If this + * method is given 1:23, 1:23 is returned. + * + * @param timeString a string of the form hh:mm[:ss] + * @return a string of the form hh:mm + */ + public static String stripSeconds(String timeString) { + checkNotNull(timeString, "timeString"); + checkArgument(!timeString.isEmpty(), "timeString can't be empty"); + + Matcher matcher = TIME_REGEXP.matcher(timeString); + if (matcher.find()) { + return matcher.group(1); + } + return timeString; + } } diff --git a/src/main/java/gov/nasa/ziggy/util/WrapperUtils.java b/src/main/java/gov/nasa/ziggy/util/WrapperUtils.java index 72e6962..70145bc 100644 --- a/src/main/java/gov/nasa/ziggy/util/WrapperUtils.java +++ b/src/main/java/gov/nasa/ziggy/util/WrapperUtils.java @@ -1,5 +1,8 @@ package gov.nasa.ziggy.util; +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + public class WrapperUtils { public static final String WRAPPER_LIBRARY_PATH_PROP_NAME_PREFIX = "wrapper.java.library.path."; @@ -25,6 +28,11 @@ public String toString() { * @param value the value of the parameter */ public static String wrapperParameter(String wrapperPropName, String value) { + checkNotNull(wrapperPropName, "wrapperPropName"); + checkArgument(!wrapperPropName.isEmpty(), "wrapperPropName can't be empty"); + checkNotNull(value, "value"); + checkArgument(!value.isEmpty(), "value can't be empty"); + StringBuilder s = new StringBuilder(); s.append(wrapperPropName).append("=").append(value); return s.toString(); @@ -38,6 +46,12 @@ public static String wrapperParameter(String wrapperPropName, String value) { * @param value the value of the parameter */ public static String wrapperParameter(String wrapperPropNamePrefix, int index, String value) { + checkNotNull(wrapperPropNamePrefix, "wrapperPropNamePrefix"); + checkArgument(!wrapperPropNamePrefix.isEmpty(), "wrapperPropNamePrefix can't be empty"); + checkNotNull(value, "value"); + checkArgument(!value.isEmpty(), "value can't be empty"); + checkArgument(index >= 0, "index must be non-negative"); + StringBuilder s = new StringBuilder(); s.append(wrapperPropNamePrefix).append(index).append("=").append(value); return s.toString(); diff --git a/src/main/java/gov/nasa/ziggy/util/StringUtils.java b/src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java similarity index 94% rename from src/main/java/gov/nasa/ziggy/util/StringUtils.java rename to src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java index 97f6957..609a0f6 100644 --- a/src/main/java/gov/nasa/ziggy/util/StringUtils.java +++ b/src/main/java/gov/nasa/ziggy/util/ZiggyStringUtils.java @@ -5,11 +5,15 @@ import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.Date; +import java.util.HashSet; import java.util.List; +import java.util.Set; import java.util.StringTokenizer; import java.util.regex.Matcher; import java.util.regex.Pattern; +import java.util.stream.Collectors; import org.apache.commons.lang3.time.DurationFormatUtils; import org.slf4j.Logger; @@ -23,8 +27,8 @@ * @author Todd Klaus * @author Thomas Han */ -public class StringUtils { - private static final Logger log = LoggerFactory.getLogger(StringUtils.class); +public class ZiggyStringUtils { + private static final Logger log = LoggerFactory.getLogger(ZiggyStringUtils.class); public static final String EMPTY = org.apache.commons.lang3.StringUtils.EMPTY; @@ -341,4 +345,16 @@ public static List breakStringAtLineTerminations(String string) { String[] splitString = string.split(System.lineSeparator()); return Arrays.asList(splitString); } + + /** + * Checks a {@link Collection} of {@link String}s for duplicates, and returns any that are + * found. + */ + public static List duplicateStrings(Collection strings) { + Set uniqueStrings = new HashSet<>(); + return strings.stream() + .filter(s -> !uniqueStrings.add(s)) + .distinct() + .collect(Collectors.toList()); + } } diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java index c232bb5..c05705d 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/DisplayModel.java @@ -4,7 +4,7 @@ import java.text.SimpleDateFormat; import java.util.Date; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; /** * Superclass for all DisplayModel classes. Contains abstract methods and print logic @@ -48,7 +48,7 @@ public void print(PrintStream ps, String title) { // print column headers for (int column = 0; column < getColumnCount(); column++) { - ps.print(StringUtils.pad(getColumnName(column), columnWidths[column])); + ps.print(ZiggyStringUtils.pad(getColumnName(column), columnWidths[column])); } ps.println(); @@ -62,7 +62,8 @@ public void print(PrintStream ps, String title) { // print table data for (int row = 0; row < getRowCount(); row++) { for (int column = 0; column < getColumnCount(); column++) { - ps.print(StringUtils.pad(getValueAt(row, column).toString(), columnWidths[column])); + ps.print( + ZiggyStringUtils.pad(getValueAt(row, column).toString(), columnWidths[column])); } ps.println(); } @@ -95,7 +96,7 @@ protected String formatDouble(double d) { return String.format("%.3f", d); } - public static String formatDate(Date d) { + public synchronized static String formatDate(Date d) { if (d.getTime() == 0) { return "-"; } diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java index a634675..18466a8 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/InstancesDisplayModel.java @@ -3,6 +3,8 @@ import java.util.LinkedList; import java.util.List; +import org.apache.commons.lang3.StringUtils; + import gov.nasa.ziggy.pipeline.definition.PipelineInstance; /** @@ -50,7 +52,8 @@ public Object getValueAt(int rowIndex, int columnIndex) { return switch (columnIndex) { case 0 -> instance.getId(); - case 1 -> instance.getPipelineDefinition().getName() + ": " + instance.getName(); + case 1 -> instance.getPipelineDefinition().getName() + + (StringUtils.isEmpty(instance.getName()) ? "" : ": " + instance.getName()); case 2 -> getStateString(instance.getState()); case 3 -> instance.elapsedTime(); default -> throw new IllegalArgumentException("Unexpected value: " + columnIndex); diff --git a/src/main/java/gov/nasa/ziggy/ui/util/models/TableModelContentClass.java b/src/main/java/gov/nasa/ziggy/util/dispmod/ModelContentClass.java similarity index 51% rename from src/main/java/gov/nasa/ziggy/ui/util/models/TableModelContentClass.java rename to src/main/java/gov/nasa/ziggy/util/dispmod/ModelContentClass.java index 9d6f1b0..e23eee6 100644 --- a/src/main/java/gov/nasa/ziggy/ui/util/models/TableModelContentClass.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/ModelContentClass.java @@ -1,13 +1,13 @@ -package gov.nasa.ziggy.ui.util.models; +package gov.nasa.ziggy.util.dispmod; /** - * Interface that provides the ability for table models to report the Java class for their content + * Interface that provides the ability for display models to report the Java class for their content * objects. All Ziggy table models must implement this interface. * * @author PT * @param Class of objects managed by the table model. */ -public interface TableModelContentClass { +public interface ModelContentClass { Class tableModelContentClass(); } diff --git a/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java b/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java index e08b02c..9088a9a 100644 --- a/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java +++ b/src/main/java/gov/nasa/ziggy/util/dispmod/PipelineStatsDisplayModel.java @@ -9,7 +9,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.PipelineTask.State; -import gov.nasa.ziggy.ui.util.models.TableModelContentClass; import gov.nasa.ziggy.util.TaskProcessingTimeStats; import gov.nasa.ziggy.util.dispmod.PipelineStatsDisplayModel.ProcessingStatistics; @@ -22,7 +21,7 @@ * @author Todd Klaus */ public class PipelineStatsDisplayModel extends DisplayModel - implements TableModelContentClass { + implements ModelContentClass { private static final String[] COLUMN_NAMES = { "Module", "State", "Count", "Sum (hrs)", "Min (hrs)", "Max (hrs)", "Mean (hrs)", "Std (hrs)", "Start", "End", "Elapsed (hrs)" }; diff --git a/src/main/java/gov/nasa/ziggy/util/io/FileUtil.java b/src/main/java/gov/nasa/ziggy/util/io/FileUtil.java index f594936..33f7b5a 100644 --- a/src/main/java/gov/nasa/ziggy/util/io/FileUtil.java +++ b/src/main/java/gov/nasa/ziggy/util/io/FileUtil.java @@ -1,5 +1,8 @@ package gov.nasa.ziggy.util.io; +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.base.Preconditions.checkNotNull; + import java.io.Closeable; import java.io.IOException; import java.io.InputStream; @@ -11,22 +14,29 @@ import java.nio.file.FileVisitResult; import java.nio.file.FileVisitor; import java.nio.file.Files; +import java.nio.file.LinkOption; import java.nio.file.Path; import java.nio.file.SimpleFileVisitor; +import java.nio.file.StandardCopyOption; import java.nio.file.attribute.BasicFileAttributes; import java.nio.file.attribute.PosixFilePermission; import java.nio.file.attribute.PosixFilePermissions; +import java.util.Collection; import java.util.HashMap; import java.util.Map; import java.util.Set; +import java.util.regex.Pattern; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import org.apache.commons.collections.CollectionUtils; import org.apache.commons.io.FileUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.google.common.collect.ImmutableSet; -import gov.nasa.ziggy.data.management.DataFileManager; +import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.util.AcceptableCatchBlock; import gov.nasa.ziggy.util.AcceptableCatchBlock.Rationale; @@ -37,6 +47,7 @@ * @author PT */ public class FileUtil { + static final int BUFFER_SIZE = 1000; private static final Logger log = LoggerFactory.getLogger(FileUtil.class); @@ -165,7 +176,7 @@ public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) { // Add the file and its real source to the map. try { - Path realFile = DataFileManager.realSourceFile(file); + Path realFile = FileUtil.realSourceFile(file); if (Files.isDirectory(realFile) || Files.isHidden(file)) { return FileVisitResult.CONTINUE; } @@ -346,4 +357,190 @@ public static void deleteDirectoryTree(Path directory, boolean force) { throw new UncheckedIOException("Unable to delete directory " + directory.toString(), e); } } + + /** + * Finds the actual source file for a given source file. If the source file is not a symbolic + * link, then that file is the actual source file. If not, the symbolic link is read to find the + * actual source file. The reading of symbolic links runs iteratively, so it produces the + * correct result even in the case of a link to a link to a link... etc. The process of + * following symbolic links stops at the first such link that is a child of the datastore root + * path. Thus the "actual source" is either a non-symlink file that the src file is a link to, + * or it's a file (symlink or regular file) that lies inside the datastore. + */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public static Path realSourceFile(Path src) { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + Path trueSrc = src; + if (Files.isSymbolicLink(src) && !src.startsWith(datastoreRoot)) { + try { + trueSrc = realSourceFile(Files.readSymbolicLink(src)); + } catch (IOException e) { + throw new UncheckedIOException("Unable to resolve symbolic link " + src.toString(), + e); + } + } + return trueSrc; + } + + /** Abstraction of the {@link Files#list(Path)} API for a fast, simple directory listing. */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public static Set listFiles(Path directory) { + return listFiles(directory, null, null); + } + + /** + * Abstraction of the {@link Files#list(Path)} API for a fast, simple directory listing that + * filters results according to a regular expression. + */ + public static Set listFiles(Path directory, String regexp) { + return listFiles(directory, Set.of(Pattern.compile(regexp)), null); + } + + /** + * Abstraction of the {@link Files#list(Path)} API for a fast, simple directory listing that + * filters results according to two collections of {@link Pattern}s. The first collection is of + * Patterns that must be matched (i.e., include patterns); the second collection is of Patterns + * that must not be matched (i.e., exclude patterns). Any file that matches both an include and + * an exclude pattern will be excluded. Either collection of Patterns can be empty, or null. + */ + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + public static Set listFiles(Path directory, Collection includePatterns, + Collection excludePatterns) { + try (Stream stream = Files.list(directory)) { + Stream filteredStream = stream; + if (!CollectionUtils.isEmpty(includePatterns)) { + for (Pattern includePattern : includePatterns) { + filteredStream = filteredStream + .filter(s -> includePattern.matcher(s.getFileName().toString()).matches()); + } + } + if (!CollectionUtils.isEmpty(excludePatterns)) { + for (Pattern excludePattern : excludePatterns) { + filteredStream = filteredStream + .filter(s -> !excludePattern.matcher(s.getFileName().toString()).matches()); + } + } + return filteredStream.collect(Collectors.toSet()); + } catch (IOException e) { + throw new UncheckedIOException(e); + } + } + + /** + * Enum-with-behavior that supports multiple different copy mechanisms that are specialized for + * use with moving files between the datastore and a working directory. The following options + * are provided: + *

        + *
      1. {@link CopyType#COPY} performs a traditional file copy operation. The copy is recursive, + * so directories are supported as well as individual files. + *
      2. {@link CopyType#LINK} makes the destination a hard link to the true source file, as + * defined by the {@link realSourceFile} method. Linking can be faster than copying and can + * consume less disk space (assuming the datastore and working directories are on the same file + * system). + *
      3. {@link CopyType#MOVE} will move the true source file to the destination; that is, it will + * follow symlinks via the {@link realSourceFile} method and move the file that is found in this + * way. In addition, if the source file is a symlink, it will be changed to a symlink to the + * moved file in its new location. In this way, the source file symlink remains valid and + * unchanged, but it now targets the moved file. to the moved file. + *
      + * In addition to all the foregoing, {@link CopyType} manages file permissions. After execution + * of any move / copy / symlink operation, the new file's permissions are set to make it + * write-protected and world-readable. If the copy / move / symlink operation is required to + * overwrite the destination file, that file's permissions will be set to allow the overwrite + * prior to execution. + *

      + * For copying files from the datastore to a subtask directory, {@link CopyType#COPY}, and + * {@link CopyType#LINK} options are available. For copies from the subtask directory to the + * datastore, {@link CopyType#MOVE} and {@link CopyType#LINK} are available. + * + * @author PT + */ + public enum CopyType { + COPY { + @Override + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + protected void copyInternal(Path src, Path dest) { + try { + checkArguments(src, dest); + if (Files.isRegularFile(src)) { + Files.copy(src, dest, StandardCopyOption.REPLACE_EXISTING); + } else { + FileUtils.copyDirectory(src.toFile(), dest.toFile()); + } + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to copy " + src.toString() + " to " + dest.toString(), e); + } + } + }, + MOVE { + @Override + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + protected void copyInternal(Path src, Path dest) { + try { + checkArguments(src, dest); + Path trueSrc = realSourceFile(src); + Files.move(trueSrc, dest, StandardCopyOption.REPLACE_EXISTING, + StandardCopyOption.ATOMIC_MOVE); + if (src != trueSrc) { + Files.createSymbolicLink(src, dest); + } + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to move " + src.toString() + " to " + dest.toString(), e); + } + } + }, + LINK { + @Override + @AcceptableCatchBlock(rationale = Rationale.EXCEPTION_CHAIN) + protected void copyInternal(Path src, Path dest) { + try { + checkArguments(src, dest); + Path trueSrc = realSourceFile(src); + if (Files.exists(dest)) { + Files.delete(dest); + } + createLink(trueSrc, dest); + } catch (IOException e) { + throw new UncheckedIOException( + "Unable to link from " + src.toString() + " to " + dest.toString(), e); + } + } + + /** Recursively copies directories and hard-links regular files. */ + private void createLink(Path src, Path dest) throws IOException { + if (Files.isDirectory(src)) { + Files.createDirectories(dest); + for (Path file : Files.list(src).collect(Collectors.toList())) { + createLink(file, dest.resolve(file.getFileName())); + } + } else { + Files.createLink(dest, src); + } + } + }; + + /** + * Copy operation that allows / forces the caller to manage any {@link IOException} that + * occurs. + */ + protected abstract void copyInternal(Path src, Path dest); + + /** + * Copy operation that manages any resulting {@link IOException}}. In this event, an + * {@link UncheckedIOException} is thrown, which terminates execution of the datastore + * operations. + */ + public void copy(Path src, Path dest) { + copyInternal(src, dest); + } + + private static void checkArguments(Path src, Path dest) { + checkNotNull(src, "src"); + checkNotNull(dest, "dest"); + checkArgument(Files.exists(src, LinkOption.NOFOLLOW_LINKS), + "Source file " + src + " does not exist"); + } + } } diff --git a/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java b/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java index e9b43d3..4c87e8a 100644 --- a/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java +++ b/src/main/java/gov/nasa/ziggy/worker/PipelineWorker.java @@ -1,5 +1,5 @@ /* - * Copyright (C) 2022-2023 United States Government as represented by the Administrator of the + * Copyright (C) 2022-2024 United States Government as represented by the Administrator of the * National Aeronautics and Space Administration. All Rights Reserved. * * NASA acknowledges the SETI Institute's primary role in authoring and producing Ziggy, a Pipeline @@ -45,18 +45,15 @@ import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; -import gov.nasa.ziggy.services.config.PropertyName; -import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; import gov.nasa.ziggy.services.logging.TaskLog; import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.KillTasksRequest; import gov.nasa.ziggy.services.messages.KilledTaskMessage; import gov.nasa.ziggy.services.messages.ShutdownMessage; -import gov.nasa.ziggy.services.messaging.ProcessHeartbeatManager; +import gov.nasa.ziggy.services.messaging.HeartbeatManager; import gov.nasa.ziggy.services.messaging.ZiggyMessenger; import gov.nasa.ziggy.services.messaging.ZiggyRmiClient; -import gov.nasa.ziggy.services.messaging.ZiggyRmiServer; import gov.nasa.ziggy.services.process.AbstractPipelineProcess; import gov.nasa.ziggy.supervisor.PipelineSupervisor; import gov.nasa.ziggy.supervisor.TaskRequestHandler; @@ -145,15 +142,11 @@ private void processTask(long taskId, RunMode runMode) { // Initialize the ProcessHeartbeatManager for this process. log.info("Initializing ProcessHeartbeatManager..."); - ProcessHeartbeatManager - .initializeInstance(new ProcessHeartbeatManager.WorkerHeartbeatManagerAssistant()); + HeartbeatManager.startInstance(); log.info("Initializing ProcessHeartbeatManager...done"); // Initialize the UiCommunicator for this process. - int rmiPort = ZiggyConfiguration.getInstance() - .getInt(PropertyName.SUPERVISOR_PORT.property(), ZiggyRmiServer.RMI_PORT_DEFAULT); - log.info("Starting ZiggyRmiClient instance with registry on port " + rmiPort + "..."); - ZiggyRmiClient.initializeInstance(rmiPort, NAME); + ZiggyRmiClient.start(NAME); ZiggyShutdownHook.addShutdownHook(() -> { // Note that we need to wait for the final status message to get sent @@ -167,7 +160,6 @@ private void processTask(long taskId, RunMode runMode) { } ZiggyRmiClient.reset(); }); - log.info("Starting ZiggyRmiClient instance ... done"); // Subscribe to messages as needed. subscribe(); diff --git a/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java b/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java index e063ce5..4636fe9 100644 --- a/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java +++ b/src/main/java/gov/nasa/ziggy/worker/TaskExecutor.java @@ -239,9 +239,9 @@ private boolean processTaskInternal() throws Exception { PipelineTask task = new PipelineTaskCrud().retrieve(taskId); Hibernate.initialize(task.getPipelineInstance().getPipelineParameterSets()); Hibernate.initialize(task.getPipelineInstanceNode().getModuleParameterSets()); - Hibernate.initialize(task.getPipelineDefinitionNode().getInputDataFileTypes()); - Hibernate.initialize(task.getPipelineDefinitionNode().getOutputDataFileTypes()); - Hibernate.initialize(task.getPipelineDefinitionNode().getModelTypes()); + Hibernate.initialize(task.pipelineDefinitionNode().getInputDataFileTypes()); + Hibernate.initialize(task.pipelineDefinitionNode().getOutputDataFileTypes()); + Hibernate.initialize(task.pipelineDefinitionNode().getModelTypes()); return task; }); @@ -277,14 +277,7 @@ private void pipelineModuleProcessTask(PipelineModule currentPipelineModule, try { // Hand off control to the PipelineModule implementation - if (currentPipelineModule.processTaskRequiresDatabaseTransaction()) { - performTransaction(() -> { - setTaskDone(currentPipelineModule.processTask()); - return null; - }); - } else { - setTaskDone(currentPipelineModule.processTask()); - } + setTaskDone(currentPipelineModule.processTask()); } finally { IntervalMetric.stop(moduleExecMetricPrefix + ".processTask", key); taskContext.setModuleExecTime(System.currentTimeMillis() - startTime); diff --git a/src/main/java/gov/nasa/ziggy/worker/WorkerResources.java b/src/main/java/gov/nasa/ziggy/worker/WorkerResources.java new file mode 100644 index 0000000..5a9b8f2 --- /dev/null +++ b/src/main/java/gov/nasa/ziggy/worker/WorkerResources.java @@ -0,0 +1,52 @@ +package gov.nasa.ziggy.worker; + +import java.io.Serializable; + +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.services.messages.WorkerResourcesMessage; +import gov.nasa.ziggy.supervisor.PipelineSupervisor; +import gov.nasa.ziggy.util.HumanReadableHeapSize; + +/** + * Represents a set of worker resources, specifically the max worker count and Java heap size. + *

      + * Note that any particular instance of {@link WorkerResources} can be one of the following: + *

        + *
      1. The configured resources for a particular {@link PipelineDefinitionNode} instance, in which + * case one or both of the values can be null, indicating that the default values should be used. + *
      2. The default values, which were set when the {@link PipelineSupervisor} was instantiated. + *
      3. A composite of the above, in which null values from the node's resources are replaced by the + * corresponding values from the default resources. This defines the current resources available to + * the node when defaults are taken into account. + *
      + * Users should be careful that they know exactly which of these three cases is represented by any + * particular instance. + *

      + * Note that the {@link WorkerResourcesMessage} is incapable of transporting a non-default instance + * of {@link WorkerResources} if the value of any resource is null. + * + * @author PT + */ +public class WorkerResources implements Serializable { + + private static final long serialVersionUID = 20231204L; + private final Integer maxWorkerCount; + private final Integer heapSizeMb; + + public WorkerResources(Integer maxWorkerCount, Integer heapSizeMb) { + this.maxWorkerCount = maxWorkerCount; + this.heapSizeMb = heapSizeMb; + } + + public Integer getMaxWorkerCount() { + return maxWorkerCount; + } + + public Integer getHeapSizeMb() { + return heapSizeMb; + } + + public HumanReadableHeapSize humanReadableHeapSize() { + return new HumanReadableHeapSize(getHeapSizeMb()); + } +} diff --git a/src/main/matlab/initialize_pipeline_configuration.m b/src/main/matlab/initialize_pipeline_configuration.m index 03e2c9b..372aa10 100644 --- a/src/main/matlab/initialize_pipeline_configuration.m +++ b/src/main/matlab/initialize_pipeline_configuration.m @@ -38,7 +38,7 @@ function initialize_pipeline_configuration( csciNamesToSkip ) % "main", "src" nPathSteps = length(thisFile) - 4 ; - buildDir = fullfile(thisFile{1:nPathSteps}); + buildDir = fullfile(thisFile{1:nPathSteps}, 'build'); thisLocation = fullfile(thisFile{1:length(thisFile)-1}) ; % add this location to the path @@ -87,11 +87,9 @@ function initialize_pipeline_configuration( csciNamesToSkip ) disp(['Setting log4j2 configuration file to ' log4jConfigFile]); java.lang.System.setProperty('log4j2.configurationFile', log4jConfigFile); log4jDestination = fullfile(pipelineProperties.get_property( ... - 'ziggy.pipeline.home.dir'), 'logs') ; - disp(['Setting log4j destination to: ',log4jDestination]) ; - % trailing slash needed - java.lang.System.setProperty('log4j.logfile.prefix', ... - [log4jDestination,'/']) ; + 'ziggy.pipeline.home.dir'), 'logs', 'matlab.log'); + disp(['Setting log4j log file to: ', log4jDestination]); + java.lang.System.setProperty('ziggy.logFile', log4jDestination); else disp('No log4j config file found') ; end diff --git a/src/main/perl/ziggy.pl b/src/main/perl/ziggy.pl index bdc44c6..ffbbdaa 100755 --- a/src/main/perl/ziggy.pl +++ b/src/main/perl/ziggy.pl @@ -108,12 +108,22 @@ sub main { # If the user-specified name is one of the pre-defined entries in the # nicknames, use information that that nickname maps to. If not, the user # will need to specify everything. + + # Convert program options to a string, preserving empty options (""). + my $programOptionsString = ""; + foreach (@$programOptions) { + if ($programOptionsString ne "") { + $programOptionsString .= " "; + } + $programOptionsString .= /^$/ ? '""' : $_; + } + if (defined($nickname)) { if (exists $nicknames{$nickname}) { my $substJvm = makeSubstitutions("$cmd $nicknames{$nickname}{jvmOptions} @$jvmOptions", %properties); $cmd = "$substJvm " . "$nicknames{$nickname}{className} " . - "$nicknames{$nickname}{programOptions} @$programOptions"; + "$nicknames{$nickname}{programOptions} $programOptionsString"; } else { print "Nickname $nickname unknown\n\n"; displayNicknames(%nicknames); @@ -121,7 +131,7 @@ sub main { } } elsif (defined($className)) { my $substJvm = makeSubstitutions("$cmd @$jvmOptions", %properties); - $cmd = "$substJvm $className @$programOptions"; + $cmd = "$substJvm $className $programOptionsString"; } else { print "Neither nickname nor class name provided"; return 1; @@ -245,7 +255,8 @@ sub makeSubstitutions { # Substitute property references. if ($s =~ /\$\{([^}]+)}/) { my $key = $1; - die "Missing Ziggy property '$key'" if (!exists($properties{$key})); + exists($properties{$key}) + or die "Missing Ziggy property '$key'"; $s =~ s/\$\{$key}/$properties{$key}/; $substitutionMade = 1; } @@ -278,7 +289,7 @@ sub makeSubstitutions { sub getNicknames { my (%properties) = @_; my %nicknames = (); - my $default_jvm_options = $properties{"ziggy.default.jvm.args"}; + my $default_jvm_options = exists($properties{'ziggy.default.jvm.args'}) ? $properties{'ziggy.default.jvm.args'} . " " : ""; foreach my $property (keys %properties) { next if ($property !~ /^ziggy\.nickname\./); @@ -292,7 +303,7 @@ sub getNicknames { my $nickname = $property =~ s/^ziggy\.nickname\.//r; $nicknames{$nickname}{className} = $fields[0]; $nicknames{$nickname}{logFile} = $fields[1]; - $nicknames{$nickname}{jvmOptions} = $default_jvm_options . " " . logFileOption($fields[1]) . " " . $fields[2]; + $nicknames{$nickname}{jvmOptions} = $default_jvm_options . logFileOption($fields[1]) . " " . $fields[2]; $nicknames{$nickname}{programOptions} = $fields[3]; } @@ -303,9 +314,12 @@ sub logFileOption { my ($logFileBasename) = @_; my ($logFileName, $logFileOption); + exists($properties{'ziggy.pipeline.results.dir'}) + or die "Missing Ziggy property ziggy.pipeline.results.dir"; + $logFileBasename = "ziggy" if $logFileBasename eq ""; $logFileName = File::Spec->catfile($properties{'ziggy.pipeline.results.dir'}, 'logs', 'cli', $logFileBasename . '.log'); - $logFileOption = "-Dziggy.logfile=" . $logFileName; + $logFileOption = "-Dziggy.logFile=" . $logFileName; return $logFileOption; } diff --git a/src/main/python/hdf5mi/hdf5.py b/src/main/python/hdf5mi/hdf5.py index 02f6131..29c7ecd 100644 --- a/src/main/python/hdf5mi/hdf5.py +++ b/src/main/python/hdf5mi/hdf5.py @@ -198,6 +198,8 @@ def _read_group(self, group): elif len(k) == 1: if isinstance(group[k[0]], h5py.Dataset): return_value = self._read_dataset(group) + else: + return_value = self._read_groups(group) else: return_value = "" return return_value diff --git a/src/test/java/gov/nasa/ziggy/ZiggyPropertyRule.java b/src/test/java/gov/nasa/ziggy/ZiggyPropertyRule.java index 281c235..b32d6b7 100644 --- a/src/test/java/gov/nasa/ziggy/ZiggyPropertyRule.java +++ b/src/test/java/gov/nasa/ziggy/ZiggyPropertyRule.java @@ -4,7 +4,7 @@ import java.nio.file.Path; -import org.apache.commons.configuration2.Configuration; +import org.apache.commons.configuration2.CompositeConfiguration; import org.junit.rules.ExternalResource; import org.junit.rules.TestRule; @@ -43,9 +43,10 @@ * public final RuleChain ruleChain = RuleChain.outerRule(directoryRule).around(fooPropertyRule); * * - * For convenience, this rule provides a {@link #getProperty} method to access the current property - * value and a {@link #getPreviousProperty} method to access the value of the property before the - * test started. + * This rule provides a {@link #getValue} method to access the current property value and a + * {@link #setValue} method to set it. The latter can be used to override the rule's initial value + * or a system property. Ziggy code should not call either {@link System#getProperty()} or + * {@link System#setProperty()}. *

      * This class is thread-safe as it uses thread-safe configuration objects. * @@ -57,7 +58,6 @@ public class ZiggyPropertyRule extends ExternalResource { private String value; private ZiggyDirectoryRule directoryRule; private String subdirectory; - private String previousValue; /** * Creates a {@code ZiggyPropertyRule} with the given property and value. See class @@ -158,42 +158,37 @@ protected void before() throws Throwable { value = directory.toString(); } - // the TEST_ENVIRONMENT property requires special handling because it needs - // to keep its value across ZiggyConfiguration resets. To accomplish that, - // it is placed into the system properties. - if (property.equals(PropertyName.TEST_ENVIRONMENT.property())) { - System.setProperty(PropertyName.TEST_ENVIRONMENT.property(), "true"); - return; - } + CompositeConfiguration seedConfig = ZiggyConfiguration.getMutableInstance(); - Configuration configuration = ZiggyConfiguration.getMutableInstance(); - previousValue = configuration.getString(property, null); - if (value != null) { - configuration.setProperty(property, value); - } else { - configuration.clearProperty(property); + if (seedConfig == null) { + seedConfig = new CompositeConfiguration(); + ZiggyConfiguration.setMutableInstance(seedConfig); } + setValue(value); } @Override protected void after() { ZiggyConfiguration.reset(); - if (property.equals(PropertyName.TEST_ENVIRONMENT.property())) { - System.clearProperty(PropertyName.TEST_ENVIRONMENT.property()); - } } /** * Returns the value of the property set by this rule. */ - public String getProperty() { + public String getValue() { return value; } /** - * Returns the previous value of the property before it was set by this rule. + * Updates the initial value of the property. This needs to be called at the beginning of the + * test method in order to affect future {@link ZiggyConfiguration#getInstance()} calls. */ - public String getPreviousProperty() { - return previousValue; + public void setValue(String value) { + this.value = value; + if (value != null) { + ZiggyConfiguration.getMutableInstance().setProperty(property, value); + } else { + ZiggyConfiguration.getMutableInstance().clearProperty(property); + } } } diff --git a/src/test/java/gov/nasa/ziggy/ZiggyPropertyRuleTest.java b/src/test/java/gov/nasa/ziggy/ZiggyPropertyRuleTest.java index 6788ba0..69803d9 100644 --- a/src/test/java/gov/nasa/ziggy/ZiggyPropertyRuleTest.java +++ b/src/test/java/gov/nasa/ziggy/ZiggyPropertyRuleTest.java @@ -27,8 +27,8 @@ public class ZiggyPropertyRuleTest { @Test public void stringConstructorTest() { - assertEquals("value", stringPropertyRule.getProperty()); + assertEquals("value", stringPropertyRule.getValue()); assertEquals("build/test/ZiggyPropertyRuleTest/stringConstructorTest", - stringDirectoryPropertyRule.getProperty()); + stringDirectoryPropertyRule.getValue()); } } diff --git a/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java b/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java index 8ccb1a2..c05dadd 100644 --- a/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java +++ b/src/test/java/gov/nasa/ziggy/ZiggyUnitTestUtils.java @@ -12,7 +12,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.security.User; /** * General utilities for unit and integration tests. @@ -62,7 +61,6 @@ public static void initializePipelineInstanceNode(PipelineInstanceNode node) { // Initialization for database items that define the pipelines: pipeline definitions, // pipeline module definitions, pipeline definition nodes public static void initializePipelineDefinition(PipelineDefinition pipelineDefinition) { - initializeUser(pipelineDefinition.getAuditInfo().getLastChangedUser()); Hibernate.initialize(pipelineDefinition.getRootNodes()); Hibernate.initialize(pipelineDefinition.getPipelineParameterSetNames()); initializePipelineDefinitionNodes(pipelineDefinition.getRootNodes()); @@ -87,12 +85,5 @@ public static void initializePipelineDefinitionNode(PipelineDefinitionNode node) public static void initializePipelineModuleDefinition( PipelineModuleDefinition moduleDefinition) { - initializeUser(moduleDefinition.getAuditInfo().getLastChangedUser()); - } - - // Utility initialization of a User instance - public static void initializeUser(User user) { - Hibernate.initialize(user.getRoles()); - Hibernate.initialize(user.getPrivileges()); } } diff --git a/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java b/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java index dbc3c32..9694d57 100644 --- a/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java +++ b/src/test/java/gov/nasa/ziggy/crud/ZiggyQueryTest.java @@ -38,7 +38,7 @@ import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.util.StringUtils; +import gov.nasa.ziggy.util.ZiggyStringUtils; import gov.nasa.ziggy.util.io.FileUtil; /** @@ -78,7 +78,7 @@ public class ZiggyQueryTest { public ZiggyPropertyRule log4jConfigProperty = new ZiggyPropertyRule( PropertyName.LOG4J2_CONFIGURATION_FILE, Paths.get("etc").resolve("log4j2.xml").toString()); - public ZiggyPropertyRule log4jLogFileProperty = new ZiggyPropertyRule("ziggy.logfile", + public ZiggyPropertyRule log4jLogFileProperty = new ZiggyPropertyRule("ziggy.logFile", directoryRule, HIBERNATE_LOG_FILE_NAME); @Rule @@ -146,9 +146,11 @@ public void sqlRetrieveTest() throws IOException { List fieldNames = new ArrayList<>(); Set lazyFieldNames = Set.of("log", "uowTaskParameters", "summaryMetrics", "execLog", "producerTaskIds", "remoteJobs"); + Set transientFieldNames = Set.of("maxFailedSubtaskCount", "maxAutoResubmits"); for (Field field : fields) { - if (!lazyFieldNames.contains(field.getName())) { + if (!lazyFieldNames.contains(field.getName()) + && !transientFieldNames.contains(field.getName())) { fieldNames.add(field.getName()); } } @@ -592,7 +594,7 @@ public void testCount() { } private List logFileContents() throws IOException { - return StringUtils + return ZiggyStringUtils .breakStringAtLineTerminations(Files.readString(logPath, FileUtil.ZIGGY_CHARSET)); } diff --git a/src/test/java/gov/nasa/ziggy/data/management/DatastoreConfigurationFileTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationFileTest.java similarity index 79% rename from src/test/java/gov/nasa/ziggy/data/management/DatastoreConfigurationFileTest.java rename to src/test/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationFileTest.java index 1f90be9..3bf09ec 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DatastoreConfigurationFileTest.java +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationFileTest.java @@ -1,4 +1,4 @@ -package gov.nasa.ziggy.data.management; +package gov.nasa.ziggy.data.datastore; import static gov.nasa.ziggy.XmlUtils.assertContains; import static gov.nasa.ziggy.XmlUtils.complexTypeContent; @@ -10,7 +10,6 @@ import java.io.IOException; import java.nio.file.Files; import java.util.List; -import java.util.Set; import javax.xml.transform.Result; import javax.xml.transform.stream.StreamResult; @@ -68,9 +67,11 @@ public void testGenerateSchema() throws JAXBException, IOException { assertContains(complexTypeContent, ""); assertContains(complexTypeContent, - ""); + ""); assertContains(complexTypeContent, - ""); + ""); + assertContains(complexTypeContent, + ""); complexTypeContent = complexTypeContent(schemaContent, ""); @@ -86,13 +87,20 @@ public void testGenerateSchema() throws JAXBException, IOException { ""); complexTypeContent = complexTypeContent(schemaContent, - ""); + ""); assertContains(complexTypeContent, ""); assertContains(complexTypeContent, - ""); + ""); + + complexTypeContent = complexTypeContent(schemaContent, + ""); + assertContains(complexTypeContent, + ""); assertContains(complexTypeContent, - ""); + ""); + assertContains(complexTypeContent, ""); + assertContains(complexTypeContent, ""); } @Test @@ -101,16 +109,8 @@ public void testUnmarshaller() throws JAXBException { Unmarshaller unmarshaller = context.createUnmarshaller(); DatastoreConfigurationFile datastoreConfigurationFile = (DatastoreConfigurationFile) unmarshaller .unmarshal(xmlUnmarshalingFile); - assertEquals(5, datastoreConfigurationFile.getDataFileTypes().size()); + assertEquals(2, datastoreConfigurationFile.getDataFileTypes().size()); assertEquals(2, datastoreConfigurationFile.getModelTypes().size()); - - Set dataFileTypes = datastoreConfigurationFile.getDataFileTypes(); - for (DataFileType dataFileType : dataFileTypes) { - if (dataFileType.getName().equals("has backslashes")) { - assertEquals("(\\S+)-(set-[0-9])-(file-[0-9]).png", - dataFileType.getFileNameRegexForTaskDir()); - } - } } private class DatastoreFileSchemaResolver extends SchemaOutputResolver { diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporterTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporterTest.java new file mode 100644 index 0000000..4b4c4f4 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreConfigurationImporterTest.java @@ -0,0 +1,337 @@ +package gov.nasa.ziggy.data.datastore; + +import static gov.nasa.ziggy.ZiggyUnitTestUtils.TEST_DATA; +import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.nio.file.Path; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.hibernate.Hibernate; +import org.junit.Rule; +import org.junit.Test; +import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; + +import com.google.common.collect.ImmutableList; + +import gov.nasa.ziggy.ZiggyDatabaseRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.crud.DataFileTypeCrud; +import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import jakarta.xml.bind.JAXBException; + +/** + * Unit test class for {@link DatastoreConfigurationImporter}. + * + * @author PT + */ +public class DatastoreConfigurationImporterTest { + + private static final Path DATASTORE = TEST_DATA.resolve("datastore"); + private static final String FILE_1 = DATASTORE.resolve("pd-test-1.xml").toString(); + private static final String FILE_2 = DATASTORE.resolve("pd-test-2.xml").toString(); + private static final String NO_SUCH_FILE = "no-such-file.xml"; + private static final String NOT_REGULAR_FILE = TEST_DATA.resolve("configuration").toString(); + private static final String INVALID_FILE_1 = DATASTORE.resolve("pd-test-invalid-type.xml") + .toString(); + private static final String INVALID_FILE_2 = DATASTORE.resolve("pd-test-invalid-xml") + .toString(); + private static final String UPDATE_FILE = DATASTORE.resolve("datastore-update.xml").toString(); + + private DataFileTypeCrud dataFileTypeCrud = Mockito.spy(DataFileTypeCrud.class); + private ModelCrud modelCrud = Mockito.spy(ModelCrud.class); + private DatastoreNodeCrud nodeCrud = Mockito.spy(DatastoreNodeCrud.class); + private DatastoreRegexpCrud regexpCrud = Mockito.spy(DatastoreRegexpCrud.class); + + @Rule + public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); + + @Rule + public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule(ZIGGY_HOME_DIR, + DirectoryProperties.ziggyCodeBuildDir().toString()); + + // Basic functionality -- multiple files, multiple definitions, get imported + @SuppressWarnings("unchecked") + @Test + public void testBasicImport() throws JAXBException { + + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, FILE_2), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + DatabaseTransactionFactory.performTransaction(() -> { + importerSpy.importConfiguration(); + return null; + }); + + Set nodesForDatabase = importerSpy.nodesForDatabase(); + assertEquals(10, nodesForDatabase.size()); + + List regexps = importerSpy.getRegexps(); + assertEquals(2, regexps.size()); + + assertEquals(3, importerSpy.getDataFileTypes().size()); + Mockito.verify(dataFileTypeCrud, Mockito.times(1)) + .persist(ArgumentMatchers. anyList()); + + assertEquals(2, importerSpy.getModelTypes().size()); + Mockito.verify(modelCrud, Mockito.times(1)).persist(ArgumentMatchers. anyList()); + + Map databaseRegexps = (Map) DatabaseTransactionFactory + .performTransaction(() -> regexpCrud.retrieveRegexpsByName()); + assertEquals(2, databaseRegexps.size()); + DatastoreRegexp regexp = databaseRegexps.get("cadenceType"); + assertNotNull(regexp); + assertEquals("(target|ffi)", regexp.getValue()); + regexp = databaseRegexps.get("sector"); + assertNotNull(regexp); + assertEquals("(sector-[0-9]{4})", regexp.getValue()); + + Map datastoreNodes = (Map) DatabaseTransactionFactory + .performTransaction(() -> { + Map nodes = nodeCrud.retrieveNodesByFullPath(); + for (DatastoreNode node : nodes.values()) { + Hibernate.initialize(node.getChildNodeFullPaths()); + } + return nodes; + }); + DatastoreNode sectorNode = testNode(datastoreNodes, "sector", true, 1, null); + DatastoreNode mdaNode = testNode(datastoreNodes, "mda", false, 2, sectorNode); + + // DR nodes + DatastoreNode drNode = testNode(datastoreNodes, "dr", false, 1, mdaNode); + DatastoreNode drPixelNode = testNode(datastoreNodes, "pixels", false, 1, drNode); + DatastoreNode drCadenceTypeNode = testNode(datastoreNodes, "cadenceType", true, 1, + drPixelNode); + testNode(datastoreNodes, "channel", false, 0, drCadenceTypeNode); + + // CAL nodes + DatastoreNode calNode = testNode(datastoreNodes, "cal", false, 1, mdaNode); + DatastoreNode calPixelNode = testNode(datastoreNodes, "pixels", false, 1, calNode); + DatastoreNode calCadenceTypeNode = testNode(datastoreNodes, "cadenceType", true, 1, + calPixelNode); + testNode(datastoreNodes, "channel", false, 0, calCadenceTypeNode); + + assertEquals(10, datastoreNodes.size()); + } + + private DatastoreNode testNode(Map datastoreNodes, String name, + boolean regexp, int expectedChildNodeCount, DatastoreNode parentNode) { + String parentFullPath = parentNode != null ? parentNode.getFullPath() : ""; + String fullPath = DatastoreConfigurationImporter.fullPathFromParentPath(name, + parentFullPath); + DatastoreNode node = datastoreNodes.get(fullPath); + assertNotNull(node); + assertEquals(name, node.getName()); + assertEquals(regexp, node.isRegexp()); + assertEquals(expectedChildNodeCount, node.getChildNodeFullPaths().size()); + if (parentNode != null) { + assertTrue(parentNode.getChildNodeFullPaths().contains(fullPath)); + } + return node; + } + + @Test + public void testUpdateDatastore() { + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, FILE_2), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + + dataFileImporter = new DatastoreConfigurationImporter(List.of(UPDATE_FILE), false); + DatastoreConfigurationImporter updaterSpy = Mockito.spy(dataFileImporter); + setMocks(updaterSpy); + updaterSpy.importConfiguration(); + + @SuppressWarnings("unchecked") + Map regexpsByName = (Map) DatabaseTransactionFactory + .performTransaction(() -> regexpCrud.retrieveRegexpsByName()); + @SuppressWarnings("unchecked") + Map nodesByFullPath = (Map) DatabaseTransactionFactory + .performTransaction(() -> { + Map nodes = nodeCrud.retrieveNodesByFullPath(); + for (DatastoreNode node : nodes.values()) { + Hibernate.initialize(node.getChildNodeFullPaths()); + } + return nodes; + }); + + assertNotNull(regexpsByName.get("sector")); + assertEquals("(sector-[0-9]{4})", regexpsByName.get("sector").getValue()); + assertNotNull(regexpsByName.get("cadenceType")); + assertEquals("(target|ffi|fast-target)", regexpsByName.get("cadenceType").getValue()); + assertEquals(2, regexpsByName.size()); + + DatastoreNode sectorNode = testNode(nodesByFullPath, "sector", true, 1, null); + DatastoreNode mdaNode = testNode(nodesByFullPath, "mda", false, 2, sectorNode); + + // DR nodes. + DatastoreNode drNode = testNode(nodesByFullPath, "dr", false, 1, mdaNode); + DatastoreNode drPixelNode = testNode(nodesByFullPath, "pixels", false, 1, drNode); + DatastoreNode drCadenceTypeNode = testNode(nodesByFullPath, "cadenceType", true, 1, + drPixelNode); + testNode(nodesByFullPath, "ccd", false, 0, drCadenceTypeNode); + + // PA nodes. + DatastoreNode paNode = testNode(nodesByFullPath, "pa", false, 1, mdaNode); + DatastoreNode paFluxNode = testNode(nodesByFullPath, "raw-flux", false, 1, paNode); + DatastoreNode paCadenceTypeNode = testNode(nodesByFullPath, "cadenceType", true, 1, + paFluxNode); + testNode(nodesByFullPath, "ccd", false, 0, paCadenceTypeNode); + + // Deleted nodes. + assertNull(nodesByFullPath.get("sector/mda/dr/pixels/cadenceType/channel")); + assertNull(nodesByFullPath.get("sector/mda/cal")); + assertNull(nodesByFullPath.get("sector/mda/cal/pixels")); + assertNull(nodesByFullPath.get("sector/mda/cal/pixels/cadenceType")); + assertNull(nodesByFullPath.get("sector/mda/cal/pixels/cadenceType/channel")); + + assertEquals(10, nodesByFullPath.size()); + } + + // Dry run test -- should import but not persist + @Test + public void testDryRun() throws JAXBException { + + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, FILE_2), true); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + + assertEquals(3, importerSpy.getDataFileTypes().size()); + Mockito.verify(dataFileTypeCrud, Mockito.times(0)) + .persist(ArgumentMatchers. anyList()); + assertEquals(2, importerSpy.getModelTypes().size()); + Mockito.verify(modelCrud, Mockito.times(0)).persist(ArgumentMatchers. anyList()); + } + + @Test + public void testDryRunOfUpdate() { + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, FILE_2), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + + dataFileImporter = new DatastoreConfigurationImporter(List.of(UPDATE_FILE), true); + DatastoreConfigurationImporter updaterSpy = Mockito.spy(dataFileImporter); + setMocks(updaterSpy); + updaterSpy.importConfiguration(); + + @SuppressWarnings("unchecked") + Map regexpsByName = (Map) DatabaseTransactionFactory + .performTransaction(() -> regexpCrud.retrieveRegexpsByName()); + @SuppressWarnings("unchecked") + Map nodesByFullPath = (Map) DatabaseTransactionFactory + .performTransaction(() -> { + Map nodes = new DatastoreNodeCrud() + .retrieveNodesByFullPath(); + for (DatastoreNode node : nodes.values()) { + Hibernate.initialize(node.getChildNodeFullPaths()); + } + return nodes; + }); + + assertEquals(2, regexpsByName.size()); + DatastoreRegexp regexp = regexpsByName.get("cadenceType"); + assertNotNull(regexp); + assertEquals("(target|ffi)", regexp.getValue()); + regexp = regexpsByName.get("sector"); + assertNotNull(regexp); + assertEquals("(sector-[0-9]{4})", regexp.getValue()); + + DatastoreNode sectorNode = testNode(nodesByFullPath, "sector", true, 1, null); + DatastoreNode mdaNode = testNode(nodesByFullPath, "mda", false, 2, sectorNode); + + // DR nodes + DatastoreNode drNode = testNode(nodesByFullPath, "dr", false, 1, mdaNode); + DatastoreNode drPixelNode = testNode(nodesByFullPath, "pixels", false, 1, drNode); + DatastoreNode drCadenceTypeNode = testNode(nodesByFullPath, "cadenceType", true, 1, + drPixelNode); + testNode(nodesByFullPath, "channel", false, 0, drCadenceTypeNode); + + // CAL nodes + DatastoreNode calNode = testNode(nodesByFullPath, "cal", false, 1, mdaNode); + DatastoreNode calPixelNode = testNode(nodesByFullPath, "pixels", false, 1, calNode); + DatastoreNode calCadenceTypeNode = testNode(nodesByFullPath, "cadenceType", true, 1, + calPixelNode); + testNode(nodesByFullPath, "channel", false, 0, calCadenceTypeNode); + + assertEquals(10, nodesByFullPath.size()); + } + + // Test with missing and non-regular files -- should still import from the present, + // regular files + @Test + public void testWithInvalidFiles() throws JAXBException { + + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, FILE_2, NO_SUCH_FILE, NOT_REGULAR_FILE), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + + assertEquals(3, importerSpy.getDataFileTypes().size()); + Mockito.verify(dataFileTypeCrud, Mockito.times(1)) + .persist(ArgumentMatchers. anyList()); + } + + // Test with a file that has an entry that is valid XML but instantiates to an + // invalid DataFileType instance + @Test + public void testWithInvalidDataFileType() throws JAXBException { + + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, INVALID_FILE_1), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + + assertEquals(2, importerSpy.getDataFileTypes().size()); + Mockito.verify(dataFileTypeCrud, Mockito.times(1)) + .persist(ArgumentMatchers. anyList()); + } + + // Test with a file that has an entry that is invalid XML + @Test + public void testWithInvalidDataXml() throws JAXBException { + + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, INVALID_FILE_2), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + + assertEquals(2, importerSpy.getDataFileTypes().size()); + Mockito.verify(dataFileTypeCrud, Mockito.times(1)) + .persist(ArgumentMatchers. anyList()); + } + + @Test(expected = IllegalStateException.class) + public void testDuplicateNames() throws JAXBException { + + DatastoreConfigurationImporter dataFileImporter = new DatastoreConfigurationImporter( + ImmutableList.of(FILE_1, FILE_1), false); + DatastoreConfigurationImporter importerSpy = Mockito.spy(dataFileImporter); + setMocks(importerSpy); + importerSpy.importConfiguration(); + } + + private void setMocks(DatastoreConfigurationImporter dataFileImporter) { + Mockito.when(dataFileImporter.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); + Mockito.when(dataFileImporter.modelCrud()).thenReturn(modelCrud); + Mockito.when(dataFileImporter.datastoreRegexpCrud()).thenReturn(regexpCrud); + Mockito.when(dataFileImporter.datastoreNodeCrud()).thenReturn(nodeCrud); + } +} diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java new file mode 100644 index 0000000..62314c3 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreFileManagerTest.java @@ -0,0 +1,677 @@ +package gov.nasa.ziggy.data.datastore; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.RuleChain; +import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; + +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager.InputFiles; +import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; +import gov.nasa.ziggy.module.AlgorithmStateFiles; +import gov.nasa.ziggy.module.SubtaskUtils; +import gov.nasa.ziggy.pipeline.PipelineExecutor; +import gov.nasa.ziggy.pipeline.definition.ModelMetadata; +import gov.nasa.ziggy.pipeline.definition.ModelRegistry; +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionProcessingOptions.ProcessingMode; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.uow.UnitOfWork; +import gov.nasa.ziggy.util.io.FileUtil; + +/** + * Unit tests for {@link DatastoreFileManager}. + * + * @author PT + */ +public class DatastoreFileManagerTest { + + private static final int SUBTASK_DIR_COUNT = 7; + public ZiggyDirectoryRule ziggyDirectoryRule = new ZiggyDirectoryRule(); + + public ZiggyPropertyRule datastoreRootProperty = new ZiggyPropertyRule( + PropertyName.DATASTORE_ROOT_DIR, ziggyDirectoryRule, "datastore"); + + public ZiggyPropertyRule taskDirRule = new ZiggyPropertyRule(PropertyName.RESULTS_DIR, + ziggyDirectoryRule, "pipeline-results"); + + @Rule + public final RuleChain testRuleChain = RuleChain.outerRule(ziggyDirectoryRule) + .around(datastoreRootProperty) + .around(taskDirRule); + + private DatastoreFileManager datastoreFileManager; + private PipelineTask pipelineTask; + private DataFileType uncalibratedSciencePixelDataFileType; + private DataFileType uncalibratedCollateralPixelDataFileType; + private DataFileType allFilesAllSubtasksDataFileType; + private DataFileType calibratedCollateralPixelDataFileType; + private Map regexpsByName; + private DatastoreWalker datastoreWalker; + private Path taskDirectory; + private PipelineDefinitionNode pipelineDefinitionNode; + private PipelineInstanceNode pipelineInstanceNode; + private PipelineInstance pipelineInstance; + private ModelRegistry modelRegistry; + private ModelMetadata modelMetadata; + private Map regexpValueByName = new HashMap<>(); + private PipelineDefinitionCrud pipelineDefinitionCrud; + private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud; + private PipelineTaskCrud pipelineTaskCrud; + private PipelineDefinition pipelineDefinition; + + @Before + public void setUp() throws IOException { + taskDirectory = DirectoryProperties.taskDataDir().toAbsolutePath(); + pipelineTask = Mockito.mock(PipelineTask.class); + datastoreFileManager = Mockito.spy(new DatastoreFileManager(pipelineTask, taskDirectory)); + Mockito.doReturn(Mockito.mock(AlertService.class)) + .when(datastoreFileManager) + .alertService(); + + // Create datastore directories. + DatastoreTestUtils.createDatastoreDirectories(); + + // Get defined DataFileTypes and add file name regular expressions. + // We use the "calibrated science pixels" to store files that we use as + // all-files-all-subtasks files for input, in the interest of not rewriting + // the entire DataFileUtils infrastructure. + Map dataFileTypes = DatastoreTestUtils.dataFileTypesByName(); + uncalibratedSciencePixelDataFileType = dataFileTypes + .get("uncalibrated science pixel values"); + uncalibratedSciencePixelDataFileType + .setFileNameRegexp("(uncalibrated-pixels-[0-9]+)\\.science\\.nc"); + uncalibratedCollateralPixelDataFileType = dataFileTypes + .get("uncalibrated collateral pixel values"); + uncalibratedCollateralPixelDataFileType + .setFileNameRegexp("(uncalibrated-pixels-[0-9]+)\\.collateral\\.nc"); + allFilesAllSubtasksDataFileType = dataFileTypes.get("calibrated science pixel values"); + allFilesAllSubtasksDataFileType.setFileNameRegexp("(everyone-needs-me-[0-9]+)\\.nc"); + allFilesAllSubtasksDataFileType.setIncludeAllFilesInAllSubtasks(true); + calibratedCollateralPixelDataFileType = dataFileTypes + .get("calibrated collateral pixel values"); + calibratedCollateralPixelDataFileType.setFileNameRegexp("(outputs-file-[0-9]+)\\.nc"); + + // Construct datastore files. + regexpsByName = DatastoreTestUtils.regexpsByName(); + datastoreWalker = new DatastoreWalker(regexpsByName, + DatastoreTestUtils.datastoreNodesByFullPath()); + Mockito.doReturn(datastoreWalker).when(datastoreFileManager).datastoreWalker(); + pipelineDefinitionCrud = Mockito.mock(PipelineDefinitionCrud.class); + Mockito.when(pipelineDefinitionCrud.retrieveProcessingMode(ArgumentMatchers.anyString())) + .thenReturn(ProcessingMode.PROCESS_ALL); + Mockito.doReturn(pipelineDefinitionCrud) + .when(datastoreFileManager) + .pipelineDefinitionCrud(); + datastoreProducerConsumerCrud = Mockito.mock(DatastoreProducerConsumerCrud.class); + Mockito.doReturn(datastoreProducerConsumerCrud) + .when(datastoreFileManager) + .datastoreProducerConsumerCrud(); + pipelineTaskCrud = Mockito.mock(PipelineTaskCrud.class); + Mockito.doReturn(pipelineTaskCrud).when(datastoreFileManager).pipelineTaskCrud(); + + // Construct the Map from regexp name to value. Note that we need to include the pixel type + // in the way that DatastoreWalker would include it. + regexpValueByName.put("sector", "sector-0002"); + regexpValueByName.put("cadenceType", "target"); + regexpValueByName.put("channel", "1:1:A"); + for (Map.Entry regexpEntry : regexpValueByName.entrySet()) { + regexpsByName.get(regexpEntry.getKey()).setInclude(regexpEntry.getValue()); + } + regexpValueByName.put("pixelType", "pixelType$science"); + + constructDatastoreFiles(uncalibratedSciencePixelDataFileType, SUBTASK_DIR_COUNT + 1, + "uncalibrated-pixels-", ".science.nc"); + constructDatastoreFiles(uncalibratedCollateralPixelDataFileType, SUBTASK_DIR_COUNT, + "uncalibrated-pixels-", ".collateral.nc"); + constructDatastoreFiles(allFilesAllSubtasksDataFileType, 2, "everyone-needs-me-", ".nc"); + + // Construct a model type and model metadata. + ModelType modelType = new ModelType(); + modelType.setType("test"); + modelMetadata = new ModelMetadata(); + modelMetadata.setModelType(modelType); + modelMetadata.setOriginalFileName("foo"); + modelMetadata.setDatastoreFileName("bar"); + Files.createDirectories(modelMetadata.datastoreModelPath().getParent()); + Files.createFile(modelMetadata.datastoreModelPath()); + + // Set up the pipeline task. + pipelineInstance = Mockito.mock(PipelineInstance.class); + pipelineInstanceNode = Mockito.mock(PipelineInstanceNode.class); + pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); + Mockito.when(pipelineInstanceNode.getPipelineDefinitionNode()) + .thenReturn(pipelineDefinitionNode); + Mockito.when(pipelineTask.getPipelineInstanceNode()).thenReturn(pipelineInstanceNode); + Mockito.when(pipelineTask.pipelineDefinitionNode()).thenReturn(pipelineDefinitionNode); + Mockito.when(pipelineTask.getPipelineInstance()).thenReturn(pipelineInstance); + Mockito.when(pipelineTask.getModuleName()).thenReturn("test module"); + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(uncalibratedSciencePixelDataFileType, + uncalibratedCollateralPixelDataFileType, allFilesAllSubtasksDataFileType)); + Mockito.when(pipelineDefinitionNode.getModelTypes()).thenReturn(Set.of(modelType)); + pipelineDefinition = Mockito.mock(PipelineDefinition.class); + Mockito.when(pipelineInstance.getPipelineDefinition()).thenReturn(pipelineDefinition); + Mockito.when(pipelineDefinition.getName()).thenReturn("test pipeline"); + + modelRegistry = Mockito.mock(ModelRegistry.class); + Mockito.when(pipelineInstance.getModelRegistry()).thenReturn(modelRegistry); + Mockito.when(modelRegistry.getModels()).thenReturn(Map.of(modelType, modelMetadata)); + + // Construct the UOW. + DatastoreDirectoryUnitOfWorkGenerator uowGenerator = Mockito + .spy(DatastoreDirectoryUnitOfWorkGenerator.class); + Mockito.doReturn(datastoreWalker).when(uowGenerator).datastoreWalker(); + List uows = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); + Mockito.doReturn(uows.get(0)).when(pipelineTask).uowTaskInstance(); + } + + /** Constructs a collection of zero-length files in the datastore. */ + private void constructDatastoreFiles(DataFileType dataFileType, int fileCount, + String filenamePrefix, String filenameSuffix) throws IOException { + Path datastorePath = datastoreWalker.pathFromLocationAndRegexpValues(regexpValueByName, + dataFileType.getLocation()); + for (int fileCounter = 0; fileCounter < fileCount; fileCounter++) { + String filename = filenamePrefix + fileCounter + filenameSuffix; + Files.createDirectories(datastorePath); + Files.createFile(datastorePath.resolve(filename)); + } + } + + /** Tests that the {@link DatastoreFileManager#filesForSubtasks()} method works as expected. */ + @Test + public void testFilesForSubtasks() { + Map> filesForSubtasks = datastoreFileManager.filesForSubtasks(); + Set subtaskBaseNames = filesForSubtasks.keySet(); + + // Check that the base names are as expected -- the uncalibrated-pixels-7 entry + // should not be present because it didn't have the right number of files. + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-0")); + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-1")); + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-2")); + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-3")); + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-4")); + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-5")); + assertTrue(subtaskBaseNames.contains("uncalibrated-pixels-6")); + assertEquals(SUBTASK_DIR_COUNT, filesForSubtasks.size()); + + // Check that every entry in the Map has the expected data files from the DR science and + // collateral pixels, plus the 2 files from the CAL science pixels. + for (Map.Entry> filesForSubtasksEntry : filesForSubtasks.entrySet()) { + String baseName = filesForSubtasksEntry.getKey(); + Set subtaskFiles = filesForSubtasksEntry.getValue(); + checkForFiles(baseName, subtaskFiles); + } + } + + /** Tests that all expected files are found in a Set of Path instances. */ + private void checkForFiles(String baseName, Set subtaskFiles) { + assertTrue(subtaskFiles.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("everyone-needs-me-0.nc"))); + assertTrue(subtaskFiles.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("everyone-needs-me-1.nc"))); + assertTrue(subtaskFiles.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve(baseName + ".science.nc"))); + assertTrue(subtaskFiles.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A") + .resolve(baseName + ".collateral.nc"))); + } + + /** Tests that filesForSubtasks acts as expected for a single-subtask use case. */ + @Test + public void testFilesForSubtasksSingleSubtask() { + Mockito.when(pipelineDefinitionNode.getSingleSubtask()).thenReturn(true); + Map> filesForSingleSubtask = datastoreFileManager.filesForSubtasks(); + assertNotNull(filesForSingleSubtask.get("Single Subtask")); + assertEquals(1, filesForSingleSubtask.size()); + Set files = filesForSingleSubtask.get("Single Subtask"); + for (int baseNameCount = 0; baseNameCount < SUBTASK_DIR_COUNT; baseNameCount++) { + String baseName = "uncalibrated-pixels-" + baseNameCount; + checkForFiles(baseName, files); + } + assertTrue(files.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("uncalibrated-pixels-7.science.nc"))); + assertEquals(17, files.size()); + } + + @Test + public void testSubtaskCount() { + assertEquals(7, datastoreFileManager.subtaskCount()); + } + + @Test + public void testModelFilesForTask() { + Map modelFilesForTask = datastoreFileManager.modelFilesForTask(); + assertNotNull(modelFilesForTask.get(modelMetadata.datastoreModelPath())); + assertEquals("foo", modelFilesForTask.get(modelMetadata.datastoreModelPath())); + assertEquals(1, modelFilesForTask.size()); + } + + @Test + public void testCopyDatastoreFilesToTaskDirectory() { + Map> filesForSubtasks = datastoreFileManager.filesForSubtasks(); + List> subtaskFiles = new ArrayList<>(filesForSubtasks.values()); + Map modelFilesForTask = datastoreFileManager.modelFilesForTask(); + Map> copiedFiles = datastoreFileManager + .copyDatastoreFilesToTaskDirectory(subtaskFiles, modelFilesForTask); + Set subtaskDirs = copiedFiles.keySet(); + + // We should wind up with 7 subtask directories. + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-0"))); + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-1"))); + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-2"))); + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-3"))); + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-4"))); + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-5"))); + assertTrue(subtaskDirs.contains(taskDirectory.resolve("st-6"))); + assertEquals(SUBTASK_DIR_COUNT, copiedFiles.size()); + + // Each subtask directory should have a file for each of the files in the + // corresponding Map value (note that the Map value is the Set of datastore + // file paths, so we have to generate the equivalent subtask directory file + // path and test for existence). + for (Map.Entry> copiedFilesEntry : copiedFiles.entrySet()) { + for (Path path : copiedFilesEntry.getValue()) { + assertTrue(Files.exists(copiedFilesEntry.getKey().resolve(path.getFileName()))); + } + + // Check that each subtask's collection of datastore files matches one of + // the ones that was produced by the copyDatastoreFilesToSubtaskDirectory method. + assertTrue(subtaskFiles.contains(copiedFilesEntry.getValue())); + } + + // Each subtask directory should have the test model in it, renamed to its original + // filename ("foo"). + for (Path subtaskDir : subtaskDirs) { + assertTrue(Files.exists(subtaskDir.resolve("foo"))); + } + } + + @Test + public void testCopyTaskDirectoryFilesToDatastore() throws IOException { + createOutputFiles(); + Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) + .thenReturn(Set.of(calibratedCollateralPixelDataFileType)); + Set copiedFiles = datastoreFileManager.copyTaskDirectoryFilesToDatastore(); + Path datastorePath = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A"); + for (int subtaskIndex = 0; subtaskIndex < SUBTASK_DIR_COUNT; subtaskIndex++) { + assertTrue(copiedFiles.contains( + datastorePath.toAbsolutePath().resolve("outputs-file-" + subtaskIndex + ".nc"))); + } + assertEquals(SUBTASK_DIR_COUNT, copiedFiles.size()); + } + + private void createOutputFiles() throws IOException { + for (int subtaskIndex = 0; subtaskIndex < SUBTASK_DIR_COUNT; subtaskIndex++) { + SubtaskUtils.createSubtaskDirectory(taskDirectory, subtaskIndex); + Path subtaskDir = taskDirectory.resolve(SubtaskUtils.subtaskDirName(subtaskIndex)); + Path outputsFile = subtaskDir.resolve("outputs-file-" + subtaskIndex + ".nc"); + Files.createFile(outputsFile); + } + } + + @Test + public void testInputFilesByOutputStatus() throws IOException { + createOutputFiles(); + createInputFiles(); + setAlgorithmStateFiles(); + Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) + .thenReturn(Set.of(calibratedCollateralPixelDataFileType)); + Files.delete(taskDirectory.resolve(SubtaskUtils.subtaskDirName(SUBTASK_DIR_COUNT - 1)) + .resolve("outputs-file-" + (SUBTASK_DIR_COUNT - 1) + ".nc")); + InputFiles inputFiles = datastoreFileManager.inputFilesByOutputStatus(); + Set strippedInputFilesWithOutputs = inputFiles.getFilesWithOutputs() + .stream() + .map(s -> DirectoryProperties.datastoreRootDir().toAbsolutePath().relativize(s)) + .collect(Collectors.toSet()); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-0.science.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-1.science.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-2.science.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-3.science.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-4.science.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-5.science.nc"))); + + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-0.collateral.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-1.collateral.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-2.collateral.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-3.collateral.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-4.collateral.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-5.collateral.nc"))); + + assertTrue(strippedInputFilesWithOutputs.contains( + Paths.get("sector-0002/mda/cal/pixels/target/science/1:1:A/everyone-needs-me-0.nc"))); + assertTrue(strippedInputFilesWithOutputs.contains( + Paths.get("sector-0002/mda/cal/pixels/target/science/1:1:A/everyone-needs-me-1.nc"))); + assertEquals(14, inputFiles.getFilesWithOutputs().size()); + + Set strippedInputFilesWithoutOutputs = inputFiles.getFilesWithoutOutputs() + .stream() + .map(s -> DirectoryProperties.datastoreRootDir().toAbsolutePath().relativize(s)) + .collect(Collectors.toSet()); + assertTrue(strippedInputFilesWithoutOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-6.science.nc"))); + assertTrue(strippedInputFilesWithoutOutputs.contains(Paths.get( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-6.collateral.nc"))); + assertEquals(2, inputFiles.getFilesWithoutOutputs().size()); + } + + private void createInputFiles() throws IOException { + for (int subtaskIndex = 0; subtaskIndex < SUBTASK_DIR_COUNT; subtaskIndex++) { + Path subtaskPath = SubtaskUtils.subtaskDirectory(taskDirectory, subtaskIndex); + Files.createFile( + subtaskPath.resolve("uncalibrated-pixels-" + subtaskIndex + ".science.nc")); + Files.createFile( + subtaskPath.resolve("uncalibrated-pixels-" + subtaskIndex + ".collateral.nc")); + Files.createFile(subtaskPath.resolve("everyone-needs-me-0.nc")); + Files.createFile(subtaskPath.resolve("everyone-needs-me-1.nc")); + } + } + + private void setAlgorithmStateFiles() { + for (int subtaskIndex = 0; subtaskIndex < SUBTASK_DIR_COUNT - 1; subtaskIndex++) { + AlgorithmStateFiles stateFile = new AlgorithmStateFiles( + SubtaskUtils.subtaskDirectory(taskDirectory, subtaskIndex).toFile()); + stateFile.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + stateFile.setOutputsFlag(); + } + new AlgorithmStateFiles( + SubtaskUtils.subtaskDirectory(taskDirectory, SUBTASK_DIR_COUNT - 1).toFile()) + .updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); + } + + @Test + public void testSingleSubtaskNoPerSubtaskFiles() { + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(allFilesAllSubtasksDataFileType)); + Mockito.doReturn(true).when(datastoreFileManager).singleSubtask(); + Map> filesForSubtasks = datastoreFileManager.filesForSubtasks(); + assertNotNull(filesForSubtasks.get(DatastoreFileManager.SINGLE_SUBTASK_BASE_NAME)); + Set paths = filesForSubtasks.get(DatastoreFileManager.SINGLE_SUBTASK_BASE_NAME); + assertTrue(paths.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("everyone-needs-me-0.nc"))); + assertTrue(paths.contains(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("everyone-needs-me-1.nc"))); + assertEquals(2, paths.size()); + assertEquals(1, filesForSubtasks.size()); + } + + @Test + public void testFilterOutFilesAlreadyProcessed() { + configureForFilteringTest(); + Map> filesForSubtasks = datastoreFileManager.filesForSubtasks(); + + // There should only be 5 Map entries, for base names uncalibrated-pixels-2 + // through uncalibrated-pixels-6. Both of the uncalibrated data files in + // uncalibrated-pixels-0 have been processed before. The collateral pixel file + // for uncalibrated-pixels-1 has been processed before. The collateral pixel + // file for uncalibrated-pixels-7 is missing. + assertNotNull(filesForSubtasks.get("uncalibrated-pixels-2")); + assertEquals(4, filesForSubtasks.get("uncalibrated-pixels-2").size()); + assertNotNull(filesForSubtasks.get("uncalibrated-pixels-3")); + assertEquals(4, filesForSubtasks.get("uncalibrated-pixels-3").size()); + assertNotNull(filesForSubtasks.get("uncalibrated-pixels-4")); + assertEquals(4, filesForSubtasks.get("uncalibrated-pixels-4").size()); + assertNotNull(filesForSubtasks.get("uncalibrated-pixels-5")); + assertEquals(4, filesForSubtasks.get("uncalibrated-pixels-5").size()); + assertNotNull(filesForSubtasks.get("uncalibrated-pixels-6")); + assertEquals(4, filesForSubtasks.get("uncalibrated-pixels-6").size()); + assertEquals(5, filesForSubtasks.size()); + } + + @Test + public void testFilteringForSingleSubtask() { + configureForFilteringTest(); + Mockito.doReturn(true).when(datastoreFileManager).singleSubtask(); + Map> filesForSubtasks = datastoreFileManager.filesForSubtasks(); + assertNotNull(filesForSubtasks.get(DatastoreFileManager.SINGLE_SUBTASK_BASE_NAME)); + Set paths = filesForSubtasks.get(DatastoreFileManager.SINGLE_SUBTASK_BASE_NAME); + assertEquals(17, paths.size()); + assertEquals(1, filesForSubtasks.size()); + } + + @Test + public void testFilteringNoPriorProcessingDetected() { + configureForFilteringTest(); + Mockito + .when( + pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode(pipelineDefinitionNode, null)) + .thenReturn(new ArrayList<>()); + Map> filesForSubtasks = datastoreFileManager.filesForSubtasks(); + assertEquals(7, filesForSubtasks.size()); + } + + private void configureForFilteringTest() { + + // Request processing of only new data. + Mockito.when(pipelineDefinitionCrud.retrieveProcessingMode(ArgumentMatchers.anyString())) + .thenReturn(ProcessingMode.PROCESS_NEW); + Set scienceDatastoreFilenames = producerConsumerTableFilenames("science"); + Set collateralDatastoreFilenames = producerConsumerTableFilenames("collateral"); + + // Set up the retrieval of earlier consumer task IDs from the database. + Mockito + .when( + pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode(pipelineDefinitionNode, null)) + .thenReturn(List.of(30L, 40L)) + .thenReturn(List.of(30L, 35L)); + + // Set up the DatastoreProducerConsumer retieval mocks. + Mockito + .when(datastoreProducerConsumerCrud.retrieveFilesConsumedByTasks(List.of(30L, 40L), + scienceDatastoreFilenames)) + .thenReturn(Set.of( + "sector-0002/mda/dr/pixels/target/science/1:1:A/uncalibrated-pixels-0.science.nc")); + Mockito + .when(datastoreProducerConsumerCrud.retrieveFilesConsumedByTasks(List.of(30L, 35L), + collateralDatastoreFilenames)) + .thenReturn(Set.of( + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-0.collateral.nc", + "sector-0002/mda/dr/pixels/target/collateral/1:1:A/uncalibrated-pixels-1.collateral.nc")); + } + + private Set producerConsumerTableFilenames(String pixelType) { + Path commonPath = DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target"); + return constructProducerConsumerPaths(commonPath.resolve(pixelType).resolve("1:1:A")); + } + + private Set constructProducerConsumerPaths(Path datastorePath) { + Set dirFiles = FileUtil.listFiles(datastorePath); + return dirFiles.stream() + .map(s -> DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .relativize(s) + .toString()) + .collect(Collectors.toSet()); + } + + /** Tests that DatastoreCopyType COPY produces a recursive copy of a directory. */ + @Test + public void testCopy() throws IOException { + FileUtil.CopyType.COPY.copy(DirectoryProperties.datastoreRootDir(), + ziggyDirectoryRule.directory().resolve("copydir")); + assertTrue(Files.isDirectory(ziggyDirectoryRule.directory().resolve("copydir"))); + assertTrue(Files.isDirectory(DirectoryProperties.datastoreRootDir())); + assertFalse(Files.isSameFile(ziggyDirectoryRule.directory().resolve("copydir"), + DirectoryProperties.datastoreRootDir())); + assertFalse(Files.isSymbolicLink(ziggyDirectoryRule.directory().resolve("copydir"))); + Path copiedFile = ziggyDirectoryRule.directory() + .resolve("copydir") + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("uncalibrated-pixels-0.science.nc"); + assertTrue(Files.isRegularFile(copiedFile)); + assertFalse(Files.isSymbolicLink(copiedFile)); + Path originalFile = DirectoryProperties.datastoreRootDir() + .resolve(ziggyDirectoryRule.directory().resolve("copydir").relativize(copiedFile)); + assertTrue(Files.isRegularFile(originalFile)); + assertFalse(Files.isSameFile(copiedFile, originalFile)); + } + + /** Tests that DatastoreCopyType MOVE moves a file or directory to a new location. */ + @Test + public void testMove() { + FileUtil.CopyType.MOVE.copy(DirectoryProperties.datastoreRootDir(), + ziggyDirectoryRule.directory().resolve("copydir")); + assertTrue(Files.isDirectory(ziggyDirectoryRule.directory().resolve("copydir"))); + assertFalse(Files.exists(DirectoryProperties.datastoreRootDir())); + assertFalse(Files.isSymbolicLink(ziggyDirectoryRule.directory().resolve("copydir"))); + Path copiedFile = ziggyDirectoryRule.directory() + .resolve("copydir") + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("uncalibrated-pixels-0.science.nc"); + assertTrue(Files.isRegularFile(copiedFile)); + assertFalse(Files.isSymbolicLink(copiedFile)); + } + + /** + * Tests that DatastoreCopyType LINK produces a hard link of a file to a new location, and + * produces copies of directories (which cannot be hard link targets). + */ + @Test + public void testLink() throws IOException { + FileUtil.CopyType.LINK.copy(DirectoryProperties.datastoreRootDir(), + ziggyDirectoryRule.directory().resolve("copydir")); + assertTrue(Files.isDirectory(ziggyDirectoryRule.directory().resolve("copydir"))); + assertTrue(Files.isDirectory(DirectoryProperties.datastoreRootDir())); + assertFalse(Files.isSameFile(ziggyDirectoryRule.directory().resolve("copydir"), + DirectoryProperties.datastoreRootDir())); + assertFalse(Files.isSymbolicLink(ziggyDirectoryRule.directory().resolve("copydir"))); + Path copiedFile = ziggyDirectoryRule.directory() + .resolve("copydir") + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("uncalibrated-pixels-0.science.nc"); + assertTrue(Files.isRegularFile(copiedFile)); + assertFalse(Files.isSymbolicLink(copiedFile)); + Path originalFile = DirectoryProperties.datastoreRootDir() + .resolve(ziggyDirectoryRule.directory().resolve("copydir").relativize(copiedFile)); + assertTrue(Files.isRegularFile(originalFile)); + assertTrue(Files.isSameFile(copiedFile, originalFile)); + } +} diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreTestUtils.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreTestUtils.java new file mode 100644 index 0000000..0c12a3a --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreTestUtils.java @@ -0,0 +1,327 @@ +package gov.nasa.ziggy.data.datastore; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.services.config.DirectoryProperties; + +/** + * Static methods that can be used to prepare datastore-related data objects for unit tests. + * + * @author PT + */ +public class DatastoreTestUtils { + + /** + * Returns datastore nodes based on a partial implementation of the TESS DR and CAL locations. + */ + public static Map datastoreNodesByFullPath() { + Map datastoreNodesByFullPath = new HashMap<>(); + + DatastoreNode sectorNode = new DatastoreNode("sector", true); + setFullPath(sectorNode, null); + datastoreNodesByFullPath.put(sectorNode.getFullPath(), sectorNode); + + DatastoreNode mdaNode = new DatastoreNode("mda", false); + setFullPath(mdaNode, sectorNode); + sectorNode.setChildNodeFullPaths(List.of("sector/mda")); + datastoreNodesByFullPath.put(mdaNode.getFullPath(), mdaNode); + + DatastoreNode drNode = new DatastoreNode("dr", false); + setFullPath(drNode, mdaNode); + datastoreNodesByFullPath.put(drNode.getFullPath(), drNode); + + DatastoreNode calNode = new DatastoreNode("cal", false); + setFullPath(calNode, mdaNode); + mdaNode.setChildNodeFullPaths(List.of("sector/mda/dr", "sector/mda/cal")); + datastoreNodesByFullPath.put(calNode.getFullPath(), calNode); + + DatastoreNode drPixelNode = new DatastoreNode("pixels", false); + setFullPath(drPixelNode, drNode); + drNode.setChildNodeFullPaths(List.of("sector/mda/dr/pixels")); + datastoreNodesByFullPath.put(drPixelNode.getFullPath(), drPixelNode); + + DatastoreNode drCadenceTypeNode = new DatastoreNode("cadenceType", true); + setFullPath(drCadenceTypeNode, drPixelNode); + drPixelNode.setChildNodeFullPaths(List.of("sector/mda/dr/pixels/cadenceType")); + datastoreNodesByFullPath.put(drCadenceTypeNode.getFullPath(), drCadenceTypeNode); + + DatastoreNode drPixelTypeNode = new DatastoreNode("pixelType", true); + setFullPath(drPixelTypeNode, drCadenceTypeNode); + drCadenceTypeNode + .setChildNodeFullPaths(List.of("sector/mda/dr/pixels/cadenceType/pixelType")); + datastoreNodesByFullPath.put(drPixelTypeNode.getFullPath(), drPixelTypeNode); + + DatastoreNode drChannelNode = new DatastoreNode("channel", true); + setFullPath(drChannelNode, drPixelTypeNode); + drPixelTypeNode + .setChildNodeFullPaths(List.of("sector/mda/dr/pixels/cadenceType/pixelType/channel")); + datastoreNodesByFullPath.put(drChannelNode.getFullPath(), drChannelNode); + + DatastoreNode calPixelNode = new DatastoreNode("pixels", false); + setFullPath(calPixelNode, calNode); + calNode.setChildNodeFullPaths(List.of("sector/mda/cal/pixels")); + datastoreNodesByFullPath.put(calPixelNode.getFullPath(), calPixelNode); + + DatastoreNode calCadenceTypeNode = new DatastoreNode("cadenceType", true); + setFullPath(calCadenceTypeNode, calPixelNode); + calPixelNode.setChildNodeFullPaths(List.of("sector/mda/cal/pixels/cadenceType")); + datastoreNodesByFullPath.put(calCadenceTypeNode.getFullPath(), calCadenceTypeNode); + + DatastoreNode calPixelTypeNode = new DatastoreNode("pixelType", true); + setFullPath(calPixelTypeNode, calCadenceTypeNode); + calCadenceTypeNode + .setChildNodeFullPaths(List.of("sector/mda/cal/pixels/cadenceType/pixelType")); + datastoreNodesByFullPath.put(calPixelTypeNode.getFullPath(), calPixelTypeNode); + + DatastoreNode calChannelNode = new DatastoreNode("channel", true); + setFullPath(calChannelNode, calPixelTypeNode); + calPixelTypeNode + .setChildNodeFullPaths(List.of("sector/mda/cal/pixels/cadenceType/pixelType/channel")); + datastoreNodesByFullPath.put(calChannelNode.getFullPath(), calChannelNode); + + return datastoreNodesByFullPath; + } + + private static void setFullPath(DatastoreNode node, DatastoreNode parent) { + String parentPath = parent != null ? parent.getFullPath() : null; + node.setFullPath( + DatastoreConfigurationImporter.fullPathFromParentPath(node.getName(), parentPath)); + } + + /** Returns regexps based on a partial implementation of DR and CAL. */ + public static Map regexpsByName() { + Map regexpsByName = new HashMap<>(); + + DatastoreRegexp sectorRegexp = new DatastoreRegexp("sector", "(sector-[0-9]{4})"); + regexpsByName.put(sectorRegexp.getName(), sectorRegexp); + + DatastoreRegexp cadenceTypeRegexp = new DatastoreRegexp("cadenceType", + "(target|ffi|fast-target)"); + regexpsByName.put(cadenceTypeRegexp.getName(), cadenceTypeRegexp); + + DatastoreRegexp pixelTypeRegexp = new DatastoreRegexp("pixelType", "(science|collateral)"); + regexpsByName.put(pixelTypeRegexp.getName(), pixelTypeRegexp); + + DatastoreRegexp channelRegexp = new DatastoreRegexp("channel", "([1-4]:[1-4]:[A-D])"); + regexpsByName.put(channelRegexp.getName(), channelRegexp); + + return regexpsByName; + } + + /** Returns data file types based on CAL inputs and outputs. */ + public static Map dataFileTypesByName() { + + Map dataFileTypesByName = new HashMap<>(); + + DataFileType uncalibratedSciencePixelType = new DataFileType( + "uncalibrated science pixel values", + "sector/mda/dr/pixels/cadenceType/pixelType$science/channel"); + dataFileTypesByName.put(uncalibratedSciencePixelType.getName(), + uncalibratedSciencePixelType); + + DataFileType uncalibratedCollateralPixelType = new DataFileType( + "uncalibrated collateral pixel values", + "sector/mda/dr/pixels/cadenceType/pixelType$collateral/channel"); + dataFileTypesByName.put(uncalibratedCollateralPixelType.getName(), + uncalibratedCollateralPixelType); + + DataFileType calibratedSciencePixelType = new DataFileType( + "calibrated science pixel values", + "sector/mda/cal/pixels/cadenceType/pixelType$science/channel"); + dataFileTypesByName.put(calibratedSciencePixelType.getName(), calibratedSciencePixelType); + + DataFileType calibratedCollateralPixelType = new DataFileType( + "calibrated collateral pixel values", + "sector/mda/cal/pixels/cadenceType/pixelType$collateral/channel"); + dataFileTypesByName.put(calibratedCollateralPixelType.getName(), + calibratedCollateralPixelType); + + return dataFileTypesByName; + } + + /** + * Creates a subset of datastore directories for CAL inputs and outputs. The resulting + * directories are created in the directory indicated by the DATASTORE_ROOT_DIR. To use this + * method, do the following in the caller: + *

        + *
      1. Use {@link ZiggyDirectoryRule} to create a directory for test artifacts. + *
      2. Use {@link ZiggyPropertyRule} to set the DATASTORE_ROOT_DIR to a subdirectory in the test + * artifact directory. + */ + public static void createDatastoreDirectories() throws IOException { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + + // Start with sector 2 uncalibrated target pixels for 1:1:A and 1:1:B. + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:B")); + + // Sector 2 uncalibrated FFI pixels for 1:1:A and 1:1:B. + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B")); + + // Sector 3 uncalibrated target pixels for 1:1:A and 1:1:B. + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:B")); + + // Sector 3 uncalibrated FFI pixels for 1:1:A and 1:1:B. + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B")); + + // Sector 2 calibrated target pixels for 1:1:A. + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A")); + + Files.createDirectories(datastoreRoot.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A")); + + // Sector 3 calibrated FFI pixels for 1:1:B. + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B")); + + Files.createDirectories(datastoreRoot.resolve("sector-0003") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B")); + } +} diff --git a/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java new file mode 100644 index 0000000..aa2410c --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/data/datastore/DatastoreWalkerTest.java @@ -0,0 +1,384 @@ +package gov.nasa.ziggy.data.datastore; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Path; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.RuleChain; + +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.config.PropertyName; + +/** + * Unit tests for the {@link DatastoreWalker} class. + * + * @author PT + */ +public class DatastoreWalkerTest { + + private DatastoreWalker datastoreWalker; + + public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); + public ZiggyPropertyRule datastoreRootPropertyRule = new ZiggyPropertyRule( + PropertyName.DATASTORE_ROOT_DIR.property(), directoryRule, "datastore"); + + @Rule + public RuleChain ruleChain = RuleChain.outerRule(directoryRule) + .around(datastoreRootPropertyRule); + + @Before + public void setUp() throws IOException { + datastoreWalker = new DatastoreWalker(DatastoreTestUtils.regexpsByName(), + DatastoreTestUtils.datastoreNodesByFullPath()); + DatastoreTestUtils.createDatastoreDirectories(); + } + + @Test + public void testLocationExists() { + assertTrue(datastoreWalker.locationExists("sector/mda/dr/pixels/cadenceType/pixelType")); + assertTrue(datastoreWalker.locationExists("sector/mda/dr/pixels/cadenceType$ffi")); + assertFalse(datastoreWalker.locationExists("sector/foo/dr")); + assertFalse(datastoreWalker.locationExists("sector/mda/cal/pixels/cadenceType$foo")); + assertFalse(datastoreWalker.locationExists("sector/mda/dr/pixels/cadenceType$ffi$target")); + } + + @Test + public void testPathsForLocation() throws IOException { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + + List paths = datastoreWalker + .pathsForLocation("sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel"); + assertEquals(4, paths.size()); + + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B"))); + + paths = datastoreWalker + .pathsForLocation("sector/mda/dr/pixels/cadenceType/pixelType/channel"); + assertEquals(16, paths.size()); + + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B"))); + + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B"))); + + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B"))); + + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:B"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A"))); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:B"))); + + // Now test with include and exclude regular expressions. + Map regexpsByName = datastoreWalker.regexpsByName(); + regexpsByName.get("sector").setInclude("sector-0002"); + regexpsByName.get("channel").setExclude("1:1:A"); + + paths = datastoreWalker + .pathsForLocation("sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel"); + assertEquals(1, paths.size()); + assertTrue(paths.contains(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B"))); + } + + @Test + public void testDatastoreDirectoryBriefState() throws IOException { + int datastoreRootPathElements = DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .getNameCount(); + + List paths = datastoreWalker + .pathsForLocation("sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel"); + + List pathElementIndices = datastoreWalker.pathElementIndicesForBriefState(paths); + assertTrue(pathElementIndices.contains(datastoreRootPathElements + 0)); + assertTrue(pathElementIndices.contains(datastoreRootPathElements + 6)); + assertEquals(2, pathElementIndices.size()); + + // Now test with include and exclude regular expressions. + Map regexpsByName = datastoreWalker.regexpsByName(); + regexpsByName.get("sector").setInclude("sector-0002"); + regexpsByName.get("channel").setExclude("1:1:A"); + + paths = datastoreWalker + .pathsForLocation("sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel"); + + pathElementIndices = datastoreWalker.pathElementIndicesForBriefState(paths); + assertTrue(pathElementIndices.isEmpty()); + } + + @Test + public void testLocationMatchesDatastore() { + assertTrue(datastoreWalker.locationMatchesDatastore("sector-0002/mda")); + assertTrue(datastoreWalker.locationMatchesDatastore("sector-0003/mda/dr/pixels")); + assertFalse(datastoreWalker.locationMatchesDatastore("sector/mda")); + assertFalse(datastoreWalker.locationMatchesDatastore("sector-0003/tba")); + assertFalse( + datastoreWalker.locationMatchesDatastore("sector-0003/mda/dr/pixels/cadenceType$ffi")); + assertFalse(datastoreWalker + .locationMatchesDatastore("sector-0003/mda/dr/pixels/ffi/collateral/1:1:A/subdir")); + } + + @Test + public void testRegexpValues() { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + Map regexpValues = datastoreWalker.regexpValues( + "sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel", + datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B")); + assertNotNull(regexpValues.get("sector")); + assertEquals("sector-0002", regexpValues.get("sector")); + assertNotNull(regexpValues.get("cadenceType")); + assertEquals("ffi", regexpValues.get("cadenceType")); + assertNotNull(regexpValues.get("pixelType")); + assertEquals("science", regexpValues.get("pixelType")); + assertNotNull(regexpValues.get("channel")); + assertEquals("1:1:B", regexpValues.get("channel")); + assertEquals(4, regexpValues.size()); + } + + @Test + public void testRegexpValuesWithLocationSuppression() { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + Map regexpValues = datastoreWalker.regexpValues( + "sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel", + datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B"), + false); + assertNotNull(regexpValues.get("sector")); + assertEquals("sector-0002", regexpValues.get("sector")); + assertNotNull(regexpValues.get("channel")); + assertEquals("1:1:B", regexpValues.get("channel")); + assertEquals(2, regexpValues.size()); + } + + @Test + public void testPathFromLocationAndRegexpValues() { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + Map regexpValues = datastoreWalker.regexpValues( + "sector/mda/dr/pixels/cadenceType$ffi/pixelType$science/channel", + datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B")); + Path constructedPath = datastoreWalker.pathFromLocationAndRegexpValues(regexpValues, + "sector/mda/dr/pixels/cadenceType$target/pixelType$collateral/channel"); + assertEquals(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:B") + .toString(), constructedPath.toString()); + } + + /** + * Tests whether the pathFromLocationAndRegexpValues method does the right thing when one of the + * regexps is missing from the Map of values but has a value assigned in the location argument. + */ + @Test + public void testPathFromRegexValuesWhenPartNotInRegexpMap() { + Path datastoreRoot = DirectoryProperties.datastoreRootDir(); + Map regexpValues = new HashMap<>(); + regexpValues.put("sector", "sector-0002"); + regexpValues.put("cadenceType", "target"); + regexpValues.put("channel", "1:1:B"); + Path constructedPath = datastoreWalker.pathFromLocationAndRegexpValues(regexpValues, + "sector/mda/dr/pixels/cadenceType$target/pixelType$collateral/channel"); + assertEquals(datastoreRoot.toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:B") + .toString(), constructedPath.toString()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java b/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java index e9a037c..e099a1a 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/AcknowledgementTest.java @@ -250,7 +250,7 @@ public void testXmlRoundTrip() @Test public void testSchema() throws IOException { - Path schemaPath = Paths.get(ziggyHomeDirPropertyRule.getProperty(), "schema", "xml", + Path schemaPath = Paths.get(ziggyHomeDirPropertyRule.getValue(), "schema", "xml", new Acknowledgement().getXmlSchemaFilename()); List schemaContent = Files.readAllLines(schemaPath, FileUtil.ZIGGY_CHARSET); diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataFileInfoTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataFileInfoTest.java deleted file mode 100644 index e667791..0000000 --- a/src/test/java/gov/nasa/ziggy/data/management/DataFileInfoTest.java +++ /dev/null @@ -1,49 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.nio.file.Path; -import java.nio.file.Paths; - -import org.junit.Test; - -import gov.nasa.ziggy.data.management.DataFileTestUtils.DataFileInfoSample1; - -/** - * Class of unit tests for the DataFileInfo class. - * - * @author PT - */ -public class DataFileInfoTest { - - @Test - public void testStringArgConstructor() { - DataFileInfoSample1 d = new DataFileInfoSample1("pa-123456789-100-results.h5"); - Path p = d.getName(); - assertEquals("pa-123456789-100-results.h5", p.toString()); - } - - @Test - public void testPathArgConstructor() { - DataFileInfoSample1 d = new DataFileInfoSample1(Paths.get("pa-123456789-100-results.h5")); - Path p = d.getName(); - assertEquals("pa-123456789-100-results.h5", p.toString()); - } - - @Test - public void testPathValid() { - DataFileInfoSample1 d = new DataFileInfoSample1(); - assertTrue(d.pathValid(Paths.get("pa-123456789-100-results.h5"))); - assertFalse(d.pathValid(Paths.get("some-other-string.h5"))); - } - - @Test - public void testCompareTo() { - DataFileInfoSample1 d1 = new DataFileInfoSample1("pa-123456789-100-results.h5"); - DataFileInfoSample1 d2 = new DataFileInfoSample1("pa-123456789-101-results.h5"); - assertTrue(d1.compareTo(d2) < 0); - assertTrue(d1.compareTo(d1) == 0); - } -} diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataFileManagerTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataFileManagerTest.java deleted file mode 100644 index 124a22b..0000000 --- a/src/test/java/gov/nasa/ziggy/data/management/DataFileManagerTest.java +++ /dev/null @@ -1,1920 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; -import static gov.nasa.ziggy.services.config.PropertyName.USE_SYMLINKS; -import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_TEST_WORKING_DIR; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.nio.file.attribute.PosixFilePermission; -import java.nio.file.attribute.PosixFilePermissions; -import java.util.ArrayList; -import java.util.Collection; -import java.util.Collections; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeSet; -import java.util.stream.Collectors; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.RuleChain; -import org.mockito.ArgumentMatchers; -import org.mockito.Mockito; - -import com.google.common.collect.Lists; -import com.google.common.collect.Sets; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.data.management.DataFileTestUtils.DataFileInfoSample1; -import gov.nasa.ziggy.data.management.DataFileTestUtils.DataFileInfoSample2; -import gov.nasa.ziggy.data.management.DataFileTestUtils.DataFileInfoSampleForDirs; -import gov.nasa.ziggy.data.management.DataFileTestUtils.DatastorePathLocatorSample; -import gov.nasa.ziggy.module.AlgorithmStateFiles; -import gov.nasa.ziggy.module.TaskConfigurationManager; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; -import gov.nasa.ziggy.uow.TaskConfigurationParameters; - -/** - * Test class for DataFileManager class. - * - * @author PT - */ -public class DataFileManagerTest { - - private String datastoreRoot; - private String taskDir; - private String subtaskDir; - private DataFileManager dataFileManager; - private DataFileManager dataFileManager2; - private static final long TASK_ID = 100L; - private static final long PROD_TASK_ID1 = 10L; - private static final long PROD_TASK_ID2 = 11L; - private String externalTempDir; - - private PipelineTask pipelineTask; - private PipelineDefinitionNode pipelineDefinitionNode; - private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud; - private PipelineTaskCrud pipelineTaskCrud; - private TaskConfigurationParameters taskConfigurationParameters; - - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( - DATASTORE_ROOT_DIR, directoryRule, "datastore"); - - @Rule - public ZiggyPropertyRule useSymlinksPropertyRule = new ZiggyPropertyRule(USE_SYMLINKS, - (String) null); - - public ZiggyPropertyRule ziggyTestWorkingDirPropertyRule = new ZiggyPropertyRule( - ZIGGY_TEST_WORKING_DIR, directoryRule, "pa-5-10" + File.separator + "st-0"); - - @Rule - public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) - .around(datastoreRootDirPropertyRule) - .around(ziggyTestWorkingDirPropertyRule); - - @Before - public void setup() throws IOException { - Path datastore = Paths.get(datastoreRootDirPropertyRule.getProperty()); - Files.createDirectories(datastore); - datastoreRoot = datastore.toString(); - - Path taskDirRoot = directoryRule.directory().resolve("taskspace"); - Files.createDirectories(taskDirRoot); - subtaskDir = ziggyTestWorkingDirPropertyRule.getProperty(); - File subtaskFile = new File(subtaskDir); - subtaskFile.mkdirs(); - taskDir = subtaskFile.getParent(); - - Path externalTemp = directoryRule.directory().resolve("tmp"); - Files.createDirectories(externalTemp); - externalTempDir = externalTemp.toAbsolutePath().toString(); - - // For some tests we will need a pipeline task and a DatastoreProducerConsumerCrud; - // set that up now. - pipelineTask = Mockito.spy(PipelineTask.class); - datastoreProducerConsumerCrud = new ProducerConsumerCrud(); - pipelineTaskCrud = Mockito.mock(PipelineTaskCrud.class); - Mockito.when(pipelineTask.getId()).thenReturn(TASK_ID); - pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); - Mockito.doReturn(pipelineDefinitionNode).when(pipelineTask).getPipelineDefinitionNode(); - - taskConfigurationParameters = new TaskConfigurationParameters(); - taskConfigurationParameters.setReprocess(true); - Mockito.doReturn(taskConfigurationParameters) - .when(pipelineTask) - .getParameters(ArgumentMatchers.eq(TaskConfigurationParameters.class)); - Mockito.doReturn(taskConfigurationParameters) - .when(pipelineTask) - .getParameters(ArgumentMatchers.eq(TaskConfigurationParameters.class), - ArgumentMatchers.anyBoolean()); - initializeDataFileManager(); - - // Now build a DataFileManager for use with DataFileType instances and Ziggy unit of work - // generators. - initializeDataFileManager2(); - DataFileTestUtils.initializeDataFileTypeSamples(); - } - - @Test - public void testDataFilesMap() throws IOException { - - // setup the task directory - constructTaskDirFiles(); - - // construct the set of DatastoreId subclasses - Set> datastoreIdClasses = new HashSet<>(); - datastoreIdClasses.add(DataFileInfoSample1.class); - datastoreIdClasses.add(DataFileInfoSample2.class); - - // construct the map - Map, Set> datastoreIdMap = new DataFileManager() - .dataFilesMap(Paths.get(taskDir), datastoreIdClasses); - - // The map should have 2 entries - assertEquals(2, datastoreIdMap.size()); - - // The DatastoreIdSample1 entry should have 2 DatastoreIds in it - @SuppressWarnings("unchecked") - Set d1Set = (Set) datastoreIdMap - .get(DataFileInfoSample1.class); - assertEquals(2, d1Set.size()); - Set names = getNamesFromDatastoreIds(d1Set); - assertTrue(names.contains("pa-001234567-20-results.h5")); - assertTrue(names.contains("pa-765432100-20-results.h5")); - - // The DatastoreIdSample2 entry should have 2 DatastoreIds in it - @SuppressWarnings("unchecked") - Set d2Set = (Set) datastoreIdMap - .get(DataFileInfoSample2.class); - assertEquals(2, d2Set.size()); - names = getNamesFromDatastoreIds(d2Set); - assertTrue(names.contains("cal-1-1-A-20-results.h5")); - assertTrue(names.contains("cal-1-1-B-20-results.h5")); - - // NB: the PDC file was ignored, as it should be. - } - - /** - * Tests the taskDirectoryDataFilesMap() method of DataFileManager. - * - * @throws IOException - */ - @Test - public void testTaskDirectoryDataFilesMap() throws IOException { - - constructTaskDirFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Construct the Map - Map> dataFileTypeMap = dataFileManager2 - .taskDirectoryDataFilesMap(dataFileTypes); - - assertEquals(2, dataFileTypeMap.size()); - - // Make sure the PA files were correctly identified - Set d1Set = dataFileTypeMap.get(DataFileTestUtils.dataFileTypeSample1); - assertEquals(2, d1Set.size()); - Set d1Names = getNamesFromPaths(d1Set); - assertTrue(d1Names.contains("pa-001234567-20-results.h5")); - assertTrue(d1Names.contains("pa-765432100-20-results.h5")); - - // Now check the CAL files - Set d2Set = dataFileTypeMap.get(DataFileTestUtils.dataFileTypeSample2); - assertEquals(2, d1Set.size()); - Set d2Names = getNamesFromPaths(d2Set); - assertTrue(d2Names.contains("cal-1-1-A-20-results.h5")); - assertTrue(d2Names.contains("cal-1-1-B-20-results.h5")); - } - - /** - * Tests the datastoreDataFilesMap method of DataFileManager. - * - * @throws IOException - * @throws InterruptedException - */ - @Test - public void testDatastoreDataFilesMap() throws IOException, InterruptedException { - - constructDatastoreFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Construct the Map - Map> dataFileTypeMap = dataFileManager2 - .datastoreDataFilesMap(Paths.get(""), dataFileTypes); - - assertEquals(2, dataFileTypeMap.size()); - - // Make sure the PA files were correctly identified - Set d1Set = dataFileTypeMap.get(DataFileTestUtils.dataFileTypeSample1); - assertEquals(2, d1Set.size()); - Set d1Names = getNamesFromPaths(d1Set); - assertTrue(d1Names.contains("pa-001234567-20-results.h5")); - assertTrue(d1Names.contains("pa-765432100-20-results.h5")); - - // Now check the CAL files - Set d2Set = dataFileTypeMap.get(DataFileTestUtils.dataFileTypeSample2); - assertEquals(2, d1Set.size()); - Set d2Names = getNamesFromPaths(d2Set); - assertTrue(d2Names.contains("cal-1-1-A-20-results.h5")); - assertTrue(d2Names.contains("cal-1-1-B-20-results.h5")); - } - - /** - * Tests the datastoreFiles() method. - * - * @throws IOException - * @throws InterruptedException - */ - @Test - public void testDatastoreFiles() throws IOException, InterruptedException { - - constructTaskDirFiles(); - - // construct the set of DatastoreId subclasses - Set> datastoreIdClasses = new HashSet<>(); - datastoreIdClasses.add(DataFileInfoSample1.class); - datastoreIdClasses.add(DataFileInfoSample2.class); - Set datastoreIds = new DataFileManager().datastoreFiles(Paths.get(taskDir), - datastoreIdClasses); - assertEquals(4, datastoreIds.size()); - Set names = getNamesFromDatastoreIds(datastoreIds); - assertTrue(names.contains("pa-001234567-20-results.h5")); - assertTrue(names.contains("pa-765432100-20-results.h5")); - assertTrue(names.contains("cal-1-1-A-20-results.h5")); - assertTrue(names.contains("cal-1-1-B-20-results.h5")); - } - - /** - * Tests the copyToTaskDirectory() methods, for individual files. - * - * @throws IOException - * @throws InterruptedException - */ - @Test - public void testCopyFilesToTaskDirectory() throws IOException, InterruptedException { - - // Set up the datastore. - constructDatastoreFiles(); - - // Create the DatastoreId objects. - Set datastoreIds = constructDatastoreIds(); - - // Perform the copy. - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager.copyToTaskDirectory(datastoreIds); - File[] endFileList = taskDirFile.listFiles(); - assertEquals(4, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList); - assertEquals(3, filenames.size()); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - - // Check that the copies are real copies, not symlinks. - for (File file : endFileList) { - assertFalse(java.nio.file.Files.isSymbolicLink(file.toPath())); - } - - // check that the originators were set correctly - Set producerTaskIds = pipelineTask.getProducerTaskIds(); - assertEquals(2, producerTaskIds.size()); - assertTrue(producerTaskIds.contains(PROD_TASK_ID1)); - assertTrue(producerTaskIds.contains(PROD_TASK_ID2)); - } - - /** - * Test the copyFilesByNameToWorkingDirectory() method in the case in which the files are - * actually copied and not just symlinked. - */ - @Test - public void testCopyFilesByNameToWorkingDirectory() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - // create the DatastoreId objects - Set datastoreIds = constructDatastoreIds(); - - // copy files to the task directory - dataFileManager.copyToTaskDirectory(datastoreIds); - File[] endFileList = new File(taskDir).listFiles(); - Set filenames = getNamesFromListFiles(endFileList); - - endFileList = new File(subtaskDir).listFiles(); - assertEquals(0, endFileList.length); - dataFileManager.copyFilesByNameFromTaskDirToWorkingDir(filenames); - endFileList = new File(subtaskDir).listFiles(); - assertEquals(3, endFileList.length); - filenames = getNamesFromListFiles(endFileList); - assertEquals(3, filenames.size()); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - - // Check that the copies are real copies, not symlinks - for (File file : endFileList) { - assertFalse(java.nio.file.Files.isSymbolicLink(file.toPath())); - } - } - - /** - * Tests the copyToTaskDirectory() methods, for individual files, in the case in which the - * method constructs symlinks instead of performing true copy operations. - * - * @throws IOException - * @throws InterruptedException - */ - @Test - public void testSymlinkFilesToTaskDirectory() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager(); - - // construct a new file in the external temp directory - Path externalFile = Paths.get(externalTempDir, "pa-001234569-20-results.h5"); - java.nio.file.Files.createFile(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), externalFile); - - // create the DatastoreId objects - Set datastoreIds = constructDatastoreIds(); - - // create the DatastoreId object - DataFileInfoSample1 pa1 = new DataFileInfoSample1("pa-001234569-20-results.h5"); - datastoreIds.add(pa1); - - // perform the copy - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager.copyToTaskDirectory(datastoreIds); - File[] endFileList = taskDirFile.listFiles(); - assertEquals(5, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("pa-001234569-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - - // Check that the copies are actually symlinks - for (File file : endFileList) { - if (!file.getName().equals("st-0")) { - assertTrue(java.nio.file.Files.isSymbolicLink(file.toPath())); - } - } - - // check that the copies are symlinks of the correct files - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234567-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-001234567-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-765432100-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-765432100-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-A-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "cal-1-1-A-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-001234569-20-results.h5"))); - - // check that the originators were set correctly - Set producerTaskIds = pipelineTask.getProducerTaskIds(); - assertEquals(2, producerTaskIds.size()); - assertTrue(producerTaskIds.contains(PROD_TASK_ID1)); - assertTrue(producerTaskIds.contains(PROD_TASK_ID2)); - } - - /** - * Tests that the search along a symlink path for the true source file doesn't go past the - * boundaries of the datastore. - * - * @throws InterruptedException - * @throws IOException - */ - @Test - public void testSymlinkFilesWithSearchLimits() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager(); - - // construct a new file in the external temp directory - Path externalFile = Paths.get(externalTempDir, "pa-001234569-20-results.h5"); - java.nio.file.Files.createFile(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), externalFile); - - // create the DatastoreId object - DataFileInfoSample1 pa1 = new DataFileInfoSample1("pa-001234569-20-results.h5"); - Set datastoreIds = new HashSet<>(); - datastoreIds.add(pa1); - - // perform the copy - dataFileManager.copyToTaskDirectory(datastoreIds); - File[] endFileList = new File(taskDir).listFiles(); - assertEquals(2, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234569-20-results.h5")); - - // Check that the file is a symlink, and that it's a symlink of the correct source file - for (File file : endFileList) { - if (!file.getName().equals("st-0")) { - assertTrue(java.nio.file.Files.isSymbolicLink(file.toPath())); - } - } - // check that the copies are symlinks of the correct files - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-001234569-20-results.h5"))); - } - - /** - * Tests that the copyFilesByNameToWorkingDirectory method properly makes symlinks rather than - * copies. - */ - @Test - public void testSymlinkFilesByNameToWorkingDirectory() - throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager(); - - // create the DatastoreId objects - Set datastoreIds = constructDatastoreIds(); - - // construct a new file in the external temp directory - Path externalFile = Paths.get(externalTempDir, "pa-001234569-20-results.h5"); - java.nio.file.Files.createFile(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), externalFile); - - // create the DatastoreId object - DataFileInfoSample1 pa1 = new DataFileInfoSample1("pa-001234569-20-results.h5"); - datastoreIds.add(pa1); - - // copy files to the task directory - dataFileManager.copyToTaskDirectory(datastoreIds); - File[] endFileList = new File(taskDir).listFiles(); - Set filenames = getNamesFromListFiles(endFileList); - - endFileList = new File(subtaskDir).listFiles(); - assertEquals(0, endFileList.length); - dataFileManager.copyFilesByNameFromTaskDirToWorkingDir(filenames); - endFileList = new File(subtaskDir).listFiles(); - assertEquals(4, endFileList.length); - filenames = getNamesFromListFiles(endFileList); - assertEquals(4, filenames.size()); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("pa-001234569-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - - // Check that the copies are real copies, not symlinks - for (File file : endFileList) { - assertTrue(java.nio.file.Files.isSymbolicLink(file.toPath())); - } - - // check that the copies are symlinks of the correct files (i.e., to the files in the - // datastore and not the files in the task directory) - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234567-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-001234567-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-765432100-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-765432100-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-001234569-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-A-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(subtaskDir, "cal-1-1-A-20-results.h5"))); - } - - @Test - public void testSymlinkDirectoriesByNameToWorkingDirectory() throws IOException { - - constructDatastoreDirectories(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager2(); - - // construct a new file in the external temp directory and a symlink to same in - // the datastore - Path externalFile = Paths.get(externalTempDir, "EO1H0230412000337112N0_WGS_01"); - java.nio.file.Files.createDirectory(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "EO1H0230412000337112N0_WGS_01"), externalFile); - - // Construct the DataFileInfo instances - Set datastoreIds = new HashSet<>(); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112N0_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112NO_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H2240632000337112NP_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230412000337112N0_WGS_01")); - - DataFileTestUtils.initializeDataFileTypeForDirectories(); - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeForDirectories); - - // Copy the files to the task directory - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - - // Now copy the files by name to the subtask directory - List dirNamesToCopy = new ArrayList<>(); - dirNamesToCopy.add("EO1H0230312000337112N0_WGS_01"); - dirNamesToCopy.add("EO1H0230312000337112NO_WGS_01"); - dirNamesToCopy.add("EO1H2240632000337112NP_WGS_01"); - dirNamesToCopy.add("EO1H0230412000337112N0_WGS_01"); - - dataFileManager2.copyFilesByNameFromTaskDirToWorkingDir(dirNamesToCopy); - - File[] endFileList = new File(subtaskDir).listFiles(); - assertEquals(4, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList, true); - assertEquals(4, filenames.size()); - assertTrue(filenames.containsAll(dirNamesToCopy)); - - // Check that the symlinks are links of the expected files - assertEquals(Paths.get(datastoreRoot, "EO1H0230312000337112N0_WGS_01"), java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "EO1H0230312000337112N0_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H0230312000337112NO_WGS_01"), java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "EO1H0230312000337112NO_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H2240632000337112NP_WGS_01"), java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "EO1H2240632000337112NP_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H0230412000337112N0_WGS_01"), java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "EO1H0230412000337112N0_WGS_01"))); - } - - /** - * Tests the deleteFromTaskDirectory() method, for individual files. - */ - @Test - public void testDeleteFilesFromTaskDirectory() throws IOException { - - Set datastoreFilenames = constructTaskDirFiles(); - - // create the DataFileInfo objects - Set datastoreIds = constructDatastoreIds(); - - // Copy the data file objects to the subtask directory. - dataFileManager.copyFilesByNameFromTaskDirToWorkingDir(datastoreFilenames); - - // The files in the datastoreIds should be gone from the task directory but still - // present in the subtask directory - dataFileManager.deleteFromTaskDirectory(datastoreIds); - Set filesInTaskDir = getNamesFromListFiles(new File(taskDir).listFiles()); - assertEquals(2, filesInTaskDir.size()); - assertTrue(filesInTaskDir.contains("cal-1-1-B-20-results.h5")); - assertTrue(filesInTaskDir.contains("pdc-1-1-20-results.h5")); - Set filesInSubtaskDir = getNamesFromListFiles(new File(subtaskDir).listFiles()); - assertEquals(5, filesInSubtaskDir.size()); - for (String filename : datastoreFilenames) { - assertTrue(filesInSubtaskDir.contains(filename)); - assertFalse(java.nio.file.Files.isSymbolicLink(Paths.get(subtaskDir, filename))); - } - } - - /** - * Tests that when symlinks are deleted from the task directory, they remain in the subtask - * directory and point to the datastore, not the task directory. - */ - @Test - public void testDeleteSymlinksFromTaskDirectory() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager(); - - // create the DatastoreId objects - Set datastoreIds = constructDatastoreIds(); - - // copy files to the task directory - dataFileManager.copyToTaskDirectory(datastoreIds); - File[] endFileList = new File(taskDir).listFiles(); - Set filenames = getNamesFromListFiles(endFileList); - - dataFileManager.copyFilesByNameFromTaskDirToWorkingDir(filenames); - - // The files in the datastoreIds should be gone from the task directory but still - // present in the subtask directory - dataFileManager.deleteFromTaskDirectory(datastoreIds); - Set filesInTaskDir = getNamesFromListFiles(new File(taskDir).listFiles()); - assertEquals(0, filesInTaskDir.size()); - Set filesInSubtaskDir = getNamesFromListFiles(new File(subtaskDir).listFiles()); - assertEquals(3, filesInSubtaskDir.size()); - assertTrue(filesInSubtaskDir.contains("pa-001234567-20-results.h5")); - assertTrue(filesInSubtaskDir.contains("pa-765432100-20-results.h5")); - assertTrue(filesInSubtaskDir.contains("cal-1-1-A-20-results.h5")); - - // The files in the working directory should be symlinks back to the datastore - Path datastorePath = Paths.get(datastoreRoot); - for (String filename : filesInSubtaskDir) { - Path subtaskPath = Paths.get(subtaskDir, filename); - assertTrue(java.nio.file.Files.isSymbolicLink(subtaskPath)); - assertTrue(java.nio.file.Files.readSymbolicLink(subtaskPath).startsWith(datastorePath)); - } - } - - /** - * Tests the moveToDatastore() method, for individual files. - * - * @throws IOException - */ - @Test - public void testMoveFilesToDatastore() throws IOException { - - constructTaskDirFiles(); - - // create the DataFileInfo objects - Set datastoreIds = constructDatastoreIds(); - - File paDatastoreFile = new File(datastoreRoot, "pa/20"); - File calDatastoreFile = new File(datastoreRoot, "cal/20"); - File taskDirFile = new File(taskDir); - paDatastoreFile.mkdirs(); - calDatastoreFile.mkdirs(); - assertEquals(0, paDatastoreFile.listFiles().length); - File[] taskFileList = taskDirFile.listFiles(); - assertEquals(6, taskFileList.length); - Set filenames = getNamesFromListFiles(taskFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(filenames.contains("pdc-1-1-20-results.h5")); - dataFileManager.moveToDatastore(datastoreIds); - File[] endFileList = paDatastoreFile.listFiles(); - assertEquals(2, endFileList.length); - filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(checkFilePermissions(endFileList, "r--r--r--")); - assertTrue(checkForSymlinks(endFileList, false)); - endFileList = calDatastoreFile.listFiles(); - assertEquals(1, endFileList.length); - assertEquals("cal-1-1-A-20-results.h5", endFileList[0].getName()); - assertTrue(checkFilePermissions(endFileList, "r--r--r--")); - assertTrue(checkForSymlinks(endFileList, false)); - taskFileList = taskDirFile.listFiles(); - assertEquals(3, taskFileList.length); - filenames = getNamesFromListFiles(taskFileList); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(filenames.contains("pdc-1-1-20-results.h5")); - } - - /** - * Tests that even when the copy mode for task dir files is symlinks, the move of files from the - * task dir to the datastore results in actual files, not symlinks. - * - * @throws IOException - */ - @Test - public void testMoveFilesToDatastoreSymlinkMode() throws IOException { - - constructTaskDirFiles(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager(); - - // create the DataFileInfo objects - Set datastoreIds = constructDatastoreIds(); - - File paDatastoreFile = new File(datastoreRoot, "pa/20"); - File calDatastoreFile = new File(datastoreRoot, "cal/20"); - File taskDirFile = new File(taskDir); - paDatastoreFile.mkdirs(); - calDatastoreFile.mkdirs(); - assertEquals(0, paDatastoreFile.listFiles().length); - File[] taskFileList = taskDirFile.listFiles(); - assertEquals(6, taskFileList.length); - Set filenames = getNamesFromListFiles(taskFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(filenames.contains("pdc-1-1-20-results.h5")); - dataFileManager.moveToDatastore(datastoreIds); - File[] endFileList = paDatastoreFile.listFiles(); - assertEquals(2, endFileList.length); - filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(checkFilePermissions(endFileList, "r--r--r--")); - assertTrue(checkForSymlinks(endFileList, false)); - endFileList = calDatastoreFile.listFiles(); - assertEquals(1, endFileList.length); - assertEquals("cal-1-1-A-20-results.h5", endFileList[0].getName()); - assertTrue(checkFilePermissions(endFileList, "r--r--r--")); - assertTrue(checkForSymlinks(endFileList, false)); - taskFileList = taskDirFile.listFiles(); - assertEquals(3, taskFileList.length); - filenames = getNamesFromListFiles(taskFileList); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(filenames.contains("pdc-1-1-20-results.h5")); - } - - /** - * Tests the copyToTaskDirectory() method when the objects to be copied are themselves - * directories. - */ - @Test - public void testCopyDirectoriesToTaskDirectory() throws IOException { - - // set up the directories in the datastore - constructDatastoreDirectories(); - - // Construct the DataFileInfo instances - Set datastoreIds = new HashSet<>(); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112N0_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112NO_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H2240632000337112NP_WGS_01")); - - // Perform the copy - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager.copyToTaskDirectory(datastoreIds); - - // check the existence of the copied directories - File[] endFileList = taskDirFile.listFiles(); - assertEquals(4, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList, true); - assertTrue(filenames.contains("EO1H0230312000337112N0_WGS_01")); - assertTrue(filenames.contains("EO1H0230312000337112NO_WGS_01")); - assertTrue(filenames.contains("EO1H2240632000337112NP_WGS_01")); - assertTrue(filenames.contains("st-0")); - - // check that the copied things are, in fact, directories - assertTrue(java.nio.file.Files - .isDirectory(taskDirFile.toPath().resolve("EO1H0230312000337112N0_WGS_01"))); - assertTrue(java.nio.file.Files - .isDirectory(taskDirFile.toPath().resolve("EO1H0230312000337112NO_WGS_01"))); - assertTrue(java.nio.file.Files - .isDirectory(taskDirFile.toPath().resolve("EO1H2240632000337112NP_WGS_01"))); - - // check that the first directory has the intended content - File[] dir1FileList = new File(taskDirFile, "EO1H0230312000337112N0_WGS_01").listFiles(); - assertEquals(3, dir1FileList.length); - filenames = getNamesFromListFiles(dir1FileList, true); - assertTrue(filenames.contains("EO12000337_00CA00C9_r1_WGS_01.L0")); - assertTrue(filenames.contains("EO12000337_00CD00CC_r1_WGS_01.L0")); - assertTrue(filenames.contains("EO12000337_00CF00CE_r1_WGS_01.L0")); - for (File f : dir1FileList) { - assertTrue(f.isFile()); - } - - // check that the third directory is empty - File[] dir3FileList = new File(taskDirFile, "EO1H2240632000337112NP_WGS_01").listFiles(); - assertEquals(0, dir3FileList.length); - - // check that the 2nd directory contains a subdirectory - File[] dir2FileList = new File(taskDirFile, "EO1H0230312000337112NO_WGS_01").listFiles(); - assertEquals(1, dir2FileList.length); - assertTrue(dir2FileList[0].isDirectory()); - assertEquals("next-level-down-subdir", dir2FileList[0].getName()); - - // get the contents of the subdir and check them - File[] dir2SubDirList = dir2FileList[0].listFiles(); - assertEquals(1, dir2SubDirList.length); - assertEquals("next-level-down-content.L0", dir2SubDirList[0].getName()); - assertTrue(dir2SubDirList[0].isFile()); - } - - @Test - public void testSymlinkDirectoriesToTaskDirectory() throws IOException { - - // set up the directories in the datastore - constructDatastoreDirectories(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager(); - - // construct a new file in the external temp directory and a symlink to same in - // the datastore - Path externalFile = Paths.get(externalTempDir, "EO1H0230412000337112N0_WGS_01"); - java.nio.file.Files.createDirectory(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "EO1H0230412000337112N0_WGS_01"), externalFile); - - // Construct the DataFileInfo instances - Set datastoreIds = new HashSet<>(); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112N0_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112NO_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H2240632000337112NP_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230412000337112N0_WGS_01")); - - // Perform the copy - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager.copyToTaskDirectory(datastoreIds); - - // check the existence of the copied directories - File[] endFileList = taskDirFile.listFiles(); - assertEquals(5, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList, true); - assertTrue(filenames.contains("EO1H0230312000337112N0_WGS_01")); - assertTrue(filenames.contains("EO1H0230312000337112NO_WGS_01")); - assertTrue(filenames.contains("EO1H2240632000337112NP_WGS_01")); - assertTrue(filenames.contains("EO1H0230412000337112N0_WGS_01")); - assertTrue(filenames.contains("st-0")); - - // check that the copied things are, in fact, symlinks - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112N0_WGS_01"))); - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112NO_WGS_01"))); - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H2240632000337112NP_WGS_01"))); - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H0230412000337112N0_WGS_01"))); - - // Check that the symlinks are links of the expected files - assertEquals(Paths.get(datastoreRoot, "EO1H0230312000337112N0_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112N0_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H0230312000337112NO_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112NO_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H2240632000337112NP_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H2240632000337112NP_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H0230412000337112N0_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H0230412000337112N0_WGS_01"))); - } - - /** - * Tests the moveToDatastore() method when the objects to be copied are themselves directories. - */ - @Test - public void testMoveDirectoriesToDatastore() throws IOException { - - // set up sub-directories in the task directory - constructTaskDirSubDirectories(); - - // Construct the DataFileInfo instances - Set datastoreIds = new HashSet<>(); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112N0_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112NO_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H2240632000337112NP_WGS_01")); - - // Copy the task dir sub-directories to the datastore - dataFileManager.moveToDatastore(datastoreIds); - - // check the existence of the copied directories - File[] endFileList = new File(datastoreRoot).listFiles(); - assertEquals(3, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList, true); - assertTrue(filenames.contains("EO1H0230312000337112N0_WGS_01")); - assertTrue(filenames.contains("EO1H0230312000337112NO_WGS_01")); - assertTrue(filenames.contains("EO1H2240632000337112NP_WGS_01")); - assertTrue(checkFilePermissions(endFileList, "r-xr-xr-x")); - - Path datastorePath = Paths.get(datastoreRoot); - // check that the copied things are, in fact, directories - assertTrue(java.nio.file.Files - .isDirectory(datastorePath.resolve("EO1H0230312000337112N0_WGS_01"))); - assertTrue(java.nio.file.Files - .isDirectory(datastorePath.resolve("EO1H0230312000337112NO_WGS_01"))); - assertTrue(java.nio.file.Files - .isDirectory(datastorePath.resolve("EO1H2240632000337112NP_WGS_01"))); - - // check that the first directory has the intended content - File[] dir1FileList = new File(datastorePath.toFile(), "EO1H0230312000337112N0_WGS_01") - .listFiles(); - assertEquals(3, dir1FileList.length); - filenames = getNamesFromListFiles(dir1FileList, true); - assertTrue(filenames.contains("EO12000337_00CA00C9_r1_WGS_01.L0")); - assertTrue(filenames.contains("EO12000337_00CD00CC_r1_WGS_01.L0")); - assertTrue(filenames.contains("EO12000337_00CF00CE_r1_WGS_01.L0")); - for (File f : dir1FileList) { - assertTrue(f.isFile()); - } - - // check that the third directory is empty - File[] dir3FileList = new File(datastorePath.toFile(), "EO1H2240632000337112NP_WGS_01") - .listFiles(); - assertEquals(0, dir3FileList.length); - - // check that the 2nd directory contains a subdirectory - File[] dir2FileList = new File(datastorePath.toFile(), "EO1H0230312000337112NO_WGS_01") - .listFiles(); - assertEquals(1, dir2FileList.length); - assertTrue(dir2FileList[0].isDirectory()); - assertEquals("next-level-down-subdir", dir2FileList[0].getName()); - - // get the contents of the subdir and check them - File[] dir2SubDirList = dir2FileList[0].listFiles(); - assertEquals(1, dir2SubDirList.length); - assertEquals("next-level-down-content.L0", dir2SubDirList[0].getName()); - assertTrue(dir2SubDirList[0].isFile()); - } - - /** - * Tests the copyDataFilesByTypeToTaskDirectory() method of DataFileManager. - */ - @Test - public void testCopyDataFilesByTypeToTaskDirectory() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // perform the copy - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - File[] endFileList = taskDirFile.listFiles(); - assertEquals(5, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(checkForSymlinks(fileList, false)); - - // check that the originators were set correctly - Set producerTaskIds = pipelineTask.getProducerTaskIds(); - assertEquals(2, producerTaskIds.size()); - assertTrue(producerTaskIds.contains(PROD_TASK_ID1)); - assertTrue(producerTaskIds.contains(PROD_TASK_ID2)); - } - - @Test - public void testDataFilesForInputsByType() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - TaskConfigurationParameters taskConfig = new TaskConfigurationParameters(); - taskConfig.setReprocess(true); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Get all the files (reprocessing use-case) - Set paths = dataFileManager2.dataFilesForInputs(Paths.get(""), dataFileTypes); - assertEquals(4, paths.size()); - Set filenames = paths.stream() - .map(s -> s.getFileName().toString()) - .collect(Collectors.toSet()); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - - // Now get only the ones that are appropriate for reprocessing - taskConfigurationParameters.setReprocess(false); - Mockito - .when(pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode( - ArgumentMatchers.> any(), - ArgumentMatchers.any(PipelineDefinitionNode.class))) - .thenReturn(Lists.newArrayList(11L, 12L)); - - paths = dataFileManager2.dataFilesForInputs(Paths.get(""), dataFileTypes); - assertEquals(2, paths.size()); - filenames = paths.stream().map(s -> s.getFileName().toString()).collect(Collectors.toSet()); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - } - - /** - * Tests the copyDataFilesByTypeToTaskDirectory method in the case in which symlinks are to be - * employed instead of true copies. - */ - @Test - public void testSymlinkDataFilesByTypeToTaskDirectory() - throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager2(); - - // construct a new file in the external temp directory - Path externalFile = Paths.get(externalTempDir, "pa-001234569-20-results.h5"); - java.nio.file.Files.createFile(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), externalFile); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // perform the copy - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - File[] endFileList = taskDirFile.listFiles(); - assertEquals(6, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-001234569-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(checkForSymlinks(fileList, true)); - - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234567-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-001234567-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234569-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-001234569-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-765432100-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-765432100-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-A-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "cal-1-1-A-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-B-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "cal-1-1-B-20-results.h5"))); - } - - @Test - public void testSymlinkDirectoriesByTypeToTaskDirectory() throws IOException { - - // set up the datastore - constructDatastoreDirectories(); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager2(); - - // construct a new file in the external temp directory and a symlink to same in - // the datastore - Path externalFile = Paths.get(externalTempDir, "EO1H0230412000337112N0_WGS_01"); - java.nio.file.Files.createDirectory(externalFile); - java.nio.file.Files.createSymbolicLink( - Paths.get(datastoreRoot, "EO1H0230412000337112N0_WGS_01"), externalFile); - - Set dataFileTypes = new HashSet<>(); - DataFileTestUtils.initializeDataFileTypeForDirectories(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeForDirectories); - - // Perform the copy - File taskDirFile = new File(taskDir); - File[] fileList = taskDirFile.listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - - // check the existence of the copied directories - File[] endFileList = taskDirFile.listFiles(); - assertEquals(5, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList, true); - assertTrue(filenames.contains("EO1H0230312000337112N0_WGS_01")); - assertTrue(filenames.contains("EO1H0230312000337112NO_WGS_01")); - assertTrue(filenames.contains("EO1H2240632000337112NP_WGS_01")); - assertTrue(filenames.contains("EO1H0230412000337112N0_WGS_01")); - assertTrue(filenames.contains("st-0")); - - // check that the copied things are, in fact, symlinks - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112N0_WGS_01"))); - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112NO_WGS_01"))); - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H2240632000337112NP_WGS_01"))); - assertTrue(java.nio.file.Files - .isSymbolicLink(taskDirFile.toPath().resolve("EO1H0230412000337112N0_WGS_01"))); - - // Check that the symlinks are links of the expected files - assertEquals(Paths.get(datastoreRoot, "EO1H0230312000337112N0_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112N0_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H0230312000337112NO_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H0230312000337112NO_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H2240632000337112NP_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H2240632000337112NP_WGS_01"))); - assertEquals(Paths.get(datastoreRoot, "EO1H0230412000337112N0_WGS_01"), java.nio.file.Files - .readSymbolicLink(taskDirFile.toPath().resolve("EO1H0230412000337112N0_WGS_01"))); - } - - /** - * Tests the deleteDataFilesByTypeFromTaskDirectory() method of DataFileManager. - * - * @throws IOException - * @throws InterruptedException - */ - @Test - public void testDeleteDataFilesByTypeFromTaskDirectory() - throws IOException, InterruptedException { - - // set up the datastore - Set datastoreFilenames = constructDatastoreFiles(); - - // setup the data file types - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Copy the files to the task directory - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - new File(taskDir, "pdc-1-1-20-results.h5").createNewFile(); - - // move to the subtask directory and copy the files to there - dataFileManager2.copyFilesByNameFromTaskDirToWorkingDir(datastoreFilenames); - - // delete the files - dataFileManager2.deleteDataFilesByTypeFromTaskDirectory(dataFileTypes); - - // The PDC file should still be present in the task directory - File[] listFiles = new File(taskDir).listFiles(); - Set filesInTaskDir = getNamesFromListFiles(listFiles); - assertEquals(1, filesInTaskDir.size()); - assertTrue(filesInTaskDir.contains("pdc-1-1-20-results.h5")); - assertTrue(checkForSymlinks(listFiles, false)); - - // all 5 files should still be present in the subtask directory - listFiles = new File(subtaskDir).listFiles(); - filesInTaskDir = getNamesFromListFiles(listFiles); - assertEquals(5, filesInTaskDir.size()); - assertTrue(filesInTaskDir.contains("pdc-1-1-20-results.h5")); - assertTrue(filesInTaskDir.contains("pa-001234567-20-results.h5")); - assertTrue(filesInTaskDir.contains("pa-765432100-20-results.h5")); - assertTrue(filesInTaskDir.contains("cal-1-1-A-20-results.h5")); - assertTrue(filesInTaskDir.contains("cal-1-1-B-20-results.h5")); - assertTrue(checkForSymlinks(listFiles, false)); - } - - @Test - public void testDeleteSymlinksByTypeFromTaskDirectory() - throws IOException, InterruptedException { - - // set up the datastore - Set datastoreFilenames = constructDatastoreFiles(); - datastoreFilenames.remove("pdc-1-1-20-results.h5"); - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager2(); - - // setup the data file types - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Copy the files to the task directory - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - - dataFileManager2.copyFilesByNameFromTaskDirToWorkingDir(datastoreFilenames); - - // delete the files - dataFileManager2.deleteDataFilesByTypeFromTaskDirectory(dataFileTypes); - - // None of the files should still be present in the task directory - Set filesInTaskDir = getNamesFromListFiles(new File(taskDir).listFiles()); - assertEquals(0, filesInTaskDir.size()); - - // all 5 files should still be present in the subtask directory, as symlinks - File[] listFiles = new File(subtaskDir).listFiles(); - filesInTaskDir = getNamesFromListFiles(listFiles); - assertEquals(4, filesInTaskDir.size()); - assertTrue(filesInTaskDir.contains("pa-001234567-20-results.h5")); - assertTrue(filesInTaskDir.contains("pa-765432100-20-results.h5")); - assertTrue(filesInTaskDir.contains("cal-1-1-A-20-results.h5")); - assertTrue(filesInTaskDir.contains("cal-1-1-B-20-results.h5")); - assertTrue(checkForSymlinks(listFiles, true)); - - // The files should be symlinks of the datastore files - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234567-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-001234567-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-765432100-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-765432100-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-A-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(subtaskDir, "cal-1-1-A-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-B-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(subtaskDir, "cal-1-1-B-20-results.h5"))); - } - - /** - * Tests the moveDataFilesByTypeToDatastore() method of DataFileManager. - */ - @Test - public void testMoveDataFilesByTypeToDatastore() throws IOException { - - // set up sub-directories in the task directory - constructTaskDirFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - File paDatastoreFile = new File(datastoreRoot, "pa/20"); - File calDatastoreFile = new File(datastoreRoot, "cal/20"); - File taskDirFile = new File(taskDir); - paDatastoreFile.mkdirs(); - calDatastoreFile.mkdirs(); - assertEquals(0, paDatastoreFile.listFiles().length); - File[] taskFileList = taskDirFile.listFiles(); - assertEquals(6, taskFileList.length); - Set filenames = getNamesFromListFiles(taskFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(filenames.contains("pdc-1-1-20-results.h5")); - - // perform the move - dataFileManager2.moveDataFilesByTypeToDatastore(dataFileTypes); - - // check both moved and unmoved files - File[] endFileList = paDatastoreFile.listFiles(); - assertEquals(2, endFileList.length); - filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(checkFilePermissions(endFileList, "r--r--r--")); - endFileList = calDatastoreFile.listFiles(); - assertEquals(2, endFileList.length); - filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - assertTrue(checkFilePermissions(endFileList, "r--r--r--")); - taskFileList = taskDirFile.listFiles(); - assertEquals(2, taskFileList.length); - filenames = getNamesFromListFiles(taskFileList); - assertTrue(filenames.contains("pdc-1-1-20-results.h5")); - } - - /** - * Tests the deleteFromTaskDirectory() method in the case in which the objects to be deleted are - * directories, including non-empty directories. - */ - @Test - public void testDeleteDirectoriesFromTaskDirectory() throws IOException { - - // set up sub-directories in the task directory - constructTaskDirSubDirectories(); - - // Construct the DataFileInfo instances - Set datastoreIds = new HashSet<>(); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112N0_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H0230312000337112NO_WGS_01")); - datastoreIds.add(new DataFileInfoSampleForDirs("EO1H2240632000337112NP_WGS_01")); - - // delete the directories - dataFileManager.deleteFromTaskDirectory(datastoreIds); - - // check that they are really gone - File[] endFileList = new File(taskDir).listFiles(); - assertEquals(1, endFileList.length); - assertEquals("st-0", endFileList[0].getName()); - } - - @Test - public void testMoveSymlinkedFileToDatastore() throws IOException { - - // Enable symlinking - System.setProperty(USE_SYMLINKS.property(), "true"); - initializeDataFileManager2(); - - new File(subtaskDir, "pa-001234567-20-results.h5").createNewFile(); - new File(subtaskDir, "pa-765432100-20-results.h5").createNewFile(); - new File(subtaskDir, "cal-1-1-A-20-results.h5").createNewFile(); - new File(subtaskDir, "cal-1-1-B-20-results.h5").createNewFile(); - - // Set up the data file types - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Set up the datastore directories - File paDatastoreFile = new File(datastoreRoot, "pa/20"); - File calDatastoreFile = new File(datastoreRoot, "cal/20"); - paDatastoreFile.mkdirs(); - calDatastoreFile.mkdirs(); - - // Copy the files to the task directory - dataFileManager2.copyDataFilesByTypeFromWorkingDirToTaskDir(dataFileTypes); - - // This should result in 4 files in the task directory, all of which are - // symlinks of the files in the working directory - File[] fileList = new File(taskDir).listFiles(); - assertTrue(checkForSymlinks(fileList, true)); - assertEquals(Paths.get(subtaskDir, "pa-001234567-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-001234567-20-results.h5"))); - assertEquals(Paths.get(subtaskDir, "pa-765432100-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "pa-765432100-20-results.h5"))); - assertEquals(Paths.get(subtaskDir, "cal-1-1-A-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "cal-1-1-A-20-results.h5"))); - assertEquals(Paths.get(subtaskDir, "cal-1-1-B-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(taskDir, "cal-1-1-B-20-results.h5"))); - - // now copy the files back to the datastore - dataFileManager2.moveDataFilesByTypeToDatastore(dataFileTypes); - - // None of these files should be present in the task directory anymore - fileList = new File(taskDir).listFiles(); - assertEquals(1, fileList.length); - assertEquals("st-0", fileList[0].getName()); - - // All of the files should be present in the subtask directory, but they should be symlinks - fileList = new File(subtaskDir).listFiles(); - assertTrue(checkForSymlinks(fileList, true)); - - // The files should be symlinks of the files in the datastore, which should themselves be - // real files - File[] paFiles = Paths.get(datastoreRoot, "pa", "20").toFile().listFiles(); - assertTrue(checkForSymlinks(paFiles, false)); - File[] calFiles = Paths.get(datastoreRoot, "cal", "20").toFile().listFiles(); - assertTrue(checkForSymlinks(calFiles, false)); - - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-001234567-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-001234567-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "pa", "20", "pa-765432100-20-results.h5"), - java.nio.file.Files - .readSymbolicLink(Paths.get(subtaskDir, "pa-765432100-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-A-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(subtaskDir, "cal-1-1-A-20-results.h5"))); - assertEquals(Paths.get(datastoreRoot, "cal", "20", "cal-1-1-B-20-results.h5"), - java.nio.file.Files.readSymbolicLink(Paths.get(subtaskDir, "cal-1-1-B-20-results.h5"))); - } - - @Test - public void testDatastoreFilesInCompletedSubtasks() throws IOException { - - // setup the task directory - constructTaskDirFiles(); - - // add a second subtask directory - File subtaskDir2 = new File(taskDir, "st-1"); - subtaskDir2.mkdirs(); - - // move one CAL file and one PA file to each directory - File pa1 = new File(taskDir, "pa-001234567-20-results.h5"); - File pa2 = new File(taskDir, "pa-765432100-20-results.h5"); - File cal1 = new File(taskDir, "cal-1-1-A-20-results.h5"); - File cal2 = new File(taskDir, "cal-1-1-B-20-results.h5"); - - Files.move(pa1.toPath(), new File(subtaskDir, pa1.getName()).toPath()); - Files.move(pa2.toPath(), new File(subtaskDir2, pa2.getName()).toPath()); - Files.move(cal1.toPath(), new File(subtaskDir, cal1.getName()).toPath()); - Files.move(cal2.toPath(), new File(subtaskDir2, cal2.getName()).toPath()); - - // mark the first subtask directory as completed - AlgorithmStateFiles asf = new AlgorithmStateFiles(new File(subtaskDir)); - asf.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); -// AlgorithmResultsState.setHasResults(new File(subtaskDir)); - - // Create and persist a TaskConfigurationManager instance - TaskConfigurationManager tcm = new TaskConfigurationManager(new File(taskDir)); - tcm.addFilesForSubtask(new TreeSet<>()); - tcm.addFilesForSubtask(new TreeSet<>()); - tcm.persist(); - - // Get the flavors of input files - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // Get the files from the completed subtasks with results (i.e., - // at this point, none of the subtasks meet both conditions) - Set filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithResults(dataFileTypes); - assertEquals(0, filesFromCompletedSubtasks.size()); - - // Now for completed subtasks without results (the first subtask - // dir) - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithoutResults(dataFileTypes); - assertEquals(2, filesFromCompletedSubtasks.size()); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample1 - .datastoreFileNameFromTaskDirFileName(pa1.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample2 - .datastoreFileNameFromTaskDirFileName(cal1.getName()))); - - // Set the first subtask directory to "has results" - new AlgorithmStateFiles(new File(subtaskDir)).setResultsFlag(); - - // The first subtask directory's files should come up when testing for - // completed subtasks with results; nothing should come up when testing for - // completed subtasks without results. - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithResults(dataFileTypes); - assertEquals(2, filesFromCompletedSubtasks.size()); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample1 - .datastoreFileNameFromTaskDirFileName(pa1.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample2 - .datastoreFileNameFromTaskDirFileName(cal1.getName()))); - - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithoutResults(dataFileTypes); - assertEquals(0, filesFromCompletedSubtasks.size()); - - // Now mark the 2nd subtask directory as completed - asf = new AlgorithmStateFiles(subtaskDir2); - asf.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - - // The first directory should have all the files for complete with results - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithResults(dataFileTypes); - assertEquals(2, filesFromCompletedSubtasks.size()); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample1 - .datastoreFileNameFromTaskDirFileName(pa1.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample2 - .datastoreFileNameFromTaskDirFileName(cal1.getName()))); - - // The second directory should have all the files for complete without results - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithoutResults(dataFileTypes); - assertEquals(2, filesFromCompletedSubtasks.size()); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample1 - .datastoreFileNameFromTaskDirFileName(pa2.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample2 - .datastoreFileNameFromTaskDirFileName(cal2.getName()))); - - // When the 2nd directory is also set to "has results," both dirs should show up - // in the search for completed with results... - new AlgorithmStateFiles(subtaskDir2).setResultsFlag(); - - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithResults(dataFileTypes); - assertEquals(4, filesFromCompletedSubtasks.size()); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample1 - .datastoreFileNameFromTaskDirFileName(pa1.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample2 - .datastoreFileNameFromTaskDirFileName(cal1.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample1 - .datastoreFileNameFromTaskDirFileName(pa2.getName()))); - assertTrue(filesFromCompletedSubtasks.contains(DataFileTestUtils.dataFileTypeSample2 - .datastoreFileNameFromTaskDirFileName(cal2.getName()))); - - // The complete without results search should return nothing - filesFromCompletedSubtasks = dataFileManager2 - .datastoreFilesInCompletedSubtasksWithoutResults(dataFileTypes); - assertEquals(0, filesFromCompletedSubtasks.size()); - } - - @Test - public void testFilesInCompletedSubtasks() throws IOException { - - // setup the task directory - constructTaskDirFiles(); - - // add a second subtask directory - File subtaskDir2 = new File(taskDir, "st-1"); - subtaskDir2.mkdirs(); - - // move one CAL file and one PA file to each directory - File pa1 = new File(taskDir, "pa-001234567-20-results.h5"); - File pa2 = new File(taskDir, "pa-765432100-20-results.h5"); - File cal1 = new File(taskDir, "cal-1-1-A-20-results.h5"); - File cal2 = new File(taskDir, "cal-1-1-B-20-results.h5"); - - Files.move(pa1.toPath(), new File(subtaskDir, pa1.getName()).toPath()); - Files.move(pa2.toPath(), new File(subtaskDir2, pa2.getName()).toPath()); - Files.move(cal1.toPath(), new File(subtaskDir, cal1.getName()).toPath()); - Files.move(cal2.toPath(), new File(subtaskDir2, cal2.getName()).toPath()); - - // mark the first subtask directory as completed - AlgorithmStateFiles asf = new AlgorithmStateFiles(new File(subtaskDir)); - asf.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - new AlgorithmStateFiles(new File(subtaskDir)).setResultsFlag(); - - // Create and persist a TaskConfigurationManager instance - TaskConfigurationManager tcm = new TaskConfigurationManager(new File(taskDir)); - tcm.addFilesForSubtask(new TreeSet<>()); - tcm.addFilesForSubtask(new TreeSet<>()); - tcm.persist(); - - // construct the set of DatastoreId subclasses - Set> datastoreIdClasses = new HashSet<>(); - datastoreIdClasses.add(DataFileInfoSample1.class); - datastoreIdClasses.add(DataFileInfoSample2.class); - - initializeDataFileManager(); - Set filenames = dataFileManager - .filesInCompletedSubtasksWithResults(datastoreIdClasses); - assertEquals(2, filenames.size()); - assertTrue( - filenames.contains(Paths.get(datastoreRoot, "pa", "20", pa1.getName()).toString())); - assertTrue( - filenames.contains(Paths.get(datastoreRoot, "cal", "20", cal1.getName()).toString())); - - // Now mark the second subtask as completed - asf = new AlgorithmStateFiles(subtaskDir2); - asf.updateCurrentState(AlgorithmStateFiles.SubtaskState.COMPLETE); - new AlgorithmStateFiles(subtaskDir2).setResultsFlag(); - - filenames = dataFileManager.filesInCompletedSubtasksWithResults(datastoreIdClasses); - assertEquals(4, filenames.size()); - assertTrue( - filenames.contains(Paths.get(datastoreRoot, "pa", "20", pa1.getName()).toString())); - assertTrue( - filenames.contains(Paths.get(datastoreRoot, "cal", "20", cal1.getName()).toString())); - assertTrue( - filenames.contains(Paths.get(datastoreRoot, "pa", "20", pa2.getName()).toString())); - assertTrue( - filenames.contains(Paths.get(datastoreRoot, "cal", "20", cal2.getName()).toString())); - } - - @Test - public void testStandardReprocessing() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - taskConfigurationParameters.setReprocess(true); - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - File[] endFileList = new File(taskDir).listFiles(); - assertEquals(5, endFileList.length); - Set filenames = getNamesFromListFiles(endFileList); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-A-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - } - - @Test - public void testKeepUpProcessing() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // set up the pipeline task CRUD - Mockito - .when(pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode( - ArgumentMatchers.> any(), - ArgumentMatchers.any(PipelineDefinitionNode.class))) - .thenReturn(Lists.newArrayList(11L, 12L)); - - taskConfigurationParameters.setReprocess(false); - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - File[] endFileList = new File(taskDir).listFiles(); - Set filenames = getNamesFromListFiles(endFileList); - assertEquals(2, filenames.size()); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - assertTrue(filenames.contains("cal-1-1-B-20-results.h5")); - } - - @Test - public void testReprocessingWithExcludes() throws IOException, InterruptedException { - - // set up the datastore - constructDatastoreFiles(); - - Set dataFileTypes = new HashSet<>(); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypes.add(DataFileTestUtils.dataFileTypeSample2); - - // set up the pipeline task CRUD - Mockito - .when(pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode( - ArgumentMatchers.> any(), - ArgumentMatchers.any(PipelineDefinitionNode.class))) - .thenReturn(Lists.newArrayList(10L, 11L, 12L)); - - taskConfigurationParameters.setReprocessingTasksExclude(new long[] { 10L }); - dataFileManager2.copyDataFilesByTypeToTaskDirectory(Paths.get(""), dataFileTypes); - File[] endFileList = new File(taskDir).listFiles(); - Set filenames = getNamesFromListFiles(endFileList); - assertEquals(1, filenames.size()); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - } - - @Test - public void testWorkingDirHasFilesOfTypes() throws IOException { - - // Put PA data files into the subtask directory - File sample1 = new File(subtaskDir, "pa-001234567-20-results.h5"); - File sample2 = new File(subtaskDir, "pa-765432100-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - - // Files of the type of the DataFileTypeSample1 should be found. - assertTrue(dataFileManager2.workingDirHasFilesOfTypes( - Collections.singleton(DataFileTestUtils.dataFileTypeSample1))); - - // Files of the type of the DataFileTypeSample2 should not be found. - assertFalse(dataFileManager2.workingDirHasFilesOfTypes( - Collections.singleton(DataFileTestUtils.dataFileTypeSample2))); - - Set dataFileTypeSet = new HashSet<>(); - dataFileTypeSet.add(DataFileTestUtils.dataFileTypeSample1); - dataFileTypeSet.add(DataFileTestUtils.dataFileTypeSample2); - - // When searching for both data types, a result of true should be - // returned. - assertTrue(dataFileManager2.workingDirHasFilesOfTypes(dataFileTypeSet)); - } - - private static List datastoreProducerConsumers() { - - List dpcs = new ArrayList<>(); - - DatastoreProducerConsumer dpc = new DatastoreProducerConsumer(1L, - "pa/20/pa-001234567-20-results.h5", DatastoreProducerConsumer.DataReceiptFileType.DATA); - dpc.setConsumers(Sets.newHashSet(10L, 11L, 12L)); - dpcs.add(dpc); - - dpc = new DatastoreProducerConsumer(1L, "pa/20/pa-765432100-20-results.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA); - dpc.setConsumers(Sets.newHashSet()); - dpcs.add(dpc); - - // Set up the 1-1-A-20 data file such that it ran in task 11 but produced no - // results. This should prevent it from being included in reprocessing because - // the pipeline module that's doing the reprocessing is the same as the module - // for tasks 11 and 12. - dpc = new DatastoreProducerConsumer(1L, "cal/20/cal-1-1-A-20-results.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA); - dpc.setConsumers(Sets.newHashSet(10L, -11L)); - dpcs.add(dpc); - - dpc = new DatastoreProducerConsumer(1L, "cal/20/cal-1-1-B-20-results.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA); - dpc.setConsumers(Sets.newHashSet(10L)); - dpcs.add(dpc); - - return dpcs; - } - - private Set constructTaskDirFiles() throws IOException { - - File taskDir = new File(this.taskDir); - Set filenames = new HashSet<>(); - // create a couple of files in the DatastoreIdSample1 pattern - File sample1 = new File(taskDir, "pa-001234567-20-results.h5"); - File sample2 = new File(taskDir, "pa-765432100-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - filenames.add(sample1.getName()); - filenames.add(sample2.getName()); - - // create a couple of files in the DatastoreIdSample2 pattern - sample1 = new File(taskDir, "cal-1-1-A-20-results.h5"); - sample2 = new File(taskDir, "cal-1-1-B-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - filenames.add(sample1.getName()); - filenames.add(sample2.getName()); - - // create a file that matches neither pattern - sample1 = new File(taskDir, "pdc-1-1-20-results.h5"); - sample1.createNewFile(); - filenames.add(sample1.getName()); - return filenames; - } - - private Set constructDatastoreFiles() throws IOException, InterruptedException { - - Set datastoreFilenames = new HashSet<>(); - - // create some directories in the datastore - File paDir = new File(datastoreRoot, "pa/20"); - File calDir = new File(datastoreRoot, "cal/20"); - File pdcDir = new File(datastoreRoot, "pdc/20"); - paDir.mkdirs(); - calDir.mkdirs(); - pdcDir.mkdirs(); - - // create the files - File sample1 = new File(paDir, "pa-001234567-20-results.h5"); - File sample2 = new File(paDir, "pa-765432100-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - datastoreFilenames.add(sample1.getName()); - datastoreFilenames.add(sample2.getName()); - - sample1 = new File(calDir, "cal-1-1-A-20-results.h5"); - sample2 = new File(calDir, "cal-1-1-B-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - datastoreFilenames.add(sample1.getName()); - datastoreFilenames.add(sample2.getName()); - - sample1 = new File(pdcDir, "pdc-1-1-20-results.h5"); - sample1.createNewFile(); - boolean dmy = true; - dmy = !dmy; - datastoreFilenames.add(sample1.getName()); - return datastoreFilenames; - } - - private Set getNamesFromDatastoreIds(Set datastoreSet) { - Set names = new HashSet<>(); - for (T d : datastoreSet) { - names.add(d.toString()); - } - return names; - } - - private Set getNamesFromListFiles(File[] files) { - return getNamesFromListFiles(files, false); - } - - private Set getNamesFromListFiles(File[] files, boolean acceptDirs) { - Set nameSet = new HashSet<>(); - for (File f : files) { - if (!f.isDirectory() || acceptDirs) { - nameSet.add(f.getName()); - } - } - return nameSet; - } - - private Set getNamesFromPaths(Set paths) { - Set nameSet = new HashSet<>(); - for (Path p : paths) { - nameSet.add(p.getFileName().toString()); - } - return nameSet; - } - - private boolean checkFilePermissions(File[] files, String permissions) throws IOException { - Set intendedPermissions = PosixFilePermissions.fromString(permissions); - boolean permissionsCorrect = true; - for (File file : files) { - Set actualPermissions = java.nio.file.Files - .getPosixFilePermissions(file.toPath()); - permissionsCorrect = permissionsCorrect - && actualPermissions.size() == intendedPermissions.size(); - for (PosixFilePermission permission : intendedPermissions) { - permissionsCorrect = permissionsCorrect && actualPermissions.contains(permission); - } - } - return permissionsCorrect; - } - - private boolean checkForSymlinks(File[] files, boolean symlinkExpected) { - boolean allFilesMatchExpected = true; - for (File file : files) { - if (!file.isDirectory()) { - allFilesMatchExpected = allFilesMatchExpected - && java.nio.file.Files.isSymbolicLink(file.toPath()) == symlinkExpected; - } - } - return allFilesMatchExpected; - } - - private void initializeDataFileManager() { - dataFileManager = new DataFileManager(new DatastorePathLocatorSample(), pipelineTask, - Paths.get(taskDir)); - dataFileManager = Mockito.spy(dataFileManager); - Mockito.when(dataFileManager.datastoreProducerConsumerCrud()) - .thenReturn(datastoreProducerConsumerCrud); - Mockito.when(dataFileManager.pipelineTaskCrud()).thenReturn(pipelineTaskCrud); - } - - private void initializeDataFileManager2() { - dataFileManager2 = new DataFileManager(Paths.get(datastoreRoot), Paths.get(taskDir), - pipelineTask); - dataFileManager2 = Mockito.spy(dataFileManager2); - Mockito.when(dataFileManager2.datastoreProducerConsumerCrud()) - .thenReturn(datastoreProducerConsumerCrud); - Mockito.when(dataFileManager2.pipelineTaskCrud()).thenReturn(pipelineTaskCrud); - } - - /** - * Constructs directories in the datastore that follow the Hyperion naming convention, with - * files inside them. - * - * @throws IOException - */ - private void constructDatastoreDirectories() throws IOException { - - // Create the directories - File dir1 = new File(datastoreRoot, "EO1H0230312000337112N0_WGS_01"); - File dir2 = new File(datastoreRoot, "EO1H0230312000337112NO_WGS_01"); - File dir3 = new File(datastoreRoot, "EO1H2240632000337112NP_WGS_01"); - dir1.mkdirs(); - dir2.mkdirs(); - dir3.mkdirs(); - - // create contents in 2 out of 3 directories - File sample1 = new File(dir1, "EO12000337_00CA00C9_r1_WGS_01.L0"); - File sample2 = new File(dir1, "EO12000337_00CD00CC_r1_WGS_01.L0"); - File sample3 = new File(dir1, "EO12000337_00CF00CE_r1_WGS_01.L0"); - sample1.createNewFile(); - sample2.createNewFile(); - sample3.createNewFile(); - - // In this directory, create a subdirectory as well - File dir4 = new File(dir2, "next-level-down-subdir"); - dir4.mkdirs(); - File sample4 = new File(dir4, "next-level-down-content.L0"); - sample4.createNewFile(); - } - - private Set constructDatastoreIds() { - DataFileInfoSample1 pa1 = new DataFileInfoSample1("pa-001234567-20-results.h5"); - DataFileInfoSample1 pa2 = new DataFileInfoSample1("pa-765432100-20-results.h5"); - Set datastoreIds = new HashSet<>(); - datastoreIds.add(pa1); - datastoreIds.add(pa2); - DataFileInfoSample2 cal1 = new DataFileInfoSample2("cal-1-1-A-20-results.h5"); - datastoreIds.add(cal1); - return datastoreIds; - } - - private void constructTaskDirSubDirectories() throws IOException { - - // Create the directories - File dir1 = new File(taskDir, "EO1H0230312000337112N0_WGS_01"); - File dir2 = new File(taskDir, "EO1H0230312000337112NO_WGS_01"); - File dir3 = new File(taskDir, "EO1H2240632000337112NP_WGS_01"); - dir1.mkdirs(); - dir2.mkdirs(); - dir3.mkdirs(); - - // create contents in 2 out of 3 directories - File sample1 = new File(dir1, "EO12000337_00CA00C9_r1_WGS_01.L0"); - File sample2 = new File(dir1, "EO12000337_00CD00CC_r1_WGS_01.L0"); - File sample3 = new File(dir1, "EO12000337_00CF00CE_r1_WGS_01.L0"); - sample1.createNewFile(); - sample2.createNewFile(); - sample3.createNewFile(); - - // In this directory, create a subdirectory as well - File dir4 = new File(dir2, "next-level-down-subdir"); - dir4.mkdirs(); - File sample4 = new File(dir4, "next-level-down-content.L0"); - sample4.createNewFile(); - } - - /** - * Provides a subclass of {@link DatastoreProducerConsumerCrud} for use in the unit tests of - * different processing modes. Necessary because it was far from obvious how to properly mock - * the behavior of the class in question via Mockito. - * - * @author PT - */ - private static class ProducerConsumerCrud extends DatastoreProducerConsumerCrud { - - @Override - public List retrieveByFilename(Set datafiles) { - - List returns = new ArrayList<>(); - List dpcs = datastoreProducerConsumers(); - for (DatastoreProducerConsumer dpc : dpcs) { - if (datafiles.contains(Paths.get(dpc.getFilename()))) { - returns.add(dpc); - } - } - return returns; - } - - @Override - public Set retrieveProducers(Set paths) { - return Sets.newHashSet(PROD_TASK_ID1, PROD_TASK_ID2); - } - - @Override - public void createOrUpdateProducer(PipelineTask pipelineTask, - Collection datastoreFiles, DatastoreProducerConsumer.DataReceiptFileType type) { - } - } -} diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java b/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java index 6182fb1..8da7573 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DataFileTestUtils.java @@ -1,19 +1,13 @@ package gov.nasa.ziggy.data.management; import java.nio.file.Path; -import java.util.Collections; -import java.util.HashMap; import java.util.HashSet; -import java.util.Map; import java.util.Set; -import java.util.regex.Pattern; -import gov.nasa.ziggy.module.PipelineInputs; -import gov.nasa.ziggy.module.PipelineOutputs; -import gov.nasa.ziggy.module.PipelineResults; -import gov.nasa.ziggy.module.TaskConfigurationManager; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.module.DatastoreDirectoryPipelineInputs; +import gov.nasa.ziggy.module.DatastoreDirectoryPipelineOutputs; +import gov.nasa.ziggy.module.SubtaskInformation; +import gov.nasa.ziggy.module.TaskConfiguration; /** * Test utilities for the data management package. In the main this is class definitions that the @@ -23,157 +17,15 @@ */ public class DataFileTestUtils { - /** - * Subclass of DataFileInfo class used to exercise the class features unit tests. It accepts - * files with the name pattern: "pa-<9-digit-number>--results.h5". - * - * @author PT - */ - public static class DataFileInfoSample1 extends DataFileInfo { - - private static final Pattern PATTERN = Pattern.compile("pa-\\d{9}-\\d+-results.h5"); - - public DataFileInfoSample1() { - } - - public DataFileInfoSample1(Path file) { - super(file); - } - - public DataFileInfoSample1(String name) { - super(name); - } - - /** - * Provides a Pattern that expects a String with a form sort of like: - * "pa-001234567-10-results.h5", where the first set of numbers is exactly 9 digits, but the - * second set can be any length. - */ - @Override - protected Pattern getPattern() { - return PATTERN; - } - } - - /** - * Subclass of DataFileInfo class used to exercise features in unit tests. It accepts files with - * the name pattern "cal-#-#-L--results.h5", where "L" is a capital letter from the set - * "ABCD". - * - * @author PT - */ - static class DataFileInfoSample2 extends DataFileInfo { - - private static final Pattern PATTERN = Pattern - .compile("cal-\\d{1}-\\d{1}-[ABCD]-\\d+-results.h5"); - - public DataFileInfoSample2() { - } - - public DataFileInfoSample2(Path file) { - super(file); - } - - public DataFileInfoSample2(String name) { - super(name); - } - - /** - * Provides a Pattern that expects a string with a form sort of like: - * "cal-1-1-A-20-results.h5". The first 2 numbers are 1 digit exactly, the letter is one of - * "ABCD", the final number can be any length. - */ - @Override - protected Pattern getPattern() { - return PATTERN; - } - } - - /** - * DataFileInfo class for testing full-directory copying. - * - * @author PT - */ - public static class DataFileInfoSampleForDirs extends DataFileInfo { - - public DataFileInfoSampleForDirs() { - } - - public DataFileInfoSampleForDirs(String name) { - super(name); - } - - public DataFileInfoSampleForDirs(Path name) { - super(name); - } - - // Use the horrendous directory pattern used by Hyperion L0 data directories - private static Pattern PATTERN = Pattern - .compile("EO1H([0-9]{6})([0-9]{4})([0-9]{3})([A-Z0-9]{5})_([A-Z]{3})_([0-9]{2})"); - - @Override - protected Pattern getPattern() { - return PATTERN; - } - } - - /** - * PipelineResults subclass for test purposes. - * - * @author PT - */ - public static class PipelineResultsSample1 extends PipelineResults { - - private int value; - - public int getValue() { - return value; - } - - public void setValue(int value) { - this.value = value; - } - } - - /** - * PipelineResults subclass for test purposes. - * - * @author PT - */ - public static class PipelineResultsSample2 extends PipelineResults { - - private float fvalue; - - public float getFvalue() { - return fvalue; - } - - public void setFvalue(float fvalue) { - this.fvalue = fvalue; - } - } - - public static class PipelineInputsSample extends PipelineInputs { + public static class PipelineInputsSample extends DatastoreDirectoryPipelineInputs { private double dvalue; - /** - * For test purposes, we will make the DatastoreIdSample1 class the only one that is - * required to populate PipelineInputsSample - */ - @Override - public Set> requiredDataFileInfoClasses() { - Set> requiredClasses = new HashSet<>(); - requiredClasses.add(DataFileInfoSample1.class); - return requiredClasses; - } - /** * Since the populateSubTaskInputs() method can do anything, we'll just have it set the * dvalue */ - @Override - public void populateSubTaskInputs() { + public void populateSubtaskInputs() { dvalue = 105.3; } @@ -182,123 +34,26 @@ public double getDvalue() { } @Override - public DatastorePathLocator datastorePathLocator(PipelineTask pipelineTask) { - return null; - } - - @Override - public void copyDatastoreFilesToTaskDirectory( - TaskConfigurationManager taskConfigurationManager, PipelineTask pipelineTask, + public void copyDatastoreFilesToTaskDirectory(TaskConfiguration taskConfigurationManager, Path taskDirectory) { } @Override - public Set findDatastoreFilesForInputs(PipelineTask pipelineTask) { - return Collections.emptySet(); + public SubtaskInformation subtaskInformation() { + return null; } } - public static class PipelineOutputsSample1 extends PipelineOutputs { - - private int[] ivalues; - - @Override - public void populateTaskResults() { - /** - * Since the populateTaskResults() value can do anything, we'll use it to set the - * ivalues - */ - ivalues = new int[] { 27, -9, 5 }; - } - - public int[] getIvalues() { - return ivalues; - } - - public void setIvalues(int[] ivalues) { - this.ivalues = ivalues; - } - - @Override - public Map pipelineResults() { - Map map = new HashMap<>(); - - // all the results files will use the DataFileInfoSample1 class and - // will be of the PipelineResultsSample1 class - - int i = 0; - for (int f : getIvalues()) { - String fname = String.format("pa-001234567-%d-results.h5", i); - DataFileInfoSample1 d = new DataFileInfoSample1(fname); - PipelineResultsSample1 p = new PipelineResultsSample1(); - p.setValue(f); - map.put(d, p); - i++; - } - - return map; - } + public static class PipelineOutputsSample1 extends DatastoreDirectoryPipelineOutputs { @Override - public void updateInputFileConsumers(PipelineInputs pipelineInputs, - PipelineTask pipelineTask, Path taskDirectory) { + public Set copyTaskFilesToDatastore() { + return new HashSet<>(); } @Override - protected boolean subtaskProducedResults() { + public boolean subtaskProducedOutputs() { return true; } } - - /** - * DatastorePathLocator implementation for test purposes. For instances of DatastoreInfoSample1, - * it returns the combination of the datastore root and the pa- as the path for the - * file; for instance of DatastoreInfoSample2, it returns the datastore root and the cal-#-#-L - * as the path for the file. - * - * @author PT - */ - public static class DatastorePathLocatorSample implements DatastorePathLocator { - - public DatastorePathLocatorSample() { - } - - @Override - public Path datastorePath(DataFileInfo dataFileInfo) { - Path datastoreRoot = DirectoryProperties.datastoreRootDir(); - Path p = null; - String s = dataFileInfo.getName().toString(); - if (dataFileInfo instanceof DataFileInfoSample1) { - p = datastoreRoot.resolve("pa").resolve("20").resolve(s); - } else if (dataFileInfo instanceof DataFileInfoSample2) { - p = datastoreRoot.resolve("cal").resolve("20").resolve(s); - } else if (dataFileInfo instanceof DataFileInfoSampleForDirs) { - p = datastoreRoot.resolve(s); - } - return p; - } - } - - public static final DataFileType dataFileTypeSample1 = new DataFileType(); - public static final DataFileType dataFileTypeSample2 = new DataFileType(); - - public static void initializeDataFileTypeSamples() { - dataFileTypeSample1.setName("pa"); - dataFileTypeSample1.setFileNameRegexForTaskDir("pa-([0-9]{9})-([0-9]{2})-results.h5"); - dataFileTypeSample1.setFileNameWithSubstitutionsForDatastore("pa/$2/pa-$1-$2-results.h5"); - dataFileTypeSample2.setName("cal"); - dataFileTypeSample2 - .setFileNameRegexForTaskDir("cal-([1-4])-([1-4])-([ABCD])-([0-9]{2})-results.h5"); - dataFileTypeSample2 - .setFileNameWithSubstitutionsForDatastore("cal/$4/cal-$1-$2-$3-$4-results.h5"); - } - - public static final DataFileType dataFileTypeForDirectories = new DataFileType(); - - public static void initializeDataFileTypeForDirectories() { - dataFileTypeForDirectories.setName("Hyperion L0"); - dataFileTypeForDirectories.setFileNameRegexForTaskDir( - "EO1H([0-9]{6})([0-9]{4})([0-9]{2})([0-9]{1})([A-Z0-9]{5})_([A-Z]{3})_([0-9]{2})"); - dataFileTypeForDirectories.setFileNameWithSubstitutionsForDatastore("EO1H$1$2$3$4$5_$6_$7"); - } } diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataFileTypeImporterTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataFileTypeImporterTest.java deleted file mode 100644 index 3947b6c..0000000 --- a/src/test/java/gov/nasa/ziggy/data/management/DataFileTypeImporterTest.java +++ /dev/null @@ -1,145 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static gov.nasa.ziggy.ZiggyUnitTestUtils.TEST_DATA; -import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; -import static org.junit.Assert.assertEquals; - -import java.nio.file.Path; - -import org.junit.Rule; -import org.junit.Test; -import org.mockito.ArgumentMatchers; -import org.mockito.Mockito; - -import com.google.common.collect.ImmutableList; - -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.pipeline.definition.ModelType; -import gov.nasa.ziggy.pipeline.definition.crud.DataFileTypeCrud; -import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; -import gov.nasa.ziggy.services.config.DirectoryProperties; -import jakarta.xml.bind.JAXBException; - -/** - * Unit test class for DataFileTypeImporter. - * - * @author PT - */ -public class DataFileTypeImporterTest { - - private static final Path DATASTORE = TEST_DATA.resolve("datastore"); - private static final String FILE_1 = DATASTORE.resolve("pd-test-1.xml").toString(); - - private static final String FILE_2 = DATASTORE.resolve("pd-test-2.xml").toString(); - private static final String NO_SUCH_FILE = "no-such-file.xml"; - private static final String NOT_REGULAR_FILE = TEST_DATA.resolve("configuration").toString(); - private static final String INVALID_FILE_1 = DATASTORE.resolve("pd-test-invalid-type.xml") - .toString(); - private static final String INVALID_FILE_2 = DATASTORE.resolve("pd-test-invalid-xml") - .toString(); - - private DataFileTypeCrud dataFileTypeCrud = Mockito.mock(DataFileTypeCrud.class); - private ModelCrud modelCrud = Mockito.mock(ModelCrud.class); - - @Rule - public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule(ZIGGY_HOME_DIR, - DirectoryProperties.ziggyCodeBuildDir().toString()); - - // Basic functionality -- multiple files, multiple definitions, get imported - @Test - public void testBasicImport() throws JAXBException { - - DataFileTypeImporter dataFileImporter = new DataFileTypeImporter( - ImmutableList.of(FILE_1, FILE_2), false); - DataFileTypeImporter importerSpy = Mockito.spy(dataFileImporter); - Mockito.when(importerSpy.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); - Mockito.when(importerSpy.modelCrud()).thenReturn(modelCrud); - importerSpy.importFromFiles(); - - assertEquals(6, importerSpy.getDataFileImportedCount()); - Mockito.verify(dataFileTypeCrud, Mockito.times(1)) - .persist(ArgumentMatchers. anyList()); - - assertEquals(2, importerSpy.getModelFileImportedCount()); - Mockito.verify(modelCrud, Mockito.times(1)).persist(ArgumentMatchers. anyList()); - } - - // Dry run test -- should import but not persist - @Test - public void testDryRun() throws JAXBException { - - DataFileTypeImporter dataFileImporter = new DataFileTypeImporter( - ImmutableList.of(FILE_1, FILE_2), true); - DataFileTypeImporter importerSpy = Mockito.spy(dataFileImporter); - Mockito.when(importerSpy.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); - Mockito.when(importerSpy.modelCrud()).thenReturn(modelCrud); - importerSpy.importFromFiles(); - - assertEquals(6, importerSpy.getDataFileImportedCount()); - Mockito.verify(dataFileTypeCrud, Mockito.times(0)) - .persist(ArgumentMatchers. anyList()); - assertEquals(2, importerSpy.getModelFileImportedCount()); - Mockito.verify(modelCrud, Mockito.times(0)).persist(ArgumentMatchers. anyList()); - } - - // Test with missing and non-regular files -- should still import from the present, - // regular files - @Test - public void testWithInvalidFiles() throws JAXBException { - - DataFileTypeImporter dataFileImporter = new DataFileTypeImporter( - ImmutableList.of(FILE_1, FILE_2, NO_SUCH_FILE, NOT_REGULAR_FILE), false); - DataFileTypeImporter importerSpy = Mockito.spy(dataFileImporter); - Mockito.when(importerSpy.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); - Mockito.when(importerSpy.modelCrud()).thenReturn(modelCrud); - importerSpy.importFromFiles(); - - assertEquals(6, importerSpy.getDataFileImportedCount()); - Mockito.verify(dataFileTypeCrud, Mockito.times(1)) - .persist(ArgumentMatchers. anyList()); - } - - // Test with a file that has an entry that is valid XML but instantiates to an - // invalid DataFileType instance - @Test - public void testWithInvalidDataFileType() throws JAXBException { - - DataFileTypeImporter dataFileImporter = new DataFileTypeImporter( - ImmutableList.of(FILE_1, INVALID_FILE_1), false); - DataFileTypeImporter importerSpy = Mockito.spy(dataFileImporter); - Mockito.when(importerSpy.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); - Mockito.when(importerSpy.modelCrud()).thenReturn(modelCrud); - importerSpy.importFromFiles(); - - assertEquals(5, importerSpy.getDataFileImportedCount()); - Mockito.verify(dataFileTypeCrud, Mockito.times(1)) - .persist(ArgumentMatchers. anyList()); - } - - // Test with a file that has an entry that is invalid XML - @Test - public void testWithInvalidDataXml() throws JAXBException { - - DataFileTypeImporter dataFileImporter = new DataFileTypeImporter( - ImmutableList.of(FILE_1, INVALID_FILE_2), false); - DataFileTypeImporter importerSpy = Mockito.spy(dataFileImporter); - Mockito.when(importerSpy.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); - Mockito.when(importerSpy.modelCrud()).thenReturn(modelCrud); - importerSpy.importFromFiles(); - - assertEquals(5, importerSpy.getDataFileImportedCount()); - Mockito.verify(dataFileTypeCrud, Mockito.times(1)) - .persist(ArgumentMatchers. anyList()); - } - - @Test(expected = IllegalStateException.class) - public void testDuplicateNames() throws JAXBException { - - DataFileTypeImporter dataFileImporter = new DataFileTypeImporter( - ImmutableList.of(FILE_1, FILE_1), false); - DataFileTypeImporter importerSpy = Mockito.spy(dataFileImporter); - Mockito.when(importerSpy.dataFileTypeCrud()).thenReturn(dataFileTypeCrud); - Mockito.when(importerSpy.modelCrud()).thenReturn(modelCrud); - importerSpy.importFromFiles(); - } -} diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataFileTypeTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataFileTypeTest.java deleted file mode 100644 index 697442d..0000000 --- a/src/test/java/gov/nasa/ziggy/data/management/DataFileTypeTest.java +++ /dev/null @@ -1,135 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.util.regex.Pattern; - -import org.junit.Before; -import org.junit.Test; - -/** - * Unit test class for DataFileType class. - * - * @author PT - */ -public class DataFileTypeTest { - - private DataFileType d; - - @Before - public void setUp() { - d = new DataFileType(); - d.setName("calibrated-pixels"); - d.setFileNameRegexForTaskDir( - "sector-([0-9]{4})-readout-([ABCD])-ccd-([1234]:[1234])-calibrated-pixels.h5"); - d.setFileNameWithSubstitutionsForDatastore( - "sector-$1/ccd-$3/cal/sector-$1/readout-$2/calibrated-pixels.h5"); - } - - @Test - public void testFileNamePatternForTaskDir() { - Pattern p = d.fileNamePatternForTaskDir(); - assertEquals("sector-([0-9]{4})-readout-([ABCD])-ccd-([1234]:[1234])-calibrated-pixels.h5", - p.pattern()); - } - - @Test - public void testFileNameRegexForDatastore() { - String s = d.fileNameRegexForDatastore(); - assertEquals( - "sector-([0-9]{4})/ccd-([1234]:[1234])/cal/sector-\\1/readout-([ABCD])/calibrated-pixels.h5", - s); - - d = new DataFileType(); - d.setName("has backslashes"); - d.setFileNameRegexForTaskDir("(\\S+)-(set-[0-9])-(file-[0-9]).png"); - d.setFileNameWithSubstitutionsForDatastore("$2/L0/$1-$3.png"); - assertEquals("(set-[0-9])/L0/(\\S+)-(file-[0-9]).png", d.fileNameRegexForDatastore()); - } - - @Test - public void testFileNamePatternForDatastore() { - Pattern p = d.fileNamePatternForDatastore(); - assertEquals( - "sector-([0-9]{4})/ccd-([1234]:[1234])/cal/sector-\\1/readout-([ABCD])/calibrated-pixels.h5", - p.pattern()); - } - - @Test - public void testFileNameInTaskDirMatches() { - String goodMatch = "sector-1234-readout-A-ccd-1:2-calibrated-pixels.h5"; - assertTrue(d.fileNameInTaskDirMatches(goodMatch)); - String badMatch = "sector-123-readout-A-ccd-1:2-calibrated-pixels.h5"; - assertFalse(d.fileNameInTaskDirMatches(badMatch)); - - d = new DataFileType(); - d.setName("has backslashes"); - d.setFileNameRegexForTaskDir("perm-(\\S+)-(set-[0-9])-(file-[0-9]).png"); - d.setFileNameWithSubstitutionsForDatastore("$2/L0/$1-$3.png"); - assertTrue(d.fileNameInTaskDirMatches("perm-nasa_logo-set-1-file-0.png")); - } - - @Test - public void testFileNameInDatastoreMatches() { - String goodMatch = "sector-1234/ccd-1:2/cal/sector-1234/readout-A/calibrated-pixels.h5"; - assertTrue(d.fileNameInDatastoreMatches(goodMatch)); - String badMatch = "sector-123/ccd-1:2/cal/sector-1234/readout-A/calibrated-pixels.h5"; - assertFalse(d.fileNameInDatastoreMatches(badMatch)); - } - - @Test - public void testDatastoreFileNameFromTaskDirFileName() { - String s = d.datastoreFileNameFromTaskDirFileName( - "sector-1234-readout-A-ccd-1:2-calibrated-pixels.h5"); - assertEquals("sector-1234/ccd-1:2/cal/sector-1234/readout-A/calibrated-pixels.h5", s); - } - - @Test - public void testTaskDirFileNameFromDatastoreFileName() { - String s = d.taskDirFileNameFromDatastoreFileName( - "sector-1234/ccd-1:2/cal/sector-1234/readout-A/calibrated-pixels.h5"); - assertEquals("sector-1234-readout-A-ccd-1:2-calibrated-pixels.h5", s); - } - - @Test - public void testGetDatastorePatternTruncatedToLevel() { - Pattern p = d.getDatastorePatternTruncatedToLevel(2); - assertEquals("sector-([0-9]{4})/ccd-([1234]:[1234])", p.pattern()); - } - - @Test - public void testGetDatastorePatternWithLowLevelsTruncated() { - Pattern p = d.getDatastorePatternWithLowLevelsTruncated(3); - assertEquals("sector-([0-9]{4})/ccd-([1234]:[1234])/cal", p.pattern()); - } - - // Now for some tests that should cause exceptions to be thrown - - @Test(expected = IllegalStateException.class) - public void testNonContiguousSubstitions() { - d.setFileNameWithSubstitutionsForDatastore( - "sector-$1/ccd-$4/cal/readout-$2/calibrated-pixels.h5"); - d.validate(); - } - - @Test(expected = IllegalStateException.class) - public void testBadSubstitutionValues() { - d.setFileNameWithSubstitutionsForDatastore( - "sector-$2/ccd-$3/cal/readout-$4/calibrated-pixels.h5"); - d.validate(); - } - - @Test(expected = IllegalStateException.class) - public void testRegexSubstitutionMismatch() { - d.setFileNameWithSubstitutionsForDatastore("sector-$1/cal/readout-$2/calibrated-pixels.h5"); - d.validate(); - } - - @Test(expected = IllegalStateException.class) - public void testNoName() { - d.setName(""); - d.validate(); - } -} diff --git a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java index b256393..e4d1ced 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DataReceiptPipelineModuleTest.java @@ -8,7 +8,6 @@ import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; import java.io.File; @@ -20,10 +19,9 @@ import java.util.Collection; import java.util.HashMap; import java.util.HashSet; +import java.util.List; import java.util.Map; import java.util.Set; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; import org.apache.commons.io.FileUtils; import org.junit.After; @@ -35,25 +33,26 @@ import org.mockito.Mockito; import org.xml.sax.SAXException; -import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableSet; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.collections.ZiggyDataType; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumer.DataReceiptFileType; +import gov.nasa.ziggy.data.datastore.DatastoreTestUtils; +import gov.nasa.ziggy.data.datastore.DatastoreWalker; import gov.nasa.ziggy.models.ModelImporter; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.pipeline.definition.ModelMetadata; -import gov.nasa.ziggy.pipeline.definition.ModelMetadataTest.ModelMetadataFixedDate; import gov.nasa.ziggy.pipeline.definition.ModelRegistry; import gov.nasa.ziggy.pipeline.definition.ModelType; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingState; import gov.nasa.ziggy.pipeline.definition.TypedParameter; import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; import gov.nasa.ziggy.uow.DataReceiptUnitOfWorkGenerator; @@ -72,15 +71,17 @@ public class DataReceiptPipelineModuleTest { private PipelineTask pipelineTask = Mockito.mock(PipelineTask.class); private Path dataImporterPath; private Path dataImporterSubdirPath; - private Path modelImporterSubdirPath; private Path datastoreRootPath; private UnitOfWork singleUow = new UnitOfWork(); private UnitOfWork dataSubdirUow = new UnitOfWork(); - private UnitOfWork modelSubdirUow = new UnitOfWork(); private PipelineDefinitionNode node = new PipelineDefinitionNode(); private ModelType modelType1, modelType2, modelType3; - - private ExecutorService execThread; + private DatastoreDirectoryDataReceiptDefinition dataReceiptDefinition; + private ModelImporter modelImporter, subdirModelImporter; + private ModelCrud modelCrud; + private PipelineInstanceCrud pipelineInstanceCrud; + private DatastoreWalker datastoreWalker; + ModelRegistry registry; public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -117,63 +118,72 @@ public void setUp() throws IOException { // set up model types setUpModelTypes(); - // Initialize the data type samples - DataFileTestUtils.initializeDataFileTypeSamples(); - // Construct the necessary directories. - dataImporterPath = Paths.get(dataReceiptDirPropertyRule.getProperty()); - dataImporterPath.toFile().mkdirs(); - datastoreRootPath = Paths.get(datastoreRootDirPropertyRule.getProperty()); - datastoreRootPath.toFile().mkdirs(); + dataImporterPath = Paths.get(dataReceiptDirPropertyRule.getValue()).toAbsolutePath(); + datastoreRootPath = Paths.get(datastoreRootDirPropertyRule.getValue()).toAbsolutePath(); dataImporterSubdirPath = dataImporterPath.resolve("sub-dir"); - dataImporterSubdirPath.toFile().mkdirs(); dataSubdirUow.addParameter(new TypedParameter( UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, DataReceiptUnitOfWorkGenerator.class.getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); - modelSubdirUow.addParameter(new TypedParameter( - UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, - DataReceiptUnitOfWorkGenerator.class.getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); singleUow.addParameter(new TypedParameter( UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, DataReceiptUnitOfWorkGenerator.class.getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); dataSubdirUow - .addParameter(new TypedParameter(DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME, + .addParameter(new TypedParameter(DirectoryUnitOfWorkGenerator.DIRECTORY_PARAMETER_NAME, "sub-dir", ZiggyDataType.ZIGGY_STRING)); - modelSubdirUow - .addParameter(new TypedParameter(DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME, - "models-sub-dir", ZiggyDataType.ZIGGY_STRING)); singleUow.addParameter(new TypedParameter( - DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME, "", ZiggyDataType.ZIGGY_STRING)); - modelImporterSubdirPath = dataImporterPath.resolve("models-sub-dir"); - modelImporterSubdirPath.toFile().mkdirs(); - - // construct the files for import - constructFilesForImport(); - - // Construct the data file type information - node.setInputDataFileTypes(ImmutableSet.of(DataFileTestUtils.dataFileTypeSample1, - DataFileTestUtils.dataFileTypeSample2)); + DirectoryUnitOfWorkGenerator.DIRECTORY_PARAMETER_NAME, "", ZiggyDataType.ZIGGY_STRING)); // construct the model type information node.setModelTypes(ImmutableSet.of(modelType1, modelType2, modelType3)); // Create the "database objects," these are actually an assortment of mocks // so we can test this without needing an actual database. - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); + Mockito.when(pipelineTask.pipelineDefinitionNode()).thenReturn(node); Mockito.when(pipelineTask.getId()).thenReturn(101L); + PipelineInstance pipelineInstance = new PipelineInstance(); + pipelineInstance.setId(2L); + Mockito.when(pipelineTask.getPipelineInstance()).thenReturn(pipelineInstance); + pipelineInstanceCrud = Mockito.mock(PipelineInstanceCrud.class); + Mockito.when(pipelineInstanceCrud.retrieve(ArgumentMatchers.anyLong())) + .thenReturn(pipelineInstance); // Put in a mocked AlertService instance. AlertService.setInstance(Mockito.mock(AlertService.class)); - // Set up the executor service - execThread = Executors.newFixedThreadPool(1); + // Set up the model importer and data receipt definition. + constructDataReceiptDefinition(); + Mockito.doReturn(pipelineInstanceCrud).when(dataReceiptDefinition).pipelineInstanceCrud(); + Mockito.doReturn(List.of(modelType1, modelType2, modelType3)) + .when(dataReceiptDefinition) + .modelTypes(); + modelCrud = Mockito.mock(ModelCrud.class); + + registry = new ModelRegistry(); + modelImporter = new ModelImporter(dataImporterPath, "unit test"); + modelImporter = Mockito.spy(modelImporter); + Mockito.doReturn(registry).when(modelImporter).unlockedRegistry(); + Mockito.doNothing() + .when(modelImporter) + .persistModelMetadata(ArgumentMatchers.any(ModelMetadata.class)); + Mockito.doReturn(1L) + .when(modelImporter) + .mergeRegistryAndReturnUnlockedId(ArgumentMatchers.any(ModelRegistry.class)); + Mockito.doReturn(modelImporter).when(dataReceiptDefinition).modelImporter(); + + subdirModelImporter = new ModelImporter(dataImporterSubdirPath, "unit test"); + subdirModelImporter = Mockito.spy(subdirModelImporter); + Mockito.doReturn(registry).when(subdirModelImporter).unlockedRegistry(); + Mockito.doNothing() + .when(subdirModelImporter) + .persistModelMetadata(ArgumentMatchers.any(ModelMetadata.class)); + Mockito.doReturn(1L) + .when(subdirModelImporter) + .mergeRegistryAndReturnUnlockedId(ArgumentMatchers.any(ModelRegistry.class)); } @After public void shutDown() throws InterruptedException, IOException { - Thread.interrupted(); - execThread.shutdownNow(); - AlertService.setInstance(null); } @@ -182,9 +192,8 @@ public void testImportFromDataReceiptDir() throws IOException, InstantiationExce IllegalAccessException, SAXException, JAXBException, IllegalArgumentException, InvocationTargetException, NoSuchMethodException, SecurityException { - // Populate the models - setUpModelsForImport(dataImporterPath); - constructManifests(); + // Populate the importer files + constructFilesForImport(dataImporterPath); Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); @@ -199,39 +208,64 @@ public void testImportFromDataReceiptDir() throws IOException, InstantiationExce assertEquals(0, module.getFailedImportsDataAccountability().size()); Set producerConsumerRecords = module .getSuccessfulImportsDataAccountability(); - assertEquals(6, producerConsumerRecords.size()); + assertEquals(7, producerConsumerRecords.size()); Map successfulImports = new HashMap<>(); for (DatastoreProducerConsumer producerConsumer : producerConsumerRecords) { successfulImports.put(producerConsumer.getFilename(), producerConsumer.getProducer()); } - assertTrue(successfulImports.containsKey("pa/20/pa-001234567-20-results.h5")); - assertEquals(Long.valueOf(101L), successfulImports.get("pa/20/pa-001234567-20-results.h5")); - assertTrue(successfulImports.containsKey("cal/20/cal-1-1-A-20-results.h5")); - assertEquals(Long.valueOf(101L), successfulImports.get("cal/20/cal-1-1-A-20-results.h5")); assertTrue(successfulImports - .containsKey("models/geometry/tess2020321141517-12345_024-geometry.xml")); + .containsKey("sector-0002/mda/dr/pixels/target/science/1:1:A/1:1:A.nc")); assertEquals(Long.valueOf(101L), - successfulImports.get("models/geometry/tess2020321141517-12345_024-geometry.xml")); + successfulImports.get("sector-0002/mda/dr/pixels/target/science/1:1:A/1:1:A.nc")); + assertTrue(successfulImports - .containsKey("models/geometry/tess2020321141517-12345_025-geometry.xml")); + .containsKey("sector-0002/mda/cal/pixels/ffi/collateral/1:1:A/1:1:A.nc")); assertEquals(Long.valueOf(101L), - successfulImports.get("models/geometry/tess2020321141517-12345_025-geometry.xml")); - assertTrue( - successfulImports.containsKey("models/ravenswood/2020-12-29.0001-simple-text.h5")); - assertEquals(Long.valueOf(101L), - successfulImports.get("models/ravenswood/2020-12-29.0001-simple-text.h5")); - assertTrue( - successfulImports.containsKey("models/calibration/2020-12-29.calibration-4.12.19.h5")); + successfulImports.get("sector-0002/mda/cal/pixels/ffi/collateral/1:1:A/1:1:A.nc")); + + assertTrue(successfulImports + .containsKey("models/geometry/tess2020321141517-12345_024-geometry.xml")); assertEquals(Long.valueOf(101L), - successfulImports.get("models/calibration/2020-12-29.calibration-4.12.19.h5")); + successfulImports.get("models/geometry/tess2020321141517-12345_024-geometry.xml")); + + assertEquals(3, registry.getModels().size()); + ModelMetadata metadata = registry.getModels().get(modelType1); + String datastoreName = Paths.get("models") + .resolve(modelType1.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + metadata = registry.getModels().get(modelType2); + datastoreName = Paths.get("models") + .resolve(modelType2.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + metadata = registry.getModels().get(modelType3); + datastoreName = Paths.get("models") + .resolve(modelType3.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); // check that all the files made it to their destinations - assertTrue(datastoreRootPath.resolve(Paths.get("pa", "20", "pa-001234567-20-results.h5")) - .toFile() - .exists()); - assertTrue(datastoreRootPath.resolve(Paths.get("cal", "20", "cal-1-1-A-20-results.h5")) - .toFile() - .exists()); + assertTrue( + datastoreRootPath + .resolve(Paths.get("sector-0002", "mda", "dr", "pixels", "target", "science", + "1:1:A", "1:1:A.nc")) + .toFile() + .exists()); + assertTrue( + datastoreRootPath + .resolve(Paths.get("sector-0002", "mda", "cal", "pixels", "ffi", "collateral", + "1:1:A", "1:1:A.nc")) + .toFile() + .exists()); assertTrue(datastoreRootPath .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_024-geometry.xml")) .toFile() @@ -240,38 +274,22 @@ public void testImportFromDataReceiptDir() throws IOException, InstantiationExce .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_025-geometry.xml")) .toFile() .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "calibration", "2020-12-29.calibration-4.12.19.h5")) + + assertTrue(datastoreRootPath.resolve("models") + .resolve(modelType1.getType()) + .resolve(registry.getModels().get(modelType1).getDatastoreFileName()) .toFile() .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "ravenswood", "2020-12-29.0001-simple-text.h5")) + assertTrue(datastoreRootPath.resolve("models") + .resolve(modelType2.getType()) + .resolve(registry.getModels().get(modelType2).getDatastoreFileName()) + .toFile() + .exists()); + assertTrue(datastoreRootPath.resolve("models") + .resolve(modelType3.getType()) + .resolve(registry.getModels().get(modelType3).getDatastoreFileName()) .toFile() .exists()); - - // Check that the files were removed from the import directories, or not, as - // appropriate - assertEquals(5, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve("sub-dir").toFile().exists()); - assertTrue(dataImporterPath.resolve("models-sub-dir").toFile().exists()); - assertTrue(dataImporterPath.resolve("pdc-1-1-22-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("data-importer-manifest.xml").toFile().exists()); - assertTrue(dataImporterPath.resolve("data-importer-manifest-ack.xml").toFile().exists()); - assertEquals(3, dataImporterSubdirPath.toFile().listFiles().length); - assertTrue(dataImporterSubdirPath.resolve("pa-765432100-20-results.h5").toFile().exists()); - assertTrue(dataImporterSubdirPath.resolve("cal-1-1-B-20-results.h5").toFile().exists()); - assertTrue( - dataImporterSubdirPath.resolve("data-importer-subdir-manifest.xml").toFile().exists()); - assertEquals(0, modelImporterSubdirPath.toFile().listFiles().length); - - // Get the manifest out of the database - Manifest dbManifest = module.getManifest(); - assertNotNull(dbManifest); - assertEquals(1L, dbManifest.getDatasetId()); - assertTrue(dbManifest.isAcknowledged()); - assertEquals(DataReceiptStatus.VALID, dbManifest.getStatus()); - assertEquals("data-importer-manifest.xml", dbManifest.getName()); - assertNotNull(dbManifest.getImportTime()); } @Test @@ -279,9 +297,10 @@ public void testImportFromDataSubdir() throws IOException, InstantiationExceptio IllegalAccessException, SAXException, JAXBException, IllegalArgumentException, InvocationTargetException, NoSuchMethodException, SecurityException { - // Populate the models - setUpModelsForImport(modelImporterSubdirPath); - constructManifests(); + // Populate the importer files + constructFilesForImport(dataImporterPath); + constructFilesForImport(dataImporterSubdirPath); + Mockito.doReturn(subdirModelImporter).when(dataReceiptDefinition).modelImporter(); // Set up the pipeline module to return the single unit of work task and the appropriate // families of model and data types @@ -296,55 +315,62 @@ public void testImportFromDataSubdir() throws IOException, InstantiationExceptio assertEquals(0, module.getFailedImportsDataAccountability().size()); Set producerConsumerRecords = module .getSuccessfulImportsDataAccountability(); - assertEquals(2, producerConsumerRecords.size()); + assertEquals(5, producerConsumerRecords.size()); Map successfulImports = new HashMap<>(); for (DatastoreProducerConsumer producerConsumer : producerConsumerRecords) { successfulImports.put(producerConsumer.getFilename(), producerConsumer.getProducer()); } - assertTrue(successfulImports.containsKey("pa/20/pa-765432100-20-results.h5")); - assertEquals(Long.valueOf(101L), successfulImports.get("pa/20/pa-765432100-20-results.h5")); - assertTrue(successfulImports.containsKey("cal/20/cal-1-1-B-20-results.h5")); - assertEquals(Long.valueOf(101L), successfulImports.get("cal/20/cal-1-1-B-20-results.h5")); + assertTrue(successfulImports + .containsKey("sector-0002/mda/dr/pixels/target/science/1:1:B/1:1:B.nc")); + assertEquals(Long.valueOf(101L), + successfulImports.get("sector-0002/mda/dr/pixels/target/science/1:1:B/1:1:B.nc")); // check that the data files made it to their destinations - assertTrue(datastoreRootPath.resolve(Paths.get("pa", "20", "pa-765432100-20-results.h5")) - .toFile() - .exists()); - assertTrue(datastoreRootPath.resolve(Paths.get("cal", "20", "cal-1-1-B-20-results.h5")) - .toFile() - .exists()); + assertTrue( + datastoreRootPath + .resolve(Paths.get("sector-0002", "mda", "dr", "pixels", "target", "science", + "1:1:B", "1:1:B.nc")) + .toFile() + .exists()); + + assertTrue(successfulImports + .containsKey("models/geometry/tess2020321141517-12345_024-geometry.xml")); + assertEquals(Long.valueOf(101L), + successfulImports.get("models/geometry/tess2020321141517-12345_024-geometry.xml")); + + assertEquals(3, registry.getModels().size()); + ModelMetadata metadata = registry.getModels().get(modelType1); + String datastoreName = Paths.get("models") + .resolve(modelType1.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + metadata = registry.getModels().get(modelType2); + datastoreName = Paths.get("models") + .resolve(modelType2.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + metadata = registry.getModels().get(modelType3); + datastoreName = Paths.get("models") + .resolve(modelType3.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); // Check that the files were removed from the import directories, or not, as // appropriate - assertEquals(5, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve("models-sub-dir").toFile().exists()); - assertTrue(dataImporterPath.resolve("pdc-1-1-22-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("pa-001234567-20-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("cal-1-1-A-20-results.h5").toFile().exists()); + assertEquals(3, dataImporterPath.toFile().listFiles().length); + assertTrue(dataImporterPath.resolve("models").toFile().exists()); + assertTrue(dataImporterPath.resolve("sector-0002").toFile().exists()); assertTrue(dataImporterPath.resolve("data-importer-manifest.xml").toFile().exists()); - assertEquals(5, modelImporterSubdirPath.toFile().listFiles().length); - assertTrue(modelImporterSubdirPath.resolve("tess2020321141517-12345_024-geometry.xml") - .toFile() - .exists()); - assertTrue(modelImporterSubdirPath.resolve("tess2020321141517-12345_025-geometry.xml") - .toFile() - .exists()); - assertTrue(modelImporterSubdirPath.resolve("calibration-4.12.19.h5").toFile().exists()); - assertTrue(modelImporterSubdirPath.resolve("simple-text.h5").toFile().exists()); - assertTrue(modelImporterSubdirPath.resolve("model-importer-subdir-manifest.xml") - .toFile() - .exists()); - // Get the manifest out of the database - Manifest dbManifest = module.getManifest(); - assertNotNull(dbManifest); - assertEquals(2L, dbManifest.getDatasetId()); - assertTrue(dbManifest.isAcknowledged()); - assertEquals(DataReceiptStatus.VALID, dbManifest.getStatus()); - assertEquals("data-importer-subdir-manifest.xml", dbManifest.getName()); - assertNotNull(dbManifest.getImportTime()); - - // The manifest and the acknowledgement should be moved to the manifests hidden + // The manifest and the acknowledgement should be moved to the manifests // directory Path manifestDir = DirectoryProperties.manifestsDir(); assertTrue(Files.exists(manifestDir)); @@ -353,103 +379,6 @@ public void testImportFromDataSubdir() throws IOException, InstantiationExceptio // The data directory should be deleted assertFalse(Files.exists(dataImporterSubdirPath)); - - // The models directory should still be present with its same file content - assertTrue(Files.exists(modelImporterSubdirPath)); - assertEquals(5, modelImporterSubdirPath.toFile().listFiles().length); - - // The parent directory should still have its same file content, - // except for the data subdirectory (but with the manifests directory - // the number of files is 6 again anyway) - assertEquals(5, dataImporterPath.toFile().listFiles().length); - } - - @Test - public void testImportFromModelsSubdir() throws IOException, InstantiationException, - IllegalAccessException, SAXException, JAXBException, IllegalArgumentException, - InvocationTargetException, NoSuchMethodException, SecurityException { - - // Populate the models - setUpModelsForImport(modelImporterSubdirPath); - constructManifests(); - - // Set up the pipeline module to return the single unit of work task and the appropriate - // families of model and data types - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(modelSubdirUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - Mockito.when(pipelineTask.getId()).thenReturn(101L); - - // Perform the import - DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, - RunMode.STANDARD); - module.processTask(); - - // Obtain the producer-consumer records and check that only the data files are listed - assertEquals(0, module.getFailedImportsDataAccountability().size()); - Set producerConsumerRecords = module - .getSuccessfulImportsDataAccountability(); - assertEquals(4, producerConsumerRecords.size()); - Map successfulImports = new HashMap<>(); - for (DatastoreProducerConsumer producerConsumer : producerConsumerRecords) { - successfulImports.put(producerConsumer.getFilename(), producerConsumer.getProducer()); - } - assertTrue(successfulImports - .containsKey("models/geometry/tess2020321141517-12345_024-geometry.xml")); - assertEquals(Long.valueOf(101L), - successfulImports.get("models/geometry/tess2020321141517-12345_024-geometry.xml")); - assertTrue(successfulImports - .containsKey("models/geometry/tess2020321141517-12345_025-geometry.xml")); - assertEquals(Long.valueOf(101L), - successfulImports.get("models/geometry/tess2020321141517-12345_025-geometry.xml")); - assertTrue( - successfulImports.containsKey("models/ravenswood/2020-12-29.0001-simple-text.h5")); - assertEquals(Long.valueOf(101L), - successfulImports.get("models/ravenswood/2020-12-29.0001-simple-text.h5")); - assertTrue( - successfulImports.containsKey("models/calibration/2020-12-29.calibration-4.12.19.h5")); - assertEquals(Long.valueOf(101L), - successfulImports.get("models/calibration/2020-12-29.calibration-4.12.19.h5")); - - // check that the model files made it to their destinations - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_024-geometry.xml")) - .toFile() - .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_025-geometry.xml")) - .toFile() - .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "calibration", "2020-12-29.calibration-4.12.19.h5")) - .toFile() - .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "ravenswood", "2020-12-29.0001-simple-text.h5")) - .toFile() - .exists()); - - // Check that the files were removed from the import directories, or not, as - // appropriate - assertEquals(5, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve("sub-dir").toFile().exists()); - assertTrue(dataImporterPath.resolve("pdc-1-1-22-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("pa-001234567-20-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("cal-1-1-A-20-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("data-importer-manifest.xml").toFile().exists()); - assertEquals(3, dataImporterSubdirPath.toFile().listFiles().length); - assertTrue(dataImporterSubdirPath.resolve("pa-765432100-20-results.h5").toFile().exists()); - assertTrue(dataImporterSubdirPath.resolve("cal-1-1-B-20-results.h5").toFile().exists()); - assertTrue( - dataImporterSubdirPath.resolve("data-importer-subdir-manifest.xml").toFile().exists()); - - // Get the manifest out of the database - Manifest dbManifest = module.getManifest(); - assertNotNull(dbManifest); - assertEquals(3L, dbManifest.getDatasetId()); - assertTrue(dbManifest.isAcknowledged()); - assertEquals(DataReceiptStatus.VALID, dbManifest.getStatus()); - assertEquals("model-importer-subdir-manifest.xml", dbManifest.getName()); - assertNotNull(dbManifest.getImportTime()); } @Test @@ -458,45 +387,33 @@ public void testImportWithErrors() throws IOException, InstantiationException, InvocationTargetException, NoSuchMethodException, SecurityException { // Populate the models - setUpModelsForImport(dataImporterPath); - constructManifests(); + constructFilesForImport(dataImporterPath); // Set up the pipeline module to return the single unit of work task and the appropriate // families of model and data types Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - Mockito.when(pipelineTask.getId()).thenReturn(101L); // Generate data and model importers that will throw IOExceptions at opportune moments - DefaultDataImporter dataImporter = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - dataImporter = Mockito.spy(dataImporter); + Path dataReceiptExceptionPath = dataImporterPath.resolve(Paths.get("sector-0002", "mda", + "cal", "pixels", "ffi", "collateral", "1:1:A", "1:1:A.nc")); + Path datastoreExceptionPath = datastoreRootPath.resolve(Paths.get("sector-0002", "mda", + "cal", "pixels", "ffi", "collateral", "1:1:A", "1:1:A.nc")); Mockito.doThrow(IOException.class) - .when(dataImporter) - .moveOrSymlink(dataImporterPath.resolve(Paths.get("pa-001234567-20-results.h5")), - datastoreRootPath.resolve(Paths.get("pa", "20", "pa-001234567-20-results.h5"))); + .when(dataReceiptDefinition) + .move(dataReceiptExceptionPath, datastoreExceptionPath); - ModelImporter modelImporter = new ModelImporterForTest(dataImporterPath.toString(), - "unit test"); - modelImporter = Mockito.spy(modelImporter); Path destFileToFlunk = Paths.get(datastoreRootPath.toString(), "models", "geometry", "tess2020321141517-12345_025-geometry.xml"); - Path srcFileToFlunk = Paths.get(dataImporterPath.toString(), + Path srcFileToFlunk = Paths.get(dataImporterPath.toString(), "models", "tess2020321141517-12345_025-geometry.xml"); Mockito.doThrow(IOException.class) .when(modelImporter) - .moveOrSymlink(srcFileToFlunk, destFileToFlunk); + .move(srcFileToFlunk, destFileToFlunk); // Install the data and model importers in the pipeline module DataReceiptModuleForTest pipelineModule = new DataReceiptModuleForTest(pipelineTask, RunMode.STANDARD); final DataReceiptModuleForTest module = Mockito.spy(pipelineModule); - Mockito.doReturn(dataImporter) - .when(module) - .dataImporter(ArgumentMatchers.any(Path.class), ArgumentMatchers.any(Path.class)); - Mockito.doReturn(modelImporter) - .when(module) - .modelImporter(ArgumentMatchers.any(Path.class), ArgumentMatchers.any(String.class)); // install a dummy alert service in the module Mockito.doReturn(Mockito.mock(AlertService.class)).when(module).alertService(); @@ -514,69 +431,107 @@ public void testImportWithErrors() throws IOException, InstantiationException, // the correct producer Set producerConsumerRecords = module .getSuccessfulImportsDataAccountability(); - assertEquals(4, producerConsumerRecords.size()); Map successfulImports = new HashMap<>(); for (DatastoreProducerConsumer producerConsumer : producerConsumerRecords) { successfulImports.put(producerConsumer.getFilename(), producerConsumer.getProducer()); } - assertTrue(successfulImports.containsKey("cal/20/cal-1-1-A-20-results.h5")); - assertEquals(Long.valueOf(101L), successfulImports.get("cal/20/cal-1-1-A-20-results.h5")); assertTrue(successfulImports - .containsKey("models/geometry/tess2020321141517-12345_024-geometry.xml")); + .containsKey("sector-0002/mda/dr/pixels/target/science/1:1:A/1:1:A.nc")); assertEquals(Long.valueOf(101L), - successfulImports.get("models/geometry/tess2020321141517-12345_024-geometry.xml")); - assertTrue( - successfulImports.containsKey("models/ravenswood/2020-12-29.0001-simple-text.h5")); + successfulImports.get("sector-0002/mda/dr/pixels/target/science/1:1:A/1:1:A.nc")); + + assertTrue(successfulImports + .containsKey("sector-0002/mda/cal/pixels/ffi/collateral/1:1:B/1:1:B.nc")); assertEquals(Long.valueOf(101L), - successfulImports.get("models/ravenswood/2020-12-29.0001-simple-text.h5")); - assertTrue( - successfulImports.containsKey("models/calibration/2020-12-29.calibration-4.12.19.h5")); + successfulImports.get("sector-0002/mda/cal/pixels/ffi/collateral/1:1:B/1:1:B.nc")); + + assertTrue(successfulImports + .containsKey("models/geometry/tess2020321141517-12345_024-geometry.xml")); assertEquals(Long.valueOf(101L), - successfulImports.get("models/calibration/2020-12-29.calibration-4.12.19.h5")); + successfulImports.get("models/geometry/tess2020321141517-12345_024-geometry.xml")); + + assertEquals(3, registry.getModels().size()); + ModelMetadata metadata = registry.getModels().get(modelType1); + String datastoreName = Paths.get("models") + .resolve(modelType1.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + metadata = registry.getModels().get(modelType2); + datastoreName = Paths.get("models") + .resolve(modelType2.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + metadata = registry.getModels().get(modelType3); + datastoreName = Paths.get("models") + .resolve(modelType3.getType()) + .resolve(metadata.getDatastoreFileName()) + .toString(); + assertTrue(successfulImports.containsKey(datastoreName)); + assertEquals(Long.valueOf(101L), successfulImports.get(datastoreName)); + + assertEquals(5, successfulImports.size()); // check that the files made it to their destinations - assertTrue(datastoreRootPath.resolve(Paths.get("cal", "20", "cal-1-1-A-20-results.h5")) - .toFile() - .exists()); + assertTrue( + datastoreRootPath + .resolve(Paths.get("sector-0002", "mda", "dr", "pixels", "target", "science", + "1:1:A", "1:1:A.nc")) + .toFile() + .exists()); assertTrue(datastoreRootPath .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_024-geometry.xml")) .toFile() .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "calibration", "2020-12-29.calibration-4.12.19.h5")) + + assertTrue(datastoreRootPath.resolve("models") + .resolve(modelType1.getType()) + .resolve(registry.getModels().get(modelType1).getDatastoreFileName()) .toFile() .exists()); - assertTrue(datastoreRootPath - .resolve(Paths.get("models", "ravenswood", "2020-12-29.0001-simple-text.h5")) + assertTrue(datastoreRootPath.resolve("models") + .resolve(modelType2.getType()) + .resolve(registry.getModels().get(modelType2).getDatastoreFileName()) + .toFile() + .exists()); + assertTrue(datastoreRootPath.resolve("models") + .resolve(modelType3.getType()) + .resolve(registry.getModels().get(modelType3).getDatastoreFileName()) .toFile() .exists()); + assertFalse(datastoreRootPath + .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_025-geometry.xml")) + .toFile() + .exists()); + assertFalse( + datastoreRootPath + .resolve(Paths.get("sector-0002", "mda", "cal", "pixels", "ffi", "collateral", + "1:1:A", "1:1:A.nc")) + .toFile() + .exists()); + // Check that the files were removed from the import directories, or not, as // appropriate - assertEquals(7, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve("sub-dir").toFile().exists()); - assertTrue(dataImporterPath.resolve("models-sub-dir").toFile().exists()); - assertTrue(dataImporterPath.resolve("pdc-1-1-22-results.h5").toFile().exists()); - assertTrue(dataImporterPath.resolve("pa-001234567-20-results.h5").toFile().exists()); - assertTrue( - dataImporterPath.resolve("tess2020321141517-12345_025-geometry.xml").toFile().exists()); - assertTrue(dataImporterPath.resolve("data-importer-manifest.xml").toFile().exists()); - assertTrue(dataImporterPath.resolve("data-importer-manifest-ack.xml").toFile().exists()); - assertEquals(3, dataImporterSubdirPath.toFile().listFiles().length); - assertTrue(dataImporterSubdirPath.resolve("pa-765432100-20-results.h5").toFile().exists()); - assertTrue(dataImporterSubdirPath.resolve("cal-1-1-B-20-results.h5").toFile().exists()); - assertTrue( - dataImporterSubdirPath.resolve("data-importer-subdir-manifest.xml").toFile().exists()); - assertEquals(0, modelImporterSubdirPath.toFile().listFiles().length); - - // Get the manifest out of the database - Manifest dbManifest = module.getManifest(); - assertNotNull(dbManifest); - assertEquals(1L, dbManifest.getDatasetId()); - assertTrue(dbManifest.isAcknowledged()); - assertEquals(DataReceiptStatus.VALID, dbManifest.getStatus()); - assertEquals("data-importer-manifest.xml", dbManifest.getName()); - assertNotNull(dbManifest.getImportTime()); + assertTrue(dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A") + .resolve("1:1:A.nc") + .toFile() + .exists()); + assertTrue(dataImporterPath.resolve("models") + .resolve("tess2020321141517-12345_025-geometry.xml") + .toFile() + .exists()); // Finally, check that the expected files are in the failed imports table. Set failedImports = module.getFailedImportsDataAccountability(); @@ -585,11 +540,97 @@ public void testImportWithErrors() throws IOException, InstantiationException, for (FailedImport failedImport : failedImports) { failedImportMap.put(failedImport.getFilename(), failedImport.getDataReceiptTaskId()); } - assertTrue(failedImportMap.containsKey("pa/20/pa-001234567-20-results.h5")); - assertEquals(Long.valueOf(101L), failedImportMap.get("pa/20/pa-001234567-20-results.h5")); - assertTrue(failedImportMap.containsKey("tess2020321141517-12345_025-geometry.xml")); + assertTrue(failedImportMap.containsKey(Paths.get("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A") + .resolve("1:1:A.nc") + .toString())); + assertEquals(Long.valueOf(101L), + failedImportMap.get(Paths.get("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A") + .resolve("1:1:A.nc") + .toString())); + + assertTrue(failedImportMap.containsKey("models/tess2020321141517-12345_025-geometry.xml")); assertEquals(Long.valueOf(101L), - failedImportMap.get("tess2020321141517-12345_025-geometry.xml")); + failedImportMap.get("models/tess2020321141517-12345_025-geometry.xml")); + } + + @Test + public void testReEntrantImportAfterError() throws InstantiationException, + IllegalAccessException, IllegalArgumentException, InvocationTargetException, + NoSuchMethodException, SecurityException, IOException, SAXException, JAXBException { + testImportWithErrors(); + + // Reconstruct the data receipt definition and model importer so that the extant versions + // don't throw IOExceptions. + constructDataReceiptDefinition(); + Mockito.doReturn(pipelineInstanceCrud).when(dataReceiptDefinition).pipelineInstanceCrud(); + Mockito.doReturn(List.of(modelType1, modelType2, modelType3)) + .when(dataReceiptDefinition) + .modelTypes(); + modelImporter = new ModelImporter(dataImporterPath, "unit test"); + modelImporter = Mockito.spy(modelImporter); + Mockito.doReturn(modelImporter).when(dataReceiptDefinition).modelImporter(); + Mockito.doReturn(registry).when(modelImporter).unlockedRegistry(); + Mockito.doNothing() + .when(modelImporter) + .persistModelMetadata(ArgumentMatchers.any(ModelMetadata.class)); + Mockito.doReturn(1L) + .when(modelImporter) + .mergeRegistryAndReturnUnlockedId(ArgumentMatchers.any(ModelRegistry.class)); + + assertFalse(Files.exists(dataImporterPath.resolve("data-importer-manifest.xml"))); + + DataReceiptModuleForTest pipelineModule = new DataReceiptModuleForTest(pipelineTask, + RunMode.STANDARD); + pipelineModule.storingTaskAction(); + + // Obtain the producer-consumer records and check that the expected files are listed with + // the correct producer + Set producerConsumerRecords = pipelineModule + .getSuccessfulImportsDataAccountability(); + Map successfulImports = new HashMap<>(); + for (DatastoreProducerConsumer producerConsumer : producerConsumerRecords) { + successfulImports.put(producerConsumer.getFilename(), producerConsumer.getProducer()); + } + + assertTrue(successfulImports + .containsKey("sector-0002/mda/cal/pixels/ffi/collateral/1:1:A/1:1:A.nc")); + assertEquals(Long.valueOf(101L), + successfulImports.get("sector-0002/mda/cal/pixels/ffi/collateral/1:1:A/1:1:A.nc")); + + assertTrue(successfulImports + .containsKey("models/geometry/tess2020321141517-12345_025-geometry.xml")); + assertEquals(Long.valueOf(101L), + successfulImports.get("models/geometry/tess2020321141517-12345_025-geometry.xml")); + + assertEquals(0, pipelineModule.getFailedImportsDataAccountability().size()); + + // check that the files made it to their destinations + assertTrue( + datastoreRootPath + .resolve(Paths.get("sector-0002", "mda", "cal", "pixels", "ffi", "collateral", + "1:1:A", "1:1:A.nc")) + .toFile() + .exists()); + assertTrue(datastoreRootPath + .resolve(Paths.get("models", "geometry", "tess2020321141517-12345_025-geometry.xml")) + .toFile() + .exists()); + + // Check that the data import directory is empty because cleanup ran successfully. + File[] files = dataImporterPath.toFile().listFiles(); + assertEquals(0, files.length); } @Test(expected = PipelineException.class) @@ -597,27 +638,30 @@ public void testCleanupFailOnNonEmptyDir() throws IOException, InstantiationExce IllegalAccessException, SAXException, JAXBException, IllegalArgumentException, InvocationTargetException, NoSuchMethodException, SecurityException { - // Populate the models - setUpModelsForImport(dataImporterPath); - constructManifests(); + constructFilesForImport(dataImporterPath); // Set up the pipeline module to return the single unit of work task and the appropriate // families of model and data types Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); + Mockito.when(pipelineTask.pipelineDefinitionNode()).thenReturn(node); Mockito.when(pipelineTask.getId()).thenReturn(101L); // Perform the import DataReceiptPipelineModule module = new DataReceiptModuleForTest(pipelineTask, RunMode.STANDARD); - module.processTask(); + module.performDirectoryCleanup(); } @Test public void testImportOnEmptyDirectory() throws IOException { - FileUtils.cleanDirectory(dataImporterPath.toFile()); + if (Files.exists(dataImporterPath)) { + FileUtils.cleanDirectory(dataImporterPath.toFile()); + Files.delete(dataImporterPath); + } + Files.createDirectories(dataImporterPath); Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + constructDataReceiptDefinition(); // Perform the import DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, RunMode.STANDARD); @@ -626,8 +670,15 @@ public void testImportOnEmptyDirectory() throws IOException { } @Test(expected = PipelineException.class) - public void testMissingManifest() { + public void testMissingManifest() throws InstantiationException, IllegalAccessException, + IllegalArgumentException, InvocationTargetException, NoSuchMethodException, + SecurityException, IOException, SAXException, JAXBException { + + constructFilesForImport(dataImporterPath); + Files.delete(dataImporterPath.resolve("data-importer-manifest.xml")); + Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); + constructDataReceiptDefinition(); // Perform the import DataReceiptModuleForTest module = new DataReceiptModuleForTest(pipelineTask, RunMode.STANDARD); @@ -660,47 +711,87 @@ private void setUpModelTypes() { modelType3.setTimestampGroup(-1); } - private Set constructFilesForImport() throws IOException { - - Set filenames = new HashSet<>(); - // create a couple of files in the DatastoreIdSample1 pattern - File sample1 = new File(dataImporterPath.toFile(), "pa-001234567-20-results.h5"); - File sample2 = new File(dataImporterSubdirPath.toFile(), "pa-765432100-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - filenames.add(sample1.getName()); - filenames.add(sample2.getName()); - - // create a couple of files in the DatastoreIdSample2 pattern - sample1 = new File(dataImporterPath.toFile(), "cal-1-1-A-20-results.h5"); - sample2 = new File(dataImporterSubdirPath.toFile(), "cal-1-1-B-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - filenames.add(sample1.getName()); - filenames.add(sample2.getName()); - - // create a file that matches neither pattern - sample1 = new File(dataImporterPath.toFile(), "pdc-1-1-22-results.h5"); - sample1.createNewFile(); - filenames.add(sample1.getName()); - return filenames; - } + /** + * Constructs the data receipt directory. Specifically, files are placed in the main DR + * directory and in each of two subdirectories, and each of the 3 directories then gets a + * manifest generated. + */ + private void constructFilesForImport(Path importerPath) + throws IOException, InstantiationException, IllegalAccessException, + IllegalArgumentException, InvocationTargetException, NoSuchMethodException, + SecurityException, SAXException, JAXBException { + + // Start with the dataImporterPath files. + if (importerPath.equals(dataImporterPath)) { + Path sample1 = dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("1:1:A.nc"); + Files.createDirectories(sample1.getParent()); + Files.createFile(sample1); + + sample1 = dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A") + .resolve("1:1:A.nc"); + Files.createDirectories(sample1.getParent()); + Files.createFile(sample1); + + sample1 = dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B") + .resolve("1:1:B.nc"); + Files.createDirectories(sample1.getParent()); + Files.createFile(sample1); + + setUpModelsForImport(dataImporterPath); + constructManifest(dataImporterPath, "data-importer-manifest.xml", -1L); + return; + } - private void setUpModelsForImport(Path modelDirPath) throws IOException { - String modelImportDir = modelDirPath.toString(); - // create the new files to be imported - new File(modelImportDir, "tess2020321141517-12345_024-geometry.xml").createNewFile(); - new File(modelImportDir, "tess2020321141517-12345_025-geometry.xml").createNewFile(); - new File(modelImportDir, "calibration-4.12.19.h5").createNewFile(); - new File(modelImportDir, "simple-text.h5").createNewFile(); + // Now do the dataImporterSubdirPath. + Path sample2 = dataImporterSubdirPath.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B") + .resolve("1:1:B.nc"); + Files.createDirectories(sample2.getParent()); + Files.createFile(sample2); + + if (importerPath.equals(dataImporterSubdirPath)) { + setUpModelsForImport(dataImporterSubdirPath); + constructManifest(dataImporterSubdirPath, "data-importer-subdir-manifest.xml", -2L); + } } - private void constructManifests() throws IOException, InstantiationException, - IllegalAccessException, SAXException, JAXBException, IllegalArgumentException, - InvocationTargetException, NoSuchMethodException, SecurityException { - constructManifest(dataImporterSubdirPath, "data-importer-subdir-manifest.xml", 2L); - constructManifest(modelImporterSubdirPath, "model-importer-subdir-manifest.xml", 3L); - constructManifest(dataImporterPath, "data-importer-manifest.xml", 1L); + private void setUpModelsForImport(Path dataImportDir) throws IOException { + Path modelImportDir = dataImportDir.resolve("models"); + // create the new files to be imported + Files.createDirectories(modelImportDir); + // create the new files to be imported + Path modelFile1 = modelImportDir.resolve("tess2020321141517-12345_024-geometry.xml"); + Files.createFile(modelFile1); + Path modelFile2 = modelImportDir.resolve("tess2020321141517-12345_025-geometry.xml"); + Files.createFile(modelFile2); + Path modelFile3 = modelImportDir.resolve("calibration-4.12.19.h5"); + Files.createFile(modelFile3); + Path modelFile4 = modelImportDir.resolve("simple-text.h5"); + Files.createFile(modelFile4); } private void constructManifest(Path dir, String name, long datasetId) @@ -714,41 +805,26 @@ private void constructManifest(Path dir, String name, long datasetId) } } - /** - * Specialized subclass of {@link ModelImporter} that produces {@link ModelMetadata} instances - * with a fixed timestamp. - * - * @author PT - */ - private class ModelImporterForTest extends ModelImporter { - - private ModelCrud crud = Mockito.mock(ModelCrud.class); - - public ModelImporterForTest(String directory, String modelDescription) { - super(directory, modelDescription); - Mockito.when(crud.retrieveAllModelTypes()) - .thenReturn(ImmutableList.of(modelType1, modelType2, modelType3)); - Mockito.when(crud.retrieveUnlockedRegistry()).thenReturn(new ModelRegistry()); - Mockito.when(crud.retrieveUnlockedRegistryId()).thenReturn(2L); - } - - @Override - protected ModelMetadata modelMetadata(ModelType modelType, String modelName, - String modelDescription, ModelMetadata currentRegistryMetadata) { - return new ModelMetadataFixedDate(modelType, modelName, modelDescription, - currentRegistryMetadata).toSuper(); - } + private void constructDataReceiptDefinition() { + dataReceiptDefinition = new DatastoreDirectoryDataReceiptDefinition(); + dataReceiptDefinition.setDataImportDirectory(dataImporterPath); + dataReceiptDefinition.setPipelineTask(pipelineTask); + dataReceiptDefinition = Mockito.spy(dataReceiptDefinition); + Mockito.doReturn(List.of(modelType1, modelType2, modelType3)) + .when(dataReceiptDefinition) + .modelTypes(); + datastoreWalker = new DatastoreWalker(DatastoreTestUtils.regexpsByName(), + DatastoreTestUtils.datastoreNodesByFullPath()); + Mockito.doReturn(datastoreWalker).when(dataReceiptDefinition).datastoreWalker(); + Mockito.doReturn(modelImporter).when(dataReceiptDefinition).modelImporter(); + Mockito.doReturn(modelCrud).when(dataReceiptDefinition).modelCrud(); + Mockito.doNothing().when(dataReceiptDefinition).updateModelRegistryForPipelineInstance(); - @Override - protected ModelCrud modelCrud() { - return crud; - } } /** - * Specialized subclass of {@link DataReceiptPipelineModule} that produces an instance of - * {@link ModelImporter} that, in turn, produces instances of {@link ModelMetadata} with fixed - * timestamps. + * Specialized subclass of {@link DataReceiptPipelineModule} that holds onto some of the data + * accountability results for later inspection. * * @author PT */ @@ -758,7 +834,6 @@ private class DataReceiptModuleForTest extends DataReceiptPipelineModule impleme private ProcessingState processingState = ProcessingState.INITIALIZING; private Set successfulImportsDataAccountability = new HashSet<>(); private Set failedImportsDataAccountability = new HashSet<>(); - private Manifest manifest; public DataReceiptModuleForTest(PipelineTask pipelineTask, RunMode runMode) { super(pipelineTask, runMode); @@ -779,18 +854,6 @@ public void processingCompleteTaskAction() { super.processingCompleteTaskAction(); } - @Override - ModelImporter modelImporter(Path importDirectory, String description) { - if (modelImporter == null) { - modelImporter = new ModelImporterForTest(importDirectory.toString(), description); - } - return modelImporter; - } - - @Override - void updateModelRegistryForPipelineInstance() { - } - @Override public void performDirectoryCleanup() { if (performDirectoryCleanupEnabled) { @@ -798,37 +861,38 @@ public void performDirectoryCleanup() { } } + @Override + DataReceiptDefinition dataReceiptDefinition() { + return dataReceiptDefinition; + } + @Override protected void persistProducerConsumerRecords(Collection successfulImports, - Collection failedImports, DataReceiptFileType fileType) { + Collection failedImports) { for (Path file : successfulImports) { - successfulImportsDataAccountability.add( - new DatastoreProducerConsumer(pipelineTask.getId(), file.toString(), fileType)); + successfulImportsDataAccountability + .add(new DatastoreProducerConsumer(pipelineTask.getId(), file.toString())); } for (Path file : failedImports) { - failedImportsDataAccountability.add(new FailedImport(pipelineTask, file, fileType)); + failedImportsDataAccountability.add(new FailedImport(pipelineTask, file)); } } /** * Returns the sequence of {@link ProcessingState} instances that are produced by the - * production version of {@link #getProcessingState()} during pipeline execution. This + * production version of {@link #databaseProcessingState()} during pipeline execution. This * allows us to live without a database connection for these tests. */ @Override - public ProcessingState getProcessingState() { + public ProcessingState databaseProcessingState() { return processingState; } @Override - public void incrementProcessingState() { + public void incrementDatabaseProcessingState() { processingState = nextProcessingState(processingState); } - @Override - protected void flushDatabase() { - } - public void disableDirectoryCleanup() { performDirectoryCleanupEnabled = false; } @@ -841,33 +905,6 @@ public Set getFailedImportsDataAccountability() { return failedImportsDataAccountability; } - @Override - ManifestCrud manifestCrud() { - return new ManifestCrudForTest(); - } - - public Manifest getManifest() { - return manifest; - } - - /** - * Version of {@link ManifestCrud} that's safe to use in testing. - * - * @author PT - */ - private class ManifestCrudForTest extends ManifestCrud { - - @Override - public void persist(Object o) { - manifest = (Manifest) o; - } - - @Override - public boolean datasetIdExists(long datasetId) { - return false; - } - } - @Override public void run() { processTask(); diff --git a/src/test/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinitionTest.java b/src/test/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinitionTest.java new file mode 100644 index 0000000..897396f --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/data/management/DatastoreDirectoryDataReceiptDefinitionTest.java @@ -0,0 +1,417 @@ +package gov.nasa.ziggy.data.management; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.RuleChain; +import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; + +import gov.nasa.ziggy.ZiggyDatabaseRule; +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.data.datastore.DatastoreTestUtils; +import gov.nasa.ziggy.data.datastore.DatastoreWalker; +import gov.nasa.ziggy.models.ModelImporter; +import gov.nasa.ziggy.pipeline.definition.ModelMetadata; +import gov.nasa.ziggy.pipeline.definition.ModelRegistry; +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.crud.ModelCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.services.config.ZiggyConfiguration; + +/** + * Unit tests for {@link DatastoreDirectoryDataReceiptDefinition} class. + * + * @author PT + */ +public class DatastoreDirectoryDataReceiptDefinitionTest { + + private Path testDirectory; + private Path dataImporterPath; + private Path datastoreRootPath; + private DatastoreDirectoryDataReceiptDefinition dataReceiptDefinition; + private ModelType modelType1, modelType2, modelType3; + private ManifestCrud manifestCrud; + private ModelCrud modelCrud; + private Path dataFile1, dataFile2, dataFile3; + private Path modelFile1, modelFile2, modelFile3, modelFile4; + private DatastoreWalker datastoreWalker; + private ModelImporter modelImporter; + + public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); + + public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( + PropertyName.DATASTORE_ROOT_DIR.property(), directoryRule, "datastore"); + + public ZiggyPropertyRule pipelineRootDirPropertyRule = new ZiggyPropertyRule( + PropertyName.RESULTS_DIR.property(), directoryRule, "pipeline-results"); + + @Rule + public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) + .around(datastoreRootDirPropertyRule) + .around(pipelineRootDirPropertyRule); + + @Rule + public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule( + PropertyName.ZIGGY_HOME_DIR.property(), DirectoryProperties.ziggyCodeBuildDir().toString()); + + @Rule + public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); + + @Before + public void setUp() throws IOException { + + // Construct the necessary directories. + testDirectory = directoryRule.directory(); + dataImporterPath = testDirectory.resolve("data-import").toAbsolutePath(); + dataImporterPath.toFile().mkdirs(); + datastoreRootPath = testDirectory.resolve("datastore").toAbsolutePath(); + datastoreRootPath.toFile().mkdirs(); + + // construct the files for import + constructFilesForImport(); + + // Construct a Spy of the definition instance. + dataReceiptDefinition = Mockito.spy(DatastoreDirectoryDataReceiptDefinition.class); + dataReceiptDefinition.setDataImportDirectory(dataImporterPath); + PipelineTask pipelineTask = new PipelineTask(); + pipelineTask.setId(1L); + PipelineInstance pipelineInstance = new PipelineInstance(); + pipelineInstance.setId(2L); + pipelineTask.setPipelineInstance(pipelineInstance); + dataReceiptDefinition.setPipelineTask(pipelineTask); + setUpModelTypes(); + manifestCrud = Mockito.mock(ManifestCrud.class); + Mockito.doReturn(manifestCrud).when(dataReceiptDefinition).manifestCrud(); + Mockito.doReturn(List.of(modelType1, modelType2, modelType3)) + .when(dataReceiptDefinition) + .modelTypes(); + datastoreWalker = new DatastoreWalker(DatastoreTestUtils.regexpsByName(), + DatastoreTestUtils.datastoreNodesByFullPath()); + Mockito.doReturn(datastoreWalker).when(dataReceiptDefinition).datastoreWalker(); + + modelCrud = Mockito.mock(ModelCrud.class); + Mockito.doReturn(modelCrud).when(dataReceiptDefinition).modelCrud(); + + modelImporter = Mockito.spy(new ModelImporter(dataImporterPath, "importerForTest")); + Mockito.doReturn(modelImporter).when(dataReceiptDefinition).modelImporter(); + Mockito.doReturn(Mockito.mock(AlertService.class)) + .when(dataReceiptDefinition) + .alertService(); + Mockito.doNothing().when(dataReceiptDefinition).updateModelRegistryForPipelineInstance(); + + PipelineInstanceCrud pipelineInstanceCrud = Mockito.mock(PipelineInstanceCrud.class); + Mockito.doReturn(pipelineInstanceCrud).when(dataReceiptDefinition).pipelineInstanceCrud(); + Mockito.when(pipelineInstanceCrud.retrieve(ArgumentMatchers.anyLong())) + .thenReturn(pipelineInstance); + } + + @Test + public void testIsConformingDirectory() { + assertTrue(dataReceiptDefinition.isConformingDelivery()); + Path manifestDir = Paths + .get(ZiggyConfiguration.getInstance().getString(PropertyName.RESULTS_DIR.property())) + .resolve("logs") + .resolve("manifests"); + assertTrue(Files.isDirectory(manifestDir)); + assertTrue(Files + .isRegularFile(manifestDir.resolve("datastore-directory-definition-manifest.xml"))); + assertTrue(Files + .isRegularFile(manifestDir.resolve("datastore-directory-definition-manifest-ack.xml"))); + } + + /** Tests that isConformingDelivery is false if there is no manifest. */ + @Test + public void testMissingManifest() throws IOException { + Files.delete(dataImporterPath.resolve("datastore-directory-definition-manifest.xml")); + assertFalse(dataReceiptDefinition.isConformingDelivery()); + } + + /** Tests that isConformingDelivery is false if the acknowledgement has invalid status. */ + @Test + public void testAckInvalid() { + Mockito.doReturn(false).when(dataReceiptDefinition).acknowledgementTransferStatus(); + assertFalse(dataReceiptDefinition.isConformingDelivery()); + } + + /** + * Tests that isConformingDelivery is false if there are files in the directory that are not in + * the manifest. + */ + @Test + public void testFilesNotInManifest() throws IOException { + Files.createFile(dataImporterPath.resolve("foo.txt")); + assertFalse(dataReceiptDefinition.isConformingDelivery()); + } + + /** Tests that isConformingDelivery is false if the dataset ID has already been used. */ + @Test + public void testManifestIdInvalid() { + Mockito.doReturn(true).when(manifestCrud).datasetIdExists(1L); + assertFalse(dataReceiptDefinition.isConformingDelivery()); + } + + /** Tests that isConformingFile performs as expected. */ + @Test + public void testIsConformingFile() { + + // Files that conform to the design. + assertTrue(dataReceiptDefinition.isConformingFile(dataFile1)); + assertTrue(dataReceiptDefinition.isConformingFile(dataFile2)); + assertTrue(dataReceiptDefinition.isConformingFile(dataFile3)); + assertTrue(dataReceiptDefinition.isConformingFile(modelFile1)); + assertTrue(dataReceiptDefinition.isConformingFile(modelFile2)); + assertTrue(dataReceiptDefinition.isConformingFile(modelFile3)); + assertTrue(dataReceiptDefinition.isConformingFile(modelFile4)); + + // Files that do not conform to the design. + assertFalse(dataReceiptDefinition.isConformingFile( + dataImporterPath.resolve("tess2020321141517-12345_024-geometry.xml"))); + assertFalse(dataReceiptDefinition.isConformingFile( + dataImporterPath.resolve("models").resolve(dataImporterPath.relativize(dataFile1)))); + } + + /** Tests that use of a relative path for the import directory throws exception. */ + @Test(expected = IllegalArgumentException.class) + public void testSetDataImportDirectoryRelativePath() { + dataReceiptDefinition.setDataImportDirectory(testDirectory); + } + + /** Tests that use of a relative path in isConformingFile throws exception. */ + @Test(expected = IllegalArgumentException.class) + public void testIsConformingFileRelativePath() { + dataReceiptDefinition.isConformingFile(testDirectory); + } + + /** Tests that filesForImport finds all files that are to be imported. */ + @Test + public void testFilesForImport() throws IOException { + + // Delete the manifest to emulate the behavior of the methods of DataReceiptDefinition + // that are called in the pipeline module prior to the filesForImport() call. + Files.delete(dataImporterPath.resolve("datastore-directory-definition-manifest.xml")); + List filesForImport = dataReceiptDefinition.filesForImport(); + assertTrue(filesForImport.contains(dataFile1)); + assertTrue(filesForImport.contains(dataFile2)); + assertTrue(filesForImport.contains(dataFile3)); + assertTrue(filesForImport.contains(modelFile1)); + assertTrue(filesForImport.contains(modelFile2)); + assertTrue(filesForImport.contains(modelFile3)); + assertTrue(filesForImport.contains(modelFile4)); + assertEquals(7, filesForImport.size()); + } + + /** Tests that dataFilesForImport finds all data files for import. */ + @Test + public void testDataFilesForImport() throws IOException { + + // Delete the manifest to emulate the behavior of the methods of DataReceiptDefinition + // that are called in the pipeline module prior to the filesForImport() call. + Files.delete(dataImporterPath.resolve("datastore-directory-definition-manifest.xml")); + List filesForImport = dataReceiptDefinition.dataFilesForImport(); + assertTrue(filesForImport.contains(dataFile1)); + assertTrue(filesForImport.contains(dataFile2)); + assertTrue(filesForImport.contains(dataFile3)); + assertEquals(3, filesForImport.size()); + } + + /** Tests that modelFilesForImport finds all model files for import. */ + @Test + public void testModelFilesForImport() throws IOException { + + // Delete the manifest to emulate the behavior of the methods of DataReceiptDefinition + // that are called in the pipeline module prior to the filesForImport() call. + Files.delete(dataImporterPath.resolve("datastore-directory-definition-manifest.xml")); + List filesForImport = dataReceiptDefinition.modelFilesForImport(); + assertTrue(filesForImport.contains(modelFile1)); + assertTrue(filesForImport.contains(modelFile2)); + assertTrue(filesForImport.contains(modelFile3)); + assertTrue(filesForImport.contains(modelFile4)); + assertEquals(4, filesForImport.size()); + } + + /** Tests the actual import of files. */ + @Test + public void testImportFiles() { + ModelRegistry registry = new ModelRegistry(); + Mockito.doReturn(registry).when(modelImporter).unlockedRegistry(); + Mockito.doNothing() + .when(modelImporter) + .persistModelMetadata(ArgumentMatchers.any(ModelMetadata.class)); + Mockito.doReturn(1L) + .when(modelImporter) + .mergeRegistryAndReturnUnlockedId(ArgumentMatchers.any(ModelRegistry.class)); + Mockito.when(modelCrud.retrieveCurrentRegistry()).thenReturn(registry); + assertTrue(dataReceiptDefinition.isConformingDelivery()); + + // All of the original files should still be in the data import directory. + assertTrue(Files.exists(dataFile1)); + assertTrue(Files.exists(dataFile2)); + assertTrue(Files.exists(dataFile3)); + assertTrue(Files.exists(modelFile1)); + assertTrue(Files.exists(modelFile2)); + assertTrue(Files.exists(modelFile3)); + assertTrue(Files.exists(modelFile4)); + + dataReceiptDefinition.importFiles(); + + // Data files should be in the datastore, in directories that match the data import + // directories but with the datastore as root rather than data import. + assertTrue(Files.exists(datastoreRootPath.resolve(dataImporterPath.relativize(dataFile1)))); + assertTrue(Files.exists(datastoreRootPath.resolve(dataImporterPath.relativize(dataFile2)))); + assertTrue(Files.exists(datastoreRootPath.resolve(dataImporterPath.relativize(dataFile3)))); + + // There should be a datastore models directory and 3 subdirs unter that. + assertTrue(Files.isDirectory(datastoreRootPath.resolve("models"))); + assertTrue(Files.isDirectory(datastoreRootPath.resolve("models").resolve("geometry"))); + assertTrue(Files.isDirectory(datastoreRootPath.resolve("models").resolve("calibration"))); + assertTrue(Files.isDirectory(datastoreRootPath.resolve("models").resolve("ravenswood"))); + + assertNotNull(registry.getModels()); + registry.populateXmlFields(); + + // The geometry model should be imported to the geometry directory with no name change. + ModelMetadata metadata = registry.getModels().get(modelType1); + assertEquals("tess2020321141517-12345_025-geometry.xml", metadata.getDatastoreFileName()); + assertEquals("tess2020321141517-12345_025-geometry.xml", metadata.getOriginalFileName()); + assertTrue(Files.isRegularFile(datastoreRootPath.resolve("models") + .resolve(modelType1.getType()) + .resolve(metadata.getDatastoreFileName()))); + + // The calibration model should be in the right place with a different name. + metadata = registry.getModels().get(modelType2); + assertEquals(modelFile3.getFileName().toString(), metadata.getOriginalFileName()); + assertNotEquals(metadata.getOriginalFileName(), metadata.getDatastoreFileName()); + assertTrue(Files.isRegularFile(datastoreRootPath.resolve("models") + .resolve(modelType2.getType()) + .resolve(metadata.getDatastoreFileName()))); + + // The "ravenswood" model should be in the right place with a different name. + metadata = registry.getModels().get(modelType3); + assertEquals(modelFile4.getFileName().toString(), metadata.getOriginalFileName()); + assertNotEquals(metadata.getOriginalFileName(), metadata.getDatastoreFileName()); + assertTrue(Files.isRegularFile(datastoreRootPath.resolve("models") + .resolve(modelType3.getType()) + .resolve(metadata.getDatastoreFileName()))); + + assertEquals(3, registry.getModels().size()); + + // None of the original files should still be in the data import directory. + assertFalse(Files.exists(dataFile1)); + assertFalse(Files.exists(dataFile2)); + assertFalse(Files.exists(dataFile3)); + assertFalse(Files.exists(modelFile1)); + assertFalse(Files.exists(modelFile2)); + assertFalse(Files.exists(modelFile3)); + assertFalse(Files.exists(modelFile4)); + } + + /** + * Creates test files for import in the data receipt directory + */ + private void constructFilesForImport() throws IOException { + + Path sample1 = dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("1:1:A.nc"); + Files.createDirectories(sample1.getParent()); + Files.createFile(sample1); + dataFile1 = sample1; + + sample1 = dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:A") + .resolve("1:1:A.nc"); + Files.createDirectories(sample1.getParent()); + Files.createFile(sample1); + dataFile2 = sample1; + + sample1 = dataImporterPath.resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("ffi") + .resolve("collateral") + .resolve("1:1:B") + .resolve("1:1:B.nc"); + Files.createDirectories(sample1.getParent()); + Files.createFile(sample1); + dataFile3 = sample1; + + // Create model files + setUpModelsForImport(); + + // Create a manifest in the data receipt directory. + Manifest manifest = Manifest.generateManifest(dataImporterPath, 1); + manifest.setName("datastore-directory-definition-manifest.xml"); + if (manifest.getFileCount() > 0) { + manifest.write(dataImporterPath); + } + } + + private void setUpModelTypes() { + // Set up the model type 1 to have a model ID in its name, which is a simple integer, + // and a timestamp in its name + modelType1 = new ModelType(); + modelType1.setFileNameRegex("tess([0-9]{13})-([0-9]{5})_([0-9]{3})-geometry.xml"); + modelType1.setType("geometry"); + modelType1.setVersionNumberGroup(3); + modelType1.setTimestampGroup(1); + modelType1.setSemanticVersionNumber(false); + + // Set up the model type 2 to have a semantic model ID in its name but no timestamp + modelType2 = new ModelType(); + modelType2.setFileNameRegex("calibration-([0-9]+\\.[0-9]+\\.[0-9]+).h5"); + modelType2.setTimestampGroup(-1); + modelType2.setType("calibration"); + modelType2.setVersionNumberGroup(1); + modelType2.setSemanticVersionNumber(true); + + // Set up the model type 3 to have neither ID nor timestamp + modelType3 = new ModelType(); + modelType3.setFileNameRegex("simple-text.h5"); + modelType3.setType("ravenswood"); + modelType3.setTimestampGroup(-1); + } + + private void setUpModelsForImport() throws IOException { + Path modelImportPath = dataImporterPath.resolve("models"); + Files.createDirectories(modelImportPath); + // create the new files to be imported + modelFile1 = modelImportPath.resolve("tess2020321141517-12345_024-geometry.xml"); + Files.createFile(modelFile1); + modelFile2 = modelImportPath.resolve("tess2020321141517-12345_025-geometry.xml"); + Files.createFile(modelFile2); + modelFile3 = modelImportPath.resolve("calibration-4.12.19.h5"); + Files.createFile(modelFile3); + modelFile4 = modelImportPath.resolve("simple-text.h5"); + Files.createFile(modelFile4); + } +} diff --git a/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrudTest.java b/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrudTest.java index 5cf164d..ed90293 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/DatastoreProducerConsumerCrudTest.java @@ -60,8 +60,7 @@ public void testCreateOrUpdateOriginator() { * First test the version that takes a single ResultsOriginator object */ DatabaseTransactionFactory.performTransaction(() -> { - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1, - DatastoreProducerConsumer.DataReceiptFileType.DATA); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1); return null; }); List r0 = (List) DatabaseTransactionFactory @@ -80,8 +79,7 @@ public void testCreateOrUpdateOriginator() { Mockito.when(pipelineTask.getId()).thenReturn(TASK_ID + 1); DatabaseTransactionFactory.performTransaction(() -> { resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, - Sets.newHashSet(PATH_1, PATH_2, PATH_3), - DatastoreProducerConsumer.DataReceiptFileType.DATA); + Sets.newHashSet(PATH_1, PATH_2, PATH_3)); return null; }); @@ -107,12 +105,9 @@ public void testCreateOrUpdateOriginator() { @Test public void retrieveOriginatorsAllSame() { DatabaseTransactionFactory.performTransaction(() -> { - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1, - DatastoreProducerConsumer.DataReceiptFileType.DATA); - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_2, - DatastoreProducerConsumer.DataReceiptFileType.DATA); - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_3, - DatastoreProducerConsumer.DataReceiptFileType.DATA); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_2); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_3); return null; }); @SuppressWarnings("unchecked") @@ -128,14 +123,11 @@ public void retrieveOriginatorsAllSame() { @Test public void retrieveOriginatorsAllDifferent() { DatabaseTransactionFactory.performTransaction(() -> { - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1, - DatastoreProducerConsumer.DataReceiptFileType.DATA); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1); Mockito.when(pipelineTask.getId()).thenReturn(TASK_ID + 1); - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_2, - DatastoreProducerConsumer.DataReceiptFileType.DATA); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_2); Mockito.when(pipelineTask.getId()).thenReturn(TASK_ID + 2); - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_3, - DatastoreProducerConsumer.DataReceiptFileType.DATA); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_3); return null; }); @SuppressWarnings("unchecked") @@ -155,12 +147,9 @@ public void testRetrieveFilesConsumedByTask() { // Create some files in the producer-consumer database table, and add consumers to them. DatabaseTransactionFactory.performTransaction(() -> { - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1, - DatastoreProducerConsumer.DataReceiptFileType.DATA); - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_2, - DatastoreProducerConsumer.DataReceiptFileType.DATA); - resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_3, - DatastoreProducerConsumer.DataReceiptFileType.DATA); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_1); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_2); + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, PATH_3); PipelineTask consumer1 = Mockito.mock(PipelineTask.class); Mockito.when(consumer1.getId()).thenReturn(31L); @@ -188,4 +177,52 @@ public void testRetrieveFilesConsumedByTask() { return null; }); } + + @SuppressWarnings("unchecked") + @Test + public void testRetrieveFilesConsumedByTasks() { + + // Put the files into the database. + DatabaseTransactionFactory.performTransaction(() -> { + resultsOriginatorCrud.createOrUpdateProducer(pipelineTask, + Sets.newHashSet(PATH_1, PATH_2, PATH_3)); + return null; + }); + + // Add comsumers. + Mockito.when(pipelineTask.getId()).thenReturn(100L); + DatabaseTransactionFactory.performTransaction(() -> { + resultsOriginatorCrud.addConsumer(pipelineTask, + new HashSet<>(Set.of(PATH_1.toString()))); + return null; + }); + Mockito.when(pipelineTask.getId()).thenReturn(110L); + DatabaseTransactionFactory.performTransaction(() -> { + resultsOriginatorCrud.addConsumer(pipelineTask, + new HashSet<>(Set.of(PATH_3.toString()))); + return null; + }); + Mockito.when(pipelineTask.getId()).thenReturn(120L); + DatabaseTransactionFactory.performTransaction(() -> { + resultsOriginatorCrud.addConsumer(pipelineTask, + new HashSet<>(Set.of(PATH_2.toString()))); + return null; + }); + + Set filenames = (Set) DatabaseTransactionFactory.performTransaction( + () -> resultsOriginatorCrud.retrieveFilesConsumedByTasks(Set.of(100L, 105L), null)); + assertTrue(filenames.contains(PATH_1.toString())); + assertEquals(1, filenames.size()); + filenames = (Set) DatabaseTransactionFactory + .performTransaction(() -> resultsOriginatorCrud + .retrieveFilesConsumedByTasks(Set.of(100L, 105L, 110L), null)); + assertTrue(filenames.contains(PATH_1.toString())); + assertTrue(filenames.contains(PATH_3.toString())); + assertEquals(2, filenames.size()); + filenames = (Set) DatabaseTransactionFactory.performTransaction( + () -> resultsOriginatorCrud.retrieveFilesConsumedByTasks(Set.of(100L, 105L, 110L), + Set.of(PATH_2.toString(), PATH_3.toString()))); + assertTrue(filenames.contains(PATH_3.toString())); + assertEquals(1, filenames.size()); + } } diff --git a/src/test/java/gov/nasa/ziggy/data/management/DefaultDataImporterTest.java b/src/test/java/gov/nasa/ziggy/data/management/DefaultDataImporterTest.java deleted file mode 100644 index f6ed58e..0000000 --- a/src/test/java/gov/nasa/ziggy/data/management/DefaultDataImporterTest.java +++ /dev/null @@ -1,301 +0,0 @@ -package gov.nasa.ziggy.data.management; - -import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; -import static gov.nasa.ziggy.services.config.PropertyName.USE_SYMLINKS; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.IOException; -import java.nio.file.DirectoryStream; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.RuleChain; -import org.mockito.Mockito; - -import com.google.common.collect.ImmutableSet; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.collections.ZiggyDataType; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.TypedParameter; -import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.uow.DataReceiptUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.UnitOfWork; -import gov.nasa.ziggy.uow.UnitOfWorkGenerator; - -/** - * Unit tests for {@link DefaultDataImporter} class. - * - * @author PT - */ -public class DefaultDataImporterTest { - - private PipelineTask pipelineTask = Mockito.mock(PipelineTask.class); - private Path testDirectory; - private Path dataImporterPath; - private Path dataImporterSubdirPath; - private Path dirForImports; - private Path datastoreRootPath; - private UnitOfWork singleUow = new UnitOfWork(); - private UnitOfWork subdirUow = new UnitOfWork(); - private PipelineDefinitionNode node = Mockito.mock(PipelineDefinitionNode.class); - - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( - DATASTORE_ROOT_DIR, directoryRule, "datastore"); - - @Rule - public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) - .around(datastoreRootDirPropertyRule); - - @Rule - public ZiggyPropertyRule useSymlinksPropertyRule = new ZiggyPropertyRule(USE_SYMLINKS, - (String) null); - - @Before - public void setUp() throws IOException { - - // Construct the necessary directories. - testDirectory = directoryRule.directory(); - dataImporterPath = testDirectory.resolve("data-import"); - dataImporterPath.toFile().mkdirs(); - datastoreRootPath = testDirectory.resolve("datastore"); - datastoreRootPath.toFile().mkdirs(); - dataImporterSubdirPath = dataImporterPath.resolve("sub-dir"); - dataImporterSubdirPath.toFile().mkdirs(); - singleUow.addParameter(new TypedParameter( - UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, - DataReceiptUnitOfWorkGenerator.class.getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); - subdirUow.addParameter(new TypedParameter( - UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, - DataReceiptUnitOfWorkGenerator.class.getCanonicalName(), ZiggyDataType.ZIGGY_STRING)); - subdirUow - .addParameter(new TypedParameter(DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME, - "sub-dir", ZiggyDataType.ZIGGY_STRING)); - singleUow.addParameter(new TypedParameter( - DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME, "", ZiggyDataType.ZIGGY_STRING)); - - // Initialize the data type samples - DataFileTestUtils.initializeDataFileTypeSamples(); - - // construct the files for import - constructFilesForImport(); - - // Construct the data file type information - Mockito.when(node.getInputDataFileTypes()) - .thenReturn(ImmutableSet.of(DataFileTestUtils.dataFileTypeSample1, - DataFileTestUtils.dataFileTypeSample2)); - } - - @Test - public void testSingleUowConstructor() { - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - assertEquals(dataImporterPath, importer.getDataImportPath()); - } - - @Test - public void testDefaultUowConstructor() { - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(subdirUow); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - assertEquals(dataImporterSubdirPath, importer.getDataImportPath()); - } - - @Test - public void testDataFilesInMainDir() throws IOException { - dirForImports = dataImporterPath; - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - Map dataFiles = importer.dataFiles(filenamesInDirectory()); - assertEquals(2, dataFiles.size()); - assertTrue(dataFiles.containsKey(Paths.get("pa-001234567-20-results.h5"))); - assertEquals(dataFiles.get(Paths.get("pa-001234567-20-results.h5")), - Paths.get("pa", "20", "pa-001234567-20-results.h5")); - assertTrue(dataFiles.containsKey(Paths.get("cal-1-1-A-20-results.h5"))); - assertEquals(dataFiles.get(Paths.get("cal-1-1-A-20-results.h5")), - Paths.get("cal", "20", "cal-1-1-A-20-results.h5")); - } - - @Test - public void testDataFilesInSubdir() throws IOException { - dirForImports = dataImporterSubdirPath; - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(subdirUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - Map dataFiles = importer.dataFiles(filenamesInDirectory()); - assertEquals(2, dataFiles.size()); - assertTrue(dataFiles.containsKey(Paths.get("pa-765432100-20-results.h5"))); - assertEquals(dataFiles.get(Paths.get("pa-765432100-20-results.h5")), - Paths.get("pa", "20", "pa-765432100-20-results.h5")); - assertTrue(dataFiles.containsKey(Paths.get("cal-1-1-B-20-results.h5"))); - assertEquals(dataFiles.get(Paths.get("cal-1-1-B-20-results.h5")), - Paths.get("cal", "20", "cal-1-1-B-20-results.h5")); - } - - @Test - public void testImportFilesFromMainDir() throws IOException { - dirForImports = dataImporterPath; - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - Map dataFiles = importer.dataFiles(filenamesInDirectory()); - Set importedFiles = importer.importFiles(dataFiles); - assertEquals(2, importedFiles.size()); - assertTrue(importedFiles.contains(Paths.get("pa-001234567-20-results.h5"))); - assertTrue(importedFiles.contains(Paths.get("cal-1-1-A-20-results.h5"))); - - // The files should be in the correct locations in the datastore - assertTrue(datastoreRootPath.resolve(Paths.get("pa", "20", "pa-001234567-20-results.h5")) - .toFile() - .exists()); - assertTrue(datastoreRootPath.resolve(Paths.get("cal", "20", "cal-1-1-A-20-results.h5")) - .toFile() - .exists()); - - // The PDC file should remain in the import directory - assertEquals(2, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve(Paths.get("sub-dir")).toFile().exists()); - assertTrue(dataImporterPath.resolve(Paths.get("pdc-1-1-21-results.h5")).toFile().exists()); - - // The subdir files should be untouched - assertEquals(2, dataImporterSubdirPath.toFile().listFiles().length); - assertTrue(dataImporterSubdirPath.resolve(Paths.get("pa-765432100-20-results.h5")) - .toFile() - .exists()); - assertTrue( - dataImporterSubdirPath.resolve(Paths.get("cal-1-1-B-20-results.h5")).toFile().exists()); - } - - @Test - public void testImportFilesFromSubdir() throws IOException { - dirForImports = dataImporterSubdirPath; - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(subdirUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - Map dataFiles = importer.dataFiles(filenamesInDirectory()); - Set importedFiles = importer.importFiles(dataFiles); - assertEquals(2, importedFiles.size()); - assertTrue(importedFiles.contains(Paths.get("pa-765432100-20-results.h5"))); - assertTrue(importedFiles.contains(Paths.get("cal-1-1-B-20-results.h5"))); - - // The files should be in the correct locations in the datastore - assertTrue(datastoreRootPath.resolve(Paths.get("pa", "20", "pa-765432100-20-results.h5")) - .toFile() - .exists()); - assertTrue(datastoreRootPath.resolve(Paths.get("cal", "20", "cal-1-1-B-20-results.h5")) - .toFile() - .exists()); - - // The top-level import directory should be untouched - assertEquals(4, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve(Paths.get("sub-dir")).toFile().exists()); - assertTrue(dataImporterPath.resolve(Paths.get("pdc-1-1-21-results.h5")).toFile().exists()); - assertTrue( - dataImporterPath.resolve(Paths.get("pa-001234567-20-results.h5")).toFile().exists()); - assertTrue( - dataImporterPath.resolve(Paths.get("cal-1-1-A-20-results.h5")).toFile().exists()); - - // The subdir should be empty - assertEquals(0, dataImporterSubdirPath.toFile().listFiles().length); - } - - @Test - public void testImportFilesWithFailure() throws IOException { - dirForImports = dataImporterPath; - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(singleUow); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(node); - DefaultDataImporter importer = new DefaultDataImporter(pipelineTask, dataImporterPath, - datastoreRootPath); - importer = Mockito.spy(importer); - Mockito.doReturn(Mockito.mock(AlertService.class)).when(importer).alertService(); - Mockito.doThrow(IOException.class) - .when(importer) - .moveOrSymlink(dataImporterPath.resolve(Paths.get("pa-001234567-20-results.h5")), - datastoreRootPath.resolve(Paths.get("pa", "20", "pa-001234567-20-results.h5"))); - Map dataFiles = importer.dataFiles(filenamesInDirectory()); - Set importedFiles = importer.importFiles(dataFiles); - assertEquals(1, importedFiles.size()); - assertTrue(importedFiles.contains(Paths.get("cal-1-1-A-20-results.h5"))); - - // The file should be in the correct locations in the datastore - assertTrue(datastoreRootPath.resolve(Paths.get("cal", "20", "cal-1-1-A-20-results.h5")) - .toFile() - .exists()); - - // The PDC and PA files should remain in the import directory - assertEquals(3, dataImporterPath.toFile().listFiles().length); - assertTrue(dataImporterPath.resolve(Paths.get("sub-dir")).toFile().exists()); - assertTrue(dataImporterPath.resolve(Paths.get("pdc-1-1-21-results.h5")).toFile().exists()); - assertTrue( - dataImporterPath.resolve(Paths.get("pa-001234567-20-results.h5")).toFile().exists()); - - // The subdir files should be untouched - assertEquals(2, dataImporterSubdirPath.toFile().listFiles().length); - assertTrue(dataImporterSubdirPath.resolve(Paths.get("pa-765432100-20-results.h5")) - .toFile() - .exists()); - assertTrue( - dataImporterSubdirPath.resolve(Paths.get("cal-1-1-B-20-results.h5")).toFile().exists()); - } - - /** - * Creates test files for import in the data receipt directory - */ - private Set constructFilesForImport() throws IOException { - - Set filenames = new HashSet<>(); - // create a couple of files in the DatastoreIdSample1 pattern - File sample1 = new File(dataImporterPath.toFile(), "pa-001234567-20-results.h5"); - File sample2 = new File(dataImporterSubdirPath.toFile(), "pa-765432100-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - filenames.add(sample1.getName()); - filenames.add(sample2.getName()); - - // create a couple of files in the DatastoreIdSample2 pattern - sample1 = new File(dataImporterPath.toFile(), "cal-1-1-A-20-results.h5"); - sample2 = new File(dataImporterSubdirPath.toFile(), "cal-1-1-B-20-results.h5"); - sample1.createNewFile(); - sample2.createNewFile(); - filenames.add(sample1.getName()); - filenames.add(sample2.getName()); - - // create a file that matches neither pattern - sample1 = new File(dataImporterPath.toFile(), "pdc-1-1-21-results.h5"); - sample1.createNewFile(); - filenames.add(sample1.getName()); - return filenames; - } - - private List filenamesInDirectory() throws IOException { - List filenamesInDirectory = new ArrayList<>(); - try (DirectoryStream stream = Files.newDirectoryStream(dirForImports)) { - for (Path path : stream) { - filenamesInDirectory.add(path.getFileName().toString()); - } - } - return filenamesInDirectory; - } -} diff --git a/src/test/java/gov/nasa/ziggy/data/management/ManifestTest.java b/src/test/java/gov/nasa/ziggy/data/management/ManifestTest.java index ce29f05..2bfdc88 100644 --- a/src/test/java/gov/nasa/ziggy/data/management/ManifestTest.java +++ b/src/test/java/gov/nasa/ziggy/data/management/ManifestTest.java @@ -207,11 +207,12 @@ public void testSchema() throws IOException { assertContains(complexTypeContent, ""); } + // TODO : fix this! private void validateManifestFile(ManifestEntry manifestFile) throws IOException { - Path file = DataFileManager.realSourceFile(testDataDir.resolve(manifestFile.getName())); - assertTrue(Files.exists(file)); - assertEquals(Files.size(file), manifestFile.getSize()); - String sha256 = checksumType.checksum(file); - assertEquals(sha256, manifestFile.getChecksum()); +// Path file = DataFileManager.realSourceFile(testDataDir.resolve(manifestFile.getName())); +// assertTrue(Files.exists(file)); +// assertEquals(Files.size(file), manifestFile.getSize()); +// String sha256 = checksumType.checksum(file); +// assertEquals(sha256, manifestFile.getChecksum()); } } diff --git a/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java b/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java new file mode 100644 index 0000000..467d89c --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/metrics/TaskMetricsTest.java @@ -0,0 +1,131 @@ +package gov.nasa.ziggy.metrics; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Map; + +import org.junit.Test; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics; +import gov.nasa.ziggy.pipeline.definition.PipelineTaskMetrics.Units; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; +import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; + +public class TaskMetricsTest { + + private static final long START_MILLIS = 1700000000000L; + private static final long HOUR_MILLIS = 60 * 60 * 1000; + private long totalDuration; + + @SuppressWarnings("unlikely-arg-type") + @Test + public void testHashCodeEquals() { + TaskMetrics taskMetrics = taskMetrics(2); + assertTrue(taskMetrics.equals(taskMetrics)); + assertFalse(taskMetrics.equals(null)); + assertFalse(taskMetrics.equals("a string")); + + assertTrue(taskMetrics(2).equals(taskMetrics(2))); + assertFalse(taskMetrics(2).equals(taskMetrics(3))); + + assertEquals(taskMetrics(2).hashCode(), taskMetrics(2).hashCode()); + assertNotEquals(taskMetrics(2).hashCode(), taskMetrics(3).hashCode()); + } + + @Test + public void testGetCategoryMetrics() { + TaskMetrics taskMetrics = taskMetrics(3); + Map categoryMetrics = taskMetrics.getCategoryMetrics(); + assertEquals(3, categoryMetrics.size()); + checkCategoryMetrics(categoryMetrics.get("module0")); + checkCategoryMetrics(categoryMetrics.get("module1")); + checkCategoryMetrics(categoryMetrics.get("module2")); + } + + private void checkCategoryMetrics(TimeAndPercentile timeAndPercentile) { + assertNotNull(timeAndPercentile); + assertEquals(0.00019, timeAndPercentile.getPercent(), 0.00001); + assertEquals(42.0, timeAndPercentile.getTimeMillis(), 0.0001); + } + + @Test + public void testGetUnallocatedTime() { + TaskMetrics taskMetrics = taskMetrics(3); + TimeAndPercentile unallocatedTime = taskMetrics.getUnallocatedTime(); + assertEquals(99.999, unallocatedTime.getPercent(), 0.01); + assertEquals(2.16E7, unallocatedTime.getTimeMillis(), 0.01E7); + } + + @Test + public void testGetTotalProcessingTimeMillis() { + TaskMetrics taskMetrics = taskMetrics(3); + long totalProcessingTimeMillis = taskMetrics.getTotalProcessingTimeMillis(); + assertEquals(totalDuration, totalProcessingTimeMillis); + } + + /** + * This test reproduces the following error: + * + * 
        +     * org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role:
+     * gov.nasa.ziggy.pipeline.definition.PipelineTask.summaryMetrics: could not initialize proxy -
+     * no Session
+     *
        + * + * This test is commented out as it is beyond the scope of this unit test and slows down the + * test by a couple of orders of magnitude. However, it provides a good example of the + * importance of creating TaskMetrics objects within the same transaction in which the pipeline + * tasks were retrieved. + */ +// @Test + public void testCreateMetricsWithDatabaseTasks() { + DatabaseTransactionFactory.performTransaction(() -> { + new PipelineTaskCrud().persist(pipelineTasks(3)); + return null; + }); + DatabaseTransactionFactory.performTransaction(() -> { + List pipelineTasks = new PipelineTaskCrud().retrieveAll(); + new TaskMetrics(pipelineTasks); + return null; + }); + } + + private TaskMetrics taskMetrics(int taskCount) { + return new TaskMetrics(pipelineTasks(taskCount)); + } + + private List pipelineTasks(int taskCount) { + // Each task starts one hour after the last. The task duration starts at one hour and each + // subsequent task is one hour longer. + ArrayList pipelineTasks = new ArrayList<>(); + long startTime = START_MILLIS; + for (int i = 0; i < taskCount; i++) { + long duration = (i + 1) * HOUR_MILLIS; + totalDuration += duration; + pipelineTasks.add( + pipelineTask("module" + i, new Date(startTime), new Date(startTime + duration))); + startTime += duration + HOUR_MILLIS; + } + return pipelineTasks; + } + + private PipelineTask pipelineTask(String moduleName, Date start, Date end) { + PipelineTask pipelineTask = new PipelineTask(); + pipelineTask.setStartProcessingTime(start); + pipelineTask.setEndProcessingTime(end); + pipelineTask.setSummaryMetrics(summaryMetrics(moduleName)); + return pipelineTask; + } + + private List summaryMetrics(String moduleName) { + return List.of(new PipelineTaskMetrics(moduleName, 42, Units.TIME)); + } +} diff --git a/src/test/java/gov/nasa/ziggy/models/ModelImporterTest.java b/src/test/java/gov/nasa/ziggy/models/ModelImporterTest.java index 4a195ab..4b93e08 100644 --- a/src/test/java/gov/nasa/ziggy/models/ModelImporterTest.java +++ b/src/test/java/gov/nasa/ziggy/models/ModelImporterTest.java @@ -56,7 +56,7 @@ public class ModelImporterTest { @Before public void setup() throws IOException { - datastoreRoot = new File(ziggyDatastorePropertyRule.getProperty()); + datastoreRoot = new File(ziggyDatastorePropertyRule.getValue()); // Set up the model type 1 to have a model ID in its name, which is a simple integer, // and a timestamp in its name modelType1 = new ModelType(); @@ -90,7 +90,10 @@ public void setup() throws IOException { ModelMetadata modelMetadata3 = new ModelMetadata(modelType3, filename3, "zinfandel", null); // Initialize the datastore - modelImportDirectory = directoryRule.directory().resolve("modelImportDirectory").toFile(); + modelImportDirectory = directoryRule.directory() + .resolve(ModelImporter.DATASTORE_MODELS_SUBDIR_NAME) + .toAbsolutePath() + .toFile(); modelImportDirectory.mkdirs(); // Create the database objects @@ -120,10 +123,10 @@ public void setup() throws IOException { public void testImportWithCurrentRegistryUnlocked() { // Import the models - ModelImporter modelImporter = new ModelImporter(modelImportDirectory.getAbsolutePath(), - "unit test"); + ModelImporter modelImporter = new ModelImporter( + modelImportDirectory.toPath().toAbsolutePath().getParent(), "unit test"); DatabaseTransactionFactory.performTransaction(() -> { - modelImporter.importModels(filenamesInDirectory()); + modelImporter.importModels(filesInDirectory()); return null; }); @@ -214,10 +217,10 @@ public void testImportWithCurrentRegistryLocked() { }); // Import the models - ModelImporter modelImporter = new ModelImporter(modelImportDirectory.getAbsolutePath(), - "unit test"); + ModelImporter modelImporter = new ModelImporter( + modelImportDirectory.toPath().toAbsolutePath().getParent(), "unit test"); DatabaseTransactionFactory.performTransaction(() -> { - modelImporter.importModels(filenamesInDirectory()); + modelImporter.importModels(filesInDirectory()); return null; }); @@ -301,8 +304,8 @@ public void testImportWithCurrentRegistryLocked() { public void testImportWithFailures() throws IOException { // For this exercise we need a spy for the importer - ModelImporter importer = new ModelImporter(modelImportDirectory.getAbsolutePath(), - "unit test"); + ModelImporter importer = new ModelImporter( + modelImportDirectory.toPath().toAbsolutePath().getParent(), "unit test"); final ModelImporter modelImporter = Mockito.spy(importer); Path destFileToFlunk = Paths.get(datastoreRoot.toString(), "models", "geometry", "tess2020321141517-12345_025-geometry.xml"); @@ -310,11 +313,11 @@ public void testImportWithFailures() throws IOException { "tess2020321141517-12345_025-geometry.xml"); Mockito.doThrow(IOException.class) .when(modelImporter) - .moveOrSymlink(srcFileToFlunk.toAbsolutePath(), destFileToFlunk); + .move(srcFileToFlunk.toAbsolutePath(), destFileToFlunk.toAbsolutePath()); // Perform the import DatabaseTransactionFactory.performTransaction(() -> { - modelImporter.importModels(filenamesInDirectory()); + modelImporter.importModels(filesInDirectory()); return null; }); @@ -334,7 +337,8 @@ public void testImportWithFailures() throws IOException { // Check that there is one file logged with the importer as failed List failedImports = modelImporter.getFailedImports(); assertEquals(1, failedImports.size()); - assertEquals("tess2020321141517-12345_025-geometry.xml", failedImports.get(0).toString()); + assertEquals("tess2020321141517-12345_025-geometry.xml", + failedImports.get(0).getFileName().toString()); // Check that the failed import is still in the source directory File[] remainingFiles = modelImportDirectory.listFiles(); @@ -394,14 +398,14 @@ public void testImportWithFailures() throws IOException { assertEquals(modelFilename, ravenswoodModels[0].getName()); } - private List filenamesInDirectory() throws IOException { - List filenamesInDirectory = new ArrayList<>(); + private List filesInDirectory() throws IOException { + List filesInDirectory = new ArrayList<>(); try (DirectoryStream stream = java.nio.file.Files .newDirectoryStream(modelImportDirectory.toPath())) { for (Path path : stream) { - filenamesInDirectory.add(path.getFileName().toString()); + filesInDirectory.add(path.toAbsolutePath()); } } - return filenamesInDirectory; + return filesInDirectory; } } diff --git a/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java index f6b6fcc..4000373 100644 --- a/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/AlgorithmExecutorTest.java @@ -3,13 +3,16 @@ import static org.junit.Assert.assertTrue; import org.junit.Test; +import org.mockito.ArgumentMatchers; import org.mockito.Mockito; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.module.remote.nas.NasExecutor; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.PipelineTask.ProcessingSummary; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.ProcessingSummaryOperations; /** @@ -37,11 +40,16 @@ public void testNewInstanceNullTask() { // returned. @Test public void testNewInstanceNullRemoteParameters() { - PipelineTask task = new PipelineTask(); + PipelineTask task = Mockito.spy(PipelineTask.class); + Mockito.doReturn(new PipelineDefinitionNode()).when(task).pipelineDefinitionNode(); ParameterSetCrud parameterSetCrud = Mockito.mock(ParameterSetCrud.class); - Mockito.when(parameterSetCrud.retrieveRemoteParameters(task)).thenReturn(null); + PipelineDefinitionNodeCrud nodeDefCrud = Mockito.mock(PipelineDefinitionNodeCrud.class); + Mockito + .when(nodeDefCrud + .retrieveExecutionResources(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(new PipelineDefinitionNodeExecutionResources("dummy", "dummy")); AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, parameterSetCrud, - new ProcessingSummaryOperations()); + nodeDefCrud, new ProcessingSummaryOperations()); assertTrue(executor instanceof LocalAlgorithmExecutor); } @@ -49,13 +57,19 @@ public void testNewInstanceNullRemoteParameters() { // is returned. @Test public void testNewInstanceRemoteDisabled() { - PipelineTask task = new PipelineTask(); + PipelineTask task = Mockito.spy(PipelineTask.class); + Mockito.doReturn(new PipelineDefinitionNode()).when(task).pipelineDefinitionNode(); ParameterSetCrud parameterSetCrud = Mockito.mock(ParameterSetCrud.class); - RemoteParameters remotePars = new RemoteParameters(); - remotePars.setEnabled(false); - Mockito.when(parameterSetCrud.retrieveRemoteParameters(task)).thenReturn(remotePars); + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setRemoteExecutionEnabled(false); + PipelineDefinitionNodeCrud nodeDefCrud = Mockito.mock(PipelineDefinitionNodeCrud.class); + Mockito + .when(nodeDefCrud + .retrieveExecutionResources(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(executionResources); AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, parameterSetCrud, - new ProcessingSummaryOperations()); + nodeDefCrud, new ProcessingSummaryOperations()); assertTrue(executor instanceof LocalAlgorithmExecutor); } @@ -65,17 +79,24 @@ public void testNewInstanceRemoteDisabled() { public void testNewInstanceTooFewSubtasks() { PipelineTask task = Mockito.mock(PipelineTask.class); Mockito.when(task.getId()).thenReturn(100L); + Mockito.when(task.pipelineDefinitionNode()).thenReturn(new PipelineDefinitionNode()); ParameterSetCrud parameterSetCrud = Mockito.mock(ParameterSetCrud.class); - RemoteParameters remotePars = new RemoteParameters(); - remotePars.setEnabled(true); - remotePars.setMinSubtasksForRemoteExecution(5); - Mockito.when(parameterSetCrud.retrieveRemoteParameters(task)).thenReturn(remotePars); + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setRemoteExecutionEnabled(true); + executionResources.setMinSubtasksForRemoteExecution(5); + PipelineDefinitionNodeCrud nodeDefCrud = Mockito.mock(PipelineDefinitionNodeCrud.class); + Mockito + .when(nodeDefCrud + .retrieveExecutionResources(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(executionResources); ProcessingSummaryOperations sumOps = Mockito.mock(ProcessingSummaryOperations.class); ProcessingSummary summary = Mockito.mock(PipelineTask.ProcessingSummary.class); Mockito.when(summary.getTotalSubtaskCount()).thenReturn(100); Mockito.when(summary.getCompletedSubtaskCount()).thenReturn(99); Mockito.when(sumOps.processingSummary(100L)).thenReturn(summary); - AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, parameterSetCrud, sumOps); + AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, parameterSetCrud, + nodeDefCrud, sumOps); assertTrue(executor instanceof LocalAlgorithmExecutor); } @@ -85,17 +106,24 @@ public void testNewInstanceTooFewSubtasks() { public void testNewInstanceRemote() { PipelineTask task = Mockito.mock(PipelineTask.class); Mockito.when(task.getId()).thenReturn(100L); + Mockito.when(task.pipelineDefinitionNode()).thenReturn(new PipelineDefinitionNode()); ParameterSetCrud parameterSetCrud = Mockito.mock(ParameterSetCrud.class); - RemoteParameters remotePars = new RemoteParameters(); - remotePars.setEnabled(true); - remotePars.setMinSubtasksForRemoteExecution(5); - Mockito.when(parameterSetCrud.retrieveRemoteParameters(task)).thenReturn(remotePars); + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setRemoteExecutionEnabled(true); + executionResources.setMinSubtasksForRemoteExecution(5); + PipelineDefinitionNodeCrud nodeDefCrud = Mockito.mock(PipelineDefinitionNodeCrud.class); + Mockito + .when(nodeDefCrud + .retrieveExecutionResources(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(executionResources); ProcessingSummaryOperations sumOps = Mockito.mock(ProcessingSummaryOperations.class); ProcessingSummary summary = Mockito.mock(PipelineTask.ProcessingSummary.class); Mockito.when(summary.getTotalSubtaskCount()).thenReturn(100); Mockito.when(summary.getCompletedSubtaskCount()).thenReturn(90); Mockito.when(sumOps.processingSummary(100L)).thenReturn(summary); - AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, parameterSetCrud, sumOps); + AlgorithmExecutor executor = AlgorithmExecutor.newInstance(task, parameterSetCrud, + nodeDefCrud, sumOps); assertTrue(executor instanceof NasExecutor); } } diff --git a/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java b/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java index 1e7f13a..751715c 100644 --- a/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/AlgorithmMonitorTest.java @@ -6,6 +6,7 @@ import java.io.IOException; import java.nio.file.Files; +import java.util.HashMap; import java.util.List; import org.apache.commons.configuration2.ex.ConfigurationException; @@ -21,15 +22,20 @@ import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.module.AlgorithmExecutor.AlgorithmType; +import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.PipelineOperations; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; +import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; import gov.nasa.ziggy.pipeline.definition.PipelineModule.RunMode; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.ProcessingState; import gov.nasa.ziggy.pipeline.definition.TaskCounts; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineInstanceNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; @@ -58,6 +64,8 @@ public class AlgorithmMonitorTest { private AlertService alertService; private ProcessingSummaryOperations attrOps; private PipelineInstanceNodeCrud nodeCrud; + private PipelineDefinitionNodeExecutionResources resources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); public TaskRequestHandlerLifecycleManager lifecycleManager = new InstrumentedTaskRequestHandlerLifecycleManager(); @@ -80,14 +88,20 @@ public void setUp() throws IOException, ConfigurationException { Mockito.when(monitor.jobMonitor()).thenReturn(jobMonitor); Mockito.when(monitor.pollingIntervalMillis()).thenReturn(50L); Mockito.doReturn(false).when(monitor).taskIsKilled(ArgumentMatchers.isA(long.class)); - pipelineTask = Mockito.mock(PipelineTask.class); - Mockito.when(pipelineTask.pipelineInstanceId()).thenReturn(50L); - Mockito.when(pipelineTask.getId()).thenReturn(100L); - Mockito.when(pipelineTask.getModuleName()).thenReturn("dummy"); - Mockito.when(pipelineTask.getPipelineInstance()) - .thenReturn(Mockito.mock(PipelineInstance.class)); - Mockito.when(pipelineTask.getPipelineDefinitionNode()) - .thenReturn(Mockito.mock(PipelineDefinitionNode.class)); + pipelineTask = Mockito.spy(PipelineTask.class); + Mockito.doReturn(50L).when(pipelineTask).pipelineInstanceId(); + Mockito.doReturn(100L).when(pipelineTask).getId(); + Mockito.doReturn("dummy").when(pipelineTask).getModuleName(); + Mockito.doReturn(Mockito.mock(PipelineInstance.class)) + .when(pipelineTask) + .getPipelineInstance(); + Mockito.doReturn(100).when(pipelineTask).exeTimeoutSeconds(); + Mockito.doReturn(new HashMap<>()) + .when(pipelineTask) + .getPipelineParameterSets(); + Mockito.doReturn(new HashMap<>()) + .when(pipelineTask) + .getModuleParameterSets(); pipelineTaskCrud = Mockito.mock(PipelineTaskCrud.class); Mockito.when(pipelineTaskCrud.retrieve(100L)).thenReturn(pipelineTask); Mockito.when(pipelineTaskCrud.merge(ArgumentMatchers.isA(PipelineTask.class))) @@ -100,9 +114,9 @@ public void setUp() throws IOException, ConfigurationException { .thenReturn(new TaskCounts(50, 50, 10, 1)); Mockito.when(monitor.pipelineOperations()).thenReturn(pipelineOperations); pipelineExecutor = Mockito.spy(PipelineExecutor.class); - pipelineExecutor.setPipelineTaskCrud(pipelineTaskCrud); - pipelineExecutor.setPipelineInstanceNodeCrud(nodeCrud); - pipelineExecutor.setPipelineOperations(pipelineOperations); + Mockito.doReturn(pipelineTaskCrud).when(pipelineExecutor).pipelineTaskCrud(); + Mockito.doReturn(nodeCrud).when(pipelineExecutor).pipelineInstanceNodeCrud(); + Mockito.doReturn(pipelineOperations).when(pipelineExecutor).pipelineOperations(); Mockito.doNothing() .when(pipelineExecutor) .removeTaskFromKilledTaskList(ArgumentMatchers.isA(long.class)); @@ -112,7 +126,16 @@ public void setUp() throws IOException, ConfigurationException { .thenReturn(Mockito.mock(PipelineInstanceNode.class)); Mockito.when(pipelineExecutor.taskRequestEnabled()).thenReturn(false); attrOps = Mockito.mock(ProcessingSummaryOperations.class); - pipelineExecutor.setPipelineInstanceCrud(Mockito.mock(PipelineInstanceCrud.class)); + Mockito.doReturn(Mockito.mock(PipelineInstanceCrud.class)) + .when(pipelineExecutor) + .pipelineInstanceCrud(); + PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud = Mockito + .mock(PipelineDefinitionNodeCrud.class); + PipelineDefinitionNode pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); + Mockito.when(monitor.pipelineDefinitionNodeCrud()).thenReturn(pipelineDefinitionNodeCrud); + Mockito.doReturn(pipelineDefinitionNode).when(pipelineTask).pipelineDefinitionNode(); + Mockito.when(pipelineDefinitionNodeCrud.retrieveExecutionResources(pipelineDefinitionNode)) + .thenReturn(resources); Mockito.when(monitor.pipelineExecutor()).thenReturn(pipelineExecutor); Mockito.when(monitor.processingSummaryOperations()).thenReturn(attrOps); Mockito.when(monitor.pipelineTaskOperations()) @@ -184,7 +207,7 @@ public void testStateFileUpdate() @Test public void testExecutionFailed() throws ConfigurationException, IOException, InterruptedException { - Mockito.when(pipelineTask.maxFailedSubtasks()).thenReturn(4); + resources.setMaxFailedSubtaskCount(4); stateFile.setState(StateFile.State.PROCESSING); stateFile.setNumComplete(90); stateFile.setNumFailed(5); @@ -214,7 +237,7 @@ public void testExecutionFailed() public void testExecutionCompleteTooManyErrors() throws ConfigurationException, IOException, InterruptedException { - Mockito.when(pipelineTask.maxFailedSubtasks()).thenReturn(4); + resources.setMaxFailedSubtaskCount(4); stateFile.setState(StateFile.State.COMPLETE); stateFile.setNumComplete(95); stateFile.setNumFailed(5); @@ -240,7 +263,7 @@ public void testExecutionCompleteTooManyErrors() public void testExecutionCompleteTooManyMissed() throws ConfigurationException, IOException, InterruptedException { - Mockito.when(pipelineTask.maxFailedSubtasks()).thenReturn(4); + resources.setMaxFailedSubtaskCount(4); stateFile.setState(StateFile.State.COMPLETE); stateFile.setNumComplete(95); stateFile.setNumFailed(0); @@ -266,7 +289,7 @@ public void testExecutionCompleteTooManyMissed() public void testExecutionComplete() throws ConfigurationException, IOException, InterruptedException { - Mockito.when(pipelineTask.maxFailedSubtasks()).thenReturn(6); + resources.setMaxFailedSubtaskCount(6); stateFile.setState(StateFile.State.COMPLETE); stateFile.setNumComplete(95); stateFile.setNumFailed(5); @@ -291,8 +314,8 @@ public void testAutoResubmit() int taskCount = lifecycleManager.taskRequestSize(); assertEquals(0, taskCount); - Mockito.when(pipelineTask.maxFailedSubtasks()).thenReturn(4); - Mockito.when(pipelineTask.maxAutoResubmits()).thenReturn(3); + resources.setMaxFailedSubtaskCount(4); + resources.setMaxAutoResubmits(3); Mockito.when(pipelineTask.getAutoResubmitCount()).thenReturn(1); Mockito.when(pipelineTask.getState()).thenReturn(PipelineTask.State.ERROR); Mockito.doNothing() @@ -328,8 +351,8 @@ public void testAutoResubmit() public void testOutOfAutoResubmits() throws ConfigurationException, IOException, InterruptedException { - Mockito.when(pipelineTask.maxFailedSubtasks()).thenReturn(4); - Mockito.when(pipelineTask.maxAutoResubmits()).thenReturn(3); + Mockito.when(pipelineTask.getMaxFailedSubtaskCount()).thenReturn(4); + Mockito.when(pipelineTask.getMaxAutoResubmits()).thenReturn(3); Mockito.when(pipelineTask.getAutoResubmitCount()).thenReturn(3); Mockito.when(pipelineTask.getState()).thenReturn(PipelineTask.State.ERROR); stateFile.setState(StateFile.State.COMPLETE); diff --git a/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java b/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java index 2b3185c..eab0db1 100644 --- a/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java +++ b/src/test/java/gov/nasa/ziggy/module/ComputeNodeMasterTest.java @@ -58,7 +58,7 @@ public class ComputeNodeMasterTest { + "-" + MODULE_NAME; private PipelineTask pipelineTask; - private TaskConfigurationManager inputsHandler; + private TaskConfiguration inputsHandler; private SubtaskServer subtaskServer; private ExecutorService subtaskMasterThreadPool; private TaskLog taskLog; @@ -101,15 +101,14 @@ public void setUp() throws Exception { DirectoryProperties.algorithmLogsDir().resolve(TASK_DIR_NAME + ".log").toString()); // Create mocked instances - inputsHandler = mock(TaskConfigurationManager.class); - when(inputsHandler.allSubTaskDirectories()).thenReturn(subtaskDirFiles); + inputsHandler = mock(TaskConfiguration.class); subtaskServer = mock(SubtaskServer.class); subtaskMasterThreadPool = mock(ExecutorService.class); // Create the ComputeNodeMaster. To be precise, create an instance of the // class that is a Mockito spy. computeNodeMaster = Mockito.spy(new ComputeNodeMaster(taskDir.toString(), taskLog)); - doReturn(inputsHandler).when(computeNodeMaster).getInputsHandler(); + doReturn(inputsHandler).when(computeNodeMaster).getTaskConfiguration(); doReturn(subtaskServer).when(computeNodeMaster).subtaskServer(); doReturn(subtaskMasterThreadPool).when(computeNodeMaster).subtaskMasterThreadPool(); doReturn(true).when(computeNodeMaster) @@ -274,7 +273,6 @@ public void testMonitoringWhenSubtasksRemain() // All of the semaphore permits should still be in use. assertEquals(0, computeNodeMaster.getSemaphorePermits()); - } /** @@ -367,7 +365,6 @@ public void testMonitoringCompletedTask() assertEquals(5, computeNodeMaster.getStateFileNumTotal()); assertEquals(3, computeNodeMaster.getStateFileNumComplete()); assertEquals(2, computeNodeMaster.getStateFileNumFailed()); - } /** @@ -393,7 +390,6 @@ public void testMonitoringSubtaskMastersDone() // The countdown latch should no longer be waiting. assertEquals(0, computeNodeMaster.getCountDownLatchCount()); - } @Test diff --git a/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java b/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java new file mode 100644 index 0000000..db6c9a0 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineInputsTest.java @@ -0,0 +1,439 @@ +package gov.nasa.ziggy.module; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.RuleChain; +import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; + +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager; +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.data.datastore.DatastoreTestUtils; +import gov.nasa.ziggy.data.datastore.DatastoreWalker; +import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; +import gov.nasa.ziggy.module.io.ProxyIgnore; +import gov.nasa.ziggy.parameters.Parameters; +import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.pipeline.PipelineExecutor; +import gov.nasa.ziggy.pipeline.definition.ClassWrapper; +import gov.nasa.ziggy.pipeline.definition.ModelMetadata; +import gov.nasa.ziggy.pipeline.definition.ModelType; +import gov.nasa.ziggy.pipeline.definition.ParameterSet; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineInstance; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.services.alert.AlertService; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.config.PropertyName; +import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; +import gov.nasa.ziggy.uow.UnitOfWork; + +/** + * Unit test class for {@link DatastoreDirectoryPipelineInputs}. + * + * @author PT + */ +public class DatastoreDirectoryPipelineInputsTest { + + private static final int EXPECTED_SUBTASK_COUNT = 7; + private PipelineTask pipelineTask; + private PipelineInstance pipelineInstance; + private PipelineInstanceNode pipelineInstanceNode; + private PipelineDefinitionNode pipelineDefinitionNode; + private PipelineInputsForTest pipelineInputs; + private Path taskDirectory; + private Map regexpsByName; + private DatastoreWalker datastoreWalker; + private DatastoreFileManager datastoreFileManager; + private Map regexpValueByName = new HashMap<>(); + private ModelMetadata modelMetadata; + private Map> filesForSubtasks; + private TaskConfiguration taskConfiguration; + private DataFileType calibratedCollateralPixelDataFileType; + + public ZiggyDirectoryRule ziggyDirectoryRule = new ZiggyDirectoryRule(); + + public ZiggyPropertyRule datastoreRootProperty = new ZiggyPropertyRule( + PropertyName.DATASTORE_ROOT_DIR, ziggyDirectoryRule, "datastore"); + + public ZiggyPropertyRule taskDirRule = new ZiggyPropertyRule(PropertyName.RESULTS_DIR, + ziggyDirectoryRule, "pipeline-results"); + + @Rule + public final RuleChain testRuleChain = RuleChain.outerRule(ziggyDirectoryRule) + .around(datastoreRootProperty) + .around(taskDirRule); + + @Before + public void setup() throws IOException { + + pipelineTask = Mockito.mock(PipelineTask.class); + taskDirectory = DirectoryProperties.taskDataDir(); + + regexpsByName = DatastoreTestUtils.regexpsByName(); + datastoreWalker = new DatastoreWalker(regexpsByName, + DatastoreTestUtils.datastoreNodesByFullPath()); + + // Create datastore directories. + DatastoreTestUtils.createDatastoreDirectories(); + + // Get and update data file types. + Map dataFileTypes = DatastoreTestUtils.dataFileTypesByName(); + DataFileType uncalibratedSciencePixelDataFileType = dataFileTypes + .get("uncalibrated science pixel values"); + uncalibratedSciencePixelDataFileType + .setFileNameRegexp("uncalibrated-pixels-[0-9]+\\.science\\.nc"); + DataFileType uncalibratedCollateralPixelDataFileType = dataFileTypes + .get("uncalibrated collateral pixel values"); + uncalibratedCollateralPixelDataFileType + .setFileNameRegexp("uncalibrated-pixels-[0-9]+\\.collateral\\.nc"); + DataFileType allFilesAllSubtasksDataFileType = dataFileTypes + .get("calibrated science pixel values"); + allFilesAllSubtasksDataFileType.setFileNameRegexp("everyone-needs-me-[0-9.nc"); + calibratedCollateralPixelDataFileType = dataFileTypes + .get("calibrated collateral pixel values"); + calibratedCollateralPixelDataFileType + .setFileNameRegexp("calibrated-pixels-[0-9]+\\.collateral\\.nc"); + + // Construct the Map from regexp name to value. + regexpValueByName.put("sector", "sector-0002"); + regexpValueByName.put("cadenceType", "target"); + regexpValueByName.put("channel", "1:1:A"); + for (Map.Entry regexpEntry : regexpValueByName.entrySet()) { + regexpsByName.get(regexpEntry.getKey()).setInclude(regexpEntry.getValue()); + } + + // Create datastore files. + constructDatastoreFiles(uncalibratedSciencePixelDataFileType, EXPECTED_SUBTASK_COUNT + 1, + "uncalibrated-pixels-", ".science.nc"); + constructDatastoreFiles(uncalibratedCollateralPixelDataFileType, EXPECTED_SUBTASK_COUNT, + "uncalibrated-pixels-", ".collateral.nc"); + constructDatastoreFiles(allFilesAllSubtasksDataFileType, 2, "everyone-needs-me-", ".nc"); + + // Construct a model type and model metadata. + ModelType modelType = new ModelType(); + modelType.setType("test"); + modelMetadata = new ModelMetadata(); + modelMetadata.setModelType(modelType); + modelMetadata.setOriginalFileName("foo"); + modelMetadata.setDatastoreFileName("bar"); + Files.createDirectories(modelMetadata.datastoreModelPath().getParent()); + Files.createFile(modelMetadata.datastoreModelPath()); + + // Create the PipelineTask. + pipelineTask = Mockito.mock(PipelineTask.class); + pipelineInstance = Mockito.mock(PipelineInstance.class); + pipelineInstanceNode = Mockito.mock(PipelineInstanceNode.class); + pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); + Mockito.when(pipelineTask.getModuleName()).thenReturn("testmod"); + Mockito.when(pipelineTask.getPipelineInstance()).thenReturn(pipelineInstance); + Mockito.when(pipelineTask.getPipelineInstanceNode()).thenReturn(pipelineInstanceNode); + Mockito.when(pipelineTask.pipelineDefinitionNode()).thenReturn(pipelineDefinitionNode); + Mockito.when(pipelineInstanceNode.getPipelineDefinitionNode()) + .thenReturn(pipelineDefinitionNode); + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(uncalibratedSciencePixelDataFileType, + uncalibratedCollateralPixelDataFileType, allFilesAllSubtasksDataFileType)); + Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) + .thenReturn(Set.of(calibratedCollateralPixelDataFileType)); + Mockito.when(pipelineDefinitionNode.getModelTypes()).thenReturn(Set.of(modelType)); + + // Create the parameter sets. + Mockito.when(pipelineInstance.getPipelineParameterSets()) + .thenReturn(pipelineParameterSets()); + Mockito.when(pipelineInstanceNode.getModuleParameterSets()) + .thenReturn(moduleParameterSets()); + + // Construct the UOW. + DatastoreDirectoryUnitOfWorkGenerator uowGenerator = Mockito + .spy(DatastoreDirectoryUnitOfWorkGenerator.class); + Mockito.doReturn(datastoreWalker).when(uowGenerator).datastoreWalker(); + List uows = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); + Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(uows.get(0)); + + // Construct mocked DatastoreFileManager. + datastoreFileManager = Mockito.mock(DatastoreFileManager.class); + Mockito.when(datastoreFileManager.taskDirectory()).thenReturn(taskDirectory); + filesForSubtasks = new HashMap<>(); + populateFilesForSubtasks(EXPECTED_SUBTASK_COUNT); + Mockito.when(datastoreFileManager.filesForSubtasks()).thenReturn(filesForSubtasks); + Map modelFilesForTask = new HashMap<>(); + modelFilesForTask.put(modelMetadata.datastoreModelPath(), "foo"); + Mockito.when(datastoreFileManager.modelFilesForTask()).thenReturn(modelFilesForTask); + Mockito + .when(datastoreFileManager.copyDatastoreFilesToTaskDirectory( + ArgumentMatchers.anyCollection(), ArgumentMatchers.anyMap())) + .thenReturn(pathsBySubtaskDirectory(EXPECTED_SUBTASK_COUNT)); + + // Construct the pipeline inputs. We can't use the standard method of a Mockito spy + // applied to a PipelineInputs instance because Mockito's spy and the HDF5 API don't + // work together. Hence we need to have a subclass of DatastoreDirectoryPipelineInputs + // that takes all the necessary arguments and makes correct use of them. + pipelineInputs = new PipelineInputsForTest(datastoreFileManager, + Mockito.mock(AlertService.class), pipelineTask); + + taskConfiguration = new TaskConfiguration(); + } + + /** Constructs a collection of zero-length files in the datastore. */ + private void constructDatastoreFiles(DataFileType dataFileType, int fileCount, + String filenamePrefix, String filenameSuffix) throws IOException { + Path datastorePath = datastoreWalker.pathFromLocationAndRegexpValues(regexpValueByName, + dataFileType.getLocation()); + for (int fileCounter = 0; fileCounter < fileCount; fileCounter++) { + String filename = filenamePrefix + fileCounter + filenameSuffix; + Files.createDirectories(datastorePath); + Files.createFile(datastorePath.resolve(filename)); + } + } + + private void populateFilesForSubtasks(int subtaskCount) { + for (int subtaskIndex = 0; subtaskIndex < subtaskCount; subtaskIndex++) { + String baseName = "uncalibrated-pixels-" + subtaskIndex; + Set subtaskFiles = new HashSet<>(); + subtaskFiles.add(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("everyone-needs-me-0.nc")); + subtaskFiles.add(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("cal") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve("everyone-needs-me-1.nc")); + subtaskFiles.add(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .resolve(baseName + ".science.nc")); + subtaskFiles.add(DirectoryProperties.datastoreRootDir() + .toAbsolutePath() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve("1:1:A") + .resolve(baseName + ".collateral.nc")); + filesForSubtasks.put(baseName, subtaskFiles); + } + } + + private Map> pathsBySubtaskDirectory(int subtaskCount) throws IOException { + Map> pathsBySubtaskDirectory = new HashMap<>(); + for (int subtaskIndex = 0; subtaskIndex < subtaskCount; subtaskIndex++) { + Path subtaskPath = taskDirectory.resolve("st-" + subtaskIndex); + Files.createDirectories(subtaskPath); + String baseName = "uncalibrated-pixels-" + subtaskIndex; + pathsBySubtaskDirectory.put(subtaskPath, filesForSubtasks.get(baseName)); + } + return pathsBySubtaskDirectory; + } + + /** Exercises the copyDatastoreFilesToTaskDirectory() method. */ + @Test + public void testCopyDatastoreFilesToTaskDirectory() throws IOException { + + // Note that we don't actually copy any files to the subtask directory. + // That capability has been fully tested in the DatastoreFileManager. + // Here we just want to see that the HDF5 file in each subtask directory + // contains what we expect to see. + pipelineInputs.copyDatastoreFilesToTaskDirectory(taskConfiguration, taskDirectory); + assertEquals(EXPECTED_SUBTASK_COUNT, taskConfiguration.getSubtaskCount()); + Hdf5ModuleInterface hdf5ModuleInterface = new Hdf5ModuleInterface(); + + for (int subtaskIndex = 0; subtaskIndex < EXPECTED_SUBTASK_COUNT; subtaskIndex++) { + assertTrue(Files + .exists(taskDirectory.resolve("st-" + subtaskIndex).resolve("testmod-inputs.h5"))); + PipelineInputsForTest storedInputs = new PipelineInputsForTest(datastoreFileManager, + Mockito.mock(AlertService.class), pipelineTask); + hdf5ModuleInterface.readFile( + taskDirectory.resolve("st-" + subtaskIndex).resolve("testmod-inputs.h5").toFile(), + storedInputs, false); + + assertTrue(storedInputs.getModelFilenames().contains("foo")); + assertEquals(1, storedInputs.getModelFilenames().size()); + + assertTrue(storedInputs.getDataFilenames() + .contains("uncalibrated-pixels-" + subtaskIndex + ".science.nc")); + assertTrue(storedInputs.getDataFilenames() + .contains("uncalibrated-pixels-" + subtaskIndex + ".collateral.nc")); + assertTrue(storedInputs.getDataFilenames().contains("everyone-needs-me-0.nc")); + assertTrue(storedInputs.getDataFilenames().contains("everyone-needs-me-1.nc")); + assertEquals(4, storedInputs.getDataFilenames().size()); + + List pars = storedInputs.getModuleParameters() + .getModuleParameters(); + if (pars.get(0) instanceof Params1) { + assertTrue(pars.get(1) instanceof Params2); + } else { + assertTrue(pars.get(0) instanceof Params2); + assertTrue(pars.get(1) instanceof Params1); + } + assertEquals(2, pars.size()); + + Collection outputDataFileTypes = PipelineInputsOutputsUtils + .deserializedOutputFileTypesFromTaskDirectory(taskDirectory); + assertTrue(outputDataFileTypes.contains(calibratedCollateralPixelDataFileType)); + assertEquals(1, outputDataFileTypes.size()); + } + } + + /** Tests the subtaskInformation() method. */ + @Test + public void testSubtaskInformation() { + + Mockito.when(datastoreFileManager.subtaskCount()).thenReturn(7); + SubtaskInformation subtaskInformation = pipelineInputs.subtaskInformation(); + assertEquals("testmod", subtaskInformation.getModuleName()); + assertEquals("[sector-0002;target;1:1:A]", subtaskInformation.getUowBriefState()); + assertEquals(7, subtaskInformation.getSubtaskCount()); + + pipelineInputs.setSingleSubtask(true); + subtaskInformation = pipelineInputs.subtaskInformation(); + assertEquals("testmod", subtaskInformation.getModuleName()); + assertEquals("[sector-0002;target;1:1:A]", subtaskInformation.getUowBriefState()); + assertEquals(1, subtaskInformation.getSubtaskCount()); + } + + private Map, ParameterSet> pipelineParameterSets() { + Map, ParameterSet> parMap = new HashMap<>(); + ClassWrapper c1 = new ClassWrapper<>(Params1.class); + ParameterSet s1 = new ParameterSet("params1"); + s1.populateFromParametersInstance(new Params1()); + parMap.put(c1, s1); + return parMap; + } + + private Map, ParameterSet> moduleParameterSets() { + Map, ParameterSet> parMap = new HashMap<>(); + ClassWrapper c2 = new ClassWrapper<>(Params2.class); + ParameterSet s2 = new ParameterSet("params2"); + s2.populateFromParametersInstance(new Params2()); + parMap.put(c2, s2); + return parMap; + } + + public static class Params1 extends Parameters { + private int dmy1 = 500; + private double dmy2 = 2856.3; + + public int getDmy1() { + return dmy1; + } + + public void setDmy1(int dmy1) { + this.dmy1 = dmy1; + } + + public double getDmy2() { + return dmy2; + } + + public void setDmy2(double dmy2) { + this.dmy2 = dmy2; + } + } + + public static class Params2 extends Parameters { + private String dmy3 = "dummy string"; + private boolean[] dmy4 = { true, false }; + + public String getDmy3() { + return dmy3; + } + + public void setDmy3(String dmy3) { + this.dmy3 = dmy3; + } + + public boolean[] getDmy4() { + return dmy4; + } + + public void setDmy4(boolean[] dmy4) { + this.dmy4 = dmy4; + } + } + + /** + * Subclass of {@link DatastoreDirectoryPipelineInputs}. This is necessary because if we create + * an instance of DatastoreDirectoryPipelineInputs and then apply a Mockito spy to it, the HDF5 + * API fails. Hence we need a subclass that has additional functionality we can use in the + * places where ordinarily we would use Mockito doReturn() ... when() calls on a spy. + * + * @author PT + */ + public static class PipelineInputsForTest extends DatastoreDirectoryPipelineInputs { + + @ProxyIgnore + private final AlertService mockedAlertService; + + @ProxyIgnore + private final DatastoreFileManager mockedDatastoreFileManager; + + @ProxyIgnore + private boolean singleSubtask; + + public PipelineInputsForTest(DatastoreFileManager datastoreFileManager, + AlertService alertService, PipelineTask pipelineTask) { + mockedAlertService = alertService; + mockedDatastoreFileManager = datastoreFileManager; + setPipelineTask(pipelineTask); + } + + @Override + AlertService alertService() { + return mockedAlertService; + } + + @Override + DatastoreFileManager datastoreFileManager() { + return mockedDatastoreFileManager; + } + + @Override + boolean singleSubtask() { + return singleSubtask; + } + + public void setSingleSubtask(boolean singleSubtask) { + this.singleSubtask = singleSubtask; + } + } +} diff --git a/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputsTest.java b/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputsTest.java new file mode 100644 index 0000000..269b107 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/module/DatastoreDirectoryPipelineOutputsTest.java @@ -0,0 +1,108 @@ +package gov.nasa.ziggy.module; + +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.RuleChain; + +import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager; +import gov.nasa.ziggy.data.datastore.DatastoreTestUtils; +import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.config.PropertyName; + +/** + * Unit tests for {@link DatastoreDirectoryPipelineOutputs} class. + *

        + * Note that the method {@link DatastoreDirectoryPipelineOutputs#copyTaskFilesToDatastore()} is not + * tested here. That method does nothing but call a method in {@link DatastoreFileManager}, so the + * unit tests of the latter class should be sufficient to guarantee that the method in the former + * class will work as expected. + * + * @author PT + */ +public class DatastoreDirectoryPipelineOutputsTest { + + private static final int EXPECTED_SUBTASK_COUNT = 7; + + public ZiggyDirectoryRule ziggyDirectoryRule = new ZiggyDirectoryRule(); + + public ZiggyPropertyRule datastoreRootProperty = new ZiggyPropertyRule( + PropertyName.DATASTORE_ROOT_DIR, ziggyDirectoryRule, "datastore"); + + public ZiggyPropertyRule taskDirRule = new ZiggyPropertyRule(PropertyName.RESULTS_DIR, + ziggyDirectoryRule, "pipeline-results"); + + @Rule + public final RuleChain testRuleChain = RuleChain.outerRule(ziggyDirectoryRule) + .around(datastoreRootProperty) + .around(taskDirRule); + + private Path taskDirectory; + private DataFileType calibratedSciencePixelsDataFileType; + private DataFileType calibratedCollateralPixelsDataFileType; + + @Before + public void setup() throws IOException { + + taskDirectory = DirectoryProperties.taskDataDir(); + + // Get and update the data file types. + Map dataFileTypes = DatastoreTestUtils.dataFileTypesByName(); + calibratedSciencePixelsDataFileType = dataFileTypes.get("calibrated science pixel values"); + calibratedSciencePixelsDataFileType + .setFileNameRegexp("calibrated-pixels-[0-9]+\\.science\\.nc"); + calibratedCollateralPixelsDataFileType = dataFileTypes + .get("calibrated collateral pixel values"); + calibratedCollateralPixelsDataFileType + .setFileNameRegexp("calibrated-pixels-[0-9]+\\.collateral\\.nc"); + + // Construct the subtask directories and the outputs files. + constructOutputsFiles("calibrated-pixels-", ".science.nc", EXPECTED_SUBTASK_COUNT); + constructOutputsFiles("calibrated-pixels-", ".collateral.nc", EXPECTED_SUBTASK_COUNT - 1); + + // Construct a directory with no outputs files. + SubtaskUtils.createSubtaskDirectory(taskDirectory, EXPECTED_SUBTASK_COUNT + 1); + + // Construct the collection of output file types in the task directory. + Set outputDataFileTypes = Set.of(calibratedSciencePixelsDataFileType, + calibratedCollateralPixelsDataFileType); + PipelineInputsOutputsUtils.serializeOutputFileTypesToTaskDirectory(outputDataFileTypes, + taskDirectory); + } + + private Set constructOutputsFiles(String fileNamePrefix, String fileNameSuffix, + int subtaskDirCount) throws IOException { + Set paths = new HashSet<>(); + for (int subtaskIndex = 0; subtaskIndex < subtaskDirCount; subtaskIndex++) { + Path subtaskDir = SubtaskUtils.createSubtaskDirectory(taskDirectory, subtaskIndex); + paths.add(Files + .createFile(subtaskDir.resolve(fileNamePrefix + subtaskIndex + fileNameSuffix))); + } + return paths; + } + + @Test + public void testSubtaskProducedOutputs() { + + DatastoreDirectoryPipelineOutputs pipelineOutputs = new DatastoreDirectoryPipelineOutputs(); + for (int subtaskIndex = 0; subtaskIndex < EXPECTED_SUBTASK_COUNT; subtaskIndex++) { + assertTrue(pipelineOutputs.subtaskProducedOutputs(taskDirectory, + SubtaskUtils.subtaskDirectory(taskDirectory, subtaskIndex))); + } + assertFalse(pipelineOutputs.subtaskProducedOutputs(taskDirectory, + SubtaskUtils.subtaskDirectory(taskDirectory, EXPECTED_SUBTASK_COUNT + 1))); + } +} diff --git a/src/test/java/gov/nasa/ziggy/module/DefaultPipelineInputsTest.java b/src/test/java/gov/nasa/ziggy/module/DefaultPipelineInputsTest.java deleted file mode 100644 index a7ef987..0000000 --- a/src/test/java/gov/nasa/ziggy/module/DefaultPipelineInputsTest.java +++ /dev/null @@ -1,838 +0,0 @@ -package gov.nasa.ziggy.module; - -import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; -import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_TEST_WORKING_DIR; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.RuleChain; -import org.mockito.ArgumentMatchers; -import org.mockito.Mockito; - -import com.google.common.collect.Sets; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.collections.ZiggyDataType; -import gov.nasa.ziggy.data.management.DataFileManager; -import gov.nasa.ziggy.data.management.DataFileType; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumer; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; -import gov.nasa.ziggy.models.ModelImporter; -import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; -import gov.nasa.ziggy.parameters.Parameters; -import gov.nasa.ziggy.parameters.ParametersInterface; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.ModelMetadata; -import gov.nasa.ziggy.pipeline.definition.ModelRegistry; -import gov.nasa.ziggy.pipeline.definition.ModelType; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineInstance; -import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.pipeline.definition.TypedParameter; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; -import gov.nasa.ziggy.services.alert.AlertService; -import gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.DirectoryUnitOfWorkGenerator; -import gov.nasa.ziggy.uow.TaskConfigurationParameters; -import gov.nasa.ziggy.uow.UnitOfWork; -import gov.nasa.ziggy.uow.UnitOfWorkGenerator; - -/** - * Unit test class for DefaultPipelineInputs. - * - * @author PT - */ -public class DefaultPipelineInputsTest { - - private DataFileType fluxDataFileType, centroidDataFileType; - private DataFileType resultsDataFileType; - private PipelineTask pipelineTask; - private PipelineDefinitionNode pipelineDefinitionNode; - private PipelineInstance pipelineInstance; - private PipelineInstanceNode pipelineInstanceNode; - private File datastore; - private File taskWorkspace; - private File taskDir; - private DataFileManager mockedDataFileManager; - private DefaultPipelineInputs defaultPipelineInputs; - private ModelType modelType1, modelType2, modelType3; - private ModelRegistry modelRegistry; - private Set modelTypes; - private UnitOfWork uow; - private File dataDir; - private AlertService alertService; - private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud; - private PipelineTaskCrud pipelineTaskCrud; - private TaskConfigurationParameters taskConfigurationParameters; - private Set centroidPaths; - private Set fluxPaths; - - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( - DATASTORE_ROOT_DIR, directoryRule, "datastore"); - - @Rule - public ZiggyPropertyRule ziggyTestWorkingDirPropertyRule = new ZiggyPropertyRule( - ZIGGY_TEST_WORKING_DIR, (String) null); - - @Rule - public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) - .around(datastoreRootDirPropertyRule); - - @Before - public void setup() throws IOException { - - uow = new UnitOfWork(); - uow.addParameter(new TypedParameter(UnitOfWorkGenerator.GENERATOR_CLASS_PARAMETER_NAME, - DatastoreDirectoryUnitOfWorkGenerator.class.getCanonicalName(), - ZiggyDataType.ZIGGY_STRING)); - uow.addParameter(new TypedParameter(DirectoryUnitOfWorkGenerator.DIRECTORY_PROPERTY_NAME, - "sector-0001/ccd-1:1/pa", ZiggyDataType.ZIGGY_STRING)); - uow.addParameter(new TypedParameter(UnitOfWork.BRIEF_STATE_PARAMETER_NAME, - "sector-0001/ccd-1:1/pa", ZiggyDataType.ZIGGY_STRING)); - uow.addParameter( - new TypedParameter(DatastoreDirectoryUnitOfWorkGenerator.SINGLE_SUBTASK_PROPERTY_NAME, - Boolean.toString(false), ZiggyDataType.ZIGGY_BOOLEAN)); - - datastore = new File(datastoreRootDirPropertyRule.getProperty()); - // Set up a temporary directory for the datastore and one for the task-directory - dataDir = new File(datastore, "sector-0001/ccd-1:1/pa"); - dataDir.mkdirs(); - taskWorkspace = directoryRule.directory().resolve("taskspace").toFile(); - taskDir = new File(taskWorkspace, "10-20-csci"); - taskDir.mkdirs(); - - // Set up the data file types - initializeDataFileTypes(); - - // set up the model registry and model files - initializeModelRegistry(); - - // Set up a mocked TaskConfigurationParameters instance - taskConfigurationParameters = Mockito.mock(TaskConfigurationParameters.class); - Mockito.when(taskConfigurationParameters.isReprocess()).thenReturn(true); - - // Set up a dummied PipelineTask and a dummied PipelineDefinitionNode - pipelineTask = Mockito.mock(PipelineTask.class); - pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(pipelineDefinitionNode); - Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) - .thenReturn(Sets.newHashSet(fluxDataFileType, centroidDataFileType)); - Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) - .thenReturn(Sets.newHashSet(resultsDataFileType)); - Mockito.when(pipelineTask.getModuleName()).thenReturn("csci"); - Mockito.when(pipelineDefinitionNode.getModelTypes()).thenReturn(modelTypes); - Mockito - .when( - pipelineTask.getParameters(ArgumentMatchers.eq(TaskConfigurationParameters.class))) - .thenReturn(taskConfigurationParameters); - Mockito - .when(pipelineTask.getParameters(ArgumentMatchers.eq(TaskConfigurationParameters.class), - ArgumentMatchers.anyBoolean())) - .thenReturn(taskConfigurationParameters); - - // Set up the dummied PipelineInstance - pipelineInstance = Mockito.mock(PipelineInstance.class); - Mockito.when(pipelineTask.getPipelineInstance()).thenReturn(pipelineInstance); - Mockito.when(pipelineInstance.getPipelineParameterSets()).thenReturn(parametersMap()); - Mockito.when(pipelineInstance.getModelRegistry()).thenReturn(modelRegistry); - - // Set up the dummied PipelineInstanceNode - pipelineInstanceNode = Mockito.mock(PipelineInstanceNode.class); - Mockito.when(pipelineTask.getPipelineInstanceNode()).thenReturn(pipelineInstanceNode); - Mockito.when(pipelineInstanceNode.getModuleParameterSets()).thenReturn(new HashMap<>()); - - // Create some "data files" for the process - initializeDataFiles(); - - // We need an instance of DatastoreProducerConsumerCrud that's mocked - datastoreProducerConsumerCrud = Mockito.mock(DatastoreProducerConsumerCrud.class); - - // Also an instance of PipelineTaskCrud that's mocked - pipelineTaskCrud = Mockito.mock(PipelineTaskCrud.class); - - // We need a DataFileManager that's had its crud methods mocked out - mockedDataFileManager = new DataFileManager(datastore.toPath(), taskDir.toPath(), - pipelineTask); - mockedDataFileManager = Mockito.spy(mockedDataFileManager); - Mockito.when(mockedDataFileManager.datastoreProducerConsumerCrud()) - .thenReturn(datastoreProducerConsumerCrud); - Mockito.when(mockedDataFileManager.pipelineTaskCrud()).thenReturn(pipelineTaskCrud); - - // We need a mocked AlertService. - AlertService alertService = Mockito.mock(AlertService.class); - - // We can't use a Spy on the DefaultPipelineInputs instance because it has to get - // serialized via HDF5, and the HDF5 module interface can't figure out how to do that - // for a mocked object. Instead we resort to the tried-and-true approach of a - // constructor that takes as argument the objects we want to replace. - defaultPipelineInputs = new DefaultPipelineInputs(mockedDataFileManager, alertService); - } - - /** - * Exercises the copyDatastoreFilesToTaskDirectory() method for the case of multiple subtasks. - */ - @Test - public void testCopyDatastoreFilesToTaskDirectory() throws IOException { - - performCopyToTaskDir(false); - - // Let's see what wound up in the task directory! - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - - // Should be 2 sub-directories - assertTrue(taskDirFileNames.contains("st-0")); - assertTrue(taskDirFileNames.contains("st-1")); - - // Should be 4 data files - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - - // Should be 2 model files, both with their original file names - assertTrue(taskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(taskDirFileNames.contains("calibration-4.12.9.h5")); - - // Should be an HDF5 file of the partial inputs - assertTrue(taskDirFileNames.contains("csci-inputs.h5")); - } - - // Load the HDF5 file - Hdf5ModuleInterface hdf5ModuleInterface = new Hdf5ModuleInterface(); - DefaultPipelineInputs storedInputs = new DefaultPipelineInputs(); - hdf5ModuleInterface.readFile(new File(taskDir, "csci-inputs.h5"), storedInputs, true); - - List pars = storedInputs.getModuleParameters().getModuleParameters(); - assertEquals(2, pars.size()); - if (pars.get(0) instanceof Params1) { - assertTrue(pars.get(1) instanceof Params2); - } else { - assertTrue(pars.get(0) instanceof Params2); - assertTrue(pars.get(1) instanceof Params1); - } - - List outputTypes = storedInputs.getOutputDataFileTypes(); - assertEquals(1, outputTypes.size()); - assertEquals("results", outputTypes.get(0).getName()); - - List modelFilenames = storedInputs.getModelFilenames(); - assertEquals(2, modelFilenames.size()); - assertTrue(modelFilenames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(modelFilenames.contains("calibration-4.12.9.h5")); - } - - @Test - public void testCopyDatastoreFilesMissingFiles() throws IOException { - - // Delete one of the flux files from the datastore - new File(dataDir, "001234567.flux.h5").delete(); - performCopyToTaskDir(false); - - // Let's see what wound up in the task directory! - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - - // Should be 1 sub-directory - assertTrue(taskDirFileNames.contains("st-0")); - assertFalse(taskDirFileNames.contains("st-1")); - - // Should be 2 data files - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - } - } - - /** - * Tests the situation in which the user wants to perform keep-up processing (i.e., do not - * reprocess any files that were already processed), but none of the files in the datastore have - * already been processed (i.e., keep-up processing requires all files to be processed anyway). - */ - @Test - public void testDatastoreKeepUpProcessingNoOldFiles() throws IOException { - - // We don't want to reprocess. - Mockito.when(taskConfigurationParameters.isReprocess()).thenReturn(false); - - // We do want to return a set of DatastoreProducerConsumer instances. - Mockito - .when(datastoreProducerConsumerCrud.retrieveByFilename(ArgumentMatchers.eq(fluxPaths))) - .thenReturn(fluxDatastoreProducerConsumers()); - Mockito - .when(datastoreProducerConsumerCrud - .retrieveByFilename(ArgumentMatchers.eq(centroidPaths))) - .thenReturn(centroidDatastoreProducerConsumers()); - - // None of the consumers will have the correct pipeline definition node. - Mockito - .when(pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode(ArgumentMatchers.anySet(), - ArgumentMatchers.eq(pipelineDefinitionNode))) - .thenReturn(new ArrayList<>()); - - // Perform the copy - performCopyToTaskDir(false); - - // Let's see what wound up in the task directory! - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - - // Should be 2 sub-directories - assertTrue(taskDirFileNames.contains("st-0")); - assertTrue(taskDirFileNames.contains("st-1")); - - // Should be 4 data files - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - - // Should be 2 model files, both with their original file names - assertTrue(taskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(taskDirFileNames.contains("calibration-4.12.9.h5")); - - // Should be an HDF5 file of the partial inputs - assertTrue(taskDirFileNames.contains("csci-inputs.h5")); - } - } - - /** - * Constructs a {@link List} of {@link DatastoreProducerConsumer} instances for the data files - * used in the unit tests. - */ - private List fluxDatastoreProducerConsumers() { - - List datastoreProducerConsumers = new ArrayList<>(); - - // Add producer-consumer instances for each data file. - datastoreProducerConsumers - .add(new DatastoreProducerConsumer(5L, "sector-0001/ccd-1:1/pa/001234567.flux.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA)); - datastoreProducerConsumers - .add(new DatastoreProducerConsumer(5L, "sector-0001/ccd-1:1/pa/765432100.flux.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA)); - - // Set the 001234567 files to have one consumer, the 765432100 files to have a - // different one. - datastoreProducerConsumers.get(0).addConsumer(6L); - datastoreProducerConsumers.get(1).addConsumer(7L); - - return datastoreProducerConsumers; - } - - /** - * Constructs a {@link List} of {@link DatastoreProducerConsumer} instances for the data files - * used in the unit tests. - */ - private List centroidDatastoreProducerConsumers() { - - List datastoreProducerConsumers = new ArrayList<>(); - - // Add producer-consumer instances for each data file. - datastoreProducerConsumers - .add(new DatastoreProducerConsumer(5L, "sector-0001/ccd-1:1/pa/001234567.centroid.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA)); - datastoreProducerConsumers - .add(new DatastoreProducerConsumer(5L, "sector-0001/ccd-1:1/pa/765432100.centroid.h5", - DatastoreProducerConsumer.DataReceiptFileType.DATA)); - - // Set the 001234567 files to have one consumer, the 765432100 files to have a - // different one. - datastoreProducerConsumers.get(0).addConsumer(6L); - datastoreProducerConsumers.get(1).addConsumer(7L); - - return datastoreProducerConsumers; - } - - /** - * Tests the situation in which the user wants to perform keep-up processing (i.e., do not - * reprocess any files that were already processed), and there are some files that have already - * been processed and don't need to be processed again. - */ - @Test - public void testDatastoreKeepUpProcessing() throws IOException { - - // We don't want to reprocess. - Mockito.when(taskConfigurationParameters.isReprocess()).thenReturn(false); - - // We do want to return a set of DatastoreProducerConsumer instances. - Mockito - .when(datastoreProducerConsumerCrud.retrieveByFilename(ArgumentMatchers.eq(fluxPaths))) - .thenReturn(fluxDatastoreProducerConsumers()); - Mockito - .when(datastoreProducerConsumerCrud - .retrieveByFilename(ArgumentMatchers.eq(centroidPaths))) - .thenReturn(centroidDatastoreProducerConsumers()); - - // None of the consumers will have the correct pipeline definition node. - Mockito - .when(pipelineTaskCrud.retrieveIdsForPipelineDefinitionNode(ArgumentMatchers.anySet(), - ArgumentMatchers.eq(pipelineDefinitionNode))) - .thenReturn(List.of(6L)); - - // Perform the copy - performCopyToTaskDir(false); - - // Let's see what wound up in the task directory! - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - - // Should be 2 data files - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - - // Should be 1 sub-directory - assertTrue(taskDirFileNames.contains("st-0")); - assertFalse(taskDirFileNames.contains("st-1")); - - // Should be 2 model files, both with their original file names - assertTrue(taskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(taskDirFileNames.contains("calibration-4.12.9.h5")); - - // Should be an HDF5 file of the partial inputs - assertTrue(taskDirFileNames.contains("csci-inputs.h5")); - } - } - - @Test - public void testFindDatastoreFilesForInputs() { - - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(uow); - - Set paths = defaultPipelineInputs.findDatastoreFilesForInputs(pipelineTask); - assertEquals(4, paths.size()); - Set filenames = paths.stream() - .map(s -> s.getFileName().toString()) - .collect(Collectors.toSet()); - assertTrue(filenames.contains("001234567.flux.h5")); - assertTrue(filenames.contains("765432100.flux.h5")); - assertTrue(filenames.contains("001234567.centroid.h5")); - assertTrue(filenames.contains("765432100.centroid.h5")); - } - - @Test - public void testSubtaskInformation() { - - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(uow); - - SubtaskInformation subtaskInformation = defaultPipelineInputs - .subtaskInformation(pipelineTask); - assertEquals("csci", subtaskInformation.getModuleName()); - assertEquals("sector-0001/ccd-1:1/pa", subtaskInformation.getUowBriefState()); - assertEquals(2, subtaskInformation.getSubtaskCount()); - assertEquals(2, subtaskInformation.getMaxParallelSubtasks()); - - TypedParameter singleSubtask = uow - .getParameter(DatastoreDirectoryUnitOfWorkGenerator.SINGLE_SUBTASK_PROPERTY_NAME); - singleSubtask.setValue(Boolean.TRUE); - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(uow); - subtaskInformation = defaultPipelineInputs.subtaskInformation(pipelineTask); - assertEquals("csci", subtaskInformation.getModuleName()); - assertEquals("sector-0001/ccd-1:1/pa", subtaskInformation.getUowBriefState()); - assertEquals(1, subtaskInformation.getSubtaskCount()); - assertEquals(1, subtaskInformation.getMaxParallelSubtasks()); - } - - /** - * Exercises the copyDatastoreFilesToTaskDirectory() method for the case of a single subtask. - */ - @Test - public void testCopyDatastoreFilesToDirectorySingleSubtask() throws IOException { - - performCopyToTaskDir(true); - - // Let's see what wound up in the task directory! - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - - // Should be 1 sub-directories - assertTrue(taskDirFileNames.contains("st-0")); - - // Should be 4 data files - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - - // Should be 2 model files, both with their original file names - assertTrue(taskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(taskDirFileNames.contains("calibration-4.12.9.h5")); - - // Should be an HDF5 file of the partial inputs - assertTrue(taskDirFileNames.contains("csci-inputs.h5")); - - // Load the HDF5 file - Hdf5ModuleInterface hdf5ModuleInterface = new Hdf5ModuleInterface(); - DefaultPipelineInputs storedInputs = new DefaultPipelineInputs(); - hdf5ModuleInterface.readFile(new File(taskDir, "csci-inputs.h5"), storedInputs, true); - - List pars = storedInputs.getModuleParameters() - .getModuleParameters(); - assertEquals(2, pars.size()); - if (pars.get(0) instanceof Params1) { - assertTrue(pars.get(1) instanceof Params2); - } else { - assertTrue(pars.get(0) instanceof Params2); - assertTrue(pars.get(1) instanceof Params1); - } - - List outputTypes = storedInputs.getOutputDataFileTypes(); - assertEquals(1, outputTypes.size()); - assertEquals("results", outputTypes.get(0).getName()); - - List modelFilenames = storedInputs.getModelFilenames(); - assertEquals(2, modelFilenames.size()); - assertTrue(modelFilenames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(modelFilenames.contains("calibration-4.12.9.h5")); - } - } - - /** - * Tests that populateSubTaskInputs() works correctly when multiple subtasks are specified. - */ - @Test - public void testPopulateSubTaskInputs() throws IOException { - - performCopyToTaskDir(false); - - // move to the st-0 subtask directory - Path subtaskDir = Paths.get(taskDir.getAbsolutePath(), "st-0"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), subtaskDir.toString()); - - new DefaultPipelineInputs(mockedDataFileManager, alertService).populateSubTaskInputs(); - - String subtask0DataFiles = null; - // Let's see what wound up in the subtask directory! - try (Stream taskDirPaths = java.nio.file.Files.list(subtaskDir)) { - List subtaskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - assertTrue(subtaskDirFileNames.contains("csci-inputs.h5")); - if (subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")) { - subtask0DataFiles = "001234567"; - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - } else { - subtask0DataFiles = "765432100"; - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - } - assertTrue(subtaskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(subtaskDirFileNames.contains("calibration-4.12.9.h5")); - } - - // Now do the same thing in the st-1 directory to make sure that all of the - // data is getting processed someplace - - subtaskDir = Paths.get(taskDir.getAbsolutePath(), "st-1"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), subtaskDir.toString()); - - new DefaultPipelineInputs(mockedDataFileManager, alertService).populateSubTaskInputs(); - - // Let's see what wound up in the subtask directory! - try (Stream taskDirPaths = java.nio.file.Files.list(subtaskDir)) { - List subtaskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - if (subtask0DataFiles.equals("001234567")) { - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - } else { - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertFalse( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - } - assertTrue(subtaskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(subtaskDirFileNames.contains("calibration-4.12.9.h5")); - } - } - - /** - * Tests that populateSubTaskInputs() works correctly in the single-subtask use case. - */ - @Test - public void testPopulateSubTaskInputsSingleSubtask() throws IOException { - - performCopyToTaskDir(true); - - // move to the st-0 subtask directory - Path subtaskDir = Paths.get(taskDir.getAbsolutePath(), "st-0"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), subtaskDir.toString()); - - defaultPipelineInputs.populateSubTaskInputs(); - - // all the data files should be in st-0 - try (Stream taskDirPaths = java.nio.file.Files.list(subtaskDir)) { - List subtaskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - assertTrue(subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertTrue(subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertTrue( - subtaskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - assertTrue(subtaskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertTrue(subtaskDirFileNames.contains("calibration-4.12.9.h5")); - } - } - - /** - * Tests the deleteTempInputsFromTaskDirectory() method. - */ - @Test - public void testDeleteTempInputsFromTaskDirectory() throws IOException { - - performCopyToTaskDir(false); - defaultPipelineInputs.deleteTempInputsFromTaskDirectory(pipelineTask, taskDir.toPath()); - - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - - assertTrue(taskDirFileNames.contains("st-0")); - assertTrue(taskDirFileNames.contains("st-1")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-flux.h5")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-centroid.h5")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-flux.h5")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-centroid.h5")); - assertFalse(taskDirFileNames.contains("tess2020234101112-12345_023-geometry.xml")); - assertFalse(taskDirFileNames.contains("calibration-4.12.9.h5")); - } - } - - /** - * Executes the copy of files to the task directory. Extracted to a separate method as all of - * the tests depend on it. - * - * @param singleSubtask indicates whether a single subtask per task is desired. - */ - private void performCopyToTaskDir(boolean singleSubtask) { - - TypedParameter singleSubtaskProp = uow - .getParameter(DatastoreDirectoryUnitOfWorkGenerator.SINGLE_SUBTASK_PROPERTY_NAME); - singleSubtaskProp.setValue(Boolean.valueOf(singleSubtask)); - - Mockito.when(pipelineTask.uowTaskInstance()).thenReturn(uow); - - // Create a TaskConfigurationManager - TaskConfigurationManager tcm = new TaskConfigurationManager(taskDir); - - defaultPipelineInputs.copyDatastoreFilesToTaskDirectory(tcm, pipelineTask, - taskDir.toPath()); - tcm.persist(); - } - - private void initializeDataFileTypes() { - - fluxDataFileType = new DataFileType(); - fluxDataFileType.setName("flux"); - fluxDataFileType.setFileNameRegexForTaskDir( - "sector-([0-9]{4})-ccd-([1234]:[1234])-tic-([0-9]{9})-flux.h5"); - fluxDataFileType.setFileNameWithSubstitutionsForDatastore("sector-$1/ccd-$2/pa/$3.flux.h5"); - - centroidDataFileType = new DataFileType(); - centroidDataFileType.setName("centroid"); - centroidDataFileType.setFileNameRegexForTaskDir( - "sector-([0-9]{4})-ccd-([1234]:[1234])-tic-([0-9]{9})-centroid.h5"); - centroidDataFileType - .setFileNameWithSubstitutionsForDatastore("sector-$1/ccd-$2/pa/$3.centroid.h5"); - - resultsDataFileType = new DataFileType(); - resultsDataFileType.setName("results"); - resultsDataFileType.setFileNameRegexForTaskDir( - "sector-([0-9]{4})-ccd-([1234]:[1234])-tic-([0-9]{9})-results.h5"); - resultsDataFileType - .setFileNameWithSubstitutionsForDatastore("sector-$1/ccd-$2/results/$3.results.h5"); - - // Set up the model type 1 to have a model ID in its name, which is a simple integer, - // and a timestamp in its name - modelType1 = new ModelType(); - modelType1.setFileNameRegex("tess([0-9]{13})-([0-9]{5})_([0-9]{3})-geometry.xml"); - modelType1.setType("geometry"); - modelType1.setVersionNumberGroup(3); - modelType1.setTimestampGroup(1); - modelType1.setSemanticVersionNumber(false); - - // Set up the model type 2 to have a semantic model ID in its name but no timestamp - modelType2 = new ModelType(); - modelType2.setFileNameRegex("calibration-([0-9]+\\.[0-9]+\\.[0-9]+).h5"); - modelType2.setTimestampGroup(-1); - modelType2.setType("calibration"); - modelType2.setVersionNumberGroup(1); - modelType2.setSemanticVersionNumber(true); - - // Set up the model type 3 to have neither ID nor timestamp - modelType3 = new ModelType(); - modelType3.setFileNameRegex("simple-text.h5"); - modelType3.setType("ravenswood"); - modelType3.setTimestampGroup(-1); - modelType3.setVersionNumberGroup(-1); - } - - private void initializeModelRegistry() throws IOException { - - // First construct the registry itself - modelRegistry = new ModelRegistry(); - - // Construct the model metadata objects - ModelMetadata m1 = new ModelMetadata(modelType1, "tess2020234101112-12345_023-geometry.xml", - "DefaultModuleParametersTest", null); - ModelMetadata m2 = new ModelMetadata(modelType2, "calibration-4.12.9.h5", - "DefaultModuleParametersTest", null); - ModelMetadata m3 = new ModelMetadata(modelType3, "simple-text.h5", - "DefaultModuleParametersTest", null); - - // add the metadata objects to the registry - Map metadataMap = modelRegistry.getModels(); - metadataMap.put(modelType1, m1); - metadataMap.put(modelType2, m2); - metadataMap.put(modelType3, m3); - - // create the files for the metadata objects in the datastore - File modelsDir = new File(datastore, ModelImporter.DATASTORE_MODELS_SUBDIR_NAME); - File geometryDir = new File(modelsDir, "geometry"); - geometryDir.mkdirs(); - new File(geometryDir, m1.getDatastoreFileName()).createNewFile(); - File calibrationDir = new File(modelsDir, "calibration"); - calibrationDir.mkdirs(); - new File(calibrationDir, m2.getDatastoreFileName()).createNewFile(); - File ravenswoodDir = new File(modelsDir, "ravenswood"); - ravenswoodDir.mkdirs(); - new File(ravenswoodDir, m3.getDatastoreFileName()).createNewFile(); - - // Make the pipeline task depend on types 1 and 2 but not 3 - modelTypes = new HashSet<>(); - modelTypes.add(modelType1); - modelTypes.add(modelType2); - } - - private void initializeDataFiles() throws IOException { - - // Create the sets of paths. - centroidPaths = new HashSet<>(); - Path dataDirRelativePath = datastore.toPath().relativize(dataDir.toPath()); - centroidPaths.add(dataDirRelativePath.resolve("001234567.centroid.h5")); - centroidPaths.add(dataDirRelativePath.resolve("765432100.centroid.h5")); - - fluxPaths = new HashSet<>(); - fluxPaths.add(dataDirRelativePath.resolve("001234567.flux.h5")); - fluxPaths.add(dataDirRelativePath.resolve("765432100.flux.h5")); - - // Create the datastore files as zero-length regular files. - for (Path path : centroidPaths) { - Files.createFile(datastore.toPath().resolve(path)); - } - - // Create the datastore files as zero-length regular files. - for (Path path : fluxPaths) { - Files.createFile(datastore.toPath().resolve(path)); - } - } - - private Map, ParameterSet> parametersMap() { - - Map, ParameterSet> parMap = new HashMap<>(); - - ClassWrapper c1 = new ClassWrapper<>(Params1.class); - ParameterSet s1 = new ParameterSet("params1"); - s1.populateFromParametersInstance(new Params1()); - parMap.put(c1, s1); - - ClassWrapper c2 = new ClassWrapper<>(Params2.class); - ParameterSet s2 = new ParameterSet("params2"); - s2.populateFromParametersInstance(new Params2()); - parMap.put(c2, s2); - - return parMap; - } - - public static class Params1 extends Parameters { - private int dmy1 = 500; - private double dmy2 = 2856.3; - - public int getDmy1() { - return dmy1; - } - - public void setDmy1(int dmy1) { - this.dmy1 = dmy1; - } - - public double getDmy2() { - return dmy2; - } - - public void setDmy2(double dmy2) { - this.dmy2 = dmy2; - } - } - - public static class Params2 extends Parameters { - private String dmy3 = "dummy string"; - private boolean[] dmy4 = { true, false }; - - public String getDmy3() { - return dmy3; - } - - public void setDmy3(String dmy3) { - this.dmy3 = dmy3; - } - - public boolean[] getDmy4() { - return dmy4; - } - - public void setDmy4(boolean[] dmy4) { - this.dmy4 = dmy4; - } - } -} diff --git a/src/test/java/gov/nasa/ziggy/module/DefaultPipelineOutputsTest.java b/src/test/java/gov/nasa/ziggy/module/DefaultPipelineOutputsTest.java deleted file mode 100644 index e5ba495..0000000 --- a/src/test/java/gov/nasa/ziggy/module/DefaultPipelineOutputsTest.java +++ /dev/null @@ -1,180 +0,0 @@ -package gov.nasa.ziggy.module; - -import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; -import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_TEST_WORKING_DIR; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.Arrays; -import java.util.List; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.RuleChain; -import org.mockito.Mockito; - -import com.google.common.collect.Sets; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.data.management.DataFileManager; -import gov.nasa.ziggy.data.management.DataFileType; -import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; -import gov.nasa.ziggy.services.alert.AlertService; - -public class DefaultPipelineOutputsTest { - - private DataFileType fluxDataFileType, centroidDataFileType; - private DataFileType resultsDataFileType; - private PipelineTask pipelineTask; - private PipelineDefinitionNode pipelineDefinitionNode; - private File datastore; - private File taskWorkspace; - private File taskDir; - private DataFileManager mockedDataFileManager; - private DefaultPipelineInputs defaultPipelineInputs; - private DefaultPipelineOutputs defaultPipelineOutputs; - - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( - DATASTORE_ROOT_DIR, directoryRule, "datastore"); - - @Rule - public ZiggyPropertyRule ziggyTestWorkingDirPropertyRule = new ZiggyPropertyRule( - ZIGGY_TEST_WORKING_DIR, (String) null); - - @Rule - public final RuleChain ruleChain = RuleChain.outerRule(directoryRule) - .around(datastoreRootDirPropertyRule); - - @Before - public void setup() throws IOException { - - datastore = new File(datastoreRootDirPropertyRule.getProperty()); - // Set up a temporary directory for the datastore and one for the task-directory - datastore.mkdirs(); - File dataDir = new File(datastore, "sector-0001/ccd-1:1/pa"); - dataDir.mkdirs(); - taskWorkspace = directoryRule.directory().resolve("taskspace").toFile(); - taskWorkspace.mkdirs(); - taskDir = new File(taskWorkspace, "10-20-csci"); - taskDir.mkdirs(); - - // Set up the data file types - initializeDataFileTypes(); - - // Set up a dummied PipelineTask and a dummied PipelineDefinitionNode - pipelineTask = Mockito.mock(PipelineTask.class); - pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); - Mockito.when(pipelineTask.getPipelineDefinitionNode()).thenReturn(pipelineDefinitionNode); - Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) - .thenReturn(Sets.newHashSet(fluxDataFileType, centroidDataFileType)); - Mockito.when(pipelineDefinitionNode.getOutputDataFileTypes()) - .thenReturn(Sets.newHashSet(resultsDataFileType)); - Mockito.when(pipelineTask.getModuleName()).thenReturn("csci"); - - // We need a DataFileManager that's had its ResultsOriginatorCrud mocked out - mockedDataFileManager = new DataFileManager(datastore.toPath(), taskDir.toPath(), - pipelineTask); - mockedDataFileManager = Mockito.spy(mockedDataFileManager); - Mockito.when(mockedDataFileManager.datastoreProducerConsumerCrud()) - .thenReturn(Mockito.mock(DatastoreProducerConsumerCrud.class)); - - // We can't use a Spy on the DefaultPipelineInputs instance because it has to get - // serialized via HDF5, and the HDF5 module interface can't figure out how to do that - // for a mocked object. Instead we resort to the tried-and-true approach of a - // constructor that takes as argument the object we want to replace. - defaultPipelineInputs = new DefaultPipelineInputs(mockedDataFileManager, - Mockito.mock(AlertService.class)); - defaultPipelineInputs.setOutputDataFileTypes(Arrays.asList(resultsDataFileType)); - defaultPipelineInputs.writeToTaskDir(pipelineTask, taskDir); - - // the DefaultPipelineOutputs has the same DataFileManager issue as DefaultPipelineInputs - defaultPipelineOutputs = new DefaultPipelineOutputs(mockedDataFileManager); - - // create the subtask directory and put a couple of results files therein - File subtaskDir = new File(taskDir, "st-0"); - subtaskDir.mkdirs(); - new File(subtaskDir, "sector-0001-ccd-1:1-tic-001234567-results.h5").createNewFile(); - new File(subtaskDir, "sector-0001-ccd-1:1-tic-765432100-results.h5").createNewFile(); - } - - /** - * Tests the populateTaskResults() method, which copies task results from the subtask directory - * to the task directory. - */ - @Test - public void testPopulateTaskResults() throws IOException { - - // pull the directory listing and make sure no results files are present - - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-results.h5")); - assertFalse(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-results.h5")); - } - - // Go to the subtask directory - Path subtaskDir = taskDir.toPath().resolve("st-0"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), subtaskDir.toString()); - - // Populate the task results - defaultPipelineOutputs.populateTaskResults(); - - // Pull the directory listing and check for results files - try (Stream taskDirPaths = java.nio.file.Files.list(taskDir.toPath())) { - List taskDirFileNames = taskDirPaths.map(s -> s.getFileName().toString()) - .collect(Collectors.toList()); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-001234567-results.h5")); - assertTrue(taskDirFileNames.contains("sector-0001-ccd-1:1-tic-765432100-results.h5")); - } - } - - /** - * Tests the copyTaskDirectoryResultsToDatastore() method, which persists results from - * processing. - */ - @Test - public void testCopyTaskDirectoryResultsToDatastore() throws IOException { - - // put the results files in the task directory - new File(taskDir, "sector-0001-ccd-1:1-tic-001234567-results.h5").createNewFile(); - new File(taskDir, "sector-0001-ccd-1:1-tic-765432100-results.h5").createNewFile(); - - // execute the copy - defaultPipelineOutputs.copyTaskDirectoryResultsToDatastore(null, pipelineTask, - taskDir.toPath()); - - // check that the files are gone from the task directory - assertFalse(new File(taskDir, "sector-0001-ccd-1:1-tic-001234567-results.h5").exists()); - assertFalse(new File(taskDir, "sector-0001-ccd-1:1-tic-765432100-results.h5").exists()); - - // check that the files are present in the datastore - String datastoreResultsPath = "sector-0001/ccd-1:1/results"; - assertTrue(java.nio.file.Files.exists( - Paths.get(datastore.getAbsolutePath(), datastoreResultsPath, "001234567.results.h5"))); - assertTrue(java.nio.file.Files.exists( - Paths.get(datastore.getAbsolutePath(), datastoreResultsPath, "765432100.results.h5"))); - } - - private void initializeDataFileTypes() { - - resultsDataFileType = new DataFileType(); - resultsDataFileType.setName("results"); - resultsDataFileType.setFileNameRegexForTaskDir( - "sector-([0-9]{4})-ccd-([1234]:[1234])-tic-([0-9]{9})-results.h5"); - resultsDataFileType - .setFileNameWithSubstitutionsForDatastore("sector-$1/ccd-$2/results/$3.results.h5"); - } -} diff --git a/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java b/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java index 71ff814..dd7acd2 100644 --- a/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java +++ b/src/test/java/gov/nasa/ziggy/module/ExternalProcessPipelineModuleTest.java @@ -22,20 +22,21 @@ import java.io.File; import java.io.IOException; import java.util.Collections; +import java.util.HashSet; import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; -import org.mockito.Mockito; +import org.mockito.ArgumentMatchers; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager; +import gov.nasa.ziggy.data.datastore.DatastoreFileManager.InputFiles; import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineInputsSample; import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineOutputsSample1; import gov.nasa.ziggy.data.management.DatastoreProducerConsumerCrud; -import gov.nasa.ziggy.module.remote.RemoteParameters; -import gov.nasa.ziggy.module.remote.TimestampFile; import gov.nasa.ziggy.module.remote.TimestampFile.Event; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; @@ -56,26 +57,30 @@ */ public class ExternalProcessPipelineModuleTest { - private PipelineTask p; - private PipelineInstance i; - private ProcessingSummaryOperations a; - private PipelineTaskCrud c; - private TaskConfigurationManager ih; - private TestAlgorithmLifecycle tal; + private PipelineTask pipelineTask; + private PipelineInstance pipelineInstance; + private ProcessingSummaryOperations processingSummaryOperations; + private PipelineTaskCrud pipelineTaskCrud; + private TaskConfiguration taskConfiguration; + private AlgorithmLifecycleManager taskAlgorithmLifecycle; private File taskDir; - private RemoteParameters r; - private PipelineInstanceNode pin; - private PipelineModuleDefinition pmd; - private DatabaseService ds; - private AlgorithmExecutor ae; - private TestPipelineModule t; - private DatastoreProducerConsumerCrud dpcc; + private PipelineInstanceNode pipelineInstanceNode; + private PipelineModuleDefinition pipelineModuleDefinition; + private DatabaseService databaseService; + private AlgorithmExecutor algorithmExecutor; + private DatastoreProducerConsumerCrud datastoreProducerConsumerCrud; + private ExternalProcessPipelineModule pipelineModule; + private TaskDirectoryManager taskDirManager; + private DatastoreFileManager datastoreFileManager; @Rule public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @Rule public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( DATASTORE_ROOT_DIR, "/dev/null"); + @Rule + public ZiggyPropertyRule pipelineResultsRule = new ZiggyPropertyRule(PropertyName.RESULTS_DIR, + "/dev/null"); @Rule public ZiggyPropertyRule piProcessingHaltStepPropertyRule = new ZiggyPropertyRule(PIPELINE_HALT, @@ -87,35 +92,68 @@ public class ExternalProcessPipelineModuleTest { @Before public void setup() { - r = new RemoteParameters(); - p = mock(PipelineTask.class); - i = mock(PipelineInstance.class); - pin = mock(PipelineInstanceNode.class); - pmd = mock(PipelineModuleDefinition.class); - ae = mock(AlgorithmExecutor.class); - when(p.getId()).thenReturn(100L); - when(p.getPipelineInstance()).thenReturn(i); - when(p.getParameters(RemoteParameters.class, false)).thenReturn(r); - when(p.getPipelineInstanceNode()).thenReturn(pin); - when(pin.getPipelineModuleDefinition()).thenReturn(pmd); - when(pmd.getInputsClass()).thenReturn(new ClassWrapper<>(PipelineInputsSample.class)); - when(pmd.getOutputsClass()).thenReturn(new ClassWrapper<>(PipelineOutputsSample1.class)); - when(i.getId()).thenReturn(50L); - a = mock(ProcessingSummaryOperations.class); - c = mock(PipelineTaskCrud.class); - when(c.retrieve(100L)).thenReturn(p); - ih = mock(TaskConfigurationManager.class); + pipelineTask = mock(PipelineTask.class); + pipelineInstance = mock(PipelineInstance.class); + pipelineInstanceNode = mock(PipelineInstanceNode.class); + pipelineModuleDefinition = mock(PipelineModuleDefinition.class); + algorithmExecutor = mock(AlgorithmExecutor.class); + when(pipelineTask.getId()).thenReturn(100L); + when(pipelineTask.getPipelineInstance()).thenReturn(pipelineInstance); + when(pipelineTask.getPipelineInstanceNode()).thenReturn(pipelineInstanceNode); + when(pipelineTask.taskBaseName()).thenReturn("50-100-test"); + when(pipelineInstanceNode.getPipelineModuleDefinition()) + .thenReturn(pipelineModuleDefinition); + when(pipelineModuleDefinition.getInputsClass()) + .thenReturn(new ClassWrapper<>(PipelineInputsSample.class)); + when(pipelineModuleDefinition.getOutputsClass()) + .thenReturn(new ClassWrapper<>(PipelineOutputsSample1.class)); + when(pipelineInstance.getId()).thenReturn(50L); + processingSummaryOperations = mock(ProcessingSummaryOperations.class); + pipelineTaskCrud = mock(PipelineTaskCrud.class); + when(pipelineTaskCrud.retrieve(100L)).thenReturn(pipelineTask); + taskConfiguration = mock(TaskConfiguration.class); taskDir = directoryRule.directory().toFile(); taskDir.mkdirs(); - tal = mock(TestAlgorithmLifecycle.class); - when(tal.getTaskDir(true)).thenReturn(taskDir); - when(tal.getTaskDir(false)).thenReturn(taskDir); - when(tal.getExecutor()).thenReturn(ae); - ds = mock(DatabaseService.class); - DatabaseService.setInstance(ds); - dpcc = mock(DatastoreProducerConsumerCrud.class); - when(dpcc.retrieveFilesConsumedByTask(100L)).thenReturn(Collections.emptySet()); - t = new TestPipelineModule(p, RunMode.STANDARD); + taskAlgorithmLifecycle = mock(AlgorithmLifecycleManager.class); + when(taskAlgorithmLifecycle.getTaskDir(true)).thenReturn(taskDir); + when(taskAlgorithmLifecycle.getTaskDir(false)).thenReturn(taskDir); + when(taskAlgorithmLifecycle.getExecutor()).thenReturn(algorithmExecutor); + taskDirManager = mock(TaskDirectoryManager.class); + when(taskDirManager.taskDir()).thenReturn(directoryRule.directory()); + databaseService = mock(DatabaseService.class); + DatabaseService.setInstance(databaseService); + datastoreProducerConsumerCrud = mock(DatastoreProducerConsumerCrud.class); + when(datastoreProducerConsumerCrud.retrieveFilesConsumedByTask(100L)) + .thenReturn(Collections.emptySet()); + + datastoreFileManager = mock(DatastoreFileManager.class); + when(datastoreFileManager.inputFilesByOutputStatus()) + .thenReturn(new InputFiles(new HashSet<>(), new HashSet<>())); + + configurePipelineModule(RunMode.STANDARD); + + // By default, mock 5 subtasks. + when(taskConfiguration.getSubtaskCount()).thenReturn(5); + } + + /** Sets up a pipeline module with a specified run mode. */ + private void configurePipelineModule(RunMode runMode) { + pipelineModule = spy(new ExternalProcessPipelineModule(pipelineTask, runMode)); + doReturn(taskConfiguration).when(pipelineModule).taskConfiguration(); + doReturn(datastoreFileManager).when(pipelineModule).datastoreFileManager(); + doReturn(new HashSet<>()).when(pipelineModule) + .datastorePathsToRelative(ArgumentMatchers.anySet()); + doReturn(new HashSet<>()).when(pipelineModule) + .datastorePathsToNames(ArgumentMatchers.anySet()); + doReturn(datastoreProducerConsumerCrud).when(pipelineModule) + .datastoreProducerConsumerCrud(); + doReturn(taskAlgorithmLifecycle).when(pipelineModule).algorithmManager(); + doReturn(processingSummaryOperations).when(pipelineModule).processingSummaryOperations(); + doReturn(pipelineTaskCrud).when(pipelineModule).pipelineTaskCrud(); + doReturn(taskDirManager).when(pipelineModule).taskDirManager(); + + // Return the database processing states in the correct order. + configureDatabaseProcessingStates(); } @After @@ -129,20 +167,21 @@ public void teardown() throws IOException { @Test public void testNextProcessingState() { - ProcessingState s = t.nextProcessingState(ProcessingState.INITIALIZING); - assertEquals(ProcessingState.MARSHALING, s); - s = t.nextProcessingState(s); - assertEquals(ProcessingState.ALGORITHM_SUBMITTING, s); - s = t.nextProcessingState(s); - assertEquals(ProcessingState.ALGORITHM_QUEUED, s); - s = t.nextProcessingState(s); - assertEquals(ProcessingState.ALGORITHM_EXECUTING, s); - s = t.nextProcessingState(s); - assertEquals(ProcessingState.ALGORITHM_COMPLETE, s); - s = t.nextProcessingState(s); - assertEquals(ProcessingState.STORING, s); - s = t.nextProcessingState(s); - assertEquals(ProcessingState.COMPLETE, s); + ProcessingState processingState = pipelineModule + .nextProcessingState(ProcessingState.INITIALIZING); + assertEquals(ProcessingState.MARSHALING, processingState); + processingState = pipelineModule.nextProcessingState(processingState); + assertEquals(ProcessingState.ALGORITHM_SUBMITTING, processingState); + processingState = pipelineModule.nextProcessingState(processingState); + assertEquals(ProcessingState.ALGORITHM_QUEUED, processingState); + processingState = pipelineModule.nextProcessingState(processingState); + assertEquals(ProcessingState.ALGORITHM_EXECUTING, processingState); + processingState = pipelineModule.nextProcessingState(processingState); + assertEquals(ProcessingState.ALGORITHM_COMPLETE, processingState); + processingState = pipelineModule.nextProcessingState(processingState); + assertEquals(ProcessingState.STORING, processingState); + processingState = pipelineModule.nextProcessingState(processingState); + assertEquals(ProcessingState.COMPLETE, processingState); } /** @@ -151,21 +190,19 @@ public void testNextProcessingState() { */ @Test(expected = PipelineException.class) public void testExceptionFinalState() { - - t.nextProcessingState(ProcessingState.COMPLETE); + pipelineModule.nextProcessingState(ProcessingState.COMPLETE); } /** - * Tests the initialize() method of ExternalProcessPipelineModule. + * Tests the ExternalProcessPipelineModule constructor. */ @Test - public void testInitialize() { - assertEquals(p, t.pipelineTask()); - assertEquals(100L, t.taskId()); - assertEquals(50L, t.instanceId()); - assertNotNull(t.algorithmManager()); - assertTrue(t.pipelineInputs() instanceof PipelineInputsSample); - assertTrue(t.pipelineOutputs() instanceof PipelineOutputsSample1); + public void testConstructor() { + assertEquals(100L, pipelineModule.taskId()); + assertEquals(50L, pipelineModule.instanceId()); + assertNotNull(pipelineModule.algorithmManager()); + assertTrue(pipelineModule.pipelineInputs() instanceof PipelineInputsSample); + assertTrue(pipelineModule.pipelineOutputs() instanceof PipelineOutputsSample1); } /** @@ -174,9 +211,11 @@ public void testInitialize() { @Test public void testProcessInitialize() { - t = Mockito.spy(t); - t.initializingTaskAction(); - verify(t).incrementProcessingState(); + doReturn(ProcessingState.INITIALIZING).when(pipelineModule).databaseProcessingState(); + pipelineModule.initializingTaskAction(); + verify(pipelineModule).incrementDatabaseProcessingState(); + assertFalse(pipelineModule.getDoneLooping()); + assertFalse(pipelineModule.isProcessingSuccessful()); } /** @@ -188,37 +227,16 @@ public void testProcessInitialize() { @Test public void testProcessMarshalingLocal() throws Exception { - t = spy(t); - boolean b = t.processMarshaling(); - assertFalse(b); - - verify(p).clearProducerTaskIds(); - verify(t).copyDatastoreFilesToTaskDirectory(eq(ih), eq(p), eq(taskDir)); - verify(ih).validate(); - verify(ih).persist(eq(taskDir)); - verify(t).incrementProcessingState(); - } - - /** - * Tests that the method that processes a task in MARSHALING state performs correctly for remote - * processing tasks. - * - * @throws Exception - */ - @Test - public void testProcessMarshalingRemote() throws Exception { - - t = spy(t); - r.setEnabled(true); - when(tal.isRemote()).thenReturn(true); - boolean b = t.processMarshaling(); - assertFalse(b); + doReturn(ProcessingState.MARSHALING).when(pipelineModule).databaseProcessingState(); + pipelineModule.marshalingTaskAction(); - verify(p).clearProducerTaskIds(); - verify(t).copyDatastoreFilesToTaskDirectory(eq(ih), eq(p), eq(taskDir)); - verify(ih).validate(); - verify(ih).persist(eq(taskDir)); - verify(t).incrementProcessingState(); + verify(pipelineTask).clearProducerTaskIds(); + verify(pipelineModule).copyDatastoreFilesToTaskDirectory(eq(taskConfiguration), + eq(taskDir)); + verify(taskConfiguration).serialize(eq(taskDir)); + verify(pipelineModule).incrementDatabaseProcessingState(); + assertFalse(pipelineModule.getDoneLooping()); + assertFalse(pipelineModule.isProcessingSuccessful()); } /** @@ -230,16 +248,17 @@ public void testProcessMarshalingRemote() throws Exception { @Test public void testProcessMarshalingNoInputs() throws Exception { - t = spy(t); - when(ih.isEmpty()).thenReturn(true); - boolean b = t.processMarshaling(); - assertTrue(b); + doReturn(ProcessingState.MARSHALING).when(pipelineModule).databaseProcessingState(); + when(taskConfiguration.getSubtaskCount()).thenReturn(0); + pipelineModule.marshalingTaskAction(); - verify(p).clearProducerTaskIds(); - verify(t).copyDatastoreFilesToTaskDirectory(eq(ih), eq(p), eq(taskDir)); - verify(ih).validate(); - verify(ih, never()).persist(eq(taskDir)); - verify(t, never()).incrementProcessingState(); + verify(pipelineTask).clearProducerTaskIds(); + verify(pipelineModule).copyDatastoreFilesToTaskDirectory(eq(taskConfiguration), + eq(taskDir)); + verify(taskConfiguration, never()).serialize(eq(taskDir)); + verify(pipelineModule, never()).incrementDatabaseProcessingState(); + assertTrue(pipelineModule.getDoneLooping()); + assertTrue(pipelineModule.isProcessingSuccessful()); } /** @@ -248,24 +267,11 @@ public void testProcessMarshalingNoInputs() throws Exception { @Test(expected = PipelineException.class) public void testProcessMarshalingError1() { - t = spy(t); - doThrow(IllegalStateException.class).when(t) - .copyDatastoreFilesToTaskDirectory(eq(ih), eq(p), eq(taskDir)); - t.processMarshaling(); - } - - /** - * Tests that the correct exception is thrown when a problem arises while trying to commit the - * database transaction. - * - * @throws Exception - */ - @Test(expected = PipelineException.class) - public void testProcessMarshalingError2() throws Exception { - - t = spy(t); - doThrow(IllegalStateException.class).when(ds).commitTransaction(); - t.processMarshaling(); + doReturn(ProcessingState.MARSHALING).when(pipelineModule).databaseProcessingState(); + pipelineModule.marshalingTaskAction(); + doThrow(IllegalStateException.class).when(pipelineModule) + .copyDatastoreFilesToTaskDirectory(eq(taskConfiguration), eq(taskDir)); + pipelineModule.marshalingTaskAction(); } /** @@ -276,21 +282,25 @@ public void testProcessMarshalingError2() throws Exception { public void testProcessAlgorithmExecuting() { // remote processing - t = spy(t); - when(tal.isRemote()).thenReturn(true); - t.executingTaskAction(); + doReturn(ProcessingState.ALGORITHM_EXECUTING).when(pipelineModule) + .databaseProcessingState(); + when(taskAlgorithmLifecycle.isRemote()).thenReturn(true); + pipelineModule.executingTaskAction(); - verify(tal).executeAlgorithm(null); - verify(t, never()).incrementProcessingState(); + verify(taskAlgorithmLifecycle).executeAlgorithm(null); + verify(pipelineModule, never()).incrementDatabaseProcessingState(); + assertTrue(pipelineModule.getDoneLooping()); + assertFalse(pipelineModule.isProcessingSuccessful()); // local execution - when(tal.isRemote()).thenReturn(false); - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - t.executingTaskAction(); + configurePipelineModule(RunMode.STANDARD); + when(taskAlgorithmLifecycle.isRemote()).thenReturn(false); + pipelineModule.executingTaskAction(); - verify(tal, times(2)).executeAlgorithm(null); - verify(t, never()).incrementProcessingState(); + verify(taskAlgorithmLifecycle, times(2)).executeAlgorithm(null); + verify(pipelineModule, never()).incrementDatabaseProcessingState(); + assertTrue(pipelineModule.getDoneLooping()); + assertFalse(pipelineModule.isProcessingSuccessful()); } /** @@ -300,13 +310,13 @@ public void testProcessAlgorithmExecuting() { @Test public void testProcessAlgorithmCompleted() { - t = spy(t); - t.algorithmCompleteTaskAction(); - verify(t).incrementProcessingState(); + doReturn(ProcessingState.ALGORITHM_COMPLETE).when(pipelineModule).databaseProcessingState(); + pipelineModule.algorithmCompleteTaskAction(); + verify(pipelineModule).incrementDatabaseProcessingState(); - when(tal.isRemote()).thenReturn(true); - t.algorithmCompleteTaskAction(); - verify(t, times(2)).incrementProcessingState(); + when(taskAlgorithmLifecycle.isRemote()).thenReturn(true); + pipelineModule.algorithmCompleteTaskAction(); + verify(pipelineModule, times(2)).incrementDatabaseProcessingState(); } /** @@ -315,32 +325,35 @@ public void testProcessAlgorithmCompleted() { @Test public void testProcessStoring() { - t = spy(t); - ProcessingFailureSummary f = mock(ProcessingFailureSummary.class); - when(f.isAllTasksSucceeded()).thenReturn(true); - when(f.isAllTasksFailed()).thenReturn(false); - doReturn(0L).when(t).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(t).timestampFileTimestamp(any(Event.class)); - doReturn(f).when(t).processingFailureSummary(); + doReturn(ProcessingState.STORING).when(pipelineModule).databaseProcessingState(); + ProcessingFailureSummary failureSummary = mock(ProcessingFailureSummary.class); + when(failureSummary.isAllTasksSucceeded()).thenReturn(true); + when(failureSummary.isAllTasksFailed()).thenReturn(false); + doReturn(0L).when(pipelineModule) + .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); + doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); + doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); // the local version performs relatively limited activities - t.storingTaskAction(); - verify(t, never()).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - verify(t, never()).timestampFileTimestamp(any(Event.class)); - verify(t, never()).valueMetricAddValue(any(String.class), any(long.class)); - verify(t).processingFailureSummary(); - verify(t).persistResultsAndDeleteTempFiles(eq(p), eq(f)); - verify(t).incrementProcessingState(); + pipelineModule.storingTaskAction(); + verify(pipelineModule, never()).timestampFileElapsedTimeMillis(any(Event.class), + any(Event.class)); + verify(pipelineModule, never()).timestampFileTimestamp(any(Event.class)); + verify(pipelineModule, never()).valueMetricAddValue(any(String.class), any(long.class)); + verify(pipelineModule).processingFailureSummary(); + verify(pipelineModule).persistResultsAndUpdateConsumers(); + verify(pipelineModule).incrementDatabaseProcessingState(); // the remote version does somewhat more - when(tal.isRemote()).thenReturn(true); - t.storingTaskAction(); - verify(t, times(3)).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - verify(t).timestampFileTimestamp(any(Event.class)); - verify(t, times(4)).valueMetricAddValue(any(String.class), any(long.class)); - verify(t, times(2)).processingFailureSummary(); - verify(t, times(2)).persistResultsAndDeleteTempFiles(eq(p), eq(f)); - verify(t, times(2)).incrementProcessingState(); + when(taskAlgorithmLifecycle.isRemote()).thenReturn(true); + pipelineModule.storingTaskAction(); + verify(pipelineModule, times(3)).timestampFileElapsedTimeMillis(any(Event.class), + any(Event.class)); + verify(pipelineModule).timestampFileTimestamp(any(Event.class)); + verify(pipelineModule, times(4)).valueMetricAddValue(any(String.class), any(long.class)); + verify(pipelineModule, times(2)).processingFailureSummary(); + verify(pipelineModule, times(2)).persistResultsAndUpdateConsumers(); + verify(pipelineModule, times(2)).incrementDatabaseProcessingState(); } /** @@ -350,15 +363,17 @@ public void testProcessStoring() { @Test(expected = PipelineException.class) public void testProcessStoringError() { - t = spy(t); - ProcessingFailureSummary f = mock(ProcessingFailureSummary.class); - when(f.isAllTasksSucceeded()).thenReturn(true); - when(f.isAllTasksFailed()).thenReturn(false); - doReturn(0L).when(t).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(t).timestampFileTimestamp(any(Event.class)); - doReturn(f).when(t).processingFailureSummary(); - doThrow(IllegalStateException.class).when(t).persistResultsAndDeleteTempFiles(eq(p), eq(f)); - t.storingTaskAction(); + doReturn(ProcessingState.STORING).when(pipelineModule).databaseProcessingState(); + ProcessingFailureSummary failureSummary = mock(ProcessingFailureSummary.class); + when(failureSummary.isAllTasksSucceeded()).thenReturn(true); + when(failureSummary.isAllTasksFailed()).thenReturn(false); + doReturn(0L).when(pipelineModule) + .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); + doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); + doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); + doThrow(IllegalStateException.class).when(pipelineModule) + .persistResultsAndUpdateConsumers(); + pipelineModule.storingTaskAction(); } /** @@ -367,15 +382,16 @@ public void testProcessStoringError() { @Test public void testPartialFailureStoreResults() { - t = spy(t); - ProcessingFailureSummary f = mock(ProcessingFailureSummary.class); - when(f.isAllTasksSucceeded()).thenReturn(false); - when(f.isAllTasksFailed()).thenReturn(false); - doReturn(0L).when(t).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(t).timestampFileTimestamp(any(Event.class)); - doReturn(f).when(t).processingFailureSummary(); - t.storingTaskAction(); - verify(t).persistResultsAndDeleteTempFiles(eq(p), eq(f)); + doReturn(ProcessingState.STORING).when(pipelineModule).databaseProcessingState(); + ProcessingFailureSummary failureSummary = mock(ProcessingFailureSummary.class); + when(failureSummary.isAllTasksSucceeded()).thenReturn(false); + when(failureSummary.isAllTasksFailed()).thenReturn(false); + doReturn(0L).when(pipelineModule) + .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); + doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); + doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); + pipelineModule.storingTaskAction(); + verify(pipelineModule).persistResultsAndUpdateConsumers(); } /** @@ -385,15 +401,16 @@ public void testPartialFailureStoreResults() { @Test(expected = PipelineException.class) public void testPartialFailureThrowException() { - System.setProperty(PropertyName.ALLOW_PARTIAL_TASKS.property(), "false"); - t = spy(t); - ProcessingFailureSummary f = mock(ProcessingFailureSummary.class); - when(f.isAllTasksSucceeded()).thenReturn(false); - when(f.isAllTasksFailed()).thenReturn(false); - doReturn(0L).when(t).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(t).timestampFileTimestamp(any(Event.class)); - doReturn(f).when(t).processingFailureSummary(); - t.storingTaskAction(); + piWorkerAllowPartialTasksPropertyRule.setValue("false"); + doReturn(ProcessingState.STORING).when(pipelineModule).databaseProcessingState(); + ProcessingFailureSummary failureSummary = mock(ProcessingFailureSummary.class); + when(failureSummary.isAllTasksSucceeded()).thenReturn(false); + when(failureSummary.isAllTasksFailed()).thenReturn(false); + doReturn(0L).when(pipelineModule) + .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); + doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); + doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); + pipelineModule.storingTaskAction(); } /** @@ -402,14 +419,15 @@ public void testPartialFailureThrowException() { @Test(expected = PipelineException.class) public void testTotalFailureThrowsException() { - t = spy(t); - ProcessingFailureSummary f = mock(ProcessingFailureSummary.class); - when(f.isAllTasksSucceeded()).thenReturn(false); - when(f.isAllTasksFailed()).thenReturn(true); - doReturn(0L).when(t).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(t).timestampFileTimestamp(any(Event.class)); - doReturn(f).when(t).processingFailureSummary(); - t.storingTaskAction(); + doReturn(ProcessingState.STORING).when(pipelineModule).databaseProcessingState(); + ProcessingFailureSummary failureSummary = mock(ProcessingFailureSummary.class); + when(failureSummary.isAllTasksSucceeded()).thenReturn(false); + when(failureSummary.isAllTasksFailed()).thenReturn(true); + doReturn(0L).when(pipelineModule) + .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); + doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); + doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); + pipelineModule.storingTaskAction(); } /** @@ -419,27 +437,38 @@ public void testTotalFailureThrowsException() { @Test public void testProcessingMainLoopLocalTask1() { - // create the pipeline module and its database - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + when(taskConfiguration.getSubtaskCount()).thenReturn(5); // setup mocking - - mockForLoopTest(t, tal, true, false); + mockForLoopTest(true, false); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertFalse(t.isProcessingSuccessful()); - verify(t).initializingTaskAction(); - verify(t).marshalingTaskAction(); - verify(t).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_SUBMITTING, t.getProcessingState()); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.ALGORITHM_SUBMITTING, + pipelineModule.databaseProcessingState()); + } + + private void configureDatabaseProcessingStates() { + // Note that getProcessingState() is called twice during normal operations: + // once in ExternalProcessPipelineModule, once in ProcessingStatePipelineModule. + doReturn(ProcessingState.INITIALIZING, ProcessingState.INITIALIZING, + ProcessingState.MARSHALING, ProcessingState.MARSHALING, + ProcessingState.ALGORITHM_SUBMITTING, ProcessingState.ALGORITHM_SUBMITTING, + ProcessingState.ALGORITHM_QUEUED, ProcessingState.ALGORITHM_QUEUED, + ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); } /** @@ -449,29 +478,26 @@ public void testProcessingMainLoopLocalTask1() { @Test public void testProcessingMainLoopLocalTask2() { - // create the pipeline module and its database - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - t.setInitialProcessingState(ProcessingState.ALGORITHM_COMPLETE); + doReturn(ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); // setup mocking - - mockForLoopTest(t, tal, true, false); + mockForLoopTest(true, false); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertTrue(t.isProcessingSuccessful()); - // verify(t, times(5)).getProcessingState(); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t).algorithmCompleteTaskAction(); - verify(t).storingTaskAction(); - assertEquals(ProcessingState.COMPLETE, t.getProcessingState()); + assertTrue(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule).algorithmCompleteTaskAction(); + verify(pipelineModule).storingTaskAction(); + assertEquals(ProcessingState.COMPLETE, pipelineModule.databaseProcessingState()); } /** @@ -481,29 +507,25 @@ public void testProcessingMainLoopLocalTask2() { @Test public void testProcessingMainLoopRemote1() { - // create the pipeline module and its database - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - // setup mocking - - mockForLoopTest(t, tal, true, true); + mockForLoopTest(true, true); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertFalse(t.isProcessingSuccessful()); - verify(t, times(5)).getProcessingState(); - verify(t, times(2)).incrementProcessingState(); - verify(t).initializingTaskAction(); - verify(t).marshalingTaskAction(); - verify(t).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_SUBMITTING, t.getProcessingState()); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule, times(5)).databaseProcessingState(); + verify(pipelineModule, times(2)).incrementDatabaseProcessingState(); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.ALGORITHM_SUBMITTING, + pipelineModule.databaseProcessingState()); } /** @@ -513,33 +535,28 @@ public void testProcessingMainLoopRemote1() { @Test public void testProcessingMainLoopRemote2() { - // create the pipeline module and its database - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + doReturn(ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); // setup mocking - - mockForLoopTest(t, tal, true, true); - - // set up the state so it's at ALGORITHM_COMPLETE, the point at which the remote - // system hands execution back to the local one - t.setInitialProcessingState(ProcessingState.ALGORITHM_COMPLETE); + mockForLoopTest(true, true); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertTrue(t.isProcessingSuccessful()); - verify(t, times(4)).getProcessingState(); - verify(t, times(2)).incrementProcessingState(); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t).algorithmCompleteTaskAction(); - verify(t).storingTaskAction(); - assertEquals(ProcessingState.COMPLETE, t.getProcessingState()); + assertTrue(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule, times(4)).databaseProcessingState(); + verify(pipelineModule, times(2)).incrementDatabaseProcessingState(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule).algorithmCompleteTaskAction(); + verify(pipelineModule).storingTaskAction(); + assertEquals(ProcessingState.COMPLETE, pipelineModule.databaseProcessingState()); } /** @@ -549,32 +566,30 @@ public void testProcessingMainLoopRemote2() { @Test public void testProcessingMainLoopRestart1() { - // create the pipeline module and its database - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + // Note that getProcessingState() is called twice during normal operations: + // once in ExternalProcessPipelineModule, once in ProcessingStatePipelineModule. + doReturn(ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); // setup mocking - - mockForLoopTest(t, tal, true, false); - - // put the state to ALGORITHM_EXECUTING (emulates a restart after - // the task failed in the middle of running) - t.setInitialProcessingState(ProcessingState.ALGORITHM_EXECUTING); + mockForLoopTest(true, false); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertFalse(t.isProcessingSuccessful()); - verify(t, times(1)).getProcessingState(); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_EXECUTING, t.getProcessingState()); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule, times(1)).databaseProcessingState(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.ALGORITHM_EXECUTING, pipelineModule.databaseProcessingState()); } /** @@ -584,32 +599,31 @@ public void testProcessingMainLoopRestart1() { @Test public void testProcessingMainLoopRestart2() { - // create the pipeline module and its database - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + // Note that getProcessingState() is called twice during normal operations: + // once in ExternalProcessPipelineModule, once in ProcessingStatePipelineModule. + doReturn(ProcessingState.ALGORITHM_QUEUED, ProcessingState.ALGORITHM_QUEUED, + ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); // setup mocking - - mockForLoopTest(t, tal, true, true); - - // put the state to ALGORITHM_EXECUTING (emulates a restart after - // the task failed in the middle of running) - t.setInitialProcessingState(ProcessingState.ALGORITHM_QUEUED); + mockForLoopTest(true, true); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertFalse(t.isProcessingSuccessful()); - verify(t, times(1)).getProcessingState(); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_QUEUED, t.getProcessingState()); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule, times(1)).databaseProcessingState(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.ALGORITHM_QUEUED, pipelineModule.databaseProcessingState()); } /** @@ -618,29 +632,25 @@ public void testProcessingMainLoopRestart2() { @Test public void testProcessingMainLoopNoTaskDirs() { - // create the pipeline module and its database - TestPipelineModule t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + when(taskConfiguration.getSubtaskCount()).thenReturn(0); // setup mocking - - mockForLoopTest(t, tal, false, false); + mockForLoopTest(false, false); // do the loop method - t.processingMainLoop(); + pipelineModule.processingMainLoop(); // check that everything we wanted to happen, happened - assertFalse(t.isProcessingSuccessful()); - verify(t, times(4)).getProcessingState(); - verify(t, times(2)).incrementProcessingState(); - verify(t).initializingTaskAction(); - verify(t).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); + assertTrue(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + verify(pipelineModule, times(3)).databaseProcessingState(); + verify(pipelineModule, times(1)).incrementDatabaseProcessingState(); } /** @@ -650,24 +660,20 @@ public void testProcessingMainLoopNoTaskDirs() { public void testHaltInitialize() { // Set the desired stopping point - System.setProperty(PIPELINE_HALT.property(), "I"); - // Set up mockery and states as though to run the main loop for local processing - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + doReturn("I").when(pipelineModule).haltStep(); PipelineException exception = assertThrows(PipelineException.class, - () -> t.processingMainLoop()); + () -> pipelineModule.processingMainLoop()); assertEquals( "Halting processing at end of step INITIALIZING due to configuration request for halt after step I", exception.getMessage()); - verify(t).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.INITIALIZING, t.getProcessingState()); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.INITIALIZING, pipelineModule.databaseProcessingState()); } /** @@ -677,24 +683,20 @@ public void testHaltInitialize() { public void testHaltMarshaling() { // Set the desired stopping point - System.setProperty(PIPELINE_HALT.property(), "M"); - // Set up mockery and states as though to run the main loop for local processing - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); + doReturn("M").when(pipelineModule).haltStep(); PipelineException exception = assertThrows(PipelineException.class, - () -> t.processingMainLoop()); + () -> pipelineModule.processingMainLoop()); assertEquals( "Halting processing at end of step MARSHALING due to configuration request for halt after step M", exception.getMessage()); - verify(t).initializingTaskAction(); - verify(t).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.MARSHALING, t.getProcessingState()); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.MARSHALING, pipelineModule.databaseProcessingState()); } /** @@ -703,26 +705,26 @@ public void testHaltMarshaling() { @Test public void testHaltAlgorithmComplete() { + doReturn(ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + // Set the desired stopping point - System.setProperty(PIPELINE_HALT.property(), "Ac"); - // Set up mockery and states as though to run the main loop for local processing - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - t.tdb.setPState(ProcessingState.ALGORITHM_COMPLETE); - when(t.algorithmManager()).thenReturn(tal); + doReturn("Ac").when(pipelineModule).haltStep(); + PipelineException exception = assertThrows(PipelineException.class, - () -> t.processingMainLoop()); + () -> pipelineModule.processingMainLoop()); assertEquals( "Halting processing at end of step ALGORITHM_COMPLETE due to configuration request for halt after step Ac", exception.getMessage()); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_COMPLETE, t.getProcessingState()); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.ALGORITHM_COMPLETE, pipelineModule.databaseProcessingState()); } /** @@ -731,24 +733,24 @@ public void testHaltAlgorithmComplete() { @Test public void testHaltStoring() { + doReturn(ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + // Set the desired stopping point - System.setProperty(PIPELINE_HALT.property(), "S"); - // Set up mockery and states as though to run the main loop for local processing - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - t.tdb.setPState(ProcessingState.ALGORITHM_COMPLETE); - when(t.algorithmManager()).thenReturn(tal); + doReturn("S").when(pipelineModule).haltStep(); + PipelineException exception = assertThrows(PipelineException.class, - () -> t.processingMainLoop()); + () -> pipelineModule.processingMainLoop()); assertEquals("Unable to persist due to sub-task failures", exception.getMessage()); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t).algorithmCompleteTaskAction(); - verify(t).storingTaskAction(); - assertEquals(ProcessingState.STORING, t.getProcessingState()); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule).algorithmCompleteTaskAction(); + verify(pipelineModule).storingTaskAction(); + assertEquals(ProcessingState.STORING, pipelineModule.databaseProcessingState()); } /** @@ -758,55 +760,22 @@ public void testHaltStoring() { public void testHaltAlgorithmSubmitting() { // Set the desired stopping point - System.setProperty(PIPELINE_HALT.property(), "As"); - // Set up mockery and states as though to run the main loop for local processing - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - mockForLoopTest(t, tal, true, true); - t.setInitialProcessingState(ProcessingState.ALGORITHM_SUBMITTING); + doReturn("As").when(pipelineModule).haltStep(); + mockForLoopTest(true, true); PipelineException exception = assertThrows(PipelineException.class, - () -> t.processingMainLoop()); + () -> pipelineModule.processingMainLoop()); assertEquals( "Halting processing at end of step ALGORITHM_SUBMITTING due to configuration request for halt after step As", exception.getMessage()); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t).submittingTaskAction(); - verify(t, never()).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_SUBMITTING, t.getProcessingState()); - } - - /** - * Tests that processing halts at the end of ALGORITHM_QUEUED when required. - */ - @Test - public void testHaltAlgorithmQueued() { - - // Set the desired stopping point - System.setProperty(PIPELINE_HALT.property(), "Aq"); - // Set up mockery and states as though to run the main loop for local processing - t = new TestPipelineModule(p, RunMode.STANDARD); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - mockForLoopTest(t, tal, true, true); - t.setInitialProcessingState(ProcessingState.ALGORITHM_QUEUED); - PipelineException exception = assertThrows(PipelineException.class, - () -> t.processingMainLoop()); - assertEquals( - "Halting processing at end of step ALGORITHM_QUEUED due to configuration request for halt after step Aq", - exception.getMessage()); - verify(t, never()).initializingTaskAction(); - verify(t, never()).marshalingTaskAction(); - verify(t, never()).submittingTaskAction(); - verify(t).queuedTaskAction(); - verify(t, never()).executingTaskAction(); - verify(t, never()).algorithmCompleteTaskAction(); - verify(t, never()).storingTaskAction(); - assertEquals(ProcessingState.ALGORITHM_QUEUED, t.getProcessingState()); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + assertEquals(ProcessingState.ALGORITHM_SUBMITTING, + pipelineModule.databaseProcessingState()); } /** @@ -817,269 +786,186 @@ public void testHaltAlgorithmQueued() { * @param successfulMarshaling * @param remote */ - private void mockForLoopTest(TestPipelineModule t, TestAlgorithmLifecycle tal, - boolean successfulMarshaling, boolean remote) { + private void mockForLoopTest(boolean successfulMarshaling, boolean remote) { - t.setDoneLoopingValue(!successfulMarshaling); - ProcessingFailureSummary f = mock(ProcessingFailureSummary.class); - when(f.isAllTasksSucceeded()).thenReturn(true); - when(f.isAllTasksFailed()).thenReturn(false); - doReturn(0L).when(t).timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); - doReturn(0L).when(t).timestampFileTimestamp(any(Event.class)); - doReturn(f).when(t).processingFailureSummary(); + ProcessingFailureSummary failureSummary = mock(ProcessingFailureSummary.class); + when(failureSummary.isAllTasksSucceeded()).thenReturn(true); + when(failureSummary.isAllTasksFailed()).thenReturn(false); + doReturn(0L).when(pipelineModule) + .timestampFileElapsedTimeMillis(any(Event.class), any(Event.class)); + doReturn(0L).when(pipelineModule).timestampFileTimestamp(any(Event.class)); + doReturn(failureSummary).when(pipelineModule).processingFailureSummary(); // mock the algorithm lifecycle manager's isRemote() call - when(tal.isRemote()).thenReturn(remote); + when(taskAlgorithmLifecycle.isRemote()).thenReturn(remote); } - /** - * Tests that the processRestart() method performs the correct actions. - */ + /** Tests restart from beginning. */ @Test - public void testProcessRestart() { - - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - when(t.processingSummaryOperations()).thenReturn(a); - doNothing().when(a).updateProcessingState(eq(100L), any(ProcessingState.class)); - - // restart from beginning - t = new TestPipelineModule(p, RunMode.RESTART_FROM_BEGINNING, true); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - when(t.processingSummaryOperations()).thenReturn(a); - t.processTask(); - assertTrue(t.isProcessingSuccessful()); - verify(t).processingMainLoop(); - verify(a).updateProcessingState(eq(100L), eq(ProcessingState.INITIALIZING)); + public void testRestartFromBeginning() { + configurePipelineModule(RunMode.RESTART_FROM_BEGINNING); + pipelineModule.processTask(); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).processingMainLoop(); + verify(pipelineModule).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + verify(pipelineModule, never()).processingCompleteTaskAction(); + verify(processingSummaryOperations).updateProcessingState(eq(100L), + eq(ProcessingState.ALGORITHM_SUBMITTING)); + } - // resume current step - t = new TestPipelineModule(p, RunMode.RESUME_CURRENT_STEP, true); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - when(t.processingSummaryOperations()).thenReturn(a); - t.processTask(); - assertTrue(t.isProcessingSuccessful()); - verify(a).updateProcessingState(eq(100L), any(ProcessingState.class)); - verify(t).processingMainLoop(); - - // resubmit to PBS -- this is a local task, so nothing at all should happen - t = new TestPipelineModule(p, RunMode.RESUBMIT, true); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - when(t.processingSummaryOperations()).thenReturn(a); - t.processTask(); - assertTrue(t.isProcessingSuccessful()); - verify(a, times(2)).updateProcessingState(eq(100L), any(ProcessingState.class)); - verify(t).processingMainLoop(); - - // resubmit to PBS for a remote task - when(tal.isRemote()).thenReturn(true); - t.processTask(); - assertTrue(t.isProcessingSuccessful()); - verify(a, times(3)).updateProcessingState(eq(100L), any(ProcessingState.class)); - verify(a, times(2)).updateProcessingState(eq(100L), + /** Tests a resubmit for a local-execution task. */ + @Test + public void testResubmitLocalTask() { + configurePipelineModule(RunMode.RESUBMIT); + doReturn(ProcessingState.ALGORITHM_SUBMITTING, ProcessingState.ALGORITHM_SUBMITTING, + ProcessingState.ALGORITHM_QUEUED, ProcessingState.ALGORITHM_QUEUED, + ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + pipelineModule.processTask(); + verify(processingSummaryOperations).updateProcessingState(eq(100L), eq(ProcessingState.ALGORITHM_SUBMITTING)); - verify(t, times(2)).processingMainLoop(); - - // restart PBS monitoring - t = new TestPipelineModule(p, RunMode.RESUME_MONITORING, true); - t = spy(t); - when(t.algorithmManager()).thenReturn(tal); - when(t.processingSummaryOperations()).thenReturn(a); - when(tal.isRemote()).thenReturn(true); - t.processTask(); - assertFalse(t.isProcessingSuccessful()); - verify(a, times(3)).updateProcessingState(eq(100L), any(ProcessingState.class)); - verify(a, times(2)).updateProcessingState(eq(100L), + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).processingMainLoop(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + verify(pipelineModule, never()).processingCompleteTaskAction(); + verify(processingSummaryOperations).updateProcessingState(eq(100L), + any(ProcessingState.class)); + } + + /** Tests a resubmit for a remote execution task. */ + @Test + public void testResubmitRemoteTask() { + configurePipelineModule(RunMode.RESUBMIT); + doReturn(ProcessingState.ALGORITHM_SUBMITTING, ProcessingState.ALGORITHM_SUBMITTING, + ProcessingState.ALGORITHM_QUEUED, ProcessingState.ALGORITHM_QUEUED, + ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + when(taskAlgorithmLifecycle.isRemote()).thenReturn(true); + pipelineModule.processTask(); + verify(processingSummaryOperations).updateProcessingState(eq(100L), eq(ProcessingState.ALGORITHM_SUBMITTING)); - verify(t, never()).processingMainLoop(); - verify(ae).resumeMonitoring(); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).processingMainLoop(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + verify(pipelineModule, never()).processingCompleteTaskAction(); } @Test - public void testProcessTask() { - - t = new TestPipelineModule(p, RunMode.STANDARD, true); - t = spy(t); - - boolean b = t.processTask(); - assertTrue(b); - verify(t).processingMainLoop(); + public void testResumeMonitoring() { + configurePipelineModule(RunMode.RESUME_MONITORING); + doReturn(ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + pipelineModule.processTask(); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(algorithmExecutor).resumeMonitoring(); + verify(pipelineModule, never()).processingMainLoop(); + verify(pipelineModule, never()).incrementDatabaseProcessingState(); + verify(pipelineModule, never()).databaseProcessingState(); + } + + /** Test resumption of the marshaling step. */ + @Test + public void testResumeMarshaling() { - t = new TestPipelineModule(p, RunMode.RESUBMIT, true); - t = spy(t); - when(tal.isRemote()).thenReturn(true); - b = t.processTask(); - assertTrue(b); - verify(t).processingMainLoop(); - verify(t).resubmit(); + // resume current step + configurePipelineModule(RunMode.RESUME_CURRENT_STEP); + doReturn(ProcessingState.MARSHALING, ProcessingState.MARSHALING, + ProcessingState.ALGORITHM_SUBMITTING, ProcessingState.ALGORITHM_SUBMITTING, + ProcessingState.ALGORITHM_QUEUED, ProcessingState.ALGORITHM_QUEUED, + ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + pipelineModule.processTask(); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).processingMainLoop(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule).marshalingTaskAction(); + verify(pipelineModule).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + verify(pipelineModule, never()).processingCompleteTaskAction(); + verify(processingSummaryOperations).updateProcessingState(eq(100L), + eq(ProcessingState.ALGORITHM_SUBMITTING)); } - /** - * Stubbed implementation of the ExternalProcessPipelineModule abstract class for test purposes. - * In addition to stubbing the methods that are actually used in normal processing (specifically - * generateInputs(), outputsClass(), processOutputs(), getModuleName(), unitOfWorkTaskType()), - * several additional methods are overridden: in some cases they are set up to return mocked - * objects, in other cases they support use of the TestAttributesDatabase in place of a real - * database. - * - * @author PT - */ - public class TestPipelineModule extends ExternalProcessPipelineModule { - - public TestAttributesDatabase tdb = new TestAttributesDatabase(); - public Boolean marshalingReturn = null; - public Boolean processingLoopSuccessState = null; - - public TestPipelineModule(PipelineTask p, RunMode r) { - super(p, r); - } - - public TestPipelineModule(PipelineTask p, RunMode r, boolean processingLoopSuccessState) { - this(p, r); - this.processingLoopSuccessState = processingLoopSuccessState; - } - - @Override - public ProcessingSummaryOperations processingSummaryOperations() { - return a; - } - - @Override - PipelineTaskCrud pipelineTaskCrud() { - return c; - } - - @Override - TaskConfigurationManager taskConfigurationManager() { - return ih; - } - - @Override - public AlgorithmLifecycle algorithmManager() { - return tal; - } - - @Override - StateFile generateStateFile() { - return new StateFile(); - } - - @Override - public ProcessingState getProcessingState() { - return tdb.getPState(); - } - - @Override - public void incrementProcessingState() { - tdb.setPState(nextProcessingState(getProcessingState())); - } - - void setInitialProcessingState(ProcessingState pState) { - tdb.setPState(pState); - } - - @Override - long timestampFileElapsedTimeMillis(TimestampFile.Event startEvent, - TimestampFile.Event finishEvent) { - return 0L; - } - - @Override - long timestampFileTimestamp(TimestampFile.Event event) { - return 0L; - } - - @Override - public void marshalingTaskAction() { - super.marshalingTaskAction(); - boolean doneLooping = marshalingReturn == null ? getDoneLooping() : marshalingReturn; - setDoneLooping(doneLooping); - } - - // For some reason attempting to use Mockito to return the value required for the - // test is not working, so we'll handle it this way: - boolean processMarshaling() { - super.marshalingTaskAction(); - return getDoneLooping(); - } - - void setDoneLoopingValue(boolean v) { - marshalingReturn = v; - } - - // Allows the processing main loop to either run normally or else skip execution - // and simply set its return value, depending on context - @Override - public void processingMainLoop() { - if (processingLoopSuccessState != null) { - processingSuccessful = processingLoopSuccessState; - return; - } - super.processingMainLoop(); - } - - public PipelineTask pipelineTask() { - return pipelineTask; - } - - @Override - DatastoreProducerConsumerCrud datastoreProducerConsumerCrud() { - return dpcc; - } + /** Test resumption of algorithm execution. */ + @Test + public void testResumeAlgorithmExecuting() { + configurePipelineModule(RunMode.RESUME_CURRENT_STEP); + doReturn(ProcessingState.ALGORITHM_EXECUTING, ProcessingState.ALGORITHM_EXECUTING, + ProcessingState.ALGORITHM_COMPLETE, ProcessingState.ALGORITHM_COMPLETE, + ProcessingState.STORING, ProcessingState.STORING, ProcessingState.COMPLETE, + ProcessingState.COMPLETE).when(pipelineModule).databaseProcessingState(); + pipelineModule.processTask(); + assertFalse(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).processingMainLoop(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule).executingTaskAction(); + verify(pipelineModule, never()).algorithmCompleteTaskAction(); + verify(pipelineModule, never()).storingTaskAction(); + verify(pipelineModule, never()).processingCompleteTaskAction(); + verify(processingSummaryOperations, never()).updateProcessingState(eq(100L), + eq(ProcessingState.ALGORITHM_EXECUTING)); } - /** - * Stubbed implementation of AlgorithmLifecycle interface for use in testing. - * - * @author PT - */ - class TestAlgorithmLifecycle implements AlgorithmLifecycle { - - @Override - public File getTaskDir(boolean cleanExisting) { - return null; - } - - @Override - public void executeAlgorithm(TaskConfigurationManager inputs) { - } - - @Override - public void doPostProcessing() { - } - - @Override - public boolean isRemote() { - return false; - } - - @Override - public AlgorithmExecutor getExecutor() { - return ae; - } + @Test + public void testResumeAlgorithmComplete() { + mockForLoopTest(false, true); + configurePipelineModule(RunMode.RESUME_CURRENT_STEP); + doReturn(ProcessingState.ALGORITHM_COMPLETE, ProcessingState.STORING, + ProcessingState.STORING, ProcessingState.COMPLETE, ProcessingState.COMPLETE) + .when(pipelineModule) + .databaseProcessingState(); + doNothing().when(pipelineModule).storingTaskAction(); + pipelineModule.processTask(); + assertTrue(pipelineModule.isProcessingSuccessful()); + verify(pipelineModule).processingMainLoop(); + verify(pipelineModule, never()).initializingTaskAction(); + verify(pipelineModule, never()).marshalingTaskAction(); + verify(pipelineModule, never()).submittingTaskAction(); + verify(pipelineModule, never()).queuedTaskAction(); + verify(pipelineModule, never()).executingTaskAction(); + verify(pipelineModule).algorithmCompleteTaskAction(); + verify(pipelineModule).storingTaskAction(); + verify(pipelineModule).processingCompleteTaskAction(); + verify(processingSummaryOperations).updateProcessingState(eq(100L), + eq(ProcessingState.COMPLETE)); } - /** - * Emulates the database that contains the processing state for pipeline tasks. - * - * @author PT - */ - class TestAttributesDatabase { - - private ProcessingState pState = null; - - public TestAttributesDatabase() { - pState = ProcessingState.INITIALIZING; - } - - public ProcessingState getPState() { - return pState; - } - - public void setPState(ProcessingState pState) { - this.pState = pState; - } + @Test + public void testProcessTask() { + boolean b = pipelineModule.processTask(); + assertFalse(b); + verify(pipelineModule).processingMainLoop(); } } diff --git a/src/test/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtilsTest.java b/src/test/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtilsTest.java index 15ece08..c7c18b6 100644 --- a/src/test/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtilsTest.java +++ b/src/test/java/gov/nasa/ziggy/module/PipelineInputsOutputsUtilsTest.java @@ -35,7 +35,8 @@ public void setup() throws IOException { taskDir = directoryRule.directory().resolve("1-2-pa"); Path workingDir = taskDir.resolve("st-12"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), workingDir.toString()); + ziggyTestWorkingDirPropertyRule.setValue(workingDir.toString()); + // Create the task dir and the subtask dir Files.createDirectories(workingDir); } diff --git a/src/test/java/gov/nasa/ziggy/module/PipelineInputsTest.java b/src/test/java/gov/nasa/ziggy/module/PipelineInputsTest.java deleted file mode 100644 index d7af428..0000000 --- a/src/test/java/gov/nasa/ziggy/module/PipelineInputsTest.java +++ /dev/null @@ -1,205 +0,0 @@ -package gov.nasa.ziggy.module; - -import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; -import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_TEST_WORKING_DIR; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.ArrayList; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.TreeSet; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.mockito.Mockito; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.data.management.DataFileInfo; -import gov.nasa.ziggy.data.management.DataFileTestUtils.DataFileInfoSample1; -import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineInputsSample; -import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineResultsSample1; -import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineResultsSample2; -import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; -import gov.nasa.ziggy.pipeline.definition.PipelineTask; - -/** - * Test class for PipelineInputs. - * - * @author PT - */ -public class PipelineInputsTest { - - private Path taskDir; - - @Rule - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - @Rule - public ZiggyPropertyRule ziggyTestWorkingDirPropertyRule = new ZiggyPropertyRule( - ZIGGY_TEST_WORKING_DIR, (String) null); - - @Rule - public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( - DATASTORE_ROOT_DIR, "/dev/null"); - - @Before - public void setup() throws IOException { - - taskDir = directoryRule.directory().resolve("1-2-pa"); - Path workingDir = taskDir.resolve("st-12"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), workingDir.toString()); - // Create the task dir and the subtask dir - Files.createDirectories(workingDir); - - PipelineResultsSample1 p = new PipelineResultsSample1(); - p.setOriginator(100L); - p.setValue(-50); - Hdf5ModuleInterface h = new Hdf5ModuleInterface(); - h.writeFile(taskDir.resolve("pa-001234567-20-results.h5").toFile(), p, true); - p = new PipelineResultsSample1(); - p.setOriginator(100L); - p.setValue(-30); - h.writeFile(taskDir.resolve("pa-765432100-20-results.h5").toFile(), p, true); - PipelineResultsSample2 p2 = new PipelineResultsSample2(); - p2.setOriginator(99L); - p2.setFvalue(92.7F); - h.writeFile(taskDir.resolve("cal-1-1-B-20-results.h5").toFile(), p, true); - - // add a task configuration file - TaskConfigurationManager taskConfigurationManager = new TaskConfigurationManager( - taskDir.toFile()); - Set wrongInstance = new TreeSet<>(); - wrongInstance.add("wrong"); - Set rightInstance = new TreeSet<>(); - rightInstance.add("right"); - for (int i = 0; i < 12; i++) { - taskConfigurationManager.addFilesForSubtask(wrongInstance); - } - taskConfigurationManager.addFilesForSubtask(rightInstance); - for (int i = 0; i < 12; i++) { - taskConfigurationManager.addFilesForSubtask(wrongInstance); - } - taskConfigurationManager.persist(); - } - - /** - * Tests the resultsFiles() method and the requiredDatastoreClasses() method. - */ - @Test - public void testResultsFiles() { - - // Start by checking that the required classes are as expected - PipelineInputsSample pipelineInputs = new PipelineInputsSample(); - Set> datastoreClasses = pipelineInputs - .requiredDataFileInfoClasses(); - assertEquals(1, datastoreClasses.size()); - assertTrue(datastoreClasses.contains(DataFileInfoSample1.class)); - - // Get the map and check its contents - Map, Set> sourcesMap = pipelineInputs - .resultsFiles(); - Set> keys = sourcesMap.keySet(); - assertEquals(1, sourcesMap.size()); - assertTrue(keys.contains(DataFileInfoSample1.class)); - - Set datastoreIds = sourcesMap.get(DataFileInfoSample1.class); - assertEquals(2, datastoreIds.size()); - Set filenames = dataFileInfosToNames(datastoreIds); - assertTrue(filenames.contains("pa-001234567-20-results.h5")); - assertTrue(filenames.contains("pa-765432100-20-results.h5")); - } - - /** - * Tests the write() method. Also exercises the go() method. - */ - @Test - public void testWrite() { - - PipelineInputsSample pipelineInputs = new PipelineInputsSample(); - pipelineInputs.populateSubTaskInputs(); - pipelineInputs.writeSubTaskInputs(); - Hdf5ModuleInterface h = new Hdf5ModuleInterface(); - PipelineInputsSample inputsFromFile = new PipelineInputsSample(); - Path subTaskDir = taskDir.resolve("st-12"); - h.readFile(subTaskDir.resolve("pa-inputs.h5").toFile(), inputsFromFile, true); - assertEquals(105.3, inputsFromFile.getDvalue(), 1e-9); - } - - /** - * Tests the subTaskIndex() method. - */ - @Test - public void testSubTaskIndex() { - PipelineInputsSample pipelineInputs = new PipelineInputsSample(); - int st = pipelineInputs.subtaskIndex(); - assertEquals(12, st); - } - - /** - * Tests the filesForSubtask method. - */ - @Test - public void testFilesForSubtask() { - PipelineInputsSample pipelineInputs = new PipelineInputsSample(); - List u = new ArrayList<>(pipelineInputs.filesForSubtask()); - assertEquals(1, u.size()); - assertEquals("right", u.get(0)); - } - - /** - * Tests the readResults() method. - */ - @Test - public void testReadResults() { - PipelineInputsSample pipelineInputs = new PipelineInputsSample(); - Map, Set> sourcesMap = pipelineInputs - .resultsFiles(); - Set datastoreIds = sourcesMap.get(DataFileInfoSample1.class); - for (DataFileInfo datastoreId : datastoreIds) { - PipelineResultsSample1 r = new PipelineResultsSample1(); - pipelineInputs.readResultsFile(datastoreId, r); - assertEquals(100L, r.getOriginator()); - } - } - - /** - * Tests the readFromTaskDir() and writeToTaskDir() methods. - */ - @Test - public void testReadWriteToTaskDir() { - PipelineInputsSample pipelineInputs = new PipelineInputsSample(); - pipelineInputs.populateSubTaskInputs(); - File taskDirFile = taskDir.toFile(); - String taskDirRoot = taskDirFile.getParent(); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), taskDirRoot); - PipelineTask pipelineTask = Mockito.mock(PipelineTask.class); - Mockito.when(pipelineTask.getModuleName()).thenReturn("pa"); - pipelineInputs.writeToTaskDir(pipelineTask, taskDirFile); - File writtenInputsFile = new File(taskDirFile, "pa-inputs.h5"); - assertTrue(writtenInputsFile.exists()); - - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), - new File(taskDirFile, "st-12").getAbsolutePath()); - pipelineInputs = new PipelineInputsSample(); - assertEquals(0.0, pipelineInputs.getDvalue(), 1e-9); - pipelineInputs.readFromTaskDir(); - assertEquals(105.3, pipelineInputs.getDvalue(), 1e-9); - } - - private Set dataFileInfosToNames(Set dataFileInfos) { - Set names = new HashSet<>(); - for (DataFileInfo d : dataFileInfos) { - names.add(d.getName().toString()); - } - return names; - } -} diff --git a/src/test/java/gov/nasa/ziggy/module/PipelineOutputsTest.java b/src/test/java/gov/nasa/ziggy/module/PipelineOutputsTest.java deleted file mode 100644 index c5117e7..0000000 --- a/src/test/java/gov/nasa/ziggy/module/PipelineOutputsTest.java +++ /dev/null @@ -1,130 +0,0 @@ -package gov.nasa.ziggy.module; - -import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; -import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_TEST_WORKING_DIR; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; - -import java.io.File; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.HashSet; -import java.util.Set; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; - -import gov.nasa.ziggy.ZiggyDirectoryRule; -import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineOutputsSample1; -import gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineResultsSample1; -import gov.nasa.ziggy.module.hdf5.Hdf5ModuleInterface; -import gov.nasa.ziggy.module.io.ModuleInterfaceUtils; -import gov.nasa.ziggy.services.config.DirectoryProperties; - -/** - * Test class for PipelineOutputs. - * - * @author PT - */ -public class PipelineOutputsTest { - - private Path taskDir; - private String filename = ModuleInterfaceUtils.outputsFileName("pa"); - - @Rule - public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); - - @Rule - public ZiggyPropertyRule datastoreRootDirPropertyRule = new ZiggyPropertyRule( - DATASTORE_ROOT_DIR, "/dev/null"); - - @Rule - public ZiggyPropertyRule ziggyTestWorkingDirPropertyRule = new ZiggyPropertyRule( - ZIGGY_TEST_WORKING_DIR, (String) null); - - @Before - public void setup() throws IOException { - - taskDir = directoryRule.directory().resolve("100-200-pa"); - Path workingDir = taskDir.resolve("st-12"); - System.setProperty(ZIGGY_TEST_WORKING_DIR.property(), workingDir.toString()); - // Create the task dir and the subtask dir - Files.createDirectories(workingDir); - - // create the outputs object and save to a file - PipelineOutputsSample1 p = new PipelineOutputsSample1(); - p.populateTaskResults(); - Hdf5ModuleInterface h = new Hdf5ModuleInterface(); - h.writeFile(DirectoryProperties.workingDir().resolve(filename).toFile(), p, true); - } - - /** - * Tests the read() method. - */ - @Test - public void testRead() { - PipelineOutputsSample1 p = new PipelineOutputsSample1(); - int[] ivalues = p.getIvalues(); - assertNull(ivalues); - p.readSubTaskOutputs(DirectoryProperties.workingDir().resolve(filename).toFile()); - ivalues = p.getIvalues(); - assertEquals(3, ivalues.length); - assertEquals(27, ivalues[0]); - assertEquals(-9, ivalues[1]); - assertEquals(5, ivalues[2]); - } - - /** - * Tests the originator() method. - */ - @Test - public void testOriginator() { - PipelineOutputsSample1 p = new PipelineOutputsSample1(); - long originator = p.originator(); - assertEquals(200L, originator); - } - - /** - * Tests the saveResults() method. - */ - @Test - public void testSaveResults() { - PipelineOutputsSample1 p = new PipelineOutputsSample1(); - p.readSubTaskOutputs(DirectoryProperties.workingDir().resolve(filename).toFile()); - p.saveResultsToTaskDir(); - int[] ivalues = p.getIvalues(); - - // The results should be saved to 3 files in the task directory, with - // names given by "pa-001234567-s", the index number, and ".h5". Each - // result should be of class PipelineResultsSample1, and should contain - // the i'th value from the ivalues array of the PipelineOutputsExample1 - // instance. - Hdf5ModuleInterface h = new Hdf5ModuleInterface(); - for (int i = 0; i < 3; i++) { - String fname = "pa-001234567-" + i + "-results.h5"; - PipelineResultsSample1 pr = new PipelineResultsSample1(); - h.readFile(taskDir.resolve(fname).toFile(), pr, true); - assertEquals(200L, pr.getOriginator()); - assertEquals(ivalues[i], pr.getValue()); - } - } - - /** - * Tests the outputFiles() method. - */ - @Test - public void testOutputFiles() { - PipelineOutputsSample1 p = new PipelineOutputsSample1(); - File[] files = p.outputFiles(); - assertEquals(1, files.length); - Set filenames = new HashSet<>(); - for (File f : files) { - filenames.add(f.getName()); - } - assertTrue(filenames.contains(filename)); - } -} diff --git a/src/test/java/gov/nasa/ziggy/module/StateFileTest.java b/src/test/java/gov/nasa/ziggy/module/StateFileTest.java index ef5b7ad..33dfc05 100644 --- a/src/test/java/gov/nasa/ziggy/module/StateFileTest.java +++ b/src/test/java/gov/nasa/ziggy/module/StateFileTest.java @@ -202,7 +202,7 @@ public void testDefaultPropertyValues() { assertEquals(StateFile.INVALID_VALUE, stateFile.getRequestedNodeCount()); assertEquals(StateFile.INVALID_VALUE, stateFile.getActiveCoresPerNode()); assertEquals(StateFile.INVALID_VALUE, stateFile.getMinCoresPerNode()); - assertEquals(StateFile.INVALID_VALUE, stateFile.getMinGigsPerNode()); + assertEquals(StateFile.INVALID_VALUE, stateFile.getMinGigsPerNode(), 1e-3); assertEquals(StateFile.INVALID_VALUE, stateFile.getPbsSubmitTimeMillis()); assertEquals(StateFile.INVALID_VALUE, stateFile.getPfeArrivalTimeMillis()); @@ -266,7 +266,7 @@ private void checkEqual(StateFile stateFile, StateFile newStateFile) { assertEquals(stateFile.getActiveCoresPerNode(), newStateFile.getActiveCoresPerNode()); assertEquals(stateFile.getRequestedNodeCount(), newStateFile.getRequestedNodeCount()); assertEquals(stateFile.getMinCoresPerNode(), newStateFile.getMinCoresPerNode()); - assertEquals(stateFile.getMinGigsPerNode(), newStateFile.getMinGigsPerNode()); + assertEquals(stateFile.getMinGigsPerNode(), newStateFile.getMinGigsPerNode(), 1e-3); assertEquals(stateFile.getPbsSubmitTimeMillis(), newStateFile.getPbsSubmitTimeMillis()); assertEquals(stateFile.getPfeArrivalTimeMillis(), newStateFile.getPfeArrivalTimeMillis()); @@ -296,7 +296,7 @@ private void testStateFileProperties(StateFile stateFile) { assertEquals(REMOTE_GROUP, stateFile.getRemoteGroup()); assertEquals(QUEUE_NAME, stateFile.getQueueName()); assertEquals(MIN_CORES_PER_NODE, stateFile.getMinCoresPerNode()); - assertEquals(MIN_GIGS_PER_NODE, stateFile.getMinGigsPerNode()); + assertEquals(MIN_GIGS_PER_NODE, stateFile.getMinGigsPerNode(), 1e-3); assertEquals(ACTIVE_CORES_PER_NODE, stateFile.getActiveCoresPerNode()); assertEquals(REQUESTED_NODE_COUNT, stateFile.getRequestedNodeCount()); assertEquals(GIGS_PER_SUBTASK, stateFile.getGigsPerSubtask(), 1e-9); diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskAllocatorTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskAllocatorTest.java index e506743..a7b883b 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskAllocatorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskAllocatorTest.java @@ -9,11 +9,11 @@ public class SubtaskAllocatorTest { - private TaskConfigurationManager taskConfigurationManager; + private TaskConfiguration taskConfigurationManager; @Before public void setup() { - taskConfigurationManager = mock(TaskConfigurationManager.class); + taskConfigurationManager = mock(TaskConfiguration.class); } /** @@ -22,7 +22,7 @@ public void setup() { */ @Test public void testAllocatorWithSingleSubtaskSet() { - when(taskConfigurationManager.numSubTasks()).thenReturn(6); + when(taskConfigurationManager.getSubtaskCount()).thenReturn(6); SubtaskAllocator allocator = new SubtaskAllocator(taskConfigurationManager); SubtaskAllocation allocation; @@ -87,6 +87,5 @@ public void testAllocatorWithSingleSubtaskSet() { allocation = allocator.nextSubtask(); assertEquals(SubtaskServer.ResponseType.NO_MORE, allocation.getStatus()); assertEquals(-1, allocation.getSubtaskIndex()); - } } diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java index 236df23..dcfd66a 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskExecutorTest.java @@ -48,7 +48,7 @@ public class SubtaskExecutorTest { private File subTaskDir; private SubtaskExecutor externalProcessExecutor; private ExternalProcess externalProcess; - private TaskConfigurationManager taskConfigurationManager = new TaskConfigurationManager(); + private TaskConfiguration taskConfigurationManager = new TaskConfiguration(); private File buildDir; private File binDir; @@ -89,13 +89,13 @@ public void setup() throws IOException, ConfigurationException { taskDir = new File(rootDir, "10-20-pa"); subTaskDir = new File(taskDir, "st-0"); subTaskDir.mkdirs(); - buildDir = new File(pipelineHomeDirPropertyRule.getProperty()); + buildDir = new File(pipelineHomeDirPropertyRule.getValue()); binDir = new File(buildDir, "bin"); binDir.mkdirs(); File paFile = new File(binDir, "pa"); paFile.createNewFile(); - new File(resultsDirPropertyRule.getProperty()).mkdirs(); + new File(resultsDirPropertyRule.getValue()).mkdirs(); // Create the state file directory Files.createDirectories(DirectoryProperties.stateFilesDir()); @@ -122,7 +122,7 @@ public void testConstructor() throws IOException { assertEquals("path1" + File.pathSeparator + "path2", e.libPath()); // with MATLAB paths defined - System.setProperty(MCRROOT.property(), "/path/to/mcr/v22"); + moduleExeMcrrootPropertyRule.setValue("/path/to/mcr/v22"); e = new SubtaskExecutor.Builder().binaryName("pa") .taskDir(taskDir) .subtaskIndex(0) @@ -175,7 +175,7 @@ public void testBinPath() throws IOException { new File(binDir3, "pa").createNewFile(); String binPath = phonyBinDir1 + File.pathSeparator + phonyBinDir2 + File.pathSeparator + binDir3; - System.setProperty(BINPATH.property(), binPath); + moduleExeBinpathPropertyRule.setValue(binPath); SubtaskExecutor e = new SubtaskExecutor.Builder().binaryName("pa") .taskDir(taskDir) .subtaskIndex(0) @@ -199,7 +199,7 @@ public void testRunInputsOutputsCommand() throws ExecuteException, IOException { [/path/to/ziggy/build/bin/ziggy, --verbose,\s\ -Djava.library.path=path1:path2:/path/to/ziggy/build/lib,\s\ -Dlog4j2.configurationFile=/path/to/ziggy/build/etc/log4j2.xml,\s\ - --class=gov.nasa.ziggy.module.TaskFileManager,\s\ + --class=gov.nasa.ziggy.module.BeforeAndAfterAlgorithmExecutor,\s\ gov.nasa.ziggy.data.management.DataFileTestUtils.PipelineInputsSample]"""; assertEquals(expectedCommandString, cmdString); assertEquals(0, retCode); @@ -227,7 +227,7 @@ public void testInputsErrorSetsErrorStatus() throws Exception { setUpMockedObjects(); Mockito.doReturn(taskConfigurationManager) .when(externalProcessExecutor) - .taskConfigurationManager(); + .taskConfiguration(); taskConfigurationManager.setInputsClass(PipelineInputsSample.class); Mockito.doReturn(1) .when(externalProcessExecutor) diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java index e76d2ef..fcb4d70 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskMasterTest.java @@ -99,7 +99,7 @@ public void testNormalExecution() throws InterruptedException, IOException { verify(subtaskMaster).releaseWriteLock(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); @@ -126,7 +126,7 @@ public void testAlgorithmFailure() throws InterruptedException, IOException { verify(subtaskMaster).releaseWriteLock(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); verify(subtaskMaster).logException(ArgumentMatchers.eq(SUBTASK_INDEX), ArgumentMatchers.any(ModuleFatalProcessingException.class)); @@ -155,7 +155,7 @@ public void testSubtaskAlreadyComplete() throws InterruptedException, IOExceptio verify(subtaskMaster).releaseWriteLock(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); @@ -184,7 +184,7 @@ public void testSubtaskAlreadyFailed() throws InterruptedException, IOException verify(subtaskMaster).releaseWriteLock(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); @@ -216,7 +216,7 @@ public void testSubtaskAlreadyProcessing() throws InterruptedException, IOExcept verify(subtaskMaster).releaseWriteLock(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); @@ -240,7 +240,7 @@ public void testUnableToObtainFileLock() throws InterruptedException, IOExceptio .getWriteLockWithoutBlocking(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); // Execute the run() method. @@ -253,7 +253,7 @@ public void testUnableToObtainFileLock() throws InterruptedException, IOExceptio verify(subtaskMaster, times(1)).releaseWriteLock(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); verify(subtaskMaster, times(0)).logException(ArgumentMatchers.any(Integer.class), ArgumentMatchers.any(Exception.class)); @@ -274,7 +274,7 @@ public void testIOException() throws InterruptedException, IOException { .getWriteLockWithoutBlocking(DirectoryProperties.taskDataDir() .resolve(TASK_DIR) .resolve("st-" + SUBTASK_INDEX) - .resolve(TaskConfigurationManager.LOCK_FILE_NAME) + .resolve(TaskConfiguration.LOCK_FILE_NAME) .toFile()); // Execute the run() method. @@ -285,8 +285,7 @@ public void testIOException() throws InterruptedException, IOException { // released (since it was never obtained), and the IOException should be logged. verify(subtaskExecutor, times(0)).execAlgorithm(); verify(subtaskMaster, times(0)).releaseWriteLock( - Paths.get(TASK_DIR, "st-" + SUBTASK_INDEX, TaskConfigurationManager.LOCK_FILE_NAME) - .toFile()); + Paths.get(TASK_DIR, "st-" + SUBTASK_INDEX, TaskConfiguration.LOCK_FILE_NAME).toFile()); verify(subtaskMaster).logException(ArgumentMatchers.eq(SUBTASK_INDEX), ArgumentMatchers.any(PipelineException.class)); assertEquals(5, completionCounter.availablePermits()); @@ -311,7 +310,7 @@ private void standardSetUp() throws InterruptedException, IOException { // Mock a successful lock of the subtask lock file. doReturn(true).when(subtaskMaster) .getWriteLockWithoutBlocking( - Paths.get(TASK_DIR, "st-" + SUBTASK_INDEX, TaskConfigurationManager.LOCK_FILE_NAME) + Paths.get(TASK_DIR, "st-" + SUBTASK_INDEX, TaskConfiguration.LOCK_FILE_NAME) .toFile()); // The subtask should have no prior algorithm state file. @@ -322,6 +321,5 @@ private void standardSetUp() throws InterruptedException, IOException { // The SubtaskExecutor should return zero. when(subtaskExecutor.execAlgorithm()).thenReturn(0); - } } diff --git a/src/test/java/gov/nasa/ziggy/module/SubtaskServerTest.java b/src/test/java/gov/nasa/ziggy/module/SubtaskServerTest.java index 59a03c0..8d298c1 100644 --- a/src/test/java/gov/nasa/ziggy/module/SubtaskServerTest.java +++ b/src/test/java/gov/nasa/ziggy/module/SubtaskServerTest.java @@ -30,7 +30,7 @@ public class SubtaskServerTest { @Before public void setUp() { subtaskAllocator = mock(SubtaskAllocator.class); - subtaskServer = spy(new SubtaskServer(50, new TaskConfigurationManager())); + subtaskServer = spy(new SubtaskServer(50, new TaskConfiguration())); doReturn(subtaskAllocator).when(subtaskServer).subtaskAllocator(); subtaskClient = new SubtaskClient(); } diff --git a/src/test/java/gov/nasa/ziggy/module/TaskConfigurationManagerTest.java b/src/test/java/gov/nasa/ziggy/module/TaskConfigurationManagerTest.java index 1bbf7e3..2fbe822 100644 --- a/src/test/java/gov/nasa/ziggy/module/TaskConfigurationManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/module/TaskConfigurationManagerTest.java @@ -6,8 +6,6 @@ import static org.junit.Assert.assertTrue; import java.io.File; -import java.util.Set; -import java.util.TreeSet; import org.junit.Before; import org.junit.Rule; @@ -20,7 +18,6 @@ public class TaskConfigurationManagerTest { private File taskDir; - private Set t1, t2, t3, t4, t5, t6, single; @Rule public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -28,13 +25,6 @@ public class TaskConfigurationManagerTest { @Before public void setup() { taskDir = directoryRule.directory().toFile(); - t1 = new TreeSet<>(); - t2 = new TreeSet<>(); - t3 = new TreeSet<>(); - t4 = new TreeSet<>(); - t5 = new TreeSet<>(); - t6 = new TreeSet<>(); - single = new TreeSet<>(); } /** @@ -42,107 +32,15 @@ public void setup() { */ @Test public void testConstructors() { - TaskConfigurationManager h = new TaskConfigurationManager(); + TaskConfiguration h = new TaskConfiguration(); assertNull(h.getTaskDir()); - h = new TaskConfigurationManager(taskDir); + h = new TaskConfiguration(taskDir); assertEquals(taskDir, h.getTaskDir()); } - /** - * Tests the addSubTaskInputs method. Also exercises the getCurrentSubTaskIndex(), - * subTaskDirectory(), and numInputs() methods. - */ - @Test - public void testAddSubTaskInputs() { - TaskConfigurationManager h = new TaskConfigurationManager(taskDir); - h.addFilesForSubtask(t1); - h.addFilesForSubtask(t2); - h.addFilesForSubtask(t3); - h.addFilesForSubtask(single); - - assertTrue(new File(taskDir, "st-0").exists()); - assertTrue(new File(taskDir, "st-1").exists()); - assertTrue(new File(taskDir, "st-2").exists()); - assertTrue(new File(taskDir, "st-3").exists()); - - assertEquals(4, h.getSubtaskCount()); - assertEquals(4, h.numInputs()); - } - - /** - * Tests the subTaskUnitOfWork method. - */ - @Test - public void testSubTaskUnitOfWork() { - TaskConfigurationManager h = new TaskConfigurationManager(taskDir); - h.addFilesForSubtask(t1); - h.addFilesForSubtask(t2); - h.addFilesForSubtask(t3); - - Set u = h.filesForSubtask(2); - assertEquals(t3, u); - } - - /** - * Tests the validate() method, including the case in which it sets the default processing to - * cover all sub-tasks in parallel. - */ - @Test - public void testValidate() { - TaskConfigurationManager h = new TaskConfigurationManager(taskDir); - h.addFilesForSubtask(t1); - h.addFilesForSubtask(t2); - h.addFilesForSubtask(t3); - h.addFilesForSubtask(t4); - h.addFilesForSubtask(t5); - h.addFilesForSubtask(t6); - h.validate(); - assertEquals(6, h.getSubtaskCount()); - } - - /** - * Exercises the subTaskDirectory() method. - */ - @Test - public void testSubTaskDirectory() { - TaskConfigurationManager h = new TaskConfigurationManager(taskDir); - File f = h.subtaskDirectory(); - assertEquals(new File(taskDir, "st-0").getAbsolutePath(), f.getAbsolutePath()); - h.addFilesForSubtask(t1); - f = h.subtaskDirectory(); - assertEquals(new File(taskDir, "st-1").getAbsolutePath(), f.getAbsolutePath()); - } - - /** - * Exercises the isEmpty() method. - */ - @Test - public void testIsEmpty() { - TaskConfigurationManager h = new TaskConfigurationManager(taskDir); - assertTrue(h.isEmpty()); - h.addFilesForSubtask(t1); - assertFalse(h.isEmpty()); - } - - /** - * Exercises the toString() method. - */ - @Test - public void testToString() { - TaskConfigurationManager h = new TaskConfigurationManager(taskDir); - h.addFilesForSubtask(t1); - h.addFilesForSubtask(t2); - h.addFilesForSubtask(t3); - h.addFilesForSubtask(t4); - h.addFilesForSubtask(t5); - h.addFilesForSubtask(t6); - String s = h.toString(); - assertEquals("SINGLE:[0,5]", s); - } - @Test public void testInputOutputClassHandling() { - TaskConfigurationManager h1 = new TaskConfigurationManager(taskDir); + TaskConfiguration h1 = new TaskConfiguration(taskDir); h1.setInputsClass(PipelineInputsSample.class); h1.setOutputsClass(PipelineOutputsSample1.class); Class ci = h1.getInputsClass(); @@ -157,19 +55,13 @@ public void testInputOutputClassHandling() { */ @Test public void testPersistRestore() { - TaskConfigurationManager h1 = new TaskConfigurationManager(taskDir); - h1.addFilesForSubtask(t1); - h1.addFilesForSubtask(t2); - h1.addFilesForSubtask(t3); - h1.addFilesForSubtask(t4); - h1.addFilesForSubtask(t5); - h1.addFilesForSubtask(t6); + TaskConfiguration h1 = new TaskConfiguration(taskDir); h1.setInputsClass(PipelineInputsSample.class); h1.setOutputsClass(PipelineOutputsSample1.class); - assertFalse(TaskConfigurationManager.isPersistedInputsHandlerPresent(h1.getTaskDir())); - h1.persist(); - assertTrue(TaskConfigurationManager.isPersistedInputsHandlerPresent(h1.getTaskDir())); - TaskConfigurationManager h2 = TaskConfigurationManager.restore(h1.getTaskDir()); + assertFalse(TaskConfiguration.isSerializedTaskConfigurationPresent(h1.getTaskDir())); + h1.serialize(); + assertTrue(TaskConfiguration.isSerializedTaskConfigurationPresent(h1.getTaskDir())); + TaskConfiguration h2 = TaskConfiguration.deserialize(h1.getTaskDir()); assertEquals(h1, h2); } } diff --git a/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java b/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java index 39b8b10..957102c 100644 --- a/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/TaskMonitorTest.java @@ -3,8 +3,6 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; -import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.when; import java.io.File; import java.io.IOException; @@ -68,10 +66,7 @@ public void setUp() throws IOException, ConfigurationException { stateFile.setNumTotal(subtaskDirectories.size()); stateFile.persist(); - TaskConfigurationManager inputsHandler = mock(TaskConfigurationManager.class); - when(inputsHandler.allSubTaskDirectories()).thenReturn(subtaskDirectories); - - taskMonitor = new TaskMonitor(inputsHandler, stateFile, taskDir.toFile()); + taskMonitor = new TaskMonitor(stateFile, taskDir.toFile()); } @Test diff --git a/src/test/java/gov/nasa/ziggy/module/hdf5/ModuleParametersHdf5ArrayTest.java b/src/test/java/gov/nasa/ziggy/module/hdf5/ModuleParametersHdf5ArrayTest.java index e65ef24..f992114 100644 --- a/src/test/java/gov/nasa/ziggy/module/hdf5/ModuleParametersHdf5ArrayTest.java +++ b/src/test/java/gov/nasa/ziggy/module/hdf5/ModuleParametersHdf5ArrayTest.java @@ -18,7 +18,6 @@ import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.module.io.Persistable; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.parameters.ModuleParameters; import gov.nasa.ziggy.parameters.Parameters; import gov.nasa.ziggy.parameters.ParametersInterface; @@ -78,13 +77,9 @@ public void testWriteAndRead() { hdf5ModuleInterface.readFile(testFile, loadArticle, true); ModuleParameters m = loadArticle.getModuleParameters(); List p = m.getModuleParameters(); - assertEquals(2, p.size()); + assertEquals(1, p.size()); Parameters d; - if (p.get(0) instanceof RemoteParameters) { - d = (Parameters) p.get(1); - } else { - d = (Parameters) p.get(0); - } + d = (Parameters) p.get(0); assertEquals(d.getName(), "test default parameters"); Set t = d.getParameters(); assertEquals(4, t.size()); @@ -113,7 +108,6 @@ public void testWriteAndRead() { private ModuleParameters populateModuleParameters() { ModuleParameters m = new ModuleParameters(); List p = m.getModuleParameters(); - p.add(new RemoteParameters()); p.add(parameters()); return m; diff --git a/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample2.java b/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample2.java index b3b2d1b..b877deb 100644 --- a/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample2.java +++ b/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample2.java @@ -53,7 +53,8 @@ public boolean equals(Object obj) { || !Objects.equals(persistableList, other.persistableList)) { return false; } - if (!Objects.equals(persistableScalar1, other.persistableScalar1) || !Objects.equals(persistableScalar2, other.persistableScalar2)) { + if (!Objects.equals(persistableScalar1, other.persistableScalar1) + || !Objects.equals(persistableScalar2, other.persistableScalar2)) { return false; } return true; diff --git a/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample3.java b/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample3.java index 5a6ac0b..b287f35 100644 --- a/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample3.java +++ b/src/test/java/gov/nasa/ziggy/module/hdf5/PersistableSample3.java @@ -84,7 +84,7 @@ public boolean equals(Object obj) { || !Objects.equals(boxedIntVar, other.boxedIntVar) || enumScalar != other.enumScalar) { return false; } - if ((intVar != other.intVar) || !Objects.equals(stringVar, other.stringVar)) { + if (intVar != other.intVar || !Objects.equals(stringVar, other.stringVar)) { return false; } return true; diff --git a/src/test/java/gov/nasa/ziggy/module/io/matlab/MatlabUtilsTest.java b/src/test/java/gov/nasa/ziggy/module/io/matlab/MatlabUtilsTest.java index cee2c91..cc31bf3 100644 --- a/src/test/java/gov/nasa/ziggy/module/io/matlab/MatlabUtilsTest.java +++ b/src/test/java/gov/nasa/ziggy/module/io/matlab/MatlabUtilsTest.java @@ -1,16 +1,24 @@ package gov.nasa.ziggy.module.io.matlab; +import static gov.nasa.ziggy.services.config.PropertyName.ARCHITECTURE; +import static gov.nasa.ziggy.services.config.PropertyName.OPERATING_SYSTEM; import static org.junit.Assert.assertEquals; +import org.junit.Rule; import org.junit.Test; -import gov.nasa.ziggy.util.os.OperatingSystemType; +import gov.nasa.ziggy.ZiggyPropertyRule; public class MatlabUtilsTest { + @Rule + public ZiggyPropertyRule osName = new ZiggyPropertyRule(OPERATING_SYSTEM, "Linux"); + + @Rule + public ZiggyPropertyRule architecture = new ZiggyPropertyRule(ARCHITECTURE, (String) null); + @Test public void testLinuxMcrPath() { - MatlabUtils.setOsType(OperatingSystemType.LINUX); String mPath = MatlabUtils.mcrPaths("/path/to/mcr/v93"); String mPathExpect = """ /path/to/mcr/v93/runtime/glnxa64:\ @@ -22,8 +30,8 @@ public void testLinuxMcrPath() { @Test public void testOsXIntelMcrPath() { - MatlabUtils.setOsType(OperatingSystemType.MAC_OS_X); - MatlabUtils.setArchitecture("x86_64"); + osName.setValue("Mac OS X"); + architecture.setValue("x86_64"); String mPath = MatlabUtils.mcrPaths("/path/to/mcr/v93"); String mPathExpect = """ /path/to/mcr/v93/runtime/maci64:\ @@ -34,8 +42,8 @@ public void testOsXIntelMcrPath() { @Test public void testOsXM1McrPath() { - MatlabUtils.setOsType(OperatingSystemType.MAC_OS_X); - MatlabUtils.setArchitecture("aarch"); + osName.setValue("Mac OS X"); + architecture.setValue("aarch"); String mPath = MatlabUtils.mcrPaths("/path/to/mcr/v93"); String mPathExpect = """ /path/to/mcr/v93/runtime/maca64:\ diff --git a/src/test/java/gov/nasa/ziggy/module/remote/PbsParametersTest.java b/src/test/java/gov/nasa/ziggy/module/remote/PbsParametersTest.java index 0d32ea6..d6d1a5a 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/PbsParametersTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/PbsParametersTest.java @@ -7,6 +7,8 @@ import org.junit.Before; import org.junit.Test; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; + /** * Unit test class for {@link PbsParameters} class. * @@ -14,25 +16,25 @@ */ public class PbsParametersTest { - private RemoteParameters remoteParameters; + private PipelineDefinitionNodeExecutionResources executionResources; private RemoteNodeDescriptor descriptor; private PbsParameters pbsParameters; @Before public void setup() { descriptor = RemoteNodeDescriptor.SANDY_BRIDGE; - remoteParameters = new RemoteParameters(); - remoteParameters.setRemoteNodeArchitecture(descriptor.getNodeName()); - remoteParameters.setEnabled(true); - remoteParameters.setGigsPerSubtask(6); - remoteParameters.setSubtaskMaxWallTimeHours(4.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); + executionResources = new PipelineDefinitionNodeExecutionResources("dummy", "dummy"); + executionResources.setRemoteNodeArchitecture(descriptor.getNodeName()); + executionResources.setRemoteExecutionEnabled(true); + executionResources.setGigsPerSubtask(6); + executionResources.setSubtaskMaxWallTimeHours(4.5); + executionResources.setSubtaskTypicalWallTimeHours(0.5); } @Test public void testSimpleCase() { pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("normal", pbsParameters.getQueueName()); assertEquals(12, pbsParameters.getRequestedNodeCount()); @@ -43,9 +45,9 @@ public void testSimpleCase() { @Test public void testNodeCountOverride() { - remoteParameters.setMaxNodes("1"); + executionResources.setMaxNodes(1); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("long", pbsParameters.getQueueName()); assertEquals(1, pbsParameters.getRequestedNodeCount()); @@ -56,9 +58,9 @@ public void testNodeCountOverride() { @Test public void testNodeCountOverrideSmallSubtaskCount() { - remoteParameters.setMaxNodes("10"); + executionResources.setMaxNodes(10); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 5); + pbsParameters.populateResourceParameters(executionResources, 5); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals(1, pbsParameters.getRequestedNodeCount()); assertEquals(5, pbsParameters.getActiveCoresPerNode()); @@ -69,9 +71,9 @@ public void testNodeCountOverrideSmallSubtaskCount() { @Test public void testSmallRamRequest() { - remoteParameters.setGigsPerSubtask(0.5); + executionResources.setGigsPerSubtask(0.5); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("normal", pbsParameters.getQueueName()); assertEquals(4, pbsParameters.getRequestedNodeCount()); @@ -82,9 +84,9 @@ public void testSmallRamRequest() { @Test public void testSmallTask() { - remoteParameters.setGigsPerSubtask(0.5); + executionResources.setGigsPerSubtask(0.5); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 10); + pbsParameters.populateResourceParameters(executionResources, 10); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("normal", pbsParameters.getQueueName()); assertEquals(1, pbsParameters.getRequestedNodeCount()); @@ -95,9 +97,9 @@ public void testSmallTask() { @Test public void testSubtaskPerCoreOverride() { - remoteParameters.setSubtasksPerCore("12"); + executionResources.setSubtasksPerCore(12.0); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("normal", pbsParameters.getQueueName()); assertEquals(9, pbsParameters.getRequestedNodeCount()); @@ -105,9 +107,9 @@ public void testSubtaskPerCoreOverride() { assertEquals(25.38, pbsParameters.getEstimatedCost(), 1e-9); assertEquals("6:00:00", pbsParameters.getRequestedWallTime()); - remoteParameters.setSubtasksPerCore("6"); + executionResources.setSubtasksPerCore(6.0); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("normal", pbsParameters.getQueueName()); assertEquals(12, pbsParameters.getRequestedNodeCount()); @@ -119,10 +121,10 @@ public void testSubtaskPerCoreOverride() { @Test public void testNodeSharingDisabled() { pbsParameters(); - remoteParameters.setSubtaskTypicalWallTimeHours(4.5); - remoteParameters.setNodeSharing(false); - remoteParameters.setWallTimeScaling(false); - pbsParameters.populateResourceParameters(remoteParameters, 500); + executionResources.setSubtaskTypicalWallTimeHours(4.5); + executionResources.setNodeSharing(false); + executionResources.setWallTimeScaling(false); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(500, pbsParameters.getRequestedNodeCount()); assertEquals("4:30:00", pbsParameters.getRequestedWallTime()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); @@ -132,10 +134,10 @@ public void testNodeSharingDisabled() { @Test public void testNodeSharingDisabledTimeScalingEnabled() { pbsParameters(); - remoteParameters.setSubtaskTypicalWallTimeHours(4.5); - remoteParameters.setNodeSharing(false); - remoteParameters.setWallTimeScaling(true); - pbsParameters.populateResourceParameters(remoteParameters, 500); + executionResources.setSubtaskTypicalWallTimeHours(4.5); + executionResources.setNodeSharing(false); + executionResources.setWallTimeScaling(true); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(500, pbsParameters.getRequestedNodeCount()); assertEquals("0:30:00", pbsParameters.getRequestedWallTime()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); @@ -144,9 +146,9 @@ public void testNodeSharingDisabledTimeScalingEnabled() { @Test public void testQueueNameOverride() { - remoteParameters.setQueueName("long"); + executionResources.setQueueName("long"); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals("long", pbsParameters.getQueueName()); assertEquals(12, pbsParameters.getRequestedNodeCount()); @@ -157,9 +159,9 @@ public void testQueueNameOverride() { @Test public void testQueueNameForReservation() { - remoteParameters.setQueueName("R14950266"); + executionResources.setQueueName("R14950266"); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); assertEquals("R14950266", pbsParameters.getQueueName()); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, pbsParameters.getArchitecture()); assertEquals(12, pbsParameters.getRequestedNodeCount()); @@ -170,29 +172,29 @@ public void testQueueNameForReservation() { @Test(expected = IllegalStateException.class) public void testBadQueueOverride() { - remoteParameters.setQueueName("low"); + executionResources.setQueueName("low"); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 500); + pbsParameters.populateResourceParameters(executionResources, 500); } @Test(expected = IllegalStateException.class) public void testNoQueuePossible() { - remoteParameters.setMaxNodes("1"); + executionResources.setMaxNodes(1); pbsParameters(); - pbsParameters.populateResourceParameters(remoteParameters, 5000); + pbsParameters.populateResourceParameters(executionResources, 5000); } @Test public void testArchitectureOverride() { pbsParameters(); - pbsParameters.populateArchitecture(remoteParameters, 500, SupportedRemoteClusters.NAS); + pbsParameters.populateArchitecture(executionResources, 500, SupportedRemoteClusters.NAS); } @Test(expected = IllegalStateException.class) public void testBadArchitectureOverride() { - remoteParameters.setGigsPerSubtask(1000); + executionResources.setGigsPerSubtask(1000); pbsParameters(); - pbsParameters.populateArchitecture(remoteParameters, 500, SupportedRemoteClusters.NAS); + pbsParameters.populateArchitecture(executionResources, 500, SupportedRemoteClusters.NAS); } /** @@ -202,11 +204,11 @@ public void testBadArchitectureOverride() { @Test public void testSelectQueueSmallJob() { pbsParameters(); - remoteParameters.setMaxNodes("5"); - remoteParameters.setSubtaskMaxWallTimeHours(0.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); - remoteParameters.setGigsPerSubtask(2.0); - pbsParameters.populateResourceParameters(remoteParameters, 50); + executionResources.setMaxNodes(5); + executionResources.setSubtaskMaxWallTimeHours(0.5); + executionResources.setSubtaskTypicalWallTimeHours(0.5); + executionResources.setGigsPerSubtask(2.0); + pbsParameters.populateResourceParameters(executionResources, 50); assertEquals(RemoteQueueDescriptor.LOW.getQueueName(), pbsParameters.getQueueName()); assertEquals("0:30:00", pbsParameters.getRequestedWallTime()); } @@ -215,10 +217,10 @@ public void testSelectQueueSmallJob() { public void testAggregatePbsParameters() { pbsParameters(); PbsParameters parameterSet1 = pbsParameters; - parameterSet1.populateResourceParameters(remoteParameters, 500); + parameterSet1.populateResourceParameters(executionResources, 500); pbsParameters(); PbsParameters parameterSet2 = pbsParameters; - parameterSet2.populateResourceParameters(remoteParameters, 500); + parameterSet2.populateResourceParameters(executionResources, 500); parameterSet2.setActiveCoresPerNode(3); parameterSet2.setRequestedWallTime("20:00:00"); parameterSet2.setQueueName("long"); @@ -233,7 +235,7 @@ public void testAggregatePbsParameters() { } private void pbsParameters() { - pbsParameters = remoteParameters.pbsParametersInstance(); + pbsParameters = executionResources.pbsParametersInstance(); pbsParameters.setMinCoresPerNode(descriptor.getMinCores()); pbsParameters .setMinGigsPerNode((int) (descriptor.getMinCores() * descriptor.getGigsPerCore())); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java b/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java index 1be89d8..af0d63f 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/QueueCommandManagerTest.java @@ -171,13 +171,13 @@ public void testDeleteJobsForPipelineTasks() { // set up the returns for the qstat commands that are looking for the tasks in // the queue -- NB, there is no job in the queue for task 3. String jobName = task1.taskBaseName(); - String[] grepArgs = { new String(jobName) }; + String[] grepArgs = { jobName }; mockQstatCall("-u user", grepArgs, qstatOutputLine(task1, 1234567L)); jobName = task2.taskBaseName(); - grepArgs = new String[] { new String(jobName) }; + grepArgs = new String[] { jobName }; mockQstatCall("-u user", grepArgs, qstatOutputLine(task1, 7654321L)); jobName = task3.taskBaseName(); - grepArgs = new String[] { new String(jobName) }; + grepArgs = new String[] { jobName }; mockQstatCall("-u user", grepArgs, (String[]) null); grepArgs = new String[] { "Job:", "Job_Owner" }; diff --git a/src/test/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizerTest.java b/src/test/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizerTest.java index 7cd0788..41657c9 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizerTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/RemoteArchitectureOptimizerTest.java @@ -10,6 +10,7 @@ import org.mockito.Mockito; import gov.nasa.ziggy.module.remote.nas.NasQueueTimeMetrics; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; /** * Unit test class for {@link RemoteArchitectureOptimizer} class. @@ -48,26 +49,27 @@ public void testOptimizeForCores() { RemoteArchitectureOptimizer optimizer = RemoteArchitectureOptimizer.CORES; List descriptors = RemoteNodeDescriptor .descriptorsSortedByRamThenCost(SupportedRemoteClusters.NAS); - RemoteParameters remoteParameters = new RemoteParameters(); - remoteParameters.setGigsPerSubtask(0.5); - RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(remoteParameters, 0, + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setGigsPerSubtask(0.5); + RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(executionResources, 0, RemoteNodeDescriptor.nodesWithSufficientRam(descriptors, - remoteParameters.getGigsPerSubtask())); + executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, descriptor); - remoteParameters.setGigsPerSubtask(3.0); - descriptor = optimizer.optimalArchitecture(remoteParameters, 0, RemoteNodeDescriptor - .nodesWithSufficientRam(descriptors, remoteParameters.getGigsPerSubtask())); + executionResources.setGigsPerSubtask(3.0); + descriptor = optimizer.optimalArchitecture(executionResources, 0, RemoteNodeDescriptor + .nodesWithSufficientRam(descriptors, executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.IVY_BRIDGE, descriptor); - remoteParameters.setGigsPerSubtask(4.2); - descriptor = optimizer.optimalArchitecture(remoteParameters, 0, RemoteNodeDescriptor - .nodesWithSufficientRam(descriptors, remoteParameters.getGigsPerSubtask())); + executionResources.setGigsPerSubtask(4.2); + descriptor = optimizer.optimalArchitecture(executionResources, 0, RemoteNodeDescriptor + .nodesWithSufficientRam(descriptors, executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.BROADWELL, descriptor); - remoteParameters.setGigsPerSubtask(10); - descriptor = optimizer.optimalArchitecture(remoteParameters, 0, RemoteNodeDescriptor - .nodesWithSufficientRam(descriptors, remoteParameters.getGigsPerSubtask())); + executionResources.setGigsPerSubtask(10); + descriptor = optimizer.optimalArchitecture(executionResources, 0, RemoteNodeDescriptor + .nodesWithSufficientRam(descriptors, executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.HASWELL, descriptor); } @@ -76,13 +78,14 @@ public void testOptimizeForCost() { RemoteArchitectureOptimizer optimizer = RemoteArchitectureOptimizer.COST; List descriptors = RemoteNodeDescriptor .descriptorsSortedByCost(SupportedRemoteClusters.NAS); - RemoteParameters remoteParameters = new RemoteParameters(); - remoteParameters.setGigsPerSubtask(6.0); - remoteParameters.setSubtaskMaxWallTimeHours(4.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); - RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(remoteParameters, 500, + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setGigsPerSubtask(6.0); + executionResources.setSubtaskMaxWallTimeHours(4.5); + executionResources.setSubtaskTypicalWallTimeHours(0.5); + RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(executionResources, 500, RemoteNodeDescriptor.nodesWithSufficientRam(descriptors, - remoteParameters.getGigsPerSubtask())); + executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.HASWELL, descriptor); } @@ -91,13 +94,14 @@ public void testOptimizeForQueueDepth() { RemoteArchitectureOptimizer optimizer = RemoteArchitectureOptimizer.QUEUE_DEPTH; List descriptors = RemoteNodeDescriptor .descriptorsSortedByCost(SupportedRemoteClusters.NAS); - RemoteParameters remoteParameters = new RemoteParameters(); - remoteParameters.setGigsPerSubtask(6.0); - remoteParameters.setSubtaskMaxWallTimeHours(4.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); - RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(remoteParameters, 500, + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setGigsPerSubtask(6.0); + executionResources.setSubtaskMaxWallTimeHours(4.5); + executionResources.setSubtaskTypicalWallTimeHours(0.5); + RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(executionResources, 500, RemoteNodeDescriptor.nodesWithSufficientRam(descriptors, - remoteParameters.getGigsPerSubtask())); + executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.ROME, descriptor); } @@ -106,13 +110,14 @@ public void testOptimizeForQueueTime() { RemoteArchitectureOptimizer optimizer = RemoteArchitectureOptimizer.QUEUE_TIME; List descriptors = RemoteNodeDescriptor .descriptorsSortedByCost(SupportedRemoteClusters.NAS); - RemoteParameters remoteParameters = new RemoteParameters(); - remoteParameters.setGigsPerSubtask(6.0); - remoteParameters.setSubtaskMaxWallTimeHours(4.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); - RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(remoteParameters, 500, + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setGigsPerSubtask(6.0); + executionResources.setSubtaskMaxWallTimeHours(4.5); + executionResources.setSubtaskTypicalWallTimeHours(0.5); + RemoteNodeDescriptor descriptor = optimizer.optimalArchitecture(executionResources, 500, RemoteNodeDescriptor.nodesWithSufficientRam(descriptors, - remoteParameters.getGigsPerSubtask())); + executionResources.getGigsPerSubtask())); assertEquals(RemoteNodeDescriptor.IVY_BRIDGE, descriptor); } } diff --git a/src/test/java/gov/nasa/ziggy/module/remote/RemoteParametersTest.java b/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutionConfigurationTest.java similarity index 54% rename from src/test/java/gov/nasa/ziggy/module/remote/RemoteParametersTest.java rename to src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutionConfigurationTest.java index 9b7825e..0184725 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/RemoteParametersTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutionConfigurationTest.java @@ -7,45 +7,50 @@ import org.apache.commons.lang3.builder.EqualsBuilder; import org.junit.Test; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; + /** - * Test class for {@link RemoteParameters} class. + * Test class for {@link PipelineDefinitionNodeExecutionResources} class. * * @author PT */ -public class RemoteParametersTest { +public class RemoteExecutionConfigurationTest { @Test public void testCopyConstructor() { - RemoteParameters r1 = new RemoteParameters(); - r1.setEnabled(true); + PipelineDefinitionNodeExecutionResources r1 = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + r1.setRemoteExecutionEnabled(true); r1.setGigsPerSubtask(2.0); - r1.setMaxNodes("3"); - r1.setMinCoresPerNode("4"); - r1.setMinGigsPerNode("5"); - r1.setOptimizer("CORES"); + r1.setMaxNodes(3); + r1.setMinCoresPerNode(4); + r1.setMinGigsPerNode(5.0); + r1.setOptimizer(RemoteArchitectureOptimizer.CORES); r1.setQueueName("low"); r1.setRemoteNodeArchitecture("bro"); r1.setSubtaskMaxWallTimeHours(9); - r1.setSubtasksPerCore("1.5"); + r1.setSubtasksPerCore(1.5); r1.setSubtaskTypicalWallTimeHours(0.5); - RemoteParameters r2 = new RemoteParameters(r1); + PipelineDefinitionNodeExecutionResources r2 = new PipelineDefinitionNodeExecutionResources( + r1); assertTrue(EqualsBuilder.reflectionEquals(r2, r1)); } @Test public void testPbsParametersInstance() { - RemoteParameters r1 = new RemoteParameters(); - r1.setEnabled(true); + PipelineDefinitionNodeExecutionResources r1 = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + r1.setRemoteExecutionEnabled(true); r1.setGigsPerSubtask(2.0); - r1.setMaxNodes("3"); - r1.setMinCoresPerNode("4"); - r1.setMinGigsPerNode("5"); - r1.setOptimizer("CORES"); + r1.setMaxNodes(3); + r1.setMinCoresPerNode(4); + r1.setMinGigsPerNode(5.0); + r1.setOptimizer(RemoteArchitectureOptimizer.CORES); r1.setQueueName("low"); r1.setRemoteNodeArchitecture("bro"); r1.setSubtaskMaxWallTimeHours(9); - r1.setSubtasksPerCore("1.5"); + r1.setSubtasksPerCore(1.5); r1.setSubtaskTypicalWallTimeHours(0.5); PbsParameters p1 = r1.pbsParametersInstance(); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java index d1cffe5..e6588bc 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/RemoteExecutorTest.java @@ -18,19 +18,23 @@ import org.junit.Rule; import org.junit.Test; import org.junit.rules.RuleChain; +import org.mockito.ArgumentMatchers; +import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.module.StateFile; -import gov.nasa.ziggy.module.TaskConfigurationManager; +import gov.nasa.ziggy.module.TaskConfiguration; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.pipeline.definition.PipelineTask.ProcessingSummary; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; +import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionNodeCrud; import gov.nasa.ziggy.pipeline.definition.crud.ProcessingSummaryOperations; import gov.nasa.ziggy.services.database.DatabaseService; import gov.nasa.ziggy.services.database.SingleThreadExecutor; -import gov.nasa.ziggy.uow.TaskConfigurationParameters; /** * Class that provides unit tests for the {@link RemoteExecutor} abstract class. @@ -39,16 +43,16 @@ */ public class RemoteExecutorTest { - private ParameterSetCrud parameterSetCrud; private GenericRemoteExecutor executor; private PipelineTask pipelineTask; private ProcessingSummaryOperations crud; - private TaskConfigurationManager taskConfigurationManager; + private TaskConfiguration taskConfigurationManager; private PipelineInstance pipelineInstance; private ProcessingSummary taskAttr; private DatabaseService databaseService; private static Future futureVoid; - private static TaskConfigurationParameters tcp; + private static PipelineDefinitionNodeCrud defNodeCrud = Mockito + .mock(PipelineDefinitionNodeCrud.class); public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -71,33 +75,25 @@ public void setup() throws InterruptedException, ExecutionException { new File(taskDir, "st-0").mkdirs(); new File(taskDir, "st-1").mkdirs(); - parameterSetCrud = mock(ParameterSetCrud.class); pipelineTask = mock(PipelineTask.class); pipelineInstance = mock(PipelineInstance.class); crud = mock(ProcessingSummaryOperations.class); - taskConfigurationManager = mock(TaskConfigurationManager.class); + taskConfigurationManager = mock(TaskConfiguration.class); taskAttr = mock(ProcessingSummary.class); executor = new GenericRemoteExecutor(pipelineTask); - executor.setParameterSetCrud(parameterSetCrud); executor.setProcessingSummaryOperations(crud); databaseService = mock(DatabaseService.class); DatabaseService.setInstance(databaseService); futureVoid = mock(Future.class); - tcp = mock(TaskConfigurationParameters.class); - when(pipelineTask.getParameters(RemoteParameters.class, false)) - .thenReturn(remoteParametersForPipelineTask()); - when(pipelineTask.getParameters(TaskConfigurationParameters.class)).thenReturn(tcp); - when(tcp.getMaxFailedSubtaskCount()).thenReturn(0); when(pipelineTask.getPipelineInstance()).thenReturn(pipelineInstance); when(pipelineTask.pipelineInstanceId()).thenReturn(10L); when(pipelineTask.getModuleName()).thenReturn("modulename"); when(pipelineTask.getId()).thenReturn(50L); when(pipelineTask.taskBaseName()).thenReturn("10-50-modulename"); + when(pipelineTask.pipelineDefinitionNode()).thenReturn(new PipelineDefinitionNode()); when(pipelineInstance.getId()).thenReturn(10L); - when(parameterSetCrud.retrieveRemoteParameters(pipelineTask)) - .thenReturn(remoteParametersFromDatabase()); - when(taskConfigurationManager.numSubTasks()).thenReturn(500); + when(taskConfigurationManager.getSubtaskCount()).thenReturn(500); when(crud.processingSummary(50L)).thenReturn(taskAttr); when(taskAttr.getTotalSubtaskCount()).thenReturn(500); when(taskAttr.getCompletedSubtaskCount()).thenReturn(400); @@ -112,12 +108,18 @@ public void teardown() throws IOException { @Test public void testExecuteAlgorithmFirstIteration() { + + when(defNodeCrud + .retrieveExecutionResources(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(remoteExecutionConfigurationForPipelineTask()); + executor.submitAlgorithm(taskConfigurationManager); // The correct call to generate PBS parameters should have occurred. GenericRemoteExecutor gExecutor = executor; assertEquals(500, gExecutor.totalSubtaskCount); - checkPbsParameterValues(remoteParametersForPipelineTask(), gExecutor.pbsParameters); + checkPbsParameterValues(remoteExecutionConfigurationForPipelineTask(), + gExecutor.pbsParameters); assertEquals(RemoteNodeDescriptor.BROADWELL, gExecutor.pbsParameters.getArchitecture()); // The correct calls should have occurred. @@ -136,24 +138,27 @@ public void testExecuteAlgorithmFirstIteration() { // Make sure that the things that should not get called did not, in fact, // get called. verify(crud, never()).processingSummary(any(long.class)); - verify(parameterSetCrud, never()).retrieveRemoteParameters(any(PipelineTask.class)); } @Test public void testExecuteAlgorithmLaterIteration() { + when(defNodeCrud + .retrieveExecutionResources(ArgumentMatchers.any(PipelineDefinitionNode.class))) + .thenReturn(remoteExecutionConfigurationFromDatabase()); + executor.submitAlgorithm(null); // The CRUDs should have been called verify(crud).processingSummary(50L); - verify(parameterSetCrud).retrieveRemoteParameters(pipelineTask); // The correct call to generate PBS parameters should have occurred, // including the fact that the number of remaining subtasks should be // lower. GenericRemoteExecutor gExecutor = executor; assertEquals(100, gExecutor.totalSubtaskCount); - checkPbsParameterValues(remoteParametersFromDatabase(), gExecutor.pbsParameters); + checkPbsParameterValues(remoteExecutionConfigurationFromDatabase(), + gExecutor.pbsParameters); assertEquals(RemoteNodeDescriptor.ROME, gExecutor.pbsParameters.getArchitecture()); // The correct calls should have occurred, including that @@ -172,58 +177,58 @@ public void testExecuteAlgorithmLaterIteration() { assertEquals(stateFile, gExecutor.monitoredStateFile); } - private void checkPbsParameterValues(RemoteParameters rParameters, PbsParameters pParameters) { + private void checkPbsParameterValues( + PipelineDefinitionNodeExecutionResources executionResources, PbsParameters pParameters) { - assertEquals(rParameters.getQueueName(), pParameters.getQueueName()); - assertEquals(rParameters.getMinCoresPerNode(), - Integer.toString(pParameters.getMinCoresPerNode())); - assertEquals(rParameters.getMinGigsPerNode(), - Integer.toString(pParameters.getMinGigsPerNode())); - assertEquals(rParameters.getMaxNodes(), - Integer.toString(pParameters.getRequestedNodeCount())); + assertEquals(executionResources.getQueueName(), pParameters.getQueueName()); + assertEquals(executionResources.getMinCoresPerNode(), pParameters.getMinCoresPerNode()); + assertEquals(executionResources.getMinGigsPerNode(), pParameters.getMinGigsPerNode(), 1e-3); + assertEquals(executionResources.getMaxNodes(), pParameters.getRequestedNodeCount()); } private void checkStateFileValues(PbsParameters pParameters, StateFile stateFile) { assertEquals(pParameters.getArchitecture().getNodeName(), stateFile.getRemoteNodeArchitecture()); assertEquals(pParameters.getQueueName(), stateFile.getQueueName()); - assertEquals(pParameters.getMinGigsPerNode(), stateFile.getMinGigsPerNode()); + assertEquals(pParameters.getMinGigsPerNode(), stateFile.getMinGigsPerNode(), 1e-3); assertEquals(pParameters.getMinCoresPerNode(), stateFile.getMinCoresPerNode()); assertEquals(pParameters.getRequestedNodeCount(), stateFile.getRequestedNodeCount()); } // Parameters that come from the PipelineTask - private RemoteParameters remoteParametersForPipelineTask() { - - RemoteParameters parameters = new RemoteParameters(); - parameters.setEnabled(true); - parameters.setSubtaskMaxWallTimeHours(5.0); - parameters.setSubtaskTypicalWallTimeHours(1.0); - parameters.setGigsPerSubtask(4.5); - parameters.setQueueName("normal"); - parameters.setRemoteNodeArchitecture("bro"); - parameters.setSubtasksPerCore("5"); - parameters.setMaxNodes("10"); - parameters.setMinGigsPerNode("100"); - parameters.setMinCoresPerNode("50"); - return parameters; + private PipelineDefinitionNodeExecutionResources remoteExecutionConfigurationForPipelineTask() { + + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setRemoteExecutionEnabled(true); + executionResources.setSubtaskMaxWallTimeHours(5.0); + executionResources.setSubtaskTypicalWallTimeHours(1.0); + executionResources.setGigsPerSubtask(4.5); + executionResources.setQueueName("normal"); + executionResources.setRemoteNodeArchitecture("bro"); + executionResources.setSubtasksPerCore(5.0); + executionResources.setMaxNodes(10); + executionResources.setMinGigsPerNode(100.0); + executionResources.setMinCoresPerNode(50); + return executionResources; } // Different parameters that come from the database - private RemoteParameters remoteParametersFromDatabase() { - RemoteParameters parameters = new RemoteParameters(); - parameters.setEnabled(true); - parameters.setSubtaskMaxWallTimeHours(6.0); - parameters.setSubtaskTypicalWallTimeHours(2.0); - parameters.setGigsPerSubtask(8); - parameters.setQueueName("long"); - parameters.setRemoteNodeArchitecture("rom_ait"); - parameters.setSubtasksPerCore("1"); - parameters.setMaxNodes("12"); - parameters.setMinGigsPerNode("101"); - parameters.setMinCoresPerNode("51"); - return parameters; + private PipelineDefinitionNodeExecutionResources remoteExecutionConfigurationFromDatabase() { + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setRemoteExecutionEnabled(true); + executionResources.setSubtaskMaxWallTimeHours(6.0); + executionResources.setSubtaskTypicalWallTimeHours(2.0); + executionResources.setGigsPerSubtask(8); + executionResources.setQueueName("long"); + executionResources.setRemoteNodeArchitecture("rom_ait"); + executionResources.setSubtasksPerCore(1.0); + executionResources.setMaxNodes(12); + executionResources.setMinGigsPerNode(101.0); + executionResources.setMinCoresPerNode(51); + return executionResources; } private static class GenericRemoteExecutor extends RemoteExecutor { @@ -245,8 +250,8 @@ public StateFile stateFile() { } @Override - public PbsParameters generatePbsParameters(RemoteParameters remoteParameters, - int totalSubtaskCount) { + public PbsParameters generatePbsParameters( + PipelineDefinitionNodeExecutionResources remoteParameters, int totalSubtaskCount) { this.totalSubtaskCount = totalSubtaskCount; pbsParameters = remoteParameters.pbsParametersInstance(); return pbsParameters; @@ -262,6 +267,11 @@ public void setParameterSetCrud(ParameterSetCrud parameterSetCrud) { super.setParameterSetCrud(parameterSetCrud); } + @Override + protected PipelineDefinitionNodeCrud pipelineDefinitionNodeCrud() { + return defNodeCrud; + } + @Override public void setProcessingSummaryOperations(ProcessingSummaryOperations crud) { super.setProcessingSummaryOperations(crud); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptorTest.java b/src/test/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptorTest.java index 1a88232..f8ea9b9 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/RemoteNodeDescriptorTest.java @@ -65,10 +65,10 @@ public void testDescriptorsSortedByRamThenCost() { assertEquals(7, descriptors.size()); assertEquals(RemoteNodeDescriptor.SANDY_BRIDGE, descriptors.get(0)); assertEquals(RemoteNodeDescriptor.IVY_BRIDGE, descriptors.get(1)); - assertEquals(RemoteNodeDescriptor.CASCADE_LAKE, descriptors.get(2)); - assertEquals(RemoteNodeDescriptor.ROME, descriptors.get(3)); - assertEquals(RemoteNodeDescriptor.BROADWELL, descriptors.get(4)); - assertEquals(RemoteNodeDescriptor.SKYLAKE, descriptors.get(5)); + assertEquals(RemoteNodeDescriptor.ROME, descriptors.get(2)); + assertEquals(RemoteNodeDescriptor.BROADWELL, descriptors.get(3)); + assertEquals(RemoteNodeDescriptor.SKYLAKE, descriptors.get(4)); + assertEquals(RemoteNodeDescriptor.CASCADE_LAKE, descriptors.get(5)); assertEquals(RemoteNodeDescriptor.HASWELL, descriptors.get(6)); descriptors = RemoteNodeDescriptor @@ -86,10 +86,10 @@ public void testNodesWithSufficientRam() { List acceptableDescriptors = RemoteNodeDescriptor .nodesWithSufficientRam(descriptors, 70.0); assertEquals(5, acceptableDescriptors.size()); - assertEquals(RemoteNodeDescriptor.CASCADE_LAKE, acceptableDescriptors.get(0)); - assertEquals(RemoteNodeDescriptor.ROME, acceptableDescriptors.get(1)); - assertEquals(RemoteNodeDescriptor.BROADWELL, acceptableDescriptors.get(2)); - assertEquals(RemoteNodeDescriptor.SKYLAKE, acceptableDescriptors.get(3)); + assertEquals(RemoteNodeDescriptor.ROME, acceptableDescriptors.get(0)); + assertEquals(RemoteNodeDescriptor.BROADWELL, acceptableDescriptors.get(1)); + assertEquals(RemoteNodeDescriptor.SKYLAKE, acceptableDescriptors.get(2)); + assertEquals(RemoteNodeDescriptor.CASCADE_LAKE, acceptableDescriptors.get(3)); assertEquals(RemoteNodeDescriptor.HASWELL, acceptableDescriptors.get(4)); } @@ -107,7 +107,7 @@ public void testGetMaxGigs() { assertEquals(32, RemoteNodeDescriptor.SANDY_BRIDGE.getMaxGigs()); assertEquals(64, RemoteNodeDescriptor.IVY_BRIDGE.getMaxGigs()); assertEquals(192, RemoteNodeDescriptor.SKYLAKE.getMaxGigs()); - assertEquals(160, RemoteNodeDescriptor.CASCADE_LAKE.getMaxGigs()); + assertEquals(192, RemoteNodeDescriptor.CASCADE_LAKE.getMaxGigs()); assertEquals(512, RemoteNodeDescriptor.ROME.getMaxGigs()); assertEquals(192, RemoteNodeDescriptor.C5.getMaxGigs()); assertEquals(384, RemoteNodeDescriptor.M5.getMaxGigs()); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/aws/AwsExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/remote/aws/AwsExecutorTest.java index 9a75083..8bb67f0 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/aws/AwsExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/aws/AwsExecutorTest.java @@ -3,13 +3,14 @@ import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_GROUP; import static org.junit.Assert.assertEquals; +import org.junit.Ignore; import org.junit.Rule; import org.junit.Test; import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.module.remote.RemoteNodeDescriptor; -import gov.nasa.ziggy.module.remote.RemoteParameters; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** @@ -23,6 +24,11 @@ public class AwsExecutorTest { @Rule public ZiggyPropertyRule groupPropertyRule = new ZiggyPropertyRule(REMOTE_GROUP, "12345"); + // We ignore this test because the AwsExecutor is (probably) obsolete, since it was originally + // written for a proof-of-concept activity and represents an approach to AWS remote execution + // that we actually don't want to use anymore. When we write the new AwsExecutor, or revive the + // current one, we'll either delete and replace this test, or un-ignore it and fix it. + @Ignore @Test public void testGeneratePbsParameters() { AwsExecutor executor = new AwsExecutor(new PipelineTask()); @@ -30,18 +36,19 @@ public void testGeneratePbsParameters() { // Start with a job that needs minimal gigs per core -- the optimization should // get us the C5 architecture, with a memory and cores configuration near the middle // of what's available on that architecture - RemoteParameters remoteParameters = new RemoteParameters(); - remoteParameters.setGigsPerSubtask(1.0); - remoteParameters.setSubtaskMaxWallTimeHours(4.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); - remoteParameters.setEnabled(true); - remoteParameters.setRemoteNodeArchitecture(""); + PipelineDefinitionNodeExecutionResources executionParameters = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionParameters.setGigsPerSubtask(1.0); + executionParameters.setSubtaskMaxWallTimeHours(4.5); + executionParameters.setSubtaskTypicalWallTimeHours(0.5); + executionParameters.setRemoteExecutionEnabled(true); + executionParameters.setRemoteNodeArchitecture(""); - PbsParameters pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + PbsParameters pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.C5, pbsParameters.getArchitecture()); assertEquals(16, pbsParameters.getActiveCoresPerNode()); assertEquals(16, pbsParameters.getMinCoresPerNode()); - assertEquals(64, pbsParameters.getMinGigsPerNode()); + assertEquals(64, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(4, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); @@ -50,12 +57,12 @@ public void testGeneratePbsParameters() { // Now try a job that requires an R5 node and can't use all of its cores due to // memory demands. - remoteParameters.setGigsPerSubtask(32.0); - pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + executionParameters.setGigsPerSubtask(32.0); + pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.R5, pbsParameters.getArchitecture()); assertEquals(8, pbsParameters.getActiveCoresPerNode()); assertEquals(16, pbsParameters.getMinCoresPerNode()); - assertEquals(256, pbsParameters.getMinGigsPerNode()); + assertEquals(256, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(7, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); @@ -64,12 +71,12 @@ public void testGeneratePbsParameters() { // Now a job that requires an R5 node and a number of cores that is greater than the // default (default currently set to max / 3) - remoteParameters.setGigsPerSubtask(384.0); - pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + executionParameters.setGigsPerSubtask(384.0); + pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.R5, pbsParameters.getArchitecture()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); assertEquals(24, pbsParameters.getMinCoresPerNode()); - assertEquals(384, pbsParameters.getMinGigsPerNode()); + assertEquals(384, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(56, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); @@ -77,12 +84,12 @@ public void testGeneratePbsParameters() { assertEquals(2733.696, pbsParameters.getEstimatedCost(), 1e-9); // Now do a test in which the minimum number of cores per node is set by the user - remoteParameters.setMinCoresPerNode("36"); - pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + executionParameters.setMinCoresPerNode(36); + pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.R5, pbsParameters.getArchitecture()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); assertEquals(36, pbsParameters.getMinCoresPerNode()); - assertEquals(384, pbsParameters.getMinGigsPerNode()); + assertEquals(384, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(56, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); @@ -90,13 +97,13 @@ public void testGeneratePbsParameters() { assertEquals(4100.544, pbsParameters.getEstimatedCost(), 1e-9); // Now a test in which the minimum amount of RAM per node is set by the user - remoteParameters.setMinCoresPerNode(""); - remoteParameters.setMinGigsPerNode("400"); - pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + executionParameters.setMinCoresPerNode(0); + executionParameters.setMinGigsPerNode(400.0); + pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.R5, pbsParameters.getArchitecture()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); assertEquals(24, pbsParameters.getMinCoresPerNode()); - assertEquals(400, pbsParameters.getMinGigsPerNode()); + assertEquals(400, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(56, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); @@ -105,12 +112,12 @@ public void testGeneratePbsParameters() { // Now a test in which both the minimum RAM and minimum cores are set // by the user - remoteParameters.setMinCoresPerNode("36"); - pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + executionParameters.setMinCoresPerNode(36); + pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.R5, pbsParameters.getArchitecture()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); assertEquals(36, pbsParameters.getMinCoresPerNode()); - assertEquals(400, pbsParameters.getMinGigsPerNode()); + assertEquals(400, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(56, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); @@ -119,12 +126,12 @@ public void testGeneratePbsParameters() { // Now a test in which the user asks for too little RAM, and the PBS parameter // calculator increases it to be sufficent to run the job. - remoteParameters.setMinGigsPerNode("200"); - pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + executionParameters.setMinGigsPerNode(200.0); + pbsParameters = executor.generatePbsParameters(executionParameters, 500); assertEquals(RemoteNodeDescriptor.R5, pbsParameters.getArchitecture()); assertEquals(1, pbsParameters.getActiveCoresPerNode()); assertEquals(36, pbsParameters.getMinCoresPerNode()); - assertEquals(384, pbsParameters.getMinGigsPerNode()); + assertEquals(384, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(56, pbsParameters.getRequestedNodeCount()); assertEquals("cloud", pbsParameters.getQueueName()); assertEquals("12345", pbsParameters.getRemoteGroup()); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/nas/NasExecutorTest.java b/src/test/java/gov/nasa/ziggy/module/remote/nas/NasExecutorTest.java index 6392eaa..af5e1b0 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/nas/NasExecutorTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/nas/NasExecutorTest.java @@ -10,7 +10,7 @@ import gov.nasa.ziggy.ZiggyPropertyRule; import gov.nasa.ziggy.module.remote.PbsParameters; import gov.nasa.ziggy.module.remote.RemoteNodeDescriptor; -import gov.nasa.ziggy.module.remote.RemoteParameters; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNodeExecutionResources; import gov.nasa.ziggy.pipeline.definition.PipelineTask; /** @@ -27,19 +27,20 @@ public class NasExecutorTest { public void testGeneratePbsParameters() { NasExecutor executor = new NasExecutor(new PipelineTask()); - RemoteParameters remoteParameters = new RemoteParameters(); - remoteParameters.setGigsPerSubtask(6.0); - remoteParameters.setSubtaskMaxWallTimeHours(4.5); - remoteParameters.setSubtaskTypicalWallTimeHours(0.5); - remoteParameters.setEnabled(true); - remoteParameters.setRemoteNodeArchitecture(""); - - PbsParameters pbsParameters = executor.generatePbsParameters(remoteParameters, 500); + PipelineDefinitionNodeExecutionResources executionResources = new PipelineDefinitionNodeExecutionResources( + "dummy", "dummy"); + executionResources.setGigsPerSubtask(6.0); + executionResources.setSubtaskMaxWallTimeHours(4.5); + executionResources.setSubtaskTypicalWallTimeHours(0.5); + executionResources.setRemoteExecutionEnabled(true); + executionResources.setRemoteNodeArchitecture(""); + + PbsParameters pbsParameters = executor.generatePbsParameters(executionResources, 500); assertTrue(pbsParameters.isEnabled()); assertEquals(RemoteNodeDescriptor.HASWELL, pbsParameters.getArchitecture()); assertEquals(pbsParameters.getArchitecture().getMaxCores(), pbsParameters.getMinCoresPerNode()); - assertEquals(128, pbsParameters.getMinGigsPerNode()); + assertEquals(128, pbsParameters.getMinGigsPerNode(), 1e-3); assertEquals(21, pbsParameters.getActiveCoresPerNode()); assertEquals("4:30:00", pbsParameters.getRequestedWallTime()); assertEquals("normal", pbsParameters.getQueueName()); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetricsTest.java b/src/test/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetricsTest.java index 53b784d..20a1622 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetricsTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/nas/NasQueueTimeMetricsTest.java @@ -45,7 +45,7 @@ public void setup() { @Test public void testArmdMetrics() { - System.setProperty(REMOTE_NASA_DIRECTORATE.property(), "ARMD"); + nasaDirectoratePropertyRule.setValue("ARMD"); instance.populate(QS_MOCK_OUTPUT_FILE); testValues(RemoteNodeDescriptor.SANDY_BRIDGE, 53.8, 1.0); testValues(RemoteNodeDescriptor.IVY_BRIDGE, 197.5, 122.4); @@ -58,7 +58,7 @@ public void testArmdMetrics() { @Test public void testHeomdMetrics() { - System.setProperty(REMOTE_NASA_DIRECTORATE.property(), "HEOMD"); + nasaDirectoratePropertyRule.setValue("HEOMD"); instance.populate(QS_MOCK_OUTPUT_FILE); testValues(RemoteNodeDescriptor.SANDY_BRIDGE, 278.4, 2.0); testValues(RemoteNodeDescriptor.IVY_BRIDGE, 218.9, 2.6); @@ -71,7 +71,7 @@ public void testHeomdMetrics() { @Test public void testSmdMetrics() { - System.setProperty(REMOTE_NASA_DIRECTORATE.property(), "SMD"); + nasaDirectoratePropertyRule.setValue("SMD"); instance.populate(QS_MOCK_OUTPUT_FILE); testValues(RemoteNodeDescriptor.SANDY_BRIDGE, 175.7, 15.6); testValues(RemoteNodeDescriptor.IVY_BRIDGE, 199.8, 31.7); diff --git a/src/test/java/gov/nasa/ziggy/module/remote/nas/RemoteExecutionPropertiesTest.java b/src/test/java/gov/nasa/ziggy/module/remote/nas/RemoteExecutionPropertiesTest.java index 3ca80d5..140229e 100644 --- a/src/test/java/gov/nasa/ziggy/module/remote/nas/RemoteExecutionPropertiesTest.java +++ b/src/test/java/gov/nasa/ziggy/module/remote/nas/RemoteExecutionPropertiesTest.java @@ -3,10 +3,10 @@ import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_GROUP; import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_HOST; import static gov.nasa.ziggy.services.config.PropertyName.REMOTE_USER; -import static gov.nasa.ziggy.services.config.PropertyName.TEST_ENVIRONMENT; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import org.apache.commons.configuration2.CompositeConfiguration; import org.junit.Rule; import org.junit.Test; @@ -30,9 +30,6 @@ public class RemoteExecutionPropertiesTest { @Rule public ZiggyPropertyRule userPropertyRule = new ZiggyPropertyRule(REMOTE_USER, "u1"); - @Rule - public ZiggyPropertyRule testEnvRule = new ZiggyPropertyRule(TEST_ENVIRONMENT, "true"); - @Test public void testPropertiesRetrieval() { @@ -46,11 +43,10 @@ public void testPropertiesRetrieval() { @Test public void testEmptyPropertiesRetrieval() { - // This clears properties set by rules, and ensures that ZiggyConfiguration doesn't read the // user's property file. ZiggyConfiguration.reset(); - ZiggyConfiguration.getMutableInstance(); + ZiggyConfiguration.setMutableInstance(new CompositeConfiguration()); assertTrue(RemoteExecutionProperties.getUser().isEmpty()); assertTrue(RemoteExecutionProperties.getGroup().isEmpty()); diff --git a/src/test/java/gov/nasa/ziggy/parameters/ParameterSetDescriptorTest.java b/src/test/java/gov/nasa/ziggy/parameters/ParameterSetDescriptorTest.java index ad096b7..7cdd1e2 100644 --- a/src/test/java/gov/nasa/ziggy/parameters/ParameterSetDescriptorTest.java +++ b/src/test/java/gov/nasa/ziggy/parameters/ParameterSetDescriptorTest.java @@ -1,11 +1,11 @@ package gov.nasa.ziggy.parameters; +import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; -import java.nio.file.Paths; import java.util.HashSet; import java.util.Map; import java.util.Set; @@ -17,7 +17,6 @@ import gov.nasa.ziggy.collections.ZiggyDataType; import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.TypedParameter; -import gov.nasa.ziggy.services.config.PropertyName; /** * Unit test class for {@link ParameterSetDescriptor} @@ -27,9 +26,8 @@ public class ParameterSetDescriptorTest { @Rule - public ZiggyPropertyRule schemaRule = new ZiggyPropertyRule( - PropertyName.ZIGGY_HOME_DIR.property(), - Paths.get(System.getProperty(PropertyName.WORKING_DIR.property()), "build").toString()); + public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule(ZIGGY_HOME_DIR, + "build"); @Test(expected = NumberFormatException.class) public void testValidationFailureDataTypes() { diff --git a/src/test/java/gov/nasa/ziggy/parameters/ParametersOperationsTest.java b/src/test/java/gov/nasa/ziggy/parameters/ParametersOperationsTest.java index 8d7ed0c..68d0144 100644 --- a/src/test/java/gov/nasa/ziggy/parameters/ParametersOperationsTest.java +++ b/src/test/java/gov/nasa/ziggy/parameters/ParametersOperationsTest.java @@ -347,7 +347,7 @@ public void testImportFromFile() throws Exception { ParametersOperations ops = new ParametersOperations(); List paramsDescriptors = ops.importParameterLibrary( TEST_DATA.resolve("paramlib").resolve("test.xml").toString(), null, ParamIoMode.NODB); - assertEquals(4, paramsDescriptors.size()); + assertEquals(3, paramsDescriptors.size()); for (ParameterSetDescriptor descriptor : paramsDescriptors) { assertEquals(ParameterSetDescriptor.State.CREATE, descriptor.getState()); } @@ -356,47 +356,13 @@ public void testImportFromFile() throws Exception { Map nameToParameterSetDescriptor = nameToParameterSetDescriptor( paramsDescriptors); - // Start with the RemoteParameters instance. - ParameterSetDescriptor descriptor = nameToParameterSetDescriptor.get("Remote Hyperion L1"); - assertEquals("gov.nasa.ziggy.module.remote.RemoteParameters", descriptor.getClassName()); - Set typedProperties = descriptor.getImportedProperties(); - assertEquals(14, typedProperties.size()); - Map nameToTypedProperty = nameToTypedPropertyMap(typedProperties); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "enabled", "false", - ZiggyDataType.ZIGGY_BOOLEAN, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "gigsPerSubtask", "0.1", - ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtaskMaxWallTimeHours", "2.1", - ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtaskTypicalWallTimeHours", - "2.1", ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "optimizer", "COST", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "minSubtasksForRemoteExecution", - "0", ZiggyDataType.ZIGGY_INT, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "remoteNodeArchitecture", "", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "queueName", "", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtasksPerCore", "", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "maxNodes", "", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "minGigsPerNode", "", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "minCoresPerNode", "", - ZiggyDataType.ZIGGY_STRING, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "nodeSharing", "true", - ZiggyDataType.ZIGGY_BOOLEAN, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "wallTimeScaling", "true", - ZiggyDataType.ZIGGY_BOOLEAN, true)); - // sample classless parameter set - descriptor = nameToParameterSetDescriptor.get("Sample classless parameter set"); + ParameterSetDescriptor descriptor = nameToParameterSetDescriptor + .get("Sample classless parameter set"); assertEquals("gov.nasa.ziggy.parameters.Parameters", descriptor.getClassName()); - typedProperties = descriptor.getImportedProperties(); + Set typedProperties = descriptor.getImportedProperties(); assertEquals(3, typedProperties.size()); - nameToTypedProperty = nameToTypedPropertyMap(typedProperties); + Map nameToTypedProperty = nameToTypedPropertyMap(typedProperties); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "z1", "100", ZiggyDataType.ZIGGY_INT, true)); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "z2", "28.56,57.12", @@ -460,40 +426,25 @@ public void testImportOverrideFromFile() { // Check the descriptor states Map nameToDescriptor = nameToParameterSetDescriptor( descriptors); - assertEquals(ParameterSetDescriptor.State.UPDATE, - nameToDescriptor.get("Remote Hyperion L1").getState()); assertEquals(ParameterSetDescriptor.State.SAME, nameToDescriptor.get("Sample classless parameter set").getState()); - assertEquals(ParameterSetDescriptor.State.LIBRARY_ONLY, + assertEquals(ParameterSetDescriptor.State.UPDATE, nameToDescriptor.get("ISOFIT module parameters").getState()); + assertEquals(ParameterSetDescriptor.State.LIBRARY_ONLY, + nameToDescriptor.get("Multiple subtask configuration").getState()); // Retrieve the parameter sets from the database and check their values ParameterSetCrud paramCrud = new ParameterSetCrud(); List parameterSets = paramCrud.retrieveLatestVersions(); - assertEquals(4, parameterSets.size()); + assertEquals(3, parameterSets.size()); Map nameToParameterSet = nameToParameterSet(parameterSets); - // The Hyperion L1 dataset has its gigs per subtask value changed - Set typedProperties = nameToParameterSet.get("Remote Hyperion L1") - .getTypedParameters(); - assertEquals(14, typedProperties.size()); - Map nameToTypedProperty = nameToTypedPropertyMap(typedProperties); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "enabled", "false", - ZiggyDataType.ZIGGY_BOOLEAN, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "gigsPerSubtask", "1.0", - ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtaskMaxWallTimeHours", "2.1", - ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtaskTypicalWallTimeHours", - "2.1", ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "optimizer", "COST", - ZiggyDataType.ZIGGY_STRING, true)); - // The sample classless parameter set no changed parameters - typedProperties = nameToParameterSet.get("Sample classless parameter set") + Set typedProperties = nameToParameterSet + .get("Sample classless parameter set") .getTypedParameters(); assertEquals(3, typedProperties.size()); - nameToTypedProperty = nameToTypedPropertyMap(typedProperties); + Map nameToTypedProperty = nameToTypedPropertyMap(typedProperties); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "z1", "100", ZiggyDataType.ZIGGY_INT, true)); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "z2", "28.56,57.12", @@ -501,16 +452,32 @@ public void testImportOverrideFromFile() { assertTrue(checkTypedPropertyValues(nameToTypedProperty, "z3", "some text", ZiggyDataType.ZIGGY_STRING, true)); - // ISOFIT classless parameter set has no changes + // ISOFIT classless parameter set has one changed parameter typedProperties = nameToParameterSet.get("ISOFIT module parameters").getTypedParameters(); assertEquals(3, typedProperties.size()); nameToTypedProperty = nameToTypedPropertyMap(typedProperties); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "n_cores", "4", + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "n_cores", "8", ZiggyDataType.ZIGGY_INT, true)); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "presolve", "true", ZiggyDataType.ZIGGY_BOOLEAN, true)); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "empirical_line", "true", ZiggyDataType.ZIGGY_BOOLEAN, true)); + + // The TaskConfigurationParameters set has no changes. + typedProperties = nameToParameterSet.get("Multiple subtask configuration") + .getTypedParameters(); + assertEquals(5, typedProperties.size()); + nameToTypedProperty = nameToTypedPropertyMap(typedProperties); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "taskDirectoryRegex", + "set-([0-9]{1})", ZiggyDataType.ZIGGY_STRING, true)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "reprocessingTasksExclude", "0", + ZiggyDataType.ZIGGY_INT, false)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "singleSubtask", "false", + ZiggyDataType.ZIGGY_BOOLEAN, true)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "maxFailedSubtaskCount", "0", + ZiggyDataType.ZIGGY_INT, true)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "reprocess", "true", + ZiggyDataType.ZIGGY_BOOLEAN, true)); } @Test @@ -534,47 +501,32 @@ public void importReplacementParameterSetsFromFile() { .toString(), null, ParamIoMode.STANDARD); }); - assertEquals(5, descriptors.size()); + assertEquals(4, descriptors.size()); // Check the descriptor states Map nameToDescriptor = nameToParameterSetDescriptor( descriptors); - assertEquals(ParameterSetDescriptor.State.UPDATE, - nameToDescriptor.get("Remote Hyperion L1").getState()); assertEquals(ParameterSetDescriptor.State.UPDATE, nameToDescriptor.get("Sample classless parameter set").getState()); assertEquals(ParameterSetDescriptor.State.SAME, nameToDescriptor.get("ISOFIT module parameters").getState()); assertEquals(ParameterSetDescriptor.State.CREATE, nameToDescriptor.get("All-new parameters").getState()); + assertEquals(ParameterSetDescriptor.State.LIBRARY_ONLY, + nameToDescriptor.get("Multiple subtask configuration").getState()); // Retrieve the parameter sets from the database and check their values ParameterSetCrud paramCrud = new ParameterSetCrud(); List parameterSets = paramCrud.retrieveLatestVersions(); - assertEquals(5, parameterSets.size()); + assertEquals(4, parameterSets.size()); Map nameToParameterSet = nameToParameterSet(parameterSets); - // The Hyperion L1 dataset has only its gigsPerSubtask set to a non-default value - Set typedProperties = nameToParameterSet.get("Remote Hyperion L1") - .getTypedParameters(); - assertEquals(14, typedProperties.size()); - Map nameToTypedProperty = nameToTypedPropertyMap(typedProperties); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "enabled", "false", - ZiggyDataType.ZIGGY_BOOLEAN, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "gigsPerSubtask", "3.0", - ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtaskMaxWallTimeHours", "0.0", - ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "subtaskTypicalWallTimeHours", - "0.0", ZiggyDataType.ZIGGY_DOUBLE, true)); - assertTrue(checkTypedPropertyValues(nameToTypedProperty, "optimizer", "CORES", - ZiggyDataType.ZIGGY_STRING, true)); - // The sample classless parameter set now has only 1 parameter - typedProperties = nameToParameterSet.get("Sample classless parameter set") + Set typedProperties = nameToParameterSet + .get("Sample classless parameter set") .getTypedParameters(); assertEquals(1, typedProperties.size()); - nameToTypedProperty = nameToTypedPropertyMap(typedProperties); + Map nameToTypedProperty = nameToTypedPropertyMap(typedProperties); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "z1", "100", ZiggyDataType.ZIGGY_STRING, true)); @@ -595,6 +547,23 @@ public void importReplacementParameterSetsFromFile() { nameToTypedProperty = nameToTypedPropertyMap(typedProperties); assertTrue(checkTypedPropertyValues(nameToTypedProperty, "parameter", "4", ZiggyDataType.ZIGGY_INT, true)); + + // The TaskConfigurationParameters set has no changes. + // The TaskConfigurationParameters set has no changes. + typedProperties = nameToParameterSet.get("Multiple subtask configuration") + .getTypedParameters(); + assertEquals(5, typedProperties.size()); + nameToTypedProperty = nameToTypedPropertyMap(typedProperties); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "taskDirectoryRegex", + "set-([0-9]{1})", ZiggyDataType.ZIGGY_STRING, true)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "reprocessingTasksExclude", "0", + ZiggyDataType.ZIGGY_INT, false)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "singleSubtask", "false", + ZiggyDataType.ZIGGY_BOOLEAN, true)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "maxFailedSubtaskCount", "0", + ZiggyDataType.ZIGGY_INT, true)); + assertTrue(checkTypedPropertyValues(nameToTypedProperty, "reprocess", "true", + ZiggyDataType.ZIGGY_BOOLEAN, true)); } // And now for a bunch of tests that exercise all the error cases diff --git a/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java b/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java index 85cbb06..7c95c0e 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/PipelineTaskInformationTest.java @@ -2,7 +2,6 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.doReturn; @@ -19,10 +18,9 @@ import org.junit.Test; import org.mockito.ArgumentMatchers; -import gov.nasa.ziggy.module.DefaultPipelineInputs; +import gov.nasa.ziggy.module.DatastoreDirectoryPipelineInputs; import gov.nasa.ziggy.module.PipelineInputs; import gov.nasa.ziggy.module.SubtaskInformation; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.parameters.Parameters; import gov.nasa.ziggy.parameters.ParametersInterface; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; @@ -56,7 +54,6 @@ public class PipelineTaskInformationTest { private ParameterSet instanceParSet1 = new ParameterSet(instancePars1Name); private ParameterSet instanceParSet2 = new ParameterSet(instancePars2Name); private String moduleParsName = "Module Pars"; - private RemoteParameters remoteParams = new RemoteParameters(); private ParameterSet moduleParSet = new ParameterSet(moduleParsName); private PipelineTask p1 = new PipelineTask(); private PipelineTask p2 = new PipelineTask(); @@ -77,8 +74,9 @@ public void setup() { // Construct the instances of pipeline infrastructure needed for these tests node = new PipelineDefinitionNode(); - PipelineModuleDefinition moduleDefinition = new PipelineModuleDefinition(null, "module"); - ClassWrapper inputsClass = new ClassWrapper<>(DefaultPipelineInputs.class); + PipelineModuleDefinition moduleDefinition = new PipelineModuleDefinition("module"); + ClassWrapper inputsClass = new ClassWrapper<>( + DatastoreDirectoryPipelineInputs.class); moduleDefinition.setInputsClass(inputsClass); node.setModuleName(moduleDefinition.getName()); pipelineDefinition = new PipelineDefinition("pipeline"); @@ -102,9 +100,7 @@ public void setup() { when(pipelineModuleDefinitionCrud.retrieveLatestVersionForName(moduleDefinition.getName())) .thenReturn(moduleDefinition); Map, String> moduleParameterNames = new HashMap<>(); - moduleParameterNames.put(new ClassWrapper<>(RemoteParameters.class), moduleParsName); node.setModuleParameterSetNames(moduleParameterNames); - moduleParSet.setTypedParameters(remoteParams.getParameters()); when(parameterSetCrud.retrieveLatestVersionForName(moduleParsName)) .thenReturn(moduleParSet); @@ -117,8 +113,8 @@ public void setup() { uowList.add(u2); doReturn(uowList).when(pipelineTaskInformation) .unitsOfWork(ArgumentMatchers.> any(), - ArgumentMatchers - ., ParametersInterface>> any()); + ArgumentMatchers. any(), + ArgumentMatchers. any()); // Set up pipeline task generation p1.setId(1L); @@ -129,8 +125,8 @@ public void setup() { any(UnitOfWork.class)); // Set up SubtaskInformation returns - s1 = new SubtaskInformation("module", "u1", 3, 3); - s2 = new SubtaskInformation("module", "u2", 5, 2); + s1 = new SubtaskInformation("module", "u1", 3); + s2 = new SubtaskInformation("module", "u2", 5); doReturn(s1).when(pipelineTaskInformation).subtaskInformation(moduleDefinition, p1); doReturn(s2).when(pipelineTaskInformation).subtaskInformation(moduleDefinition, p2); } @@ -148,31 +144,11 @@ public void testBasicFunctionality() { assertTrue(subtaskInfo.contains(s1)); assertTrue(subtaskInfo.contains(s2)); - String psn = PipelineTaskInformation.remoteParameters(node); - assertEquals(moduleParsName, psn); - // Resetting it should cause it to disappear again PipelineTaskInformation.reset(node); assertFalse(PipelineTaskInformation.hasPipelineDefinitionNode(node)); } - @Test - public void testRemoteParamsAtInstanceLevel() { - - pipelineDefinition.getPipelineParameterSetNames() - .put(new ClassWrapper<>(RemoteParameters.class), moduleParsName); - node.getModuleParameterSetNames().clear(); - String psn = PipelineTaskInformation.remoteParameters(node); - assertEquals(moduleParsName, psn); - } - - @Test - public void testNoRemoteParams() { - node.getModuleParameterSetNames().clear(); - String psn = PipelineTaskInformation.remoteParameters(node); - assertNull(psn); - } - public static class InstancePars1 extends Parameters { private int intParam; diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/ClassWrapperTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/ClassWrapperTest.java index 4d5dea2..2fb6352 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/ClassWrapperTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/ClassWrapperTest.java @@ -4,7 +4,6 @@ import org.junit.Test; -import gov.nasa.ziggy.module.remote.RemoteParameters; import gov.nasa.ziggy.parameters.Parameters; import gov.nasa.ziggy.parameters.ParametersInterface; @@ -69,14 +68,15 @@ public void testParameterSetConstructor() { @Test public void testParameterSetSubclassConstructor() { ParameterSet paramSet = new ParameterSet(); - RemoteParameters params = new RemoteParameters(); + TestParameters params = new TestParameters(); paramSet.populateFromParametersInstance(params); paramSet.setName("foo"); ClassWrapper paramWrapper = new ClassWrapper<>(paramSet); - assertEquals("gov.nasa.ziggy.module.remote.RemoteParameters", paramWrapper.getClassName()); - assertEquals("gov.nasa.ziggy.module.remote.RemoteParameters", + assertEquals("gov.nasa.ziggy.pipeline.definition.ClassWrapperTest$TestParameters", + paramWrapper.getClassName()); + assertEquals("gov.nasa.ziggy.pipeline.definition.ClassWrapperTest$TestParameters", paramWrapper.unmangledClassName()); - assertEquals(RemoteParameters.class, paramWrapper.getClazz()); + assertEquals(TestParameters.class, paramWrapper.getClazz()); } public static class Foo { @@ -84,4 +84,8 @@ public static class Foo { public static class Bar extends Foo { } + + public static class TestParameters extends Parameters { + + } } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/DatastoreProducerConsumerTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/DatastoreProducerConsumerTest.java index 3463d97..f41732b 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/DatastoreProducerConsumerTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/DatastoreProducerConsumerTest.java @@ -25,8 +25,7 @@ public void testResultsOriginatorMethods() { PipelineTask p = Mockito.mock(PipelineTask.class); Mockito.when(p.getId()).thenReturn(TASK_ID); - DatastoreProducerConsumer r = new DatastoreProducerConsumer(p, Paths.get(FILE_SPEC), - DatastoreProducerConsumer.DataReceiptFileType.DATA); + DatastoreProducerConsumer r = new DatastoreProducerConsumer(p, Paths.get(FILE_SPEC)); assertEquals(FILE_SPEC, r.getFilename()); assertEquals(TASK_ID, r.getProducer()); } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/FakePipelineTaskFactory.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/FakePipelineTaskFactory.java index f346dd6..512ed6a 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/FakePipelineTaskFactory.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/FakePipelineTaskFactory.java @@ -1,7 +1,5 @@ package gov.nasa.ziggy.pipeline.definition; -import java.util.Date; - import gov.nasa.ziggy.crud.SimpleCrud; import gov.nasa.ziggy.pipeline.definition.crud.ParameterSetCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineDefinitionCrud; @@ -10,8 +8,6 @@ import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.services.security.UserCrud; import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; import gov.nasa.ziggy.uow.UnitOfWork; @@ -27,7 +23,6 @@ public PipelineTask newTask() { } public PipelineTask newTask(boolean inDb) { - UserCrud userCrud = new UserCrud(); PipelineDefinitionCrud pipelineDefinitionCrud = new PipelineDefinitionCrud(); @@ -40,15 +35,8 @@ public PipelineTask newTask(boolean inDb) { return (PipelineTask) DatabaseTransactionFactory.performTransaction(() -> { - // create users - User testUser = new User("unit-test", "Unit-Test", "unit-test@example.com", "x111"); - if (inDb) { - userCrud.createUser(testUser); - } - // create a module param set def - ParameterSet parameterSet = new ParameterSet(new AuditInfo(testUser, new Date()), - "test mps1"); + ParameterSet parameterSet = new ParameterSet("test mps1"); parameterSet.setTypedParameters(new TestModuleParameters().getParameters()); if (inDb) { parameterSet = parameterSetCrud.merge(parameterSet); @@ -56,8 +44,7 @@ public PipelineTask newTask(boolean inDb) { // create a module def PipelineModuleDefinition moduleDef = new PipelineModuleDefinition("Test-1"); - PipelineDefinition pipelineDef = new PipelineDefinition( - new AuditInfo(testUser, new Date()), "test pipeline name"); + PipelineDefinition pipelineDef = new PipelineDefinition("test pipeline name"); if (inDb) { moduleDef = pipelineModuleDefinitionCrud.merge(moduleDef); pipelineDef = pipelineDefinitionCrud.merge(pipelineDef); @@ -66,8 +53,7 @@ public PipelineTask newTask(boolean inDb) { // create some pipeline def nodes PipelineDefinitionNode pipelineDefNode1 = new PipelineDefinitionNode( moduleDef.getName(), pipelineDef.getName()); - pipelineDefNode1 - .setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); + moduleDef.setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); pipelineDefNode1 = new SimpleCrud<>().merge(pipelineDefNode1); pipelineDef.getRootNodes().add(pipelineDefNode1); if (inDb) { diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionFileTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionFileTest.java index 7da3322..5024031 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionFileTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionFileTest.java @@ -79,7 +79,7 @@ public void testGenerateSchema() throws JAXBException, IOException { assertContains(complexTypeContent, ""); assertContains(complexTypeContent, - ""); + ""); complexTypeContent = complexTypeContent(schemaContent, ""); @@ -106,8 +106,6 @@ public void testGenerateSchema() throws JAXBException, IOException { ""); assertContains(complexTypeContent, ""); - assertContains(complexTypeContent, - ""); assertContains(complexTypeContent, ""); assertContains(complexTypeContent, diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java index b4ea54e..a8e7699 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionNodeTest.java @@ -27,7 +27,6 @@ import gov.nasa.ziggy.pipeline.xml.XmlReference.ModelTypeReference; import gov.nasa.ziggy.pipeline.xml.XmlReference.OutputTypeReference; import gov.nasa.ziggy.pipeline.xml.XmlReference.ParameterSetReference; -import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; import gov.nasa.ziggy.util.io.FileUtil; import jakarta.xml.bind.JAXBContext; import jakarta.xml.bind.JAXBException; @@ -64,12 +63,14 @@ public void setUp() { // Construct a new node for the test node = new Node("module 1", null); - node.setUnitOfWorkGenerator(new ClassWrapper<>(SingleUnitOfWorkGenerator.class)); node.setChildNodeNames("module 2, module 3"); + node.setHeapSizeMb(2); + node.populateXmlFields(); Set xmlReferences = new HashSet<>(); xmlReferences.add(new ParameterSetReference("Remote execution")); xmlReferences.add(new ParameterSetReference("Convergence criteria")); xmlReferences.add(new InputTypeReference("flight L0 data")); + xmlReferences.add(new InputTypeReference("target pixel data")); xmlReferences.add(new OutputTypeReference("flight L1 data")); xmlReferences.add(new ModelTypeReference("calibration constants")); node.setXmlReferences(xmlReferences); @@ -83,15 +84,15 @@ public void testMarshaller() throws JAXBException, IOException { marshaller.marshal(node, xmlFile); assertTrue(xmlFile.exists()); List xmlContent = Files.readAllLines(xmlFile.toPath(), FileUtil.ZIGGY_CHARSET); - assertEquals(8, xmlContent.size()); + assertEquals(9, xmlContent.size()); List nodeContent = nodeContent(xmlContent, - ""); + ""); String[] xmlLines = { "", "", "", "", + "", "" }; for (String xmlLine : xmlLines) { assertContains(nodeContent, xmlLine); @@ -108,9 +109,11 @@ public void testUnmarshaller() throws JAXBException { assertEquals(2, node.getParameterSetNames().size()); assertTrue(node.getParameterSetNames().contains("Remote execution")); assertTrue(node.getParameterSetNames().contains("Convergence criteria")); - assertEquals(1, node.getInputDataFileTypeReferences().size()); + assertEquals(2, node.getInputDataFileTypeReferences().size()); assertTrue(node.getInputDataFileTypeReferences() .contains(new InputTypeReference("flight L0 data"))); + assertTrue(node.getInputDataFileTypeReferences() + .contains(new InputTypeReference("target pixel table"))); assertEquals(1, node.getOutputDataFileTypeReferences().size()); assertTrue(node.getOutputDataFileTypeReferences() .contains(new OutputTypeReference("flight L1 data"))); @@ -138,9 +141,10 @@ public void testGenerateSchema() throws JAXBException, IOException { "", "", "", - "", "", + "", "", + "", "" }; for (String nodeString : nodeStrings) { assertContains(nodeContent, nodeString); diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionTest.java index 26f0b5e..732d2cc 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineDefinitionTest.java @@ -31,7 +31,6 @@ import gov.nasa.ziggy.pipeline.xml.XmlReference.ModelTypeReference; import gov.nasa.ziggy.pipeline.xml.XmlReference.OutputTypeReference; import gov.nasa.ziggy.pipeline.xml.XmlReference.ParameterSetReference; -import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; import gov.nasa.ziggy.util.io.FileUtil; import jakarta.xml.bind.JAXBContext; import jakarta.xml.bind.JAXBException; @@ -69,10 +68,10 @@ public void setUp() { // Create some nodes PipelineDefinitionNode node1 = new PipelineDefinitionNode("module 1", null); - node1.setUnitOfWorkGenerator(new ClassWrapper<>(SingleUnitOfWorkGenerator.class)); node1.addXmlReference(new ParameterSetReference("Remote execution")); node1.addXmlReference(new ParameterSetReference("Convergence criteria")); node1.addXmlReference(new InputTypeReference("flight L0 data")); + node1.addXmlReference(new InputTypeReference("target pixel table")); node1.addXmlReference(new OutputTypeReference("flight L1 data")); node1.addXmlReference(new ModelTypeReference("calibration constants")); @@ -121,29 +120,28 @@ public void testMarshaller() throws JAXBException, IOException { ""); assertContains(pipelineContents, ""); - List nodeContents = nodeContent(pipelineContents, - ""); + List nodeContents = nodeContent(pipelineContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); + assertContains(nodeContents, ""); assertContains(nodeContents, ""); - nodeContents = nodeContent(pipelineContents, ""); + nodeContents = nodeContent(pipelineContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); nodeContents = nodeContent(pipelineContents, - ""); + ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); nodeContents = nodeContent(pipelineContents, - ""); + ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); assertContains(nodeContents, ""); @@ -183,7 +181,6 @@ private void checkNode(PipelineDefinitionNode node, } } assertFalse(groundTruthNode == null); - assertEquals(groundTruthNode.getUnitOfWorkGenerator(), node.getUnitOfWorkGenerator()); assertEquals(groundTruthNode.getChildNodeNames(), node.getChildNodeNames()); compareXmlReferences(groundTruthNode.getModelTypeReferences(), node.getModelTypeReferences()); @@ -228,8 +225,6 @@ public void testGenerateSchema() throws IOException, JAXBException { ""); assertContains(complexTypeContent, ""); - assertContains(complexTypeContent, - ""); assertContains(complexTypeContent, ""); assertContains(complexTypeContent, diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java index 5dcfb5b..58e51f8 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/PipelineModuleDefinitionTest.java @@ -21,7 +21,7 @@ import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.data.management.DataFileTestUtils; -import gov.nasa.ziggy.module.ExternalProcessPipelineModuleTest; +import gov.nasa.ziggy.module.ExternalProcessPipelineModule; import gov.nasa.ziggy.util.io.FileUtil; import jakarta.xml.bind.JAXBContext; import jakarta.xml.bind.JAXBException; @@ -57,35 +57,32 @@ public void setUp() { xmlUnmarshalingFile1 = xmlUnmarshalingPath.resolve("module1.xml").toFile(); xmlUnmarshalingFile2 = xmlUnmarshalingPath.resolve("module2.xml").toFile(); xmlFile = directoryRule.directory().resolve("modules.xml").toFile(); - schemaFile = directoryRule.directory().resolve("modules.xml").toFile(); + schemaFile = directoryRule.directory().resolve("modules.xsd").toFile(); // Module 1 uses defaults for everything possible module1 = new PipelineMod("module 1"); module1.setDescription("first module"); module1.setExeTimeoutSecs(100); - module1.setMinMemoryMegaBytes(200); module1XmlString = """ """; + inputsClass="gov.nasa.ziggy.module.DatastoreDirectoryPipelineInputs" \ + outputsClass="gov.nasa.ziggy.module.DatastoreDirectoryPipelineOutputs" \ + exeTimeoutSecs="100" minMemoryMegabytes="0"/>"""; // Module 2 uses no defaults module2 = new PipelineMod("module 2"); module2.setDescription("second module"); module2.setExeTimeoutSecs(300); - module2.setMinMemoryMegaBytes(400); module2.setInputsClass(new ClassWrapper<>(DataFileTestUtils.PipelineInputsSample.class)); module2.setOutputsClass(new ClassWrapper<>(DataFileTestUtils.PipelineOutputsSample1.class)); - module2.setPipelineModuleClass( - new ClassWrapper<>(ExternalProcessPipelineModuleTest.TestPipelineModule.class)); + module2.setPipelineModuleClass(new ClassWrapper<>(ExternalProcessPipelineModule.class)); module2XmlString = """ """; + exeTimeoutSecs="300" minMemoryMegabytes="0"/>"""; } @Test @@ -134,7 +131,7 @@ public void testGenerateSchema() throws JAXBException, IOException { "", "", "", - "" }; + "" }; for (String moduleAttribute : moduleDefAttributes) { assertContains(moduleDefContent, moduleAttribute); } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrudTest.java index 604b1b7..1f06a64 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineDefinitionCrudTest.java @@ -4,7 +4,6 @@ import static org.junit.Assert.assertNotEquals; import java.lang.reflect.Field; -import java.util.Date; import java.util.List; import org.junit.Before; @@ -17,8 +16,6 @@ import gov.nasa.ziggy.crud.SimpleCrud; import gov.nasa.ziggy.crud.ZiggyQuery; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.AuditInfo; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; @@ -26,9 +23,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.TestModuleParameters; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.services.security.UserCrud; -import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; /** * Tests for {@link PipelineDefinitionCrud} Tests that objects can be stored, retrieved, and edited @@ -40,11 +34,6 @@ public class PipelineDefinitionCrudTest { private static final String TEST_PIPELINE_NAME_1 = "Test Pipeline 1"; - private UserCrud userCrud; - - private User adminUser; - private User operatorUser; - private PipelineDefinitionCrud pipelineDefinitionCrud; private PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud; @@ -61,7 +50,6 @@ public class PipelineDefinitionCrudTest { @Before public void setUp() { - userCrud = new UserCrud(); pipelineDefinitionCrud = new PipelineDefinitionCrud(); pipelineModuleDefinitionCrud = new PipelineModuleDefinitionCrud(); parameterSetCrud = new ParameterSetCrud(); @@ -78,15 +66,8 @@ private PipelineDefinition populateObjects() { return (PipelineDefinition) DatabaseTransactionFactory.performTransaction(() -> { - // create users - adminUser = new User("admin", "Administrator", "admin@example.com", "x111"); - userCrud.createUser(adminUser); - - operatorUser = new User("ops", "Operator", "ops@example.com", "x112"); - userCrud.createUser(operatorUser); - // create a module param set def - expectedParamSet = new ParameterSet(new AuditInfo(adminUser, new Date()), "test mps1"); + expectedParamSet = new ParameterSet("test mps1"); expectedParamSet.setTypedParameters(new TestModuleParameters().getParameters()); expectedParamSet = parameterSetCrud.merge(expectedParamSet); @@ -107,18 +88,13 @@ private PipelineDefinition populateObjects() { } private PipelineDefinition createPipelineDefinition() { - PipelineDefinition pipelineDef = new PipelineDefinition( - new AuditInfo(adminUser, new Date()), TEST_PIPELINE_NAME_1); + PipelineDefinition pipelineDef = new PipelineDefinition(TEST_PIPELINE_NAME_1); PipelineDefinitionNode pipelineNode1 = new PipelineDefinitionNode( expectedModuleDef1.getName(), pipelineDef.getName()); PipelineDefinitionNode pipelineNode2 = new PipelineDefinitionNode( expectedModuleDef2.getName(), pipelineDef.getName()); pipelineNode1.getNextNodes().add(pipelineNode2); - pipelineNode1.setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); - - pipelineNode2.setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); - pipelineDef.addRootNode(pipelineNode1); return pipelineDef; @@ -191,8 +167,6 @@ private PipelineDefinition copyOf(PipelineDefinition original) throws NoSuchFiel PipelineDefinition copy = new PipelineDefinition(); copy.setName(original.getName()); copy.setDescription(original.getDescription()); - copy.setAuditInfo(original.getAuditInfo()); - copy.setGroup(original.group()); setOptimisticLockValue(copy, original.getOptimisticLockValue()); setVersion(copy, original.getId(), original.getVersion(), original.isLocked()); List rootNodes = copy.getRootNodes(); @@ -318,8 +292,7 @@ public void testOptimisticLocking() { */ private void editPipelineDef(PipelineDefinition pipelineDef) { pipelineDef.setDescription("new description"); - pipelineDef.getAuditInfo().setLastChangedTime(new Date()); - pipelineDef.getAuditInfo().setLastChangedUser(operatorUser); + pipelineDef.updateAuditInfo(); } @Test @@ -402,7 +375,6 @@ private PipelineDefinitionNode editPipelineDefAddNextNode(PipelineDefinition pip PipelineDefinitionNode newPipelineNode = new PipelineDefinitionNode( expectedModuleDef3.getName(), pipelineDef.getName()); pipelineDef.getRootNodes().get(0).getNextNodes().get(0).getNextNodes().add(newPipelineNode); - newPipelineNode.setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); return newPipelineNode; } @@ -454,7 +426,6 @@ private PipelineDefinitionNode editPipelineDefAddBranchNode(PipelineDefinition p PipelineDefinitionNode newPipelineNode = new PipelineDefinitionNode( expectedModuleDef3.getName(), pipelineDef.getName()); pipelineDef.getRootNodes().get(0).getNextNodes().add(newPipelineNode); - newPipelineNode.setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); return newPipelineNode; } @@ -636,9 +607,7 @@ public void testNewInstance() { assertEquals(null, pipelineDefinitionCopy.getId()); assertEquals(null, pipelineDefinitionCopy.getId()); - assertEquals(pipelineDefinition.getAuditInfo(), pipelineDefinitionCopy.getAuditInfo()); assertEquals(pipelineDefinition.getDescription(), pipelineDefinitionCopy.getDescription()); - assertEquals(pipelineDefinition.group(), pipelineDefinitionCopy.group()); assertEquals(pipelineDefinition.getInstancePriority(), pipelineDefinitionCopy.getInstancePriority()); assertEquals(pipelineDefinition.getPipelineParameterSetNames(), diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineInstanceTaskCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineInstanceTaskCrudTest.java index 7951456..ce4e702 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineInstanceTaskCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineInstanceTaskCrudTest.java @@ -23,8 +23,8 @@ import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyUnitTestUtils; import gov.nasa.ziggy.module.PipelineException; +import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.PipelineOperations; -import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinition; @@ -40,8 +40,6 @@ import gov.nasa.ziggy.pipeline.definition.TestPipelineParameters; import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrud.ClearStaleStateResults; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.services.security.UserCrud; import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; import gov.nasa.ziggy.uow.UnitOfWork; @@ -58,10 +56,6 @@ public class PipelineInstanceTaskCrudTest { private static final String TEST_PIPELINE_NAME = "Test Pipeline"; private static final String TEST_WORKER_NAME = "TestWorker"; - private UserCrud userCrud; - - private User adminUser; - private PipelineDefinitionCrud pipelineDefinitionCrud; private PipelineInstanceCrud pipelineInstanceCrud; private PipelineInstanceNodeCrud pipelineInstanceNodeCrud; @@ -91,8 +85,6 @@ public class PipelineInstanceTaskCrudTest { @Before public void setUp() { - userCrud = new UserCrud(); - pipelineDefinitionCrud = new PipelineDefinitionCrud(); pipelineInstanceCrud = new PipelineInstanceCrud(); pipelineInstanceNodeCrud = new PipelineInstanceNodeCrud(); @@ -105,39 +97,26 @@ public void setUp() { private void populateObjects() { - DatabaseTransactionFactory.performTransaction(() -> { - // create users - adminUser = new User("admin", "Administrator", "admin@example.com", "x111"); - userCrud.createUser(adminUser); - return null; - }); - DatabaseTransactionFactory.performTransaction(() -> { // create a module param set def - parameterSet = new ParameterSet(new AuditInfo(adminUser, new Date()), "test mps1"); + parameterSet = new ParameterSet("test mps1"); parameterSet.setTypedParameters(new TestModuleParameters().getParameters()); parameterSet = parameterSetCrud.merge(parameterSet); // create a module def - moduleDef = new PipelineModuleDefinition(new AuditInfo(adminUser, new Date()), - "Test-1"); + moduleDef = new PipelineModuleDefinition("Test-1"); moduleDef = pipelineModuleDefinitionCrud.merge(moduleDef); // Create a pipeline definition. - pipelineDef = new PipelineDefinition(new AuditInfo(adminUser, new Date()), - TEST_PIPELINE_NAME); + pipelineDef = new PipelineDefinition(TEST_PIPELINE_NAME); // create some pipeline def nodes pipelineDefNode1 = new PipelineDefinitionNode(moduleDef.getName(), pipelineDef.getName()); - pipelineDefNode1 - .setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); pipelineDefNode2 = new PipelineDefinitionNode(moduleDef.getName(), pipelineDef.getName()); - pipelineDefNode2 - .setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); pipelineDef.getRootNodes().add(pipelineDefNode1); pipelineDefNode1.getNextNodes().add(pipelineDefNode2); @@ -189,7 +168,8 @@ private PipelineInstanceNode createPipelineInstanceNode(PipelineDefinitionNode p private PipelineTask createPipelineTask(PipelineInstanceNode parentPipelineInstanceNode) throws PipelineException { PipelineTask pipelineTask = new PipelineTask(pipelineInstance, parentPipelineInstanceNode); - UnitOfWork uow = new SingleUnitOfWorkGenerator().generateUnitsOfWork(null).get(0); + UnitOfWork uow = PipelineExecutor.generateUnitsOfWork(new SingleUnitOfWorkGenerator(), null) + .get(0); pipelineTask.setUowTaskParameters(uow.getParameters()); pipelineTask.setWorkerHost(TEST_WORKER_NAME); pipelineTask.setSoftwareRevision("42"); @@ -235,6 +215,7 @@ public void testStoreAndRetrieveInstance() throws Exception { }); ReflectionEquals comparer = new ReflectionEquals(); + comparer.excludeField(".*\\.lastChangedUser"); comparer.excludeField(".*\\.lastChangedTime"); comparer.assertEquals("PipelineInstance", pipelineInstance, actualPipelineInstance); @@ -262,6 +243,8 @@ public void testStoreAndRetrieveInstanceNode() throws Exception { }); ReflectionEquals comparer = new ReflectionEquals(); + comparer.excludeField(".*\\.lastChangedUser"); + comparer.excludeField(".*\\.lastChangedTime"); comparer.assertEquals("PipelineInstanceNode", pipelineInstanceNode1, actualPipelineInstanceNode); } @@ -292,6 +275,8 @@ public void testStoreAndRetrieveAllInstanceNodes() throws Exception { actualPipelineInstanceNodes.size()); ReflectionEquals comparer = new ReflectionEquals(); + comparer.excludeField(".*\\.lastChangedUser"); + comparer.excludeField(".*\\.lastChangedTime"); comparer.assertEquals("PipelineInstanceNode", pipelineInstanceNode1, actualPipelineInstanceNodes.get(0)); } diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrudTest.java index 22a4a3a..e3ef7b0 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineModuleDefinitionCrudTest.java @@ -5,7 +5,6 @@ import static org.junit.Assert.assertNull; import java.lang.reflect.Field; -import java.util.Date; import java.util.List; import org.junit.Before; @@ -17,13 +16,10 @@ import gov.nasa.ziggy.ZiggyUnitTestUtils; import gov.nasa.ziggy.crud.ZiggyQuery; import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.pipeline.definition.AuditInfo; import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.TestModuleParameters; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.services.security.User; -import gov.nasa.ziggy.services.security.UserCrud; /** * Tests for {@link PipelineModuleDefinitionCrud} Tests that objects can be stored, retrieved, and @@ -39,10 +35,6 @@ public class PipelineModuleDefinitionCrudTest { private static final String MISSING_MODULE = "I DONT EXIST"; - private UserCrud userCrud; - - private User adminUser; - private User operatorUser; private ReflectionEquals comparer; private PipelineModuleDefinitionCrud pipelineModuleDefinitionCrud; @@ -53,7 +45,6 @@ public class PipelineModuleDefinitionCrudTest { @Before public void setUp() { - userCrud = new UserCrud(); pipelineModuleDefinitionCrud = new PipelineModuleDefinitionCrud(); parameterSetCrud = new ParameterSetCrud(); comparer = new ReflectionEquals(); @@ -66,13 +57,6 @@ public void setUp() { private PipelineModuleDefinition populateObjects() { return (PipelineModuleDefinition) DatabaseTransactionFactory.performTransaction(() -> { - // create users - adminUser = new User("admin", "Administrator", "admin@example.com", "x111"); - userCrud.createUser(adminUser); - - operatorUser = new User("ops", "Operator", "ops@example.com", "x112"); - userCrud.createUser(operatorUser); - ParameterSet paramSet = createParameterSet(TEST_PARAM_SET_NAME_1); parameterSetCrud.persist(paramSet); @@ -82,14 +66,13 @@ private PipelineModuleDefinition populateObjects() { } private ParameterSet createParameterSet(String name) { - ParameterSet parameterSet = new ParameterSet(new AuditInfo(adminUser, new Date()), name); + ParameterSet parameterSet = new ParameterSet(name); parameterSet.populateFromParametersInstance(new TestModuleParameters(1)); return parameterSet; } private PipelineModuleDefinition createPipelineModuleDefinition() { - return new PipelineModuleDefinition(new AuditInfo(adminUser, new Date()), - TEST_MODULE_NAME_1); + return new PipelineModuleDefinition(TEST_MODULE_NAME_1); } private int pipelineModuleDefinitionCount() { @@ -120,12 +103,9 @@ private int moduleNameCount() { PipelineModuleDefinition copy(PipelineModuleDefinition original) throws NoSuchFieldException, SecurityException, IllegalArgumentException, IllegalAccessException { PipelineModuleDefinition copy = new PipelineModuleDefinition(original.getName()); - copy.setGroup(original.getGroup()); - copy.setAuditInfo(original.getAuditInfo()); copy.setDescription(original.getDescription()); copy.setPipelineModuleClass(original.getPipelineModuleClass()); copy.setExeTimeoutSecs(original.getExeTimeoutSecs()); - copy.setMinMemoryMegaBytes(original.getMinMemoryMegaBytes()); Field versionField = original.getClass().getSuperclass().getDeclaredField("version"); versionField.setAccessible(true); versionField.set(copy, original.getVersion()); @@ -255,8 +235,7 @@ public void testEditPipelineModuleDefinition() throws Exception { private void editModuleDef(PipelineModuleDefinition moduleDef) { // moduleDef.setName(TEST_MODULE_NAME_2); moduleDef.setDescription("new description"); - moduleDef.getAuditInfo().setLastChangedTime(new Date()); - moduleDef.getAuditInfo().setLastChangedUser(operatorUser); + moduleDef.updateAuditInfo(); } @Test @@ -283,8 +262,6 @@ public void testEditPipelineModuleParameterSetChangeParam() throws Exception { List actualParamSets = parameterSetCrud .retrieveAllVersionsForName(TEST_PARAM_SET_NAME_1); assertEquals("paramSets size", 1, actualParamSets.size()); - ParameterSet parameterSet = actualParamSets.get(0); - ZiggyUnitTestUtils.initializeUser(parameterSet.getAuditInfo().getLastChangedUser()); return actualParamSets.get(0); }); diff --git a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrudTest.java b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrudTest.java index 379919a..bb12c17 100644 --- a/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrudTest.java +++ b/src/test/java/gov/nasa/ziggy/pipeline/definition/crud/PipelineTaskCrudTest.java @@ -26,7 +26,6 @@ import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; import gov.nasa.ziggy.pipeline.definition.PipelineTask; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; -import gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator; /** * Implements unit tests for {@link PipelineTaskCrud}. @@ -180,7 +179,6 @@ private PipelineDefinition createPipelineDefinition(String pipelineName, List nodes = Stream.of(modules).map(module -> { PipelineDefinitionNode node = new PipelineDefinitionNode(module.getName(), pipelineDef.getName()); - node.setUnitOfWorkGenerator(new ClassWrapper<>(new SingleUnitOfWorkGenerator())); path.add(0); node.setPath(new PipelineDefinitionNodePath(path)); new SimpleCrud<>().persist(node); diff --git a/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java b/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java index 9dca8f9..0a2b653 100644 --- a/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java +++ b/src/test/java/gov/nasa/ziggy/services/config/ZiggyConfigurationTest.java @@ -2,6 +2,8 @@ import static gov.nasa.ziggy.services.config.PropertyName.ALLOW_PARTIAL_TASKS; import static gov.nasa.ziggy.services.config.PropertyName.DATABASE_PORT; +import static gov.nasa.ziggy.services.config.PropertyName.OPERATING_SYSTEM; +import static gov.nasa.ziggy.services.config.PropertyName.ZIGGY_HOME_DIR; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; @@ -9,7 +11,6 @@ import java.math.BigDecimal; import java.math.BigInteger; -import java.nio.file.Paths; import java.util.ArrayList; import java.util.List; import java.util.NoSuchElementException; @@ -43,6 +44,13 @@ public class ZiggyConfigurationTest { /** The number of threads accessing a property that does not exist. */ private static final int NUM_NONEXISTENT_PROPERTY_READERS = 4; + @Rule + public ZiggyPropertyRule ziggyHomeDirPropertyRule = new ZiggyPropertyRule(ZIGGY_HOME_DIR, + "build"); + + @Rule + public ZiggyPropertyRule osName = new ZiggyPropertyRule(OPERATING_SYSTEM, (String) null); + // DATABASE_PORT is a random property that normally takes numbers. @Rule public ZiggyPropertyRule fooPropertyRule = new ZiggyPropertyRule(DATABASE_PORT, "1"); @@ -53,11 +61,6 @@ public class ZiggyConfigurationTest { @Rule public ZiggyPropertyRule barPropertyRule = new ZiggyPropertyRule(ALLOW_PARTIAL_TASKS, "true"); - @Rule - public ZiggyPropertyRule schemaRule = new ZiggyPropertyRule( - PropertyName.ZIGGY_HOME_DIR.property(), - Paths.get(System.getProperty(PropertyName.WORKING_DIR.property()), "build").toString()); - private static final String BOOLEAN_PROPERTY = ALLOW_PARTIAL_TASKS.property(); @Test @@ -66,6 +69,7 @@ public void testGetInstance() { ImmutableConfiguration configuration2 = ZiggyConfiguration.getInstance(); assertNotNull(configuration1); assertTrue(configuration1 == configuration2); + assertEquals(BigDecimal.ONE, configuration1.getBigDecimal(NUMERIC_PROPERTY)); assertEquals(BigInteger.ONE, configuration1.getBigInteger(NUMERIC_PROPERTY)); assertEquals(Byte.parseByte("1"), configuration1.getByte(NUMERIC_PROPERTY)); @@ -121,15 +125,14 @@ public void testConfigurationThrowIfMissing() { @Test public void testSystemProperty() { + // This test should be the only exception to the rule of not calling System.setProperty() as + // we are testing that getInstance() reads system properties. System.setProperty("my.string.property", "foo"); System.setProperty("my.boolean.property", "true"); System.setProperty("my.int.property", "42"); System.setProperty("my.double.property", "42.42"); - // Force getInstance() to read from system properties. - ZiggyConfiguration.reset(); ImmutableConfiguration config = ZiggyConfiguration.getInstance(); - assertEquals("foo", config.getString("my.string.property")); assertEquals(true, config.getBoolean("my.boolean.property")); assertEquals(42, config.getInt("my.int.property")); @@ -147,13 +150,13 @@ public void testFilePropertyOverride() { @Test public void testSystemPropertyOverride() { + osName.setValue("foobar"); + assertEquals("foobar", + ZiggyConfiguration.getInstance().getString(OPERATING_SYSTEM.property())); } @Test public void testDefaultFileProperty() { - // Force getInstance() to read from ziggy.properties. - ZiggyConfiguration.getMutableInstance(); - assertEquals(ZIGGY_PROPERTIES_VALUE, ZiggyConfiguration.getInstance().getString(PropertyName.TEST_FILE.property())); } diff --git a/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java b/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java index 080b94a..814f951 100644 --- a/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java +++ b/src/test/java/gov/nasa/ziggy/services/events/ZiggyEventHandlerTest.java @@ -19,7 +19,6 @@ import java.util.ArrayList; import java.util.Comparator; import java.util.List; -import java.util.Set; import org.hibernate.Hibernate; import org.junit.Before; @@ -31,13 +30,12 @@ import org.xml.sax.SAXException; import com.google.common.collect.ImmutableList; -import com.google.common.collect.Sets; import gov.nasa.ziggy.TestEventDetector; import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.data.management.DataFileTypeImporter; +import gov.nasa.ziggy.data.datastore.DatastoreConfigurationImporter; import gov.nasa.ziggy.data.management.DataReceiptPipelineModule; import gov.nasa.ziggy.module.PipelineException; import gov.nasa.ziggy.parameters.ParameterLibraryImportExportCli.ParamIoMode; @@ -45,7 +43,6 @@ import gov.nasa.ziggy.pipeline.PipelineExecutor; import gov.nasa.ziggy.pipeline.PipelineOperations; import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.ParameterSet; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionOperations; import gov.nasa.ziggy.pipeline.definition.PipelineInstance; @@ -57,6 +54,7 @@ import gov.nasa.ziggy.pipeline.xml.ValidatingXmlManager; import gov.nasa.ziggy.services.alert.AlertService; import gov.nasa.ziggy.services.config.DirectoryProperties; +import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.config.ZiggyConfiguration; import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; import gov.nasa.ziggy.supervisor.PipelineSupervisor; @@ -81,7 +79,6 @@ public class ZiggyEventHandlerTest { private Path testDataDir; private ZiggyEventHandler ziggyEventHandler; private String pipelineName = "sample"; - private String testInstanceName = "test-instance-name"; private Path readyIndicator1, readyIndicator2a, readyIndicator2b; private PipelineOperations pipelineOperations = Mockito.spy(PipelineOperations.class); private PipelineExecutor pipelineExecutor = Mockito.spy(PipelineExecutor.class); @@ -104,7 +101,7 @@ public class ZiggyEventHandlerTest { @Before public void setUp() throws IOException { - testDataDir = Paths.get(dataReceiptDirPropertyRule.getProperty()); + testDataDir = Paths.get(dataReceiptDirPropertyRule.getValue()); testDataDir.toFile().mkdirs(); readyIndicator1 = testDataDir.resolve("gazelle.READY.mammal.1"); readyIndicator2a = testDataDir.resolve("psittacus.READY.bird.2"); @@ -123,7 +120,6 @@ public void setUp() throws IOException { // pipeline classes and a shortened interval between checks for the ready-indicator // file. ziggyEventHandler = Mockito.spy(ZiggyEventHandler.class); - Mockito.doReturn(testInstanceName).when(ziggyEventHandler).instanceName(); Mockito.doReturn(100L).when(ziggyEventHandler).readyFileCheckIntervalMillis(); Mockito.doReturn(Mockito.mock(AlertService.class)).when(ziggyEventHandler).alertService(); @@ -144,9 +140,9 @@ public void setUp() throws IOException { DatabaseTransactionFactory.performTransaction(() -> { new ParametersOperations().importParameterLibrary( new File(TEST_DATA_SRC, "pl-event.xml"), null, ParamIoMode.STANDARD); - new DataFileTypeImporter( + new DatastoreConfigurationImporter( ImmutableList.of(new File(TEST_DATA_SRC, "pt-event.xml").toString()), false) - .importFromFiles(); + .importConfiguration(); new PipelineModuleDefinitionCrud() .merge(DataReceiptPipelineModule.createDataReceiptPipelineForDb()); new PipelineDefinitionOperations().importPipelineConfiguration( @@ -215,6 +211,14 @@ public void testStartPipeline() throws IOException, InterruptedException { // create the ready-indicator file Files.createFile(readyIndicator1); + + // Create the manifest file. + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("gazelle") + .resolve("test-manifest.xml")); + ziggyEventHandler.run(); List events = (List) DatabaseTransactionFactory @@ -226,24 +230,6 @@ public void testStartPipeline() throws IOException, InterruptedException { assertEquals("test-event", event.getEventHandlerName()); assertTrue(event.getEventTime() != null); - // Get the instance out of the database and check its values. - PipelineInstance instance = (PipelineInstance) DatabaseTransactionFactory - .performTransaction(() -> { - PipelineInstance pipelineInstance = new PipelineInstanceCrud().retrieve(1L); - Hibernate.initialize(pipelineInstance.getPipelineParameterSets()); - return pipelineInstance; - }); - - assertEquals(testInstanceName, instance.getName()); - assertEquals(1, instance.getPipelineParameterSets().size()); - ParameterSet parameterSet = instance.getPipelineParameterSets() - .get(new ClassWrapper<>(ZiggyEventLabels.class)); - ZiggyEventLabels eventLabels = (ZiggyEventLabels) parameterSet.parametersInstance(); - assertEquals("test-event", eventLabels.getEventHandlerName()); - assertEquals("mammal", eventLabels.getEventName()); - assertEquals(1, eventLabels.getEventLabels().length); - assertEquals("gazelle", eventLabels.getEventLabels()[0]); - // Get the task out of the database and check its values. List tasks = (List) DatabaseTransactionFactory .performTransaction(this::retrievePipelineTasks); @@ -251,13 +237,9 @@ public void testStartPipeline() throws IOException, InterruptedException { assertEquals(1, tasks.size()); PipelineTask task = tasks.get(0); UnitOfWork uow = task.uowTaskInstance(); - assertEquals("gazelle", DirectoryUnitOfWorkGenerator.directory(uow)); - ParameterSet labelsParamSet = task.getParameterSet(ZiggyEventLabels.class); - eventLabels = (ZiggyEventLabels) labelsParamSet.parametersInstance(); - assertEquals("test-event", eventLabels.getEventHandlerName()); - assertEquals("mammal", eventLabels.getEventName()); - assertEquals(1, eventLabels.getEventLabels().length); - assertEquals("gazelle", eventLabels.getEventLabels()[0]); + assertEquals( + DirectoryProperties.dataReceiptDir().toAbsolutePath().resolve("gazelle").toString(), + DirectoryUnitOfWorkGenerator.directory(uow)); // The ready indicator file should be gone assertFalse(Files.exists(readyIndicator1)); @@ -288,6 +270,13 @@ public void testPreExistingReadyFile() throws IOException, InterruptedException // create the ready-indicator file Files.createFile(readyIndicator1); + // Create a manifest in the data receipt directory. + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("gazelle") + .resolve("test-manifest.xml")); + // There should be no indication that the event handler acted. DatabaseTransactionFactory.performTransaction(() -> { assertTrue(new ZiggyEventCrud().retrieveAllEvents().isEmpty()); @@ -337,6 +326,14 @@ public void testEventWithTwoReadyFiles() throws IOException, InterruptedExceptio // create one ready-indicator file Files.createFile(readyIndicator2a); + + // Create the manifest file. + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("psittacus") + .resolve("test-manifest.xml")); + ziggyEventHandler.run(); // At this point, there should be no entries in the events database table @@ -350,6 +347,14 @@ public void testEventWithTwoReadyFiles() throws IOException, InterruptedExceptio // When the second one is created, the event handler should act. Files.createFile(readyIndicator2b); + + // Create the manifest file. + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("archosaur") + .resolve("test-manifest.xml")); + ziggyEventHandler.run(); @SuppressWarnings("unchecked") @@ -362,26 +367,6 @@ public void testEventWithTwoReadyFiles() throws IOException, InterruptedExceptio assertEquals("test-event", event.getEventHandlerName()); assertTrue(event.getEventTime() != null); - // Get the instance out of the database and check its values. - PipelineInstance instance = (PipelineInstance) DatabaseTransactionFactory - .performTransaction(() -> { - PipelineInstance pipelineInstance = new PipelineInstanceCrud().retrieve(1L); - Hibernate.initialize(pipelineInstance.getPipelineParameterSets()); - return pipelineInstance; - }); - - assertEquals(testInstanceName, instance.getName()); - assertEquals(1, instance.getPipelineParameterSets().size()); - ParameterSet parameterSet = instance.getPipelineParameterSets() - .get(new ClassWrapper<>(ZiggyEventLabels.class)); - ZiggyEventLabels eventLabels = (ZiggyEventLabels) parameterSet.parametersInstance(); - assertEquals("test-event", eventLabels.getEventHandlerName()); - assertEquals("bird", eventLabels.getEventName()); - Set labels = Sets.newHashSet(eventLabels.getEventLabels()); - assertEquals(2, labels.size()); - assertTrue(labels.contains("psittacus")); - assertTrue(labels.contains("archosaur")); - // Get the tasks out of the database and check their values. @SuppressWarnings("unchecked") List tasks = (List) DatabaseTransactionFactory @@ -392,17 +377,10 @@ public void testEventWithTwoReadyFiles() throws IOException, InterruptedExceptio for (PipelineTask task : tasks) { UnitOfWork uow = task.uowTaskInstance(); uowStrings.add(DirectoryUnitOfWorkGenerator.directory(uow)); - ParameterSet labelsParamSet = task.getParameterSet(ZiggyEventLabels.class); - eventLabels = (ZiggyEventLabels) labelsParamSet.parametersInstance(); - assertEquals("test-event", eventLabels.getEventHandlerName()); - assertEquals("bird", eventLabels.getEventName()); - labels = Sets.newHashSet(eventLabels.getEventLabels()); - assertEquals(2, labels.size()); - assertTrue(labels.contains("psittacus")); - assertTrue(labels.contains("archosaur")); } - assertTrue(uowStrings.contains("psittacus")); - assertTrue(uowStrings.contains("archosaur")); + Path dataReceiptDir = DirectoryProperties.dataReceiptDir().toAbsolutePath(); + assertTrue(uowStrings.contains(dataReceiptDir.resolve("psittacus").toString())); + assertTrue(uowStrings.contains(dataReceiptDir.resolve("archosaur").toString())); } @Test @@ -412,6 +390,23 @@ public void testSimultaneousEvents() throws IOException, InterruptedException { Files.createFile(readyIndicator2b); Files.createFile(readyIndicator1); + // Create a manifest in each data receipt directory. + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("gazelle") + .resolve("test-manifest.xml")); + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("psittacus") + .resolve("test-manifest.xml")); + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("archosaur") + .resolve("test-manifest.xml")); + ziggyEventHandler.run(); @SuppressWarnings("unchecked") @@ -428,6 +423,23 @@ public void testRetrieveByInstance() throws IOException, InterruptedException { Files.createFile(readyIndicator2b); Files.createFile(readyIndicator1); + // Create the manifest files. + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("psittacus") + .resolve("test-manifest.xml")); + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("gazelle") + .resolve("test-manifest.xml")); + Files.createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("archosaur") + .resolve("test-manifest.xml")); + ziggyEventHandler.run(); List instances = (List) DatabaseTransactionFactory @@ -455,15 +467,15 @@ public void testRetrieveByInstance() throws IOException, InterruptedException { @Test public void testNullEventLabel() throws IOException, InterruptedException { - // Start by updating the task directory regex for data receipt to be "". - DatabaseTransactionFactory.performTransaction(() -> { - new ParametersOperations().importParameterLibrary( - new File(TEST_DATA_SRC, "pl-event-override.xml"), null, ParamIoMode.STANDARD); - return null; - }); - // create one ready-indicator file. Files.createFile(testDataDir.resolve("READY.mammal.1")); + + // Create a manifest in the data receipt directory. + Files + .createFile(Paths + .get(ZiggyConfiguration.getInstance() + .getString(PropertyName.DATA_RECEIPT_DIR.property())) + .resolve("test-manifest.xml")); ziggyEventHandler.run(); @SuppressWarnings("unchecked") @@ -483,14 +495,7 @@ public void testNullEventLabel() throws IOException, InterruptedException { Hibernate.initialize(pipelineInstance.getPipelineParameterSets()); return pipelineInstance; }); - assertEquals(testInstanceName, instance.getName()); - assertEquals(1, instance.getPipelineParameterSets().size()); - ParameterSet parameterSet = instance.getPipelineParameterSets() - .get(new ClassWrapper<>(ZiggyEventLabels.class)); - ZiggyEventLabels eventLabels = (ZiggyEventLabels) parameterSet.parametersInstance(); - assertEquals("test-event", eventLabels.getEventHandlerName()); - assertEquals("mammal", eventLabels.getEventName()); - assertEquals(0, eventLabels.getEventLabels().length); + assertEquals(0, instance.getPipelineParameterSets().size()); // Get the task out of the database and check its values. @SuppressWarnings("unchecked") @@ -500,12 +505,8 @@ public void testNullEventLabel() throws IOException, InterruptedException { assertEquals(1, tasks.size()); PipelineTask task = tasks.get(0); UnitOfWork uow = task.uowTaskInstance(); - assertEquals("", DirectoryUnitOfWorkGenerator.directory(uow)); - ParameterSet labelsParamSet = task.getParameterSet(ZiggyEventLabels.class); - eventLabels = (ZiggyEventLabels) labelsParamSet.parametersInstance(); - assertEquals("test-event", eventLabels.getEventHandlerName()); - assertEquals("mammal", eventLabels.getEventName()); - assertEquals(0, eventLabels.getEventLabels().length); + assertEquals(directoryRule.directory().toAbsolutePath().resolve("events").toString(), + DirectoryUnitOfWorkGenerator.directory(uow)); } private void validateEventHandler(ZiggyEventHandler handler) { diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/ProcessHeartbeatManagerTest.java b/src/test/java/gov/nasa/ziggy/services/messaging/HeartbeatManagerTest.java similarity index 68% rename from src/test/java/gov/nasa/ziggy/services/messaging/ProcessHeartbeatManagerTest.java rename to src/test/java/gov/nasa/ziggy/services/messaging/HeartbeatManagerTest.java index 86efc85..3ea74ea 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/ProcessHeartbeatManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/HeartbeatManagerTest.java @@ -1,6 +1,5 @@ package gov.nasa.ziggy.services.messaging; -import static gov.nasa.ziggy.services.config.PropertyName.DATABASE_SOFTWARE; import static gov.nasa.ziggy.services.config.PropertyName.HEARTBEAT_INTERVAL; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; @@ -8,8 +7,6 @@ import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; import static org.mockito.Mockito.mock; -import static org.mockito.Mockito.times; -import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; import java.util.concurrent.ScheduledThreadPoolExecutor; @@ -21,22 +18,19 @@ import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.services.messaging.HeartbeatManager.NoHeartbeatException; import gov.nasa.ziggy.services.messaging.MessagingTestUtils.InstrumentedHeartbeatMessage; -import gov.nasa.ziggy.services.messaging.ProcessHeartbeatManager.HeartbeatManagerAssistant; -import gov.nasa.ziggy.services.messaging.ProcessHeartbeatManager.NoHeartbeatException; import gov.nasa.ziggy.ui.ClusterController; -import gov.nasa.ziggy.ui.status.Indicator; import gov.nasa.ziggy.util.SystemProxy; /** - * Unit tests for {@link ProcessHeartbeatManager} class. + * Unit tests for {@link HeartbeatManager} class. * * @author PT */ -public class ProcessHeartbeatManagerTest { +public class HeartbeatManagerTest { - private ProcessHeartbeatManager manager; - private HeartbeatManagerAssistant assistant; + private HeartbeatManager manager; private ClusterController clusterController; private InstrumentedHeartbeatMessage heartbeatMessage = new InstrumentedHeartbeatMessage(); @@ -44,17 +38,14 @@ public class ProcessHeartbeatManagerTest { public ZiggyPropertyRule heartbeatIntervalPropertyRule = new ZiggyPropertyRule( HEARTBEAT_INTERVAL, Long.toString(0)); - @Rule - public ZiggyPropertyRule databaseSoftwarePropertyRule = new ZiggyPropertyRule(DATABASE_SOFTWARE, - "postgresql"); - @Before public void setup() { - assistant = Mockito.mock(HeartbeatManagerAssistant.class); + manager = Mockito.spy(HeartbeatManager.class); + Mockito.doNothing().when(manager).restartZiggyRmiClient(); clusterController = mock(ClusterController.class); when(clusterController.isDatabaseAvailable()).thenReturn(true); when(clusterController.isSupervisorRunning()).thenReturn(true); - ProcessHeartbeatManager.setInitializeInThread(false); + HeartbeatManager.setInitializeInThread(false); } @After @@ -72,14 +63,11 @@ public void teardown() throws InterruptedException { */ @Test public void testGoodStart() throws InterruptedException, NoHeartbeatException { - manager = new ProcessHeartbeatManager(assistant, clusterController); manager.setHeartbeatTime(1L); SystemProxy.setUserTime(5L); - manager.initializeHeartbeatManager(); + manager.start(); assertNotNull(manager.getHeartbeatListener()); assertFalse(manager.getHeartbeatListener().isShutdown()); - verify(assistant).setRmiIndicator(Indicator.State.NORMAL); - verify(assistant, times(0)).setRmiIndicator(Indicator.State.ERROR); } /** @@ -89,12 +77,11 @@ public void testGoodStart() throws InterruptedException, NoHeartbeatException { */ @Test public void testGoodRunning() throws InterruptedException, NoHeartbeatException { - manager = new ProcessHeartbeatManager(assistant, clusterController); // Set conditions such that initialization believes that a heartbeat has been detected. manager.setHeartbeatTime(1L); SystemProxy.setUserTime(5L); - manager.initializeHeartbeatManager(); + manager.start(); // Send 2 additional heartbeats at later times. SystemProxy.setUserTime(105L); @@ -102,7 +89,6 @@ public void testGoodRunning() throws InterruptedException, NoHeartbeatException SystemProxy.setUserTime(205L); manager.setHeartbeatTime(205L); manager.checkForHeartbeat(); - verify(assistant, times(0)).setRmiIndicator(Indicator.State.WARNING); // Send 2 more heartbeats at even later times. SystemProxy.setUserTime(305L); @@ -111,7 +97,6 @@ public void testGoodRunning() throws InterruptedException, NoHeartbeatException manager.setHeartbeatTime(405L); ZiggyMessenger.publish(heartbeatMessage); manager.checkForHeartbeat(); - verify(assistant, times(0)).setRmiIndicator(Indicator.State.WARNING); assertFalse(manager.getHeartbeatListener().isShutdown()); } @@ -121,13 +106,10 @@ public void testGoodRunning() throws InterruptedException, NoHeartbeatException */ @Test public void testBadStart() { - manager = new ProcessHeartbeatManager(assistant, clusterController); try { - manager.initializeHeartbeatManager(); + manager.start(); } catch (NoHeartbeatException e) { assertNull(manager.getHeartbeatListener()); - verify(assistant).setRmiIndicator(Indicator.State.ERROR); - verify(assistant, times(0)).setRmiIndicator(Indicator.State.NORMAL); } } @@ -144,16 +126,10 @@ public void testHeartbeatDetectorHearsNothing() // Set conditions such that initialization believes that a heartbeat has been detected. SystemProxy.setUserTime(5L); - manager = new ProcessHeartbeatManager(assistant, clusterController); manager.setHeartbeatTime(1L); - manager.initializeHeartbeatManager(); - verify(assistant).setRmiIndicator(Indicator.State.NORMAL); + manager.start(); manager.checkForHeartbeat(); assertNotNull(manager.getHeartbeatListener()); - verify(assistant).restartClientCommunicator(); - verify(assistant).setRmiIndicator(Indicator.State.NORMAL); - verify(assistant).setRmiIndicator(Indicator.State.WARNING); - verify(assistant).setRmiIndicator(Indicator.State.ERROR); assertTrue(manager.getHeartbeatListener().isShutdown()); } @@ -167,9 +143,8 @@ public void testHeartbeatDetectorHearsNothing() public void testHeartbeatDetectorSuccessfulRestart() throws InterruptedException, NoHeartbeatException { SystemProxy.setUserTime(5L); - manager = new ProcessHeartbeatManager(assistant, clusterController); manager.setHeartbeatTime(5L); - manager.initializeHeartbeatManager(); + manager.start(); // Note that we don't want to automatically go to reinitialization. Instead we want to // simulate waiting in the initializer for a new heartbeat. @@ -183,11 +158,8 @@ public void testHeartbeatDetectorSuccessfulRestart() manager.setHeartbeatTime(300L); // Here is where we simulate detecting the new heartbeat in the initializer. - manager.initializeHeartbeatManager(); + manager.start(); assertNotNull(manager.getHeartbeatListener()); - verify(assistant).setRmiIndicator(Indicator.State.WARNING); - verify(assistant, times(0)).setRmiIndicator(Indicator.State.ERROR); - verify(assistant, times(2)).setRmiIndicator(Indicator.State.NORMAL); assertFalse(manager.getHeartbeatListener().isShutdown()); } } diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/MessagingTestUtils.java b/src/test/java/gov/nasa/ziggy/services/messaging/MessagingTestUtils.java index 51622a7..c9d1704 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/MessagingTestUtils.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/MessagingTestUtils.java @@ -7,9 +7,7 @@ import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.PipelineMessage; import gov.nasa.ziggy.services.messages.SpecifiedRequestorMessage; -import gov.nasa.ziggy.services.messaging.ProcessHeartbeatManager.HeartbeatManagerAssistant; import gov.nasa.ziggy.ui.ClusterController; -import gov.nasa.ziggy.ui.status.Indicator; import gov.nasa.ziggy.util.SystemProxy; /** @@ -43,13 +41,13 @@ public boolean isDatabaseAvailable() { } /** - * A {@link ProcessHeartbeatManager} that provides additional information about the inner - * workings of the class. This should only be used in test, as the means by which the additional - * information is provided will degrade the long-term performance of the manager. + * A {@link HeartbeatManager} that provides additional information about the inner workings of + * the class. This should only be used in test, as the means by which the additional information + * is provided will degrade the long-term performance of the manager. * * @author PT */ - public static class InstrumentedWorkerHeartbeatManager extends ProcessHeartbeatManager { + public static class InstrumentedWorkerHeartbeatManager extends HeartbeatManager { private static final long SYS_TIME_SCALING = 100_000L; @@ -72,12 +70,11 @@ private synchronized static long systemTimeOffset() { private long systemTimeOffset = systemTimeOffset(); public InstrumentedWorkerHeartbeatManager() { - super(new MockedHeartbeatManagerAssistant(), new ClusterControllerStub(100, 1)); } @Override - public void initializeHeartbeatManager() throws NoHeartbeatException { - super.initializeHeartbeatManager(); + public void start() throws NoHeartbeatException { + super.start(); messageHandlerStartTimes.add(getHeartbeatTime() - systemTimeOffset); localStartTimes.add(getPriorHeartbeatTime() - systemTimeOffset); } @@ -129,26 +126,6 @@ public void setHeartbeatTime(long heartbeatTime) { } } - /** - * Subclass of {@link HeartbeatManagerAssistant} that does nothing. - * - * @author PT - */ - public static class MockedHeartbeatManagerAssistant implements HeartbeatManagerAssistant { - - @Override - public void setRmiIndicator(Indicator.State state) { - } - - @Override - public void setSupervisorIndicator(Indicator.State state) { - } - - @Override - public void setDatabaseIndicator(Indicator.State state) { - } - } - /** * Basic message class that gets sent from a client. * diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/RmiClientInstantiator.java b/src/test/java/gov/nasa/ziggy/services/messaging/RmiClientInstantiator.java index 3a088ab..50bd30e 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/RmiClientInstantiator.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/RmiClientInstantiator.java @@ -19,9 +19,9 @@ public class RmiClientInstantiator { public static final String SERIALIZE_MESSAGE_MAP_COMMAND_FILE_NAME = "serialize"; public static final String SERIALIZED_MESSAGE_MAP_FILE_NAME = "message-map.ser"; - public void startClient(int port, String clientReadyDir) throws IOException { + public void startClient(String clientReadyDir) throws IOException { - ZiggyRmiClient.initializeInstance(port, "external process client"); + ZiggyRmiClient.start("external process client"); ZiggyRmiClient.setUseMessenger(false); if (clientReadyDir == null) { return; @@ -51,9 +51,8 @@ public void startClient(int port, String clientReadyDir) throws IOException { public static void main(String[] args) throws IOException { - int port = Integer.parseInt(args[0]); - String clientReadyDir = args[1]; + String clientReadyDir = args[0]; - new RmiClientInstantiator().startClient(port, clientReadyDir); + new RmiClientInstantiator().startClient(clientReadyDir); } } diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/RmiInterProcessCommunicationTest.java b/src/test/java/gov/nasa/ziggy/services/messaging/RmiInterProcessCommunicationTest.java index c45ae09..2f58481 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/RmiInterProcessCommunicationTest.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/RmiInterProcessCommunicationTest.java @@ -23,6 +23,8 @@ import gov.nasa.ziggy.RunByNameTestCategory; import gov.nasa.ziggy.TestEventDetector; import gov.nasa.ziggy.ZiggyDirectoryRule; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.PipelineMessage; import gov.nasa.ziggy.services.messaging.MessagingTestUtils.Message1; @@ -50,6 +52,10 @@ public class RmiInterProcessCommunicationTest { @Rule public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); + @Rule + public ZiggyPropertyRule portRule = new ZiggyPropertyRule(PropertyName.SUPERVISOR_PORT, + Integer.toString(port)); + @Before public void setup() { serverProcess = null; @@ -95,7 +101,7 @@ public void testInterProcessCommunication() throws IOException, InterruptedExcep directoryRule.directory().resolve(RmiServerInstantiator.SERVER_READY_FILE_NAME)))); // now start the client. - ZiggyRmiClient.initializeInstance(port, "test client"); + ZiggyRmiClient.start("test client"); ZiggyRmiClient.setUseMessenger(false); Registry registry = ZiggyRmiClient.getRegistry(); @@ -144,7 +150,7 @@ public void testMultipleClients() throws IOException, ClassNotFoundException { assertTrue(TestEventDetector.detectTestEvent(1000L, () -> Files.exists( directoryRule.directory().resolve(RmiClientInstantiator.CLIENT_READY_FILE_NAME)))); - ZiggyRmiClient.initializeInstance(port, "test client"); + ZiggyRmiClient.start("test client"); ZiggyRmiClient.setUseMessenger(false); // Send an instance of Message1 diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/RmiIntraProcessCommunicationTest.java b/src/test/java/gov/nasa/ziggy/services/messaging/RmiIntraProcessCommunicationTest.java index 4dcf308..e27cb2c 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/RmiIntraProcessCommunicationTest.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/RmiIntraProcessCommunicationTest.java @@ -16,11 +16,14 @@ import org.junit.After; import org.junit.Before; +import org.junit.Rule; import org.junit.Test; import org.junit.experimental.categories.Category; import gov.nasa.ziggy.RunByNameTestCategory; import gov.nasa.ziggy.TestEventDetector; +import gov.nasa.ziggy.ZiggyPropertyRule; +import gov.nasa.ziggy.services.config.PropertyName; import gov.nasa.ziggy.services.messages.HeartbeatMessage; import gov.nasa.ziggy.services.messages.PipelineMessage; import gov.nasa.ziggy.services.messaging.MessagingTestUtils.Message1; @@ -40,6 +43,10 @@ public class RmiIntraProcessCommunicationTest { private int port = 4788; private Registry registry; + @Rule + public ZiggyPropertyRule portRule = new ZiggyPropertyRule(PropertyName.SUPERVISOR_PORT, + Integer.toString(port)); + @Before public void setup() { registry = null; @@ -64,10 +71,10 @@ public void teardown() throws RemoteException, InterruptedException { @Test public void testInitialize() { - ZiggyRmiServer.initializeInstance(port); + ZiggyRmiServer.start(); Set clientStubs = ZiggyRmiServer.getClientServiceStubs(); assertTrue(clientStubs.isEmpty()); - ZiggyRmiClient.initializeInstance(port, "test client"); + ZiggyRmiClient.start("test client"); ZiggyRmiClient.setUseMessenger(false); clientStubs = ZiggyRmiServer.getClientServiceStubs(); @@ -81,13 +88,13 @@ public void testInitialize() { @Test public void testReinitializeServer() { - ZiggyRmiServer.initializeInstance(port); + ZiggyRmiServer.start(); // Note that for this test we need to preserve a reference to the registry from // the ZiggyRmiServer instance that started it; this will be used to shut down // the registry when the test completes. registry = ZiggyRmiServer.getRegistry(); - ZiggyRmiClient.initializeInstance(port, "test client"); + ZiggyRmiClient.start("test client"); ZiggyRmiClient.setUseMessenger(false); ZiggyRmiServer.addToBroadcastQueue(new Message1("first message")); Map, List> messagesDetected = ZiggyRmiClient @@ -98,7 +105,7 @@ public void testReinitializeServer() { // Emulate a server crashing and coming back by resetting it and running the // initializer again ZiggyRmiServer.reset(); - ZiggyRmiServer.initializeInstance(port); + ZiggyRmiServer.start(); ZiggyRmiClient.setUseMessenger(false); // This instance should have no MessageHandler service references from clients @@ -145,8 +152,8 @@ public void testReinitializeServer() { */ @Test public void testReinitializeClient() { - ZiggyRmiServer.initializeInstance(port); - ZiggyRmiClient.initializeInstance(port, "test client 1"); + ZiggyRmiServer.start(); + ZiggyRmiClient.start("test client 1"); ZiggyRmiClient.setUseMessenger(false); Map, List> messagesDetected = ZiggyRmiClient .messagesDetected(); @@ -162,7 +169,7 @@ public void testReinitializeClient() { ZiggyRmiClient.reset(); // Emulate the start of a new client - ZiggyRmiClient.initializeInstance(port, "test client 2"); + ZiggyRmiClient.start("test client 2"); ZiggyRmiClient.setUseMessenger(false); // There should now be 2 client stubs -- one from the original client, one from the @@ -202,13 +209,13 @@ public void testReinitializeClient() { public void testIntraProcessCommunication() throws IOException { RmiServerInstantiator serverTest = new RmiServerInstantiator(); - serverTest.startServer(port, 2, null); + serverTest.startServer(2, null); // broadcast a message before the client has been initialized Message1 m1 = new Message1("telecaster"); broadcastAndWait(m1); - ZiggyRmiClient.initializeInstance(port, "test client"); + ZiggyRmiClient.start("test client"); ZiggyRmiClient.setUseMessenger(false); Map, List> messagesDetected = ZiggyRmiClient .messagesDetected(); @@ -248,8 +255,8 @@ public void testIntraProcessCommunication() throws IOException { @Test public void testSendWithCountdownLatch() throws IOException { RmiServerInstantiator serverTest = new RmiServerInstantiator(); - serverTest.startServer(port, 2, null); - ZiggyRmiClient.initializeInstance(port, "test client"); + serverTest.startServer(2, null); + ZiggyRmiClient.start("test client"); ZiggyRmiClient.setUseMessenger(false); CountDownLatch countdownLatch = new CountDownLatch(1); diff --git a/src/test/java/gov/nasa/ziggy/services/messaging/RmiServerInstantiator.java b/src/test/java/gov/nasa/ziggy/services/messaging/RmiServerInstantiator.java index 4532a3c..49eb206 100644 --- a/src/test/java/gov/nasa/ziggy/services/messaging/RmiServerInstantiator.java +++ b/src/test/java/gov/nasa/ziggy/services/messaging/RmiServerInstantiator.java @@ -18,9 +18,8 @@ public class RmiServerInstantiator { public static final String SHUT_DOWN_FILE_NAME = "shutdown"; public static final String SHUT_DOWN_DETECT_FILE_NAME = "shutdown-detected"; - public void startServer(int port, int nMessagesExpected, String serverReadyDir) - throws IOException { - ZiggyRmiServer.initializeInstance(port); + public void startServer(int nMessagesExpected, String serverReadyDir) throws IOException { + ZiggyRmiServer.start(); if (serverReadyDir == null) { return; @@ -38,12 +37,11 @@ public void startServer(int port, int nMessagesExpected, String serverReadyDir) public static void main(String[] args) throws IOException { - int port = Integer.parseInt(args[0]); - int expectedMessageCount = Integer.parseInt(args[1]); + int expectedMessageCount = Integer.parseInt(args[0]); String serverReadyDir = null; - if (args.length > 2) { - serverReadyDir = args[2]; + if (args.length > 1) { + serverReadyDir = args[1]; } - new RmiServerInstantiator().startServer(port, expectedMessageCount, serverReadyDir); + new RmiServerInstantiator().startServer(expectedMessageCount, serverReadyDir); } } diff --git a/src/test/java/gov/nasa/ziggy/services/security/RoleTest.java b/src/test/java/gov/nasa/ziggy/services/security/RoleTest.java deleted file mode 100644 index ef6bfae..0000000 --- a/src/test/java/gov/nasa/ziggy/services/security/RoleTest.java +++ /dev/null @@ -1,136 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; - -import java.util.Date; -import java.util.LinkedList; -import java.util.List; - -import org.junit.Before; -import org.junit.Test; - -/** - * Tests the {@link Role} class. - * - * @author Bill Wohler - */ -public class RoleTest { - private Role role; - - @Before - public void setUp() { - role = new Role("manager"); - } - - @Test - public void testConstructors() { - Role role = new Role("operator"); - assertEquals("operator", role.getName()); - - User user = new User("bar", "Bar", "foo@bar", "x4242"); - role = new Role("foo", user); - assertEquals("foo", role.getName()); - assertEquals(user, role.getCreatedBy()); - } - - @Test - public void testPrivileges() { - assertEquals(0, role.getPrivileges().size()); - - List pList = new LinkedList<>(); - pList.add(Privilege.PIPELINE_MONITOR.toString()); - role.setPrivileges(pList); - assertEquals(Privilege.PIPELINE_MONITOR.toString(), role.getPrivileges().get(0)); - - role.addPrivilege(Privilege.PIPELINE_MONITOR.toString()); - assertEquals(1, role.getPrivileges().size()); - assertEquals(Privilege.PIPELINE_MONITOR.toString(), role.getPrivileges().get(0)); - - role.addPrivilege(Privilege.PIPELINE_OPERATIONS.toString()); - assertEquals(2, role.getPrivileges().size()); - assertEquals(Privilege.PIPELINE_OPERATIONS.toString(), role.getPrivileges().get(1)); - - assertTrue(role.hasPrivilege(Privilege.PIPELINE_MONITOR.toString())); - assertTrue(role.hasPrivilege(Privilege.PIPELINE_OPERATIONS.toString())); - assertFalse(role.hasPrivilege(Privilege.PIPELINE_CONFIG.toString())); - assertFalse(role.hasPrivilege(Privilege.USER_ADMIN.toString())); - } - - @Test - public void testAddPrivileges() { - Role src1 = new Role("src1"); - src1.addPrivilege("a"); - src1.addPrivilege("b"); - Role src2 = new Role("src2"); - src2.addPrivilege("1"); - src2.addPrivilege("2"); - Role src3 = new Role("src3"); // no privileges - - Role dest = new Role("dest"); - assertEquals(0, dest.getPrivileges().size()); - dest.addPrivileges(src1); - assertEquals(2, dest.getPrivileges().size()); - assertTrue(dest.hasPrivilege("a")); - assertTrue(dest.hasPrivilege("b")); - assertFalse(dest.hasPrivilege("1")); - assertFalse(dest.hasPrivilege("2")); - dest.addPrivileges(src2); - assertEquals(4, dest.getPrivileges().size()); - assertTrue(dest.hasPrivilege("a")); - assertTrue(dest.hasPrivilege("b")); - assertTrue(dest.hasPrivilege("1")); - assertTrue(dest.hasPrivilege("2")); - dest.addPrivileges(src3); // no privileges - assertEquals(4, dest.getPrivileges().size()); - assertTrue(dest.hasPrivilege("a")); - assertTrue(dest.hasPrivilege("b")); - assertTrue(dest.hasPrivilege("1")); - assertTrue(dest.hasPrivilege("2")); - dest.addPrivileges(src2); // avoid duplicate privileges - assertEquals(4, dest.getPrivileges().size()); - assertTrue(dest.hasPrivilege("a")); - assertTrue(dest.hasPrivilege("b")); - assertTrue(dest.hasPrivilege("1")); - assertTrue(dest.hasPrivilege("2")); - } - - @Test - public void testName() { - assertEquals("manager", role.getName()); - - String s = "a string"; - role.setName(s); - assertEquals(s, role.getName()); - } - - @Test - public void testCreated() { - assertTrue(role.getCreated() != null); - - Date date = new Date(System.currentTimeMillis()); - role.setCreated(date); - assertEquals(date, role.getCreated()); - } - - @Test - public void testCreatedBy() { - assertNull(role.getCreatedBy()); - - User user = new User(); - role.setCreatedBy(user); - assertEquals(user, role.getCreatedBy()); - } - - @Test - public void testToString() { - assertEquals("manager", role.toString()); - } - - @Test - public void testEqualsObject() { - assertEquals(role, new Role("manager")); - } -} diff --git a/src/test/java/gov/nasa/ziggy/services/security/SecurityOperationsTest.java b/src/test/java/gov/nasa/ziggy/services/security/SecurityOperationsTest.java deleted file mode 100644 index 2bef631..0000000 --- a/src/test/java/gov/nasa/ziggy/services/security/SecurityOperationsTest.java +++ /dev/null @@ -1,88 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; - -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; - -import gov.nasa.ziggy.ZiggyDatabaseRule; -import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; - -/** - * Tests the {@link SecurityOperations} class. - * - * @author Bill Wohler - */ -public class SecurityOperationsTest { - private SecurityOperations securityOperations; - - @Rule - public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); - - @Before - public void setUp() { - securityOperations = new SecurityOperations(); - } - - private void populateObjects() { - DatabaseTransactionFactory.performTransaction(() -> { - TestSecuritySeedData testSecuritySeedData = new TestSecuritySeedData(); - testSecuritySeedData.loadSeedData(); - return null; - }); - } - - @Test - public void testValidateLogin() { - // Don't need to test validateLogin(User, String) explicitly since that - // method is tested indirectly by validateLogin(String, String). - populateObjects(); - - assertTrue("valid user/password", securityOperations.validateLogin("admin")); - assertFalse("invalid user", securityOperations.validateLogin("foo")); - assertFalse("null user", securityOperations.validateLogin((String) null)); - assertFalse("empty user", securityOperations.validateLogin("")); - } - - @Test - public void testHasPrivilege() { - populateObjects(); - - UserCrud userCrud = new UserCrud(); - User user = userCrud.retrieveUser("admin"); - assertTrue("admin has create", - securityOperations.hasPrivilege(user, Privilege.PIPELINE_OPERATIONS.toString())); - assertTrue("admin has modify", - securityOperations.hasPrivilege(user, Privilege.PIPELINE_MONITOR.toString())); - assertTrue("admin has monitor", - securityOperations.hasPrivilege(user, Privilege.PIPELINE_CONFIG.toString())); - assertTrue("admin has operations", - securityOperations.hasPrivilege(user, Privilege.USER_ADMIN.toString())); - - user = userCrud.retrieveUser("joe"); - assertFalse("joe does not have create", - securityOperations.hasPrivilege(user, Privilege.PIPELINE_OPERATIONS.toString())); - assertFalse("joe does not have modify", - securityOperations.hasPrivilege(user, Privilege.PIPELINE_MONITOR.toString())); - assertTrue("joe has monitor", - securityOperations.hasPrivilege(user, Privilege.PIPELINE_CONFIG.toString())); - assertTrue("joe has operations", - securityOperations.hasPrivilege(user, Privilege.USER_ADMIN.toString())); - } - - @Test - public void testGetCurrentUser() { - populateObjects(); - assertNull("user is null", securityOperations.getCurrentUser()); - securityOperations.validateLogin("foo"); - assertNull("user is null", securityOperations.getCurrentUser()); - securityOperations.validateLogin("admin"); - assertEquals("admin", securityOperations.getCurrentUser().getLoginName()); - securityOperations.validateLogin("joe"); - assertEquals("joe", securityOperations.getCurrentUser().getLoginName()); - } -} diff --git a/src/test/java/gov/nasa/ziggy/services/security/TestSecuritySeedData.java b/src/test/java/gov/nasa/ziggy/services/security/TestSecuritySeedData.java deleted file mode 100644 index a09e77d..0000000 --- a/src/test/java/gov/nasa/ziggy/services/security/TestSecuritySeedData.java +++ /dev/null @@ -1,88 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.services.database.DatabaseService; -import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; - -/** - * This class populates seed data for {@link User}s and {@link Role}s. - */ -public class TestSecuritySeedData { - private final UserCrud userCrud; - - public TestSecuritySeedData() { - userCrud = new UserCrud(); - } - - /** - * Loads initial security data into {@link User} and {@link Role} tables. Use - * {@link #deleteAllUsersAndRoles()} to clear these tables before running this method. The - * caller is responsible for calling {@link DatabaseService#beginTransaction()} and - * {@link DatabaseService#commitTransaction()}. - * - * @throws PipelineException if there were problems inserting records into the database. - */ - public void loadSeedData() { - insertAll(); - } - - private void insertAll() { - Role superRole = new Role("Super User"); - for (Privilege privilege : Privilege.values()) { - superRole.addPrivilege(privilege.toString()); - } - userCrud.createRole(superRole); - - Role opsRole = new Role("Pipeline Operator"); - opsRole.addPrivilege(Privilege.USER_ADMIN.toString()); - opsRole.addPrivilege(Privilege.PIPELINE_CONFIG.toString()); - userCrud.createRole(opsRole); - - Role managerRole = new Role("Pipeline Manager"); - managerRole.addPrivilege(Privilege.PIPELINE_OPERATIONS.toString()); - managerRole.addPrivilege(Privilege.PIPELINE_MONITOR.toString()); - userCrud.createRole(managerRole); - - User admin = new User("admin", "Administrator", "admin@example.com", "x111"); - admin.addRole(superRole); - userCrud.createUser(admin); - - User joeOps = new User("joe", "Joe Operator", "joe@example.com", "x222"); - joeOps.addRole(opsRole); - userCrud.createUser(joeOps); - - User tonyOps = new User("tony", "Tony Trainee", "tony@example.com", "x444"); - // Since Tony is only a trainee, we'll just give him monitor privs for - // now... - tonyOps.addPrivilege(Privilege.PIPELINE_CONFIG.toString()); - userCrud.createUser(tonyOps); - - User MaryMgr = new User("mary", "Mary Manager", "mary@example.com", "x333"); - MaryMgr.addRole(managerRole); - userCrud.createUser(MaryMgr); - } - - public void deleteAllUsersAndRoles() { - for (User user : userCrud.retrieveAllUsers()) { - userCrud.deleteUser(user); - } - for (Role role : userCrud.retrieveAllRoles()) { - userCrud.deleteRole(role); - } - } - - /** - * This function runs the tests declared in this class. - * - * @param args - * @throws PipelineException - */ - public static void main(String[] args) { - TestSecuritySeedData testSecuritySeedData = new TestSecuritySeedData(); - - DatabaseTransactionFactory.performTransaction(() -> { - testSecuritySeedData.loadSeedData(); - return null; - }); - } -} diff --git a/src/test/java/gov/nasa/ziggy/services/security/UserCrudTest.java b/src/test/java/gov/nasa/ziggy/services/security/UserCrudTest.java deleted file mode 100644 index 12a5b92..0000000 --- a/src/test/java/gov/nasa/ziggy/services/security/UserCrudTest.java +++ /dev/null @@ -1,184 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertThrows; -import static org.junit.Assert.assertTrue; - -import java.util.List; - -import org.hibernate.exception.ConstraintViolationException; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import gov.nasa.ziggy.ZiggyDatabaseRule; -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; - -public class UserCrudTest { - private static final Logger log = LoggerFactory.getLogger(UserCrudTest.class); - - private UserCrud userCrud = null; - - private Role superUserRole; - private Role operatorRole; - private Role monitorRole; - private User adminUser; - private User joeOperator; - private User maryMonitor; - - @Rule - public ZiggyDatabaseRule databaseRule = new ZiggyDatabaseRule(); - - @Before - public void setUp() throws Exception { - userCrud = new UserCrud(); - } - - private void createRoles(User createdBy) throws PipelineException { - superUserRole = new Role("superuser", createdBy); - operatorRole = new Role("operator", createdBy); - monitorRole = new Role("monitor", createdBy); - - userCrud.createRole(superUserRole); - userCrud.createRole(operatorRole); - userCrud.createRole(monitorRole); - } - - private void createAdminUser() throws PipelineException { - adminUser = new User("admin", "Administrator", "admin@example.com", "x111"); - - userCrud.createUser(adminUser); - } - - private void seed() throws PipelineException { - createAdminUser(); - createRoles(adminUser); - - adminUser.addRole(superUserRole); - - joeOperator = new User("joe", "Joe Operator", "joe@example.com", "x112"); - joeOperator.addRole(operatorRole); - - maryMonitor = new User("mary", "Mary Monitor", "mary@example.com", "x113"); - maryMonitor.addRole(monitorRole); - - userCrud.createUser(joeOperator); - userCrud.createUser(maryMonitor); - } - - @Test - public void testCreateRetrieve() throws Exception { - log.info("START TEST: testCreateRetrieve"); - - DatabaseTransactionFactory.performTransaction(() -> { - // store - seed(); - - // retrieve - - List roles = userCrud.retrieveAllRoles(); - - for (Role role : roles) { - log.info(role.toString()); - } - - assertEquals("BeforeCommit: roles.size()", 3, roles.size()); - assertTrue("BeforeCommit: contains superUserRole", roles.contains(superUserRole)); - assertTrue("BeforeCommit: contains operatorRole", roles.contains(operatorRole)); - assertTrue("BeforeCommit: contains monitorRole", roles.contains(monitorRole)); - - List users = userCrud.retrieveAllUsers(); - - for (User user : users) { - log.info(user.toString()); - } - - assertEquals("BeforeCommit: users.size()", 3, users.size()); - assertTrue("BeforeCommit: contains adminUser", users.contains(adminUser)); - assertTrue("BeforeCommit: contains joeOperator", users.contains(joeOperator)); - assertTrue("BeforeCommit: contains maryMonitor", users.contains(maryMonitor)); - - assertEquals("AfterCommit: roles.size()", 3, roles.size()); - assertTrue("AfterCommit: contains superUserRole", roles.contains(superUserRole)); - assertTrue("AfterCommit: contains operatorRole", roles.contains(operatorRole)); - assertTrue("AfterCommit: contains monitorRole", roles.contains(monitorRole)); - - assertEquals("AfterCommit: users.size()", 3, users.size()); - assertTrue("AfterCommit: contains adminUser", users.contains(adminUser)); - assertTrue("AfterCommit: contains joeOperator", users.contains(joeOperator)); - assertTrue("AfterCommit: contains maryMonitor", users.contains(maryMonitor)); - return null; - }); - } - - @Test - public void testDeleteRoleConstraintViolation() throws Throwable { - Throwable e = assertThrows(PipelineException.class, () -> { - - log.info("START TEST: testDeleteRoleConstraintViolation"); - - DatabaseTransactionFactory.performTransaction(() -> { - // store - seed(); - - // delete - List roles = userCrud.retrieveAllRoles(); - - /* - * This should fail because there is a User (maryMonitor) using this Role - */ - userCrud.deleteRole(roles.get(roles.indexOf(monitorRole))); - return null; - }); - }); - assertEquals(ConstraintViolationException.class, e.getCause().getClass()); - } - - /** - * Verify that we can delete a Role after we have deleted all users that reference that Role - * - * @throws Exception - */ - @Test - public void testDeleteUserAndRole() throws Exception { - log.info("START TEST: testDeleteUserAndRole"); - - DatabaseTransactionFactory.performTransaction(() -> { - // store - seed(); - - // delete User - - List users = userCrud.retrieveAllUsers(); - userCrud.deleteUser(users.get(users.indexOf(maryMonitor))); - - // delete Role - - List roles = userCrud.retrieveAllRoles(); - userCrud.deleteRole(roles.get(roles.indexOf(monitorRole))); - - // retrieve Users - - users = userCrud.retrieveAllUsers(); - - for (User user : users) { - log.info(user.toString()); - } - - assertEquals("users.size()", 2, users.size()); - assertTrue("contains adminUser", users.contains(adminUser)); - assertTrue("contains joeOperator", users.contains(joeOperator)); - - // retrieve Roles - roles = userCrud.retrieveAllRoles(); - assertEquals("AfterCommit: roles.size()", 2, roles.size()); - assertTrue("AfterCommit: contains superUserRole", roles.contains(superUserRole)); - assertTrue("AfterCommit: contains operatorRole", roles.contains(operatorRole)); - - return null; - }); - } -} diff --git a/src/test/java/gov/nasa/ziggy/services/security/UserTest.java b/src/test/java/gov/nasa/ziggy/services/security/UserTest.java deleted file mode 100644 index e1721f7..0000000 --- a/src/test/java/gov/nasa/ziggy/services/security/UserTest.java +++ /dev/null @@ -1,139 +0,0 @@ -package gov.nasa.ziggy.services.security; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertTrue; - -import java.util.Date; -import java.util.LinkedList; -import java.util.List; - -import org.junit.Before; -import org.junit.Test; - -/** - * Tests the {@link User} class. - * - * @author Bill Wohler - */ -public class UserTest { - private User user; - - @Before - public void setUp() { - user = createUser(); - } - - private User createUser() { - return new User("jamesAdmin", "Administrator", "james@example.com", "x555"); - } - - @Test - public void testConstructors() { - User user = new User(); - assertTrue(user.getCreated() != null); - - user = createUser(); - assertEquals("jamesAdmin", user.getLoginName()); - assertEquals("Administrator", user.getDisplayName()); - assertEquals("james@example.com", user.getEmail()); - assertEquals("x555", user.getPhone()); - } - - @Test - public void testLoginName() { - assertEquals("jamesAdmin", user.getLoginName()); - String s = "a string"; - user.setLoginName(s); - assertEquals(s, user.getLoginName()); - } - - @Test - public void testDisplayName() { - assertEquals("Administrator", user.getDisplayName()); - String s = "a string"; - user.setDisplayName(s); - assertEquals(s, user.getDisplayName()); - } - - @Test - public void testEmail() { - assertEquals("james@example.com", user.getEmail()); - String s = "a string"; - user.setEmail(s); - assertEquals(s, user.getEmail()); - } - - @Test - public void testPhone() { - assertEquals("x555", user.getPhone()); - String s = "a string"; - user.setPhone(s); - assertEquals(s, user.getPhone()); - } - - @Test - public void testRoles() { - assertEquals(0, user.getRoles().size()); - - Role role = new Role("operator"); - List rList = new LinkedList<>(); - rList.add(role); - user.setRoles(rList); - assertEquals(1, user.getRoles().size()); - assertEquals(role, user.getRoles().get(0)); - - role = new Role("galley-slave"); - user.addRole(role); - assertEquals(2, user.getRoles().size()); - assertEquals(role, user.getRoles().get(1)); - } - - @Test - public void testPrivileges() { - assertEquals(0, user.getPrivileges().size()); - - String privilege = Privilege.PIPELINE_MONITOR.toString(); - List pList = new LinkedList<>(); - pList.add(privilege); - user.setPrivileges(pList); - assertEquals(1, user.getPrivileges().size()); - assertEquals(privilege, user.getPrivileges().get(0)); - - privilege = Privilege.PIPELINE_OPERATIONS.toString(); - user.addPrivilege(privilege); - assertEquals(2, user.getPrivileges().size()); - assertEquals(privilege, user.getPrivileges().get(1)); - - assertTrue(user.hasPrivilege(Privilege.PIPELINE_OPERATIONS.toString())); - assertTrue(user.hasPrivilege(Privilege.PIPELINE_MONITOR.toString())); - assertFalse(user.hasPrivilege(Privilege.PIPELINE_CONFIG.toString())); - assertFalse(user.hasPrivilege(Privilege.USER_ADMIN.toString())); - } - - @Test - public void testCreated() { - assertTrue(user.getCreated() != null); - - Date date = new Date(System.currentTimeMillis()); - user.setCreated(date); - assertEquals(date, user.getCreated()); - } - - @Test - public void testHashCode() { - int hashCode = user.hashCode(); - hashCode = createUser().hashCode(); - assertEquals(hashCode, createUser().hashCode()); - } - - @Test - public void testEquals() { - assertEquals(user, createUser()); - } - - @Test - public void testToString() { - assertEquals("Administrator", user.toString()); - } -} diff --git a/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java b/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java index 8860c19..4a78d89 100644 --- a/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/supervisor/PipelineInstanceManagerTest.java @@ -191,13 +191,13 @@ public void testFireTriggerMultipleTimesAndSucceed() { assertEquals(4, pipelineInstanceManager.getRepeats()); assertEquals(3, pipelineInstanceManager.getStatusChecks()); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-0", startNode, endNode, null); + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 1/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-1", startNode, endNode, null); + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 2/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-2", startNode, endNode, null); + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 3/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-3", startNode, endNode, null); + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 4/4", startNode, endNode, null); } /** @@ -236,20 +236,20 @@ public void testFireTriggerMultipleTimesWithErrorsRunning() { assertEquals(2, pipelineInstanceManager.getRepeats()); assertEquals(2, pipelineInstanceManager.getStatusChecks()); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-0", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 1/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-1", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 2/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(0)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-2", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 3/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(0)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-3", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 4/4", startNode, endNode, null); }); - assertEquals(exception.getMessage(), - "Unable to start pipeline repeat 2 due to errored status of pipeline repeat 1"); + assertEquals("Unable to start pipeline repeat 2 due to errored status of pipeline repeat 1", + exception.getMessage()); } /** @@ -289,16 +289,16 @@ public void testFireTriggerMultipleTimesWithErrorsStalled() { assertEquals(2, pipelineInstanceManager.getRepeats()); assertEquals(2, pipelineInstanceManager.getStatusChecks()); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-0", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 1/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(1)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-1", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 2/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(0)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-2", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 3/4", startNode, endNode, null); Mockito.verify(pipelineOperations, Mockito.times(0)) - .fireTrigger(pipelineDefinition, INSTANCE_NAME + ":-3", startNode, endNode, + .fireTrigger(pipelineDefinition, INSTANCE_NAME + " 4/4", startNode, endNode, null); }); assertEquals("Unable to start pipeline repeat 2 due to errored status of pipeline repeat 1", diff --git a/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java b/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java index a7dee73..f8eb4bb 100644 --- a/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java +++ b/src/test/java/gov/nasa/ziggy/supervisor/TaskRequestHandlerLifecycleManagerTest.java @@ -32,8 +32,8 @@ import gov.nasa.ziggy.services.database.DatabaseService; import gov.nasa.ziggy.services.messages.KillTasksRequest; import gov.nasa.ziggy.services.messages.TaskRequest; -import gov.nasa.ziggy.services.messages.WorkerResources; import gov.nasa.ziggy.util.Requestor; +import gov.nasa.ziggy.worker.WorkerResources; /** * Unit tests for {@link TaskRequestHandlerLifecycleManager} class. diff --git a/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java b/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java new file mode 100644 index 0000000..8a46104 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/ui/instances/InstanceCostEstimateDialogTest.java @@ -0,0 +1,35 @@ +package gov.nasa.ziggy.ui.instances; + +import static gov.nasa.ziggy.ui.instances.InstanceCostEstimateDialog.instanceCost; +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.mock; + +import java.util.List; + +import org.junit.Test; +import org.mockito.Mockito; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; + +public class InstanceCostEstimateDialogTest { + + @Test + public void testInstanceCost() { + List pipelineTasks = List.of(createPipelineTask(0.000123)); + assertEquals("0.0001", instanceCost(pipelineTasks)); + pipelineTasks = List.of(createPipelineTask(1.00123)); + assertEquals("1.001", instanceCost(pipelineTasks)); + pipelineTasks = List.of(createPipelineTask(10.0123)); + assertEquals("10.01", instanceCost(pipelineTasks)); + pipelineTasks = List.of(createPipelineTask(100.123)); + assertEquals("100.1", instanceCost(pipelineTasks)); + pipelineTasks = List.of(createPipelineTask(1001.23)); + assertEquals("1001.2", instanceCost(pipelineTasks)); + } + + private PipelineTask createPipelineTask(double costEstimate) { + PipelineTask pipelineTask = mock(PipelineTask.class); + Mockito.when(pipelineTask.costEstimate()).thenReturn(costEstimate); + return pipelineTask; + } +} diff --git a/src/test/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGeneratorTest.java b/src/test/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGeneratorTest.java index 7ea0458..e1063e0 100644 --- a/src/test/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGeneratorTest.java +++ b/src/test/java/gov/nasa/ziggy/uow/DataReceiptUnitOfWorkGeneratorTest.java @@ -8,10 +8,8 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.util.HashMap; import java.util.HashSet; import java.util.List; -import java.util.Map; import java.util.Set; import org.junit.Before; @@ -21,8 +19,7 @@ import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.parameters.ParametersInterface; -import gov.nasa.ziggy.services.events.ZiggyEventLabels; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; /** * Unit tests for the {@link DataReceiptUnitOfWorkGenerator} class. @@ -32,8 +29,7 @@ public class DataReceiptUnitOfWorkGeneratorTest { private Path dataImporterPath; - Map, ParametersInterface> parametersMap; - TaskConfigurationParameters taskConfig; + private PipelineInstanceNode pipelineInstanceNode; public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -47,7 +43,7 @@ public class DataReceiptUnitOfWorkGeneratorTest { @Before public void setUp() throws IOException { - dataImporterPath = Paths.get(dataReceiptDirPropertyRule.getProperty()); + dataImporterPath = Paths.get(dataReceiptDirPropertyRule.getValue()).toAbsolutePath(); // Create the data receipt main directory. dataImporterPath.toFile().mkdirs(); @@ -55,56 +51,68 @@ public void setUp() throws IOException { Files.createDirectory(dataImporterPath.resolve("subdir-1")); Files.createDirectory(dataImporterPath.resolve("subdir-2")); Files.createDirectory(dataImporterPath.resolve("bad-name")); - Files.createDirectory(dataImporterPath.resolve(".manifests")); - // Create the parameters map - parametersMap = new HashMap<>(); - taskConfig = new TaskConfigurationParameters(); - taskConfig.setTaskDirectoryRegex("(subdir-[0-9]+)"); - parametersMap.put(TaskConfigurationParameters.class, taskConfig); + // Create the pipeline instance + pipelineInstanceNode = new PipelineInstanceNode(); } @Test - public void testMultipleUnitsOfWork() { + public void testMultipleUnitsOfWork() throws IOException { + Files.createFile(dataImporterPath.resolve("subdir-1").resolve("test-manifest.xml")); + Files.createFile(dataImporterPath.resolve("subdir-2").resolve("test-manifest.xml")); List unitsOfWork = new DataReceiptUnitOfWorkGenerator() - .generateTasks(parametersMap); + .unitsOfWork(pipelineInstanceNode); assertEquals(2, unitsOfWork.size()); Set dirStrings = new HashSet<>(); dirStrings.add(unitsOfWork.get(0).getParameter("directory").getString()); dirStrings.add(unitsOfWork.get(1).getParameter("directory").getString()); - assertTrue(dirStrings.contains("subdir-1")); - assertTrue(dirStrings.contains("subdir-2")); + assertTrue(dirStrings.contains(dataImporterPath.resolve("subdir-1").toString())); + assertTrue(dirStrings.contains(dataImporterPath.resolve("subdir-2").toString())); + Set briefStates = new HashSet<>(); + briefStates.add(unitsOfWork.get(0).briefState()); + briefStates.add(unitsOfWork.get(1).briefState()); + assertTrue(briefStates.contains("subdir-1")); + assertTrue(briefStates.contains("subdir-2")); } @Test - public void testSingleUnitOfWork() { - taskConfig.setTaskDirectoryRegex(""); + public void testSingleUnitOfWork() throws IOException { + Files.createFile(dataImporterPath.resolve("test-manifest.xml")); List unitsOfWork = new DataReceiptUnitOfWorkGenerator() - .generateTasks(parametersMap); + .unitsOfWork(pipelineInstanceNode); assertEquals(1, unitsOfWork.size()); - assertEquals("", unitsOfWork.get(0).getParameter("directory").getString()); + assertEquals(dataImporterPath.toString(), + unitsOfWork.get(0).getParameter("directory").getString()); + assertEquals("data-import", unitsOfWork.get(0).briefState()); } @Test - public void testEventHandlerLimitingUows() { - ZiggyEventLabels eventLabels = new ZiggyEventLabels(); - eventLabels.setEventLabels(new String[] { "subdir-1" }); - parametersMap.put(ZiggyEventLabels.class, eventLabels); + public void testSingleUnitOfWorkWithLabel() throws IOException { + Files.createFile(dataImporterPath.resolve("test-manifest.xml")); List unitsOfWork = new DataReceiptUnitOfWorkGenerator() - .generateTasks(parametersMap); + .unitsOfWork(pipelineInstanceNode, new HashSet<>()); assertEquals(1, unitsOfWork.size()); - assertEquals("subdir-1", unitsOfWork.get(0).getParameter("directory").getString()); + assertEquals(dataImporterPath.toString(), + unitsOfWork.get(0).getParameter("directory").getString()); + assertEquals("data-import", unitsOfWork.get(0).briefState()); } @Test - public void testEmptyEventLabels() { - ZiggyEventLabels eventLabels = new ZiggyEventLabels(); - eventLabels.setEventLabels(new String[0]); - parametersMap.put(ZiggyEventLabels.class, eventLabels); - taskConfig.setTaskDirectoryRegex(""); + public void testEventHandlerLimitingUows() throws IOException { + Files.createFile(dataImporterPath.resolve("subdir-1").resolve("test-manifest.xml")); + Files.createFile(dataImporterPath.resolve("subdir-2").resolve("test-manifest.xml")); List unitsOfWork = new DataReceiptUnitOfWorkGenerator() - .generateTasks(parametersMap); + .unitsOfWork(pipelineInstanceNode, Set.of("subdir-1")); assertEquals(1, unitsOfWork.size()); - assertEquals("", unitsOfWork.get(0).getParameter("directory").getString()); + assertEquals(dataImporterPath.resolve("subdir-1").toString(), + unitsOfWork.get(0).getParameter("directory").getString()); + assertEquals("subdir-1", unitsOfWork.get(0).briefState()); + } + + @Test + public void testNoDataReceiptDirectories() { + List unitsOfWork = new DataReceiptUnitOfWorkGenerator() + .generateUnitsOfWork(pipelineInstanceNode, new HashSet<>()); + assertEquals(0, unitsOfWork.size()); } } diff --git a/src/test/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkTest.java b/src/test/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkTest.java index 6fb3b77..6db3215 100644 --- a/src/test/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkTest.java +++ b/src/test/java/gov/nasa/ziggy/uow/DatastoreDirectoryUnitOfWorkTest.java @@ -2,11 +2,11 @@ import static gov.nasa.ziggy.services.config.PropertyName.DATASTORE_ROOT_DIR; import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; -import java.io.File; -import java.nio.file.Path; +import java.io.IOException; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -16,10 +16,18 @@ import org.junit.Rule; import org.junit.Test; import org.junit.rules.RuleChain; +import org.mockito.Mockito; import gov.nasa.ziggy.ZiggyDirectoryRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.parameters.ParametersInterface; +import gov.nasa.ziggy.data.datastore.DataFileType; +import gov.nasa.ziggy.data.datastore.DatastoreRegexp; +import gov.nasa.ziggy.data.datastore.DatastoreTestUtils; +import gov.nasa.ziggy.data.datastore.DatastoreWalker; +import gov.nasa.ziggy.pipeline.PipelineExecutor; +import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.services.config.DirectoryProperties; /** * Unit test class for DatastoreDirectoryUnitOfWorkGenerator. @@ -28,9 +36,10 @@ */ public class DatastoreDirectoryUnitOfWorkTest { - private Path datastoreRoot; - private Map, ParametersInterface> parametersMap; - private TaskConfigurationParameters taskConfigurationParameters; + private PipelineInstanceNode pipelineInstanceNode; + private PipelineDefinitionNode pipelineDefinitionNode; + private DatastoreDirectoryUnitOfWorkGenerator uowGenerator; + private DataFileType drSciencePixels; public ZiggyDirectoryRule directoryRule = new ZiggyDirectoryRule(); @@ -42,50 +51,28 @@ public class DatastoreDirectoryUnitOfWorkTest { .around(datastoreRootDirPropertyRule); @Before - public void setup() { + public void setup() throws IOException { - datastoreRoot = directoryRule.directory(); // Create the datastore. - File datastore = datastoreRoot.toFile(); - - // create some directories within the datastore - File sector0001 = new File(datastore, "sector-0001"); - sector0001.mkdirs(); - File sector0002 = new File(datastore, "sector-0002"); - sector0002.mkdirs(); - - File cal0001 = new File(sector0001, "cal"); - cal0001.mkdirs(); - File cal0002 = new File(sector0002, "cal"); - cal0002.mkdirs(); - File pa0002 = new File(sector0002, "pa"); - pa0002.mkdirs(); - - File ccd11 = new File(cal0001, "ccd-1:1"); - ccd11.mkdirs(); - File ccd12 = new File(cal0001, "ccd-1:2"); - ccd12.mkdirs(); - File ccd21 = new File(cal0001, "ccd-2:1"); - ccd21.mkdirs(); - ccd11 = new File(cal0002, "ccd-1:1"); - ccd11.mkdirs(); - ccd12 = new File(cal0002, "ccd-1:2"); - ccd12.mkdirs(); - ccd21 = new File(cal0002, "ccd-2:1"); - ccd21.mkdirs(); - ccd11 = new File(pa0002, "ccd-1:1"); - ccd11.mkdirs(); - ccd12 = new File(pa0002, "ccd-1:2"); - ccd12.mkdirs(); - ccd21 = new File(pa0002, "ccd-2:1"); - ccd21.mkdirs(); - - // Construct the task configuration parameters and the parameters map - taskConfigurationParameters = new TaskConfigurationParameters(); - taskConfigurationParameters.setSingleSubtask(false); - taskConfigurationParameters.setTaskDirectoryRegex("(sector-[0-9]{4})/cal/ccd-(1:[1234])"); - parametersMap = new HashMap<>(); - parametersMap.put(TaskConfigurationParameters.class, taskConfigurationParameters); + DatastoreTestUtils.createDatastoreDirectories(); + + // Create data file types. + drSciencePixels = new DataFileType("dr science pixels", + "sector/mda/dr/pixels/cadenceType/pixelType$science/channel"); + + // Create the pipeline instance node and pipeline definition node. + pipelineInstanceNode = Mockito.mock(PipelineInstanceNode.class); + pipelineDefinitionNode = Mockito.mock(PipelineDefinitionNode.class); + Mockito.when(pipelineInstanceNode.getPipelineDefinitionNode()) + .thenReturn(pipelineDefinitionNode); + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(drSciencePixels)); + + // Create the datastore walker and the UOW generator. + DatastoreWalker datastoreWalker = new DatastoreWalker(DatastoreTestUtils.regexpsByName(), + DatastoreTestUtils.datastoreNodesByFullPath()); + uowGenerator = Mockito.spy(DatastoreDirectoryUnitOfWorkGenerator.class); + Mockito.doReturn(datastoreWalker).when(uowGenerator).datastoreWalker(); } /** @@ -95,70 +82,396 @@ public void setup() { @Test public void testGenerateUnitsOfWork() { - DatastoreDirectoryUnitOfWorkGenerator uowGenInstance = new DatastoreDirectoryUnitOfWorkGenerator(); - List uowList = uowGenInstance.generateUnitsOfWork(parametersMap); - assertEquals(4, uowList.size()); + List uowList = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); // construct a map of expected results Map uowMap = new HashMap<>(); for (UnitOfWork uow : uowList) { uowMap.put(DirectoryUnitOfWorkGenerator.directory(uow), uow.briefState()); - assertFalse(DatastoreDirectoryUnitOfWorkGenerator.singleSubtask(uow)); } - Set uowKeys = uowMap.keySet(); - - // check for the expected results - assertTrue(uowKeys.contains("sector-0001/cal/ccd-1:1")); - assertEquals("sector-0001,1:1", uowMap.get("sector-0001/cal/ccd-1:1")); - assertTrue(uowKeys.contains("sector-0002/cal/ccd-1:1")); - assertEquals("sector-0002,1:1", uowMap.get("sector-0002/cal/ccd-1:1")); - assertTrue(uowKeys.contains("sector-0001/cal/ccd-1:2")); - assertEquals("sector-0001,1:2", uowMap.get("sector-0001/cal/ccd-1:2")); - assertTrue(uowKeys.contains("sector-0002/cal/ccd-1:2")); - assertEquals("sector-0002,1:2", uowMap.get("sector-0002/cal/ccd-1:2")); + + // Check the contents of the Map + String path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0002;target;1:1:A]"); + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0002;target;1:1:B]"); + + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0002;ffi;1:1:A]"); + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0002;ffi;1:1:B]"); + + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0003;target;1:1:A]"); + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0003;target;1:1:B]"); + + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:A") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0003;ffi;1:1:A]"); + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0003") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("ffi") + .resolve("science") + .resolve("1:1:B") + .toAbsolutePath() + .toString(); + assertTrue(uowMap.containsKey(path)); + assertEquals(uowMap.get(path), "[sector-0003;ffi;1:1:B]"); + + assertEquals(8, uowMap.size()); } - /** - * Tests the generation of tasks that will have a single subtask - */ + // Test that include and exclude restrictions on the DatastoreRegexps are correctly + // handled. @Test - public void testGenerateTasksSingleSubtask() { + public void testGenerateUnitsOfWorkWithIncludesAndExcludes() { + Map regexpsByName = DatastoreTestUtils.regexpsByName(); + DatastoreRegexp regexp = regexpsByName.get("sector"); + regexp.setInclude("sector-0002"); + regexp = regexpsByName.get("cadenceType"); + regexp.setExclude("ffi"); + + Mockito + .doReturn( + new DatastoreWalker(regexpsByName, DatastoreTestUtils.datastoreNodesByFullPath())) + .when(uowGenerator) + .datastoreWalker(); + + List uowList = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); - DatastoreDirectoryUnitOfWorkGenerator uowGenInstance = new DatastoreDirectoryUnitOfWorkGenerator(); - taskConfigurationParameters.setSingleSubtask(true); - List uowList = uowGenInstance.generateUnitsOfWork(parametersMap); + // construct a map of expected results + Map briefStateByDirectory = new HashMap<>(); + Map uowByBriefState = new HashMap<>(); for (UnitOfWork uow : uowList) { - assertTrue(DatastoreDirectoryUnitOfWorkGenerator.singleSubtask(uow)); + briefStateByDirectory.put(DirectoryUnitOfWorkGenerator.directory(uow), + uow.briefState()); + uowByBriefState.put(uow.briefState(), uow); } + // Check the contents of the Map + String path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:A") + .toAbsolutePath() + .toString(); + assertTrue(briefStateByDirectory.containsKey(path)); + assertEquals(briefStateByDirectory.get(path), "[1:1:A]"); + path = DirectoryProperties.datastoreRootDir() + .resolve("sector-0002") + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve("1:1:B") + .toAbsolutePath() + .toString(); + assertTrue(briefStateByDirectory.containsKey(path)); + assertEquals(briefStateByDirectory.get(path), "[1:1:B]"); + + // Test the capture of regexp values. + UnitOfWork uow = uowByBriefState.get("[1:1:A]"); + assertNotNull(uow.getParameter("sector")); + assertEquals("sector-0002", uow.getParameter("sector").getString()); + assertNotNull(uow.getParameter("cadenceType")); + assertEquals("target", uow.getParameter("cadenceType").getString()); + assertNotNull(uow.getParameter("pixelType")); + assertEquals("science", uow.getParameter("pixelType").getString()); + assertNotNull(uow.getParameter("channel")); + assertEquals("1:1:A", uow.getParameter("channel").getString()); + + // Test the capture of regexp values. + uow = uowByBriefState.get("[1:1:B]"); + assertNotNull(uow.getParameter("sector")); + assertEquals("sector-0002", uow.getParameter("sector").getString()); + assertNotNull(uow.getParameter("cadenceType")); + assertEquals("target", uow.getParameter("cadenceType").getString()); + assertNotNull(uow.getParameter("pixelType")); + assertEquals("science", uow.getParameter("pixelType").getString()); + assertNotNull(uow.getParameter("channel")); + assertEquals("1:1:B", uow.getParameter("channel").getString()); } - /** - * Tests the generation of tasks for which the "brief state" is the full directory - */ + // Tests UOW generation with multiple directories per UOW. @Test - public void testGenerateFullBriefState() { + public void testUowMultipleDirectories() { + + // Create two data file types: target and collateral. + DataFileType targetSciencePixels = new DataFileType("target science pixels", + "sector/mda/dr/pixels/cadenceType$target/pixelType$science/channel"); + DataFileType collateralSciencePixels = new DataFileType("collateral science pixels", + "sector/mda/dr/pixels/cadenceType$target/pixelType$collateral/channel"); - DatastoreDirectoryUnitOfWorkGenerator uowGenInstance = new DatastoreDirectoryUnitOfWorkGenerator(); - taskConfigurationParameters.setTaskDirectoryRegex("sector-[0-9]{4}/cal/ccd-1:[1234]"); - List uowList = uowGenInstance.generateUnitsOfWork(parametersMap); + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(targetSciencePixels, collateralSciencePixels)); + List uowList = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); + Map uowsByName = new HashMap<>(); + for (UnitOfWork uow : uowList) { + uowsByName.put(uow.briefState(), uow); + } + + testUow(uowsByName.get("[sector-0002;1:1:A]"), "sector-0002", "1:1:A"); + testUow(uowsByName.get("[sector-0002;1:1:B]"), "sector-0002", "1:1:B"); + testUow(uowsByName.get("[sector-0003;1:1:A]"), "sector-0003", "1:1:A"); + testUow(uowsByName.get("[sector-0003;1:1:B]"), "sector-0003", "1:1:B"); assertEquals(4, uowList.size()); + } - Map uowMap = new HashMap<>(); + /** Performs all necessary tests on a {@link UnitOfWork} instance. */ + private void testUow(UnitOfWork uow, String sector, String channel) { + assertNotNull(uow); + + // Test that the correct directories are present. + List directories = DirectoryUnitOfWorkGenerator.directories(uow); + assertTrue(directories.contains(DirectoryProperties.datastoreRootDir() + .resolve(sector) + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve(channel) + .toAbsolutePath() + .toString())); + assertTrue(directories.contains(DirectoryProperties.datastoreRootDir() + .resolve(sector) + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve(channel) + .toAbsolutePath() + .toString())); + assertNotNull(DirectoryUnitOfWorkGenerator.directory(uow)); + assertEquals(2, directories.size()); + + // Test that the mapping from data file type to directory is correct. + Map directoriesByDataFileType = DirectoryUnitOfWorkGenerator + .directoriesByDataFileType(uow); + assertEquals(2, directoriesByDataFileType.size()); + String dataFileTypeDirectory = directoriesByDataFileType.get("target science pixels"); + assertNotNull(dataFileTypeDirectory); + assertEquals(DirectoryProperties.datastoreRootDir() + .resolve(sector) + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("science") + .resolve(channel) + .toAbsolutePath() + .toString(), dataFileTypeDirectory); + dataFileTypeDirectory = directoriesByDataFileType.get("collateral science pixels"); + assertNotNull(dataFileTypeDirectory); + assertEquals(DirectoryProperties.datastoreRootDir() + .resolve(sector) + .resolve("mda") + .resolve("dr") + .resolve("pixels") + .resolve("target") + .resolve("collateral") + .resolve(channel) + .toAbsolutePath() + .toString(), dataFileTypeDirectory); + } + + @Test + public void testUowMultipleDirectoriesWithIncludes() { + + // Create two data file types: target and collateral. + DataFileType targetSciencePixels = new DataFileType("target science pixels", + "sector/mda/dr/pixels/cadenceType$target/pixelType$science/channel"); + DataFileType collateralSciencePixels = new DataFileType("collateral science pixels", + "sector/mda/dr/pixels/cadenceType$target/pixelType$collateral/channel"); + + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(targetSciencePixels, collateralSciencePixels)); + + // Create an include restriction. + Map regexpsByName = DatastoreTestUtils.regexpsByName(); + DatastoreRegexp regexp = regexpsByName.get("sector"); + regexp.setInclude("sector-0002"); + + Mockito + .doReturn( + new DatastoreWalker(regexpsByName, DatastoreTestUtils.datastoreNodesByFullPath())) + .when(uowGenerator) + .datastoreWalker(); + + List uowList = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); + Map uowsByName = new HashMap<>(); for (UnitOfWork uow : uowList) { - uowMap.put(DirectoryUnitOfWorkGenerator.directory(uow), uow.briefState()); - assertFalse(DatastoreDirectoryUnitOfWorkGenerator.singleSubtask(uow)); + uowsByName.put(uow.briefState(), uow); } - Set uowKeys = uowMap.keySet(); - - // check for the expected results - assertTrue(uowKeys.contains("sector-0001/cal/ccd-1:1")); - assertEquals("sector-0001/cal/ccd-1:1", uowMap.get("sector-0001/cal/ccd-1:1")); - assertTrue(uowKeys.contains("sector-0002/cal/ccd-1:1")); - assertEquals("sector-0002/cal/ccd-1:1", uowMap.get("sector-0002/cal/ccd-1:1")); - assertTrue(uowKeys.contains("sector-0001/cal/ccd-1:2")); - assertEquals("sector-0001/cal/ccd-1:2", uowMap.get("sector-0001/cal/ccd-1:2")); - assertTrue(uowKeys.contains("sector-0002/cal/ccd-1:2")); - assertEquals("sector-0002/cal/ccd-1:2", uowMap.get("sector-0002/cal/ccd-1:2")); + + testUow(uowsByName.get("[1:1:A]"), "sector-0002", "1:1:A"); + testUow(uowsByName.get("[1:1:B]"), "sector-0002", "1:1:B"); + assertEquals(2, uowList.size()); + + // The pixel type regexp value should be missing, since we are using both science and + // collateral pixels in this UOW. + UnitOfWork uow = uowsByName.get("[1:1:A]"); + assertNotNull(uow.getParameter("sector")); + assertEquals("sector-0002", uow.getParameter("sector").getString()); + assertNotNull(uow.getParameter("cadenceType")); + assertEquals("target", uow.getParameter("cadenceType").getString()); + assertNull(uow.getParameter("pixelType")); + assertNotNull(uow.getParameter("channel")); + assertEquals("1:1:A", uow.getParameter("channel").getString()); + assertNull(uow.getParameter("pixelType")); + + uow = uowsByName.get("[1:1:B]"); + assertNotNull(uow.getParameter("sector")); + assertEquals("sector-0002", uow.getParameter("sector").getString()); + assertNotNull(uow.getParameter("cadenceType")); + assertEquals("target", uow.getParameter("cadenceType").getString()); + assertNull(uow.getParameter("pixelType")); + assertNotNull(uow.getParameter("channel")); + assertEquals("1:1:B", uow.getParameter("channel").getString()); + assertNull(uow.getParameter("pixelType")); + } + + @Test + public void testGenerateUowsSingleUowSingleDataFileType() { + Map regexpsByName = DatastoreTestUtils.regexpsByName(); + DatastoreRegexp regexp = regexpsByName.get("sector"); + regexp.setInclude("sector-0002"); + regexp = regexpsByName.get("cadenceType"); + regexp.setExclude("ffi"); + regexp = regexpsByName.get("channel"); + regexp.setInclude("1:1:A"); + + Mockito + .doReturn( + new DatastoreWalker(regexpsByName, DatastoreTestUtils.datastoreNodesByFullPath())) + .when(uowGenerator) + .datastoreWalker(); + + List uowList = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); + assertEquals(1, uowList.size()); + UnitOfWork uow = uowList.get(0); + assertEquals("[sector-0002;target;1:1:A]", uow.briefState()); + assertEquals("sector-0002", uow.getParameter("sector").getString()); + assertEquals("target", uow.getParameter("cadenceType").getString()); + assertEquals("1:1:A", uow.getParameter("channel").getString()); + assertEquals("science", uow.getParameter("pixelType").getString()); + } + + @Test + public void testGenerateUowsSingleUowMultipleDataFileTypes() { + Map regexpsByName = DatastoreTestUtils.regexpsByName(); + DatastoreRegexp regexp = regexpsByName.get("sector"); + regexp.setInclude("sector-0002"); + regexp = regexpsByName.get("cadenceType"); + regexp.setExclude("ffi"); + regexp = regexpsByName.get("channel"); + regexp.setInclude("1:1:A"); + + Mockito + .doReturn( + new DatastoreWalker(regexpsByName, DatastoreTestUtils.datastoreNodesByFullPath())) + .when(uowGenerator) + .datastoreWalker(); + + // Create another file type. + DataFileType drCollateralPixels = new DataFileType("dr science pixels", + "sector/mda/dr/pixels/cadenceType/pixelType$collateral/channel"); + + Mockito.when(pipelineDefinitionNode.getInputDataFileTypes()) + .thenReturn(Set.of(drSciencePixels, drCollateralPixels)); + + List uowList = PipelineExecutor.generateUnitsOfWork(uowGenerator, + pipelineInstanceNode); + assertEquals(1, uowList.size()); + UnitOfWork uow = uowList.get(0); + assertEquals("[sector-0002;target;1:1:A]", uow.briefState()); + assertEquals("sector-0002", uow.getParameter("sector").getString()); + assertEquals("target", uow.getParameter("cadenceType").getString()); + assertEquals("1:1:A", uow.getParameter("channel").getString()); + assertNull(uow.getParameter("pixelType")); } } diff --git a/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkGeneratorTest.java b/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkGeneratorTest.java index 8cbd0ad..0386fda 100644 --- a/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkGeneratorTest.java +++ b/src/test/java/gov/nasa/ziggy/uow/UnitOfWorkGeneratorTest.java @@ -5,25 +5,14 @@ import java.util.ArrayList; import java.util.List; -import java.util.Map; import org.junit.Rule; import org.junit.Test; import gov.nasa.ziggy.ZiggyDatabaseRule; import gov.nasa.ziggy.ZiggyPropertyRule; -import gov.nasa.ziggy.data.management.DataReceiptPipelineModule; -import gov.nasa.ziggy.module.ExternalProcessPipelineModule; -import gov.nasa.ziggy.module.PipelineException; -import gov.nasa.ziggy.parameters.ParametersInterface; -import gov.nasa.ziggy.pipeline.definition.ClassWrapper; -import gov.nasa.ziggy.pipeline.definition.PipelineDefinitionNode; -import gov.nasa.ziggy.pipeline.definition.PipelineModule; -import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineModuleDefinitionCrud; -import gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrudTest; -import gov.nasa.ziggy.services.config.ZiggyConfiguration; -import gov.nasa.ziggy.services.database.DatabaseTransactionFactory; +import gov.nasa.ziggy.pipeline.PipelineExecutor; +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; /** * Unit tests for {@link UnitOfWorkGenerator} class. @@ -48,7 +37,7 @@ public class UnitOfWorkGeneratorTest { public void testGenerateUnitsOfWork() { SampleUnitOfWorkGenerator generator = new SampleUnitOfWorkGenerator(); - List uowList = generator.generateUnitsOfWork(null); + List uowList = PipelineExecutor.generateUnitsOfWork(generator, null); assertEquals(1, uowList.size()); UnitOfWork uow = uowList.get(0); assertEquals("sample brief state", uow.briefState()); @@ -56,67 +45,6 @@ public void testGenerateUnitsOfWork() { uow.getParameter("uowGenerator").getString()); } - /** - * Tests that the Ziggy-side default UOW is correctly identified. - */ - @Test - public void testDefaultUnitOfWorkGenerator() { - - Class generator = UnitOfWorkGenerator - .defaultUnitOfWorkGenerator(ExternalProcessPipelineModule.class); - assertEquals(DatastoreDirectoryUnitOfWorkGenerator.class, generator); - generator = UnitOfWorkGenerator.defaultUnitOfWorkGenerator(DataReceiptPipelineModule.class); - assertEquals(DataReceiptUnitOfWorkGenerator.class, generator); - } - - /** - * Tests that the correct exception is thrown when a class lacks a default UOW generator. - */ - @Test(expected = PipelineException.class) - public void testNoDefaultGenerator() { - // Clear property set in property rule. - ZiggyConfiguration.reset(); - UnitOfWorkGenerator.defaultUnitOfWorkGenerator(PipelineTaskCrudTest.TestModule.class); - } - - /** - * Tests that an external UOW generator is correctly handled. - */ - @Test - public void testExternalDefaultIdentifier() { - Class generator = UnitOfWorkGenerator - .defaultUnitOfWorkGenerator(PipelineTaskCrudTest.TestModule.class); - assertEquals(SingleUnitOfWorkGenerator.class, generator); - generator = UnitOfWorkGenerator.defaultUnitOfWorkGenerator(DataReceiptPipelineModule.class); - assertEquals(DataReceiptUnitOfWorkGenerator.class, generator); - } - - /** - * Tests that the UOW generator is correctly retrieved in both the normal and default cases. - */ - @Test - public void testUowGeneratorFromNodeDefinition() { - - PipelineDefinitionNode node = new PipelineDefinitionNode(); - node.setUnitOfWorkGenerator(new ClassWrapper<>(SingleUnitOfWorkGenerator.class)); - ClassWrapper generator = UnitOfWorkGenerator.unitOfWorkGenerator(node); - assertEquals("gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator", generator.getClassName()); - - // Now test a node with no UOW generator specified and ensure that the default is retrieved. - node.setUnitOfWorkGenerator(null); - PipelineModuleDefinition modDef = new PipelineModuleDefinition("the-module"); - modDef.setPipelineModuleClass(new ClassWrapper<>(ExternalProcessPipelineModule.class)); - node.setModuleName(modDef.getName()); - DatabaseTransactionFactory.performTransaction(() -> { - PipelineModuleDefinitionCrud crud = new PipelineModuleDefinitionCrud(); - crud.persist(modDef); - return null; - }); - generator = UnitOfWorkGenerator.unitOfWorkGenerator(node); - assertEquals("gov.nasa.ziggy.uow.DatastoreDirectoryUnitOfWorkGenerator", - generator.getClassName()); - } - /** * Super-basic UOW generator. * @@ -125,13 +53,7 @@ public void testUowGeneratorFromNodeDefinition() { private static class SampleUnitOfWorkGenerator implements UnitOfWorkGenerator { @Override - public List> requiredParameterClasses() { - return new ArrayList<>(); - } - - @Override - public List generateTasks( - Map, ParametersInterface> parameters) { + public List generateUnitsOfWork(PipelineInstanceNode pipelineInstanceNode) { UnitOfWork uow = new UnitOfWork(); List uowList = new ArrayList<>(); uowList.add(uow); @@ -139,30 +61,8 @@ public List generateTasks( } @Override - public String briefState(UnitOfWork uow) { - return "sample brief state"; - } - } - - /** - * Super-basic default UOW generator identifier. - * - * @author PT - */ - @SuppressWarnings("unused") - private static class SampleUnitOfWorkIdentifier extends DefaultUnitOfWorkIdentifier { - - public SampleUnitOfWorkIdentifier() { - } - - @Override - public Class defaultUnitOfWorkGeneratorForClass( - Class module) { - Class defaultUowGenerator = null; - if (module.equals(PipelineTaskCrudTest.TestModule.class)) { - return SingleUnitOfWorkGenerator.class; - } - return defaultUowGenerator; + public void setBriefState(UnitOfWork uow, PipelineInstanceNode pipelineInstanceNode) { + uow.setBriefState("sample brief state"); } } } diff --git a/src/test/java/gov/nasa/ziggy/util/ClasspathScannerTest.java b/src/test/java/gov/nasa/ziggy/util/ClasspathScannerTest.java new file mode 100644 index 0000000..1ed8b76 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/ClasspathScannerTest.java @@ -0,0 +1,174 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + +import org.junit.Before; +import org.junit.Test; + +public class ClasspathScannerTest { + + private static final String ZIGGY_JAR_REGEXP = "ziggy-[\\d.]+\\.jar"; + + private Path ziggyJarFile = findZiggyJar(); + private ClasspathScannerListener classpathScannerListener; + private boolean called; + private Set classes = new HashSet<>(); + + @Before + public void setUp() { + called = false; + classes.clear(); + } + + @Test + public void testListener() { + ClasspathScanner classpathScanner = classpathScanner(); + + // addListener() called by classpathScanner(). + classpathScanner.scanForClasses(); + assertTrue(called); + called = false; + + classpathScanner.removeListener(classpathScannerListener); + classpathScanner.scanForClasses(); + assertFalse(called); + } + + @Test + public void testIncludeJarFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + Set includeJarFilters = Set.of(ZIGGY_JAR_REGEXP); + classpathScanner.setIncludeJarFilters(includeJarFilters); + assertEquals(includeJarFilters, classpathScanner.getIncludeJarFilters()); + classpathScanner.scanForClasses(); + checkForZiggyClasses(); + } + + @Test + public void testIncludeNonexistentJarFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + classpathScanner.setIncludeJarFilters(Set.of("i-dont-exist")); + classpathScanner.scanForClasses(); + assertTrue(classes.isEmpty()); + } + + @Test + public void testExcludeJarFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + Set excludeJarFilters = Set.of(ZIGGY_JAR_REGEXP); + classpathScanner.setExcludeJarFilters(excludeJarFilters); + assertEquals(excludeJarFilters, classpathScanner.getExcludeJarFilters()); + classpathScanner.scanForClasses(); + assertTrue(classes.isEmpty()); + } + + @Test + public void testExcludeNonexistentJarFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + classpathScanner.setExcludeJarFilters(Set.of("i-dont-exist")); + classpathScanner.scanForClasses(); + checkForZiggyClasses(); + } + + @Test + public void testIncludePackageFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + Set includePackageFilters = Set.of("gov\\.nasa\\.ziggy\\.util\\..*"); + classpathScanner.setIncludePackageFilters(includePackageFilters); + assertEquals(includePackageFilters, classpathScanner.getIncludePackageFilters()); + classpathScanner.scanForClasses(); + checkForClassesInPackage("gov.nasa.ziggy.util"); + } + + @Test + public void testIncludeNonexistentPackageFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + Set includePackageFilters = Set.of("foo.bar.baz"); + classpathScanner.setIncludePackageFilters(includePackageFilters); + classpathScanner.scanForClasses(); + assertTrue(classes.isEmpty()); + } + + @Test + public void testExcludePackageFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + Set excludePackageFilters = Set.of("gov\\.nasa\\.ziggy\\..*"); + classpathScanner.setExcludePackageFilters(excludePackageFilters); + assertEquals(excludePackageFilters, classpathScanner.getExcludePackageFilters()); + classpathScanner.scanForClasses(); + assertTrue(classes.isEmpty()); + } + + @Test + public void testExcludeNonexistentPackageFilters() { + ClasspathScanner classpathScanner = classpathScanner(); + Set excludePackageFilters = Set.of("foo.bar.baz"); + classpathScanner.setExcludePackageFilters(excludePackageFilters); + assertEquals(excludePackageFilters, classpathScanner.getExcludePackageFilters()); + classpathScanner.scanForClasses(); + checkForZiggyClasses(); + } + + private ClasspathScanner classpathScanner() { + ClasspathScanner classpathScanner = new ClasspathScanner(); + Set classPathToScan = Set.of(ziggyJarFile.toString()); + classpathScanner.setClassPathToScan(classPathToScan); + assertEquals(classPathToScan, classpathScanner.getClassPathToScan()); + classpathScannerListener = classpathScannerListener(); + classpathScanner.addListener(classpathScannerListener); + + return classpathScanner; + } + + private ClasspathScannerListener classpathScannerListener() { + return classFile -> { + // System.out.println(classFile.getName()); + + called = true; + classes.add(classFile.getName()); + }; + } + + // Returns the path to build/libs/ziggy-m.n.p.jar. + private Path findZiggyJar() { + try { + List paths = Files.list(Paths.get("build/libs")) + .filter(path -> path.getFileName().toString().matches(ZIGGY_JAR_REGEXP)) + .collect(Collectors.toList()); + if (paths.size() != 1) { + throw new IllegalStateException( + "Could not find one match of ziggy-[\\d.]+\\.jar in build/libs: " + paths); + } + return paths.get(0); + } catch (IOException e) { + throw new IllegalStateException("Can't open build/libs"); + } + } + + private void checkForZiggyClasses() { + checkForClassesInPackage("gov.nasa.ziggy"); + } + + private void checkForClassesInPackage(String packageName) { + assertTrue(classes.size() > 0); + + for (String clazz : classes) { + // Avoid unnecessary string concatenation in assertTrue(message, condition) call. + if (!clazz.startsWith(packageName)) { + fail(clazz + " doesn't start with " + packageName); + } + } + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java b/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java new file mode 100644 index 0000000..65b0fd8 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/CollectionFiltersTest.java @@ -0,0 +1,45 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Set; + +import org.junit.Test; + +public class CollectionFiltersTest { + + @Test + public void testFilterToList() { + List list = List.of(new Object(), new Date(), Double.valueOf(1.0), + Integer.valueOf(1)); + assertEquals(List.of(new Date()), CollectionFilters.filterToList(list, Date.class)); + assertEquals(List.of(Double.valueOf(1.0), Integer.valueOf(1)), + CollectionFilters.filterToList(list, Number.class)); + } + + @Test + public void testFilterToSet() { + Set set = Set.of(new Object(), new Date(), Double.valueOf(1.0), Integer.valueOf(1)); + assertEquals(Set.of(new Date()), CollectionFilters.filterToSet(set, Date.class)); + assertEquals(Set.of(Double.valueOf(1.0), Integer.valueOf(1)), + CollectionFilters.filterToSet(set, Number.class)); + } + + @Test + public void testRemoveTypeFromCollection() { + List list = new ArrayList<>(); + Object object = new Object(); + list.add(object); + Date date = new Date(); + list.add(date); + list.add(Double.valueOf(1.0)); + list.add(Integer.valueOf(1)); + CollectionFilters.removeTypeFromCollection(list, Number.class); + assertEquals(List.of(object, date), list); + CollectionFilters.removeTypeFromCollection(list, Date.class); + assertEquals(List.of(object), list); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/HostNameUtilsTest.java b/src/test/java/gov/nasa/ziggy/util/HostNameUtilsTest.java new file mode 100644 index 0000000..36a8531 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/HostNameUtilsTest.java @@ -0,0 +1,47 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; + +import org.junit.Test; + +public class HostNameUtilsTest { + + @Test + public void testHostName() { + String hostName = HostNameUtils.hostName(); + assertNotNull(hostName); + assertFalse(hostName.isEmpty()); + } + + @Test + public void testShortHostName() { + String hostName = HostNameUtils.shortHostName(); + assertNotNull(hostName); + assertFalse(hostName.isEmpty()); + assertFalse(hostName.contains(".")); + } + + @Test + public void testShortHostNameFromHostName() { + String hostName = HostNameUtils.shortHostNameFromHostName("foo"); + assertNotNull(hostName); + assertEquals("foo", hostName); + + hostName = HostNameUtils.shortHostNameFromHostName("foo.bar.baz"); + assertNotNull(hostName); + assertEquals("foo", hostName); + } + + @Test + public void testCallerHostNameOrLocalhost() { + String hostName = HostNameUtils.callerHostNameOrLocalhost("foo"); + assertNotNull(hostName); + assertEquals("foo", hostName); + + hostName = HostNameUtils.callerHostNameOrLocalhost("foo.bar.baz"); + assertNotNull(hostName); + assertEquals("foo", hostName); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/HumanReadableHeapSizeTest.java b/src/test/java/gov/nasa/ziggy/util/HumanReadableHeapSizeTest.java new file mode 100644 index 0000000..6102447 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/HumanReadableHeapSizeTest.java @@ -0,0 +1,59 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +import gov.nasa.ziggy.util.HumanReadableHeapSize.HeapSizeUnit; + +public class HumanReadableHeapSizeTest { + + @Test + public void testGetHumanReadableHeapSize() { + assertEquals(0.0, new HumanReadableHeapSize(0).getHumanReadableHeapSize(), 0.0001); + assertEquals(999.0, new HumanReadableHeapSize(999).getHumanReadableHeapSize(), 0.0001); + assertEquals(1.0, new HumanReadableHeapSize(1000).getHumanReadableHeapSize(), 0.0001); + assertEquals(1000.0, new HumanReadableHeapSize(1000000).getHumanReadableHeapSize(), 0.0001); + assertEquals(1.000001, new HumanReadableHeapSize(1000001).getHumanReadableHeapSize(), + 0.0001); + } + + @Test + public void testGetHeapSizeUnit() { + assertEquals(HeapSizeUnit.MB, new HumanReadableHeapSize(0).getHeapSizeUnit()); + assertEquals(HeapSizeUnit.MB, new HumanReadableHeapSize(999).getHeapSizeUnit()); + assertEquals(HeapSizeUnit.GB, new HumanReadableHeapSize(1000).getHeapSizeUnit()); + assertEquals(HeapSizeUnit.GB, new HumanReadableHeapSize(1000000).getHeapSizeUnit()); + assertEquals(HeapSizeUnit.TB, new HumanReadableHeapSize(1000001).getHeapSizeUnit()); + + assertEquals(HeapSizeUnit.MB, + new HumanReadableHeapSize(0, HeapSizeUnit.MB).getHeapSizeUnit()); + assertEquals(HeapSizeUnit.GB, + new HumanReadableHeapSize(0, HeapSizeUnit.GB).getHeapSizeUnit()); + assertEquals(HeapSizeUnit.TB, + new HumanReadableHeapSize(0, HeapSizeUnit.TB).getHeapSizeUnit()); + } + + @Test + public void testHeapSizeMb() { + assertEquals(0, new HumanReadableHeapSize(0).heapSizeMb()); + assertEquals(1, new HumanReadableHeapSize(1).heapSizeMb()); + assertEquals(999, new HumanReadableHeapSize(999).heapSizeMb()); + assertEquals(1000, new HumanReadableHeapSize(1000).heapSizeMb()); + assertEquals(1001, new HumanReadableHeapSize(1001).heapSizeMb()); + assertEquals(999999, new HumanReadableHeapSize(999999).heapSizeMb()); + assertEquals(1000000, new HumanReadableHeapSize(1000000).heapSizeMb()); + // TODO Determine why 1000001 fails and 1000002 passes + // assertEquals(1000001, new HumanReadableHeapSize(1000001).heapSizeMb()); + assertEquals(1000002, new HumanReadableHeapSize(1000002).heapSizeMb()); + } + + @Test + public void testToString() { + assertEquals("0.0 MB", new HumanReadableHeapSize(0).toString()); + assertEquals("999.0 MB", new HumanReadableHeapSize(999).toString()); + assertEquals("1.0 GB", new HumanReadableHeapSize(1000).toString()); + assertEquals("1000.0 GB", new HumanReadableHeapSize(1000000).toString()); + assertEquals("1.000001 TB", new HumanReadableHeapSize(1000001).toString()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/Iso8601FormatterTest.java b/src/test/java/gov/nasa/ziggy/util/Iso8601FormatterTest.java new file mode 100644 index 0000000..579af11 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/Iso8601FormatterTest.java @@ -0,0 +1,52 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import java.util.Calendar; +import java.util.Date; + +import org.junit.Before; +import org.junit.Test; + +public class Iso8601FormatterTest { + + private Date date; + + @Before + public void setUp() { + Calendar calendar = Calendar.getInstance(); + calendar.set(2024, 00, 03, 16, 52, 40); // local time + calendar.set(Calendar.MILLISECOND, 0); + date = calendar.getTime(); + } + + @Test + public void testDateFormatter() { + assertEquals("2024-01-04", Iso8601Formatter.dateFormatter().format(date)); + + // Second call adds coverage for the cached formatter code. + assertEquals("2024-01-04", Iso8601Formatter.dateFormatter().format(date)); + } + + @Test + public void testDateTimeFormatter() { + assertEquals("2024-01-04T00:52:40Z", Iso8601Formatter.dateTimeFormatter().format(date)); + } + + @Test + public void testDateTimeMillisFormatter() { + assertEquals("2024-01-04T00:52:40.000Z", + Iso8601Formatter.dateTimeMillisFormatter().format(date)); + } + + @Test + public void testDateTimeLocalFormatter() { + assertEquals("20240103T165240", Iso8601Formatter.dateTimeLocalFormatter().format(date)); + } + + @Test + public void testJavaDateTimeSansMillisLocalFormatter() { + assertEquals("2024-01-03 16:52:40", + Iso8601Formatter.javaDateTimeSansMillisLocalFormatter().format(date)); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/ReflectionUtilsTest.java b/src/test/java/gov/nasa/ziggy/util/ReflectionUtilsTest.java new file mode 100644 index 0000000..9cd60ed --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/ReflectionUtilsTest.java @@ -0,0 +1,59 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.lang.reflect.Field; +import java.util.List; +import java.util.stream.Collectors; + +import org.junit.Test; + +import gov.nasa.ziggy.module.io.ProxyIgnore; + +public class ReflectionUtilsTest { + + @Test + public void testGetAllFieldsOfClass() { + checkFields(ReflectionUtils.getAllFields(ReflectionSample.class, false), false); + checkFields(ReflectionUtils.getAllFields(ReflectionSample.class, true), true); + } + + @Test + public void testGetAllFieldsOfObject() { + checkFields(ReflectionUtils.getAllFields(new ReflectionSample(), false), false); + checkFields(ReflectionUtils.getAllFields(new ReflectionSample(), true), true); + } + + private void checkFields(List fields, boolean includeProxyIgnoreFields) { + assertEquals(includeProxyIgnoreFields ? 5 : 3, fields.size()); + + List fieldNames = fields.stream().map(Field::getName).collect(Collectors.toList()); + + assertTrue(fieldNames.contains("stringValue")); + assertTrue(fieldNames.contains("intValue")); + assertTrue(fieldNames.contains("floatValue")); + + if (includeProxyIgnoreFields) { + assertTrue(fieldNames.contains("ignoredString")); + assertTrue(fieldNames.contains("ignoredInt")); + } + } + + private static class ReflectionSample { + @SuppressWarnings("unused") + private static final int SOME_STATIC_CONSTANT = 42; + + @SuppressWarnings("unused") + private String stringValue; + @SuppressWarnings("unused") + private Integer intValue; + @SuppressWarnings("unused") + private Float floatValue; + + @ProxyIgnore + private String ignoredString; + @ProxyIgnore + private int ignoredInt; + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/RegexBackslashManagerTest.java b/src/test/java/gov/nasa/ziggy/util/RegexBackslashManagerTest.java new file mode 100644 index 0000000..56efdf8 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/RegexBackslashManagerTest.java @@ -0,0 +1,20 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class RegexBackslashManagerTest { + + @Test + public void testToSingleBackslash() { + assertEquals("foo bar", RegexBackslashManager.toSingleBackslash("foo bar")); + assertEquals("foo\\bar", RegexBackslashManager.toSingleBackslash("foo\\bar")); + } + + @Test + public void testToDoubleBackslash() { + assertEquals("foo bar", RegexBackslashManager.toDoubleBackslash("foo bar")); + assertEquals("foo\\\\bar", RegexBackslashManager.toDoubleBackslash("foo\\bar")); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/RegexGroupCounterTest.java b/src/test/java/gov/nasa/ziggy/util/RegexGroupCounterTest.java new file mode 100644 index 0000000..391e0d8 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/RegexGroupCounterTest.java @@ -0,0 +1,17 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class RegexGroupCounterTest { + + @Test + public void testGroupCount() { + assertEquals(0, RegexGroupCounter.groupCount("foobar")); + assertEquals(1, RegexGroupCounter.groupCount("(foo)bar")); + assertEquals(2, RegexGroupCounter.groupCount("(foo)(bar)")); + // TODO Fix the method so the next test passes + // assertEquals(3, RegexGroupCounter.groupCount("before ((foo)(bar)) after")); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/StringUtilsTest.java b/src/test/java/gov/nasa/ziggy/util/StringUtilsTest.java index 07fea56..babd917 100644 --- a/src/test/java/gov/nasa/ziggy/util/StringUtilsTest.java +++ b/src/test/java/gov/nasa/ziggy/util/StringUtilsTest.java @@ -10,7 +10,7 @@ import org.junit.Test; /** - * Tests the {@link StringUtils} class. + * Tests the {@link ZiggyStringUtils} class. * * @author Bill Wohler * @author Forrest Girouard @@ -18,7 +18,7 @@ public class StringUtilsTest { @Test(expected = NullPointerException.class) public void testConstantToHyphenSeparatedLowercaseNull() { - StringUtils.constantToHyphenSeparatedLowercase(null); + ZiggyStringUtils.constantToHyphenSeparatedLowercase(null); } @Test @@ -41,186 +41,186 @@ public void testTrimListWhitespace() { } private void verifyTrimListWhitespace(String value, String expected) { - assertEquals(expected, StringUtils.trimListWhitespace(value)); + assertEquals(expected, ZiggyStringUtils.trimListWhitespace(value)); } @Test public void testConstantToHyphenSeparatedLowercase() { - assertEquals("", StringUtils.constantToHyphenSeparatedLowercase("")); - assertEquals("foo", StringUtils.constantToHyphenSeparatedLowercase("foo")); - assertEquals("foo-bar", StringUtils.constantToHyphenSeparatedLowercase("foo_bar")); - assertEquals("-foo-bar-", StringUtils.constantToHyphenSeparatedLowercase("_foo_bar_")); - assertEquals("foo", StringUtils.constantToHyphenSeparatedLowercase("FOO")); - assertEquals("foo-bar", StringUtils.constantToHyphenSeparatedLowercase("FOO_BAR")); - assertEquals("-foo-bar-", StringUtils.constantToHyphenSeparatedLowercase("_FOO_BAR_")); + assertEquals("", ZiggyStringUtils.constantToHyphenSeparatedLowercase("")); + assertEquals("foo", ZiggyStringUtils.constantToHyphenSeparatedLowercase("foo")); + assertEquals("foo-bar", ZiggyStringUtils.constantToHyphenSeparatedLowercase("foo_bar")); + assertEquals("-foo-bar-", ZiggyStringUtils.constantToHyphenSeparatedLowercase("_foo_bar_")); + assertEquals("foo", ZiggyStringUtils.constantToHyphenSeparatedLowercase("FOO")); + assertEquals("foo-bar", ZiggyStringUtils.constantToHyphenSeparatedLowercase("FOO_BAR")); + assertEquals("-foo-bar-", ZiggyStringUtils.constantToHyphenSeparatedLowercase("_FOO_BAR_")); } @Test public void testToHexString() { - String s = StringUtils.toHexString(new byte[0], 0, 0); + String s = ZiggyStringUtils.toHexString(new byte[0], 0, 0); assertEquals("", s); byte[] md5 = { (byte) 0xcd, (byte) 0xe1, (byte) 0xb9, (byte) 0x6c, (byte) 0x1b, (byte) 0x79, (byte) 0xfc, (byte) 0x62, (byte) 0x18, (byte) 0x55, (byte) 0x28, (byte) 0x3e, (byte) 0xae, (byte) 0x37, (byte) 0x0d, (byte) 0x0c }; assertEquals(16, md5.length); - s = StringUtils.toHexString(md5, 0, md5.length); + s = ZiggyStringUtils.toHexString(md5, 0, md5.length); assertEquals("cde1b96c1b79fc621855283eae370d0c", s); } @Test(expected = java.lang.IllegalArgumentException.class) public void testToHexStringBadLen() { - StringUtils.toHexString(new byte[2], 0, 3); + ZiggyStringUtils.toHexString(new byte[2], 0, 3); } @Test(expected = java.lang.IllegalArgumentException.class) public void testToHexStringBadOff() { - StringUtils.toHexString(new byte[2], 10, 1); + ZiggyStringUtils.toHexString(new byte[2], 10, 1); } @Test public void testTruncate() { - assertEquals(null, StringUtils.truncate(null, 10)); - assertSame("s", StringUtils.truncate("s", 10)); - assertEquals("012345", StringUtils.truncate("0123456789", 6)); + assertEquals(null, ZiggyStringUtils.truncate(null, 10)); + assertSame("s", ZiggyStringUtils.truncate("s", 10)); + assertEquals("012345", ZiggyStringUtils.truncate("0123456789", 6)); } @Test public void testConvertStringArray() { - String[] array = StringUtils.convertStringArray("a, b, c"); + String[] array = ZiggyStringUtils.convertStringArray("a, b, c"); assertArrayEquals(new String[] { "a", "b", "c" }, array); } @Test(expected = NullPointerException.class) public void testConvertStringArrayWithNullString() { - StringUtils.convertStringArray(null); + ZiggyStringUtils.convertStringArray(null); } @Test public void testConstantToAcronym() { - String acronym = StringUtils.constantToAcronym("FOO_BAR"); + String acronym = ZiggyStringUtils.constantToAcronym("FOO_BAR"); assertEquals("fb", acronym); } @Test public void testConstantToAcronymWithLeadingUnderscore() { - String acronym = StringUtils.constantToAcronym("_FOO_BAR"); + String acronym = ZiggyStringUtils.constantToAcronym("_FOO_BAR"); assertEquals("fb", acronym); } @Test public void testConstantToAcronymWithEmptyString() { - String acronym = StringUtils.constantToAcronym(""); + String acronym = ZiggyStringUtils.constantToAcronym(""); assertEquals("", acronym); } @Test(expected = NullPointerException.class) public void testConstantToAcronymWithNullString() { - StringUtils.constantToAcronym(null); + ZiggyStringUtils.constantToAcronym(null); } @Test public void testConstantToCamel() { - String camel = StringUtils.constantToCamel("FOO_BAR"); + String camel = ZiggyStringUtils.constantToCamel("FOO_BAR"); assertEquals("fooBar", camel); } @Test public void testConstantToCamelWithLeadingUnderscore() { - String camel = StringUtils.constantToCamel("_FOO_BAR"); + String camel = ZiggyStringUtils.constantToCamel("_FOO_BAR"); assertEquals("fooBar", camel); } @Test public void testConstantToCamelWithUnderscoreDigit() { - String camel = StringUtils.constantToCamel("FOO_BAR_1_2"); + String camel = ZiggyStringUtils.constantToCamel("FOO_BAR_1_2"); assertEquals("fooBar-1-2", camel); - camel = StringUtils.constantToCamel("FOO_BAR_1ABC_2"); + camel = ZiggyStringUtils.constantToCamel("FOO_BAR_1ABC_2"); assertEquals("fooBar-1abc-2", camel); - camel = StringUtils.constantToCamel("COVARIANCE_MATRIX_1_2"); + camel = ZiggyStringUtils.constantToCamel("COVARIANCE_MATRIX_1_2"); assertEquals("covarianceMatrix-1-2", camel); } @Test public void testConstantToCamelWithEmptyString() { - String camel = StringUtils.constantToCamel(""); + String camel = ZiggyStringUtils.constantToCamel(""); assertEquals("", camel); } @Test(expected = NullPointerException.class) public void testConstantToCamelWithNullString() { - StringUtils.constantToCamel(null); + ZiggyStringUtils.constantToCamel(null); } @Test public void testConstantToCamelWithSpaces() { - String camel = StringUtils.constantToCamelWithSpaces("FOO_BAR"); + String camel = ZiggyStringUtils.constantToCamelWithSpaces("FOO_BAR"); assertEquals("Foo Bar", camel); } @Test public void testConstantToCamelWithSpacesWithLeadingUnderscore() { - String camel = StringUtils.constantToCamelWithSpaces("_FOO_BAR"); + String camel = ZiggyStringUtils.constantToCamelWithSpaces("_FOO_BAR"); assertEquals("Foo Bar", camel); } @Test public void testConstantToCamelWithSpacesWithUnderscoreDigit() { - String camel = StringUtils.constantToCamelWithSpaces("FOO_BAR_1_2"); + String camel = ZiggyStringUtils.constantToCamelWithSpaces("FOO_BAR_1_2"); assertEquals("Foo Bar 1 2", camel); - camel = StringUtils.constantToCamelWithSpaces("FOO_BAR_1ABC_2"); + camel = ZiggyStringUtils.constantToCamelWithSpaces("FOO_BAR_1ABC_2"); assertEquals("Foo Bar 1abc 2", camel); } @Test public void testConstantToCamelWithSpacesWithEmptyString() { - String camel = StringUtils.constantToCamelWithSpaces(""); + String camel = ZiggyStringUtils.constantToCamelWithSpaces(""); assertEquals("", camel); } @Test(expected = NullPointerException.class) public void testConstantToCamelWithSpacesWithNullString() { - StringUtils.constantToCamelWithSpaces(null); + ZiggyStringUtils.constantToCamelWithSpaces(null); } @Test public void testConstantToSentenceWithSpacesWithLeadingUnderscore() { - String sentence = StringUtils.constantToSentenceWithSpaces("_FOO_BAR"); + String sentence = ZiggyStringUtils.constantToSentenceWithSpaces("_FOO_BAR"); assertEquals("Foo bar", sentence); } @Test public void testConstantToSentenceWithSpacesWithUnderscoreDigit() { - String sentence = StringUtils.constantToSentenceWithSpaces("FOO_BAR_1_2"); + String sentence = ZiggyStringUtils.constantToSentenceWithSpaces("FOO_BAR_1_2"); assertEquals("Foo bar 1 2", sentence); - sentence = StringUtils.constantToSentenceWithSpaces("FOO_BAR_1ABC_2"); + sentence = ZiggyStringUtils.constantToSentenceWithSpaces("FOO_BAR_1ABC_2"); assertEquals("Foo bar 1abc 2", sentence); } @Test public void testConstantToSentenceWithSpacesWithEmptyString() { - String sentence = StringUtils.constantToSentenceWithSpaces(""); + String sentence = ZiggyStringUtils.constantToSentenceWithSpaces(""); assertEquals("", sentence); } @Test(expected = NullPointerException.class) public void testConstantToSentenceWithSpacesWithNullString() { - StringUtils.constantToSentenceWithSpaces(null); + ZiggyStringUtils.constantToSentenceWithSpaces(null); } @Test public void testElapsedTime() { - String elapsedTime = StringUtils.elapsedTime(1000, 2000); + String elapsedTime = ZiggyStringUtils.elapsedTime(1000, 2000); assertEquals("00:00:01", elapsedTime); } @Test public void testElapsedTimeFromStartToCurrent() { - String elapsedTime = StringUtils.elapsedTime(1000, 0); + String elapsedTime = ZiggyStringUtils.elapsedTime(1000, 0); // The exact string is unknown, so just check that it is something large. assertTrue(elapsedTime.length() > 11); @@ -228,23 +228,23 @@ public void testElapsedTimeFromStartToCurrent() { @Test public void testElapsedTimeWithUninitializedStartTime() { - String elapsedTime = StringUtils.elapsedTime(0, 2000); + String elapsedTime = ZiggyStringUtils.elapsedTime(0, 2000); assertEquals("-", elapsedTime); } @Test public void testElapsedTimeWithDates() { - String elapsedTime = StringUtils.elapsedTime(new Date(1000), new Date(2000)); + String elapsedTime = ZiggyStringUtils.elapsedTime(new Date(1000), new Date(2000)); assertEquals("00:00:01", elapsedTime); } @Test(expected = NullPointerException.class) public void testElapsedTimeWithDatesWithNullStartTime() { - StringUtils.elapsedTime(null, new Date(2000)); + ZiggyStringUtils.elapsedTime(null, new Date(2000)); } @Test(expected = NullPointerException.class) public void testElapsedTimeWithDatesWithNullEndTime() { - StringUtils.elapsedTime(new Date(1000), null); + ZiggyStringUtils.elapsedTime(new Date(1000), null); } } diff --git a/src/test/java/gov/nasa/ziggy/util/SystemProxyTest.java b/src/test/java/gov/nasa/ziggy/util/SystemProxyTest.java new file mode 100644 index 0000000..08a17db --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/SystemProxyTest.java @@ -0,0 +1,26 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import org.junit.Test; + +public class SystemProxyTest { + + private static final long EARLY_IN_2024 = 1704391820427L; + + @Test + public void testCurrentTimeMillis() { + assertTrue(SystemProxy.currentTimeMillis() > EARLY_IN_2024); + + SystemProxy.setUserTime(EARLY_IN_2024); + assertEquals(EARLY_IN_2024, SystemProxy.currentTimeMillis()); + } + + @Test + public void testExit() { + SystemProxy.disableExit(); + SystemProxy.exit(42); + assertEquals(Integer.valueOf(42), SystemProxy.getLatestExitCode()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/TaskProcessingTimeStatsTest.java b/src/test/java/gov/nasa/ziggy/util/TaskProcessingTimeStatsTest.java new file mode 100644 index 0000000..f8f1606 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/TaskProcessingTimeStatsTest.java @@ -0,0 +1,51 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import java.util.Date; +import java.util.List; + +import org.junit.Test; + +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTask.State; + +public class TaskProcessingTimeStatsTest { + + private static final long START_MILLIS = 1700000000000L; + private static final long HOUR_MILLIS = 60 * 60 * 1000; + + @Test + public void test() { + TaskProcessingTimeStats taskProcessingTimeStats = TaskProcessingTimeStats + .of(pipelineTasks()); + assertEquals(3, taskProcessingTimeStats.getCount()); + assertEquals(2.0, taskProcessingTimeStats.getMax(), 0.0001); + assertEquals(new Date(START_MILLIS + 5 * HOUR_MILLIS), taskProcessingTimeStats.getMaxEnd()); + assertEquals(1.0, taskProcessingTimeStats.getMean(), 0.0001); + assertEquals(0.0, taskProcessingTimeStats.getMin(), 0.0001); + assertEquals(new Date(START_MILLIS), taskProcessingTimeStats.getMinStart()); + // TODO Calculator.net says stddev of 0, 1, 2 is 0.81649658092773, not 1.0 + assertEquals(1.0, taskProcessingTimeStats.getStddev(), 0.0001); + assertEquals(3.0, taskProcessingTimeStats.getSum(), 0.0001); + assertEquals(5.0, taskProcessingTimeStats.getTotalElapsed(), 0.0001); + } + + private List pipelineTasks() { + // The first task took two hours and the second task started after the first and took one + // hour. + return List.of( + pipelineTask("module1", new Date(START_MILLIS), + new Date(START_MILLIS + 2 * HOUR_MILLIS), State.COMPLETED), + pipelineTask("module2", new Date(START_MILLIS + 4 * HOUR_MILLIS), + new Date(START_MILLIS + 5 * HOUR_MILLIS), State.COMPLETED), + pipelineTask("module3", new Date(0), new Date(0), State.SUBMITTED)); + } + + private PipelineTask pipelineTask(String moduleName, Date start, Date end, State state) { + PipelineTask pipelineTask = new PipelineTask(); + pipelineTask.setStartProcessingTime(start); + pipelineTask.setEndProcessingTime(end); + return pipelineTask; + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/TasksStatesTest.java b/src/test/java/gov/nasa/ziggy/util/TasksStatesTest.java new file mode 100644 index 0000000..4de758a --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/TasksStatesTest.java @@ -0,0 +1,174 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.junit.Test; + +import gov.nasa.ziggy.pipeline.definition.PipelineInstanceNode; +import gov.nasa.ziggy.pipeline.definition.PipelineModuleDefinition; +import gov.nasa.ziggy.pipeline.definition.PipelineTask; +import gov.nasa.ziggy.pipeline.definition.PipelineTask.ProcessingSummary; +import gov.nasa.ziggy.pipeline.definition.PipelineTask.State; +import gov.nasa.ziggy.util.TasksStates.Summary; + +public class TasksStatesTest { + + @Test + public void testUpdate() { + TasksStates tasksStates = new TasksStates(); + assertEquals(new HashMap<>(), tasksStates.getModuleStates()); + + tasksStates.update(pipelineTasks(), taskAttributes()); + testModuleStates(tasksStates); + } + + @Test + public void testGetModuleStates() { + TasksStates tasksStates = new TasksStates(); + assertEquals(new HashMap<>(), tasksStates.getModuleStates()); + + tasksStates = new TasksStates(pipelineTasks(), taskAttributes()); + testModuleStates(tasksStates); + } + + @Test + public void testGetModuleNames() { + TasksStates tasksStates = new TasksStates(); + assertEquals(new ArrayList<>(), tasksStates.getModuleNames()); + + tasksStates = new TasksStates(pipelineTasks(), taskAttributes()); + assertEquals(5, tasksStates.getModuleNames().size()); + assertEquals("module1", tasksStates.getModuleNames().get(0)); + assertEquals("module2", tasksStates.getModuleNames().get(1)); + assertEquals("module3", tasksStates.getModuleNames().get(2)); + assertEquals("module4", tasksStates.getModuleNames().get(3)); + assertEquals("module5", tasksStates.getModuleNames().get(4)); + } + + @Test + public void testGetTotalSubmittedCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalSubmittedCount()); + } + + @Test + public void testGetTotalProcessingCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalProcessingCount()); + } + + @Test + public void testGetTotalErrorCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalErrorCount()); + } + + @Test + public void testGetTotalCompletedCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalCompletedCount()); + } + + @Test + public void testGetTotalSubTaskTotalCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalSubTaskTotalCount()); + } + + @Test + public void testGetTotalSubTaskCompleteCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalSubTaskCompleteCount()); + } + + @Test + public void testGetTotalSubTaskFailedCount() { + TasksStates tasksStates = new TasksStates(); + assertEquals(0, tasksStates.getTotalSubTaskFailedCount()); + } + + private List pipelineTasks() { + return List.of(pipelineTask("module1", 1L, 10, State.INITIALIZED), + pipelineTask("module2", 2L, 20, State.SUBMITTED), + pipelineTask("module3", 3L, 30, State.PROCESSING), + pipelineTask("module4", 4L, 40, State.ERROR), + pipelineTask("module5", 5L, 50, State.COMPLETED)); + } + + private PipelineTask pipelineTask(String moduleName, Long id, int attributeSeed, State state) { + PipelineTask pipelineTask = new PipelineTask(); + PipelineInstanceNode pipelineInstanceNode = new PipelineInstanceNode(); + pipelineInstanceNode.setPipelineModuleDefinition(new PipelineModuleDefinition(moduleName)); + pipelineTask.setPipelineInstanceNode(pipelineInstanceNode); + pipelineTask.setId(id); + pipelineTask.setTotalSubtaskCount(attributeSeed); + pipelineTask.setCompletedSubtaskCount(attributeSeed - (int) (0.1 * attributeSeed)); + pipelineTask.setFailedSubtaskCount((int) (0.1 * attributeSeed)); + pipelineTask.setState(state); + return pipelineTask; + } + + private Map taskAttributes() { + List pipelineTasks = pipelineTasks(); + return Map.of(pipelineTasks.get(0).getId(), new ProcessingSummary(pipelineTasks.get(0)), + pipelineTasks.get(1).getId(), new ProcessingSummary(pipelineTasks.get(1)), + pipelineTasks.get(2).getId(), new ProcessingSummary(pipelineTasks.get(2)), + pipelineTasks.get(3).getId(), new ProcessingSummary(pipelineTasks.get(3)), + pipelineTasks.get(4).getId(), new ProcessingSummary(pipelineTasks.get(4))); + } + + private void testModuleStates(TasksStates tasksStates) { + Map moduleStates = tasksStates.getModuleStates(); + assertEquals(5, moduleStates.size()); + + Summary summary = moduleStates.get("module1"); + assertEquals(0, summary.getCompletedCount()); + assertEquals(0, summary.getErrorCount()); + assertEquals(0, summary.getProcessingCount()); + assertEquals(0, summary.getSubmittedCount()); + assertEquals(9, summary.getSubTaskCompleteCount()); + assertEquals(1, summary.getSubTaskFailedCount()); + assertEquals(10, summary.getSubTaskTotalCount()); + + summary = moduleStates.get("module2"); + assertEquals(0, summary.getCompletedCount()); + assertEquals(0, summary.getErrorCount()); + assertEquals(0, summary.getProcessingCount()); + assertEquals(1, summary.getSubmittedCount()); + assertEquals(18, summary.getSubTaskCompleteCount()); + assertEquals(2, summary.getSubTaskFailedCount()); + assertEquals(20, summary.getSubTaskTotalCount()); + + summary = moduleStates.get("module3"); + assertEquals(0, summary.getCompletedCount()); + assertEquals(0, summary.getErrorCount()); + assertEquals(1, summary.getProcessingCount()); + assertEquals(0, summary.getSubmittedCount()); + assertEquals(27, summary.getSubTaskCompleteCount()); + assertEquals(3, summary.getSubTaskFailedCount()); + assertEquals(30, summary.getSubTaskTotalCount()); + + summary = moduleStates.get("module4"); + assertEquals(0, summary.getCompletedCount()); + assertEquals(1, summary.getErrorCount()); + assertEquals(0, summary.getProcessingCount()); + assertEquals(0, summary.getSubmittedCount()); + assertEquals(36, summary.getSubTaskCompleteCount()); + assertEquals(4, summary.getSubTaskFailedCount()); + assertEquals(40, summary.getSubTaskTotalCount()); + + summary = moduleStates.get("module5"); + assertEquals(1, summary.getCompletedCount()); + assertEquals(0, summary.getErrorCount()); + assertEquals(0, summary.getProcessingCount()); + assertEquals(0, summary.getSubmittedCount()); + assertEquals(45, summary.getSubTaskCompleteCount()); + assertEquals(5, summary.getSubTaskFailedCount()); + assertEquals(50, summary.getSubTaskTotalCount()); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/TimeFormatterTest.java b/src/test/java/gov/nasa/ziggy/util/TimeFormatterTest.java new file mode 100644 index 0000000..89eb8af --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/TimeFormatterTest.java @@ -0,0 +1,93 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +public class TimeFormatterTest { + + private static final double TIME_HOURS = 12.0 + 34.0 / 60 + 56.0 / 3600; + private static final int TIME_SECONDS = 12 * 3600 + 34 * 60 + 56; + private static final String TIME_STRING = "12:34:56"; + private static final String TIME_STRING_NO_SECONDS = "12:34"; + + private static final int ZERO_TIME = 0; + private static final String ZERO_TIME_STRING = "0:00:00"; + private static final String ZERO_TIME_STRING_NO_SECONDS = "0:00"; + + @Test(expected = NullPointerException.class) + public void testNullTimeStringHhMmSsToTimeInHours() { + TimeFormatter.timeStringHhMmSsToTimeInHours(null); + } + + @Test(expected = IllegalArgumentException.class) + public void testEmptyTimeStringHhMmSsToTimeInHours() { + TimeFormatter.timeStringHhMmSsToTimeInHours(""); + } + + @Test + public void testTimeStringHhMmSsToTimeInHours() { + assertEquals(TIME_HOURS, TimeFormatter.timeStringHhMmSsToTimeInHours(TIME_STRING), 0.0001); + assertEquals(ZERO_TIME, TimeFormatter.timeStringHhMmSsToTimeInHours(ZERO_TIME_STRING), + 0.0001); + } + + @Test(expected = NullPointerException.class) + public void testNullTimeStringHhMmSsToTimeInSeconds() { + TimeFormatter.timeStringHhMmSsToTimeInSeconds(null); + } + + @Test(expected = IllegalArgumentException.class) + public void testEmptyTimeStringHhMmSsToTimeInSeconds() { + TimeFormatter.timeStringHhMmSsToTimeInSeconds(""); + } + + @Test + public void testTimeStringHhMmSsToTimeInSeconds() { + assertEquals(TIME_SECONDS, TimeFormatter.timeStringHhMmSsToTimeInSeconds(TIME_STRING), + 0.0001); + assertEquals(ZERO_TIME, TimeFormatter.timeStringHhMmSsToTimeInSeconds(ZERO_TIME_STRING), + 0.0001); + } + + @Test + public void testTimeInHoursToStringHhMmSs() { + assertEquals(TIME_STRING, TimeFormatter.timeInHoursToStringHhMmSs(TIME_HOURS)); + assertEquals(ZERO_TIME_STRING, TimeFormatter.timeInHoursToStringHhMmSs(ZERO_TIME)); + } + + @Test(expected = IllegalArgumentException.class) + public void testNegativeTimeInHoursToStringHhMmSs() { + assertEquals(TIME_STRING, TimeFormatter.timeInHoursToStringHhMmSs(-1.5)); + } + + @Test + public void testTimeInSecondsToStringHhMmSs() { + assertEquals(TIME_STRING, TimeFormatter.timeInSecondsToStringHhMmSs(TIME_SECONDS)); + assertEquals(ZERO_TIME_STRING, TimeFormatter.timeInSecondsToStringHhMmSs(ZERO_TIME)); + } + + @Test(expected = IllegalArgumentException.class) + public void testNegativeTimeInSecondsToStringHhMmSs() { + assertEquals(TIME_STRING, TimeFormatter.timeInSecondsToStringHhMmSs(-3661)); + } + + @Test(expected = NullPointerException.class) + public void testNullStripSeconds() { + TimeFormatter.stripSeconds(null); + } + + @Test(expected = IllegalArgumentException.class) + public void testEmptyStripSeconds() { + TimeFormatter.stripSeconds(""); + } + + @Test + public void testStripSeconds() { + assertEquals(TIME_STRING_NO_SECONDS, TimeFormatter.stripSeconds(TIME_STRING)); + assertEquals(TIME_STRING_NO_SECONDS, TimeFormatter.stripSeconds(TIME_STRING_NO_SECONDS)); + assertEquals(ZERO_TIME_STRING_NO_SECONDS, TimeFormatter.stripSeconds(ZERO_TIME_STRING)); + assertEquals(ZERO_TIME_STRING_NO_SECONDS, + TimeFormatter.stripSeconds(ZERO_TIME_STRING_NO_SECONDS)); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/TimeRangeTest.java b/src/test/java/gov/nasa/ziggy/util/TimeRangeTest.java new file mode 100644 index 0000000..e62e85f --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/TimeRangeTest.java @@ -0,0 +1,52 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; + +import java.util.Date; + +import org.junit.Test; + +public class TimeRangeTest { + + private static final long HOUR_MILLISECONDS = 60 * 60 * 1000; + + @SuppressWarnings("unlikely-arg-type") + @Test + public void testHashCodeEquals() { + TimeRange timeRange = timeRange(12345); + assertTrue(timeRange.equals(timeRange)); + assertFalse(timeRange.equals(null)); + assertFalse(timeRange.equals("a string")); + + assertTrue(timeRange(12345).equals(timeRange(12345))); + assertFalse(timeRange(12345).equals(timeRange(54321))); + + assertEquals(timeRange(12345).hashCode(), timeRange(12345).hashCode()); + assertNotEquals(timeRange(12345).hashCode(), timeRange(54321).hashCode()); + } + + @Test + public void testGetStartTimestamp() { + assertEquals(new Date(12345), timeRange(12345).getStartTimestamp()); + assertNotEquals(new Date(54321), timeRange(12345).getStartTimestamp()); + + assertNull(new TimeRange(null, null).getStartTimestamp()); + } + + @Test + public void testGetEndTimestamp() { + assertEquals(new Date(12345 + HOUR_MILLISECONDS), timeRange(12345).getEndTimestamp()); + assertNotEquals(new Date(54321 + HOUR_MILLISECONDS), timeRange(12345).getEndTimestamp()); + + assertNull(new TimeRange(null, null).getEndTimestamp()); + } + + /** Returns a range from the startSeed to one hour after. */ + private TimeRange timeRange(long startSeed) { + return new TimeRange(new Date(startSeed), new Date(startSeed + HOUR_MILLISECONDS)); + } +} diff --git a/src/test/java/gov/nasa/ziggy/util/WrapperUtilsTest.java b/src/test/java/gov/nasa/ziggy/util/WrapperUtilsTest.java new file mode 100644 index 0000000..5ee2e64 --- /dev/null +++ b/src/test/java/gov/nasa/ziggy/util/WrapperUtilsTest.java @@ -0,0 +1,76 @@ +package gov.nasa.ziggy.util; + +import static org.junit.Assert.assertEquals; + +import org.junit.Test; + +import gov.nasa.ziggy.util.WrapperUtils.WrapperCommand; + +public class WrapperUtilsTest { + + @Test + public void testWrapperParameter() { + assertEquals("wrapper.app.parameter=-Dfoo", + WrapperUtils.wrapperParameter("wrapper.app.parameter", "-Dfoo")); + } + + @Test(expected = NullPointerException.class) + public void testWrapperParameterNullProp() { + WrapperUtils.wrapperParameter(null, "-Dfoo"); + } + + @Test(expected = IllegalArgumentException.class) + public void testWrapperParameterEmptyProp() { + WrapperUtils.wrapperParameter("", "-Dfoo"); + } + + @Test(expected = NullPointerException.class) + public void testWrapperParameterNullValue() { + WrapperUtils.wrapperParameter("wrapper.app.parameter", null); + } + + @Test(expected = IllegalArgumentException.class) + public void testWrapperParameterEmptyValue() { + WrapperUtils.wrapperParameter("wrapper.app.parameter", ""); + } + + @Test + public void testIndexedWrapperParameter() { + assertEquals("wrapper.app.parameter.0=-Dfoo", + WrapperUtils.wrapperParameter("wrapper.app.parameter.", 0, "-Dfoo")); + assertEquals("wrapper.app.parameter.1=-Dfoo", + WrapperUtils.wrapperParameter("wrapper.app.parameter.", 1, "-Dfoo")); + } + + @Test(expected = NullPointerException.class) + public void testIndexedWrapperParameterNullProp() { + WrapperUtils.wrapperParameter(null, 1, "-Dfoo"); + } + + @Test(expected = IllegalArgumentException.class) + public void testIndexedWrapperParameterEmptyProp() { + WrapperUtils.wrapperParameter("", 1, "-Dfoo"); + } + + @Test(expected = NullPointerException.class) + public void testIndexedWrapperParameterNullValue() { + WrapperUtils.wrapperParameter("wrapper.app.parameter", 1, null); + } + + @Test(expected = IllegalArgumentException.class) + public void testIndexedWrapperParameterEmptyValue() { + WrapperUtils.wrapperParameter("wrapper.app.parameter", 1, ""); + } + + @Test(expected = IllegalArgumentException.class) + public void testIndexedWrapperParameterNegativeIndex() { + WrapperUtils.wrapperParameter("wrapper.app.parameter", -1, "-Dfoo"); + } + + @Test + public void testWrapperCommandEnum() { + assertEquals("start", WrapperCommand.START.toString()); + assertEquals("stop", WrapperCommand.STOP.toString()); + assertEquals("status", WrapperCommand.STATUS.toString()); + } +} diff --git a/test/data/EventPipeline/pd-event.xml b/test/data/EventPipeline/pd-event.xml index 2486fce..451a13d 100644 --- a/test/data/EventPipeline/pd-event.xml +++ b/test/data/EventPipeline/pd-event.xml @@ -1,31 +1,30 @@ - + - - - - - - - - - - - - - + + + + + + + + + + + + diff --git a/test/data/EventPipeline/pe-test.xml b/test/data/EventPipeline/pe-test.xml index a6ad507..5d1a18e 100644 --- a/test/data/EventPipeline/pe-test.xml +++ b/test/data/EventPipeline/pe-test.xml @@ -2,10 +2,10 @@ - - - + - \ No newline at end of file + + + diff --git a/test/data/EventPipeline/pl-event-override.xml b/test/data/EventPipeline/pl-event-override.xml index 88f3207..219bb88 100644 --- a/test/data/EventPipeline/pl-event-override.xml +++ b/test/data/EventPipeline/pl-event-override.xml @@ -1,8 +1,8 @@ - - - - - \ No newline at end of file + + + + + diff --git a/test/data/EventPipeline/pl-event.xml b/test/data/EventPipeline/pl-event.xml index b0a45aa..15f6b38 100644 --- a/test/data/EventPipeline/pl-event.xml +++ b/test/data/EventPipeline/pl-event.xml @@ -1,83 +1,23 @@ - + - - - - - - - - - - + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/test/data/EventPipeline/pt-event.xml b/test/data/EventPipeline/pt-event.xml index 238a954..e17b188 100644 --- a/test/data/EventPipeline/pt-event.xml +++ b/test/data/EventPipeline/pt-event.xml @@ -1,54 +1,48 @@ - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file +data types, the directory structure of the datastore is implicitly defined as +well. --> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/test/data/classwrapper/pd-two-default-param-sets.xml b/test/data/classwrapper/pd-two-default-param-sets.xml index 4387231..c77b7ff 100644 --- a/test/data/classwrapper/pd-two-default-param-sets.xml +++ b/test/data/classwrapper/pd-two-default-param-sets.xml @@ -3,14 +3,14 @@ + pipelineModuleClass="gov.nasa.ziggy.pipeline.definition.crud.PipelineTaskCrudTest$TestModule" + exeTimeoutSecs="2000000" minMemoryMegabytes="0" + uowGenerator="gov.nasa.ziggy.uow.SingleUnitOfWorkGenerator"/> - + - + diff --git a/test/data/classwrapper/pl-two-default-param-sets.xml b/test/data/classwrapper/pl-two-default-param-sets.xml index a66279d..bffcb1b 100644 --- a/test/data/classwrapper/pl-two-default-param-sets.xml +++ b/test/data/classwrapper/pl-two-default-param-sets.xml @@ -1,17 +1,17 @@ - - - - - - - - - - - - + + + + + + + + + + + + diff --git a/test/data/configuration/invalid-pipeline-definition.xml b/test/data/configuration/invalid-pipeline-definition.xml index 1b903f4..35ebaed 100644 --- a/test/data/configuration/invalid-pipeline-definition.xml +++ b/test/data/configuration/invalid-pipeline-definition.xml @@ -2,51 +2,51 @@ - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/test/data/configuration/mixed-pipeline-config.xml b/test/data/configuration/mixed-pipeline-config.xml index 96388c4..da315f8 100644 --- a/test/data/configuration/mixed-pipeline-config.xml +++ b/test/data/configuration/mixed-pipeline-config.xml @@ -1,21 +1,21 @@ + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/pipeline/definition file:../../../schema/xml/pipeline-definition.xsd"> - + - + - + diff --git a/test/data/configuration/module1.xml b/test/data/configuration/module1.xml index 034f81e..589d2a7 100644 --- a/test/data/configuration/module1.xml +++ b/test/data/configuration/module1.xml @@ -1,2 +1,2 @@ - + diff --git a/test/data/configuration/module2.xml b/test/data/configuration/module2.xml index dd23822..1907d3a 100644 --- a/test/data/configuration/module2.xml +++ b/test/data/configuration/module2.xml @@ -1,6 +1,6 @@ - + diff --git a/test/data/configuration/node.xml b/test/data/configuration/node.xml index 848e2ee..dbcdf82 100644 --- a/test/data/configuration/node.xml +++ b/test/data/configuration/node.xml @@ -1,8 +1,9 @@ - - - - - + + + + + + diff --git a/test/data/configuration/pd-hyperion.xml b/test/data/configuration/pd-hyperion.xml index 44b5cb7..399cb71 100644 --- a/test/data/configuration/pd-hyperion.xml +++ b/test/data/configuration/pd-hyperion.xml @@ -33,8 +33,8 @@ + exeTimeoutSecs="2000000" minMemoryMegabytes="0" /> + exeTimeoutSecs="2000000" minMemoryMegabytes="0" /> diff --git a/test/data/configuration/pe-test.xml b/test/data/configuration/pe-test.xml index a6ad507..5d1a18e 100644 --- a/test/data/configuration/pe-test.xml +++ b/test/data/configuration/pe-test.xml @@ -2,10 +2,10 @@ - - - + - \ No newline at end of file + + + diff --git a/test/data/configuration/pipeline-bad-xml.xml b/test/data/configuration/pipeline-bad-xml.xml index ac65aa4..263983d 100644 --- a/test/data/configuration/pipeline-bad-xml.xml +++ b/test/data/configuration/pipeline-bad-xml.xml @@ -1,11 +1,11 @@ + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/pipeline/definition file:../../../schema/xml/pipeline-definition.xsd"> - + - - + + diff --git a/test/data/configuration/pipeline-definition.xml b/test/data/configuration/pipeline-definition.xml index 0eeab0c..438557e 100644 --- a/test/data/configuration/pipeline-definition.xml +++ b/test/data/configuration/pipeline-definition.xml @@ -1,30 +1,32 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/test/data/configuration/pipeline-does-not-match-schema.xml b/test/data/configuration/pipeline-does-not-match-schema.xml index bfb0f5d..58324a2 100644 --- a/test/data/configuration/pipeline-does-not-match-schema.xml +++ b/test/data/configuration/pipeline-does-not-match-schema.xml @@ -1,11 +1,11 @@ + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/pipeline/definition file:../../../schema/xml/pipeline-definition.xsd"> - - + + - + diff --git a/test/data/configuration/pl-bad-xml.xml b/test/data/configuration/pl-bad-xml.xml index 0e41be4..62b3fc9 100644 --- a/test/data/configuration/pl-bad-xml.xml +++ b/test/data/configuration/pl-bad-xml.xml @@ -1,11 +1,11 @@ - + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/parameters file:../../../schema/xml/parameter-library.xsd"> + - \ No newline at end of file + diff --git a/test/data/configuration/pl-does-not-match-schema.xml b/test/data/configuration/pl-does-not-match-schema.xml index 3cdb9e6..6550d56 100644 --- a/test/data/configuration/pl-does-not-match-schema.xml +++ b/test/data/configuration/pl-does-not-match-schema.xml @@ -1,12 +1,11 @@ - + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/parameters file:../../../schema/xml/parameter-library.xsd"> + - \ No newline at end of file diff --git a/test/data/configuration/pl-sample.xml b/test/data/configuration/pl-sample.xml index 56264fc..c2d76bc 100644 --- a/test/data/configuration/pl-sample.xml +++ b/test/data/configuration/pl-sample.xml @@ -1,14 +1,13 @@ - + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/parameters file:../../../schema/xml/parameter-library.xsd"> + - + - \ No newline at end of file diff --git a/test/data/configuration/pt-hyperion.xml b/test/data/configuration/pt-hyperion.xml index 0f8cc02..558d117 100644 --- a/test/data/configuration/pt-hyperion.xml +++ b/test/data/configuration/pt-hyperion.xml @@ -1,27 +1,27 @@ - - - - - - - - - - - - - - + + + - \ No newline at end of file + + + + + + + + + + + + diff --git a/test/data/configuration/single-module.xml b/test/data/configuration/single-module.xml index 39f58bc..74bb7a3 100644 --- a/test/data/configuration/single-module.xml +++ b/test/data/configuration/single-module.xml @@ -1,18 +1,18 @@ + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/pipeline/definition file:../../../schema/xml/pipeline-definition.xsd"> - - + + - + diff --git a/test/data/configuration/single-pipeline.xml b/test/data/configuration/single-pipeline.xml index 8e2391b..4dc290c 100644 --- a/test/data/configuration/single-pipeline.xml +++ b/test/data/configuration/single-pipeline.xml @@ -1,11 +1,11 @@ + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://ziggy.nasa.gov/pipeline/definition file:../../../schema/xml/pipeline-definition.xsd"> - + - + diff --git a/test/data/datastore/datastore-update.xml b/test/data/datastore/datastore-update.xml new file mode 100644 index 0000000..0a4a18c --- /dev/null +++ b/test/data/datastore/datastore-update.xml @@ -0,0 +1,26 @@ + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/test/data/datastore/pd-test-1.xml b/test/data/datastore/pd-test-1.xml index 3ef6984..862b21a 100644 --- a/test/data/datastore/pd-test-1.xml +++ b/test/data/datastore/pd-test-1.xml @@ -1,30 +1,39 @@ - - - - - - - - - - - - - \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/test/data/datastore/pd-test-2.xml b/test/data/datastore/pd-test-2.xml index 9330a5a..067aad6 100644 --- a/test/data/datastore/pd-test-2.xml +++ b/test/data/datastore/pd-test-2.xml @@ -1,8 +1,7 @@ - + - \ No newline at end of file + diff --git a/test/data/datastore/pd-test-invalid-type.xml b/test/data/datastore/pd-test-invalid-type.xml index 7e0e57b..02c54ce 100644 --- a/test/data/datastore/pd-test-invalid-type.xml +++ b/test/data/datastore/pd-test-invalid-type.xml @@ -1,10 +1,8 @@ + - - - - \ No newline at end of file + diff --git a/test/data/datastore/pd-test-invalid-xml.xml b/test/data/datastore/pd-test-invalid-xml.xml index 8f120d7..2ac7ea8 100644 --- a/test/data/datastore/pd-test-invalid-xml.xml +++ b/test/data/datastore/pd-test-invalid-xml.xml @@ -1,7 +1,7 @@ - \ No newline at end of file + diff --git a/test/data/paramlib/params-mismatch.xml b/test/data/paramlib/params-mismatch.xml index 2c526ca..7fda87a 100644 --- a/test/data/paramlib/params-mismatch.xml +++ b/test/data/paramlib/params-mismatch.xml @@ -1,8 +1,8 @@ - - - + + + diff --git a/test/data/paramlib/pl-hyperion.xml b/test/data/paramlib/pl-hyperion.xml index 8de04bb..346c008 100644 --- a/test/data/paramlib/pl-hyperion.xml +++ b/test/data/paramlib/pl-hyperion.xml @@ -1,47 +1,47 @@ - - - - - - - - - + + + + + + + + + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - diff --git a/test/data/paramlib/pl-override-bad-type.xml b/test/data/paramlib/pl-override-bad-type.xml index d069fed..fe682de 100644 --- a/test/data/paramlib/pl-override-bad-type.xml +++ b/test/data/paramlib/pl-override-bad-type.xml @@ -1,7 +1,7 @@ - - - + + + - \ No newline at end of file + diff --git a/test/data/paramlib/pl-override-mismatch.xml b/test/data/paramlib/pl-override-mismatch.xml index 16bb153..23d55d7 100644 --- a/test/data/paramlib/pl-override-mismatch.xml +++ b/test/data/paramlib/pl-override-mismatch.xml @@ -1,7 +1,7 @@ - - - + + + - \ No newline at end of file + diff --git a/test/data/paramlib/pl-override-new-param-set.xml b/test/data/paramlib/pl-override-new-param-set.xml index 3c3e2e7..9e059cf 100644 --- a/test/data/paramlib/pl-override-new-param-set.xml +++ b/test/data/paramlib/pl-override-new-param-set.xml @@ -1,7 +1,7 @@ - - - + + + - \ No newline at end of file + diff --git a/test/data/paramlib/pl-overrides.xml b/test/data/paramlib/pl-overrides.xml index 2cc1d66..63bf0db 100644 --- a/test/data/paramlib/pl-overrides.xml +++ b/test/data/paramlib/pl-overrides.xml @@ -1,11 +1,11 @@ - - - - - - - + + + + + + + diff --git a/test/data/paramlib/pl-replacement-param-sets.xml b/test/data/paramlib/pl-replacement-param-sets.xml index e5e66b4..c5a1b8a 100644 --- a/test/data/paramlib/pl-replacement-param-sets.xml +++ b/test/data/paramlib/pl-replacement-param-sets.xml @@ -1,22 +1,17 @@ - - - + + + - - - - - - - - - + + + + + - - - + + + diff --git a/test/data/paramlib/test.xml b/test/data/paramlib/test.xml index de2a320..5a6950f 100644 --- a/test/data/paramlib/test.xml +++ b/test/data/paramlib/test.xml @@ -1,33 +1,23 @@ - - - - - - - + + + + + - - - - - - - - - - - + + + + + - - - - - - - + + + + + + +