Validate fails to read files on Windows systems after Internal Reference check updates (#308)

[jennifergward]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

For a CSV containing multiple tables, validate 3.1.1 reports the following error on a table that we think is properly defined in the label:

ERROR  [error.label.invalid_object_definition]   line 2: Invalid object definition for object #2.  The previously defined object ends at byte 354958. Offset defined in label: 25
ERROR  [error.label.invalid_object_definition]   line 3: Invalid object definition for object #3.  The previously defined object ends at byte 354958. Offset defined in label: 7751  
Note: The PDS4Viewer opens this file with no issues.

We tried to check with validate 3.4.1 to see if this issue has been fixed already, but we get another error:

FAIL: file:/X:/staff-folders/ward/clipper/RSS001E01_2031066T0241_EURGRVL20XXXXXXGSFC_COV010.xml
      ERROR  [error.validation.internal_error]   Uncaught exception while validating: null

🕵️ Expected behavior

We think the CSV and XML are valid and validate shouldn't be reporting any errors.

📜 To Reproduce

validate RSS001E01_2031066T0241_EURGRVL20XXXXXXGSFC_COV010.xml -R pds4.label -r report.txt
(note: you will get errors due to the absence of the Clipper Mission dictionary which isn't online yet. I can send a copy if helpful.)

🖥 Environment Info

• Version of this software: v3.1.1, v3.4.1
• Operating System: Windows

📚 Version of Software Used

Validate v3.1.1 and Validate v3.4.1

🩺 Test Data / Additional context

validate_bug.zip

🦄 Related requirements

🦄 #308

User also indicated seeing this on the discussion board: #811

⚙️ Engineering Details

After some investigation and testing on Windows, we started getting these errors on Windows when trying to resolve a path:

Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 2: 
...

which can be traced to https://github.com/NASA-PDS/validate/blob/main/src/main/java/gov/nasa/pds/tools/validate/CrossLabelFileAreaReferenceChecker.java#L19

Per https://stackoverflow.com/questions/43972777/java-nio-file-invalidpathexception-illegal-char-at-index-2, on Windows, we need to translate the URL to a URI vs. a path.

I&T

TestRail Test ID: T8681197

[jordanpadams]

@jennifergward it doesn't look like the CLIPPER SCH/XSD are posted online yet. Can you provide those as well?

[jordanpadams]

@jennifergward for v3.4.1, if I remove the CLIPPER references, and validate, it appears to execute successfully:

~/test/validate-3.4.1/bin/validate -t ~/Downloads/validate_bug/RSS001E01_2031066T0241_EURGRVL20XXXXXXGSFC_COV010.xml

My guess is it may have to do with your catalog file. Can you also supply that?

[jennifergward]

When I remove the Clipper references, I get the attached error in report_noclipper.txt. I'm using your same command above with version 3.4.1.

Attached is my catalog file and also the Clipper dictionary. Thanks for the help!
attachments.zip

[jordanpadams]

@jennifergward can you confirm you have tested this with the latest v3.4.1 of validate? I am having difficulty replicating this error.

[jennifergward]

@jordanpadams Confirmed. We also tested v3.4.1 on other files and bundles, and confirmed that this issue is bigger than the EC file. Attached are validate reports after running 3.1.1 vs. 3.4.1 on a bundle. We're thinking this is a Windows issue.
arcdr_bundle_3-1-1.txt
arcdr_bundle_3-4-1.txt

[jordanpadams]

Thanks @jennifergward we will take a look.

[jordanpadams]

@jennifergward I have a beat on a fix. Just awaiting a review from someone on the team and we should be good to go. Will tag and release with the update as soon as we have it.

#818

[jordanpadams]

@jennifergward can you try out the latest SNAPSHOT of validate and let me know if it works? https://github.com/NASA-PDS/validate/releases/tag/v3.5.0-SNAPSHOT

[jennifergward]

@jordanpadams It still doesn't work.

validate_report_240212.txt

[jordanpadams]

@jennifergward copy! Sorry about that. I wasn't running it exactly as you were. But looks like a different error, which is great! :-)

[jordanpadams]

@jennifergward what version of java are you running?

java -version

[jennifergward]

@jordanpadams
Looks like the machine I had just used has a pretty old version of java. This one produces the errors I sent this morning.
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

Using a different machine with a newer version produces the old errors from my initial ticket
(validate_report_240212_java11.0.12.txt)
openjdk version "11.0.12" 2021-07-20 LTS
OpenJDK Runtime Environment Zulu11.50+19-CA (build 11.0.12+7-LTS)
OpenJDK 64-Bit Server VM Zulu11.50+19-CA (build 11.0.12+7-LTS, mixed mode)
I've been told by various sources that this error isn't seen when a non-Windows machine is used.

[jordanpadams]

@jennifergward ok. I have been testing all of this on a windows virtual machine and it works for some reason. I will revisit in the morning and see if there is something I’m missing.

[jennifergward]

Looks like I may have goofed. I see validate version 3.1.1 in this last validation report. I'll double-check in the morning.

…

________________________________ From: Jordan Padams ***@***.***> Sent: Monday, February 12, 2024 6:32:50 PM To: NASA-PDS/validate ***@***.***> Cc: Ward, Jennifer ***@***.***>; Mention ***@***.***> Subject: Re: [NASA-PDS/validate] Validate fails to read files on Windows systems after Internal Reference check updates (#308) (Issue #809) @jennifergward<https://github.com/jennifergward> ok. I have been testing all of this on a windows virtual machine and it works for some reason. I will revisit in the morning and see if there is something I’m missing. — Reply to this email directly, view it on GitHub<#809 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR3WF5NQAUBGPEBXAEZQKPTYTKRCFAVCNFSM6AAAAABBYU274GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZG44DENJUGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jennifergward]

I did goof and now confirm that I get that same java error with the new 3.5.0 Snapshot no matter which version of java.

[jordanpadams]

@jennifergward I think something weird may be happening here. Can you verify if you have a JAVA_HOME set in your environment variables that points to Java 8?

It appears your PATH may have the correct Java 11 loaded, but your JAVA_HOME may be still pointed at Java 8.

[jordanpadams]

echo %JAVA_HOME%

[jennifergward]

@jordanpadams Is Java 11 required to run the newest validate?

[jordanpadams]

@jennifergward Yes. Unfortunately, Java 8 was end of life in 2019, so we upgraded ~1-2 years ago. We just started using some of the new methods in Java 11+ in the past 6-12 months, which is why I believe you encountered this exception.

For more information on the system requirements see: https://nasa-pds.github.io/validate/install/index.html#java-runtime-environment