Validate variable length record detection does not report as expected on collection product

[jpl-jengelke]

🐛 Describe the bug identified during I&T

Unexpectedly, validate does not report the requisite error, “Validate should report that *.tab files must have fixed-length fields”.

🥼 Related Test Case(s)

B12.1/VAL.6: #390

🔁 : Related issues

#406

---

➕ Additional Details

📜 To Reproduce

Steps to reproduce the behavior:

1. Run ‘validate -t collection_calib_freq.xml’
2. Pass: Ensure that an error is reported that *.tab files must have fixed-length fields.
3. Fail because it did not report the error on the bad data set.

🕵️ Expected behavior

Error is reported that *.tab files must have fixed-length fields.

📚 Version of Software Used

validate 2.2.0

🩺 Test Data / Additional context

https://github.com/NASA-PDS/validate/files/6970348/validate-tab.tar.gz
https://github.com/NASA-PDS/validate/files/6970349/validate-csv.tar.gz

🏞Screenshots

🖥 System Info

• OS: Mac OS X Big Sur 11.6.5
• Browser N/A
• Version N/A

---

🦄 Related requirements

⚙️ Engineering Details

[jordanpadams]

@jpl-jengelke thanks! this is a bug, but we will not fix this at the moment since I believe it only applies to collection products. Can you confirm this is handled correctly for all other products?

[jpl-jengelke]

@jordanpadams I tested with validate #423 by modifying the file gbo.ast.alcdef-database_v1.0_20211021_checksum_manifest_v1.0.tab to reduce the size of one value (and removing the file size from gbo.ast.alcdef-database_v1.0_20211021_aip_v1.0.xml to let it run).

validate -R pds4.label -t testdata2/gbo.ast.alcdef-database_v1.0_20211021_aip_v1.0.xml

There were no complaints about fixed-length fields in the *.tab files. I believe it affects all products if that test is valid.

[jordanpadams]

@jpl-jengelke good catch, but this is actually a requirement that has not been implemented yet. The content validation currently only validates Table_Characters within a subset of File_Area_*. See #229

Here is an example of where this validation works (it should fail per the first line of the .tab being one character short).
table_character_failure.tar.gz

[jpl-jengelke]

@jordanpadams Just to cover the bases for documentation purposes, is there a separate ticket to address this for collection products?

[jordanpadams]

@jpl-jengelke that should be captured by #229 since a collection product has a File_Area_Inventory but will add that explicitly

[jordanpadams]

reopening since #390 was incorrectly closed and original implementation was invalid

[jordanpadams]

closing this again. looks like this ticket is invalid because validate should not check variable length records in collection inventories. collection inventories are delimited tables, not fixed width tabls

[rchenatjpl]

I guess you (@jordanpadams) figured out it was a File_Area_Inventory. We do and we must allow that to be a Table_Delimited, but if Dick's arguing that any file with extension .TAB should be fixed width, that's got merit, but there might be hundreds of collection*.tab with variable width.

[rchenatjpl]

So am I still needed to test something?

[jordanpadams]

@rchenatjpl not at this time.

Additionally, per the other thread on #390 (sorry for jumping back and forth), I'm not sure we can, or should, assume fixed width and delimited at the same time based upon a file name.