Number of records found by validate in .TAB file does not match number of lines

[tloubrieu-jpl]

Hello,

I am validating a label and its data file with the following command:

% ./bin/validate -t ~/PycharmProjects/pge-python/adpTestDir/output/pds4/ECM000XXX_2022173T165156_HK_LR_SOLARWIND_CAL_ASC_RAW010.xml --skip-context-validation

PDS Validate Tool Report

Configuration:
   Version                       2.2.0
   Date                          2022-06-22T16:53:25Z

Parameters:
   Targets                       [file:/Users/loubrieu/PycharmProjects/pge-python/adpTestDir/output/pds4/ECM000XXX_2022173T165156_HK_LR_SOLARWIND_CAL_ASC_RAW010.xml]
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /Users/loubrieu/Downloads/validate-2.2.0/resources/registered_context_products.json



Product Level Validation Results

  FAIL: file:/Users/loubrieu/PycharmProjects/pge-python/adpTestDir/output/pds4/ECM000XXX_2022173T165156_HK_LR_SOLARWIND_CAL_ASC_RAW010.xml
      ERROR  [error.label.schematron]   line 86, 37: The attribute clipper:mission_phase_name must be nulled or equal to one of the following values 'Capture Orbit', 'Inner Cruise 1', 'Jupiter Approach', 'Orbits 1-2', 'Outer Cruise', 'Period Reduction Maneuver', 'Quiet Cruise', 'Science Orbits'.
    Begin Content Validation: file:/Users/loubrieu/PycharmProjects/pge-python/adpTestDir/output/pds4/ECM000XXX_2022173T165156_HK_LR_SOLARWIND_CAL_ASC_RAW010.TAB
      ERROR  [error.table.records_mismatch]   table 1: Number of records read is not equal to the defined number of records in the label (expected 944, got 750).
    End Content Validation: file:/Users/loubrieu/PycharmProjects/pge-python/adpTestDir/output/pds4/ECM000XXX_2022173T165156_HK_LR_SOLARWIND_CAL_ASC_RAW010.TAB
        1 product validation(s) completed

Summary:

  2 error(s)
  0 warning(s)

  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    1            error.label.schematron
    1            error.table.records_mismatch

End of Report
Completed execution in 38754 ms

I don't understand the error:
ERROR [error.table.records_mismatch] table 1: Number of records read is not equal to the defined number of records in the label (expected 944, got 750).

Since the data file contains 944 lines.

I am attaching the tested label and data file: Archive.zip

Can you help me with that issue ?

Thanks,

Thomas

[msbentley]

I imagine this is triggered because the record length is wrong. If you fix the record length, you start to see other bugs in the label (incorrect field offsets etc.).

[jordanpadams]

I think this may be the issue. Validate should probably throw an error that the record length is wrong instead, but I think the record count error gets thrown first and then jumps out of the checks.

Here are a couple quirks of how the PDS DSV is defined:

1. Validate reads in a line based upon the line length defined, NOT the record delimiter. Why? Because you are actually allowed to embed record delimiters within a given record (e.g. a b c CRLF d f CRLF could be valid record).
2. You can also have "undefined" content after your last field.

So I think what may be happening is a few records are too long, validate is then reading past the expected CRLF for that record, and then reads all the way to the next CRLF and then throws everything off.

A quick awk execution (which I think works?) shows there are a few lines that are off:

awk '{print length}' ECM000XXX_2022173T165156_HK_LR_SOLARWIND_CAL_ASC_RAW010.TAB | sort -n | uniq -c

[jordanpadams]

@tloubrieu-jpl ☝️

[tloubrieu-jpl]

Thanks @jordanpadams , that is really helpful

[jordanpadams]

@tloubrieu-jpl thanks should go to @msbentley . I actually originally thought this was just a bug :-) . definitely comment back if it turns out that was the issue. so we can make sure we track this to completion.

[tloubrieu-jpl]

Thanks @msbentley and @jordanpadams I was able to make validate happy with my file, then I am happy too !

I had a few formatting issues which made the tab file to have inconsistent line lengths. I could have spotted that visually since all the columns were not aligned as they were supposed to but I missed it.

Thanks again,

[jordanpadams]

🎉 excellent! I will create a ticket to track this as an improvement though, because validate should catch the line length error first. the other errors may follow, but should definitely notice at least the first time the record delimiter does not occur when the record is supposed to end.

[tloubrieu-jpl]

Thanks @jordanpadams

[msbentley]

That would be great - this is one of those cases that you think there is just one error, but squash it and realise there are a bunch more hidden under the surface

[jordanpadams]

@tloubrieu-jpl opening and inspecting this with PDS4 Viewer may help as well.

[miguelp1986]

@jordanpadams what is the expected error for the line length issue? I may need some help finding it in my output.
output.txt

[jordanpadams]

@miguelp1986 what issue is this in reference to? I can't seem to track is down from this discussion thread

[miguelp1986]

@jordanpadams this issue is related to #519. "validate appears to be missing checking that the record delimiter occurs at the end of the expected record length"

I'm using the attached test files. My output is using that data.

[miguelp1986]

@jordanpadams Looks like I found it.

ERROR [error.label.table_definition_problem] line 1: Data Object #1 (Identifier Not Specified): Data object is truncated. Expected bytes as defined by label: 645696 (944 records times 684 bytes per record), Actual bytes in file: 513353 (750.51605 records times 684 bytes per record) OR (944 records times 543.80615 bytes per record)