Improved validation when missing required sub array elements

[jpjpjp]

I had a few syntax errors in my Portman config file this week, which led to some errors that were a little tricky to track down. I'm not sure if it's realistic to improve the validation, but I'd thought I'd submit this as an idea.

I often add custom pre and post request scripts in my tests. Here is an example of a simple configuration that does this:

version: 1.0

# Define the tests to run 
tests:
  integrationTests:
    - name: badTest2
      operations:
      - openApiOperationId: leadsAdd
        openApiResponse: '200'
        variations:
          - name: badOverwriteRequestBody
            overwrites:
              - overwriteRequestBody:
                  - key: foo
                    value: bar
            tests:
              extendTests:
                - tests:
                  - file:/tmp/extend-test.js
              contentTests:
                - responseBodyTests:
                    - key: foo
                      value:  bar

If I run this config, portman will generate a file with my pre-request script:

However, if I have a syntax error and forget to include the - tests directive under the extendTests directive, portman will not notify me of this syntax error. Instead, it basically ignores the directive, which results in a collection with no pre-request script. Here's the invalid configuration:

version: 1.0

# Define the tests to run 
tests:
  integrationTests:
    - name: badTest2
      operations:
      - openApiOperationId: leadsAdd
        openApiResponse: '200'
        variations:
          - name: badOverwriteRequestBody
            overwrites:
              - overwriteRequestBody:
                  - key: foo
                    value: bar
            tests:
              extendTests:
                  - file:/tmp/extend-test.js
              contentTests:
                - responseBodyTests:
                    - key: foo
                      value:  bar

Portman runs without a syntax error, the generated collection does NOT contain my script:

For what it's worth, you get the same behavior if you have the entire extendTests directive in the wrong place, ie: in this config its a peer of tests: instead of a child:

version: 1.0

# Define the tests to run 
tests:
  integrationTests:
    - name: badTest2
      operations:
      - openApiOperationId: leadsAdd
        openApiResponse: '200'
        variations:
          - name: badOverwriteRequestBody
            overwrites:
              - overwriteRequestBody:
                  - key: foo
                    value: bar
            extendTests:
              - tests:
                - file:/tmp/extend-test.js
            tests:
              contentTests:
                - responseBodyTests:
                    - key: foo
                      value:  bar

Since the collection is generated, it can be hard to determine that some tests are missing unless you happen to navigate to the scripts tab and explicitly look for them.

It would be nice if portman generated some kind of a validation error instead of ignoring this. Interestingly, it does exactly this if you forget to include the - script child of the operationPreRequestScripts directive, ie:

version: 1.0

# Define the tests to run 
tests:
  integrationTests:
    - name: badTest2
      operations:
      - openApiOperationId: leadsAdd
        openApiResponse: '200'
        variations:
          - name: badOverwriteRequestBody
            operationPreRequestScripts:
                - file:/tmp/extend-test.js
            overwrites:
              - overwriteRequestBody:
                  - key: foo
                    value: bar
            tests:
              contentTests:
                - responseBodyTests:
                    - key: foo
                      value:  bar

npx @apideck/portman -l /tmp/crm.yml -c /tmp/invalid-operation-prerequest-scripts.yaml 
====================================================================================================================
 Local Path:            /tmp/crm.yml
 Portman Config:        /tmp/invalid-operation-prerequest-scripts.yaml
 Postman Config:        postman-config.default.json
 Environment:           .env
 Inject Tests:          true
 Run Newman:            false
 Newman Iteration Data: false
 Upload to Postman:     false  
====================================================================================================================
 Invalid Portman Config:                /tmp/invalid-operation-prerequest-scripts.yaml
[
  {
    message: "'0' property type must be object",
    path: '{base}.tests.integrationTests.0.operations.0.variations.0.operationPreRequestScripts.0',
    context: { errorType: 'type' }
  }
]
====================================================================================================================

Unzip the attached in /tmp to replicate:
invalid-external-tests.zip

npx @apideck/portman -l /tmp/crm.yml -c /tmp/invalid-extend-tests.yaml to generate collection with missing tests
npx @apideck/portman -l /tmp/crm.yml -c /tmp/invalid-operation-prerequest-scripts.yaml to get syntax errors