Getting 'UNRESOLVABLE_REFERENCE' Error even though the definition is present in the swagger spec

[amarzavery]

Output

{ errors:
   [ { code: 'INVALID_RESPONSE_BODY',
       errors:
        [ { code: 'UNRESOLVABLE_REFERENCE',
            params: [ '#/definitions/RouteFilter' ],
            message: 'Reference could not be resolved: #/definitions/RouteFilter',
            path: [] } ],
       message: 'Invalid body: Reference could not be resolved: #/definitions/RouteFilter',
       path: [] } ],
  warnings: [] }
###########################################################
.....Printing the important parts of operation object for reference...........
###########################################################

operation.responseObjects[0].definitionFullyResolved.schema: { '$ref': '#/definitions/RouteFilter' }

operation.definitionFullyResolved.responses['200'].schema: { '$ref': '#/definitions/RouteFilter' }

• The definition "RouteFilter" is actually present in the definitions section of the spec. It can be seen over here.
• I have debugged this to the point where helper.validateAgainstSchema(validator, schema, value) is called.
• The problem is that schema is not fully resolved by json-refs. Hence z-schema validator gives the 'UNRESOLVABLE_REFERENCE' error.

Can you please help me find out what is going on here?

Link to the spec is here.

Please run this script to verify the behavior.

var Sway = require('sway'),
  util = require('util');

var specUrl = 'https://gist.githubusercontent.com/amarzavery/0a27e2bb3558ed3ef81744ed9c3fa5a3/raw/786f1c4245adcfe69266b8af65d79466a141c43e/spec.json';
var options = { definition: specUrl };

Sway.create(options)
  .then(function (api) {
    let operations = api.getOperations();
    let operation = operations[1];
    if (operation.operationId === 'RouteFilters_Get') {
      let responseWrapper = {
        statusCode: '200',
        body: operation['x-ms-examples']['RouteFilterGet']['responses']['200']['body'],
        headers: {
          'content-type': 'application/json'
        }
      };
      let result = operation.validateResponse(responseWrapper);
      console.dir(result, {depth: null, colors: true});
      console.log('###########################################################');
      console.log('.....Printing the operation object for reference...........')
      console.log('###########################################################\n');
      console.log('operation.responseObjects[0].definitionFullyResolved.schema: ' + util.inspect(operation.responseObjects[0].definitionFullyResolved.schema, {depth: null}));
      console.log('\noperation.definitionFullyResolved.responses[\'200\'].schema: ' + util.inspect(operation.definitionFullyResolved.responses['200'].schema, {depth: null}));
    } else {
      console.log( operation.operationId + ' is not the operation we are looking for.');
    }
  }, function (err) {
    console.error(err.stack);
  });

[whitlockjc]

I can confirm this. I'll look into it.

[amarzavery]

@whitlockjc - Sorry for bugging you. But any update on this? This will be a blocker for us. We are planning to take a dependency on this in our CI process before merging swaggers.

Any update would be really helpful.

Interesting feature in z-schema that can be used in Sway

Btw, I found another way to validate examples against model definitions.

• z-schema provides a mechanism to validate against sub-schemas. In our case (as in for Sway) swagger spec is the schema and the model definition in the definitions section is the sub-scehma.

I followed the example provided in the z-schema repo over here.

let valid = validator.validate(responseBodyExample, schema, { schemaPath: "definitions.RouteFilter" });
if (!valid) {
  let error = validator.getLastErrors();
  console.dir(error, {depth: null, colors: true});
}

where:

• responseBodyExample is the actual data
• schema is the parsed swagger spec
• schemaPath is the path to the model definition in the spec against which the sample needs to be validated

If we use this approach then one need not depend on json-refs to fullyResolve references. It makes life simple when circular references are involved.

I ran the above changes locally in my script and it did the right thing for me.

What do you think about it?

[amarzavery]

@whitlockjc - Any thoughts on the alternative approach (z-schema feature "validating against a sub-schema") provided in the previous comment?

[whitlockjc]

I implemented that at one point honestly and it worked fine but I needed the full schema to generate sample data anyways so I just went with one approach. The fully-resolved approach should just work once the json-refs bugs are fixed. A complete rewrite of the json-refs resolver is almost done and it should fix all bugs and performance issues brought up in the past. I just need a little more time.

[amarzavery]

I see. No worries. I have forked your repo and made changes based on the alternate solution. I am testing different scenarios to make sure everything works. Once things look good, will send a PR. Would love to take your feedback on it :).

[whitlockjc]

Did a PR ever come from this?

[whitlockjc]

With the latest on master, json-refs has been updated and should be fully-resolving everything. Is this still an issue?

[whitlockjc]

Closing this until there is more information.

[drej1]

Hi @whitlockjc just one question! I'm dealing with this same issue. it's fine that it is fixed in master, but how can I tell to my node.js application to use sway from master? I can only tell to use version 1.0.0 which doesn't include this fix. Will there be soon, I mean very soon some new release? Thanks

[whitlockjc]

You can use npm install --save apigee-127/sway#master.

[drej1]

Thanks! My fault, didn't check npm documentation in detail :/

[drej1]

Hi @whitlockjc I still have a problem with UNRESOLVABLE_REFERENCE.
I have the following swagger model defined:

CategoryModel:
    type: object
    properties:
      _id:
        type: string
        description: category name (id)
      displayName:
       type: string
       description: category display name
      imgUrl:
        type: string
        description: category image url
      parent:
        $ref: "#/definitions/CategoryModel"
      ancestors:
        type: array
        items: 
          $ref: "#/definitions/CategoryModel"

While sending a request I get the following error:

{
  "code": "INVALID_REQUEST_PARAMETER",
  "errors": [
    {
      "code": "UNRESOLVABLE_REFERENCE",
      "params": [
        "#/definitions/CategoryModel"
      ],
      "message": "Reference could not be resolved: #/definitions/CategoryModel",
      "path": [
        "properties",
        "ancestors",
        "items"
      ]
    },
    {
      "code": "UNRESOLVABLE_REFERENCE",
      "params": [
        "#/definitions/CategoryModel"
      ],
      "message": "Reference could not be resolved: #/definitions/CategoryModel",
      "path": [
        "properties",
        "parent"
      ]
    }
  ],
  "in": "body",
  "message": "Invalid parameter (CategoryModel): Value failed JSON Schema validation",
  "name": "CategoryModel",
  "path": [
    "paths",
    "/categories",
    "post",
    "parameters",
    "0"
  ]
}

Please could you take a look at it what am I missing?
Thank you very much!

[whitlockjc]

I'll need more information. What I see suggests there is no issue but I've no idea what you document(s) structure looks like.

[drej1]

Here's my whole swagger.yaml
swagger.zip

And I'm sending the following documents in body:
example1:

{
  "_id": "cat1"
}

example2:

{
  "_id": "cat2",
  "parent": {
    "_id": "cat1"
  }
}

If you need further information, just tell
Thanks

[whitlockjc]

Can you tell me how the document is being used? If I validate the document, it validates without error. Are you using some APIs that might not be working properly?

[drej1]

I use swagger-node with updated swagger-node-runner and swagger-express-mw and sway.
It fails in my tests while sending request to the swagger-node endpoint via superagent.
Does this help?

This week I will try to debug the process if this doesn't help.
Thanks

[whitlockjc]

swagger-node-runner uses an old version of json-refs which had issues. But in all honesty, I'm not sure how sway is being used so it's hard to tell. It seems like you're validating a request which means sway isn't fully resolving documents (fixed in [email protected]+) or sway is using the wrong schema (the not-fully-resolved one). Without a reproduction recipe, it will be hard to fix since I cannot reproduce it.

[drej1]

Hi, I found the issue:)
In sway you are calling helpers.removeCirculars(results.definitionFullyResolved); while the removeCirculars function assumes the definition is fully resolved but it is not, because json-refs is not called with option resolveCirculars=true and by default it is set to false, so the circulars are detected but not resolved. This way there are no repeating nodes inside removeCirculars function, nothing is replaces with {}.
And UNRESOLVABLE_REFERENCE is thrown by z-schema schema compilation.

Please can you fix sway to call json-refs with resolveCirculars=true option?

Or maybe change the logic of the removeCirculars method to use json-refs's circular=true metadata flag even when the circulars are not resolved?

Thanks.

[whitlockjc]

I'll file a new issue. If you haven't seen it by the time you read this, feel free to do the same.

[drej1]

Hi, thanks. I filed issue #165

[whitlockjc]

You can already use options.jsonRefs in Sway.create(options) to alter how json-refs is invoked. But the reason sway isn't released with proper [email protected]+ support is because of things like this.

[amarzavery]

Sway is a fantastic library for doing different swagger validations. However, in the initial days I had several issues with resolving json reference around relative paths or circular references. I ended up writing my swagger spec resolver such that all the references are local to the document. I then give the resolved swagger to sway for doing semantic and model validation. After that everything works like a charm.
You can take a look at my project over here http://github.com/azure/oav.

You can use the command oav resolve -spec <spec-path> [optional parameters] and then give that resolved spec to sway as an input.

[whitlockjc]

I'm not sure this is the proper way to address concerns. I'd rather see issues that can be fixed than discussing things in one issue about other issues. Also, json-refs' issues that you mentioned about resolving references should be fixed.

[amarzavery]

I have opened :

• Validating Polymorphic model types #169 (validating polymorphic types)
• Validating nullable types #170 (validating nullable types) as separate issues as directed. Thank you :).

[penx]

I'm getting this issue at the moment and am trying to debug to get more info for you.

From what I can tell, it's from the following lines:

sway/lib/types/parameter-value.js

Lines 122 to 125 in ee83295

	if (!skipValidation) {
	// Validate against JSON Schema
	result = helpers.validateAgainstSchema(helpers.getJSONSchemaValidator(), parameterObject.schema, value);
	}

The call to validateAgainstSchema passes parameterObject.schema, which is only a subset of the schema and doesn't contain the #/definitions node.

[penx]

this is with [email protected], sway@master and [email protected]

@whitlockjc are you saying above that parameterObject.schema should not contain any references ($ref keys) as these should be fully resolved by json-refs before callingvalidateAgainstSchema?

Or, if it can contain references, where is it supposed to look up the definitions.

If the former, I guess the issue I'm having is with swagger-node-runner (the call to operation:validateRequest from fittings/swagger_validator) as it should be passing in an already resolved schema for the parameter but is not.

[penx]

would it help if I create an example that reproduces this?

[whitlockjc]

Parameter.schema should be fully-resolved and based on the code, that is the case. If you're using [email protected], it is possible that a json-refs issue is causing the problem. If you're using sway@master, you might want to file a bug with a reproduction recipe.

[whitlockjc]

Since you mention swagger-node-runner, I'm assuming you're using a version of sway less than v1.0.0 and most certainly less than master. So I am not surprised you're running into a situation where the document isn't fully resolved.

[penx]

I’m using a fork of swagger node runner and am using sway@master

[whitlockjc]

Then I'll need to know more about the reproduction. json-refs resolver, the one being used in sway@master, doesn't have any resolution issues I'm aware of but there were some things it use to resolve that it shouldn't had and that could be related.

[penx]

After writing the example, I am pretty sure it's the same as issue #165