Variable Escaping Proposal

[TotalTechGeek]

Background

In JSON Logic, there are a handful of keys that are inaccessible from JSON Logic.

Examples include:

• Empty Keys { "": 7 }
• Keys with Dots in them { "hello.world": 7 }, { ".": 8 }
• Any combinations of those two, { ".": { "": 9 } }

I'd like to discuss backwards compatible solutions that would allow us to access these

Why not RFC6901

I'd ideally like to retain backwards compatibility with JSON Logic, and also address scopes in a separate proposal.

My concern with RFC6901 is that it adjusts how we traverse properties and it invalidates all existing implementations of JSON Logic.

If this was more greenfield, I'd think RFC6901 would be a solid choice.

Suggestion

What if we take notes from dot-prop notation, which essentially allows for escaping certain characters in the path with \.

Escaped Characters would be \, ., and potentially @ (useful for scope proposal).

Alongside this proposal, I would like to suggest a second operator, val, which is a simpler, more explicit way to reference values. val would be more AST friendly, and not involve a parsed path.

Using RFC6901 Paths to explain pathing,

{ val: 'hello' } would reference /hello, { "val": ["hello", "world"] } would reference /hello/world. { val: '' } would reference /

{ val: null } would reference the entire context.

Proposed Test Suite

var retrofit

[
    "Test Specification for Handling esoteric path traversal",
    {
        "description": "Fetches a value from an empty key",
        "rule": { "var": "." },
        "data": { "" : 1 },
        "result": 1
    },
    {
        "description": "Fetches a value from a nested empty key",
        "rule": { "var": ".." },
        "data": { "" : { "": 2 } },
        "result": 2
    },
    {
        "description": "Fetches a value from a doubly nested empty key",
        "rule": { "var": "..." },
        "data": { "" : { "": { "": 3 } } },
        "result": 3
    },
    {
        "description": "Fetches a value from a key that is purely a dot",
        "rule": { "var": "\\." },
        "data": { "." : 20 },
        "result": 20
    },
    {
        "description": "Fetches a value from a key with a dot in it",
        "rule": { "var": "\\.key" },
        "data": { ".key" : 4 },
        "result": 4
    },
    {
        "description":"Fetches a value from a key with a dot in it (2)",
        "rule": { "var": "hello\\.world" },
        "data": { "hello.world" : 5 },
        "result": 5
    },
    {
        "description": "Fetches a value from a key inside an empty key with a dot in it",
        "rule": { "var": ".\\.key" },
        "data": { "": { ".key" : 6 } },
        "result": 6
    },
    {
        "description": "Going a few levels deep",
        "rule": { "var": "..\\.key." },
        "data": { "": { "": { ".key": { "": 7 }} }},
        "result": 7
    },
    {
        "description": "Escape / as well, which is useful for the scope proposal",
        "rule": { "var": "\\/" },
        "data": { "/" : 8 },
        "result": 8
    },
    {
        "description": "Though / doesn't inherently need to be escaped", 
        "rule": { "var": "/" },
        "data": { "/" : 9 },
        "result": 9
    },
    {
        "description": "Dot then empty key",
        "rule": { "var": "\\.." },
        "data": { "." : { "" : 10 } },
        "result": 10
    },
    {
        "description": "Empty key then dot",
        "rule": { "var": ".\\." },
        "data": { "" : { "." : 11 } },
        "result": 11
    },
    {
        "description": "Can use backslack in name, too",
        "rule": { "var": "\\\\.Hello" },
        "data": { "\\" : { "Hello" : 12 } },
        "result": 12
    }
]

val operator

[
    "Test Specification for val",
    {
        "description": "Fetches a value from an empty key",
        "rule": { "val": "" },
        "data": { "" : 1 },
        "result": 1
    },
    {
        "description": "Fetches a value from a nested empty key",
        "rule": { "val": ["", ""] },
        "data": { "" : { "": 2 } },
        "result": 2
    },
    {
        "description": "Fetches a value from a doubly nested empty key",
        "rule": { "val": ["", "", ""] },
        "data": { "" : { "": { "": 3 } } },
        "result": 3
    },
    {
        "description": "Fetches a value from a key that is purely a dot",
        "rule": { "val": "." },
        "data": { "." : 20 },
        "result": 20
    },
    {
        "description": "Fetches the entire context",
        "rule": { "val": null },
        "data": { "": 21 },
        "result": { "": 21 }
    },
    {
        "description": "Fetches the entire context for a nested key",
        "rule": { "val": "" },
        "data": { "": { "": 22 } },
        "result": { "": 22 }
    },
    {
        "description": "Using val in a map (remember that null gets the current context, not empty string)",
        "rule": { "map": [[1,2,3], { "+": [{ "val": null }, 1] }] },
        "data": null,
        "result": [2,3,4]
    },
    {
        "description": "Using val in a map (and remember [null] and null are the same)",
        "rule": { "map": [[1,2,3], { "+": [{ "val": [null] }, 1] }] },
        "data": null,
        "result": [2,3,4]
    }
]

Anticipated Questions

Why not make var do what val is proposed to do?

I think for this initiative to be successful, we must try to maintain backwards compatibility with current mainline JSON Logic.

If we break backwards compatibility, I fear this will fragment the ecosystem and keep people from updating.

Do we need both var and val?

If we retrofit var with dot-prop, it's likely not a necessity, but having an AST friendly way to reference variables would be quite nice and simple to implement.

I also don't believe it would add much project maintenance overhead, as folks could set up var to parse the path, and call the code set up for val.

Because val functions distinctly from var, I believe it has a right to exist, and by introducing a new operator, we have a path to deprecate the old operator and list var and val support in compatibility tables.

Truthfully, I don't think we'll get away from var, but val will make it easier for folks to build grammars that compile down to JSON Logic, and execute it across different environments.

[codetiger]

If the use case valid one, or a typo?

{
      "description": "Can use backslack in name, too",
      "rule": { "var": "\\\\.Hello" },
      "data": { "\\" : { "Hello" : 12 } },
      "result": 12
  }

I guess you mean without the dot like below
"rule": { "var": "\\\\Hello" },

[codetiger]

Please ignore the comment. The use case is valid. :)

[rmannibucau]

Hi

If becoming "standard" I think it makes sense to stick to the spirit and value of jsonlogic, ie staying an ast, if the ast needs other parsers it defeats the purpose of the spec and breaks interoperability and portability (graphql is one good example it doesnt work and stays a single vendor thing).

So my 2cts would be no escaping but a preparsed tree in plain json.

Not sure backward compat is a huge concern, I'd rather say the portability and multivendor agreement has more value and vendors can still enable backward compat in the impl (=not needed at spec level)....also means the json needs a version at root level for future upgrades (like jsonschema for ex).

[TotalTechGeek]

@rmannibucau Hypothetically speaking, is this how you would imagine it'd work?

[
    "Test Specification for val (a better AST-like var operator)",
    {
        "description": "Fetches a value from an empty key",
        "rule": { "val": "" },
        "data": { "" : 1 },
        "result": 1
    },
    {
        "description": "Fetches a value from a nested empty key",
        "rule": { "val": ["", ""] },
        "data": { "" : { "": 2 } },
        "result": 2
    },
    {
        "description": "Fetches a value from a doubly nested empty key",
        "rule": { "val": ["", "", ""] },
        "data": { "" : { "": { "": 3 } } },
        "result": 3
    },
    {
        "description": "Fetches a value from a key that is purely a dot",
        "rule": { "val": "." },
        "data": { "." : 20 },
        "result": 20
    },
    {
        "description": "Fetches the entire context",
        "rule": { "val": null },
        "data": { "": 21 },
        "result": { "": 21 }
    },
    {
        "description": "Fetches the entire context for a nested key",
        "rule": { "val": "" },
        "data": { "": { "": 22 } },
        "result": { "": 22 }
    },
    {
        "description": "Using val in a map (remember that null gets the current context, not empty string)",
        "rule": { "map": [[1,2,3], { "+": [{ "val": null }, 1] }] },
        "data": null,
        "result": [2,3,4]
    },
    {
        "description": "Using val in a map (and remember [null] and null are the same)",
        "rule": { "map": [[1,2,3], { "+": [{ "val": [null] }, 1] }] },
        "data": null,
        "result": [2,3,4]
    }
]

[rmannibucau]

something like that + numbers for arrays (["foo", 0] will read first item of foo array for ex). I don't know if null is the best to use the current value (I would have used an empty object cause it looks more like some pattern matching in some language) but that's a detail.

[TotalTechGeek]

✅ For sure on the number part,

{} is handled awkwardly by different JSON Logic implementations, and it requires a bit of interrogation by different languages to figure out "Oh hey, you're an object, can I execute you? Are you an empty object?".

I see the appeal, I just think it'd be a bit of a more divergent choice than using a value like null, which in this case kinda slots in nicely as "Oh, I'm not traversing anything!".

I created a simple (unoptimized implementation) here.

I don't see a "clean" way yet that I like to handle traversing scope with val quite yet, but I definitely see the appeal of having something a bit more pure available.

[TotalTechGeek]

Because it was brought up, one way to retrofit val with scopes is the following:

[
    "Testing out scopes",
    {
        "description": "Climb up to get adder",
        "rule": { "map": [[1,2,3], { "+": [{ "val": null }, { "val": [[-2], "adder"] }] }] },
        "data": { "adder": 10 },
        "result": [11,12,13]
    },
    {
        "description": "Climb up to get index",
        "rule": { "map": [[1,2,3], { "+": [{ "val": null }, { "val": [[-1], "index"] }] }] },
        "data": { "adder": 10 },
        "result": [1,3,5]
    },
    {
        "description": "Nested get adder",
        "rule": {
            "map": [["Test"], { "map": [[1,2,3], { "+": [{"val": null}, {"val": [[-4], "adder"]}] }]} ]
        },
        "data": { "adder": 10 },
        "result": [[11,12,13]]
    }
]

Basically, you can address the scope by doing [-2] as your first argument to say "I want to climb up twice".

This allows val to exist as a standalone operator without the need to implement other operators.

This retrofit also allows you to address the current scope with { "val" : [[0]] }

[TotalTechGeek]

This will cause controversy:

Because of the defaultValue piece of var, I don't see a way to effectively consolidate these two approaches as the same operator.

But my current 😬 take is:

• var is upgraded to a modified RFC6901 (. and / equivalency) that would maintain backwards compatibility, but allow folks to use RFC6901.
• We can introduce a val operator as described above for explicit paths that is much more AST friendly.

Documentation for var should be updated to get folks to use JSON Pointer syntax.
val can be introduced as a simpler way to reference variables.

"If we introduced val, why bother retrofitting var with RFC6901 rather than keep it simple"

Not an illogical gripe, but many folks do use JSON Logic as a DSL, and having a convenient way to access data would be quite nice.

"Ew, two ways to get variables"

Sure; in which case I'd prefer the modified RFC6901 approach.

But there are advantages to having an AST friendly / parse-free way of representing variable access.

Implementations could also call the val code from var, since var would process the JSON Pointer.

It could be called astVar but then that might discourage folks from using it. Open to other names.

tl;dr

• var - remains a way to specify a parsed string-based path to their data. Retrofitted to accommodate RFC6901.
• val - is introduced to allow explicit paths to the data.

---

Then there's a path to deprecate or implement operators, and show that on compat tables.

[dslmeinte]

I think adding semantics to paths to the var operator is quite a step, because var is so central to JSON Logic, so it's a potentially functionality-breaking change, depending on how the various implementations currently interpret occurrences of sub paths like "..". Also, such “relative” paths tend to be somewhat brittle in the face of nested iterators.

As an alternative, I've experimented with altering the implementation of the reduce operator by passing an object { current, accumulator, data } as the context to the iterated function (the 2nd argument passed to the reduce operator), where data is the data context of the call to reduce — crucially, this is not the data context resulting from applying the 1st argument. This absolves the implementation of having to track sub scopes, as well as the need to expand the var operator.

See the patch here: https://github.com/ehn-dcc-development/eu-dcc-business-rules/blob/main/jsonlogic/javascript/patch/patched-logic.js

This is a “benign” (non-breaking) change, because the extra data sub object passed doesn't clash with anything.

I only experimented with the reduce operator, because 1) that's the one we would've needed, 2) we ended up not needing this at all (because we went from a situation with potentially multiple sub objects, to one which guaranteed exactly one sub object), 3) we ended up not using JSON Logic but a strictly specified sub set — CertLogic.

[TotalTechGeek]

I think adding semantics to paths to the var operator is quite a step, because var is so central to JSON Logic

JSON Logic already has path semantics because . is used to access subkeys of objects.

Both this proposal and my modified RFC6901 are backwards compatible with all JSON Logic tests, and pretty much every logic snippet I've come across 😅

I've implemented both.

Also, such “relative” paths tend to be somewhat brittle in the face of nested iterators.

Those are in the scope of a separate proposal, but in practice they've worked rather well for json-logic-engine. I've also seen it work rather well for handlebars

For JSON Logic to be more generally useful as an AST for other grammars, I think giving operators the ability to add scope frames is necessary.

This is a “benign” (non-breaking) change, because the extra data sub object passed doesn't clash with anything.

This works for reduce because it receives an object argument, but this approach doesn't work for some of the other iterators like filter, map, etc.

It also doesn't work as well for being able to handle nested iterators.

While this approach is simpler, I don't believe it scales well to being able to support nested contexts, nor would it allow me to compile something like the Handlebars grammar down to JSON Logic and execute it.

[TotalTechGeek]

PR that uses the modified RFC6901 approach: json-logic/json-logic-engine#38

The tests run use the suite of tests provided from https://jsonlogic.com/tests.json, as well as a handful of other logic snippets I've come across.

I think either of the proposed approaches would be fairly backwards compatible with real-world usage.

[dslmeinte]

I think adding semantics to paths to the var operator is quite a step, because var is so central to JSON Logic

JSON Logic already has path semantics because . is used to access subkeys of objects.

I know — I meant: “adding additional semantics to paths on top of the semantics of the .”.

I've also seen it work rather well for handlebars

That's exactly where I've seen it being very brittle, and hard to understand and debug.

While this approach is simpler, I don't believe it scales well to being able to support nested contexts, nor would it allow me to compile something like the Handlebars grammar down to JSON Logic and execute it.

The question might also be: to what extent is it necessary/desirable to support nested contexts?

I guess you mean “compiling something that conforms to the Handlebars grammar down to JSON Logic”, right? (Otherwise, I don't understand what you mean.) Is that something that users are typically asking for?

The reason I'm asking this is that even if it's relatively easy to add functionality to a language, it's often difficult to gauge the consequences, and then the cat's already out of the proverbial bag.

Ah well, just my €0.02

[TotalTechGeek]

to what extent is it necessary/desirable to support nested contexts?

I believe scoping capabilities are necessary to an extent for general use. I've seen a handful of issues pop up (these aren't the only ones)

• filter does not work with the second parameter as var json-logic-engine#28
• Accessing data using var path inside an array diegoholiveira/jsonlogic#87
• Data is not used by "some" jwadhams/json-logic-js#48
• use the data "var" in side "filter" jwadhams/json-logic-js#61
• Referencing data values within a map jwadhams/json-logic-js#124

That require access to at least one level of scope tracking.

And I feel that it does indeed open a can of worms, "If you support adding the scope, can I get the iterator index? Can I get the iterator index above from a map inside of a map?"

Being able to support it makes JSON Logic more generally useful, but I feel it's good to discuss it, but also why I separated it into a separate proposal.

But I generally think if you're going to resolve it with one layer of depth, it's not so difficult to generalize it.

Your patch would likely work for a good chunk of use-cases, though it doesn't apply in a backwards compatible to some of the other iterators.

[TotalTechGeek]

RFC6901 Retrofit Proposal

What if we retrofit RFC6901 onto the var operator. We use RFC6901, but introduce:

• An equivalency between . and /
• A new escape code, where ~2 becomes .
• Implicit /, "hello" would be understood as "/hello"

This retrofit would work with pretty much most existing logic, enable folks to use RFC6901, and resolve the inaccessible keys.

Flaws with the Retrofit

Because of the implicit /, while this would allow RFC6901, this would not allow for relative json pointers as used by JSON Schema, because there would be an ambiguity.

Backwards Compatibility Concerns

This retrofit would require folks to escape paths for ~ and /, but I don't believe those are very common use-cases compared to the ocean of existing logic.

I think this upgrade would be manageable.

Proposed Test Suite

[
    "Test Specification for Handling esoteric path traversal",
    {
        "description": "Fetches a value from an empty key",
        "rule": { "var": "/" },
        "data": { "" : 1 },
        "result": 1
    },
    {
        "description": "Fetches a value from a nested empty key",
        "rule": { "var": "//" },
        "data": { "" : { "": 2 } },
        "result": 2
    },
    {
        "description": "Fetches a value from a doubly nested empty key",
        "rule": { "var": "///" },
        "data": { "" : { "": { "": 3 } } },
        "result": 3
    },
    {
        "description": "Fetches a value from a key that is purely a dot",
        "rule": { "var": "~2" },
        "data": { "." : 20 },
        "result": 20
    },
    {
        "description": "Fetches a value from a key with a dot in it",
        "rule": { "var": "~2key" },
        "data": { ".key" : 4 },
        "result": 4
    },
    {
        "description":"Fetches a value from a key with a dot in it (2)",
        "rule": { "var": "hello~2world" },
        "data": { "hello.world" : 5 },
        "result": 5
    },
    {
        "description": "Fetches a value from a key inside an empty key with a dot in it",
        "rule": { "var": "//~2key" },
        "data": { "": { ".key" : 6 } },
        "result": 6
    },
    {
        "description": "Going a few levels deep",
        "rule": { "var": "///~2key/" },
        "data": { "": { "": { ".key": { "": 7 }} }},
        "result": 7
    },
    {
        "description": "Escape / as well, which is useful for the scope proposal",
        "rule": { "var": "~1" },
        "data": { "/" : 8 },
        "result": 8
    },
    {
        "description": "Dot then empty key",
        "rule": { "var": "~2." },
        "data": { "." : { "" : 10 } },
        "result": 10
    },
    {
        "description": "Empty key then dot",
        "rule": { "var": "//~2" },
        "data": { "" : { "." : 11 } },
        "result": 11
    },
    {
        "description": "Can use backslack in name, too",
        "rule": { "var": "\\.Hello" },
        "data": { "\\" : { "Hello" : 12 } },
        "result": 12
    },
    {
        "description": "Can escape tilde",
        "rule": { "var": "~0" },
        "data": { "~" : 13 },
        "result": 13
    },
    {
        "description": "Fetches a value from an empty key, equivalence",
        "rule": { "var": "." },
        "data": { "" : 1 },
        "result": 1
    },
    {
        "description": "Fetches a value from a nested empty key, equivalence",
        "rule": { "var": ".." },
        "data": { "" : { "": 2 } },
        "result": 2
    },
    {
        "description": "Fetches a value from a doubly nested empty key, equivalence",
        "rule": { "var": "..." },
        "data": { "" : { "": { "": 3 } } },
        "result": 3
    },
    {
        "description": "Old format still works",
        "rule": { "var": "hello.world" },
        "data": { "hello": { "world": 5 } },
        "result": 5
    },
    {
        "description": "New format works too",
        "rule": { "var": "hello/world" },
        "data": { "hello": { "world": 5 } },
        "result": 5
    }
]

[gregsdennis]

What if we retrofit RFC6901 onto the var operator.

If you're going to use something that's not RFC6901, then don't mention RFC6901. Just make your own thing. Referencing a specification should only be done when using that specification as-is. Don't reference a spec only to change it.

[TotalTechGeek]

Fair enough; my preference for the RFC6901 modification was actually contingent upon your input 😂

[TotalTechGeek]

I believe I'll be closing this proposal and introducing a new one, based on misc discussions we've had in different threads.