How to merge two slists and remove duplicate entries?

[craigcomstock]

I have two slists:

vars:
  "list1" slist => { "one", "two" };
  "list2" slist => { "one" };

And would like to merge those two together but remove the duplicate entry: "one".

I see the mergedata and unique functions but I can't figure out how to use them together in one statement.

So far I have

bundle agent main
{
  vars:
    "list1" slist => { "one", "two" };
    "list2" slist => { "one" };
    "merged" data => mergedata("list1","list2");
    "unique" slist => unique("merged");

  reports:
    "merged is $(with)" with => storejson("merged");
    "unique is $(with)" with => storejson("unique");
}

Which gives me:

R: merged is [
  "one",
  "two",
  "one"
]
R: unique is [
  "one",
  "two"
]

But I can't figure out how to do the merged and unique steps in one statement so as to avoid creating yet another variable in between.

I tried this:

  "unique" slist => unique(mergedata( { @(list1), @(list2) } ));

But that gives me a bunch of parser errors. I suppose that mergedata doesn't parse an "inline" list?

[olehermanse]

Using the return value of mergedata() directly, as an argument to unique() works:

bundle agent main
{
  vars:
    "list1"
      slist => { "one", "two" };
    "list2"
      slist => { "one", "three" };
    "result"
      slist => unique(mergedata(list1, list2));

  reports:
    "result is $(with)" with => storejson("result");
}

(I like to not quote string args like list1 and list2, to hint that they are variables, and not just string values. To CFEngine it doesn't really matter).

Output:

$ cf-agent -K test.cf
R: result is [
  "one",
  "two",
  "three"
]

More information in the docs:

• unique()
• Collecting functions
• mergedata()

[craigcomstock]

As a follow-up, I also needed to merge in a scalar. With a bit of experimentation I found a way that seems to work! :O Although it requires an awful lot of "special character wrapping" around the scalar ;)

bundle agent main
{
  vars:
    "list1" slist => { "one", "two" };
    "scalar1" string => "one";
    "unique" slist => unique(mergedata('["$(scalar1)"]', list1));
  reports:
    "unique is $(with)" with => storejson("unique");
}

[craigcomstock]

Right. The most obvious syntax that I would expect to work would be something like

unique(mergedata([$(scalar1)],list1))

But it seemed I needed to start the JSON expression with a single-tick (or any other quote character like double-quotes or back-tick) and then add the other quotes around the $(scalar1) so that it became a string inside of a list.

But as you say, it is a "collecting function" and so the way to get a list or map into there is to send it as a JSON string.

Without the quotes around the $(scalar1) I get no output, so it's likely not being evaluated?

"unique" slist => unique(mergedata('[$(scalar1)]', list1));

and this fixes that:

"unique" slist => unique(mergedata('["$(scalar1)"]', list1));

[craigcomstock]

I tried switching around the quote characters I was using and it doesn't seem to work unless I use single-ticks to start the JSON string and then double-quotes for the list item.

[tzz]

But it seemed I needed to start the JSON expression with a single-tick (or any other quote character like double-quotes or back-tick) and then add the other quotes around the $(scalar1) so that it became a string inside of a list.

(CFE has no first-class JSON support, it's all inside a string. It also has no way to create an slist on the fly, inside another function's arguments. The parser was pretty nasty when I worked on this area so I chose to have all the "collecting" functions parse JSON dynamically from strings.)

Unfortunately if scalar1 contained a double quote ", the JSON parser would reject that dynamically constructed string. You could work around that with the regex escape function but it's not a good situation.

I think the generally proper way is to add the scalar and sublists to the top slist and then unique():

"merged_with_scalar" slist => { @(list1), @(list2), $(scalar1) };
"unique" slist => unique(merged_with_scalar);

[olehermanse]

I tried switching around the quote characters I was using and it doesn't seem to work unless I use single-ticks to start the JSON string and then double-quotes for the list item.

@craigcomstock JSON requires double quotes for strings:

https://www.json.org/

[tzz]

Hi Ole and Craig! So... I remembered, years ago, I wrote something into the JSON parser to interpolate naked variables into it. And it works for this case! Like this:

"unique" slist => unique(mergedata(list1, '[scalar1]'));

Why does this work? Because any time the CFEngine JSON parser sees something that doesn't look like JSON, it will try to replace it with the JSON version of a variable name. Here, scalar1 was replaced with the JSON version of its contents, which is a properly quoted string.

This also allows things like { "somekey": mycontainer } and so on. Here, mycontainer will be replaced with a possibly multilevel JSON data structure. Because JSON inside JSON nests cleanly, this works, and again, it quotes correctly.

If the variable name is not in the vartable, the interpolation fails and the JSON is invalid.

@nickanderson this may be one of those hidden CFEngine features that maybe need more documentation?