Manipulating map keys on arrays #529

mtrin-wooliesx · 2021-10-30T06:27:43Z

mtrin-wooliesx
Oct 30, 2021

I have tried quite a few different approaches here, but struggling to get to the final solution.

Basically I have a list of URIs I want to use as the main values to transform different keys and values, depending on where.

values.yaml

#@data/values
---

request_methods:
  - "/foo/bar/v2/app1"
  - "/bar/v3/foo/app/os"
  - "etc"

The intended target yaml is:

config:
  - _foo_bar_v2_app1_count: "count_defaulttemplate.json"
  - _foo_bar_v2_app1_time: "time_defaulttemplate.json"
  - _bar_v3_foo_app_os_count: "count_defaulttemplate.json"
  - _bar_v3_foo_app_os_time: "time_defaulttemplate.json"

_foo_bar_v2_app1_count:
  - param1: "123"
  - param2: "456"

_foo_bar_v2_app1_time:
  - param1: "123"
  - param2: "456"

_bar_v3_foo_app_os_count:
  - param1: "123"
  - param2: "456"

etc

Here is what I'm currently trying

template.yaml

#@ load("@ytt:data", "data")
#@ load("@ytt:template", "template")

#@ def maps():
  #@ return {"{}:".format(data.values.request_methods).replace("/", "_"): "value"}
#@ end

config:
  _ #@ template.replace(maps())

Result

'[]:': value

Trying a for loop straight from loaded data:

config:
config:
#! for items in data.values.request_methods:
  - #@ "prefix".format(data.values.request_methods).replace("/", "_")
#! end

Result:

config:
- prefix

Closest I got, but it wraps in single quotes:

config:
  #@ for method in data.values.request_methods: 
  #@ x=method.replace("/", "_")
  - #@ "test_prefix{}: {}".format(x, "value")
  #@ end

Result:

config:
- 'test_prefixfoo: value'
- 'test_prefixbar: value'
- 'test_prefixfoo: value'
- 'test_prefixbar: value'

Also tried

- #@ template.replace(data.values.request_methods)

but can't make it work with the values.

Any ideas?

Answered by mtrin-wooliesx

Nov 11, 2021

Just for reference, this is what ended up working for me 🙂
thanks for the community!

#@ load("@ytt:data", "data")
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:template", "template")

#@ def path_to_key(name, metric):
#@   return name.replace("/", "_") + "_" + metric
#@ end

config:
#@ for method in data.values.request_methods:
#@   for metric in data.values.request_methods[method]:
#@yaml/text-templated-strings
- REQUEST_RT_SLO(@= path_to_key(method, metric) @): #@ "REQUEST_RT_SLO_" + metric + ".json"
#@   end
#@ end

#@ def additions(method):
#@overlay/match expects=0
REQUEST_NAME: #@ method
#@ end

#@ for method in data.values.request_methods:
#@   for metric in data.values.request_…

View full answer

pivotaljohn · 2021-11-01T14:23:17Z

pivotaljohn
Nov 1, 2021

Hey @mtrin-wooliesx. Thanks for posting your question.

I believe you're looking to do something like this: https://carvel.dev/ytt/#gist:https://gist.github.com/pivotaljohn/d095513775852285015f1e1a45b807f7

Getting there took a key insight: ytt converts Starlark dictionaries into YAML maps (it also converts Starlark lists into YAML arrays).

Is that solution I linked to going in the direction you were hoping?

Looking down the road, I suspect you have four variables: the route, the metric, the metric template, and the route+metric params. Extracting these into variables and then composing the template output through a small set of functions can keep the overall template versatile/flexible.

.
.
.

We can certainly continue our conversation here. If you'd prefer, you can also get timely help in our channel on the Kubernetes Slack: #carvel. If you are not already in that Slack workspace, you can sign up here: https://slack.k8s.io/.

Cheers!

2 replies

mtrin-wooliesx Nov 2, 2021
Author

Hi @pivotaljohn

That is great, thank you so much. This really put me in the right direction. I will join the slack channel as well, but just contributing further for future reference.

Was thinking of the cleanest way of doing it, and seems like I could use a simpler values format.
I played around a bit and tried many things, from using maps to dicts on the values yaml.

request_methods:
  /foo/bar/foo/bar: 
    - count: 
        name: 1
        REQUEST_NAME: 2
        tsmMetricKey: 3
        tagKey: 4
        tagValue: 5
        DT_PROD_ENV: 6
    - threshold: 
        name: 1
        REQUEST_NAME: 1
        tsmMetricKey: 2
        tagKey: 4
        tagValue: 5
        slo: 6
        DT_PROD_ENV: 1

This worked well with this first bit:

#@ load("@ytt:data", "data")
#@ load("@ytt:template", "template")
#@ load("@ytt:struct", "struct")

#@ def path_to_key(name, metric):
#@   return name.replace("/", "_") + "_" + metric
#@ end

#@ def config_headers(route, metric):
#@   return {path_to_key(route, metric): metric + "_defaulttemplate.json"}
#@ end

#@ def config_items(route, metric):
#@   return path_to_key(route, metric)
#@ end


config:
#@ for method in struct.encode(data.values.request_methods):
- #@ config_headers(method, "count")
- #@ config_headers(method, "threshold")
#@ end

However, I can't do the param bit, not sure where to use encode,decode,make, but I get Go errors.

#@ def route_config(method, params):
#@ return {method: params}
#@ end

params_test:
#@ for param in data.values.request_methods:
_: #@ template.replace(route_config(struct.encode(param), param.count))
#@ end

$ ytt -f mapping.yaml -f values.yaml  --output-files result.yaml
ytt: Error: 
- (p) unknown type *starlark.Builtin for conversion to go value (backtrace: goroutine 1 [running]:
    in <toplevel>
      mapping.yaml:30 | _: #@ template.replace(route_config(struct.encode(param), param.count))

    reason:
     runtime/debug.Stack(0x1515240, 0x14eb940, 0xc0002b81f0)
        runtime/debug/stack.go:24 +0x9f
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1.1(0xc00020ca40)
        github.com/k14s/ytt/pkg/template/core/errs.go:23 +0x15c
     panic(0x14eb940, 0xc0002b81f0)
        runtime/panic.go:965 +0x1b9
     github.com/k14s/ytt/pkg/template/core.StarlarkValue.asInterface(0x16e38d8, 0xc0002b3400, 0x16e3898, 0xc0002ac4b0, 0x14eb940, 0xc0002b81e0, 0x0, 0x0)
        github.com/k14s/ytt/pkg/template/core/starlark_value.go:108 +0x7ca
     github.com/k14s/ytt/pkg/template/core.StarlarkValue.dictAsInterface(0x16e38d8, 0xc0002b3400, 0xc0002b3400, 0x0, 0x0, 0x1014d00, 0xc0002aea60)
        github.com/k14s/ytt/pkg/template/core/starlark_value.go:122 +0x30d
     github.com/k14s/ytt/pkg/template/core.StarlarkValue.asInterface(0x16e38d8, 0xc0002b3400, 0x16e38d8, 0xc0002b3400, 0x4, 0x4, 0x1, 0x5)
        github.com/k14s/ytt/pkg/template/core/starlark_value.go:93 +0x26e
     github.com/k14s/ytt/pkg/template/core.StarlarkValue.AsGoValue(...)
        github.com/k14s/ytt/pkg/template/core/starlark_value.go:32
     github.com/k14s/ytt/pkg/template.(*EvaluationCtx).TplReplace(0xc00025f080, 0xc00025d540, 0xc000219890, 0xc0002b81c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/k14s/ytt/pkg/template/evaluation_ctx.go:53 +0xfe
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).TplReplaceNode(0xc00027e3c0, 0xc00025d540, 0xc000219890, 0xc0002b81c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x100d73b, ...)
        github.com/k14s/ytt/pkg/template/compiled_template.go:294 +0xb1
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1(0xc00025d540, 0xc000219890, 0xc0002b81c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/k14s/ytt/pkg/template/core/errs.go:28 +0xe6
     github.com/k14s/starlark-go/starlark.(*Builtin).CallInternal(0xc000219890, 0xc00025d540, 0xc0002b81c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x16e38d8, 0x1f03858, ...)
        github.com/k14s/[email protected]/starlark/value.go:653 +0x8b
     github.com/k14s/starlark-go/starlark.Call(0xc00025d540, 0x16e3898, 0xc000219890, 0xc0002b81c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x16e38d8, ...)
        github.com/k14s/[email protected]/starlark/eval.go:1028 +0x19d
     github.com/k14s/starlark-go/starlark.(*Function).CallInternal(0xc0002a3880, 0xc00025d540, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0xc00027e840, ...)
        github.com/k14s/[email protected]/starlark/interp.go:305 +0x4014
     github.com/k14s/starlark-go/starlark.Call(0xc00025d540, 0x16e3918, 0xc0002a3880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3, ...)
        github.com/k14s/[email protected]/starlark/eval.go:1028 +0x19d
     github.com/k14s/starlark-go/starlark.(*Program).Init(0xc00000eaa0, 0xc00025d540, 0xc000219a10, 0x0, 0x0, 0xc0002a3140)
        github.com/k14s/[email protected]/starlark/eval.go:358 +0x89
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).eval(0xc00027e3c0, 0xc00025d540, 0xc000219a10, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/k14s/ytt/pkg/template/compiled_template.go:164 +0x225
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).Eval(0xc00027e3c0, 0xc00025d540, 0x16dd118, 0xc00027e000, 0xc000074640, 0xc00025d540, 0xc000219800, 0x0, 0x0)
        github.com/k14s/ytt/pkg/template/compiled_template.go:121 +0x6e6
     github.com/k14s/ytt/pkg/workspace.(*TemplateLoader).EvalYAML(0xc00027e000, 0xc0000746e0, 0xc0000746e0, 0xc000074640, 0xc00027a590, 0x2, 0x2, 0xc00020d510)
        github.com/k14s/ytt/pkg/workspace/template_loader.go:188 +0x769
     github.com/k14s/ytt/pkg/workspace.(*LibraryExecution).eval(0xc00020d9c0, 0xc00025af80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc00000e528, 0x1, ...)
        github.com/k14s/ytt/pkg/workspace/library_execution.go:234 +0xbb6
     github.com/k14s/ytt/pkg/workspace.(*LibraryExecution).Eval(0xc00020d9c0, 0xc00025af80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/k14s/ytt/pkg/workspace/library_execution.go:186 +0x9e
     github.com/k14s/ytt/pkg/cmd/template.(*Options).RunWithFiles(0xc0001446e0, 0xc0000678d0, 0x2, 0x2, 0x16e2da8, 0xc000189260, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/k14s/ytt/pkg/cmd/template/cmd.go:146 +0x52a
     github.com/k14s/ytt/pkg/cmd/template.(*Options).Run(0xc0001446e0, 0x0, 0x0)
        github.com/k14s/ytt/pkg/cmd/template/cmd.go:93 +0x58b
     github.com/k14s/ytt/pkg/cmd/template.NewCmd.func1(0xc000174a00, 0xc00006aa20, 0x0, 0x6, 0x0, 0x0)
        github.com/k14s/ytt/pkg/cmd/template/cmd.go:58 +0x2a
     github.com/cppforlife/cobrautil.WrapRunEForCmd.func1.1(0xc000174a00, 0xc00006aa20, 0x0, 0x6, 0x0, 0x0)
        github.com/cppforlife/[email protected]/misc.go:45 +0xb2
     github.com/cppforlife/cobrautil.WrapRunEForCmd.func1.1(0xc000174a00, 0xc00006aa20, 0x0, 0x6, 0x0, 0x0)
        github.com/cppforlife/[email protected]/misc.go:45 +0xb2
     github.com/spf13/cobra.(*Command).execute(0xc000174a00, 0xc00001e080, 0x6, 0x6, 0xc000174a00, 0xc00001e080)
        github.com/spf13/[email protected]/command.go:856 +0x472
     github.com/spf13/cobra.(*Command).ExecuteC(0xc000174a00, 0xc000174a00, 0x19862c0, 0x1a70108)
        github.com/spf13/[email protected]/command.go:974 +0x375
     github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:902
     main.main()
        github.com/k14s/ytt/cmd/ytt/ytt.go:21 +0x117
     )

pivotaljohn Nov 2, 2021

Delighted to help. Glad to see you're making way. 👊🏻

Iterating over a Map's Keys

When one iterates over a map (e.g. the map in data.values.request_methods):

...
#@  for method in data.values.request_methods:
...

The value for method will be the key of each entry in that map (e.g. "/foo/bar/foo/bar" <-- string).

This leads to two points:

no need to struct.encode(), the value is already a string 👍🏻
that key (here, the value in method) can be used to look up a value in its map (data.values.request_methods).

Here's that second point illustrated:

config:
#@ for method in data.values.request_methods:
#@   for metric in data.values.request_methods[method]:
- #@ config_headers(method, metric)
#@   end
#@ end

`count` is a Method on Strings

That Go backtrace is happening for a completely unrelated reason.

It turns out that the string type in Starlark has a method on it: count — that counts the number of times a given character occurs within the string (ref: https://carvel.dev/ytt/docs/latest/lang-ref-string/).

In Starlark, functions are values. So writing "something".count("o") invokes count on the string "something", but "something".count is the function-value that — when invoked — would do that counting.

The error message is ytt blowing up attempting to convert a function value into a YAML type (no such thing: 💥 ).

Typically, the way out is to use the "index notation" to refer to the value:

...
_: #@ template.replace(route_config(struct.encode(param), param["count"]))   # better, but still won't work, see below.
...

However, this brings up the next sticking point which is that the contents of each method is an Array, not a Map... so the indexes are integers, not strings.

"metrics" as a Map, Rather Than an Array

In your situation, given that each metric will have a unique name... and with the point above "Iterating over a Map's Keys", simply changing the shape of your data values can really help:

#@data/values
---
request_methods:
  /foo/bar/foo/bar: 
    count: 
      name: 1
      REQUEST_NAME: 2
      tsmMetricKey: 3
      tagKey: 4
      tagValue: 5
      DT_PROD_ENV: 6
    threshold: 
      name: 1
      REQUEST_NAME: 1
      tsmMetricKey: 2
      tagKey: 4
      tagValue: 5
      slo: 6
      DT_PROD_ENV: 1

Given that construction of data values, you can more easily produce the desired output:

#@ for method in data.values.request_methods:
#@   for metric in data.values.request_methods[method]:
_: #@ template.replace(route_config(path_to_key(method, metric), data.values.request_methods[method][metric]))
#@   end
#@ end

Here's all of these suggestions in a Playground instance: https://carvel.dev/ytt/#gist:https://gist.github.com/pivotaljohn/3bed571765df6b833159787dd25bedd7

p.s. Usefulness of `@ytt:struct`

It's interesting to note that — in the end — there's no need for struct.encode().

When writing templates in ytt, picking up the @ytt:struct module is generally not necessary. This is in part because ytt already automatically converts any struct value into YAML (e.g. in the last point, notice how the parameters seem to just fall into place as map items even though, under the covers it's a struct).

In general, the @ytt:struct module's sweet spot is collecting a bunch of values together to pass around within templates. Since that's what the @ytt:data module is doing on your behalf (and is the primary case of this "collecting a bunch of values together"), there's usually not a need for grabbing @ytt:struct, yourself.

mtrin-wooliesx · 2021-11-04T01:05:47Z

mtrin-wooliesx
Nov 4, 2021
Author

Amazing! Thank you so much for this. It really helped me understand more on how feature rich this is.

Last question, if I wanted to build a schema.yaml, how would that work? It doesn't seem to like the fact we have a key as a variable?

Again, thank you so much.

0 replies

pivotaljohn · 2021-11-04T12:13:09Z

pivotaljohn
Nov 4, 2021

Amazing! Thank you so much for this. It really helped me understand more on how feature rich this is.

Woohoo!

Last question, if I wanted to build a schema.yaml, how would that work? It doesn't seem to like the fact we have a key as a variable?

Hmmm. I'm not sure what you're referring to. Can you give an example of what you're attempting?

I was able to take that last Playground link and change "@data/values" to "@data/values-schema" and it just worked, for me.

0 replies

mtrin-wooliesx · 2021-11-09T14:44:52Z

mtrin-wooliesx
Nov 9, 2021
Author

Hello @pivotaljohn sorry for the late followup
I had to put the schema question aside as I've been trying to figure out how to make the next step on my project to work, but I'm struggling. I'm trying to update one of the values on the metric params, because it will always be the same.

https://carvel.dev/ytt/#gist:https://gist.github.com/mtrin-wooliesx/6eb6678a8106c1140c3f4c0c3915c453

I have tried quite a few different things, but here is where I'm at.
By reading starlark and ytt documentation I believe I should be able to use dict.update?

all the best

0 replies

mtrin-wooliesx · 2021-11-11T05:16:04Z

mtrin-wooliesx
Nov 11, 2021
Author

Just for reference, this is what ended up working for me 🙂
thanks for the community!

#@ load("@ytt:data", "data")
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:template", "template")

#@ def path_to_key(name, metric):
#@   return name.replace("/", "_") + "_" + metric
#@ end

config:
#@ for method in data.values.request_methods:
#@   for metric in data.values.request_methods[method]:
#@yaml/text-templated-strings
- REQUEST_RT_SLO(@= path_to_key(method, metric) @): #@ "REQUEST_RT_SLO_" + metric + ".json"
#@   end
#@ end

#@ def additions(method):
#@overlay/match expects=0
REQUEST_NAME: #@ method
#@ end

#@ for method in data.values.request_methods:
#@   for metric in data.values.request_methods[method]:
#@yaml/text-templated-strings
REQUEST_RT_SLO(@= path_to_key(method, metric) @).staging: #@ overlay.apply(data.values.request_methods[method][metric], additions(method))
#@   end
#@ end

#@ for method in data.values.request_methods:
#@   for metric in data.values.request_methods[method]:
#@yaml/text-templated-strings
REQUEST_RT_SLO(@= path_to_key(method, metric) @).prod: #@ overlay.apply(data.values.request_methods[method][metric], additions(method))
#@   end
#@ end

2 replies

pivotaljohn Nov 16, 2021

Ah! WAAAAAAY simpler. 🏆

It's valuable to note the trade-off too: in this situation, looks like there's a minimal amount of formatting, so the changes of writing a template that does not generate YAML is low.... and the more explicit text templating is shining, here.

If there were more layers (e.g. each config has multiple chunks or needs to have more layers explicitly in the template), then staying within structural templating (as shown earlier in this thread) can help avoid irritating bugs that can creep.

pivotaljohn Nov 16, 2021

(related Slack thread: https://kubernetes.slack.com/archives/CH8KCCKA5/p1636508283349700)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manipulating map keys on arrays #529

{{title}}

Replies: 5 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Manipulating map keys on arrays #529

mtrin-wooliesx Oct 30, 2021

Replies: 5 comments · 4 replies

pivotaljohn Nov 1, 2021

mtrin-wooliesx Nov 2, 2021 Author

pivotaljohn Nov 2, 2021

Iterating over a Map's Keys

count is a Method on Strings

"metrics" as a Map, Rather Than an Array

p.s. Usefulness of @ytt:struct

mtrin-wooliesx Nov 4, 2021 Author

pivotaljohn Nov 4, 2021

mtrin-wooliesx Nov 9, 2021 Author

mtrin-wooliesx Nov 11, 2021 Author

pivotaljohn Nov 16, 2021

pivotaljohn Nov 16, 2021

mtrin-wooliesx
Oct 30, 2021

Replies: 5 comments 4 replies

pivotaljohn
Nov 1, 2021

mtrin-wooliesx Nov 2, 2021
Author

`count` is a Method on Strings

p.s. Usefulness of `@ytt:struct`

mtrin-wooliesx
Nov 4, 2021
Author

pivotaljohn
Nov 4, 2021

mtrin-wooliesx
Nov 9, 2021
Author

mtrin-wooliesx
Nov 11, 2021
Author