Generate YAML from Go struct #60

saswatamcode · 2021-07-12T11:54:03Z

This PR allows mdox to generate YAML from Go structs, using the following semantics(for now).
Thus, structs like below,

type Config struct {
	Version int `yaml:"version"`

	Validator []ValidatorConfig `yaml:"validators"`
	Ignore []IgnoreConfig `yaml:"ignore"`
}

type ValidatorConfig struct {
	Type  string `yaml:"type"`
	Regex string `yaml:"regex"`
	Token string `yaml:"token"`

	r *regexp.Regexp
}

type IgnoreConfig struct {
	Url   string `yaml:"url"`
	ID    int    `yaml:"id"`
	Token string `yaml:"token"`
}

Result in markdown like below,

```yaml mdox-gen-go-struct="main.go:Config"
version: 0
validators:
    - type: ""
      regex: ""
      token: ""
ignore:
    - url: ""
      id: 0
      token: ""

```yaml mdox-gen-go-struct="main.go:ValidatorConfig"
type: ""
regex: ""
token: ""

Dependencies added: jennifer, structtag

Few TODOs:

// TODO(saswatamcode): Add tests.
// TODO(saswatamcode): Check jennifer code for some safety.
// TODO(saswatamcode): Add mechanism for caching output from generated code.
// TODO(saswatamcode): Better errors.

Resolves #23.

bwplotka

That's some next level magic

AI programmer!

LGTM, just let's add tests 👍🏽

bwplotka · 2021-08-03T10:44:16Z

pkg/yamlgen/yamlgen.go

+// TODO(saswatamcode): Currently takes file names, need to make it module based(something such as https://golang.org/pkg/cmd/go/internal/list/).
+
+// GenGoCode generates Go code for yaml gen from structs in src file.
+func GenGoCode(src []byte) (string, error) {


Good for start, but I wonder if we can scope to just one desired structure, to limit imports. Maybe it's later optimization - so fine for now

Can we add todo for that?

bwplotka · 2021-08-03T10:46:45Z

pkg/yamlgen/yamlgen.go

+	init = append(init, jen.Id("configs").Op(":=").Map(jen.String()).Interface().Values())
+
+	// Loop through declarations in file.
+	for _, decl := range f.Decls {


I think I would expect some recursion here (:

bwplotka · 2021-08-03T10:48:47Z

pkg/yamlgen/yamlgen.go

+	generatedCode.Func().Id("main").Params().Block(
+		init...,
+	)


Suggested change

generatedCode.Func().Id("main").Params().Block(

init...,

)

generatedCode.Func().Id("main").Params().Block(init...)

bwplotka · 2021-08-03T10:50:43Z

pkg/yamlgen/yamlgen.go

+	cmd = exec.CommandContext(ctx, "go", "mod", "tidy")
+	cmd.Dir = tmpDir
+	if err := cmd.Run(); err != nil {
+		return nil, errors.Wrapf(err, "run %v", cmd)


Suggested change

return nil, errors.Wrapf(err, "run %v", cmd)

return nil, errors.Wrapf(err, "mod %v", cmd)

bwplotka · 2021-08-03T10:50:53Z

pkg/yamlgen/yamlgen.go

+	cmd := exec.CommandContext(ctx, "go", "mod", "init", "structgen")
+	cmd.Dir = tmpDir
+	if err := cmd.Run(); err != nil {
+		return nil, errors.Wrapf(err, "run %v", cmd)


Suggested change

return nil, errors.Wrapf(err, "run %v", cmd)

return nil, errors.Wrapf(err, "mod init %v", cmd)

bwplotka · 2021-08-03T10:53:07Z

pkg/yamlgen/yamlgen.go

+	return checkForOmitEmptyTagOptionRec(reflect.ValueOf(obj))
+}
+
+func checkForOmitEmptyTagOptionRec(v reflect.Value) error {


Do we need to check for emit empty if we fill all the structs? I think in this case we need to also set non default values for other things that slices... so maybe for next iteration

bwplotka

LGTM, let's add recursion and understand what map will work, what not. 👍🏽

bwplotka · 2021-08-24T10:20:58Z

pkg/yamlgen/yamlgen.go

+// TODO(saswatamcode): Currently takes file names, need to make it module based(something such as https://golang.org/pkg/cmd/go/internal/list/).
+
+// GenGoCode generates Go code for yaml gen from structs in src file.
+func GenGoCode(src []byte) (string, error) {


Can we add todo for that?

bwplotka · 2021-08-24T10:25:30Z

pkg/yamlgen/yamlgen_test.go

+	})
+
+	t.Run("struct with unexported field", func(t *testing.T) {
+		source := []byte("package main\n\nimport \"regexp\"\n\ntype ValidatorConfig struct {\n\tType  string `yaml:\"type\"`\n\tRegex string `yaml:\"regex\"`\n\tToken string `yaml:\"token\"`\n\n\tr *regexp.Regexp\n}\n")


I would use ` and add + backtick + , something like that. Still easier than long string manually crafted.

Another idea - put all in files - that would do as well.

Dentrax · 2022-05-10T06:54:47Z

Any update on this? @bwplotka

bwplotka · 2022-05-10T20:39:22Z

I think @saswatamcode has some comments to be addressed, but quite close.

What do you need this for @Dentrax? Curious about your use case and if this PR fixes it (:

Dentrax · 2022-05-11T12:55:18Z

I've created a related issue thanos-io/thanos#4751 (comment) and closed due to inactivity. We want to add support for both comments and default values while generating YAML.

Dentrax · 2023-01-26T06:42:43Z

Kind ping, still looking for this feature! @saswatamcode @bwplotka

bwplotka · 2023-06-07T08:17:24Z

cc @saswatamcode

Signed-off-by: Saswata Mukherjee <[email protected]>

saswatamcode · 2023-06-13T16:36:38Z

I'm trying to look into this, this week and bring it to a nice usable state, rebasing and correcting some of the old code. Plan to clean it up in the next few commits! 🙂

saswatamcode requested a review from bwplotka July 12, 2021 11:54

saswatamcode self-assigned this Jul 12, 2021

saswatamcode force-pushed the struct-yaml branch from 34b021c to 48335e8 Compare August 2, 2021 02:51

bwplotka reviewed Aug 3, 2021

View reviewed changes

saswatamcode force-pushed the struct-yaml branch from 48335e8 to cd9a223 Compare August 23, 2021 08:07

saswatamcode mentioned this pull request Aug 23, 2021

Add yamlgen/cfggen #79

Merged

bwplotka reviewed Aug 24, 2021

View reviewed changes

saswatamcode mentioned this pull request Nov 13, 2021

cfggen: recognize comments in the struct while generating output thanos-io/thanos#4751

Closed

olivierlemasle mentioned this pull request Jan 16, 2023

feature: Get certain structure or function from file to md snippet. #92

Open

saswatamcode added 5 commits June 10, 2023 08:43

Add yamlgen

1507889

Signed-off-by: Saswata Mukherjee <[email protected]>

Add support for array in struct

10983db

Signed-off-by: Saswata Mukherjee <[email protected]>

Add support for string types

f6511ac

Signed-off-by: Saswata Mukherjee <[email protected]>

Implement suggestions; Change cfggen

aa0018e

Signed-off-by: Saswata Mukherjee <[email protected]>

Add basic tests and fix pkg import and struct initialization

021aec7

Signed-off-by: Saswata Mukherjee <[email protected]>

saswatamcode force-pushed the struct-yaml branch from b5520ae to 021aec7 Compare June 13, 2023 16:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate YAML from Go struct #60

Generate YAML from Go struct #60

saswatamcode commented Jul 12, 2021 •

edited

Loading

bwplotka left a comment

bwplotka Aug 3, 2021

bwplotka Aug 24, 2021

bwplotka Aug 3, 2021

bwplotka Aug 3, 2021

saswatamcode Aug 23, 2021

bwplotka Aug 3, 2021

saswatamcode Aug 23, 2021

bwplotka Aug 3, 2021

saswatamcode Aug 23, 2021

bwplotka Aug 3, 2021

bwplotka left a comment

bwplotka Aug 24, 2021

bwplotka Aug 24, 2021

Dentrax commented May 10, 2022

bwplotka commented May 10, 2022

Dentrax commented May 11, 2022

Dentrax commented Jan 26, 2023

bwplotka commented Jun 7, 2023

saswatamcode commented Jun 13, 2023

	return nil, errors.Wrapf(err, "run %v", cmd)
	return nil, errors.Wrapf(err, "mod %v", cmd)

	return nil, errors.Wrapf(err, "run %v", cmd)
	return nil, errors.Wrapf(err, "mod init %v", cmd)

Generate YAML from Go struct #60

Are you sure you want to change the base?

Generate YAML from Go struct #60

Conversation

saswatamcode commented Jul 12, 2021 • edited Loading

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dentrax commented May 10, 2022

bwplotka commented May 10, 2022

Dentrax commented May 11, 2022

Dentrax commented Jan 26, 2023

bwplotka commented Jun 7, 2023

saswatamcode commented Jun 13, 2023

saswatamcode commented Jul 12, 2021 •

edited

Loading