Go modules in conflict with wuffRoot

[elazarl]

Recently, go module was introduced, which sounds like the way to go (pardon the pun) with golang.

However this is in conflict with wuff's design, which assumes some source artifacts are available through $GOPATH/src/github.com/wuffs.

For example, wuffs gen is hardcoded to consume files from wuffs source root.

A more reasonable design would be not to depend on the wuffs source dir in the modules era, but to package required resources in the executable, and accept path for the sources to generate.

Even if wuffsRoot is still deemed necessary, I think we should explicitly copy the required resources to $GOPATH/src in build-all.sh, in order to have a clean copy of cloned wuffs simply work.

[nigeltao]

Yeah, it needs fixing at some point, so that we don't hard-code the wuffs root. But it hasn't been a high priority so far.

Still, I'm not sure that "packaging the required resources" in the executable would work. What resources did you have in mind?

Note that build-all.sh is not part of the regular workflow. It is only run infrequently, typically as a sanity check just before cutting a release.

[elazar-lb]

@nigeltao I mean to use something like go.rice to push a zip file with the current source code. Use a virtual FS in the code that fetches the file from the in-memory zip by default.

I think it should be more robust.

[nigeltao]

By "the current source code", do you mean the *.go code or the *.wuffs code?

If it's the *.go code, then, well, the compiler already bundles it: it's a static binary.

If it's the *.wuffs code, then, well, two points:

1. wuffs gen has to write the output somewhere. If you're configuring the output directory, you might as well configure the input directory too (to the same Wuffs root), so bundling the *.wuffs files into the binary doesn't really win you anything.
2. The long term plan is for wuffs gen your/wuffs/code/outside/of/the/standard/library to work, the same way that go build foo/bar works. Obviously, we can't bundle your/wuffs/code/etc into the binary, so once again, if we'll cater for that, bundling the *.wuffs files into the binary doesn't really win you anything.

[nigeltao]

One more point: the envisaged workflow, when working on the Wuffs standard library, is:

1. Edit std/foo/bar.wuffs
2. Run wuffs test std/foo

Note that, nowhere in that workflow is "rebuild the wuffs binary". Obviously, if we bundled resources, we'd need an extra "rebuild" step.

[elazar-lb]

The vfs could be used only of the wuffRoot is not set.

The development workflow is important, but to gain users I think the user workflow should also be as smooth as possible.

In my eyes the user workflow should be prioritized over the development workflow, but this is irrelevant here, since they're not in conflict.

[nigeltao]

The vfs could be used only of the wuffRoot is not set.

How does this alleviate "wuffs gen has to write the output somewhere"? You can't write to the VFS bundled into the binary.

I still don't know if, by "bundling resources", you mean *.go files, *.wuffs files, or something else.

[elazar-lb]

A user is expecting the wuffs binary to simply work on his code, you can choose to have an obsession process like go does, that puts wuff source and binary in a known path, and infers the source path.

Or bundle out as I suggested. Or maybe other solution.

I must reflect to you, that as someone trying to promote wuff in the organization, the current state of affair is not helping me convincing others that wuff is a stable product ready to consume.

It might be OK for you, as wuff might be for internal use only, and you might not care, but I'm just reflecting my experience to you.

[elazar-lb]

Wuff gen should accept a path, or work on the current directory, like `go build ` does. I meant packaging the wuff source required. So that wuff gen `my-file-just using-wuff.wuff` would just work.

…

On Thu, Jun 13, 2019, 08:37 Nigel Tao ***@***.***> wrote: The vfs could be used only of the wuffRoot is not set. How does this alleviate "wuffs gen has to write the output somewhere"? You can't *write* to the VFS bundled into the binary. I still don't know if, by "bundling resources", you mean *.go files, *.wuffs files, or something else. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AJ5HAWFYDUGKLFHV7XB52BTP2HMIPA5CNFSM4HSCELZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXSSDBI#issuecomment-501555589>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ5HAWGXVBSG7ECZUMW5PBLP2HMIPANCNFSM4HSCELZA> .

[nigeltao]

If you aren't already aware, you should be able to do something like:

wuffs-c gen -package_name foo foo/*.wuffs > foo.c

That might solve what your actual problem is. Note that it's wuffs-c, not wuffs.

[nigeltao]

Wuff gen should accept a path, or work on the current directory, like go build does.

Well, I already said that:

The long term plan is for wuffs gen your/wuffs/code/outside/of/the/standard/library to work, the same way that go build foo/bar works.

It just hasn't been the highest priority feature on the long list of things to work on so far.

[elazar-lb]

I'm not aware of that, that should be fine. Still I'm not sure what wuff is useful for, is it only for the development of wuff? As I said, I expected `wuff gen` to just work in a directory with wuff files, like go always did. At any rate, if wuff-c is what a user who do not intend to hack on wuff, is expected to use, I suggest to document out clearly in the beginning of the README file.

…

On Thu, Jun 13, 2019, 08:44 Nigel Tao ***@***.***> wrote: If you aren't already aware, you should be able to do something like: wuffs-c gen -package_name foo foo/*.wuffs > foo.c That might solve what your actual problem is. Note that it's wuffs-c, not wuffs. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AJ5HAWHMTWWVPIPZL5ZNA5DP2HNCNA5CNFSM4HSCELZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXSSNCY#issuecomment-501556875>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ5HAWAQEYNVU4LPDUGGBG3P2HNCNANCNFSM4HSCELZA> .

[elazar-lb]

Bottom line, would you accept patch checking for wuffs path in the current dir of the binary by default, and a clear explanation that users are expected to develop their wuff file in tree for the time being?

…

On Thu, Jun 13, 2019, 08:48 Nigel Tao ***@***.***> wrote: Wuff gen should accept a path, or work on the current directory, like go build does. Well, I already said that: The long term plan is for wuffs gen your/wuffs/code/outside/of/the/standard/library to work, the same way that go build foo/bar works. It just hasn't been the highest priority feature on the long list of things to work on so far. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AJ5HAWFWSQB4KMKEMSRLUETP2HNRLA5CNFSM4HSCELZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXSSTUA#issuecomment-501557712>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ5HAWEJCGNYGWENAQII273P2HNRLANCNFSM4HSCELZA> .

[nigeltao]

Still I'm not sure what wuff is useful for, is it only for the development of wuff?

"Wuffs" is an overloaded term:

• It is a command line tool: the wuffs in wuffs gen etc and wuffs test etc.
• It is a programming language: the language that the *.wuffs files are written in.
• It is a standard library, written in Wuffs (the programming language) and transpiled to C.

Other C/C++ projects can use Wuffs (the standard library), without caring at all about Wuffs (the command line tool) or Wuffs (the programming language). For example, Skia (a C/C++ project) uses Wuffs (the standard library), without requiring Wuffs (the command line tool) or in fact any Go code in this repository: https://github.com/google/skia/blob/master/third_party/wuffs/BUILD.gn

When you ask what wuffs is useful for, which wuffs are you referring to?

The initial focus (remember that we have still not hit a version 0.2 release) is to develop a useful standard library. The priority for Wuffs (the command line tool) is to support developing a useful standard library.

As I said, long term, we'd like to have Wuffs (the command line tool) work with third party Wuffs (the programming language) code, but as I again said, it hasn't been the highest priority feature.

---

Bottom line, would you accept patch checking for wuffs path in the current dir of the binary by default, and a clear explanation that users are expected to develop their wuff file in tree for the time being?

I'm open to having Wuffs (the command line tool) work better for third party foo/bar.wuffs code, but I'm still not exactly sure what you are proposing: does "develop their wuff file in tree" mean in the Wuffs tree or in a separate tree? If it's in the Wuffs tree, why do you need to specify the CWD, instead of automatically finding the Wuffs tree?

In any case, it might not be as trivial as it first seems, if you have multiple third party Wuffs (the programming language) packages, some of which depend on others, or depend on standard library packages.

[nigeltao]

BTW, the overloaded term is "wuffs" with an "s", not "wuff".

[nigeltao]

I must reflect to you, that as someone trying to promote wuff in the organization, the current state of affair is not helping me convincing others that wuff is a stable product ready to consume.

To set expectations: it is not a stable product, certainly for third party *.wuffs code. As I said, we have still not hit a version 0.2 release, and the immediate focus is for first party (standard library) *.wuffs code.

Even after v0.2 is released, the language and the tools will undoubtedly change, in backwards incompatible ways.

I have users, but they are using the standard library (in its "generated C code" edition). They are not writing their own *.wuffs code. Also, once a final release/c/wuffs-v0.2.c is cut, it won't change (other than bug fixes), even if Wuffs (the programming language) changes after v0.2. The standard library "generated C code edition" will be relatively stable; the Wuffs language and tools will be relatively unstable.

OTOH, longer term, I'd like to support third party *.wuffs code, and I'm curious what you want to write in Wuffs (the programming language). That might be better discussed at #13. Note that you'd be one of the earliest adopters, AFAIK, which is why you're finding the tools so unsupportive.

[elazar-lb]

My understanding of the project, since it was called puffs, is that the main product is the wuffs programming language. User who needs a parser should write their wuffs files, and generate a safe portable parser.

Do you mean by "standard library" "wuffs code compiled into C from the standard library"?

What wuffs offers now is the set of safer parsers already exist in the standard library? I didn't understand that from the README, but I think I now have a better understanding of how do you expect wuffs to be consumed.

Non-backwards compatibility is not a problem for me, but ease of usage is. If one want to write a safe binary parser in wuffs, it must be easy for the team to, at least, use it. But I now understand that the wuffs language is only meant to be used internally at this point.

[nigeltao]

the main product is the wuffs programming language

A product is the programming language, but another product is a C library, with better safety guarantees than almost all C libraries, which can be used in existing C/C++ projects without requiring their build systems to incorporate the Wuffs and Go toolchains. People can use the second product without the first, and vice versa.

So far, I have more users for the second product than the first product. That's not to say that the first product isn't useful or kept in mind, just that the second product, being where my users have been, has been the higher priority.

Do you mean by "standard library" "wuffs code compiled into C from the standard library"?

The standard library is the *.wuffs code in the std directory:
https://github.com/google/wuffs/tree/master/std

It is also available in its "generated C code" edition, in the release/c directory:
https://github.com/google/wuffs/tree/master/release/c

What wuffs offers now is the set of safer parsers already exist in the standard library?

The standard library is one of the products on offer: the second product mentioned above. The first product is still there, but hasn't received as much attention as the second one.

I'll admit that the README could be clearer about this.

the wuffs language is only meant to be used internally at this point.

I wouldn't say "meant to", just that that's where the customers (so far) have been.

---

Bottom line, from what I know so far of your project, I think you'd be better off invoking wuffs-c instead of wuffs. wuffs-c doesn't really care about the internal / external distinction.