Uncertain how to effectively use gx-workspace for go-ipfs

[rob-deutsch]

This could be simple, but I'm having problems using gx-workspace with go-ipfs.

It's probably PEBCAC, but if I can get my head around it I'll happily write some documentation.

What I'm Trying To Do

I've made some changes to ipfs/go-ipld-format, and I want to incorporate these into a new ipfs/go-ipfs binary for testing.

The changes to ipfs/go-ipld-format are going to impact a few other packages in a small way, but I'm not sure how yet.

How I'm doing this

These are the steps I'm using:

1. Make the changes to ipfs/go-ipld-format and run gx-workspace release
2. Go to ipfs/go-ipfs and run gx-workspace update start ipfs-ipld-format
3. Repeatedly run gx-workspace update next until it errors for a certain package.
4. Fix that certain package.
5. Go back to ipfs/go-ipfs and continue running gx-workspace update next

The problem

Firstly, I have verified that PRIOR to running gx-workspace I am able to build and install ipfs/go-ipfs.

However, after doing the gx-workspace procedure I described above I have two problems:

1. After fixing the certain package, and going back to ipfs/go-ipfs to run gx-workspace update next I can't help but notice that gx-workspace claims that the fixed package doesn't need to be updated. Is this maybe because it had previously updated package.json?

2. When I try to make install or make build my new ipfs/go-ipfs package I get a huge amount of errors about missing packages. Things like, for example github.com/Kubuxu/go-os-helper

[rob-deutsch]

Ah! I found part of the problem, but I dont understand why it's occurring.

For some reason gx-workspace replaced (for example) ipfs/go-ipfs/blob/master/cmd/ipfs/main.go#L33.

It started as: osh "gx/ipfs/QmXuBJ7DR6k3rmUEKtvVMhwjmXDuJgXXPUt4LQXKBMsU93/go-os-helper"
But after running gx-workspace it was: osh "github.com/Kubuxu/go-os-helper"

Running the following command fixed it and allowed ipfs/go-ipfs to compile:

gx-go rewrite

[schomatis]

It's probably PEBCAC, but if I can get my head around it I'll happily write some documentation.

Hey @rob-deutsch, so, first things first, if you're willing to write documentation about anything related to Gx I can definitely commit to helping you through any problem you may have :)

(Btw, PEBCAC 😄, didn't knew this one)

I've made some changes to ipfs/ipfs-ipld-format, and I want to incorporate these into a new ipfs/go-ipfs binary for testing.

It sound to me that gx-go link would be the suitable tool for the job, that is, make changes to a package and propagate them locally to perform some tests, gx-workspace (which I haven't used yet so I'm no expert here) is used when you're done with the changes and want to publish them to every package that depends on it to commit the changes in a PR (this is probably inaccurate, the gist of it is that you do it when you're done making changes and want to make them globally public).

The changes to ipfs/ipfs-ipld-format are going to impact a few other packages in a small way, but I'm not sure how yet.

Yes, this is never easy to know in advance, we're working on a new iteration of Gx that will simplify that task, quoting another developer:

Basically, for me at least, the biggest time-sink is the lack of a dependency resolution system.

It started as: osh "gx/ipfs/QmXuBJ7DR6k3rmUEKtvVMhwjmXDuJgXXPUt4LQXKBMsU93/go-os-helper"
But after running gx-workspace it was: osh "github.com/Kubuxu/go-os-helper"

Running the following command fixed it and allowed ipfs/go-ipfs to compile:

gx-go rewrite

So, go-ipfs is the only repo (I think) that has the Gx paths hard-coded (another thing we're working to fix) and I suppose (emphasis on that word because from this point forward I have no idea what I'm talking about) gx-workspace wants to undo what it is assuming has been rewrriten as part of the development process, but in this particular case go-ipfs has those Gx paths from the get-go, so they shouldn't be un-rewritten.

So my recommendation is, before worrying about how to publish you're modifications, while you're developing and experimenting with (what I'm assuming you meant to write) go-ipld-format is to use the gx-go link command to make a local link between your git-cloned go-ipld-format repo and the go-ipfs repo so any changes you make to the first are reflected in the second automatically without having to repeatedly perform the gx-workspace update steps you listed above every time you make a change.

[schomatis]

Aaaand I just realized there is no gx-go link documentation in the gx-go page.. and the command doc is not really that useful, see what I mean when I say any documentation is extremely welcome 😁 (especially since I've been opening Gx documentation issues for two months now but haven't actually provided any new documentation..)

So, you would do something like (untested):

go get go-ipld-format
cd go-ipfs
gx-go link go-ipld-format
# linked QmQA5mdxru8Bh6dpC9PJfSkumqnmHgJX7knxSgBo5Lpime /home/user/go/src/github.com/ipfs/go-ipld-format

# And now the directory `$GOPATH/src/gx/ipfs/<long-hash>/go-ipld-format`
# that would normally be used by `go-ipfs` will point to the directory where
# you're `go-ipld-format` git-cloned repo is.

Also you can check the doc in a PR that modifies the gx-go link a bit and adds some more documentation to it.

Ping me if you have any problems.

[rob-deutsch]

Thanks for the responses! I've sectioned my reply below to make it a bit easier to parse

My issue, gx-go link, and documentation

Thanks for the info on gx-go link! I knew that such a thing should exist, but had no idea that it did.

At the moment I'm thinking that the most valuable piece of documentation, and a good starting point, would be a "Want to hack on go-ipfs? This HOWTO explains the development process!"

I'm thinking it could be a Markdown file in the ipfs repo.

Does this sound reasonable to you? I'm happy to bring this up (and volunteer, if someone can review it) on the go-ipfs page.

go-ipfs and hardcoded links

So, go-ipfs is the only repo (I think) that has the Gx paths hard-coded (another thing we're working to fix)

Isn't this easy to fix? Just run gx-go rewrite --undo and everything will be golden! I assume I'm missing something?

[schomatis]

So, that file already exists, although more in spirit than in actual text, so any contributions you could make to it would be extremely valuable, especially coming from the perspective of a new developer. When I first read your issue I though "how could he possible have started with gx-workspace instead of gx-go link, that so is anti-intuitive", but then I remembered than when I started working with go-ipfs the gx-go link tool was personally suggested to me in an unrelated issue (whyrusleeping/gx#161) when I didn't know such thing existed, and that's why I have it so present in my mind when developing, but we actually are doing very little to divulge this tool and how important it is (especially now that so many packages are being extracted from the main go-ipfs repo). Anyway, submit a PR about it and ping me, I'll help you write it and review it.

Isn't this easy to fix? Just run gx-go rewrite --undo and everything will be golden! I assume I'm missing something?

It would seem so but no: ipfs/kubo#4831, but we're getting there! (ipfs/kubo#5435)

[schomatis]

I'm going to close this issue now but feel free to raise new issues in https://github.com/whyrusleeping/gx-go if you have trouble with the link command.