Prepare for Yotta Registry deprecation

[jaustin]

In order to make life easy for users of these targets once the yotta registry goes away, we need to change the way people use them.

This requires at least two changes...

The most simple change is that that target 'inheritance' on mbed-gcc needs to move to a GitHub link - for example!

diff --git a/bbc-microbit-classic-gcc/target.json b/bbc-microbit-classic-gcc/target.json
index beb35e2..ce20ab0 100644
--- a/bbc-microbit-classic-gcc/target.json
+++ b/bbc-microbit-classic-gcc/target.json
@@ -9,7 +9,7 @@
     }
   ],
   "inherits": {
-    "mbed-gcc": "0.1.3"
+    "mbed-gcc": "ARMmbed/target-mbed-gcc#v0.1.3"
   },
   "keywords": [
     "mbed-target:nrf51822",

This works fine

Secondly, we need to change how people select the target.

Yotta supports (at least) two ways to specify targets without using the registry

1. yotta link-target - this is really intended for local dev
2. yotta target targetname@location where location can be GitHub (yt target mbed-gcc@ARMmbed/target-mbed-gcc#0.1.3) or I believe a git@URL too which might be more resilient to GitHub auth going away (although, auth is only required for public private repos, I think?)

I've just tested method one with the patch above and it builds fine, as far as I can tell without touching the registry, but once we've picked an option I'll kill the registry in my hosts file to check.

The former is more involved as it requires checking out more than just microbit-samples. The latter would require splitting up this repo.

I would prefer to split this repo. I'd consider also creating a standalone bbc-microibit-classic-gcc and bbc-microbit-classic-gcc-nosd as standalones and then renaming this to microbit-extra-targets and require yt link-target with these (removing the now-standalone targets and leaving pointers). FWIW I don't think the armcc one builds.

@jamesadevine @finneyj @carlosperate thoughts? Other projects that will need to change their build instructions are MicroPython and MakeCode. I think the minimal change for them is changing the string input to yt target

CC @mmoskal @thegecko @dpgeorge

[jamesadevine]

@jaustin So a user would have to clone the target repository and run yotta link-target in that target repository?

[carlosperate]

2. yotta target targetname@location where location can be GitHub (yt target mbed-gcc@ARMmbed/target-mbed-gcc#0.1.3) or I believe a git@URL too which might be more resilient to GitHub auth going away (although, auth is only required for public repos, I think?

Did you mean "only required for private repos"?

GitHub Auth goes via Yotta servers, and it trigger after reaching a number of fetches from the GitHub API without authenticating. So if the "login with GH" Yotta service goes away then people won't be able to continue fetching dependencies.
When building MicroPython for v1 on a VM or docker clean slate, I generally end up needing to login on the 2nd or 3rd build (fetching all dependencies every time), so it doesn't take that much to need auth.

On the other hand git@url should just work, as that is just a normal git clone.

I guess yotta could be updated to fallback into git@urls when the GitHub API starts to rate-limit. Also, although less discoverable, and less "new developer friendly", it would also be useful if yotta could fetch a Github token from something like an environmental variable, this way a developer can generate one and continue using yotta with GitHub HTTPS URLs without issue.

[jaustin]

@jamesadevine

@jaustin So a user would have to clone the target repository and run yotta link-target in that target repository?

Only in option one, which I don't like much as option 2.
@carlosperate

> 2. yotta target targetname@location where location can be GitHub (yt target mbed-gcc@ARMmbed/target-mbed-gcc#0.1.3) or I believe a git@URL too which might be more resilient to GitHub auth going away (although, auth is only required for public repos, I think?

Did you mean "only required for private repos"?

Whoops, I do. Edited. Thanks

When building MicroPython for v1 on a VM or docker clean slate, I generally end up needing to login on the 2nd or 3rd build (fetching all dependencies every time), so it doesn't take that much to need auth.

I think that makes git@ superior here. Also clearer what the actual source is. The format I showed is a bit 'magic'

I'd say then, preference is

option 2 with git@URLs all the way down (up?) the dependency chain. This would still require breaking up this repo.

[carlosperate]

I think that makes git@ superior here. Also clearer what the actual source is. The format I showed is a bit 'magic'

Yes, also I think (I might be wrong, so take this with a pinch of salt) the ARMmbed/target-mbed-gcc#v0.1.3 format first checks the Yotta registry and then the GitHub API.

It's probably safe to assume that current Yotta will be able to deal correctly with 400 or 500 errors from the Yotta registry when it disapears, but git@ should bypass all that anyway.

[jamesadevine]

git@ sounds good @jaustin!

[finneyj]

Thanks all. Yes, agree, git@... looks like the best option.

[finneyj]

How about:

lancaster-university/yotta-target-bbc-microbit-classic-gcc
lancaster-university/yotta-target-bbc-microbit-classic-gcc-nosd

[jaustin]

Okay, a bit of testing here shows that if you've never configured ssh for Github access, using the git@ URLS means you have have to do that. Even for public repositories it fails for a user with a blank .ssh directory.

I think we should use the API for now, for a much smoother experience. I have used yt logout to make sure I'm not relying on the yt login to fetch from GitHub and it's still working.

[carlosperate]

Okay, a bit of testing here shows that if you've never configured ssh for Github access, using the git@ URLS means you have have to do that. Even for public repositories it fails for a user with a blank .ssh directory.

Do you have a PR to test?

I think we should use the API for now, for a much smoother experience. I have used yt logout to make sure I'm not relying on the yt login to fetch from GitHub and it's still working.

How many times were you able to git clone and build before requiring login?

[jaustin]

I've just been applying the following diff

diff --git a/bbc-microbit-classic-gcc/target.json b/bbc-microbit-classic-gcc/target.json
index beb35e2..06680d0 100644
--- a/bbc-microbit-classic-gcc/target.json
+++ b/bbc-microbit-classic-gcc/target.json
@@ -9,8 +9,8 @@
     }
   ],
   "inherits": {
-    "mbed-gcc": "0.1.3"
-  },
+    "mbed-gcc": "[email protected]:microbit-foundation/target-mbed-gcc.git#v0.1.3"
+},
   "keywords": [
     "mbed-target:nrf51822",
     "mbed-official",

Then I'm using yt link-target to use the local change and testing

I think we should use the API for now, for a much smoother experience. I have used yt logout to make sure I'm not relying on the yt login to fetch from GitHub and it's still working.

How many times were you able to git clone and build before requiring login?

I'll try next, but based on https://docs.github.com/en/free-pro-team@latest/github/using-git/which-remote-url-should-i-use#cloning-with-ssh-urls I think that confirms that they expect SSH URLs require a key pair. I don't know how it's working for you in places where you don't have keys? Perhaps SSH forwarding with SSH agent?

If that's the case, what other options do you think we have? I've tried with HTTPS and that gets converted to API requests.

[jaustin]

Okay @abchatra and @pelikhan I've just done the following

• Updated inheritance to use GitHub not registry for mbed-gcc
• Broken out the bbc-microbit-classic-gcc target to it's own repo

Done a clean build of microbit-samples.

Where you would previously have done
yt target bbc-microbit-classic-gcc you now need to do
yotta target bbc-microbit-classic-gcc@https://github.com/lancaster-university/yotta-target-bbc-microbit-classic-gcc

This built for me.
@carlosperate you can try things on #7 with yt link-target. If the single broken out registry works for others I'll do the nosd target too for MicroPython