Testnet RPCs under ubq.fi domain

[0x4007]

If we want to set up the temp Anvil deployments under a project domain then there is some setup required with the Cloudflare account that has that domain registered: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/

This action shows the secrets needed:
https://github.com/AnimMouse/setup-cloudflared

Originally posted by @acaldas in #442 (comment)

[0x4007]

This bounty will require another bounty hunter to set it up on their own cloudflare instance, and then walk me through how to set it up on our website.

[Keyrxng]

I was dabbling around with this today and made a bit of progress but hit a wall with Foundry.

Tunnel set up and connected to instance and hosted on my domain but any interaction with it always returned "Connection header did not include 'upgrade'". Which I believe is coming from ws.rs in the Anvil crate. Tried to reverse proxy a connection but met with same error.

The error is mentioned once as far as I can see but for something I feel has no grounds here, (Foundry#2978).

Once I figure a way round that, connecting and exposing it through the tunnel should be straightforward.

[0x4007]

I would suggest logging the output of the current request headers and compare them to functional request headers.

[gitcoindev]

I did an investigation, before any work can be done we must know which https://www.cloudflare.com/plans/#overview ubiquity is using, or going to use. For free accounts only full DNS management on Cloudflare side is available. This means that ubq.fi would have to transfer and reconfigure DNS.

For paid options the following is available https://developers.cloudflare.com/dns/zone-setups/

Are you on a Free or Pro plan?
If you are on a Free or Pro plan, full setup is the only one available. This is the recommended and most common option.

Will you be using Cloudflare with other DNS providers?
If you are on a Business or Enterprise plan, you can use partial (CNAME) setup to keep your primary DNS provider and only proxy individual subdomains through Cloudflare.

If you are on an Enterprise plan, you also have the option to use zone transfers to set up Cloudflare as either a primary or a secondary DNS provider.

Do you need to manage subdomains separately?
If you are on an Enterprise plan, you can use subdomain setup to manage the Cloudflare settings for one or more subdomains separately from your domain apex.

Perhaps this is also why some features did not work for @Keyrxng . If full DNS was used on a private domain then this is another case.

@pavlovcik which option would be considered here? An ubq.fi subdomain would require paid services. A standalone new domain could be managed for free, if bought separately and fully DNS configured by Cloudflare.

[0x4007]

We can use another domain, we have a few.

[gitcoindev]

/start

[ubiquibot]

Deadline	Tue, 03 Oct 2023 08:48:35 UTC
Registered Wallet	0x7e92476D69Ff1377a8b45176b1829C4A5566653a

Tips:

• Use /wallet 0x0000...0000 if you want to update your registered payment wallet address @user.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the bounty.

[gitcoindev]

Good news, I was able to make this running under my own domain. I had to wait 24 hours to transfer DNS servers from my domain provider (OVH) to Cloudflare, then I configured a simple Python server, then Anvil on my Cloudflare tunnel connected to a custom domain.

I will now open a PR with an example solution, provide QA and describe all steps needed to configure.

[gitcoindev]

Pull request opened.

QA: gitcoindev#1 (comment) (and comments below).

[gitcoindev]

Instructions

At first, it is important to understand that the whole setup consists of three independent steps.

Transferring DNS servers from current provider to Cloudflare

This may take 24 hours (it took exactly 24 hours for me). Transferring requires a prerequisite: disable DNSSEC, then follow instructions and put exact server names as provided by Cloudflare:

change this on your local domain provider as in the example below:

Then wait up to 24 hours, until a confirmation e-mail is received in the mailbox:

[gitcoindev]

Setting up the tunnel to a custom domain

The second step is to set up a Cloudflared tunnel to the custom domain (Anvil exposed on 8545 port in our case) and create configuration files and GitHub action secret variables.

Important notice: creating a tunnel means that Cloudflare will always redirect it, and if Anvil is not running, it will simply return 1033 error code.

In order to set up the tunnel from command line, one needs to install cloudflared daemon. Instructions are provided at:

https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-local-tunnel/

After the daemon is installed, and your free Cloudflared account was created before, it is needed to log in (the URL will be different for you):

$ cloudflared tunnel login
A browser window should have opened at the following URL:

https://dash.cloudflare.com/argotunnel?aud=&callback=https%3A%2F%2Flogin.cloudflareaccess.org%2FjqfYjgnRQJQi_VSi7qR-Neyj2Kvbt6C7k1mbbBvXOAE%3D

If the browser failed to open, please visit the URL above directly in your browser.

then cloudflared tunnel create command can be used to create a tunnel.

I used 'anvil' as name of the tunnel. This means that if a custom domain is used e.g. pavlovcik.com , the tunnel will be available as anvil.pavlovcik.com

$ cloudflared tunnel create anvil
Tunnel credentials written to /home/korrrba/.cloudflared/9d0f8828-8649-4de9-a6b7-15425fb99200.json. cloudflared chose this file based on where your origin certificate was found. Keep this file secret. To revoke these credentials, delete the tunnel.

Created tunnel anvil with id 9d0f8828-8649-4de9-a6b7-15425fb99200

Each tunnel has a name, and a unique ID which in my case was 9d0f8828-8649-4de9-a6b7-15425fb99200.

Creating a tunnel can be verified with a cloudflared tunnel list command:

$ cloudflared tunnel list
You can obtain more detailed information for each tunnel with `cloudflared tunnel info <name/uuid>`
ID                                   NAME  CREATED              CONNECTIONS 
9d0f8828-8649-4de9-a6b7-15425fb99200 anvil 2023-10-02T07:38:10Z 

Now, an important step. We have a named tunnel with a unique UUID, and now have to connect it to a custom domain.

This can be achieved with cloudflared tunnel route dns command:

$ cloudflared tunnel route dns 9d0f8828-8649-4de9-a6b7-15425fb99200 anvil.yourdomain.xyz
2023-10-02T08:32:16Z INF Added CNAME anvil.yourdomain.xyz which will route to this tunnel tunnelID=9d0f8828-8649-4de9-a6b7-15425fb99200

Now after pointing web browser to the URL provided, one should be able to see

And this is very good news, as it confirms that a) tunnel was created, b) domain works correctly.
It is now time to set up GitHub action with running Anvil on port 8545 (or any other network) to connect to the same tunnel.

[gitcoindev]

Preparing secret variables to connect to Cloudflared tunnel via GitHub actions

Three secret variables must be created in order to connect correctly:

CLOUDFLARE_TUNNEL_ID

This variable holds UUID of the tunnel name. Simply copy UUID and paste it to CLOUDFLARE_TUNNEL_ID secret.

CLOUDFLARE_TUNNEL_CREDENTIAL

This variable contains secret credentials in uuid json file, in base64 encoding. Use base64 tool like in the example below, just replace UUID.

$ base64 -w 0 ~/.cloudflared/9d0f8828-8649-4de9-a6b7-15425fb99200.json 
eyJBY2NvdW50VGFnIjoiNDY3ZGQ1ZTJkOGYxYWJkNWFjNjczMTM3NzhmOTg2MzkiLCJUdW5uZWxTZWNyZXQiOiIrMWRk...
EIjoiOWQwZjg4MjgtODY0OS00ZGU5LWE2YjctMTU0MjVmYjk5MjAwIn0

Copy output of base64 command into CLOUDFLARE_TUNNEL_CREDENTIAL secret in GitHub.

CLOUDFLARE_TUNNEL_CONFIGURATION

This variable holds base64 encoded tunnel configuration file. First , create a text file with the following content , and replace uuid with the one generated:

url: http://localhost:8545
tunnel: 9d0f8828-8649-4de9-a6b7-15425fb99200
credentials-file: /home/runner/.cloudflared/9d0f8828-8649-4de9-a6b7-15425fb99200.json

This configuration says: take whatever is running on port 8545 (e.g. Anvil) and connect tunnel with given UUID to it.

After file is created, base64 encode it:

$ base64 -w 0 config.yaml 
dXJsOiBodHRwOi8vbG9jYWxob3N0Ojg1NDUKdHVubmVsOiA5ZDBmODgyOC04NjQ5LTRkZTktYTZiNy0xNTQyNWZiOTkyMDAKY3JlZGVudGlhbHMtZmlsZTogL2hvbWUvcnVubmVyLy5jbG91ZGZsYXJlZC85
...
DBmODgyOC04NjQ5LTRkZTktYTZiNy0xNTQyNWZiOTkyMDAuanNvbgo=

and paste into CLOUDFLARE_TUNNEL_CONFIGURATION secret.

When those steps are complete, all required secrets should be set up:

[gitcoindev]

Configuring a GitHub Action workflow to use the tunnel

The last step is to configure a GitHub Action workflow that will connect the tunnel to test RPC like in the PR #799

I also added there comments to QA, which can be removed.

When the action is running, the tunnel is active and testnet RPC can be accessed with a custom domain, like in my QA gitcoindev#1 (comment) . I later replaced my domain name with domain.xyz just to show the example.

QA: accessing deployed GitHub actions Anvil instance ona custom domain, mining two blocks and getting the eth_blockNumber:

$ curl --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST https://anvil.domain.xyz
{"jsonrpc":"2.0","id":1,"result":"0x0"}10:43:56 ~$ 
$ curl --data '{"method":"evm_mine","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST https://anvil.domain.xyz
{"jsonrpc":"2.0","id":1,"result":"0x0"}10:44:19 ~$ 
$ curl --data '{"method":"evm_mine","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST https://anvil.domain.xyz
{"jsonrpc":"2.0","id":1,"result":"0x0"}10:44:21 ~$ 
$ curl --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST https://anvil.domain.xyz
{"jsonrpc":"2.0","id":1,"result":"0x2"}10:44:41 ~$ 

This sums up the instructions.

[rndquu]

We can use another domain, we have a few.

So we have 3 options:

1. Transfer ubq.fi DNS servers to cloudflare
2. Use some other domain (which uses cloudflare's DNS servers) for RPC testnets
3. Use cloudflare's paid plan to use a subdomain of ubq.fi for RPC testnets

As far as I understand we're going to use a separate domain. @pavlovcik You can either transfer a new domain yourself or somehow give me access on cloudflare and I'll handle it.

[molecula451]

i see so this is at the cloudfare level

[molecula451]

I think pavlovcik will still need help to set up this after all the instructions, we could either allow a partial permit and give time to pavlovcik to face all the stuff, or best to pass it to rndqnuu hands (cloudflare), setup, test, QA and mark as complete

[molecula451]

the instructions seems solid, altho the 24 hours to debug after setting up

[molecula451]

Let's mark this as completed in favor of #799 (a utility reference PR, that will be closed), but serves for the tasks purpose and this issue

[ubiquibot]

Task Assignee Reward

[ CLAIM 200 WXDAI ]

0x7e92476D...A5566653a

If you've enjoyed your experience in the DevPool, we'd appreciate your support. Follow Ubiquity on GitHub and star this repo. Your endorsement means the world to us and helps us grow!

We are excited to announce that the DevPool and UbiquiBot are now available to partners! Our ideal collaborators are globally distributed crypto-native organizations, who actively work on open source on GitHub, and excel in research & development. If you can introduce us to the repository maintainers in these types of companies, we have a special bonus in store for you!

[ubiquibot]

Task Creator Reward

pavlovcik: [ CLAIM 25.2 WXDAI ]