Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: advertise http endpoint as extended provider #27

Merged
merged 11 commits into from
May 19, 2023
Merged

Conversation

olizilla
Copy link
Contributor

@olizilla olizilla commented May 5, 2023

Add support for announcing special ExtendedProvider advertisements. The goal is to advertise our content as being available over bitswap and http.

This implementation hardcodes support for processing messages with the magic string ExtendedProviderHTTP which is a signal to publish the next advert in the chain as having the bitswap & http provider details in an ExtendedProvider block.

No context or entries are specified on that message so the indexer should interpret it as a request to announce our extra http provider info for all previous and future entries published on this chain.

The logic is hardcoded as the Advertisement must be signed with the private keys of each of the providers. There is no way to pre-sign the extra provider info as the previous advert CID is part of the signature, which we can't guess ahead of time. It is assumed that we will only use this specific code path once (1) (!) to announce this http endpoint. We could use it to announce another http endpoint by updating the env, and the peerId-http.json, but in general this whole repo should be replaced with an sst managed esm implementation before that happens.

There is currently no support for removing provider info, so we need to be happy to support our http endpoint domain for some time until ipni/storetheindex#1745 lands

The logic of how to serialise and sign advertisements is extracted out to our ipni module: https://github.com/web3-storage/ipni

NOTE This module needs refactoring and updating to the (dag) haus style. I have not tackled that at part of this PR, but have added js-doc and updated modules where i had to.

License: MIT

@olizilla olizilla force-pushed the extended-provider branch 2 times, most recently from 64b2e35 to ecf33b3 Compare May 5, 2023 12:46
This is a first pass at adding http as provider to our advertisments for the IPNI indexers. The goal is to advertise our content as being available over bitswap and http.

**NOTE** This module needs refactoring and updating to the (dag) haus style. I have not tackled that yet.
This PR is currently makes only the minimal changes required to add Extended Providers.

The logic of how to serialise and sign a Provider was derivied from the exiting signing code in this module and the implementation in go-libipni

see: https://github.com/ipni/go-libipni/blob/afe2d8ea45b86c2a22f756ee521741c8f99675e5/ingest/schema/envelope.go#L125-L167

License: MIT
Signed-off-by: Oli Evans <[email protected]>
Comment on lines 104 to 114
async function providerSignature (previous, peerId, cid, contextId, provider, extendedProviderOverride = false) {
const sigBuf = Buffer.concat([
previous ? Buffer.from(CID.parse(previous['/']).bytes) : Buffer.alloc(0),
Buffer.from(cid.bytes),
Buffer.from(peerId.toString(), 'utf-8'),
contextId,
Buffer.from(provider.ID, 'utf-8'),
...provider.Addresses.map(a => Buffer.from(a, 'utf-8')),
provider.Metadata,
extendedProviderOverride ? Buffer.from([1]) : Buffer.from([0])
])
Copy link
Contributor Author

@olizilla olizilla May 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec for this says

  • The Signature for each of the Providers within an ExtendedProvider is signed by their
    corresponding private key.
    * The full advertisement object is serialized, with all instances of Signature replaced with
    an empty array of bytes.
    * This serialization is then hashed, and the hash is then signed.
    * The Provider from the encapsulating advertisement must be present in the Providers of
    the ExtendedProvider object, and must sign in this way as well. It may omit Metadata
    and Addresses if they match the values already set at the encapsulating advertisement.
    However, Signature must be present.
    https://github.com/ipni/specs/blob/main/IPNI.md#extendedprovider

In the implementing here I choose to explicitly duplicate all the info from bitswap provider at the advertisment level into the extended provider rather than rely on a fallback to the parent element.

In order to find out what The full advertisement object is serialized means I had to go to the reference impl:
https://github.com/ipni/go-libipni/blob/afe2d8ea45b86c2a22f756ee521741c8f99675e5/ingest/schema/envelope.go#L125

@olizilla
Copy link
Contributor Author

olizilla commented May 5, 2023

TODO: figure out if this change is sufficient to get the indexers to apply return the additional http provider for all previous entries.


from the spec

The provider addresses in the indexer are always updated by the latest advertisement received.

...here we're adding extended providers rather than updating the top level provider info, but one would assume that the same logic applies to extended providers.

@olizilla
Copy link
Contributor Author

olizilla commented May 5, 2023

There does not appear to be any tests here that check the shape of the advertisements!

peerId and friends are all a little different now

License: MIT
Signed-off-by: Oli Evans <[email protected]>
License: MIT
Signed-off-by: Oli Evans <[email protected]>
License: MIT
Signed-off-by: Oli Evans <[email protected]>
License: MIT
Signed-off-by: Oli Evans <[email protected]>
License: MIT
Signed-off-by: Oli Evans <[email protected]>
@olizilla olizilla marked this pull request as ready for review May 17, 2023 12:55
Copy link
Member

@alanshaw alanshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

src/handlers/advertisement.js Outdated Show resolved Hide resolved
src/handlers/advertisement.js Outdated Show resolved Hide resolved
@olizilla
Copy link
Contributor Author

I have

  • add env var HTTP_PEER_MULTIADDR=/dns4/freeway-staging.dag.haus/tcp/443/https to the dev lambda via the aws console.
  • created a staging peerid-http.json and added it to the dev peerid bucket us-west-2-dev-ep-bitswap-config

next up i'm deploying the dev branch, and will create some test message in the sqs queue.

@olizilla olizilla temporarily deployed to dev May 18, 2023 12:23 — with GitHub Actions Inactive
@olizilla olizilla temporarily deployed to dev May 18, 2023 12:23 — with GitHub Actions Inactive
@olizilla
Copy link
Contributor Author

olizilla commented May 18, 2023

Code is deployed. I have put a test multibase prefixed multihash zQmWzuLhMZH2RSrxpEYBtE2dWsVPPJMdYxdEkLzr7pbMHQz from the CIDQmWzuLhMZH2RSrxpEYBtE2dWsVPPJMdYxdEkLzr7pbMHQz through the dev multihash sqs queue.

/head has been updated with

{
  "head": {
    "/": "baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q"
  },
  "pubkey": {
    "/": {
      "bytes": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDuuLoNO5mhRCrLIeKE/zVBX3r3Y9SLbalOtwbYvvh/ounJYq+xatHHCXCx/3OnSDsMfHQxSuDWN8o1YleYPGaoZoLyMy5BvUOMgrRFFbTsiQIbPHo2p+wpRPXSq4wofudz7wBRUdjAPsYkMgUxi/757Z1dpjq+D9FpzwPHqx9nYBGzsKBaqNAcQjb3EbYFrU/xYrl+bYqNCYXgAhXfTtGCKnZAjOiuRHMl2cIy39X50kMfFnXm/gMiVJVf0pxGUsLSfl8HHc0hYLQdYgbSOYBQLamTvzYQJHmfDjgZfmHuuGv+Emh3a/Tq8PNDIgZqf5sAELXQAwIt+Fl/hJK+QgHHAgMBAAE"
    }
  },
  "sig": {
    "/": {
      "bytes": "fGmvYUeZc0LBt0d2w+F6iQnRBbVaCkGS2OnPR2UmhkOlrY/u6/aH/xqplVWGOcfdQtGFMB89Qsh23AMi0fSimdi0TjtAlY52E/8UNpyLrz5E4DMIW527FnJK8xw+x4si6S41z80b3VvLm5UVEr+KadsN6lXTgFb7q37+Q5PNggtyZ5+uKFVyh/wSlJIK6Q4OjrPXm081EhItLLYSH5H0nADexIrWXb6cc0ILCcMnI+exRkAjehC2UJUA5u6Uu7qMQaL9z6nfn3/P1aQGNowNk1Avr4PBfwpbz4e6g61nvhQ7F32vrw+vvyFCiJs7DhMPXM/d9zeuKqdZsmZQ2NZqZg"
    }
  }
}

The head baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q file is

{
  "Addresses": [
    "/dns4/elastic-dev.dag.house/tcp/443/wss"
  ],
  "ContextID": {
    "/": {
      "bytes": "YmFndXFlZXJhamtzMnJxcmQ1N2NlZHl4YW9lYXU3cjdtcmRudDVxbW83dGx5NXlnaHAybnB3YmxqM3NqYQ"
    }
  },
  "Entries": {
    "/": "baguqeerajks2rqrd57cedyxaoeau7r7mrdnt5qmo7tly5yghp2npwblj3sja"
  },
  "IsRm": false,
  "Metadata": {
    "/": {
      "bytes": "gBI"
    }
  },
  "PreviousID": {
    "/": "baguqeerafcg5fsqu3ei4uy6rrepfth56npmnteuryxixbo5sostozg2o22aa"
  },
  "Provider": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem",
  "Signature": {
    "/": {
      "bytes": "CqsCCAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDuuLoNO5mhRCrLIeKE/zVBX3r3Y9SLbalOtwbYvvh/ounJYq+xatHHCXCx/3OnSDsMfHQxSuDWN8o1YleYPGaoZoLyMy5BvUOMgrRFFbTsiQIbPHo2p+wpRPXSq4wofudz7wBRUdjAPsYkMgUxi/757Z1dpjq+D9FpzwPHqx9nYBGzsKBaqNAcQjb3EbYFrU/xYrl+bYqNCYXgAhXfTtGCKnZAjOiuRHMl2cIy39X50kMfFnXm/gMiVJVf0pxGUsLSfl8HHc0hYLQdYgbSOYBQLamTvzYQJHmfDjgZfmHuuGv+Emh3a/Tq8PNDIgZqf5sAELXQAwIt+Fl/hJK+QgHHAgMBAAESGy9pbmRleGVyL2luZ2VzdC9hZFNpZ25hdHVyZRoiEiAcn2knDT4D30v0e0m65ATwk2ZNfVnaPx6hJkf44V6yaCqAArszYLsDj1YsJc7TavpKFMfzL14zNZGoF1RA2oFHWMIQ9IhkazUTVpoJD1onhFWHeW0eok2j/DUlgg+JZEXJs7q5eoLmH+nZ3knAQAlhC1t04Ujqg28rhqKHUbhm5PIwdNCt+lKeDxmezdfU+2rNB2CpMv8zXCFBhr7iUdBI911zXyOC5jR3aojVN23IWYKMlD027+rENH0V1z6n86+oHhAa92F6Jp5v6kk8CcxJDCQRwKoKM+PGFXgbeV8iuWH0pvmBi0lmyq1oAjBR72pk0Fe3oiu/YN2+QOr8DPgljPpG9++EKYe4tEL49RQFjwLsX/F2quLZ77a7JlZ/J+cs5+w"
    }
  }
}

and the entries file baguqeerajks2rqrd57cedyxaoeau7r7mrdnt5qmo7tly5yghp2npwblj3sja

{
  "Entries": [
    {
      "/": {
        "bytes": "EiCAqhv8l/LACHU4bqm0egQhgMphMw+l3BOFtUxfFhGaTw=="
      }
    }
  ]
}

@olizilla
Copy link
Contributor Author

I've posted an AnnounceHTTP message to the advetisement sqs queue. The /head is now

{
  "head": {
    "/": "baguqeeraaieajw4fu7jexdwhzadrke5tsdmod3zgh4vujpuyrjtq4kukzppa"
  },
  "pubkey": {
    "/": {
      "bytes": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDuuLoNO5mhRCrLIeKE/zVBX3r3Y9SLbalOtwbYvvh/ounJYq+xatHHCXCx/3OnSDsMfHQxSuDWN8o1YleYPGaoZoLyMy5BvUOMgrRFFbTsiQIbPHo2p+wpRPXSq4wofudz7wBRUdjAPsYkMgUxi/757Z1dpjq+D9FpzwPHqx9nYBGzsKBaqNAcQjb3EbYFrU/xYrl+bYqNCYXgAhXfTtGCKnZAjOiuRHMl2cIy39X50kMfFnXm/gMiVJVf0pxGUsLSfl8HHc0hYLQdYgbSOYBQLamTvzYQJHmfDjgZfmHuuGv+Emh3a/Tq8PNDIgZqf5sAELXQAwIt+Fl/hJK+QgHHAgMBAAE"
    }
  },
  "sig": {
    "/": {
      "bytes": "wVlDejB7yjAEflRS8+lSWrs93ajakVcIS0/wFIbFR7eic5hk99mxcibKTm29oLHOOu1CHkXcgmsdW8mJT0LePYIPZErx+Qv5CV/I94R/1tsJ17+qvH/Wzzd2WPiONu45LkqEr1TICfHS6Ndh2R0VxrNJpiRDGHe4Vgqic6LGW+t1zq0uj7IfQvXc/9mCmaA7VVQ5vjDiGHccX0VMg+UmwNdMaG8vKCn0pa5IYN7cngr8PvLEp05XTRyDrf4HfmowZngR1I89ruc43wluyLQ1wCWqakDfy8nyg0zGwAwva/NfOi9aYGq6h4NslEMaUlAeqcSdU/8vrXBMJ2mDcxW9SA"
    }
  }
}

the head advertisement at baguqeeraaieajw4fu7jexdwhzadrke5tsdmod3zgh4vujpuyrjtq4kukzppa is

{
  "Addresses": [
    "/dns4/elastic-dev.dag.house/tcp/443/wss"
  ],
  "ContextID": {
    "/": {
      "bytes": ""
    }
  },
  "Entries": {
    "/": "bafkreehdwdcefgh4dqkjv67uzcmw7oje"
  },
  "ExtendedProvider": {
    "Override": false,
    "Providers": [
      {
        "Addresses": [
          "/dns4/elastic-dev.dag.house/tcp/443/wss"
        ],
        "ID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem",
        "Metadata": {
          "/": {
            "bytes": "gBI"
          }
        },
        "Signature": {
          "/": {
            "bytes": "CqsCCAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDuuLoNO5mhRCrLIeKE/zVBX3r3Y9SLbalOtwbYvvh/ounJYq+xatHHCXCx/3OnSDsMfHQxSuDWN8o1YleYPGaoZoLyMy5BvUOMgrRFFbTsiQIbPHo2p+wpRPXSq4wofudz7wBRUdjAPsYkMgUxi/757Z1dpjq+D9FpzwPHqx9nYBGzsKBaqNAcQjb3EbYFrU/xYrl+bYqNCYXgAhXfTtGCKnZAjOiuRHMl2cIy39X50kMfFnXm/gMiVJVf0pxGUsLSfl8HHc0hYLQdYgbSOYBQLamTvzYQJHmfDjgZfmHuuGv+Emh3a/Tq8PNDIgZqf5sAELXQAwIt+Fl/hJK+QgHHAgMBAAESKS9pbmRleGVyL2luZ2VzdC9leHRlbmRlZFByb3ZpZGVyU2lnbmF0dXJlGiISIKIWH9hxiGgjAa7yILGiLDfrU9hgPrD0MiHR+crO2ywdKoACf5QvcnRXs7AKQg7DCuHfKp+VJ7woLVA4kpyu5afwHa729kPGGBfGgZ5tD/3nXTskpr+mPrZEyIt5gRLMEaWB2tG9I8zJA7How6d1q7XIA0BI/NcnhlBhocqSjTv/aeyVk9OMCYqXyerJwjeHdyXyMu9YOEHTjvMgui8z2lHCT+Gcx9ZxGjoA5cVHQwzrwTgb/9VqEckiKYLNsOqGRn+Ag69w6l8Kl6j6MWQGuPT0eHBe0Si6jIpOoC0ngIwadYTfxZtRxaV37dbc/ZvfeSnsiN0GxRDxUWmMJklqKJe3Tnm4WMmuMqZD1jLLuJV7/Hn5wu3aQqgbmoRFV8lt2BsBOg"
          }
        }
      },
      {
        "Addresses": [
          "/dns4/freeway-staging.dag.haus/tcp/443/https"
        ],
        "ID": "QmcbxXyuJsVCGyYSKfWJHWXbuUH7meVM3UtEwvwxDtBmX6",
        "Metadata": {
          "/": {
            "bytes": "gID0AQ"
          }
        },
        "Signature": {
          "/": {
            "bytes": "CqsCCAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCo2qsb0s6rA3XvDC1/f/vAUV/cmktKMlTUC5ouxi2HQxqyxwJKp/DvzZ4duaMtbRvynVTJSDm6iqoyzguo9+OVDFrH+7i9kituM4SrAoZyDxP14qSZIq0XHjg6FvcJjAKs2IIFJ/fQRBU9Nx8Wd3Nhzd4dk33VgooscPZCRm7XAQwLvZX/eRAXn8LZPisOl23jhFaxE/BTMbNTjK08ry1lMdN6Fug54KDNVk/XAHfp+GlXGTX6lPt3qUhuWyds3jB6H47E9fHAgOHdXOvWHOBatkNiKSCp2Sb4dwZ2/VbldeVklDBfhZZ8ZyEWuLbIOiglJWla1s/2Fv5auC59TnM9AgMBAAESKS9pbmRleGVyL2luZ2VzdC9leHRlbmRlZFByb3ZpZGVyU2lnbmF0dXJlGiISIGVRGIvY/NQukI1IRiJICPBbVhHNcv4KtZavOF3Acb5ZKoACEBTEs7HjP2i0h769+Vr9KWX80VSklpWKszy3NDchOvHm1fKCPTffPgojnwdaDmc7xFhImqMxDMGUvWFsn4GWyLBrhstdi97d5/XFgOAhEKLYsHLyWQxltz32yBrIHkM+jSG/K7r23rTG1ogvZG1v1I9xgKj/JrIsMWVe3oERKz4QQfeMAozV0lRiRnBQNqvPLDB7u47mtSzlvpOeaUAqpRmmwdK0BclJjU88SCfZt/08eGckaPNDfuTFkwWIJRvnP0NQ0AqRY3nLtyh3NO990YwE1/kGIPF/iUI/ay9V09tQSe4XJeFHmg5QJk/w1PKIMesbhqkEnBpzh71WcbpVIQ"
          }
        }
      }
    ]
  },
  "IsRm": false,
  "Metadata": {
    "/": {
      "bytes": "gBI"
    }
  },
  "PreviousID": {
    "/": "baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q"
  },
  "Provider": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem",
  "Signature": {
    "/": {
      "bytes": "CqsCCAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDuuLoNO5mhRCrLIeKE/zVBX3r3Y9SLbalOtwbYvvh/ounJYq+xatHHCXCx/3OnSDsMfHQxSuDWN8o1YleYPGaoZoLyMy5BvUOMgrRFFbTsiQIbPHo2p+wpRPXSq4wofudz7wBRUdjAPsYkMgUxi/757Z1dpjq+D9FpzwPHqx9nYBGzsKBaqNAcQjb3EbYFrU/xYrl+bYqNCYXgAhXfTtGCKnZAjOiuRHMl2cIy39X50kMfFnXm/gMiVJVf0pxGUsLSfl8HHc0hYLQdYgbSOYBQLamTvzYQJHmfDjgZfmHuuGv+Emh3a/Tq8PNDIgZqf5sAELXQAwIt+Fl/hJK+QgHHAgMBAAESGy9pbmRleGVyL2luZ2VzdC9hZFNpZ25hdHVyZRoiEiD/tWd5d5+hYU0YiA+mUyoCHq4NgpzlNObHnQwJevYAaiqAAg7xHkXd2Ttjziw2lx7KRkXOr36zvGysfq7IP3sT+uwv7FSrFbaLNxZ5rGyLS6kjXP5VvSOOF+IuGcLchB7sLn4DN2gtHZYUvIAejop3a6xAJF7ZVfDTQSCYYN1Y/mdLefwMaVT9psMnUU9v+UsDtIodaDLAX4o/y6VHofQ7CqBXfejtzlGIV4uQmcHsQjcc7u9prGk3MPOw7lyCUwhZtexOeYtHKS3yJDPv12ecGidHAyDyP+GAVQktxU/peXhnBSNsYq85FcNTlFBLlG1+AjQCVRqvzafp+gjFgSBvXcJDkso+u6ApHft/joM0StmAqLOX4lme5p10Td50M5Rkvmo"
    }
  }
}

@olizilla
Copy link
Contributor Author

Running storetheindex locally against that throws a invalid advertisement signature

# ingest the /head from the dev adverts bucket
./storetheindex admin sync -p QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem --addr "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"

storetheindex logs

2023-05-18T14:31:08.038+0100	INFO	indexer/admin	http/handler.go:323	Syncing with peer	{"peerID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "address": "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"}
2023-05-18T14:31:08.038+0100	INFO	indexer/ingest	ingest/ingest.go:454	Explicitly syncing the latest advertisement from peer	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "addrs": ["/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"], "depth": 0, "resync": false}
2023-05-18T14:31:08.941+0100	INFO	dagsync	dagsync/subscriber.go:454	Sync queried head CID	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeeraaieajw4fu7jexdwhzadrke5tsdmod3zgh4vujpuyrjtq4kukzppa"}
2023-05-18T14:31:08.941+0100	INFO	dagsync	dagsync/subscriber.go:462	Start sync	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeeraaieajw4fu7jexdwhzadrke5tsdmod3zgh4vujpuyrjtq4kukzppa"}
2023-05-18T14:31:09.124+0100	ERROR	indexer/ingest	ingest/linksystem.go:105	Advertisement signature verification failed	{"err": "invalid signature"}
2023-05-18T14:31:09.124+0100	ERROR	dagsync/httpsync	httpsync/sync.go:284	Failed to commit	{"err": "invalid advertisement signature"}
2023-05-18T14:31:09.124+0100	ERROR	indexer/admin	http/handler.go:343	Cannot sync with peer	{"peerID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "address": "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "err": "failed to sync: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeeraaieajw4fu7jexdwhzadrke5tsdmod3zgh4vujpuyrjtq4kukzppa: failed to fetch block for cid baguqeeraaieajw4fu7jexdwhzadrke5tsdmod3zgh4vujpuyrjtq4kukzppa: invalid advertisement signature"}

@olizilla
Copy link
Contributor Author

Rolling back /head to the previous advetisment also gives a signing error (head was for an extended provider ad, after the roll back it's just for a regular single provider with entries, but the result is the same.)

./storetheindex admin sync -p QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem --addr "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"

storetheindex logs

2023-05-18T15:05:39.972+0100	INFO	indexer/admin	http/handler.go:323	Syncing with peer	{"peerID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "address": "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"}
2023-05-18T15:05:39.972+0100	INFO	indexer/ingest	ingest/ingest.go:454	Explicitly syncing the latest advertisement from peer	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "addrs": ["/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"], "depth": 0, "resync": false}
2023-05-18T15:05:40.694+0100	INFO	dagsync	dagsync/subscriber.go:454	Sync queried head CID	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q"}
2023-05-18T15:05:40.694+0100	INFO	dagsync	dagsync/subscriber.go:462	Start sync	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q"}
2023-05-18T15:05:40.887+0100	ERROR	indexer/ingest	ingest/linksystem.go:105	Advertisement signature verification failed	{"err": "invalid signature"}
2023-05-18T15:05:40.887+0100	ERROR	dagsync/httpsync	httpsync/sync.go:284	Failed to commit	{"err": "invalid advertisement signature"}
2023-05-18T15:05:40.887+0100	ERROR	indexer/admin	http/handler.go:343	Cannot sync with peer	{"peerID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "address": "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "err": "failed to sync: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q: failed to fetch block for cid baguqeera7izo6lfcvowkpqg2jjuo2hgglxy5bfbtwla7i4if6zxuhhe3yu4q: invalid advertisement signature"}

@ischasny
Copy link

ischasny commented May 18, 2023

The issue might be with the way how signatures are composed. The line that it fails one invokes VerifySignature method that has no external inputs. Let me know if I can help any further.

This was subtling breaking everything. It returned { '/': 'string cid' } instead of a cid instance.

That string doesn't have a bytes field so it doesn't get signed correctly when used as a PreviousID.

I need to update the lib to error when not passed a CID.

License: MIT
Signed-off-by: Oli Evans <[email protected]>
@olizilla
Copy link
Contributor Author

I've added a verify command to @web3-storage/ipni to show that it produces advertisements that can be verified with go-libipni storacha/ipni#11 and it works! That helped me narrow it down to the calling code in this PR, which is untyped so wasn't warning me that fetchHeadCid() was returning an a dag-json encoded object rather than a CID instance.

That's fixed now, and I'm gonna run the tests again.

@olizilla olizilla temporarily deployed to dev May 18, 2023 22:12 — with GitHub Actions Inactive
@olizilla olizilla temporarily deployed to dev May 18, 2023 22:12 — with GitHub Actions Inactive
@olizilla
Copy link
Contributor Author

Sweet. ran the test again and it blends!

❯ go run main.go ~/Downloads/baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya.json 
Valid! Signed by QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem 

and it can be ingested by a locally running storetheindex node:

$ ./storetheindex admin sync -p QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem --addr "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"
2023-05-18T23:27:31.407+0100	INFO	indexer/admin	http/handler.go:323	Syncing with peer	{"peerID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "address": "/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"}
2023-05-18T23:27:31.407+0100	INFO	indexer/ingest	ingest/ingest.go:454	Explicitly syncing the latest advertisement from peer	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "addrs": ["/dns4/dev-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https/p2p/QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem"], "depth": 0, "resync": false}
2023-05-18T23:27:32.125+0100	INFO	dagsync	dagsync/subscriber.go:454	Sync queried head CID	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya"}
2023-05-18T23:27:32.125+0100	INFO	dagsync	dagsync/subscriber.go:462	Start sync	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya"}
2023-05-18T23:27:32.499+0100	INFO	dagsync	dagsync/subscriber.go:923	Segmented sync completed	{"cid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya", "peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "syncedCidCount": 2}
2023-05-18T23:27:32.499+0100	INFO	indexer/ingest	ingest/ingest.go:1010	Advertisement chain synced	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "length": 2}
2023-05-18T23:27:32.500+0100	INFO	indexer/ingest	ingest/ingest.go:1187	Running worker on ad stack	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "headAdCid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya", "numAdsToProcess": 2}
2023-05-18T23:27:32.500+0100	INFO	indexer/ingest	ingest/ingest.go:1254	Processing advertisement	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "adCid": "baguqeerafo2522ma3vakogdl4654tblnqz2gsyi3jt6bhsn7ndz6lf75jg3q", "progress": "1 of 2", "lag": 1}
2023-05-18T23:27:32.502+0100	INFO	dagsync	dagsync/subscriber.go:462	Start sync	{"peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "cid": "baguqeerajks2rqrd57cedyxaoeau7r7mrdnt5qmo7tly5yghp2npwblj3sja"}
2023-05-18T23:27:32.679+0100	INFO	dagsync	dagsync/subscriber.go:868	Non-segmented sync completed	{"cid": "baguqeerajks2rqrd57cedyxaoeau7r7mrdnt5qmo7tly5yghp2npwblj3sja", "peer": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "syncedCidCount": 1}
2023-05-18T23:27:32.679+0100	INFO	indexer/ingest	ingest/linksystem.go:715	Put multihashes in entry chunk	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "adCid": "baguqeerafo2522ma3vakogdl4654tblnqz2gsyi3jt6bhsn7ndz6lf75jg3q", "contextID": "YmFndXFlZXJhamtzMnJxcmQ1N2NlZHl4YW9lYXU3cjdtcmRudDVxbW83dGx5NXlnaHAybnB3YmxqM3NqYQ==", "entriesKind": "EntryChunk", "count": 1}
2023-05-18T23:27:32.685+0100	INFO	indexer/ingest	ingest/linksystem.go:163	Finished syncing advertisement	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "adCid": "baguqeerafo2522ma3vakogdl4654tblnqz2gsyi3jt6bhsn7ndz6lf75jg3q", "contextID": "YmFndXFlZXJhamtzMnJxcmQ1N2NlZHl4YW9lYXU3cjdtcmRudDVxbW83dGx5NXlnaHAybnB3YmxqM3NqYQ==", "elapsed": "184.837834ms", "multihashes": 1}
2023-05-18T23:27:32.694+0100	INFO	indexer/ingest	ingest/ingest.go:1254	Processing advertisement	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "adCid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya", "progress": "2 of 2", "lag": 0}
2023-05-18T23:27:32.697+0100	INFO	indexer/ingest	ingest/linksystem.go:309	Advertisement is metadata update only	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "adCid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya", "contextID": ""}
2023-05-18T23:27:32.697+0100	INFO	indexer/ingest	ingest/linksystem.go:163	Finished syncing advertisement	{"publisher": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem", "adCid": "baguqeera72jbvbgvhm45l2xefmpluy7twhhso4vruj6deprolw4htefaalya", "contextID": "", "elapsed": "2.862709ms", "multihashes": 0}

@ischasny
Copy link

Glad that I worked for you 🙏

@olizilla
Copy link
Contributor Author

I have

  • add env var HTTP_PEER_MULTIADDR=/dns4/freeway-staging.dag.haus/tcp/443/https to the staging lambda via the aws console.
  • created a staging peerid-http.json and added it to the dev peerid bucket us-west-2-staging-ep-bitswap-config

@olizilla
Copy link
Contributor Author

local query on storetheindex of that original test CID works and shows both bitswap and http endpoints

❯ curl http://localhost:3000/multihash/QmWzuLhMZH2RSrxpEYBtE2dWsVPPJMdYxdEkLzr7pbMHQz -Ss | jq
{
  "MultihashResults": [
    {
      "Multihash": "EiCAqhv8l/LACHU4bqm0egQhgMphMw+l3BOFtUxfFhGaTw==",
      "ProviderResults": [
        {
          "ContextID": "YmFndXFlZXJhamtzMnJxcmQ1N2NlZHl4YW9lYXU3cjdtcmRudDVxbW83dGx5NXlnaHAybnB3YmxqM3NqYQ==",
          "Metadata": "gBI=",
          "Provider": {
            "ID": "QmQPF8BeQF5GW3qmXcZLqGp2HqrFmscXaGA7ef8vN2sEem",
            "Addrs": [
              "/dns4/elastic-dev.dag.house/tcp/443/wss"
            ]
          }
        },
        {
          "ContextID": "YmFndXFlZXJhamtzMnJxcmQ1N2NlZHl4YW9lYXU3cjdtcmRudDVxbW83dGx5NXlnaHAybnB3YmxqM3NqYQ==",
          "Metadata": "gID0AQ==",
          "Provider": {
            "ID": "QmcbxXyuJsVCGyYSKfWJHWXbuUH7meVM3UtEwvwxDtBmX6",
            "Addrs": [
              "/dns4/freeway-staging.dag.haus/tcp/443/https"
            ]
          }
        }
      ]
    }
  ]
}

@olizilla
Copy link
Contributor Author

olizilla commented May 19, 2023

Created production peerId for http as peerId-http.json and uploaded it to the prod peer bucket.

Created a test local advert for that peerid and verified it signs correctly.

❯ go run main.go baguqeerazr7hxi6edgmkqc5xw3qlhtexusfwvmhdm5tqzqaln6i2gfmyyaza
Valid! Signed by QmUA9D3H7HeCYsirB3KmPSvZh3dNXMZas6Lwgr4fv1HTTp 

@olizilla
Copy link
Contributor Author

The staging publisher is creating valid adverts but failing to update https://staging.cid.contact/. I am informed that there is no such staging deployment of cid.contract, and dev.cid.contract is the one true test deployment, so i am going to update our staging deployment to use that.

@olizilla
Copy link
Contributor Author

I've had to create new peerIDs for staging publisher and e-ipfs, as both dev and staging are pointing to the same dev.cid.contact.

@olizilla olizilla merged commit 32a86c7 into main May 19, 2023
@olizilla olizilla deleted the extended-provider branch May 19, 2023 14:27
@olizilla
Copy link
Contributor Author

Merging to get this code deployed to staging

@olizilla
Copy link
Contributor Author

olizilla commented May 19, 2023

end-to-end test on staging worked ✨ 🎷 🐩

an extended provider for http is listed

❯ curl "https://dev.cid.contact/providers/Qmc5vg9zuLYvDR1wtYHCaxjBHenfCNautRwCjG3n5v5fbs" -sS | jq
{
  "AddrInfo": {
    "ID": "Qmc5vg9zuLYvDR1wtYHCaxjBHenfCNautRwCjG3n5v5fbs",
    "Addrs": [
      "/dns4/elastic-staging.dag.house/tcp/443/wss"
    ]
  },
  "LastAdvertisement": {
    "/": "baguqeeramobzcdq64sypa5x2hgikf5jjkw5lpgmtcld6eiinfjjpjsa6tv2a"
  },
  "LastAdvertisementTime": "2023-05-19T15:08:03Z",
  "Publisher": {
    "ID": "Qmc5vg9zuLYvDR1wtYHCaxjBHenfCNautRwCjG3n5v5fbs",
    "Addrs": [
      "/dns4/staging-ep-ipfs-advertisement.s3.us-west-2.amazonaws.com/tcp/443/https"
    ]
  },
  "IndexCount": 6,
  "ExtendedProviders": {
    "Providers": [
      {
        "ID": "Qmc5vg9zuLYvDR1wtYHCaxjBHenfCNautRwCjG3n5v5fbs",
        "Addrs": [
          "/dns4/elastic-staging.dag.house/tcp/443/wss"
        ]
      },
      {
        "ID": "QmX5D2SHKfJEjMfUwKAJZtgXhFzmYvA66Hcc9L3ExuVDXs",
        "Addrs": [
          "/dns4/freeway-staging.dag.haus/tcp/443/https"
        ]
      }
    ],
    "Metadatas": [
      "gBI=",
      "gID0AQ=="
    ]
  },
  "FrozenAt": null
}

the extended provider appears in query results

❯ curl "https://dev.cid.contact/cid/bafybeigqf6fljfatmqi65y3y3h57uqq5aneim43tq6a2wvznwww63y6ela" -sS | jq
{
  "MultihashResults": [
    {
      "Multihash": "EiDQL4q0lBNkEe7jeNn7+kIdA0iGc3OHgatXLbWt7ePEWA==",
      "ProviderResults": [
        {
          "ContextID": "YmFndXFlZXJhbnRrMmpkdmg3ZmtkeWxuZTJ6cmp6ZWljaDZmZHRqc2xxZTRrdnh6cDVrcjNvaWNnZm54cQ==",
          "Metadata": "gBI=",
          "Provider": {
            "ID": "Qmc5vg9zuLYvDR1wtYHCaxjBHenfCNautRwCjG3n5v5fbs",
            "Addrs": [
              "/dns4/elastic-staging.dag.house/tcp/443/wss"
            ]
          }
        },
        {
          "ContextID": "YmFndXFlZXJhbnRrMmpkdmg3ZmtkeWxuZTJ6cmp6ZWljaDZmZHRqc2xxZTRrdnh6cDVrcjNvaWNnZm54cQ==",
          "Metadata": "gID0AQ==",
          "Provider": {
            "ID": "QmX5D2SHKfJEjMfUwKAJZtgXhFzmYvA66Hcc9L3ExuVDXs",
            "Addrs": [
              "/dns4/freeway-staging.dag.haus/tcp/443/https"
            ]
          }
        }
      ]
    }
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants