This repository has been archived by the owner on Jan 8, 2024. It is now read-only.
feat!: support DNS over HTTPS and DNS-JSON over HTTPS #55
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,3 +7,6 @@ node_modules | |
| package-lock.json | ||
| yarn.lock | ||
| .vscode | ||
| .env | ||
| .envrc | ||
| .tool-versions | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| import { MAX_RECURSIVE_DEPTH, recursiveResolveDnslink } from '../utils/dns.js' | ||
| import resolve from './resolver.js' | ||
| import type { DNSResolver, ResolveDnsLinkOptions } from '../index.js' | ||
|
|
||
| export function defaultResolver (): DNSResolver { | ||
| return async (domain: string, options: ResolveDnsLinkOptions = {}): Promise<string> => { | ||
SgtPooki marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| return recursiveResolveDnslink(domain, MAX_RECURSIVE_DEPTH, resolve, options) | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| /* eslint-env browser */ | ||
|
|
||
| import PQueue from 'p-queue' | ||
| import { CustomProgressEvent } from 'progress-events' | ||
| import { type DNSResponse, MAX_RECURSIVE_DEPTH, recursiveResolveDnslink, ipfsPathAndAnswer } from '../utils/dns.js' | ||
| import { TLRU } from '../utils/tlru.js' | ||
| import type { ResolveDnsLinkOptions, DNSResolver } from '../index.js' | ||
|
|
||
| // Avoid sending multiple queries for the same hostname by caching results | ||
| const cache = new TLRU<string>(1000) | ||
| // This TTL will be used if the remote service does not return one | ||
| const ttl = 60 * 1000 | ||
|
|
||
| /** | ||
| * Uses the RFC 8427 'application/dns-json' content-type to resolve DNS queries. | ||
| * | ||
| * Supports and server that uses the same schema as Google's DNS over HTTPS | ||
| * resolver. | ||
| * | ||
| * This resolver needs fewer dependencies than the regular DNS-over-HTTPS | ||
| * resolver so can result in a smaller bundle size and consequently is preferred | ||
| * for browser use. | ||
| * | ||
| * @see https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/make-api-requests/dns-json/ | ||
| * @see https://github.com/curl/curl/wiki/DNS-over-HTTPS#publicly-available-servers | ||
| * @see https://dnsprivacy.org/public_resolvers/ | ||
| * @see https://datatracker.ietf.org/doc/html/rfc8427 | ||
| */ | ||
| export function dnsJsonOverHttps (url: string): DNSResolver { | ||
| // browsers limit concurrent connections per host, | ||
| // we don't want preload calls to exhaust the limit (~6) | ||
| const httpQueue = new PQueue({ concurrency: 4 }) | ||
|
|
||
| const resolve = async (fqdn: string, options: ResolveDnsLinkOptions = {}): Promise<string> => { | ||
| const searchParams = new URLSearchParams() | ||
| searchParams.set('name', fqdn) | ||
| searchParams.set('type', 'TXT') | ||
|
|
||
| const query = searchParams.toString() | ||
|
|
||
| // try cache first | ||
| if (options.nocache !== true && cache.has(query)) { | ||
| const response = cache.get(query) | ||
|
|
||
| if (response != null) { | ||
| options.onProgress?.(new CustomProgressEvent<string>('dnslink:cache', { detail: response })) | ||
| return response | ||
| } | ||
| } | ||
|
Comment on lines
+42
to
+49
There was a problem hiding this comment. fwiw there is dns-over-http-resolver which is already used by js-mutiaddr which is already pulled in (i would assume) by Helia users. There was a problem hiding this comment. Added the logic here: vasco-santos/dns-over-http-resolver#102 FWIW, I am not sure how much of a performance hit this is, I think having one good way of doing this should be acceptable, |
||
|
|
||
| options.onProgress?.(new CustomProgressEvent<string>('dnslink:query', { detail: fqdn })) | ||
|
|
||
| // query DNS-JSON over HTTPS server | ||
| const response = await httpQueue.add(async () => { | ||
| const res = await fetch(`${url}?${searchParams}`, { | ||
| headers: { | ||
| accept: 'application/dns-json' | ||
| }, | ||
| signal: options.signal | ||
| }) | ||
|
|
||
| if (res.status !== 200) { | ||
| throw new Error(`Unexpected HTTP status: ${res.status} - ${res.statusText}`) | ||
| } | ||
|
|
||
| const query = new URL(res.url).search.slice(1) | ||
| const json: DNSResponse = await res.json() | ||
|
|
||
| options.onProgress?.(new CustomProgressEvent<DNSResponse>('dnslink:answer', { detail: json })) | ||
|
|
||
| const { ipfsPath, answer } = ipfsPathAndAnswer(fqdn, json) | ||
|
|
||
| cache.set(query, ipfsPath, answer.TTL ?? ttl) | ||
|
|
||
| return ipfsPath | ||
| }, { | ||
| signal: options.signal | ||
| }) | ||
|
|
||
| if (response == null) { | ||
| throw new Error('No DNS response received') | ||
| } | ||
|
|
||
| return response | ||
| } | ||
|
|
||
| return async (domain: string, options: ResolveDnsLinkOptions = {}) => { | ||
| return recursiveResolveDnslink(domain, MAX_RECURSIVE_DEPTH, resolve, options) | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| /* eslint-env browser */ | ||
|
|
||
| import { Buffer } from 'buffer' | ||
| import dnsPacket, { type DecodedPacket } from 'dns-packet' | ||
| import { base64url } from 'multiformats/bases/base64' | ||
| import PQueue from 'p-queue' | ||
| import { CustomProgressEvent } from 'progress-events' | ||
| import { toString as uint8ArrayToString } from 'uint8arrays/to-string' | ||
| import { type DNSResponse, MAX_RECURSIVE_DEPTH, recursiveResolveDnslink, ipfsPathAndAnswer } from '../utils/dns.js' | ||
| import { TLRU } from '../utils/tlru.js' | ||
| import type { ResolveDnsLinkOptions, DNSResolver } from '../index.js' | ||
|
|
||
| // Avoid sending multiple queries for the same hostname by caching results | ||
| const cache = new TLRU<string>(1000) | ||
| // This TTL will be used if the remote service does not return one | ||
| const ttl = 60 * 1000 | ||
|
|
||
| /** | ||
| * Uses the RFC 1035 'application/dns-message' content-type to resolve DNS | ||
| * queries. | ||
| * | ||
| * This resolver needs more dependencies than the non-standard | ||
| * DNS-JSON-over-HTTPS resolver so can result in a larger bundle size and | ||
| * consequently is not preferred for browser use. | ||
| * | ||
| * @see https://datatracker.ietf.org/doc/html/rfc1035 | ||
| * @see https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/make-api-requests/dns-wireformat/ | ||
| * @see https://github.com/curl/curl/wiki/DNS-over-HTTPS#publicly-available-servers | ||
| * @see https://dnsprivacy.org/public_resolvers/ | ||
| */ | ||
| export function dnsOverHttps (url: string): DNSResolver { | ||
| // browsers limit concurrent connections per host, | ||
| // we don't want preload calls to exhaust the limit (~6) | ||
| const httpQueue = new PQueue({ concurrency: 4 }) | ||
|
|
||
| const resolve = async (fqdn: string, options: ResolveDnsLinkOptions = {}): Promise<string> => { | ||
| const dnsQuery = dnsPacket.encode({ | ||
| type: 'query', | ||
| id: 0, | ||
| flags: dnsPacket.RECURSION_DESIRED, | ||
| questions: [{ | ||
| type: 'TXT', | ||
| name: fqdn | ||
| }] | ||
| }) | ||
|
|
||
| const searchParams = new URLSearchParams() | ||
| searchParams.set('dns', base64url.encode(dnsQuery).substring(1)) | ||
|
|
||
| const query = searchParams.toString() | ||
|
|
||
| // try cache first | ||
| if (options.nocache !== true && cache.has(query)) { | ||
| const response = cache.get(query) | ||
|
|
||
| if (response != null) { | ||
| options.onProgress?.(new CustomProgressEvent<string>('dnslink:cache', { detail: response })) | ||
| return response | ||
| } | ||
| } | ||
SgtPooki marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| options.onProgress?.(new CustomProgressEvent<string>('dnslink:query', { detail: fqdn })) | ||
|
|
||
| // query DNS over HTTPS server | ||
| const response = await httpQueue.add(async () => { | ||
| const res = await fetch(`${url}?${searchParams}`, { | ||
| headers: { | ||
| accept: 'application/dns-message' | ||
| }, | ||
| signal: options.signal | ||
| }) | ||
|
|
||
| if (res.status !== 200) { | ||
| throw new Error(`Unexpected HTTP status: ${res.status} - ${res.statusText}`) | ||
| } | ||
|
|
||
| const query = new URL(res.url).search.slice(1) | ||
| const buf = await res.arrayBuffer() | ||
| // map to expected response format | ||
| const json = toDNSResponse(dnsPacket.decode(Buffer.from(buf))) | ||
|
|
||
| options.onProgress?.(new CustomProgressEvent<DNSResponse>('dnslink:answer', { detail: json })) | ||
|
|
||
| const { ipfsPath, answer } = ipfsPathAndAnswer(fqdn, json) | ||
|
|
||
| cache.set(query, ipfsPath, answer.TTL ?? ttl) | ||
|
|
||
| return ipfsPath | ||
| }, { | ||
| signal: options.signal | ||
| }) | ||
|
|
||
| if (response == null) { | ||
| throw new Error('No DNS response received') | ||
| } | ||
|
|
||
| return response | ||
| } | ||
|
|
||
| return async (domain: string, options: ResolveDnsLinkOptions = {}) => { | ||
| return recursiveResolveDnslink(domain, MAX_RECURSIVE_DEPTH, resolve, options) | ||
| } | ||
| } | ||
|
|
||
| function toDNSResponse (response: DecodedPacket): DNSResponse { | ||
| const txtType = 16 | ||
|
|
||
| return { | ||
| Status: 0, | ||
|
There was a problem hiding this comment. Do we need the |
||
| TC: response.flag_tc ?? false, | ||
| RD: response.flag_rd ?? false, | ||
| RA: response.flag_ra ?? false, | ||
| AD: response.flag_ad ?? false, | ||
| CD: response.flag_cd ?? false, | ||
| Question: response.questions?.map(q => ({ | ||
| name: q.name, | ||
| type: txtType | ||
| })) ?? [], | ||
SgtPooki marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Answer: response.answers?.map(a => { | ||
| if (a.type !== 'TXT' || a.data.length < 1) { | ||
| return { | ||
| name: a.name, | ||
| type: txtType, | ||
| TTL: 0, | ||
| data: 'invalid' | ||
| } | ||
| } | ||
|
|
||
| if (!Buffer.isBuffer(a.data[0])) { | ||
| return { | ||
| name: a.name, | ||
| type: txtType, | ||
| TTL: a.ttl ?? ttl, | ||
| data: String(a.data[0]) | ||
| } | ||
| } | ||
|
|
||
| return { | ||
| name: a.name, | ||
| type: txtType, | ||
| TTL: a.ttl ?? ttl, | ||
| data: uint8ArrayToString(a.data[0]) | ||
| } | ||
| }) ?? [] | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| export { dnsOverHttps } from './dns-over-https.js' | ||
| export { dnsJsonOverHttps } from './dns-json-over-https.js' | ||
SgtPooki marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be left in place. Each package in a monorepo should declare all of it's dependencies, otherwise we have some deps here, some deps there, and then it becomes hard to reason about where things are coming from and why.