-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ref(nextjs): Use proxy loader for wrapping all data-fetching functions (
#5602) This changes the experimental auto-wrapping feature to use a templated proxy module for all wrapping. (Previously, the wrapping was done using a different mix of code parsing, AST manipulation, proxying, templating, and string concatenation for each function.) Not only should this be easier to reason about and maintain (because it's one unified method of wrapping), it also solves a number of the disadvantages inherent in various parts of the previous approach. Specifically, using a template for everything rather than storing code as strings lets us take advantage of linting and syntax highlighting, and using a proxy loader rather than dealing with the AST directly puts the onus of handling syntax variations and edge cases on tools which are actually designed for that purpose. At a high level, the proxy loader works like this: - During the nextjs build phase, webpack loads all of the files in the `pages` directory, and feeds each one to our loader. - The loader derives the parameterized route from the page file's path, and fills it and the path itself into the template. - In the template, we load the page module, replace any data-fetching methods we find with wrapped versions of themselves, and then re-export everything. - The contents of the template is returned by the loader in place of the original contents of the page module. Previously, when working directly with the page module's AST, we had to account for the very many ways functions can be defined and exported. By contrast, doing the function wrapping in a separate module allows us to take advantage of the fact that imported modules have a single, known structure, which we can modify directly in the template code. Notes: - For some reason, nextjs won't accept data fetchers which are exported as part of an `export * from '...'` statement. Therefore, the "re-export everything" part of the third step above needs to be of the form `export { all, of, the, things, which, the, page, module, exports, listed, individually } from './pageModule'`. This in turn requires knowing the full list of each page module's exports, since, unfortunately, `export { ...importedPageModule }` isn't a thing. As it turns out, one of the noticeable differences between our published code before and after the build process revamp in the spring is that where `tsc` leaves `export *` statements untouched, Rollup splits them out into individual exports - exactly what's needed here! The loader therefore uses Rollup's JS API to process the proxy module code before returning it. Further, in order that Rollup be able to understand the page module code (which will be written in either `jsx` or `tsx`), we first use Sucrase to transpile the code to vanilla JS. Who knew the build process work would come in so handy? - Given that we replace a page module's contents with the proxy code the first time webpack tries to load it, we need webpack to load the same module a second time, in order to be able to process and bundle the page module itself. We therefore attach a query string to the end of the page module's path wherever it's referenced in the template, because this makes Webpack think it is a different, as-yet-unloaded module, causing it to perform the second load. The query string also acts like a flag for us, so that the second time through we know we've already handled the file and can let it pass through the loader untouched. - Rollup expects the entry point to be given as a path to a file on disk, not as raw code. We therefore create a temporary file for each page's proxy module, which we then delete as soon as rollup has read it. The easiest way to make sure that relative paths are preserved when things are re-exported is to put each temporary file alongside the page module it's wrapping, in the `pages/` directory. Fortunately, by the time our loader is running, Webpack has already decided what files it needs to load, so these temporary files don't get caught up in the bundling process. - In order to satisfy the linter in the template file, the SDK itself has been added as a dev dependency. Fortunately this seems not to confuse yarn. - Just to keep things manageable, this stops using but doesn't yet delete the previous loader (and associated files/code). Once this is merged, I'll do that in a separate PR. Ref #5505
- Loading branch information
1 parent
0bb0092
commit f7241c4
Showing
11 changed files
with
326 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,3 @@ | ||
| export { default as prefixLoader } from './prefixLoader'; | ||
| export { default as dataFetchersLoader } from './dataFetchersLoader'; | ||
| export { default as proxyLoader } from './proxyLoader'; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| import { escapeStringForRegex } from '@sentry/utils'; | ||
| import * as fs from 'fs'; | ||
| import * as path from 'path'; | ||
|
|
||
| import { rollupize } from './rollup'; | ||
| import { LoaderThis } from './types'; | ||
|
|
||
| type LoaderOptions = { | ||
| pagesDir: string; | ||
| }; | ||
|
|
||
| /** | ||
| * Replace the loaded file with a proxy module "wrapping" the original file. In the proxy, the original file is loaded, | ||
| * any data-fetching functions (`getInitialProps`, `getStaticProps`, and `getServerSideProps`) it contains are wrapped, | ||
| * and then everything is re-exported. | ||
| */ | ||
| export default async function proxyLoader(this: LoaderThis<LoaderOptions>, userCode: string): Promise<string> { | ||
| // We know one or the other will be defined, depending on the version of webpack being used | ||
| const { pagesDir } = 'getOptions' in this ? this.getOptions() : this.query; | ||
|
|
||
| // Get the parameterized route name from this page's filepath | ||
| const parameterizedRoute = path | ||
| // Get the path of the file insde of the pages directory | ||
| .relative(pagesDir, this.resourcePath) | ||
| // Add a slash at the beginning | ||
| .replace(/(.*)/, '/$1') | ||
| // Pull off the file extension | ||
| .replace(/\.(jsx?|tsx?)/, '') | ||
| // Any page file named `index` corresponds to root of the directory its in, URL-wise, so turn `/xyz/index` into | ||
| // just `/xyz` | ||
| .replace(/\/index$/, '') | ||
| // In case all of the above have left us with an empty string (which will happen if we're dealing with the | ||
| // homepage), sub back in the root route | ||
| .replace(/^$/, '/'); | ||
|
|
||
| // TODO: For the moment we skip API routes. Those will need to be handled slightly differently because of the manual | ||
| // wrapping we've already been having people do using `withSentry`. | ||
| if (parameterizedRoute.startsWith('api')) { | ||
| return userCode; | ||
| } | ||
|
|
||
| // We don't want to wrap twice (or infinitely), so in the proxy we add this query string onto references to the | ||
| // wrapped file, so that we know that it's already been processed. (Adding this query string is also necessary to | ||
| // convince webpack that it's a different file than the one it's in the middle of loading now, so that the originals | ||
| // themselves will have a chance to load.) | ||
| if (this.resourceQuery.includes('__sentry_wrapped__')) { | ||
| return userCode; | ||
| } | ||
|
|
||
| const templatePath = path.resolve(__dirname, '../templates/proxyLoaderTemplate.js'); | ||
| let templateCode = fs.readFileSync(templatePath).toString(); | ||
| // Make sure the template is included when runing `webpack watch` | ||
| this.addDependency(templatePath); | ||
|
|
||
| // Inject the route into the template | ||
| templateCode = templateCode.replace(/__ROUTE__/g, parameterizedRoute); | ||
|
|
||
| // Fill in the path to the file we're wrapping and save the result as a temporary file in the same folder (so that | ||
| // relative imports and exports are calculated correctly). | ||
| // | ||
| // TODO: We're saving the filled-in template to disk, however temporarily, because Rollup expects a path to a code | ||
| // file, not code itself. There is a rollup plugin which can fake this (`@rollup/plugin-virtual`) but the virtual file | ||
| // seems to be inside of a virtual directory (in other words, one level down from where you'd expect it) and that | ||
| // messes up relative imports and exports. Presumably there's a way to make it work, though, and if we can, it would | ||
| // be cleaner than having to first write and then delete a temporary file each time we run this loader. | ||
| templateCode = templateCode.replace(/__RESOURCE_PATH__/g, this.resourcePath); | ||
| const tempFilePath = path.resolve(path.dirname(this.resourcePath), `temp${Math.random()}.js`); | ||
| fs.writeFileSync(tempFilePath, templateCode); | ||
|
|
||
| // Run the proxy module code through Rollup, in order to split the `export * from '<wrapped file>'` out into | ||
| // individual exports (which nextjs seems to require), then delete the tempoary file. | ||
| let proxyCode = await rollupize(tempFilePath, this.resourcePath); | ||
| fs.unlinkSync(tempFilePath); | ||
|
|
||
| if (!proxyCode) { | ||
| // We will already have thrown a warning in `rollupize`, so no need to do it again here | ||
| return userCode; | ||
| } | ||
|
|
||
| // Add a query string onto all references to the wrapped file, so that webpack will consider it different from the | ||
| // non-query-stringged version (which we're already in the middle of loading as we speak), and load it separately from | ||
| // this. When the second load happens this loader will run again, but we'll be able to see the query string and will | ||
| // know to immediately return without processing. This avoids an infinite loop. | ||
| const resourceFilename = path.basename(this.resourcePath); | ||
| proxyCode = proxyCode.replace( | ||
| new RegExp(`/${escapeStringForRegex(resourceFilename)}'`, 'g'), | ||
| `/${resourceFilename}?__sentry_wrapped__'`, | ||
| ); | ||
|
|
||
| return proxyCode; | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| import type { RollupSucraseOptions } from '@rollup/plugin-sucrase'; | ||
| import sucrase from '@rollup/plugin-sucrase'; | ||
| import { logger } from '@sentry/utils'; | ||
| import * as path from 'path'; | ||
| import type { InputOptions as RollupInputOptions, OutputOptions as RollupOutputOptions } from 'rollup'; | ||
| import { rollup } from 'rollup'; | ||
|
|
||
| const getRollupInputOptions: (proxyPath: string, resourcePath: string) => RollupInputOptions = ( | ||
| proxyPath, | ||
| resourcePath, | ||
| ) => ({ | ||
| input: proxyPath, | ||
| plugins: [ | ||
| // For some reason, even though everything in `RollupSucraseOptions` besides `transforms` is supposed to be | ||
| // optional, TS complains that there are a bunch of missing properties (hence the typecast). Similar to | ||
| // https://github.com/microsoft/TypeScript/issues/20722, though that's been fixed. (In this case it's an interface | ||
| // exporting a `Pick` picking optional properties which is turning them required somehow.)' | ||
| sucrase({ | ||
| transforms: ['jsx', 'typescript'], | ||
| } as unknown as RollupSucraseOptions), | ||
| ], | ||
|
|
||
| // We want to process as few files as possible, so as not to slow down the build any more than we have to. We need the | ||
| // proxy module (living in the temporary file we've created) and the file we're wrapping not to be external, because | ||
| // otherwise they won't be processed. (We need Rollup to process the former so that we can use the code, and we need | ||
| // it to process the latter so it knows what exports to re-export from the proxy module.) Past that, we don't care, so | ||
| // don't bother to process anything else. | ||
| external: importPath => importPath !== proxyPath && importPath !== resourcePath, | ||
|
|
||
| // Prevent rollup from stressing out about TS's use of global `this` when polyfilling await. (TS will polyfill if the | ||
| // user's tsconfig `target` is set to anything before `es2017`. See https://stackoverflow.com/a/72822340 and | ||
| // https://stackoverflow.com/a/60347490.) | ||
| context: 'this', | ||
|
|
||
| // Rollup's path-resolution logic when handling re-exports can go wrong when wrapping pages which aren't at the root | ||
| // level of the `pages` directory. This may be a bug, as it doesn't match the behavior described in the docs, but what | ||
| // seems to happen is this: | ||
| // | ||
| // - We try to wrap `pages/xyz/userPage.js`, which contains `export { helperFunc } from '../../utils/helper'` | ||
| // - Rollup converts '../../utils/helper' into an absolute path | ||
| // - We mark the helper module as external | ||
| // - Rollup then converts it back to a relative path, but relative to `pages/` rather than `pages/xyz/`. (This is | ||
| // the part which doesn't match the docs. They say that Rollup will use the common ancestor of all modules in the | ||
| // bundle as the basis for the relative path calculation, but both our temporary file and the page being wrapped | ||
| // live in `pages/xyz/`, and they're the only two files in the bundle, so `pages/xyz/`` should be used as the | ||
| // root. Unclear why it's not.) | ||
| // - As a result of the miscalculation, our proxy module will include `export { helperFunc } from '../utils/helper'` | ||
| // rather than the expected `export { helperFunc } from '../../utils/helper'`, thereby causing a build error in | ||
| // nextjs.. | ||
| // | ||
| // It's not 100% clear why, but telling it not to do the conversion back from absolute to relative (by setting | ||
| // `makeAbsoluteExternalsRelative` to `false`) seems to also prevent it from going from relative to absolute in the | ||
| // first place, with the result that the path remains untouched (which is what we want.) | ||
| makeAbsoluteExternalsRelative: false, | ||
| }); | ||
|
|
||
| const rollupOutputOptions: RollupOutputOptions = { | ||
| format: 'esm', | ||
|
|
||
| // Don't create a bundle - we just want the transformed entrypoint file | ||
| preserveModules: true, | ||
| }; | ||
|
|
||
| /** | ||
| * Use Rollup to process the proxy module file (located at `tempProxyFilePath`) in order to split its `export * from | ||
| * '<wrapped file>'` call into individual exports (which nextjs seems to need). | ||
| * | ||
| * @param tempProxyFilePath The path to the temporary file containing the proxy module code | ||
| * @param resourcePath The path to the file being wrapped | ||
| * @returns The processed proxy module code, or undefined if an error occurs | ||
| */ | ||
| export async function rollupize(tempProxyFilePath: string, resourcePath: string): Promise<string | undefined> { | ||
| let finalBundle; | ||
|
|
||
| try { | ||
| const intermediateBundle = await rollup(getRollupInputOptions(tempProxyFilePath, resourcePath)); | ||
| finalBundle = await intermediateBundle.generate(rollupOutputOptions); | ||
| } catch (err) { | ||
| __DEBUG_BUILD__ && | ||
| logger.warn( | ||
| `Could not wrap ${resourcePath}. An error occurred while processing the proxy module template:\n${err}`, | ||
| ); | ||
| return undefined; | ||
| } | ||
|
|
||
| // The module at index 0 is always the entrypoint, which in this case is the proxy module. | ||
| let { code } = finalBundle.output[0]; | ||
|
|
||
| // Rollup does a few things to the code we *don't* want. Undo those changes before returning the code. | ||
| // | ||
| // Nextjs uses square brackets surrounding a path segment to denote a parameter in the route, but Rollup turns those | ||
| // square brackets into underscores. Further, Rollup adds file extensions to bare-path-type import and export sources. | ||
| // Because it assumes that everything will have already been processed, it always uses `.js` as the added extension. | ||
| // We need to restore the original name and extension so that Webpack will be able to find the wrapped file. | ||
| const resourceFilename = path.basename(resourcePath); | ||
| const mutatedResourceFilename = resourceFilename | ||
| // `[\\[\\]]` is the character class containing `[` and `]` | ||
| .replace(new RegExp('[\\[\\]]', 'g'), '_') | ||
| .replace(/(jsx?|tsx?)$/, 'js'); | ||
| code = code.replace(new RegExp(mutatedResourceFilename, 'g'), resourceFilename); | ||
|
|
||
| return code; | ||
| } |
51 changes: 51 additions & 0 deletions
51
packages/nextjs/src/config/templates/proxyLoaderTemplate.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| /** | ||
| * This file is a template for the code which will be substituted when our webpack loader handles non-API files in the | ||
| * `pages/` directory. | ||
| * | ||
| * We use `__RESOURCE_PATH__` as a placeholder for the path to the file being wrapped. Because it's not a real package, | ||
| * this causes both TS and ESLint to complain, hence the pragma comments below. | ||
| */ | ||
|
|
||
| // @ts-ignore See above | ||
| // eslint-disable-next-line import/no-unresolved | ||
| import * as wrapee from '__RESOURCE_PATH__'; | ||
| import * as Sentry from '@sentry/nextjs'; | ||
| import type { GetServerSideProps, GetStaticProps, NextPage as NextPageComponent } from 'next'; | ||
|
|
||
| type NextPageModule = { | ||
| default: { getInitialProps?: NextPageComponent['getInitialProps'] }; | ||
| getStaticProps?: GetStaticProps; | ||
| getServerSideProps?: GetServerSideProps; | ||
| }; | ||
|
|
||
| const userPageModule = wrapee as NextPageModule; | ||
|
|
||
| const pageComponent = userPageModule.default; | ||
|
|
||
| const origGetInitialProps = pageComponent.getInitialProps; | ||
| const origGetStaticProps = userPageModule.getStaticProps; | ||
| const origGetServerSideProps = userPageModule.getServerSideProps; | ||
|
|
||
| if (typeof origGetInitialProps === 'function') { | ||
| pageComponent.getInitialProps = Sentry.withSentryGetInitialProps( | ||
| origGetInitialProps, | ||
| '__ROUTE__', | ||
| ) as NextPageComponent['getInitialProps']; | ||
| } | ||
|
|
||
| export const getStaticProps = | ||
| typeof origGetStaticProps === 'function' | ||
| ? Sentry.withSentryGetStaticProps(origGetStaticProps, '__ROUTE__') | ||
| : undefined; | ||
| export const getServerSideProps = | ||
| typeof origGetServerSideProps === 'function' | ||
| ? Sentry.withSentryGetServerSideProps(origGetServerSideProps, '__ROUTE__') | ||
| : undefined; | ||
|
|
||
| export default pageComponent; | ||
|
|
||
| // Re-export anything exported by the page module we're wrapping. When processing this code, Rollup is smart enough to | ||
| // not include anything whose name matchs something we've explicitly exported above. | ||
| // @ts-ignore See above | ||
| // eslint-disable-next-line import/no-unresolved | ||
| export * from '__RESOURCE_PATH__'; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.