Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add iai_callgrind benchmark for measuring memory #64569

Closed
wants to merge 1 commit into from
Closed

Add iai_callgrind benchmark for measuring memory #64569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ inherits = "release"
debug = true
lto = "thin"

[profile.bench-with-debug]
inherits = "bench"
debug = true
strip = false

[workspace.dependencies]
# Workspace crates
next-api = { path = "packages/next-swc/crates/next-api" }
Expand Down
4 changes: 4 additions & 0 deletions packages/next-swc/crates/next-build-test/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,7 @@ turbopack-binding = { workspace = true, features = ["__turbo_tasks_build"] }
[dev-dependencies]
iai-callgrind = "0.10.2"
tempdir = "0.3.7"

[[bench]]
name = "bench"
harness = false
158 changes: 158 additions & 0 deletions packages/next-swc/crates/next-build-test/benches/bench.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
use std::{
cmp::max,
time::{Duration, Instant},
};

use anyhow::Context;
use next_api::project::{ProjectContainer, ProjectOptions};
use turbo_tasks::TurboTasks;
use turbo_tasks_malloc::TurboMalloc;
use turbopack_binding::turbo::tasks_memory::MemoryBackend;

#[global_allocator]
static ALLOC: turbo_tasks_malloc::TurboMalloc = turbo_tasks_malloc::TurboMalloc;

/// Clones a repo down into a tmp folder and produces a ProjectOptions that can
/// build it.
fn init_bench(
src: &str,
install_command: &str,
package_path: Option<&str>,
next_config: String,
js_config: String,
) -> (ProjectOptions, tempdir::TempDir) {
let tmp = tempdir::TempDir::new("turbo-bench").unwrap();

let clone = std::process::Command::new("git")
.arg("clone")
.arg("--depth")
.arg("1")
.arg(src)
.arg(tmp.path())
.status()
.unwrap();

std::process::Command::new("sh")
.arg("-c")
.current_dir(tmp.path())
.arg(install_command)
.status()
.unwrap();

let project_path = package_path
.map(|pp| tmp.path().join(pp))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

map_or_else should combine this with the .unwrap_or_else below it.

https://doc.rust-lang.org/std/option/enum.Option.html#method.map_or_else

I thought there was a clippy lint for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don' think clippy runs on benchmark code (or examples).

.unwrap_or_else(|| tmp.path().to_owned());

(
ProjectOptions {
project_path: project_path.to_str().unwrap().to_owned(),
root_path: tmp.path().to_str().unwrap().to_owned(),
env: vec![],
define_env: next_api::project::DefineEnv {
client: vec![],
edge: vec![],
nodejs: vec![],
},
watch: false,
dev: false,
next_config,
js_config,
},
tmp,
)
}

fn do_in_runtime() -> impl FnOnce(
Pin<
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this type can be cleaned up a bit so that it doesn't get split into so many lines:

Box<
dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>
+ 'static
+ Send
+ Sync,
>,
>,
) {
next_build_test::register();

let rt = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.on_thread_stop(|| {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't this just be

.on_thread_stop(TurboMalloc::thread_stop)

(and shouldn't clippy detect this? maybe I'm missing something?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. There is another place in the codebase where it is initialized without eta reduction so I just kept it consistent.

TurboMalloc::thread_stop();
})
.build()
.unwrap();
let tt = TurboTasks::new(MemoryBackend::new(usize::MAX));

move |fut| {
rt.block_on(async {
tt.run_once::<()>(fut).await;
});
}
}

fn get_container() -> (turbo_tasks::Vc<ProjectContainer>, tempdir::TempDir) {
let (options, dir) = init_bench(
"https://github.com/shadcn-ui/ui.git",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want a reproducible benchmark, this should have a commit hash associated with it. If you don't want to hardcode it, maybe take the commit hash in as an environment variable?

"pnpm i",
Some("apps/www"),
"{\n \"env\": {},\n \"webpack\": {},\n \"eslint\": {\n \"ignoreDuringBuilds\": false\n },\n \"typescript\": {\n \"ignoreBuildErrors\": false,\n \"tsconfigPath\": \"tsconfig.json\"\n },\n \"distDir\": \".next\",\n \"cleanDistDir\": true,\n \"assetPrefix\": \"\",\n \"cacheMaxMemorySize\": 52428800,\n \"configOrigin\": \"next.config.mjs\",\n \"useFileSystemPublicRoutes\": true,\n \"generateBuildId\": null,\n \"generateEtags\": true,\n \"pageExtensions\": [\n \"tsx\",\n \"ts\",\n \"jsx\",\n \"js\"\n ],\n \"poweredByHeader\": true,\n \"compress\": true,\n \"analyticsId\": \"\",\n \"images\": {\n \"deviceSizes\": [\n 640,\n 750,\n 828,\n 1080,\n 1200,\n 1920,\n 2048,\n 3840\n ],\n \"imageSizes\": [\n 16,\n 32,\n 48,\n 64,\n 96,\n 128,\n 256,\n 384\n ],\n \"path\": \"/_next/image\",\n \"loader\": \"default\",\n \"loaderFile\": \"\",\n \"domains\": [],\n \"disableStaticImages\": false,\n \"minimumCacheTTL\": 60,\n \"formats\": [\n \"image/webp\"\n ],\n \"dangerouslyAllowSVG\": false,\n \"contentSecurityPolicy\": \"script-src 'none'; frame-src 'none'; sandbox;\",\n \"contentDispositionType\": \"inline\",\n \"remotePatterns\": [\n {\n \"protocol\": \"https\",\n \"hostname\": \"avatars.githubusercontent.com\"\n },\n {\n \"protocol\": \"https\",\n \"hostname\": \"images.unsplash.com\"\n }\n ],\n \"unoptimized\": false\n },\n \"devIndicators\": {\n \"buildActivity\": true,\n \"buildActivityPosition\": \"bottom-right\"\n },\n \"onDemandEntries\": {\n \"maxInactiveAge\": 3600000,\n \"pagesBufferLength\": 5\n },\n \"amp\": {\n \"canonicalBase\": \"\"\n },\n \"basePath\": \"\",\n \"sassOptions\": {},\n \"trailingSlash\": false,\n \"i18n\": null,\n \"productionBrowserSourceMaps\": false,\n \"optimizeFonts\": true,\n \"excludeDefaultMomentLocales\": true,\n \"serverRuntimeConfig\": {},\n \"publicRuntimeConfig\": {},\n \"reactProductionProfiling\": false,\n \"reactStrictMode\": true,\n \"httpAgentOptions\": {\n \"keepAlive\": true\n },\n \"outputFileTracing\": true,\n \"staticPageGenerationTimeout\": 60,\n \"swcMinify\": true,\n \"modularizeImports\": {\n \"@mui/icons-material\": {\n \"transform\": \"@mui/icons-material/{{member}}\"\n },\n \"lodash\": {\n \"transform\": \"lodash/{{member}}\"\n }\n },\n \"experimental\": {\n \"prerenderEarlyExit\": false,\n \"serverMinification\": true,\n \"serverSourceMaps\": false,\n \"linkNoTouchStart\": false,\n \"caseSensitiveRoutes\": false,\n \"clientRouterFilter\": true,\n \"clientRouterFilterRedirects\": false,\n \"fetchCacheKeyPrefix\": \"\",\n \"middlewarePrefetch\": \"flexible\",\n \"optimisticClientCache\": true,\n \"manualClientBasePath\": false,\n \"cpus\": 9,\n \"memoryBasedWorkersCount\": false,\n \"isrFlushToDisk\": true,\n \"workerThreads\": false,\n \"optimizeCss\": false,\n \"nextScriptWorkers\": false,\n \"scrollRestoration\": false,\n \"externalDir\": false,\n \"disableOptimizedLoading\": false,\n \"gzipSize\": true,\n \"craCompat\": false,\n \"esmExternals\": true,\n \"fullySpecified\": false,\n \"outputFileTracingRoot\": \"/Users/arlyon/Programming/ui\",\n \"swcTraceProfiling\": false,\n \"forceSwcTransforms\": false,\n \"largePageDataBytes\": 128000,\n \"adjustFontFallbacks\": false,\n \"adjustFontFallbacksWithSizeAdjust\": false,\n \"typedRoutes\": false,\n \"instrumentationHook\": false,\n \"bundlePagesExternals\": false,\n \"parallelServerCompiles\": false,\n \"parallelServerBuildTraces\": false,\n \"ppr\": false,\n \"missingSuspenseWithCSRBailout\": true,\n \"optimizeServerReact\": true,\n \"useEarlyImport\": false,\n \"staleTimes\": {\n \"dynamic\": 30,\n \"static\": 300\n },\n \"optimizePackageImports\": [\n \"lucide-react\",\n \"date-fns\",\n \"lodash-es\",\n \"ramda\",\n \"antd\",\n \"react-bootstrap\",\n \"ahooks\",\n \"@ant-design/icons\",\n \"@headlessui/react\",\n \"@headlessui-float/react\",\n \"@heroicons/react/20/solid\",\n \"@heroicons/react/24/solid\",\n \"@heroicons/react/24/outline\",\n \"@visx/visx\",\n \"@tremor/react\",\n \"rxjs\",\n \"@mui/material\",\n \"@mui/icons-material\",\n \"recharts\",\n \"react-use\",\n \"@material-ui/core\",\n \"@material-ui/icons\",\n \"@tabler/icons-react\",\n \"mui-core\",\n \"react-icons/ai\",\n \"react-icons/bi\",\n \"react-icons/bs\",\n \"react-icons/cg\",\n \"react-icons/ci\",\n \"react-icons/di\",\n \"react-icons/fa\",\n \"react-icons/fa6\",\n \"react-icons/fc\",\n \"react-icons/fi\",\n \"react-icons/gi\",\n \"react-icons/go\",\n \"react-icons/gr\",\n \"react-icons/hi\",\n \"react-icons/hi2\",\n \"react-icons/im\",\n \"react-icons/io\",\n \"react-icons/io5\",\n \"react-icons/lia\",\n \"react-icons/lib\",\n \"react-icons/lu\",\n \"react-icons/md\",\n \"react-icons/pi\",\n \"react-icons/ri\",\n \"react-icons/rx\",\n \"react-icons/si\",\n \"react-icons/sl\",\n \"react-icons/tb\",\n \"react-icons/tfi\",\n \"react-icons/ti\",\n \"react-icons/vsc\",\n \"react-icons/wi\"\n ]\n },\n \"configFile\": \"/Users/arlyon/Programming/ui/apps/www/next.config.mjs\",\n \"configFileName\": \"next.config.mjs\",\n \"_originalRedirects\": [\n {\n \"source\": \"/components\",\n \"destination\": \"/docs/components/accordion\",\n \"permanent\": true\n },\n {\n \"source\": \"/docs/components\",\n \"destination\": \"/docs/components/accordion\",\n \"permanent\": true\n },\n {\n \"source\": \"/examples\",\n \"destination\": \"/examples/mail\",\n \"permanent\": false\n },\n {\n \"source\": \"/docs/primitives/:path*\",\n \"destination\": \"/docs/components/:path*\",\n \"permanent\": true\n },\n {\n \"source\": \"/figma\",\n \"destination\": \"/docs/figma\",\n \"permanent\": true\n },\n {\n \"source\": \"/docs/forms\",\n \"destination\": \"/docs/components/form\",\n \"permanent\": false\n },\n {\n \"source\": \"/docs/forms/react-hook-form\",\n \"destination\": \"/docs/components/form\",\n \"permanent\": false\n }\n ],\n \"exportPathMap\": {}\n}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is illegible, and seems like it might be difficult to update. Can we move these long strings into separate json files and pull them in with include_str!()?

.to_owned(),
"{\"compilerOptions\":{\"composite\":false,\"declaration\":true,\"declarationMap\":true,\"esModuleInterop\":true,\"forceConsistentCasingInFileNames\":true,\"inlineSources\":false,\"isolatedModules\":true,\"moduleResolution\":2,\"noUnusedLocals\":false,\"noUnusedParameters\":false,\"preserveWatchOutput\":true,\"skipLibCheck\":true,\"strict\":true,\"target\":1,\"lib\":[\"lib.dom.d.ts\",\"lib.dom.iterable.d.ts\",\"lib.esnext.d.ts\"],\"allowJs\":true,\"noEmit\":true,\"incremental\":true,\"module\":99,\"resolveJsonModule\":true,\"jsx\":1,\"baseUrl\":\"/Users/arlyon/Programming/ui/apps/www\",\"paths\":{\"@/*\":[\"./*\"],\"contentlayer/generated\":[\"./.contentlayer/generated\"]},\"plugins\":[{\"name\":\"next\"}],\"pathsBasePath\":\"/Users/arlyon/Programming/ui/apps/www\"}}"
.to_owned(),
);

(ProjectContainer::new(options), dir)
}

use std::{future::Future, pin::Pin};

#[iai_callgrind::library_benchmark]
#[bench::collect(0, do_in_runtime())]
#[bench::build_1(1, do_in_runtime())]
#[bench::build_10(10, do_in_runtime())]
pub fn build_pages(
pages: usize,
rt: impl FnOnce(Pin<Box<dyn Future<Output = Result<(), anyhow::Error>> + Send + Sync>>),
) {
// decide the pages to benchmark ahead of time
const SELECTED_PAGES: [&str; 10] = [
"/legal/event-code-of-conduct",
"/docs/functions/og-image-generation/og-image-api",
"/docs/image-optimization/managing-image-optimization-costs",
"/new/import/card",
"/try/share/[slug]",
"/app-future/[lang]/[teamSlug]/~/account/invoices-new",
"/docs/workflow-collaboration/conformance/rules/
NEXTJS_MISSING_REACT_STRICT_MODE",
"/app-future/[lang]/[teamSlug]/~/
ai/models/[modelName]/getting-started",
"/app-future/[lang]/
[teamSlug]/~/usage/[plan]/[planIteration]",
"/docs/
workflow-collaboration/conformance/rules/REQUIRE_CARET_DEPENDENCIES",
];

rt(Box::pin(async move {
let (container, _dir) = get_container();

let entrypoints = container.entrypoints().await.unwrap();

let routes = entrypoints.routes.clone().into_iter().take(pages);
// .filter(|route| SELECTED_PAGES.contains(&route.0.as_str()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover comment?


let count =
next_build_test::render_routes(routes, next_build_test::Strategy::Concurrent, 10, 10)
.await;

assert_eq!(count, SELECTED_PAGES.len());

Ok(())
}))
}

iai_callgrind::library_benchmark_group!(
name = group;
benchmarks = build_pages
);

iai_callgrind::main!(library_benchmark_groups = group);
Loading