Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: cache 404s indefinitely for bot probes #2668

Merged
merged 3 commits into from
Oct 15, 2024
Merged

feat: cache 404s indefinitely for bot probes #2668

merged 3 commits into from
Oct 15, 2024
+121 −85

Conversation

orinokai
Copy link
Contributor

Description

There are a number of routes ending in .php that are commonly probed by bots looking for security vulnerabilities. This PR ensures that the Runtime caches any 404 response resulting from a request to a URL ending in .php.

It is designed to help stop excessive function invocations and is a interim solution until we are able to constrain the paths that the SSR function runs on (waiting for regex support).

Tests

Integration tests added

Relevant links (GitHub issues, etc.) or a picture of cute animal

Partial solution for FRB-1344

@orinokai orinokai requested a review from a team as a code owner October 15, 2024 12:05
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 15, 2024 •
edited
Loading

📊 Package size report   0%↑

File Before (Size / Gzip) After (Size / Gzip)
dist/run/handlers/server.js 141.1 kB / 33.0 kB -0.01%↓141.1 kB / 0.01%↑33.0 kB
dist/run/headers.js 7.3 kB / 2.2 kB 3%↑7.5 kB / 2%↑2.3 kB
Total (Includes all files) 5.3 MB / 959.2 kB 0%↑5.3 MB / 0.01%↑959.3 kB
Tarball size 912.2 kB 0.01%↑912.2 kB
Unchanged files
File Size (Size / Gzip)
dist/build/advanced-api-routes.js 4.3 kB / 1.4 kB
dist/build/cache.js 1.0 kB / 414 B
dist/build/content/next-shims/telemetry-storage.cjs 1.6 kB / 659 B
dist/build/content/prerendered.js 8.6 kB / 2.6 kB
dist/build/content/server.js 8.7 kB / 2.8 kB
dist/build/content/static.js 3.6 kB / 1.2 kB
dist/build/functions/edge.js 20.2 kB / 5.3 kB
dist/build/functions/server.js 5.0 kB / 1.6 kB
dist/build/image-cdn.js 54.0 kB / 11.1 kB
dist/build/plugin-context.js 9.2 kB / 2.7 kB
dist/build/templates/handler-monorepo.tmpl.js 1.6 kB / 671 B
dist/build/templates/handler.tmpl.js 1.5 kB / 622 B
dist/build/verification.js 4.5 kB / 1.5 kB
dist/esm-chunks/chunk-5QSXBV7L.js 2.4 kB / 842 B
dist/esm-chunks/chunk-APO262HE.js 61.2 kB / 11.1 kB
dist/esm-chunks/chunk-GNGHTHMQ.js 55.6 kB / 9.7 kB
dist/esm-chunks/chunk-KGYJQ2U2.js 186.5 kB / 32.9 kB
dist/esm-chunks/chunk-OEQOKJGE.js 2.3 kB / 977 B
dist/esm-chunks/package-HKPEB5O7.js 3.6 kB / 1.4 kB
dist/index.js 3.4 kB / 1.1 kB
dist/run/config.js 1.2 kB / 595 B
dist/run/constants.js 516 B / 308 B
dist/run/handlers/cache.cjs 16.7 kB / 4.5 kB
dist/run/handlers/request-context.cjs 4.6 kB / 1.5 kB
dist/run/handlers/tracer.cjs 29.9 kB / 6.3 kB
dist/run/handlers/tracing.js 3.0 MB / 418.4 kB
dist/run/next.cjs 23.4 kB / 5.8 kB
dist/run/regional-blob-store.cjs 19.5 kB / 5.4 kB
dist/run/revalidate.js 1.0 kB / 475 B
dist/shared/blobkey.js 742 B / 399 B
dist/shared/cache-types.cjs 1.3 kB / 566 B
edge-runtime/lib/headers.ts 1.9 kB / 841 B
edge-runtime/lib/logging.ts 115 B / 121 B
edge-runtime/lib/middleware.ts 1.9 kB / 815 B
edge-runtime/lib/next-request.ts 3.3 kB / 1.1 kB
edge-runtime/lib/response.ts 9.2 kB / 2.9 kB
edge-runtime/lib/routing.ts 15.1 kB / 3.9 kB
edge-runtime/lib/util.test.ts 1.6 kB / 356 B
edge-runtime/lib/util.ts 3.7 kB / 1.3 kB
edge-runtime/matchers.json 3 B / 23 B
edge-runtime/middleware.ts 2.4 kB / 1.0 kB
edge-runtime/next.config.json 3 B / 23 B
edge-runtime/README.md 992 B / 509 B
edge-runtime/shim/index.js 1.5 kB / 717 B
edge-runtime/vendor.ts 807 B / 330 B
edge-runtime/vendor/deno.land/[email protected]/fmt/colors.ts 11.9 kB / 2.5 kB
edge-runtime/vendor/deno.land/[email protected]/testing/_diff.ts 9.6 kB / 3.0 kB
edge-runtime/vendor/deno.land/[email protected]/testing/asserts.ts 24.7 kB / 5.9 kB
edge-runtime/vendor/deno.land/[email protected]/_util/asserts.ts 854 B / 461 B
edge-runtime/vendor/deno.land/[email protected]/_util/os.ts 644 B / 355 B
edge-runtime/vendor/deno.land/[email protected]/async/abortable.ts 4.0 kB / 1.0 kB
edge-runtime/vendor/deno.land/[email protected]/async/deadline.ts 974 B / 544 B
edge-runtime/vendor/deno.land/[email protected]/async/debounce.ts 2.2 kB / 956 B
edge-runtime/vendor/deno.land/[email protected]/async/deferred.ts 1.5 kB / 798 B
edge-runtime/vendor/deno.land/[email protected]/async/delay.ts 1.8 kB / 845 B
edge-runtime/vendor/deno.land/[email protected]/async/mod.ts 465 B / 241 B
edge-runtime/vendor/deno.land/[email protected]/async/mux_async_iterator.ts 2.5 kB / 1.1 kB
edge-runtime/vendor/deno.land/[email protected]/async/pool.ts 3.2 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/async/retry.ts 2.4 kB / 1.0 kB
edge-runtime/vendor/deno.land/[email protected]/async/tee.ts 2.1 kB / 924 B
edge-runtime/vendor/deno.land/[email protected]/bytes/index_of_needle.ts 1.4 kB / 668 B
edge-runtime/vendor/deno.land/[email protected]/crypto/timing_safe_equal.ts 875 B / 442 B
edge-runtime/vendor/deno.land/[email protected]/datetime/to_imf.ts 1.3 kB / 681 B
edge-runtime/vendor/deno.land/[email protected]/encoding/base64.ts 2.5 kB / 1.0 kB
edge-runtime/vendor/deno.land/[email protected]/encoding/base64url.ts 2.0 kB / 872 B
edge-runtime/vendor/deno.land/[email protected]/flags/mod.ts 22.6 kB / 5.9 kB
edge-runtime/vendor/deno.land/[email protected]/fmt/colors.ts 12.4 kB / 2.7 kB
edge-runtime/vendor/deno.land/[email protected]/fmt/printf.ts 27.7 kB / 7.7 kB
edge-runtime/vendor/deno.land/[email protected]/http/cookie.ts 11.5 kB / 3.6 kB
edge-runtime/vendor/deno.land/[email protected]/node/_core.ts 2.3 kB / 716 B
edge-runtime/vendor/deno.land/[email protected]/node/_events.d.ts 27.2 kB / 5.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/_events.mjs 28.0 kB / 7.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/_global.d.ts 1.7 kB / 650 B
edge-runtime/vendor/deno.land/[email protected]/node/_next_tick.ts 5.0 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/_process/exiting.ts 138 B / 138 B
edge-runtime/vendor/deno.land/[email protected]/node/_process/process.ts 3.8 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/_process/stdio.mjs 336 B / 233 B
edge-runtime/vendor/deno.land/[email protected]/node/_process/streams.mjs 4.0 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/_stream.d.ts 53.2 kB / 11.9 kB
edge-runtime/vendor/deno.land/[email protected]/node/_stream.mjs 91.2 kB / 25.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/_util/_util_callbackify.ts 4.3 kB / 1.7 kB
edge-runtime/vendor/deno.land/[email protected]/node/_utils.ts 5.9 kB / 2.0 kB
edge-runtime/vendor/deno.land/[email protected]/node/assert.ts 23.1 kB / 4.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/assertion_error.ts 19.6 kB / 6.1 kB
edge-runtime/vendor/deno.land/[email protected]/node/async_hooks.ts 7.7 kB / 2.1 kB
edge-runtime/vendor/deno.land/[email protected]/node/buffer.ts 262 B / 204 B
edge-runtime/vendor/deno.land/[email protected]/node/events.ts 303 B / 221 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/_libuv_winerror.ts 7.8 kB / 1.9 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/_listen.ts 561 B / 342 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/_node.ts 443 B / 335 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/_timingSafeEqual.ts 479 B / 268 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/_utils.ts 2.4 kB / 938 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/_winerror.ts 354.4 kB / 64.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/ares.ts 2.4 kB / 1.1 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/async_wrap.ts 4.0 kB / 1.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/buffer.ts 3.5 kB / 1.3 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/cares_wrap.ts 15.2 kB / 3.9 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/config.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/connection_wrap.ts 2.6 kB / 1.3 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/constants.ts 21.5 kB / 5.1 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/contextify.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/credentials.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/crypto.ts 448 B / 244 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/errors.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/fs_dir.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/fs_event_wrap.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/fs.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/handle_wrap.ts 1.8 kB / 1.0 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/heap_utils.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/http_parser.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/icu.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/inspector.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/js_stream.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/messaging.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/mod.ts 3.1 kB / 955 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/module_wrap.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/native_module.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/natives.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/node_file.ts 2.9 kB / 1.5 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/node_options.ts 1.8 kB / 989 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/options.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/os.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/performance.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/pipe_wrap.ts 10.4 kB / 3.3 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/process_methods.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/report.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/serdes.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/signal_wrap.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/spawn_sync.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/stream_wrap.ts 9.3 kB / 2.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/string_decoder.ts 504 B / 261 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/symbols.ts 1.4 kB / 828 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/task_queue.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/tcp_wrap.ts 13.1 kB / 3.7 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/timers.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/tls_wrap.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/trace_events.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/tty_wrap.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/types.ts 5.7 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/udp_wrap.ts 12.4 kB / 3.6 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/url.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/util.ts 4.0 kB / 1.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/uv.ts 20.1 kB / 3.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/v8.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/worker.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal_binding/zlib.ts 87 B / 104 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/buffer.d.ts 73.6 kB / 12.1 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/buffer.mjs 66.1 kB / 10.6 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/crypto/_keys.ts 463 B / 262 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/crypto/constants.ts 252 B / 173 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/error_codes.ts 322 B / 250 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/errors.ts 78.9 kB / 17.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/fixed_queue.ts 4.4 kB / 1.2 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/hide_stack_frames.ts 550 B / 377 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/net.ts 3.1 kB / 1.5 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/normalize_encoding.mjs 2.1 kB / 500 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/options.ts 1.7 kB / 959 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/primordials.mjs 1.8 kB / 431 B
edge-runtime/vendor/deno.land/[email protected]/node/internal/process/per_thread.mjs 7.8 kB / 2.3 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/readline/callbacks.mjs 3.8 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/readline/utils.mjs 14.3 kB / 3.7 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/streams/destroy.mjs 6.9 kB / 1.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/streams/end-of-stream.mjs 7.1 kB / 1.9 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/streams/utils.mjs 5.9 kB / 1.2 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/util.mjs 4.0 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/util/comparisons.ts 16.6 kB / 3.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/util/debuglog.ts 3.2 kB / 1.4 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/util/inspect.mjs 71.5 kB / 19.8 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/util/types.ts 3.7 kB / 1.3 kB
edge-runtime/vendor/deno.land/[email protected]/node/internal/validators.mjs 8.0 kB / 2.1 kB
edge-runtime/vendor/deno.land/[email protected]/node/process.ts 19.4 kB / 5.2 kB
edge-runtime/vendor/deno.land/[email protected]/node/stream.ts 671 B / 346 B
edge-runtime/vendor/deno.land/[email protected]/node/string_decoder.ts 10.3 kB / 3.3 kB
edge-runtime/vendor/deno.land/[email protected]/node/util.ts 7.8 kB / 2.2 kB
edge-runtime/vendor/deno.land/[email protected]/node/util/types.ts 199 B / 153 B
edge-runtime/vendor/deno.land/[email protected]/path/_constants.ts 2.0 kB / 727 B
edge-runtime/vendor/deno.land/[email protected]/path/_interface.ts 728 B / 369 B
edge-runtime/vendor/deno.land/[email protected]/path/_util.ts 5.0 kB / 1.6 kB
edge-runtime/vendor/deno.land/[email protected]/path/common.ts 1.2 kB / 607 B
edge-runtime/vendor/deno.land/[email protected]/path/glob.ts 12.7 kB / 3.9 kB
edge-runtime/vendor/deno.land/[email protected]/path/mod.ts 1.4 kB / 690 B
edge-runtime/vendor/deno.land/[email protected]/path/posix.ts 13.9 kB / 3.7 kB
edge-runtime/vendor/deno.land/[email protected]/path/separator.ts 259 B / 209 B
edge-runtime/vendor/deno.land/[email protected]/path/win32.ts 28.5 kB / 6.4 kB
edge-runtime/vendor/deno.land/[email protected]/streams/write_all.ts 2.2 kB / 598 B
edge-runtime/vendor/deno.land/[email protected]/testing/_diff.ts 11.6 kB / 3.6 kB
edge-runtime/vendor/deno.land/[email protected]/testing/_format.ts 705 B / 462 B
edge-runtime/vendor/deno.land/[email protected]/testing/asserts.ts 25.5 kB / 5.7 kB
edge-runtime/vendor/deno.land/[email protected]/types.d.ts 4.2 kB / 1.2 kB
edge-runtime/vendor/deno.land/x/[email protected]/index.ts 4.5 kB / 1.7 kB
edge-runtime/vendor/deno.land/x/[email protected]/vendor/asyncify.js 2.6 kB / 931 B
edge-runtime/vendor/deno.land/x/[email protected]/vendor/html_rewriter.d.ts 2.7 kB / 622 B
edge-runtime/vendor/deno.land/x/[email protected]/vendor/html_rewriter.js 28.4 kB / 4.5 kB
edge-runtime/vendor/deno.land/x/[email protected]/index.ts 15.4 kB / 4.2 kB
edge-runtime/vendor/import_map.json 365 B / 180 B
edge-runtime/vendor/raw.githubusercontent.com/worker-tools/resolvable-promise/master/index.ts 1.8 kB / 657 B
edge-runtime/vendor/v1-7-0--edge-utils.netlify.app/logger/logger.ts 3.2 kB / 747 B
edge-runtime/vendor/v1-7-0--edge-utils.netlify.app/logger/mod.ts 29 B / 49 B
manifest.yml 31 B / 51 B
package.json 3.2 kB / 1.1 kB
README.md 2.7 kB / 1.1 kB

🤖 This report was automatically generated by pkg-size-action

mrstork
mrstork reviewed Oct 15, 2024
View reviewed changes
src/run/headers.ts Outdated Show resolved Hide resolved
mrstork
mrstork reviewed Oct 15, 2024
View reviewed changes
src/run/headers.ts Outdated Show resolved Hide resolved
mrstork
mrstork previously approved these changes Oct 15, 2024
View reviewed changes
@kodiakhq kodiakhq bot merged commit 60885d2 into main Oct 15, 2024
36 checks passed
@kodiakhq kodiakhq bot deleted the rs/cache-php-404s branch October 15, 2024 14:23
@token-generator-app token-generator-app bot mentioned this pull request Oct 15, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrstork mrstork mrstork left review comments

@pieh pieh pieh approved these changes

Assignees
No one assigned
Labels
automerge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@orinokai @pieh @mrstork