High Data Cache Writes usage without proper usage analytics

[Ahmedsaed]

Summary

I am getting a high Data Cache Writes usage that is exceeding the plan threshold. I want to understand what is driving this high usage but all I am getting from the Usage dashboard is that it is caused by Incremental Site Regeneration and the Data Cache card always displays No Data. There's no data available for your selection. I would like to have more information about what is causing the high usage, like which routes are causing the high usage, which cached files are being regenerated , etc.

Here is my usage analytics (excluding unused ones)

Fast Data Transfer
862.3 MB / 100 GB

Fast Origin Transfer
3.81 GB / 10 GB

Edge Requests
75K / 1M

Edge Request CPU Duration
1 s / 1 h

Data Cache Reads
352K / 1M

Data Cache Writes
333K / 200K

Function Invocations
6.1K / 100K

Function Duration
3.2 GB-Hrs / 100 GB-Hrs

Web Analytics Events
2.4K / 2.5K

Speed Insights Data Points
4.4K / 10K

Image Optimization
145 / 1,000

Bandwidth
862.3 MB / 100 GB

The website is a simple blog post that utilizes ISR (with on-demand revalidation).
I am using Nextjs 14 with pages router.

Example

https://tdqiq.com

Steps to Reproduce

1. Start a nextjs app with pages router
2. Make multiple pages with ISR and on-demand revalidation
3. host the website on vercel
4. watch the Data Cache Writes explode in the usage dashboard

[g-rasse]

We are also seeing millions of write units over the included 2M and have no way of telling where it's coming from.
All the dashboard tells us is ISR, but we've increased the reval time to 18h (for testing purposes), and that did not change anything.

We're still getting thousands of write units written every hour. Waiting for a response to the ticket.

Similar issue here: https://github.com/orgs/vercel/discussions/7108

[farhanulju]

Same issue.
Getting 100k+ Data writes with less than 1000 views per day. There is no way to understand what is causing it.
No revalidation timer set. On-demand 13 revalidations were made.

[Ahmedsaed]

I wonder if the spikes are connected to other metrics.

Have you observed any correlations?

[Ahmedsaed]

I just received a response on my support ticket

Hello there,

Thank you for reaching out to Vercel Support! I'm happy to assist here.

ISR was previously included in Bandwidth, but now it is its own separate metric - this can be found in our announcement here: https://vercel.com/blog/improved-infrastructure-pricing

The data cache write units are different to the number of validations made. A single write unit is calculated as 8KB of data and so the total data cache write units can be calculated as the number of revalidations in combination with the size of the page.

ISR can be turned off by adding adding export const dynamic = 'force-dynamic' in your layouts. This will greatly reduce your usage for Data Cache Writes.

We are working on introducing more observability around Data Cache Writes which we will provide updates on at a later date.

I hope this information is beneficial - please let me know if you have additional questions we can assist with and I would be happy to help!

Kind regards,

Sen
Senior Customer Success Engineer ? Vercel

While the response doesn't provide any new information or specific technical assistance, it does indicate that Vercel is working on improving usage analytics for ISR.

Also, disabling ISR with export const dynamic = 'force-dynamic' defeats the whole purpose of using ISR. I'm looking for ways to manage and optimize ISR without turning it off entirely.

[Ahmedsaed]

But I am quite confused with the calculations here. My usage analytics states that I have used 333K of Data Cache Writes and each unit is equal to 8KB of data which means that I have used 333000 * 8 = 2664000KB = 2664MB = 2.664GB of data

2.6 GB of Data Cache is a lot and doesn't make sense for my small blog. What kind of data is being cached and which parts is being re-validated? Does the cache include images?

My blog contains about 60 pages so if we divide 2664MB / 60 we get 44MB of revalidation data per page (this is just an average and doesn't reflect real life because many of these pages rarely gets revalidated).

Looking at my build logs (includes all pages)

 ✓ Generating static pages (66/66)
 ✓ Collecting build traces    
 ✓ Finalizing page optimization    

Route (pages)                                       Size     First Load JS
┌ ● / (1901 ms)                                     3.74 kB         144 kB
├   └ css/dd5aaa092f891106.css                      2.66 kB
├   /_app                                           0 B             122 kB
├ ○ /404                                            180 B           122 kB
├ ○ /about-us                                       1.76 kB         124 kB
├ ƒ /api/revalidate                                 0 B             122 kB
├ ● /article (412 ms)                               2.56 kB         125 kB
├   └ css/ffe4754d579efbc5.css                      898 B
├ ● /article/[slug] (4928 ms)                       4.12 kB         357 kB
├   └ css/399e78f9ff52c7bc.css                      2.6 kB
├   ├ /article/trading-courses (1129 ms)
├   ├ /article/report-scam-forex-brokers (685 ms)
├   ├ /article/العملات-الرقمية (617 ms)
├   ├ /article/منصات-الفوركس (531 ms)
├   ├ /article/التداول-حلال-ام-حرام (520 ms)
├   ├ /article/ما-هو-الفوركس (511 ms)
├   ├ /article/trading-signals (450 ms)
├   └ /article/cfds-vs-stocks-trading (387 ms)
├ ● /author (1053 ms)                               1.37 kB         124 kB
├   └ css/1d56e029bd989b0a.css                      536 B
├ ● /author/[slug] (2932 ms)                        1.97 kB         125 kB
├   └ css/ec23d0a7f139f15c.css                      708 B
├   ├ /author/يوسف-راسل (1106 ms)
├   ├ /author/احمد-اسماعيل (929 ms)
├   └ /author/ahmed-saed (871 ms)
├ ○ /brokers-review-and-rating-methodology          4.57 kB         127 kB
├ ● /company-review (1749 ms)                       2.35 kB         143 kB
├   └ css/f60f3e1083606cb3.css                      2.35 kB
├ ● /company-review/[slug] (13144 ms)               8.11 kB         237 kB
├   └ css/bc26d38aa67ae61c.css                      1.81 kB
├   ├ /company-review/tentrade (760 ms)
├   ├ /company-review/xtrade (671 ms)
├   ├ /company-review/evest (640 ms)
├   ├ /company-review/accuindex (637 ms)
├   ├ /company-review/equity (630 ms)
├   ├ /company-review/avatrade (625 ms)
├   ├ /company-review/fxtm (619 ms)
├   └ [+18 more paths] (avg 463 ms)
├ ○ /contact-us                                     3.49 kB         264 kB
├   └ css/c2339c5304c8faf5.css                      3.08 kB
├ ○ /cookie-policy                                  2.21 kB         124 kB
├ ● /guide (306 ms)                                 3.14 kB         125 kB
├   └ css/29a2e86015b6d8be.css                      1.5 kB
├ ● /guide/[slug] (36920 ms)                        1.55 kB         230 kB
├   └ css/b0fb5bdee9d6e74d.css                      244 B
├   ├ /guide/شركات-التداول-في-الامارات (4305 ms)
├   ├ /guide/شركات-التداول-في-العرااق (3710 ms)
├   ├ /guide/شركات-التداول-في-الكويت (3370 ms)
├   ├ /guide/منصات-تداول-العملات-الرقمية (3236 ms)
├   ├ /guide/شركات-التداول-في-تركيا (3152 ms)
├   ├ /guide/شركات-التداول-سلطنة-عمان (3069 ms)
├   ├ /guide/شركات-التداول-في-مصر (2998 ms)
├   └ [+5 more paths] (avg 2583 ms)
├ ○ /privacy-policy                                 2.42 kB         125 kB
├ ● /sitemap (956 ms)                               1.72 kB         124 kB
├   └ css/130a30ad3e1634a3.css                      764 B
├ ƒ /sitemap.xml                                    248 B           122 kB
├ ○ /terms-of-use                                   3.5 kB          126 kB
└ ƒ /xml/[slug]                                     247 B           122 kB
+ First Load JS shared by all                       127 kB
  ├ chunks/framework-ecc4130bc7a58a64.js            45.2 kB
  ├ chunks/main-b1ca1c5258d8ccb3.js                 32.2 kB
  ├ chunks/pages/_app-31e05d62da6563a4.js           43.9 kB
  └ other shared chunks (total)                     5.57 kB

○  (Static)   prerendered as static content
●  (SSG)      prerendered as static HTML (uses getStaticProps)
ƒ  (Dynamic)  server-rendered on demand

We can see that all of the pages are just a couple of kilo bytes which means that some of these pages gets at least hundreds of revalidations in the past month to reach that level of usage

[farhanulju]

I am having the same issue. My page size is 1.1 MB.

I'm getting hit with 330k data cache writes per day. For that to happen,
I would need 330,0008/(1.11024) = 2344 revalidation everyday.

I don't have ISR enabled. And On-demand regeneration is not hitting even 20 requests. If I calculate with 20 requests, that is 129 MB per page.

Something is very wrong with this system. The fact, that they are charging us for something they have no observability on, is insane.

[Ahmedsaed]

The fact, that they are charging us for something they have no observability on, is insane.

I believe you don't get charged for Data Cache Writes. It's because it's one of the beta features and all beta features are not paid.
I have been using the free tier and was never charged for exceeding the Data Cache

Source: https://vercel.com/docs/infrastructure/data-cache, https://vercel.com/docs/release-phases#beta

[leerob]

Hey there, happy to assist here.

You mentioned that your Usage dashboard says this usage is coming from Incremental Static Regeneration. ISR, a feature used in frameworks like Next.js or SvelteKit, is charged through the Data Cache billable item on Vercel. Next.js also has a more granular cache for the App Router, called the Data Cache, which is in beta on Vercel and not charged.

Since you are using the Pages Router, it makes sense that you are not using the granular Data Cache which is only part of the App Router. That is why that card is empty.

You mentioned you are using on-demand revalidation – that's great. Can you confirm there are no other places in your codebase where you're using revalidate with getStaticProps? The next step to decrease the number of writes for ISR would be to not trigger a revalidation on every single change, or decrease the frequency at which these revalidations are happening. This logic would be inside of your API route which is doing the on-demand revalidation.

Let me know if that helps narrow down where to optimize.

[Ahmedsaed]

Thanks for the detailed explanation. That clarifies why the Data Cache card in the Usage dashboard was never populated.

Regarding revalidation in getStaticProps, I conducted a global search for revalidate and can confirm it doesn't appear anywhere in the codebase other than the /revalidate API endpoint.

I wish the Usage analytics were more transparent about ISR for the Pages Router. Having basic analytics, such as which route is being re-rendered, how many times it gets re-rendered, and what kind of files are being cached, would greatly help in optimizing usage patterns.

[leerob]

Can you add some logging inside of your API route? You should still be able to see the number of requests to your /revalidate endpoint on the Usage page, but to inspect the request body (to see the path to revalidate or other information), this is where logging that information in the function would be helpful.

You can see what files have been cached by looking at the response headers for a given request. There is an x-vercel-cache header that would be HIT if cached. This would be files like HTML, CSS, and JavaScript files.

[Ahmedsaed]

Thank you for your response.

These tips are incredibly valuable, and I’m hopeful they will help me resolve the issue.

[github-actions[bot]]

This discussion was automatically locked because the community moved to a new site. Please join us at vercel.community