Codecov is unable to handle large files

[bqqbarbhg]

Hi! I've been happily using Codecov for multiple years now. The service has been able to handle large files in the past, the performance even had improved at one point. However, lately the performance has been dropping substantially, taking up to 60 seconds to display a large file (mostly in CSS layout according to the dev tools). Now it seems that the site is not able to display large files at all:

https://app.codecov.io/gh/ufbx/ufbx/blob/master/ufbx.c

This results in an "There was a problem getting the source code from your provider. Unable to show line by line coverage." error for me. The page used to work and the repository is configured correctly, you can see that the header loads correctly:

https://app.codecov.io/gh/ufbx/ufbx/blob/master/ufbx.h

This leads me to assume that the size of the file is the issue here. This regression unfortunately makes Codecov unusable for my project, even though it used to work. Is there anything that can be done about this?

[eliatcodecov]

In the immediate term, you can view this coverage just fine in the github ui using our browser extension (if you use Chrome or Firefox):

This is, of course, not the ideal answer but I'm providing it here in the event that you just need to see line coverage immediately while we work on a solution to get this supported in the ui. My guess is you're correct, this is a 30,000 LoC file which is an edge case for us but is still something we should try to render.

We'll dig into this and try to see if we can figure out the bottleneck to loading a file this large.

Question, has this file worked recently in the UI? When was the last time you viewed this file in the Codecov UI it rendered properly?

[bqqbarbhg]

Thanks a ton for the very quick reply!

The browser extension works great, really happy to see it also supports the flags as well, as that is what I was currently working on! It is also very responsive even with the large file, which is always nice to see.

Somewhat unrelated, but when adjusting the Chrome extension to be enabled on-click instead of globally the coverage button gets a ⚠️ icon and the following is output to the console, just in case this was not a known issue:

Last time I have documented that I used the Codecov UI on ufbx.c was on 11.20.2023, as I was doing some light profiling on it to see if I could see an obvious reason why it took 55 seconds to load. Here's the profile by the way if you're interested:

As I mentioned in the original post, it seems to be mostly bound by CSS styling, probably due to the amount of syntax highlighting and classes per line of code.

Thanks again for the reply and looking into the issue, let me know if there's anything I can do on my end to help!

[bqqbarbhg]

Oh, also slight follow-up to the extension issue: it persist even if changing the extension back to being globally enabled. The only fix I've found was to remove and re-install the extension. Firefox also seems to exhibit the same issue:

[drazisil-codecov]

@bqqbarbhg This is actually due to the GitHub API. Here is what is returned when we query that file:

{
  "name": "ufbx.c",
  "path": "ufbx.c",
  "sha": "81009c8fd38fe78956355a10f5719c401301994d",
  "size": 1102106,
  "url": "https://api.github.com/repos/ufbx/ufbx/contents/ufbx.c?ref=master",
  "html_url": "https://github.com/ufbx/ufbx/blob/master/ufbx.c",
  "git_url": "https://api.github.com/repos/ufbx/ufbx/git/blobs/81009c8fd38fe78956355a10f5719c401301994d",
  "download_url": "https://raw.githubusercontent.com/ufbx/ufbx/master/ufbx.c",
  "type": "file",
  "content": "",
  "encoding": "none",
  "_links": {
    "self": "https://api.github.com/repos/ufbx/ufbx/contents/ufbx.c?ref=master",
    "git": "https://api.github.com/repos/ufbx/ufbx/git/blobs/81009c8fd38fe78956355a10f5719c401301994d",
    "html": "https://github.com/ufbx/ufbx/blob/master/ufbx.c"
  }
}

Please note from GitHub's docs:

https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#get-repository-content

Between 1-100 MB: Only the raw or object custom media types are supported. Both will work as normal, except that when using the object media type, the content field will be an empty string and the encoding field will be "none". To get the contents of these larger files, use the raw media type.

Your ufbx.c file is over one MB. Note that the contents value is "".

In order to be able to fetch this file, we would need to refactor our API client to request the file in "raw". I am not sure what the work required to do this would involve.

FYI: @eliatcodecov

[bqqbarbhg]

Thanks for looking into it!

The file has probably grown just over 1MiB recently then. Having huge source files is somewhat common in single-file C libraries, so I think there would still be value in supporting them aside from ufbx.

Looks like an older commit still fits barely under the limit, in case you are interested in the loading performance:
https://app.codecov.io/gh/ufbx/ufbx/commit/dee5a64813ae62662fc9ad88ef13143031acf9b1/blob/ufbx.c

It seems that the site is responsive enough to use in Firefox, and even changing the flags is functional. In Chrome it still takes nearly a minute for me to load the page and switching the selected flag hangs the tab indefinitely. So it looks like you may be hitting some pathological case in CSS styling in Chrome, as even heavy styling should never take that long if it were to scale linearly.

[bqqbarbhg]

This seems to be fixed, the reports load and render quite fast even given the large source size.

Thank you and great work, I really appreciate the care for performance!