Reading Large Additional Files in Incremental Source Generators

[wsugarman]

Hi!

I am writing an IIncrementalGenerator that reads a "large" (~7 MB) file downloaded from the web, as an "additional file," and generates a C# source file from its contents. While I am able to successfully process the file, my builds consistently take much longer than I would think in Visual Studio and on the command line. I notice the "slow" build when either "rebuilding" in VS or invoking dotnet build after a simple code change (like changing a parameter name). I hoped I was writing my transformation pipeline correctly, but I must be missing something when it comes to caching.

Here are the broad strokes of my code:

IncrementalValueProvider<string> fileHeader = context
    .AnalyzerConfigOptionsProvider
    .Select(static (provider, _) => /* Return file_header_template from .globalconfig */);

IncrementalValueProvider<ImmutableArray</* Table Row */>> dataElements = context
    .AdditionalTextsProvider
    .Where(static text => string.Equals(Path.GetFileName(text.Path), "<my file>", StringComparison.Ordinal))
    .SelectMany(static (text, token) => /* Parse rows from HTML table */)
    .Collect();

context.RegisterSourceOutput(
    dataElements.Combine(fileHeader),
    static (c, pair) => c.AddSource($"<source>.g.cs", /* Writes source based on header and rows */));

Some things I checked or thought of include:

• The local downloaded file is not being changed. Its last modified time is a few days ago.
• The "row" class being returned in the SelectMany implements IEquatable<T>
• I saw that the comparer for AdditionalText reads the file, so I tried to replace the comparison by using my own custom IEqualityComparer<T>. My custom comparer used only the path name and the last modified date (to the chagrin of the analyzer), and I added it via WithComparer (e.g. context.AdditionalTextsProvider.WithComparer(/* comparer */)) but that didn't help either.

Is there an idiomatic way or best practice for dealing with large files?

[jasonmalinowski]

invoking dotnet build after a simple code change (like changing a parameter name)

The incremental pipeline is only used in Visual Studio for running your generator as a part of the IDE editing experience; a command line build like dotnet build will always run the generator from scratch since we don't have a place to really persist that between runs. If you don't need to consume anything from the compilation, then you might be better off instead moving this to an MSBuild task where you could try to do version checking and reuse a previously generated result.

Have you tried running your generator under a profiler? Which part is slow?

[wsugarman]

Thank you for the reply. I didn't realize that the incremental aspect of these Roslyn components was for the IDE editing experience. However, naively, I didn't think to profile my code-gen and unsurprisingly, there were some low-hanging optimizations I could make to significantly speed up my code. I am quite impressed with how easy it was to set up my analyzer as a "Start Up" project, run the profiler, and iterate on my code.

Kudos to the team for this experience!