Symbol upload for Flutter framework

[vaind]

After some testing, It seems like the native profiler could be the way to go, at least for iOS and macOS. However, I'm having issues with the symbolication - all the flutter symbols (e.g. referencing /private/var/containers/Bundle/Application/2D6824B6-DC93-4F60-AFC2-ADE201585EC6/Runner.app/Frameworks/Flutter.framework/Flutter) in this profile are unresolved (symbol not found). I don't know - was there some custom handling in the symbolicator for flutter-specific symbols? Maybe that doesn't get triggered when the transaction comes from swift... Also I've tested triggering an error in swift and the image seems resolved but the frames say redacted. Do you know what that is about @marandaneto ?

OK so at least in errors, the issue of some stack frames not being symbolicated is due to dSYMs missing for the Flutter.framework (or FlutterMacOS.framework). They're currently not shipped with Flutter at the moment so the dart plugin won't upload them to Sentry and thus they can't be used for symbolication, see flutter/flutter#117404 (comment)

Originally posted by @vaind in #1106 (comment)

Possible solutions:

• upload in the dart plugin
• use craft, although this is normally used to upload during an SDK release: https://github.com/getsentry/craft#symbol-collector-symbol-collector
• introduce a flutter-specific symbol server - https://github.com/getsentry/team-processing/issues/83

[marandaneto]

Relates to getsentry/sentry-dart-plugin#148
Maybe we can close getsentry/sentry-dart-plugin#148 in favor of this one?

[vaind]

What's the status of the internal (team-processing) issue for symbol upload? This is pretty important for profiling UX. I don't have access to that repo.

[vaind]

@krystofwoldrich & @Swatinem - replying here so others who track this issue may be able to pitch in.

Is there a way to link the debug meta image that the SDK sends to the engine version hash needed to download the correct symbols?

No, I don't think we can - symbol files sources described here: https://github.com/flutter/flutter/wiki/Crashes are based on an engine build ID which isn't present anywhere in the native profiler payload (because it's not a debug ID of any of the debug images).

Although the engine build ID string is present in the built app binary, checked as:

flutter --version
Flutter 3.13.2 • channel stable • [email protected]:flutter/flutter.git
Framework • revision ff5b5b5fa6 (12 days ago) • 2023-08-24 08:12:28 -0500
Engine • revision b20183e040
Tools • Dart 3.1.0 • DevTools 2.25.0
...
strings build/ios/..../Flutter.framework/Flutter | grep b20183e040

it doesn't seem to be available in native code. So while in theory we could make changes to extract this at runtime and propagate it in all the SDKs, it seems a bit fragile. Together with having to do custom handling on the server, it doesn't seem like a best way to go.
Another alternative (to those I've mentioned above) - maybe just get all the stable-channel releases of Flutter SDKs and push them to a shared symbol server?

[bruno-garcia]

use craft, although this is normally used to upload during an SDK release: getsentry/craft#symbol-collector-symbol-collector

Symbol Collector today is only used for Android symbols (you can actually upload Mach-O files too but they won't be fetched from anywhere. Because the service is hooked up only to the Sentry built in Android "symbol server" (it's a GCS bucket behind that). We can change that, if we need to upload those "externally". But I'm guessing a job running 'internally' with direct GCS access might be easier, such as we do for the Sentry built-in Apple "symbol server": https://github.com/getsentry/apple-system-symbols-upload

[vaind]

I've had a look at the scripts in apple-system-symbols-upload and also a bit into the craft target & the symbol-collector CLI.

For Flutter, I imagine the workflow would look like this:

1. fetch all tags from https://github.com/flutter/flutter (optimized: if run in CI, we should only fetch stuff we haven't processed yet)
2. for each tag, get the engine version from https://raw.githubusercontent.com/flutter/flutter/3.13.4/bin/internal/engine.version - say 9064459a8b0dcd32877107f6002cc429a71659d1 in case of Flutter v3.13.4
3. Head over to https://console.cloud.google.com/storage/browser/flutter_infra_release/flutter/9064459a8b0dcd32877107f6002cc429a71659d1 and look for platforms and symbols we care about
   I'd say it's mostly iOS, macOS and Android at this time, but we may want to just slip in other platforms if there are symbol servers that can be easily integrated
4. download zip files, extract symbols and upload to appropriate symbol servers

I'd create a (pwsh/dart) script to fetch stuff from the Flutter server and call the symbol-collector CLI to do the actual upload. This script would run in CI periodically. I assume there will be some changes necessary in the symbol-collector (cli, server?) to make this work, but to me it seems like the best value for the effort - extending an existing solution instead of creating a new one.

Yes, we could have yet another script/repo that uploads to all platform-specific symbol servers, but I don't really see why symbol-collector shouldn't be doing that instead. If it did, it would also be useful for other SDKs, for example IIRC we're uploading some shared Unity symbols but doing so for each app build instead of having this centralized.

WDYT @bruno-garcia? You know symbol-collector best.

[bruno-garcia]

the built-in Sentry Apple symbol server will lookup in the Symbol Collector symbol server once we merge: https://github.com/getsentry/getsentry/pull/11913

[vaind]

We still need to upload symbols so let's keep this tracking issue until it's actually finished and verified e2e

[vaind]

Works fine for iOS/macOS with:

• profiles: https://sentry-sdks.sentry.io/profiling/profile/sentry-flutter/a45ec177220b452d885b6a241dec844d/flamegraph/?colorCoding=by+system+vs+application+frame&query=&sorting=call+order&tid=42499&view=top+down
• and errors: https://sentry-sdks.sentry.io/issues/4339846487/events/9eadfaa14fea41f2901abb9454481616/

We're also collecting Android symbols although I wasn't able to find an error that would have them in a stack trace...