Hunts too large for GRPC in master/minion

[lukerosser]

Running master/minion config we're getting these errors:

{"level":"error","msg":"While connecting to remote datastore: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4423419 vs. 4194304)","time":"2025-01-08T10:50:32Z"}
{"level":"error","msg":"Unable to sync hunts: gRPC Timeout in Remote datastore","time":"2025-01-08T10:53:19Z"}

Clients are getting a 502 but the minions do connect and register as frontends with the master.

[scudette]

It looks like the remote data store rpc is exceeding its size limits - how many hunts do you have? how large are the hunts (types of artifacts - you can see it in the request tab)? how many clients?

Try ls -l /opt/velociraptor/hunts and note how many files there are - how large are the .db files generally?

[lukerosser]

Number of hunts varies across the orgs but on average it's 25 and the highest currently is 37

A lot of hunts are single artifacts but payloads are large due to the artifact type, I'm not sure which of those is a factor. An example of a hunt with multiple artifacts is:

  "artifacts": [
    "DetectRaptor.Windows.Detection.Applications",
    "DetectRaptor.Windows.Detection.Evtx",
    "DetectRaptor.Windows.Detection.MFT",
    "DetectRaptor.Windows.Detection.NamedPipes",
    "DetectRaptor.Windows.Detection.Powershell.ISEAutoSave",
    "DetectRaptor.Windows.Detection.Powershell.PSReadline",
    "DetectRaptor.Windows.Detection.Webhistory",
    "DetectRaptor.Windows.Detection.YaraProcessWin",
    "DetectRaptor.Windows.Detection.ZoneIdentifier",
    "DetectRaptor.Generic.Detection.YaraWebshell"
  ],

There's a single artifact hunt with Windows.KapeFiles.Targets that has 2195 lines in the request.

The largest .db file in datastore/orgs/*/hunts I can find is 2.4MB

[scudette]

This might be fixed by #4037. This PR gets the minions to refresh hunts directly from the datastore and not use RPC. Also this PR introduces a new parameter API.max_grpc_recv_size which allows the default grpc limit to be increased.

Can you please check if the latest CI build works better in your deployment?

[lukerosser]

Thanks Mike - I've just tested it and still having issues with the minions crashing out with a timeout. Is max_grpc_recv_size set in bytes? I will try increasing it to see if it helps

Edit: this could be something else as the minion error differs slightly, and hunts in the GUI that were previously too large load correctly now.

The new error is just a grpc timeout from the frontend rather than anything specific to hunts. I'll do some more tests and see if I can find anything more useful in the logs.