Expand heap dump and flight recording capture capabilities

[dmikusa]

Right now you can enable heap dumps on OOME or with the Java Memory Assistant. The heap dumps are written to the local file system by default, or can be directed to another location such as a mounted volume.

The same concerns apply to Java Flight Recordings. The recording files are written locally, or to a mounted volume.

There are limitations to what we can do in the container. For example, when there is an OOME, it's very likely that we're close to the memory limit for the container and we do not want to go above that limit and risk crashing the container. Heap dumps can also be rather large (up to the size of the heap) and container disk limits may be lower, to the point where there isn't even enough space to write the heap dump file (Java requires the heap dump to be written to disk somewhere, there's no way to stream the contents out).

There is an opportunity to explore the use of sidecars here to expand on the capabilities we offer. A sidecar could share the file system with the actual Java process but run under a different memory limit so that it could do things like watch the file system and upload heap dumps or flight recordings to an S3-compatible endpoint or copy the file to an SFTP server.

Rather than require users to configure all of this, we believe that buildpacks can assist by packaging generic tools to monitor a location for files of interest (heap dumps, flight recordings, etc..) and efficiently upload those to some remote storage. The buildpack could install a helper utility into the image & set up a process type to run it. The user would then just need to define the sidecar in their K8s YAML and launch the collector/uploader process in the sidecar container.

I'm putting this here under libjvm, as this is to investigate possibilities. This will likely spawn other work, possibly even other buildpacks, depending on what turns up while investigating.