FileExporter in production

[EliyaCohen]

My configuration does not allow direct network communication to the collector, only file transfer. I'd like to use the FileExporter to save the data locally, and then transfer the files to the collector.
Unfortunately, the FileExporter readme states:

Please note that there is no guarantee that exact field names will remain stable. This intended for primarily for debugging Collector without setting up backends.

Do you know of any other solutions that will allow me to save the files locally?

[bogdandrutu]

Hi @EliyaCohen,

Unfortunately we do not have a FileExporter that would much this use-case, but I think it is a very important one. Also our current fileexporter writes json (probably better to write binary to save space and overhead), we do not have file rotation etc.

But I would like to see this possible, I see that you filed https://github.com/open-telemetry/opentelemetry-collector/issues/2450, I think we should start by resolving that, then probably add an "encoding/serialization" flag to decide between proto binary vs json, even consider to support jaeger, zipkin, etc in the future.

Also you may need a receiver for this as well. Unfortunately @tigrannajaryan was a bit lazy :) and only added some kind of file receiver support for testbed open-telemetry/opentelemetry-collector#2444, and not as a standard receiver.

I will support this, but I would like to have an "umbrella issue" that covers all the needs for this to be supported where we document the high-level needs, then individual issues where we track details about those.

[EliyaCohen]

Great! I'll work on the issue documenting all the high-level needs.
Thanks @bogdandrutu

[tigrannajaryan]

@bogdandrutu I believe we need a formal specification for OTLP/File that defines how OTLP records are stored in a file. I filed an issue to work on it: open-telemetry/opentelemetry-specification#1443

[bogdandrutu]

@tigrannajaryan I know for sure that you know about https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver. Is it a possibility to have a simpler version of this which only supports file rotation, tail parsing, etc of files that contain traces/metrics/logs in some standard formats like otlp [binary/json] (initially should be enough)? Is this functionality split into a library that we can reuse?

[bogdandrutu]

@tigrannajaryan great, but I think we should update the current FileExporter to support binary and all the other features that we need. Is that in scope of that work?

[tigrannajaryan]

Unless we think it is urgently needed I would prefer to first decide on the format, have it in the spec and then after that spend time implementing it in the Collector. Adding support to dump binary Protobufs to a file in FileExporter is easy once we know what the format is (e.g. how do you delineate Protobuf records in the file? Do we prefix each record with size? How do we know which Protobuf message it is? Anything else in the prefix? Do we have a file header? etc).

[bogdandrutu]

There are other features that we need and you are adding them in the logs:

• File rotation
• Tail reads support

Do you think we can have these things discussed now?

[tigrannajaryan]

I spoke with @djaglowski today and he says it is not possible today with log-collection library but he believes can be refactored as a separately useable functionality in the future.

Issue recorded here https://github.com/open-telemetry/opentelemetry-log-collection/issues/19

[bogdandrutu]

Related to this https://github.com/open-telemetry/opentelemetry-collector/issues/3230