Neural Embedding Compression For Efficient Multi-Task Earth Observation Modelling

[brunosan]

2403.17886.pdf
8 Apr 2024

OpenAI candid summary:

The paper introduces Neural Embedding Compression (NEC) as a method to efficiently transfer compressed multi-task embeddings for earth observation (EO) data instead of the raw data itself. Key points:

EO data repositories are huge and transferring raw data is costly. The paper proposes transferring compressed embeddings generated by a foundation model (FM) encoder.
The FM encoder is adapted using learned neural compression to produce compressible embeddings while minimizing loss across downstream tasks. Only a small fraction (~10%) of FM parameters are updated.
NEC is evaluated on scene classification and semantic segmentation EO tasks. Compared to compressing the raw data, NEC achieves similar accuracy with 75-90% data reduction.
Even at 99.7% compression, NEC performance drops by only 5% for scene classification.
In summary, NEC provides a data-efficient way to share EO data via compressed embeddings with minimal performance loss, greatly reducing transfer and storage costs compared to using raw data. The approach is novel and results are promising for efficient multi-task EO modeling.

[brunosan]

This paper presents a method to increase the compression rate of EO embeddings in a way that maximizes the accuracy of downstream tasks (versus just trying to mimic the input images).

The idea is that if you need to do a specific task with EO, like labels, you can download the whole EO image to do your work, or you can download the embeddings made with a foundational model and save a ton of bandwidth. But doing so will worsen your results, not only becuase the embeddings are not perfect, especially coming from a general model. Moreover we can tweak an existing foundational model (10% of it for 1% of the time) in a way that further lowers the bits downloaded. This is the work presented here, called NEC. The performace of this method further increases if aside from the embedding, you also share the 1st layer of the model decoder.

The full image, and full image compressed ( RDC) are the dots. From right to left the precision of the task lowers as you restrict the bandwidth available.

Results:

• The most important result is the general validation of the concept that embeddings can replace working with input images in many cases. Having a full archive of embeddings greatly complements an EO archive.
• NEC seems very important for bandwidths below ~20KB per compressed embedding (~4$%-~1% of the size if input is 500Kb-1.5Mb). Sharing the Head of the decoder becomes crucial for very low bandwidths. For better bandwidths, the proposed NEC method is similar to just quantizing the embeddings, or keeping them raw.

  NEC is able to achieve 94% accuracy at an average embedding size of 0.47KB, a reduction of only 5% in accuracy for a size reduction of ∼700x, greatly outperforming 3 and 2-bit UQE.

Note:
*The model used here is not EO specific, but a ViT.

[lukaskondmann]

Cool summary :) I agree that this is only really required if bandwidth is very limited. However, we have a couple of cases in EO where this is actually the case in my opinion. I am primarily thinking of on-orbit processing here. For downlink or processing bottlenecks on orbit a compression like this could come in very handy and I am happy to see this field progressing :)