WIP: Parrallel import #1588

joshmeranda · 2023-07-20T14:57:03Z

Support parallel read and write with internal data buffer

plugins/metrics/pkg/agent/runner.go

joshmeranda added 3 commits July 19, 2023 15:19

parralelize import read and push

778427f

give each import task its own remote reader

66ea27f

fix runner tests

89573ab

joshmeranda requested review from kralicky and jan-law July 20, 2023 14:57

joshmeranda and others added 2 commits July 20, 2023 15:09

fix linting issues

e2f91a1

add import buffer volume

ba90671

jan-law reviewed Jul 21, 2023

View reviewed changes

plugins/metrics/pkg/agent/runner.go Outdated Show resolved Hide resolved

joshmeranda added 3 commits July 21, 2023 21:40

add disk buffer

b01a0df

consolodate each task buffer into one

9508a7b

properlly handle buffer not found err

fa3f594

joshmeranda force-pushed the parrallel-import branch from 0c2172d to 047c03a Compare July 22, 2023 06:52

reconcile chunks on pod restart

bac6b02

joshmeranda force-pushed the parrallel-import branch from 047c03a to bac6b02 Compare July 22, 2023 06:55

joshmeranda and others added 2 commits July 22, 2023 05:22

fix data race with err

5468970

use in-memory buffer when volume is not mounted

e65f9f5

Provide feedback