Panic when carbon-relay connects to carbon-cache on ARM/V7

[rbuffi]

Hello,

My carbon-relay container panics when connecting to carbon-cache container on k3s (arm/v7):

[2020-08-21 11:31:05] check if the port 3003 for 'carbon-cache-service' is available
[2020-08-21 11:31:06] ----------------------------------------------------
[2020-08-21 11:31:06] carbon-relay-ng - build:
[2020-08-21 11:31:06] ----------------------------------------------------
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11fd0]
goroutine 34 [running]:
github.com/grafana/carbon-relay-ng/statsmt.(*LatencyHistogram15s32).Value(0x2db6000, 0x29936, 0x0)
/opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/latencyhistogram15s32.go:26 +0x5c
github.com/grafana/carbon-relay-ng/statsmt.(*Graphite).writer(0x2d80480)
/opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:118 +0x29c
created by github.com/grafana/carbon-relay-ng/statsmt.NewGraphite
/opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:52 +0x35c

Maybe you can point me in the right direction.
Im running a K3s cluster on RPI4 (ARM/V7).
A container with carbon-cache is already running.
Now i'm trying to get carbon-relay running by doing the following:

git clone https://github.com/bodsch/docker-carbon-relay-ng
cd docker-carbon-relay-ng
docker buildx build --platform linux/arm/v7 -t containerregistrydcsl.azurecr.io/metrics/docker-carbon-relay-ng --push .

And creating the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
name: carbon-relay-ng
labels:
app: metrics
spec:
replicas: 2
selector:
matchLabels:
app: metrics
template:
metadata:
labels:
app: metrics
spec:
containers:

• name: carbon-relay-ng
  image: containerregistrydcsl.azurecr.io/metrics/docker-carbon-relay-ng
  env:
• name: GRAPHITE_HOST
  value: "carbon-cache-service"
• name: GRAPHITE_PORT
  value: "3003"
  ports:
• containerPort: 2003
• containerPort: 2004
• containerPort: 8081
  imagePullSecrets:
• name: containerregistrydcsl.azurecr.io

When I start the carbon-relay container from docker on my Mac (x86) it connect to carbon-cache without problems:

docker run -d
-p 2003:2003
-p 2004:2004
--name=docker-carbon-relay
-e GRAPHITE_HOST='192.168.0.202'
-e GRAPHITE_PORT='3003'
-d containerregistrydcsl.azurecr.io/metrics/docker-carbon-relay-ng

Thanks in advance,

Ronald

[rbuffi]

This is the output with verbose logging:

2020-08-22 18:50:19.847 [INFO] ===== carbon-relay-ng instance 'carbon-relay-ng-7b898f7c97-8jlsj' starting. (version 0.13.0-4-gc9044b5) =====
2020-08-22 18:50:19.847 [DEBUG] setting GOMAXPROCS to 4
2020-08-22 18:50:19.853 [INFO] initializing routing table...
2020-08-22 18:50:19.870 [INFO] ===========================
2020-08-22 18:50:19.870 [INFO] ========== TABLE ==========
2020-08-22 18:50:19.870 [INFO] ===========================
2020-08-22 18:50:19.870 [INFO]
2020-08-22 18:50:19.870 [INFO] ## Rewriters:
2020-08-22 18:50:19.870 [INFO] old new not max
2020-08-22 18:50:19.870 [INFO] ==================
2020-08-22 18:50:19.870 [INFO] /^/ -1
2020-08-22 18:50:19.870 [INFO]
2020-08-22 18:50:19.871 [INFO] ## Blacklist:
2020-08-22 18:50:19.871 [INFO] prefix notPrefix sub notSub regex notRegex
2020-08-22 18:50:19.871 [INFO] ===============================================
2020-08-22 18:50:19.871 [INFO]
2020-08-22 18:50:19.871 [INFO] ## Aggregations:
2020-08-22 18:50:19.871 [INFO] key func regex notRegex prefix notPrefix sub notSub outFmt cache interval wait dropRaw
2020-08-22 18:50:19.871 [INFO] ==================================================================================================
2020-08-22 18:50:19.871 [INFO]
2020-08-22 18:50:19.871 [INFO] ## Routes:
2020-08-22 18:50:19.871 [INFO] type key prefix notPrefix sub notSub regex notRegex
2020-08-22 18:50:19.871 [INFO] ================================================================================================================================================
2020-08-22 18:50:19.871 [INFO] > sendAllMatch carbon-default
2020-08-22 18:50:19.871 [INFO] prefix notPrefix sub notSub regex notRegex addr spoolDir spool pickle online
2020-08-22 18:50:19.871 [INFO] ----------------------------------------------------------------------------------------------------------------------------------------------
2020-08-22 18:50:19.871 [INFO] carbon-cache-service:3003 /var/spool/carbon-relay-ng-7b898f7c97-8jlsj false false false
2020-08-22 18:50:19.871 [INFO]
2020-08-22 18:50:19.871 [INFO]
2020-08-22 18:50:19.874 [INFO] listening on 0.0.0.0:2013/tcp
2020-08-22 18:50:19.874 [INFO] listening on 0.0.0.0:2003/udp
2020-08-22 18:50:19.874 [INFO] listening on 0.0.0.0:2003/tcp
2020-08-22 18:50:19.875 [DEBUG] dest carbon-default_carbon-cache-service_3003 (re)connecting to carbon-cache-service:3003
2020-08-22 18:50:19.874 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: false spooling: false slowLastloop: false, slowNow: false spoolQueue: false
2020-08-22 18:50:19.875 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: false spooling: false slowLastloop: false, slowNow: false spoolQueue: false
2020-08-22 18:50:19.875 [INFO] admin TCP listener starting on 0.0.0.0:2004
2020-08-22 18:50:19.875 [INFO] listening on 0.0.0.0:2013/udp
2020-08-22 18:50:19.880 [INFO] admin HTTP listener starting on 0.0.0.0:8081
2020-08-22 18:50:19.888 [DEBUG] dest carbon-default_carbon-cache-service_3003 connected to carbon-cache-service:3003
2020-08-22 18:50:19.888 [INFO] dest carbon-default_carbon-cache-service_3003 new conn online
2020-08-22 18:50:19.888 [DEBUG] conn carbon-default_carbon-cache-service_3003 .up query responded with true
2020-08-22 18:50:19.888 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: true spooling: false slowLastloop: false, slowNow: false spoolQueue: false
2020-08-22 18:50:19.889 [DEBUG] conn carbon-default_carbon-cache-service_3003 .up query responded with true
2020-08-22 18:50:19.889 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: true spooling: false slowLastloop: false, slowNow: false spoolQueue: false
2020-08-22 18:50:20.000 [DEBUG] stats flushing for 2020-08-22 18:50:20 +0000 UTC m=+0.176198045 to graphite
2020-08-22 18:50:20.890 [DEBUG] conn carbon-default_carbon-cache-service_3003 HandleData: c.buffered auto-flushing...
2020-08-22 18:50:20.890 [DEBUG] conn carbon-default_carbon-cache-service_3003 HandleData c.buffered auto-flush done without error
2020-08-22 18:50:20.890 [DEBUG] conn carbon-default_carbon-cache-service_3003 HandleData auto-flush 317.774µs (total iter 1.000484875s) (use this to tune your In buffering)
2020-08-22 18:50:21.000 [DEBUG] stats flushing for 2020-08-22 18:50:21 +0000 UTC m=+1.176197712 to graphite
2020-08-22 18:50:21.014 [INFO] stats now connected to carbon-cache-service:3003
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11fd0]

goroutine 8 [running]:
github.com/grafana/carbon-relay-ng/statsmt.(*LatencyHistogram15s32).Value(0x210c5a0, 0x32df0, 0x0)
/opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/latencyhistogram15s32.go:26 +0x5c
github.com/grafana/carbon-relay-ng/statsmt.(*Graphite).writer(0x200b9a0)
/opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:118 +0x29c
created by github.com/grafana/carbon-relay-ng/statsmt.NewGraphite
/opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:52 +0x35c