Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: runtime error: index out of range [0] with length 0 #28

Closed
Antiarchitect opened this issue Feb 10, 2022 · 12 comments
Closed

panic: runtime error: index out of range [0] with length 0 #28

Antiarchitect opened this issue Feb 10, 2022 · 12 comments
Labels
bug Something isn't working

Comments

@Antiarchitect
Copy link

Antiarchitect commented Feb 10, 2022

What happened:

Facing panic: runtime error: index out of range [0] with length 0

What you expected to happen:

No runtime panic.

How to reproduce it (as minimally and precisely as possible):

Do not know - happens periodically.

Anything else we need to know?:

Dockerfile:

FROM golang:1.17.6-bullseye AS builder

ARG COREDNS_VERSION=1.9.0
ARG UNBOUND_PLUGIN_VERSION=0.0.7

ARG CGO_ENABLED=1

WORKDIR /coredns

RUN apt-get update \
    && apt-get install -y --no-install-recommends libunbound-dev \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/* \
    && curl -L -s https://github.com/coredns/coredns/archive/refs/tags/v${COREDNS_VERSION}.tar.gz | tar --strip-components=1 -xzf - -C . \
    && go get "github.com/coredns/unbound@v${UNBOUND_PLUGIN_VERSION}" \
    && echo "unbound:github.com/coredns/unbound" >> plugin.cfg \
    && go generate \
    && go build


FROM debian:11.2-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates libunbound8 \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /coredns/coredns /opt/coredns

CMD ["/opt/coredns"]

Environment:

  • the version of CoreDNS: 1.9.0
  • Corefile:
# Managed by puppet
(common) {
  errors
  log
  prometheus 0.0.0.0:9253
}

(acl_trusted) {
  acl {
    # Some allow net with private and public IPs which I will not reveal
    block
  }
}

. {
    import common
    reload
    cache 3600
    unbound
    debug
}

local. {
    import common
    import acl_trusted
}

xxx.local. {
    import common
    import acl_trusted
    cache 3600
    forward xxx.local. 10.1.1.101 10.1.1.102
}

acme.local. {
    import common
    file zones/acme.local.db {
        reload 10s
    }
}
  • logs, if applicable:
2022-02-10T14:12:17.2839 193.9.249.145 coredns  panic: runtime error: index out of range [0] with length 0                          
2022-02-10T14:12:17.2839 193.9.249.145 coredns
2022-02-10T14:12:17.2839 193.9.249.145 coredns  goroutine 879 [running]:
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.key({0xc000748258, 0x9b87f4}, 0xc00000e5e8, 0x0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/cache.go:78 +0x85
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.(*ResponseWriter).WriteMsg(0xc00016a6c0, 0xc0005cf5f0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/cache.go:149 +0xd9
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/unbound.(*Unbound).ServeDNS(0xc00014ae10, {0x22c4df8, 0xc000710a20}, {0x22f
6fc8, 0xc00016a6c0}, 0xc0007093b0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /go/pkg/mod/github.com/coredns/[email protected]/unbound.go:140 +0x538
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ebf0d6, 0xc000103000}, {0x22a7df8, 0xc0001
4ae10}, {0x22c4df8, 0xc000710a20}, {0x22f6fc8, 0xc00016a6c0}, 0x1e55440)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.(*Cache).doRefresh(0xc000623ef0, {0x22c4df8, 0xc000710
a20}, {0xc0007093b0, {0x22f7230, 0xc000d0d540}, {0x0, 0x0}, 0x200, 0x0, ...}, ...)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/handler.go:77 +0xd9
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.(*Cache).ServeDNS(0xc000623ef0, {0x22c4df8, 0xc000710a
20}, {0x22f7230, 0xc000d0d540}, 0xc000079740)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/handler.go:43 +0x90e
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ebdb91, 0x1}, {0x22a7718, 0xc000623ef0}, {
0x22c4df8, 0xc000710a20}, {0x22f7230, 0xc000d0d540}, 0x4444e5)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/log.Logger.ServeDNS({{0x22a7718, 0xc000623ef0}, {0xc00064c9f
0, 0x1, 0x1}, {}}, {0x22c4df8, 0xc000710a20}, {0x22f7180, 0xc000d0d500}, ...)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/log/log.go:36 +0x346
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ec067e, 0xc000748258}, {0x22ad230, 0xc0006
8f2f0}, {0x22c4df8, 0xc000710a20}, {0x22f7180, 0xc000d0d500}, 0x400)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/errors.(*errorHandler).ServeDNS(0xc00064c900, {0x22c4df8, 0x
c000710a20}, {0x22f7180, 0xc000d0d500}, 0xc000709320)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/errors/errors.go:84 +0x87
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ec6d92, 0x1}, {0x22a77b8, 0xc00064c900}, {
0x22c4df8, 0xc000710a20}, {0x22f7180, 0xc000d0d500}, 0x0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/coredns/coredns/plugin/metrics.(*Metrics).ServeDNS(0xc000622cf0, {0x22c4df710a20}, {0x22f7338, 0xc00000e5e8}, 0xc000709320)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /coredns/plugin/metrics/handler.go:27 +0x255
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/coredns/coredns/core/dnsserver.(*Server).ServeDNS(0xc00012ef60, {0x22c4df8, 0xc0007
10a20}, {0x22f7338, 0xc00000e5e8}, 0xc000709320)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /coredns/core/dnsserver/server.go:280 +0x64d
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/coredns/coredns/core/dnsserver.(*Server).ServePacket.func1({0x22f8e08, 0xc000862a80
}, 0xc000709301)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /coredns/core/dnsserver/server.go:128 +0x9b
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/miekg/dns.HandlerFunc.ServeDNS(0xc0002dea00, {0x22f8e08, 0xc000862a80}, 0xc00070932
0)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:37 +0x2f
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/miekg/dns.(*Server).serveDNS(0xc000712360, {0xc0002dea00, 0x27, 0x0}, 0xc000862a80)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:659 +0x43e
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/miekg/dns.(*Server).serveUDPPacket(0xc000712360, 0x0, {0xc0002dea00, 0x27, 0x200},
{0x22ea3f0, 0xc000010198}, 0xc0003ec6c0, {0x0, 0x0})
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:603 +0x1dc
2022-02-10T14:12:17.2842 193.9.249.145 coredns  created by github.com/miekg/dns.(*Server).serveUDP
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:533 +0x485
  • OS (e.g: cat /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Others:

docker version:

Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:51:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:22:32 2019
  OS/Arch:          linux/amd64
  Experimental:     false
@Antiarchitect Antiarchitect added the bug Something isn't working label Feb 10, 2022
@chrisohaver
Copy link
Member

Per stack trace, unbound is trying to write an invalid response without the question in it.
Looks like an unbound problem.

@chrisohaver chrisohaver transferred this issue from coredns/coredns Feb 10, 2022
@Antiarchitect
Copy link
Author

#25 Seems related

@chrisohaver
Copy link
Member

chrisohaver commented Feb 10, 2022

#25 Seems related

It's the same mode of failure. unbound.Unbound.Resolve() is returning a result with an AnswerPacket that does not contain a Question section.

Probably best to figure out why that happens in the first place in github.com/miekg/unbound, but a hacky-fix here would be to add the section if it's missing. E.g. something along the lines of ...

if len(res.AnswerPacket.Question) == 0 {
   res.AnswerPacket.Question = r.Question
}

Although it would be a safer hack to SERVFAIL when len(res.AnswerPacket.Question) == 0. It's likely that in these cases that it's not just the Question that is missing (e.g. default msg struct), and the response is not valid even with the question hacked back in. e.g. lines 121-123:

if err != nil || len(res.AnswerPacket.Question) == 0 { 
 	return dns.RcodeServerFailure, err 
 } 

@Antiarchitect
Copy link
Author

Will you make a PR? I'm not good with golang.

@miekg
Copy link
Member

miekg commented Feb 12, 2022

do you have a test case? I think I can paper over it in miekg/unbound, but not sure if that is the best approach

@chrisohaver
Copy link
Member

chrisohaver commented Feb 23, 2022

I think I can paper over it in miekg/unbound, but not sure if that is the best approach

A papered over solution probably better to live here in the plugin - e.g. to return SERVFAIL if the AnswerPacket is missing a Question.

@chrisohaver
Copy link
Member

proper fix upstream with miekg/unbound#18. Thanks miekg!

@Dart2112
Copy link

Dart2112 commented Aug 5, 2023

Can the fix be applied in this repo since the upstream fix has stalled before being merged? It has been over a year, so I assume it won't be.

@hbzhost
Copy link

hbzhost commented Oct 18, 2023

I think my issue might be related to this one since I'm getting same error message "[ERROR] Recovered from panic in server: "dns://:53" runtime error: index out of range [0] with length 0" with unbound plugin when I try to resolve IBM Cloud Kafka broker instance - broker-5-48d5s71514qnmqz3.kafka.svc11.us-south.eventstreams.cloud.ibm.com. Seems like it might be related to trying to resolve long sub-domains, but again not sure.

When I was testing this 9/10 times it throws error message above and only once time it resolves and returns CNAME.

@jdoss
Copy link

jdoss commented Jun 10, 2024

I echo the need for miekg/unbound#18 to be patched in this repo since upstream has stalled on merging in the fix. For others that are looking for a quick not ideal workaround you can manually patch the three line change with the following:

Grab the patch from the PR:

curl -sL -o pr18.patch https://patch-diff.githubusercontent.com/raw/miekg/unbound/pull/18.patch

If you are using a container to build a custom CoreDNS binary with this plugin you can apply it with the following:

RUN go mod download
COPY plugin.cfg $GOPATH/src/github.com/coredns/coredns
RUN make gen
COPY pr18.patch /root/go/pkg/mod/github.com/miekg/unbound\@v0.0.0-20210309082708-dbeefb4cdb29
WORKDIR /root/go/pkg/mod/github.com/miekg/unbound\@v0.0.0-20210309082708-dbeefb4cdb29
RUN patch -p1 < pr18.patch
WORKDIR $GOPATH/src/github.com/coredns/coredns

I have tested this with the latest CoreDNS 1.11.3 pre-release and it works as expected:

$ podman exec coredns drill A 2.0.0.127.zen.spamhaus.org
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 51610
;; flags: qr rd ra ; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; 2.0.0.127.zen.spamhaus.org.	IN	A

;; ANSWER SECTION:
2.0.0.127.zen.spamhaus.org.	60	IN	A	127.0.0.4
2.0.0.127.zen.spamhaus.org.	60	IN	A	127.0.0.2
2.0.0.127.zen.spamhaus.org.	60	IN	A	127.0.0.10

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 392 msec
;; SERVER: 127.0.0.1
;; WHEN: Mon Jun 10 18:09:10 2024
;; MSG SIZE  rcvd: 170

@miekg
Copy link
Member

miekg commented Jun 13, 2024

I clicked a button

@miekg miekg closed this as completed Jun 13, 2024
@jdoss
Copy link

jdoss commented Jun 13, 2024

@miekg you are the best! Thanks dude!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants