Sentry: db.go:817: log.Fatal: pebble: fatal commit error: no data available (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com/cockroachdb...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5888971341/?referrer=webhooks_plugin

Panic Message:

db.go:817: log.Fatal: pebble: fatal commit error: no data available
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble.(*DB).applyInternal
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:817
  | github.com/cockroachdb/pebble.(*DB).Apply
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:752
  | github.com/cockroachdb/pebble.(*Batch).Commit
  | 	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1155
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit
  | 	github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:556
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:272
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:144
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:979
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:725
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
  | no data available
  | (1) no data available
  | Error types: (1) syscall.Errno
Wraps: (3) log.Fatal: pebble: fatal commit error: no data available
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: pebble: fatal commit error: no data available
*secondary.withSecondaryError
db.go:817: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595

cockroach/pkg/util/stop/stopper.go

Lines 469 to 471 in 6bd6e0c

	sp.UpdateGoroutineIDToCurrent()
	f(ctx)
	}()

https://github.com/cockroachdb/cockroach/blob/6bd6e0c65a2eba531e7a8412d63988abb5b50120/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303
https://github.com/cockroachdb/cockroach/blob/6bd6e0c65a2eba531e7a8412d63988abb5b50120/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396
https://github.com/cockroachdb/cockroach/blob/6bd6e0c65a2eba531e7a8412d63988abb5b50120/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647
https://github.com/cockroachdb/cockroach/blob/6bd6e0c65a2eba531e7a8412d63988abb5b50120/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L724-L726
https://github.com/cockroachdb/cockroach/blob/6bd6e0c65a2eba531e7a8412d63988abb5b50120/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L978-L980

cockroach/pkg/kv/kvserver/logstore/logstore.go

Lines 143 to 145 in 6bd6e0c

	batch := newStoreEntriesBatch(s.Engine)
	return s.storeEntriesAndCommitBatch(ctx, state, m, cb, stats, batch)
	}

cockroach/pkg/kv/kvserver/logstore/logstore.go

Lines 271 to 273 in 6bd6e0c

	} else {
	if err := batch.Commit(willSync); err != nil {
	const expl = "while committing batch"

cockroach/pkg/storage/pebble_batch.go

Lines 555 to 557 in 6bd6e0c

	}
	err := p.batch.Commit(opts)
	if err != nil {

github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L1154-L1156
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L751-L753
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L816-L818

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 725
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 979
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 144
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 272
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).Commit at line 556
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).Commit at line 1155
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Apply at line 752
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 817

Tags

Tag	Value
Command	server
Environment	v23.1.10
Go Version	go1.19.10
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.10
Cockroach SHA	6bd6e0c
# of CPUs	8
# of Goroutines	530

Jira issue: CRDB-42426

[yuzefovich]

unactionable