x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time

[xcolwell]

Go version

go version go1.22.5 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/brien/Library/Caches/go-build'
GOENV='/Users/brien/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/brien/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/brien/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/r3/v_3z60rx2cxg0s1r9tl9fbmw0000gn/T/go-build1492278578=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Build a native aar for Android as:

	gomobile bind \
		-ldflags "-X client.Version=$$WARP_VERSION" \
		-target=android -androidapi 19 \
		-javapkg com.bringyour \
		-trimpath \
		-o build/android/BringYourClient.aar \

The code has interface callbacks that call from Go -> Kotlin -> Go.

What did you see happen?

Works up to go 1.22.4. On 1.22.5, the interface callback raises the error below. This happens in a Kotlin interface implementation called from Go, that is making another call to Go.

2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime: newstack at runtime.printlock+0x78 sp=0x400005eb00 stack=[0x400005e000, 0x400005f000]
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	morebuf={pc:0x6f97d94980 sp:0x400005eb00 lr:0x0}
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	sched={pc:0x6f97dca218 sp:0x400005eb00 lr:0x6f97d94980 ctxt:0x0}
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.callbackUpdateSystemStack(0x4000052008, 0x7fd7e4b9d0, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:241 +0x90 fp=0x400005eb60 sp=0x400005eb00 pc=0x6f97d94980
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815db90, 0x7fd7e4bac8, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:306 +0x68 fp=0x400005ebe0 sp=0x400005eb60 pc=0x6f97d94b48
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815db90, 0x7fd7e4bac8, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	<autogenerated>:1 +0x1c fp=0x400005ec10 sp=0x400005ebe0 pc=0x6f97e0163c
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallback(0x400005ec88, 0x6f98136f90, 0x6f9817bd10)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:1130 +0xb0 fp=0x400005ec40 sp=0x400005ec10 pc=0x6f97dff980
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.systemstack_switch()
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:200 +0x8 fp=0x400005ec50 sp=0x400005ec40 pc=0x6f97dfd548
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocall(0x6f9817bd10, 0x400005ecc8)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:175 +0x70 fp=0x400005ec90 sp=0x400005ec50 pc=0x6f97d94890
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  main._Cfunc_cproxyclient_ProvideChangeListener_ProvideChanged(0x2a, 0x1)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	_cgo_gotypes.go:550 +0x30 fp=0x400005ecc0 sp=0x400005ec90 pc=0x6f98136f90
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  main.(*proxyclient_ProvideChangeListener).ProvideChanged(0x40002fc204, 0x1)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/private/var/folders/r3/v_3z60rx2cxg0s1r9tl9fbmw0000gn/T/gomobile-work-233819857/src-android-arm64/gobind/go_clientmain.go:7243 +0x58 fp=0x400005ece0 sp=0x400005ecc0 pc=0x6f981560a8
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/client.(*BringYourDevice).provideChanged.func1()
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/bringyour/client/device.go:241 +0x2c fp=0x400005ed00 sp=0x400005ece0 pc=0x6f9812423c
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/connect.HandleError(0x40001280c0?, {0x0?, 0x400005edb8?, 0x6f981247a8?})
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/connect/connect/trace.go:65 +0x70 fp=0x400005ed60 sp=0x400005ed00 pc=0x6f980f0e70
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/client.(*BringYourDevice).provideChanged(0x4000314240?, 0x1)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/bringyour/client/device.go:240 +0x70 fp=0x400005edc0 sp=0x400005ed60 pc=0x6f981241e0
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/client.(*BringYourDevice).SetProvideMode(0x4000314240, 0x6f97d94f88?)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/bringyour/client/device.go:311 +0x34 fp=0x400005ede0 sp=0x400005edc0 pc=0x6f981247b4
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  main.proxyclient_BringYourDevice_SetProvideMode(0x5ee01?, 0x3)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/private/var/folders/r3/v_3z60rx2cxg0s1r9tl9fbmw0000gn/T/gomobile-work-233819857/src-android-arm64/gobind/go_clientmain.go:1610 +0x40 fp=0x400005ee10 sp=0x400005ede0 pc=0x6f9813e7b0
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  _cgoexp_5858fad8bb02_proxyclient_BringYourDevice_SetProvideMode(0x400005ee88?)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	_cgo_gotypes.go:2126 +0x28 fp=0x400005ee30 sp=0x400005ee10 pc=0x6f9815df18
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg1(0x6f9815def0, 0x7fd7e50960, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:420 +0x228 fp=0x400005ef00 sp=0x400005ee30 pc=0x6f97d94ec8
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815def0, 0x7fd7e50960, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:339 +0x10c fp=0x400005ef80 sp=0x400005ef00 pc=0x6f97d94bec
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815def0, 0x7fd7e50960, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	<autogenerated>:1 +0x1c fp=0x400005efb0 sp=0x400005ef80 pc=0x6f97e0163c
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallback(0x0, 0x0, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:1130 +0xb0 fp=0x400005efe0 sp=0x400005efb0 pc=0x6f97dff980
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.goexit({})
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400005efe0 sp=0x400005efe0 pc=0x6f97dffa74
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  fatal error: runtime: stack split at bad time

What did you expect to see?

No error.

[gabyhelp]

Related Issues and Documentation

• runtime: Crash on package init with fatal error: bad pointer in write barrier #15831 (closed)
• x/mobile: App crashed on startup. #26009 (closed)
• x/mobile, runtime: libgojni.so crashes on ARM64 devices for methods defined in Go code #65057 (closed)
• runtime: Android shared lib panic #15856 (closed)
• syscall: accept(2) syscall blocked on Android, use accept4(2) instead #45767 (closed)
• cmd/cgo: jmethodID/jfieldID is not mapped to uintptr if building with the Android NDK #40954 (closed)
• x/mobile: Process crashed as JNI exception isn't cleaned before next call. #21033
• affected/package: #63512 (closed)
• x/mobile: Crash gomobile-app with gomobile bind #25293 (closed)
• question: ARM v5 not working on Android with clang #29729 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

cc @cherrymui @golang/runtime

[prattmic]

This is presumably a bug in https://go.googlesource.com/go/+/3560cf0afb3c29300a6c88ccd98256949ca7a6f6, which is in go1.22.5.

@xcolwell could you test with this commit reverted? Something like this should do the trick:

$ git clone https://go.googlesource.com/go
$ cd go
$ git checkout release-branch.go1.22
$ git revert 3560cf0afb3c29300a6c88ccd98256949ca7a6f6
$ cd src
$ ./make.bash

Then build with the toolchain in $CHECKOUT/bin. (I guess this means have this in PATH when running gomobile? I'm not familiar with gomobile)

[xcolwell]

@prattmic that worked. I changed the installed go to the patched on with the below, and verified that the gomobile build worked when using the patched go, and failed when using 1.22.5.

cd /Users/brien/bringyour/temp/go-issue-68760
# your steps
sudo cp -r /Users/brien/bringyour/temp/go-issue-68760/go /usr/local/go.patched
sudo mv /usr/local/go /usr/local/go.1.22.5
sudo mv /usr/local/go.patched /usr/local/go
go version
> go version go1.22.6 darwin/arm64

As a summary of what the mobile code looks like, we have a gomobile lib that lets us register callbacks from Kotlin. There is a sequence like below. When the Kotlin callback is called from within the gomobile lib, the first function call back into the gomobile lib crashes with the error.

// byDevice is a Go object in the gomobile lib
deviceProvideSub = byDevice?.addProvideChangeListener { provideEnabled ->
    runBlocking(Dispatchers.Main.immediate) {
        updateVpnService()
    }
}

private fun updateVpnService() {
  val byDevice = byDevice ?: return
  
  // the first call back into the gomobile lib crashes with the error
  val provideEnabled = byDevice.isProvideEnabled
  // ...
}

[ignoramous]

As a summary of what the mobile code looks like, we have a gomobile lib that lets us register callbacks from Kotlin. There is a sequence like below. When the Kotlin callback is called from within the gomobile lib, the first function call back into the gomobile lib crashes with the error.

Can confirm. We see the same error but in go1.23: celzero/firestack#84

[ClementFr49]

What did you do?

Build a native aar for Android with cmd :

mkdir -p target
go install golang.org/x/mobile/cmd/gomobile@latest
go get -d golang.org/x/mobile/cmd/gomobile
gomobile init
gomobile bind -target android/arm64 -androidapi 26 -o target/gomobile-test-go.aar -javapkg com.gomobile.test ./cmd

Go version

• Tested on darwin/arm64
• Go <= 1.22.4 : all my code is working as expected

1. This code cause error only with go > 1.22.4

Gomobile go code

var ReqLauncher = IToBeImplemented.Generate

// instanciated by mobile
type IToBeImplemented interface {
	Generate(test ITest)
}

type IGoImpl interface {
	TriggerGoAndLaunchTest()
}
type GoImpl struct {
	impl IToBeImplemented
}

func InstanceGetter(mobileImpl IToBeImplemented) IGoImpl {
	return GoImpl{impl: mobileImpl}
}

func (i GoImpl) TriggerGoAndLaunchTest() {
	ReqLauncher(i.impl, TestImplemented{valueExample: "the go lib is on " + runtime.Version()})
}

type ITest interface {
	Info() string
}
type TestImplemented struct {
	valueExample string
}

func (t TestImplemented) Info() string {
	return t.valueExample
}

Gomobile android code

var res by remember { mutableStateOf("") }
val inst = Cmd.instanceGetter(object : IToBeImplemented {
    override fun generate(test: ITest) {
        res = test.info()
    }
})
Surface(modifier = Modifier.fillMaxSize()) {
    Column(verticalArrangement = Arrangement.Center, horizontalAlignment = Alignment.CenterHorizontally) {
        Text(text = res)
        Button(onClick = {inst.triggerGoAndLaunchTest()}) { Text("call go") }
    }
}

Error

runtime: newstack at runtime.printlock+0x78 sp=0x4000056bb0 stack=[0x4000056000, 0x4000057000]
	morebuf={pc:0x7175880670 sp:0x4000056bb0 lr:0x0}
	sched={pc:0x71758b1828 sp:0x4000056bb0 lr:0x7175880670 ctxt:0x0}
runtime.callbackUpdateSystemStack(0x4000046008, 0x7fc994ff60, 0x0)
	/Users/.../go/go1.23.2/src/runtime/cgocall.go:257 +0x90 fp=0x4000056c10 sp=0x4000056bb0 pc=0x7175880670
runtime.cgocallbackg(0x717591c080, 0x7fc9950058, 0x0)
	/Users/.../go/go1.23.2/src/runtime/cgocall.go:322 +0x68 fp=0x4000056c60 sp=0x4000056c10 pc=0x7175880838
runtime.cgocallbackg(0x717591c080, 0x7fc9950058, 0x0)
	<autogenerated>:1 +0x1c fp=0x4000056c90 sp=0x4000056c60 pc=0x71758eecbc
runtime.cgocallback(0x4000056d08, 0x717591b050, 0x717591c748)
	/Users/.../go/go1.23.2/src/runtime/asm_arm64.s:1131 +0xb0 fp=0x4000056cc0 sp=0x4000056c90 pc=0x71758ed040
runtime.systemstack_switch()
	/Users/.../go/go1.23.2/src/runtime/asm_arm64.s:201 +0x8 fp=0x4000056cd0 sp=0x4000056cc0 pc=0x71758eac08
runtime.cgocall(0x717591c748, 0x4000056d48)
	/Users/.../go/go1.23.2/src/runtime/cgocall.go:185 +0x70 fp=0x4000056d10 sp=0x4000056cd0 pc=0x71758e11b0
main._Cfunc_cproxycmd_IToBeImplemented_Generate(0x2a, 0xffffffe7)
	_cgo_gotypes.go:141 +0x30 fp=0x4000056d40 sp=0x4000056d10 pc=0x717591b050
main.(*proxycmd_IToBeImplemented).Generate(0x4000010018, {0x71759425e8?, 0x400009e090?})
	/private/var/folders/4d/c_sr4j810rd6f56r6jx6w80c0000gn/T/gomobile-work-1694386097/src-android-arm64/gobind/go_cmdmain.go:116 +0x88 fp=0x4000056d70 sp=0x4000056d40 pc=0x717591b8a8
testgomobile/cmd.IToBeImplemented.Generate({0x71759425a8?, 0x4000010018?}, {0x71759425e8?, 0x400009e090?})
	<autogenerated>:1 +0x44 fp=0x4000056da0 sp=0x4000056d70 pc=0x717591a454
testgomobile/cmd.GoImpl.TriggerGoAndLaunchTest(...)
	/Users/.../workspace/projets/gomobile/test/testgomobile/cmd/cmd.go:24
testgomobile/cmd.(*GoImpl).TriggerGoAndLaunchTest(0x4000012060?)
	<autogenerated>:1 +0x8c fp=0x4000056e00 sp=0x4000056da0 pc=0x717591a53c
main.proxycmd_IGoImpl_TriggerGoAndLaunchTest(0xffffff01?)
	/private/var/folders/4d/c_sr4j810rd6f56r6jx6w80c0000gn/T/gomobile-work-1694386097/src-android-arm64/gobind/go_cmdmain.go:55 +0x84 fp=0x4000056e40 sp=0x4000056e00 pc=0x717591b454
_cgoexp_84a54eed484f_proxycmd_IGoImpl_TriggerGoAndLaunchTest(0x4000056eb8?)
	_cgo_gotypes.go:227 +0x20 fp=0x4000056e60 sp=0x4000056e40 pc=0x717591c060
runtime.cgocallbackg1(0x717591c040, 0x7fc9951138, 0x0)
	/Users/.../go/go1.23.2/src/runtime/cgocall.go:442 +0x21c fp=0x4000056f30 sp=0x4000056e60 pc=0x7175880b6c
runtime.cgocallbackg(0x717591c040, 0x7fc9951138, 0x0)
	/Users/.../go/go1.23.2/src/runtime/cgocall.go:361 +0xf4 fp=0x4000056f80 sp=0x4000056f30 pc=0x71758808c4
runtime.cgocallbackg(0x717591c040, 0x7fc9951138, 0x0)
	<autogenerated>:1 +0x1c fp=0x4000056fb0 sp=0x4000056f80 pc=0x71758eecbc
runtime.cgocallback(0x0, 0x0, 0x0)
	/Users/.../go/go1.23.2/src/runtime/asm_arm64.s:1131 +0xb0 fp=0x4000056fe0 sp=0x4000056fb0 pc=0x71758ed040
runtime.goexit({})
	/Users/.../go/go1.23.2/src/runtime/asm_arm64.s:1223 +0x4 fp=0x4000056fe0 sp=0x4000056fe0 pc=0x71758ed134
fatal error: runtime: stack split at bad time

2. This code work as expected (tested on go 1.22.4 & 1.22.5 & 1.22.6 & 1.23.2)

Gomobile go code (ok)

var ReqLauncher = IToBeImplemented.Generate

// instanciated by mobile
type IToBeImplemented interface {
	Generate(test string)
}

type IGoImpl interface {
	TriggerGoAndLaunchTest()
}
type GoImpl struct {
	impl IToBeImplemented
}

func InstanceGetter(mobileImpl IToBeImplemented) IGoImpl {
	return GoImpl{impl: mobileImpl}
}

func (i GoImpl) TriggerGoAndLaunchTest() {
	ReqLauncher(i.impl, TestImplemented{}.Info())
}

type ITest interface {
	Info() string
}
type TestImplemented struct{}

func (t TestImplemented) Info() string {
	return "the go lib is on " + runtime.Version()
}

Gomobile android code (ok)

var res by remember { mutableStateOf("") }
val inst = Cmd.instanceGetter(object : IToBeImplemented {
    override fun generate(test: String) {
        res = test
    }
})
Surface(modifier = Modifier.fillMaxSize()) {
    Column(verticalArrangement = Arrangement.Center, horizontalAlignment = Alignment.CenterHorizontally) {
        Text(text = res)
        Button(onClick = {inst.triggerGoAndLaunchTest()}) { Text("call go") }
    }
}

The change between 2 tests

The only change is passing interface or string as parameters:

• in go IToBeImplemented :
  • fun generate(test: ITest) to fun generate(test: String)

Error potential cause

When calling Kotlin (Cmd.instanceGetter) -> Go (IToBeImplemented.Generate(interface)) -> Kotlin (interface.getter) -> Go (failed here when getting interface.getter value)

[gurisxie]

how can I fix it? Is there any follow-up...