First time using gdbstub, lldb stuck during setup

[evmar]

I attempted to integrate gdbstub into my emulator, following the basics in the (very extensive!) docs. I sprinkled some todo!()s and logs to see which codepaths I'll need to implement next. I'm now trying to run things for the first time and things are failing silently -- lldb is hung, and I see no logs or todos.

You can see my code (mostly a cut'n'paste from the docs) here.

I tried invoking lldb via the script in the above, and here's a trace of what it prints:

$ lldb -s lldb-script 
(lldb) command source -s 0 'lldb-script'
Executing commands in '/Users/evmar/win/rs/lldb-script'.
(lldb) platform select remote-windows
  Platform: remote-windows
 Connected: no
(lldb) platform connect connect://localhost:9001
  Platform: remote-windows
  Hostname: (null)
 Connected: yes
(lldb) target create exe/winapi/winapi.exe
Current executable set to '/Users/evmar/win/rs/exe/winapi/winapi.exe' (i386).
(lldb) r

and here's what's printed in the process running gdbstub:

INFO cli/src/debugger.rs:125 Waiting for a GDB connection on "localhost:9001"...
INFO cli/src/debugger.rs:128 Debugger connected from [::1]:53464

and both print nothing further. How can I debug what lldb is attempting? Do you have docs on how you actually should connect a remote debugger for the first time, like which gdb commands to invoke?

(I suspect I may have possibly misunderstood how run_blocking is supposed to work?)

If I break my gdbstub process in a debugger (oof, debugging a process that is itself a remote debugging target, haha) it appears stuck here:

    frame #3: 0x0000000100278edc retrowin32`std::os::unix::net::datagram::UnixDatagram::recv::h0f6a339d34c3b81a at datagram.rs:385:9 [opt]
    frame #4: 0x000000010024d7d8 retrowin32`std::io::default_read_exact::h63573348629e30b8(this=0x000000016fdfcd78, buf=(data_ptr = "", length = 1)) at mod.rs:560:15
    frame #5: 0x000000010024d990 retrowin32`std::io::Read::read_exact::h43d12378c6226a36(self=0x000000016fdfcd78, buf=(data_ptr = "", length = 1)) at mod.rs:948:9
    frame #6: 0x000000010024f270 retrowin32`gdbstub::conn::impls::tcpstream::_$LT$impl$u20$gdbstub..conn..ConnectionExt$u20$for$u20$std..net..tcp..TcpStream$GT$::read::hd812327a3efb48cd(self=0x000000016fdfcd78) at tcpstream.rs:39:15
    frame #7: 0x000000010000fe98 retrowin32`gdbstub::stub::GdbStub$LT$T$C$C$GT$::run_blocking::ha03ebcdac8d4acf5(self=<unavailable>, target=0x000000016fdfd5c8) at mod.rs:170:32
    frame #8: 0x000000010002a924 retrowin32`retrowin32::debugger::main::h68b1195b63a99e9a(machine=<unavailable>) at debugger.rs:137:18

which indicates there is some protocol communication going on, but none of my code I think.

[daniel5151]

Oh hey, I remember reading about your project at some point.
Very cool stuff!

Before I take a closer look, could you re-run your emulator with TRACE level logs for gdbstub? Check out the pull request template for more details.

(One of these days I should probably add a Bug template as well... but that's besides the fact)

Also, you should configure lldb to log any packets its sending as well. A cursory google search suggests this is as simple as including log enable gdb-remote packets at the start of your lldb script (see https://lldb.llvm.org/resources/debugging.html#logging-packets)

Once we have those logs, we can start digging into where things might be getting stuck.

[evmar]

$ export RUST_LOG=trace
$ target/debug/retrowin32 --win32-trace '*' --debugger exe/winapi/winapi.exe 
INFO win32/src/winapi/kernel32/dll.rs:173 kernel32/dll/LoadLibraryA(filename:Some("kernel32.dll"))
INFO cli/src/debugger.rs:125 Waiting for a GDB connection on "localhost:9001"...
INFO cli/src/debugger.rs:128 Debugger connected from [::1]:60810
^C

(do I need to enable a feature or something?)

(lldb) platform connect connect://localhost:9001
 <   1> send packet: +
 history[1] tid=0x0103 <   1> send packet: +
 <  19> send packet: $QStartNoAckMode#b0
 <   1> read packet: +
 <   6> read packet: $OK#9a
 <   1> send packet: +
 <  13> send packet: $qHostInfo#9b
 <   4> read packet: $#00
 <  18> send packet: $qGetWorkingDir#91
 <   4> read packet: $#00
 <  19> send packet: $qQueryGDBServer#cb
 <   4> read packet: $#00
  Platform: remote-windows
  Hostname: (null)
 Connected: yes
(lldb) target create exe/winapi/winapi.exe
 < 139> send packet: $qModuleInfo:2f55736572732f65766d61722f77696e2f72732f6578652f77696e6170692f77696e6170692e657865;693338362d70632d77696e646f77732d6d737663#95
 <   4> read packet: $#00
Current executable set to '/Users/evmar/win/rs/exe/winapi/winapi.exe' (i386).
(lldb) r
 <  40> send packet: $qLaunchGDBServer;host:giraffe.local;#8b
 <   4> read packet: $#00

and hangs there. After maybe 5 seconds, it prints

error: invalid host:port specification: '[localhost]'
(lldb) 

[evmar]

In case it's helpful, in the hung state it's attempting to read here

frame #7: 0x000000010001107c retrowin32`gdbstub::stub::GdbStub$LT$T$C$C$GT$::run_blocking::ha03ebcdac8d4acf5(self=<unavailable>, target=0x000000016fdfd5e8) at mod.rs:170:32
   167              gdb = match gdb {
   168                  state_machine::GdbStubStateMachine::Idle(mut gdb) => {
   169                      // needs more data, so perform a blocking read on the connection
-> 170                      let byte = gdb.borrow_conn().read().map_err(InternalError::conn_read)?;
   171                      gdb.incoming_data(target, byte)?
   172                  }
   173 

[daniel5151]

Based on some of the lldb logs, it certainly looks like you're getting some response from gdbstub, which is a good sign.
That said - I'm seeing a lot of lldb specific stuff in these logs that I'm not super familiar with...

Could you attempt to connect using regular gdb?
While lldb should work (based on feedback I've gotten from other users of gdbstub), we should start by making sure stock gdb is working as expected.

The command to enable packet debugging in GDB is set debug remote 1

---

(do I need to enable a feature or something?)

What logging frontend are you using in your project?

RUST_LOG applies to projects using env_logger (which is what I use in the in-tree examples), but if you're using something else, then the specific way to tweak log levels will vary.

[evmar]

I'm using an in-tree simple logger, which doesn't have any configuration, all my logs are enabled. But it's blocked within gdbstub so I don't expect any of those logs. Is there some mechanism where gdbstub logs the packets it gets? I think maybe I am misunderstanding the intent of the RUST_LOG=trace recommendation. :)

[evmar]

I thought to try gdb but:

$ brew install gdb
gdb: The x86_64 architecture is required for this software.

But I just discovered there are arm builds of gdb (targeting elf, but that is irrelevant for remote debugging purposes), and that successfully triggers my code! So it does appear to be something about lldb in particular.

(gdb) target remote localhost:9001
Remote debugging using localhost:9001
warning: while parsing target description (at line 1): Target description specified unknown architecture "i386:intel"
warning: Could not load XML target description; ignoring
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
Remote connection closed

INFO cli/src/debugger.rs:125 Waiting for a GDB connection on "localhost:9001"...
INFO cli/src/debugger.rs:128 Debugger connected from [::1]:61096
thread 'main' panicked at cli/src/debugger.rs:43:9:
not yet implemented

(and the TODO is expected)

    fn read_registers(
        &mut self,
        regs: &mut <Self::Arch as gdbstub::arch::Arch>::Registers,
    ) -> gdbstub::target::TargetResult<(), Self> {
        todo!()
    }

[daniel5151]

Is there some mechanism where gdbstub logs the packets it gets

Yes, this is precisely why I am suggesting you enable trace-level logging in your application :)

gdbstub has many tracing statements that will dump incoming / outgoing protocol bytes. e.g:

gdbstub/src/protocol/recv_packet.rs

Line 59 in 927a961

trace!("<-- {}", String::from_utf8_lossy(buf.as_slice()));

As you can imagine, getting a clear picture of what each party is sending / receiving really helps when it comes to debugging.

---

Good to see that stock gdb seems to have connected successfully!

As I touch on in #99, gdbstub hasn't really prioritized deep lldb compatibility, which means I don't have much insight / advice to give on why lldb might not be connecting. It could be related to that target create thing you're doing - not super sure what that's all about? Just a guess though.

If you get to the bottom of it, feel free to let me know (in this thread, or via a new issue), and we can consider landing any missing functionality to unblock your particular usage of lldb.

In the meantime though, you might just want to stick to using stock gdb to get the basic stub up-and-running.

[evmar]

Ah, I think what I was missing is that gdbstub using the log crate requires my executable to provide some hooks for it to log anything. I will investigate that and get back to you.

[evmar]

Startup, using the same lldb script as above:

INFO cli/src/debugger.rs:125 Waiting for a GDB connection on "localhost:9001"...
INFO cli/src/debugger.rs:128 Debugger connected from [::1]:62746
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- +
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- $QStartNoAckMode#b0
TRACE gdbstub-0.7.1/src/protocol/response_writer.rs:74 --> $OK#9a
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- +
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- $qHostInfo#9b
INFO gdbstub-0.7.1/src/stub/core_impl.rs:247 Unknown command: Ok("qHostInfo")
TRACE gdbstub-0.7.1/src/protocol/response_writer.rs:74 --> $#00
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- $qGetWorkingDir#91
INFO gdbstub-0.7.1/src/stub/core_impl.rs:247 Unknown command: Ok("qGetWorkingDir")
TRACE gdbstub-0.7.1/src/protocol/response_writer.rs:74 --> $#00
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- $qQueryGDBServer#cb
INFO gdbstub-0.7.1/src/stub/core_impl.rs:247 Unknown command: Ok("qQueryGDBServer")
TRACE gdbstub-0.7.1/src/protocol/response_writer.rs:74 --> $#00
TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- $qModuleInfo:2f55736572732f65766d61722f77696e2f72732f6578652f77696e6170692f77696e6170692e657865;693338362d70632d77696e646f77732d6d737663#95
INFO gdbstub-0.7.1/src/stub/core_impl.rs:247 Unknown command: Ok("qModuleInfo:2f55736572732f65766d61722f77696e2f72732f6578652f77696e6170692f77696e6170692e657865;693338362d70632d77696e646f77732d6d737663")
TRACE gdbstub-0.7.1/src/protocol/response_writer.rs:74 --> $#00

and then after hit r in lldb:

TRACE gdbstub-0.7.1/src/protocol/recv_packet.rs:59 <-- $qLaunchGDBServer;host:giraffe.local;#8b
INFO gdbstub-0.7.1/src/stub/core_impl.rs:247 Unknown command: Ok("qLaunchGDBServer;host:giraffe.local;")
TRACE gdbstub-0.7.1/src/protocol/response_writer.rs:74 --> $#00

( pair with #146 (comment) )

[evmar]

Dug a bit in the llvm github repo (the search is broken, yuck) and found I think the relevant code

[evmar]

I looked into this a little more. For reasons not clear to me it appears lldb expects there to be an additional process in the mix. See discussion of 'debugserver' on https://lldb.llvm.org/use/remote.html .

In practice I think what this means is the initial connection is used to negotiate some metadata (see e.g. qHostInfo above) and then spawn an additional gdb server(!), and lldb in my above trace is failing because it is expecting connection info such as a port in the response which prompts the invalid host:port specification: '[localhost]' error. (See this commit for some related codepaths.)

In all, I conclude supporting lldb might be a decent amount of work. Given we figured out the gdb things here, feel free to close this bug.

[daniel5151]

Given that I've definitely confirmed that lldb works with gdbstub in some capacity (gdbstub already implements various lldb-specific packets that folks have upstreamed, after all), I strongly suspect you're somehow configuring lldb in such a way that it assumes a far more "featureful" GDB server to be on the other end of the connection, instead of a simple GDB "stub".

In any case - I'll go ahead and close out this particular issue for now, since there's nothing really actionable here.

If you run into any other issues, feel free to open a new issue / start a discussion thread.

Cheers!

[evmar]

I also was misled by the LLDB code in this repo, given that this is lldb straight out of the box without configuration.

Their docs mention:

In order to reduce code complexity and improve remote debugging experience LLDB on Linux and macOS uses the remote debugging stub even when debugging a process locally. This is achieved by spawning a remote stub process locally and communicating with it over the loopback interface. In the case of local debugging this whole process is transparent to the user. The platform binary is not used in this case, since no file transfers are needed.

The "platform binary" is the third process involved here, so perhaps it's something about how it determines whether you're using "local debugging" or not.

[evmar]

Sorry to keep updating this, but I figure it will be helpful for you in case someone else shows up with my questions:

I was using the lldb command platform connect ... which (apparently) is the one that makes it expect to connect to a process that multiplexes subsequent gdb connections.

If I instead use the lldb command gdb-remote ... then it appears to open with a string of commands that look more like a plain gdb connection.