Panda DFU recovery

[dzid26]

Panda (C3X) doesn't always enter DFU when running recover.py.

I tried rebooting and entering dfu manually. After that it would either work many times and or repeatedly not find dfu.

Steps to reproduce:

1. Turn on device and connect via ssh.
2. Run /data/openpilot/panda/board/recover.py
3. Gets stuck forever. Pressing Ctrl+C gives:

comma@comma-609a2742:/data/openpilot/panda/board$ ./recover.py
scons: Entering directory `/data/openpilot/panda'
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `board' is up to date.
scons: done building targets.
^CTraceback (most recent call last):
  File "/data/openpilot/panda/board/./recover.py", line 13, in <module>
    for s in Panda.list():
             ^^^^^^^^^^^^
  File "/data/pythonpath/panda/python/__init__.py", line 399, in list
    ret += cls.spi_list()
           ^^^^^^^^^^^^^^
  File "/data/pythonpath/panda/python/__init__.py", line 423, in spi_list
    _, _, serial, _, _ = cls.spi_connect(None, ignore_version=True)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/pythonpath/panda/python/__init__.py", line 323, in spi_connect
    dat = handle.get_protocol_version()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/pythonpath/panda/python/spi.py", line 270, in get_protocol_version
    with self.dev.acquire() as spi:
  File "/usr/local/pyenv/versions/3.11.4/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/data/pythonpath/panda/python/spi.py", line 111, in acquire
    fcntl.flock(self._spidev, fcntl.LOCK_EX)
KeyboardInterrupt
^C

[adeebshihadeh]

Are you connected to a PC? Might be a race condition with USB enumeration in the ST bootloader.

https://www.st.com/resource/en/application_note/an2606-stm32-microcontroller-system-memory-boot-mode-stmicroelectronics.pdf

[dzid26]

This is on C3X. I have it connected to 12V for this test.

Seems to me this is an SPI communication issue:

• I found that ./recover.py only gets stuck when pandad.py is active.
• pandad.py errors out when that happens.

selfdrive/pandad/spi.cc:   1 / 0x0 / 7 / 12 / tx: 14141414141414
selfdrive/pandad/spi.cc: SPI: timed out waiting for ACK, waiting for 0x1f
selfdrive/pandad/spi.cc:   1 / 0x0 / 7 / 12 / tx: 14141414141414
selfdrive/pandad/spi.cc: SPI: timed out waiting for ACK, waiting for 0x1f
selfdrive/pandad/spi.cc:   1 / 0x0 / 7 / 12 / tx: 14141414141414
selfdrive/pandad/spi.cc: SPI: timed out waiting for ACK, waiting for 0x1f
selfdrive/pandad/spi.cc:   1 / 0x0 / 7 / 12 / tx: 14141414141414
selfdrive/pandad/spi.cc: SPI: timed out waiting for ACK, waiting for 0x1f
selfdrive/pandad/spi.cc:   1 / 0x0 / 7 / 12 / tx: 14141414141414
and so on...........

• After killing panda.py, the ./recover.py will show 0 dfu devices, even after trying again
• Only after relaunching pandad.py and then killing it, the ./recover.py will show list dfu device again.
• Resetting panda via nrst pin also fixes "recover.py showing 0 dfu devices"

Looks like some clash when accessing SPI.

[adeebshihadeh]

Oh yeah, this is expected since the ST bootloader state machine is super sensitive. recover.py doesn't take an exclusive lock over the SPI device, so pandad's comms still go through.