ESP8266: WebREPL truncates pasted data #2497
Comments
cpopp
changed the title
|
I noticed the same problem with the pyboard when I wrote usb-ser-mon.py so I added a 2 millisecond delay after reading each character from stdin and sending it to the usb-serial port, and that seemed to solve the problem. Something similar might need to be done on the webrepl client side. |
|
I confirm that this is a problem with WebREPL, and that it doesn't happen at the serial (uart) prompt. With the help from some debug printf's, it is indeed an overflow in the ringbuf, when filling it in dupterm_task_handler. One solution is to put flow control in the client so that it doesn't send more than 256 bytes at once. The other solution is to put flow control in the host so that it doesn't process incoming chars if there's no room in the ringbuf. |
|
I like the idea of flow control on the host so that each client that makes use of dupterm doesn't have to worry about implementing it. |
|
Just a note that I recently stumbled about this with the 1.10 firmware (while trying to add a WebREPL transport to tools/pyboard.py), however I found that messages were truncated to 255 bytes, not 256. Only sending that much at a time (changing 256 to 255 in |
|
@cwalther, do I understand correctly, that you were able to submit longer inputs by chunking it? I tried the same but did not succeed. Can I see your code? |
|
having the same problem on ESP32 when sending 263/264 bytes of data.
|
|
Yes, I think I was able to send longer inputs by reducing the chunk size from 256 to 255. My code was a mangled version of pyboard.py and I don’t remember whether I left it in a working state, but here’s the diff in case it helps (note the last hunk): diff --git a/tools/pyboard.py b/tools/pyboard.py
index d90623506..1d3ab13aa 100755
--- a/tools/pyboard.py
+++ b/tools/pyboard.py
@@ -85,6 +85,66 @@ def stdout_write_bytes(b):
class PyboardError(Exception):
pass
+class WebsocketToSerial:
+ def __init__(self, url, password):
+ # pip install websocket-client
+ import websocket
+ # ESP8266 WebREPL puts binary data into WebSocket text messages even if it's not UTF-8, which causes a conformant recipient to close the connection - fortunately this client has an option to be non-conformant
+ self.ws = websocket.WebSocket(skip_utf8_validation=True)
+ self.ws.connect(url, timeout=5)
+ self.fd = self.ws.fileno()
+ prompt = self.ws.recv()
+ if prompt != 'Password: ':
+ raise PyboardError('Expected password prompt, got {}'.format(prompt))
+ self.ws.send(password + '\r\n')
+ try:
+ prompt = self.ws.recv()
+ except ConnectionResetError:
+ raise PyboardError('Incorrect password')
+ self.timeoutException = websocket.WebSocketTimeoutException
+ from collections import deque
+ self.fifo = deque()
+
+ def close(self):
+ self.ws.close()
+
+ def read(self, size=1):
+# print('<** enter read', size, '**>')
+ while len(self.fifo) < size:
+ try:
+ self.ws.settimeout(10)
+ r = self.ws.recv()
+# print('<** recv', len(r), repr(r), '**>')
+ self.fifo.extend(bytes(r, 'utf-8'))
+ except (self.timeoutException, BlockingIOError):
+# print('<** recv timeout **>')
+ break
+ data = b''
+ while len(data) < size and len(self.fifo) > 0:
+ data += bytes([self.fifo.popleft()])
+# print('<** done read', repr(data), '**>')
+ return data
+
+ def write(self, data):
+ self.ws.settimeout(10)
+ self.ws.send(data)
+ return len(data)
+
+ def inWaiting(self):
+# print('<** enter inWaiting **>')
+ n_waiting = len(self.fifo)
+ if not n_waiting:
+ try:
+ self.ws.settimeout(0)
+ self.fifo.extend(bytes(self.ws.recv(), 'utf-8'))
+ except (self.timeoutException, BlockingIOError):
+ pass
+ n_waiting = len(self.fifo)
+# print('<** done inWaiting', n_waiting, '**>')
+ return n_waiting
+
+ closed_by_reset = True
+
class TelnetToSerial:
def __init__(self, ip, user, password, read_timeout=None):
self.tn = None
@@ -234,6 +294,8 @@ class Pyboard:
elif device and device[0].isdigit() and device[-1].isdigit() and device.count('.') == 3:
# device looks like an IP address
self.serial = TelnetToSerial(device, user, password, read_timeout=10)
+ elif device.startswith('ws:'):
+ self.serial = WebsocketToSerial(device, password)
else:
import serial
delayed = False
@@ -261,6 +323,7 @@ class Pyboard:
self.serial.close()
def read_until(self, min_num_bytes, ending, timeout=10, data_consumer=None):
+# print('<## enter read_until', repr(ending), '##>')
# if data_consumer is used then data is not accumulated and the ending must be 1 byte long
assert data_consumer is None or len(ending) == 1
@@ -269,6 +332,7 @@ class Pyboard:
data_consumer(data)
timeout_count = 0
while True:
+# print('<## data', repr(data), '##>')
if data.endswith(ending):
break
elif self.serial.inWaiting() > 0:
@@ -284,6 +348,7 @@ class Pyboard:
if timeout is not None and timeout_count >= 100 * timeout:
break
time.sleep(0.01)
+# print('<## done read_until', repr(data), '##>')
return data
def enter_raw_repl(self):
@@ -296,16 +361,17 @@ class Pyboard:
n = self.serial.inWaiting()
self.serial.write(b'\r\x01') # ctrl-A: enter raw REPL
- data = self.read_until(1, b'raw REPL; CTRL-B to exit\r\n>')
- if not data.endswith(b'raw REPL; CTRL-B to exit\r\n>'):
- print(data)
- raise PyboardError('could not enter raw repl')
-
- self.serial.write(b'\x04') # ctrl-D: soft reset
- data = self.read_until(1, b'soft reboot\r\n')
- if not data.endswith(b'soft reboot\r\n'):
- print(data)
- raise PyboardError('could not enter raw repl')
+ if not getattr(self.serial, 'closed_by_reset', False):
+ data = self.read_until(1, b'raw REPL; CTRL-B to exit\r\n>')
+ if not data.endswith(b'raw REPL; CTRL-B to exit\r\n>'):
+ print(data)
+ raise PyboardError('could not enter raw repl')
+
+ self.serial.write(b'\x04') # ctrl-D: soft reset
+ data = self.read_until(1, b'soft reboot\r\n')
+ if not data.endswith(b'soft reboot\r\n'):
+ print(data)
+ raise PyboardError('could not enter raw repl')
# By splitting this into 2 reads, it allows boot.py to print stuff,
# which will show up after the soft reboot and before the raw REPL.
data = self.read_until(1, b'raw REPL; CTRL-B to exit\r\n')
@@ -344,8 +410,9 @@ class Pyboard:
raise PyboardError('could not enter raw repl')
# write command
- for i in range(0, len(command_bytes), 256):
- self.serial.write(command_bytes[i:min(i + 256, len(command_bytes))])
+ # ESP8266 WebREPL only supports up to 255 bytes (#2497)
+ for i in range(0, len(command_bytes), 255):
+ self.serial.write(command_bytes[i:min(i + 255, len(command_bytes))])
time.sleep(0.01)
self.serial.write(b'\x04')
|
|
@cwalther, thanks! I was able to use your trick with ESP8266. Unfortunately ESP32 requires 0.3-0.5 second delay between blocks of 255 in order to work reliably (compared to 0.01 seconds, which is sufficient with ESP8266). Tried both MP 1.11 and 1.12. Looking forward to flow control on the host! |
|
@cwalther , thanks this approach works also for me :-) @aivarannamaa , thanks for the timeout hints :-) finally after several months i picked it up again ;-) |
|
In case the host-based flow control is hard, what about adding an optional CRC checker for the raw REPL? A tool using the raw REPL could opt in by starting the command with a specific marker byte followed by CRC bytes and actual input. If the REPL fails to validate the input, it would raise a specific exception. This would allow tools to catch more miscommunications. Also, they could start with bigger chunks of input (or shorter delays between chunks) but adapt dynamically when the error rate is too big. |
|
This problem has been plaguing me for a couple of years, so I finally got around to trying find a solution. I have two versions of the webrepl client. The first had these credits: Copyright (c) 2012-2013, Christopher Jeffrey (MIT License), and Copyright (c) 2016, Paul Sokolovsky. The second one I found at https://github.com/micropython/webrepl/tree/master. I modified webrepl.js in this second version to add a 100 ms delay between each chunk of 250 pasted characters, and pasting large blocks of code works with my ESP01s. I'm not sure how to post things to git, so I just created an issue with the changes attached. |
I'm running 1.8.4 and I notice that data pasted into the WebREPL is truncated.
Start up the WebREPL and trying pasting the following
"*******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************"
The result is that only around 256 characters of what is pasted actually shows up. This is most troublesome when trying to use paste mode (CTRL-E) to bring in some existing code.
This might be related to the dupterm ringbuffer getting filled: https://github.com/micropython/micropython/blob/master/esp8266/esp_mphal.c#L39
The text was updated successfully, but these errors were encountered: