Not "pickle", even "json" over "sockets"

[nastiliano]

Hi, everyone.

With Micropython 4.19.1 in my ESP32 I'm not able to implement "pickle", even "json".

I just want to send a string list over Wifi. PC (Tx) to ESP32(Rx).

In my ESP32 importing "pickle" the console output simply is: the module not exist
And using "json" the follows doesn't work:

PC code:
...
list = ["hello" , "world"]
...
socket.sendall(json.dumps(list))

ESP32 code:
...
client_socket , addre_port = listening_socket.accept()
data1 = client_socket.recv(1024)
data2 = json.loads(data1)
print(data2)

ESP32 console output:
...
ValueError: syntax error in JSON

Could someone help me?
Thanks in advance.

[peterhinch]

I suggest you also print data1, then we can see why there is a syntax error.

[nastiliano]

I suggest you also print data1, then we can see why there is a syntax error.

Hello, thanks for advance

ESP32 console output for print(data1):

b''

ESP32 console output for the rest of the code:

Unhandled exception in thread started by <function Wifi at 0x3ffe5660>
Traceback (most recent call last):
File "", line 199, in Wifi
ValueError: syntax error in JSON

And is not a problem sending with sockets because using "sendall" with a simple string like "Hello" is sended fine.

[karfas]

I assume your client_socket.recv(1024) doesn't receive anything, so you are doing the equivalent of

>>> a=b''
>>> json.loads(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: syntax error in JSON

Are you sure you are connecting and sending anything ? json.dump([1,2]) (as in your server code) gives me a parameter error.
This should be json.dumps(..), I think.

[nastiliano]

I assume your client_socket.recv(1024) doesn't receive anything, so you are doing the equivalent of

>>> a=b''
>>> json.loads(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: syntax error in JSON

Are you sure you are connecting and sending anything ? json.dump([1,2]) (as in your server code) gives me a parameter error. This should be json.dumps(..), I think.

Thanks for advance.

It wasn't the error because was an errata when writing the post. The code is good (dumps())

[karfas]

Then I would recommend some error checking. Not everything running without exception is error-free.

In your case, you use socket.recv() immediate after you accept the connection.
A connection in TCP (=SYN request) doesn't mean there is data ready for reading.
Maybe a sleep(0.1) or the like after accept() helps ?

[peterhinch]

@nastiliano socket.recv(n) returns immediately with as much data is available at the time it is called. The result of the print(data1) test confirms the observations of @karfas - you are not allowing time for the server to respond. If the data you are expecting is newline-terminated, consider using socket.readline() which will pause until a complete line is read.

[nastiliano]

Thank you very much guys, but I'm still a little bit desperate :)

I tried all adding Sleep:
data1 = client_socket.recv(1024)
time.sleep(5)
print(data1)
data2 = json.loads(data1)
print(data2)

And I tried:
data1 = client_socket.readline()
time.sleep(5)
print(data1)
data2 = json.loads(data1)
print(data2)

Data1 always is empty. I think my problem is on client code. I think nothind is sended with this syntax:
...
list = ["hello" , "world"]
...
socket.sendall(json.dumps(list))

[robert-hh]

I would have expected the time.sleep() statement before the recv() statement, not after it. But not have sent anything could be a reason as well.

[nastiliano]

Hello,

The same adding:

time.sleep(5)
data1 = client_socket.recv(1024)
time.sleep(5)
print(data1)
data2 = json.loads(data1)
print(data2)

I think the client is not sending anything. May be an error only occured in my ESP32 model? Because is very strange that I'm not able to use "import pickle" and "import json" is so problematic...

[bixb922]

I think there is no problem with json but with the socket communication. Are you using non blocking send /receive? If you just send a string, does it arrive well?

[karfas]

How exactly do you create your listening socket ?

To debug this, you can print() the result of every system call in sight and compare the results (very thoroughly) to the values you expect.

A second method might be to try to connect to your ESP32 server using some other program (telnet, putty, netcat, nmap, even your browser) and/or use a network sniffer.
Unfortunately, this requires at least an idea how common network protocols like TCP or UDP are working.

[nastiliano]

Thank you for your responses.
Here is my code:

CLIENT:
def sending(self , *args):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:
client_socket.settimeout(3)
try:
# IP and PORT
client_socket.connect((args[1] , 19204))
# type of connection, user, password (all as string)
client_socket.sendall(str([args[0] , args[2] , args[3]]).encode("UTF-8"))
# If we receive an strig like "ack" (all ok)
if client_socket.recv(1024).decode() == "True":

                # Here is (I think) the problem. Anything is sended                    
                client_socket.sendall({"first":"You're", "second":"Awsome!"})
                # If we receive an strig like "ack" (the same than before)
                if socol_client.recv(1024).decode() == "True":
                    return True
                else:
                    return False
            else:
                return False
        except:
            client_socket.close()
            return False

SERVER:
...
listening_socket = socket.socket(socket.AF_INET , socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET , socket.SO_REUSEADDR , 1)
client_socket.bind((IP.decode() , 19204))
client_socket , addr_port = listening_socket.accept()

    deserialised_connection_data = client_socket.recv(1024).decode().strip("[]").replace("'", "").split(", ")
    ...
        if deserialised_connection_data[1] == user.decode():
            if deserialised_connection_dataconnexio_deserialitzada[2] == contrasenya.decode():
                client_socket.send(bytes("True" , "utf-8"))
                time.sleep(5)
                jsonReceived = client_socket.recv(1024)
                time.sleep(5)
                print(jsonReceived)
                data = json.loads(jsonReceived)
                print(data)
                client_socket.send(bytes("True" , "utf-8"))
                machine.reset()'''
    listening_socket.close()
    client_socket.close()

[karfas]

Your server code:

listening_socket = socket.socket(socket.AF_INET , socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET , socket.SO_REUSEADDR , 1)
listening_socket.listen(1)

client_socket , addr_port = listening_socket.accept()
deserialised_connection_data = client_socket.recv(...)....

obviously nowhere tells your listening socket on which port the server should run.

Perhaps the socket.bind() call is not there ?

May I suggest you read (and try to understand) at least a little about TCP/IP ?
The http server example in the micropython-docs looks like a good starting point to steal basic server code from.

[nastiliano]

Sorry. Yes it was in my code. But I'm translating and summarizing...
I edited again my code.

Sorry.

[karfas]

And what does IP.decode() return ? Why aren't you using 0.0.0.0 to listen on any interface (at least for testing) ?

[nastiliano]

And what does IP.decode() return ? Why aren't you using 0.0.0.0 to listen on any interface (at least for testing) ?

Thanks for your response.

But is not a problem connecting sockets. I've been sending strings and all is fine. The problem is something in my JSON syntax. I don't know exactly how to send a right JSON dictionary and recover it, for this reason my output is "JSON syntax error", because in the server I just receive an empy data.

[nastiliano]

SOLVED!

Thanks to everyone for your responses.
The problem was encoding/decoding the JSON String

Final code:

CLIENT:

def sending(self , *args):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client_socket:
client_socket.settimeout(3)

try:
    client_socket.connect((args[1] , 19204)) # IP and PORT
    client_socket.sendall(str([args[0] , args[2] , args[3]]).encode("UTF-8")) # type of connection, user, password (all as string)

    if client_socket.recv(1024).decode() == "True":  # If we receive a string like "ack" (all ok)
            client_socket.sendall(json.dumps(["Hello" , "World"]).encode("UTF-8"))  #HERE was the problem, we need encode this JSON String 

            if client_socket.recv(1024).decode() == "True": # If we receive an strig like "ack" (the same than before)
                return True
            else:
                return False
        else:
            return False
    except:
        client_socket.close()
        return False

SERVER:
...
listening_socket = socket.socket(socket.AF_INET , socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET , socket.SO_REUSEADDR , 1)
client_socket.bind((IP.decode() , 19204))
client_socket , addr_port = listening_socket.accept()

deserialised_connection_data = client_socket.recv(1024).decode().strip("[]").replace("'", "").split(", ")
...
    if deserialised_connection_data[1] == user.decode():
        if deserialised_connection_dataconnexio_deserialitzada[2] == contrasenya.decode():
            client_socket.send(bytes("True" , "utf-8"))
            jsonReceived = json.loads(client_socket.recv(1024).decode()) #HERE was the problem
            print(jsonReceived)
            client_socket.send(bytes("True" , "utf-8"))
listening_socket.close()
client_socket.close()

[karfas]

But is not a problem connecting sockets. I've been sending strings and all is fine. The problem is something in my JSON syntax. I don't know exactly how to send a right JSON dictionary and recover it.

Back to square one, then. Please review your posts here. Nowhere I can find any indication that your server is indeed working.
Everybody here assumed you are not receiving anything.

Next time, please prepare a minimal, but working case (so someone can copy/paste most of it in his own device) and respond to wrong assumptions immediate.