Cannot plot a plotly chart from pandas dataframe with DateTimeIndex

[Nikodemski2002]

Hi. As in the title, I have a problem with plotting the plotly chart from pandas DataFrame with the DateTimeIndex.
When I use an incremental index (from 0 to 5000) the plot shows normally, but when I replace it with the DateTimeIndex I get the following error in the terminal:

Task exception was never retrieved
future: <Task finished name='Task-49' coro=<AsyncServer._emit_internal() done, defined at C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py:427> exception=TypeError('unsupported datatype in numpy array')>
Traceback (most recent call last):
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py", line 437, in _emit_internal
    await self._send_packet(sid, self.packet_class(
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py", line 442, in _send_packet
    encoded_packet = pkt.encode()
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\packet.py", line 64, in encode
    encoded_packet += self.json.dumps(data, separators=(',', ':'))
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\nicegui\json\orjson_wrapper.py", line 28, in dumps
    return orjson.dumps(obj, option=opts).decode('utf-8')
TypeError: unsupported datatype in numpy array
Task exception was never retrieved
future: <Task finished name='Task-51' coro=<AsyncServer._emit_internal() done, defined at C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py:427> exception=TypeError('unsupported datatype in numpy array')>
Traceback (most recent call last):
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py", line 437, in _emit_internal
    await self._send_packet(sid, self.packet_class(
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py", line 442, in _send_packet
    encoded_packet = pkt.encode()
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\packet.py", line 64, in encode
    encoded_packet += self.json.dumps(data, separators=(',', ':'))
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\nicegui\json\orjson_wrapper.py", line 28, in dumps
    return orjson.dumps(obj, option=opts).decode('utf-8')
TypeError: unsupported datatype in numpy array
Task exception was never retrieved
future: <Task finished name='Task-53' coro=<AsyncServer._emit_internal() done, defined at C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py:427> exception=TypeError('unsupported datatype in numpy array')>
Traceback (most recent call last):
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py", line 437, in _emit_internal
    await self._send_packet(sid, self.packet_class(
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\asyncio_server.py", line 442, in _send_packet
    encoded_packet = pkt.encode()
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\socketio\packet.py", line 64, in encode
    encoded_packet += self.json.dumps(data, separators=(',', ':'))
  File "C:\Users\nikon\PycharmProjects\NICEGUI_APP\venv\lib\site-packages\nicegui\json\orjson_wrapper.py", line 28, in dumps
    return orjson.dumps(obj, option=opts).decode('utf-8')
TypeError: unsupported datatype in numpy array

This error doesn't show when I use matplotlib pyplot instead of plotly, chart shows correctly with the DateTimeIndex on the x-axis.

Packages versions:

• plotly: 5.13.1
• pandas: 1.5.3
• numpy: 1.24.2
• matplotlib: 3.7.1
• python-socketio: 5.7.2
• fastapi-socketio: 0.0.10

Kind regards,
Nikodem

[falkoschindler]

Can you provide some minimum example code to reproduce the problem? That would help a lot. Thanks!

[Nikodemski2002]

Sure. So the pandas dataframe is created from csv file separated with semicolons.
Below example is a working code that uses padnas dataframe with original incremental index and plotly:

df = pd.read_csv('T1.csv', sep=';')

chart = go.Figure()
chart.add_trace(go.Scatter(x=df.index, y=df['ActivePower'], mode='lines', name='Active Power [kW]'))
chart.update_layout(margin=dict(l=10, r=10, t=10, b=10), plot_bgcolor="white")
chart.update_yaxes(showline=True, gridcolor='#DEDEDE')
ui.plotly(chart).style('width: 100%')

Here example of working code that uses pandas dataframe with DateTimeIndex and matplotlib plot:

df = pd.read_csv('T1.csv', sep=';')
df = df.set_index(pd.to_datetime(df['Date/Time'], format='%d.%m.%Y %H:%M'))

with ui.pyplot(figsize=(15,3)).style('margin-left: auto; margin-right: auto'):
	x = df.index
	y = df['ActivePower']
	plt.plot(x, y, '-')

And finally an example of code that uses pandas dataframe with DateTimeIndex and plotly, which gives an error mentioned in the first post:

df = pd.read_csv('T1.csv', sep=';')
df = df.set_index(pd.to_datetime(df['Date/Time'], format='%d.%m.%Y %H:%M'))

chart = go.Figure()
chart.add_trace(go.Scatter(x=df.index, y=df['ActivePower'],mode='lines',name='Active power [kW]'))
chart.update_layout(margin=dict(l=10, r=10, t=10, b=10), plot_bgcolor="white")
chart.update_yaxes(showline=True, gridcolor='#DEDEDE')
ui.plotly(chart).style('width: 100%')

I also attach the T1.csv file so you can reproduce the code:
T1.csv

Have a good day!
Nikodem

[falkoschindler]

Ok, I boiled it down to this:

from io import StringIO

import matplotlib.pyplot as plt
import pandas as pd
from plotly import graph_objects as go

from nicegui import ui

csv = '''Date;Value
01.01.2018 00:00;100.0
01.01.2018 00:10;120.0
01.01.2018 00:20;110.0
'''

df = pd.read_csv(StringIO(csv), sep=';')
df = df.set_index(pd.to_datetime(df['Date'], format=r'%d.%m.%Y %H:%M'))  # this line breaks the plotly chart

chart = go.Figure(go.Scatter(x=df.index, y=df['Value']))
ui.plotly(chart)

with ui.pyplot():
    plt.plot(df.index, df['Value'], '-')  # the pyplot still works

ui.run()

Apparently df.index is of type DatetimeIndex, which is not JSON-serializable, even with orjson.
But I found the following workaround:

go.Scatter(x=df.index.to_pydatetime().tolist(), ...)

With to_pydatetime() we convert the DatetimeIndex to a NumPy array of datetime.datetime. This is still not serializable (I don't understand why). But after converting it .tolist() the serialization works.

@rbeeli Do you have experience with orjson and DatetimeIndex? Is there a better way to do it? Do you understand why orjson can't serialize a NumPy array of datetime.datetime?

[rbeeli]

DatetimeIndex is a pandas construct, not numpy.

But you can simply do this to get the underlying numpy array, and it should fix it:
go.Scatter(x=df.index.values, y=df['Value'])

[falkoschindler]

Nice! Even simpler than my hacky "solution". Thanks!

[Nikodemski2002]

@falkoschindler, @rbeeli thank you guys so much. I've repleaced .values with pandas recomended .to_numpy now it works great.
Have a great day,
Nikodem