What makes loading the NumPy array 5x slower than the Python list? #4732
Answered
by
alex
christianrickert
asked this question in
Questions
Replies: 2 comments 2 replies
-
I don't see anything about |
Beta Was this translation helpful? Give feedback.
1 reply
-
The reason is that to go from a numpy array to a Vec of f64, a new
pyinteger object is allocated and then unboxed for each value. With a list
the pyobjects already exist
…On Tue, Nov 26, 2024, 5:52 PM Christian Rickert ***@***.***> wrote:
thank you @davidhewitt <https://github.com/davidhewitt>
I don't see anything about ndarray in your snippet, so I'm a bit confused.
# load_data.py
np.random.rand(10, 1024 * 1024 * 100) # type() returns: <class 'numpy.ndarray'>np.random.rand(10, 1024 * 1024 * 100).tolist() # type() returns: <class 'list'>
I didn't want to copy/paste for a single function call tolist(), but I
understand that it is too easy to miss. - Apologies!
In general anything with 2d arrays you will want to pass them around as
numpy / ndarray arrays, not vecs-of-vecs.
I would really like to continue using Vec<Vec<f64>> on the Rust side if
at all possible.
However, there must be a reason (implementation detail like repetitive
type checks or iterative memory allocations) why the conversion to
Vec<Vec<f64>> performs significantly worse for an ndarray than for a list
that I am currently missing. Quite frankly, it doesn't make sense to me.
—
Reply to this email directly, view it on GitHub
<#4732 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBHLU6NHVFUCHLYGV3T2CT3RNAVCNFSM6AAAAABSRD7C2SVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMZYHA4DMMI>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
1 reply
Answer selected by
christianrickert
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I've run into an unexpected performance bottleneck with
pyo3
andnumpy
: In short, loading a large 2-dimensionalndarray
into a Rust function takes about five times longer than a nested Python list.It takes about 25 seconds if I pass the
ndarray
to thereturn_vector
Rust function:In contrast, it only takes about 6 seconds to load the same data as a Python list:
However, converting the
ndarray
to a Python list comes at the cost of (mostly) making a copy in memory, even without returning the processed data. - I did have a look at the simple example forrust-numpy
, but it adds a significant level of verbosity to the Rust code.I'm glad that
pyo3
works out of the box with bothndarray
and Python lists - even without any changes to the Rust code! But is there something I missed that could explain the difference in performance?Beta Was this translation helpful? Give feedback.
All reactions