Running XDF.jl

[lgarbar]

The read_xdf function seems to work without problem for some .xdf files and not others. Listed below is the full error message from my Pluto notebook. Importantly, for the files that fail and the ones that succeed don't seem to have any reliable difference in size (given that they're from the same task performed by different participants) and compile the same number of streams and headers. Any insight would be appreciated.

Error message:
MethodError: no method matching read(::IOStream, ::Nothing)
Closest candidates are:
read(::IOStream)
@ Base iostream.jl:519
read(::IOStream, !Matched::Type{Float64})
@ Base iostream.jl:422
read(::IOStream, !Matched::Type{Float32})
@ Base iostream.jl:421
...
(::Main.var"workspace#18".myXDF.var"#1#3"{Dict{Int64, Int64}, Dict{Int64, Any}})(::IOStream)@my_XDF.jl:130
var"#open#424"(::Base.Pairs{Symbol, Union{}, Tuple{}, @NamedTuple{}}, ::typeof(open), ::Main.var"workspace#18".myXDF.var"#1#3"{Dict{Int64, Int64}, Dict{Int64, Any}}, ::String)@io.jl:396
open(::Function, ::String)@io.jl:393
read_xdf(::String, ::Bool)@my_XDF.jl:36
read_xdf(::String)@my_XDF.jl:33
top-level scope@Local: 8

[cbrnr]

Can you provide a minimal reproducible example with a file that does not work (i.e. no notebook) and then post the stack trace?

Did you try if e.g. pyxdf can read your files?

[lgarbar]

Hello,

The datasets that I've seen fail are participant data, so I won't be able to share any of those data and certainly not through a public forum. Perhaps I'll check with my supervisor if it's acceptable to share fully de-identified and encrypted data with you privately so you can take a look. Otherwise, the issue appears for some files and not others, so I can try to record some dummy data to see if I can reproduce the error and share that instead.

Otherwise, I have had no issues with pyxdf, but it is significantly slower in my experience than unpacking with Julia, so we're looking to potentially make a switch.

[cbrnr]

It would be great if you recorded some dummy data to reproduce the problem. Otherwise, it will be almost impossible to determine what the issue is.

[lgarbar]

How might you suggest sharing the data file? The dummy data I have here is >100mb, so I can't attach it here nor push it to a repo.

[cbrnr]

Do you have Google Drive, OneDrive, Dropbox, etc.? If not, I can share an upload link.

[lgarbar]

Sorry, I guess I meant if you had a preference. Here's a onedrive link with the data, jl script, and stack trace listed in a txt file

[cbrnr]

This link doesn't work, it seems to be restricted for internal use.

[lgarbar]

Makes sense. Try this link

[cbrnr]

Yes, that link worked!

[cbrnr]

Something's wrong when reading chunk 9 (stream ID 5, chunk type 3). Somehow, there's a length of 17 bytes where there should be either 1, 4, or 8 bytes. This could be due to two reasons (I can think of): (1) it's a bug or (2) string streams are not supported yet. I'm still trying to figure it out...

[lgarbar]

Hmm that's interesting. I imagine the issue itself isn't with the datatype being strings since I'm able to unpack our other data of the same format and with string streams.

[cbrnr]

It definitely has to do with your data containing a regularly sampled string stream. I have never seen this in the wild (and hence I don't have any testing data), so I did not implement support for it (yet). Of course, since these are perfectly valid in XDF, these should be supported, but someone will need to implement it. Currently, I don't have the time to do that, but I'll at least mention this limitation in the README.

[lgarbar]

Interesting. I'll try collecting a set of dummy data that can be successfully read to see if you can identify any systemic differences between the two datasets.