dlsnxs2cbf: produces invalid files from i04-1 Eiger 9M

[graeme-winter]

These dimensions are wrong:

_array_data.data
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_BYTE_OFFSET"
Content-Transfer-Encoding: BINARY
X-Binary-Size: 9659686
X-Binary-ID: 1
X-Binary-Element-Type: "signed 32-bit integer"
X-Binary-Element-Byte-Order: LITTLE_ENDIAN
X-Binary-Number-of-Elements: 9659664
X-Binary-Size-Fastest-Dimension: 3108
X-Binary-Size-Second-Dimension: 3108
X-Binary-Size-Padding: 4095

indicated by

Ethics-Gradient work :( [main] $ ~/xtal/xds/xds_par 

 ***** XDS ***** (VERSION Jun 30, 2023  BUILT=20230630)  20-Mar-2024
 Author: Wolfgang Kabsch
 Copy licensed until 31-Aug-2024 to
  academic users for non-commercial applications
 No redistribution.

  !!! ERROR !!! INCORRECT SPECIFICATION OF NX= or NY=

[graeme-winter]

NeXus file contains the right values:

Ethics-Gradient work :) [main] $ h5ls -rvd ../../Insulin_6_2.nxs/entry/instrument/detector/module/data_size
Opened "../../Insulin_6_2.nxs" with sec2 driver.
entry/instrument/detector/module/data_size Dataset {2/2}
    Location:  1:22972
    Links:     1
    Storage:   8 logical bytes, 8 allocated bytes, 100.00% utilization
    Type:      native int
    Data:
         3108, 3262

[graeme-winter]

X-Binary-Size-Fastest-Dimension: {data.focus()[1]}
X-Binary-Size-Second-Dimension: {data.focus()[0]}

is in the f-string => I would expect this to be correct - data comes from

(data,) = dxtbx.nexus.get_raw_data(nxdata, nxdetector, j)

=> can start from there

[graeme-winter]

🤨 15 minutes in and already at the "how did this ever work" and "how even does dials work?" phase of the process

[graeme-winter]

Ethics-Gradient i04-1-run3-ins :) [main] $ dials.import Insulin_6_2.nxs ; dials.show imported.expt | grep image_size
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1103-gea4fc09a8
The following parameters have been modified:

input {
  experiments = <image files>
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatNXmxDLS.FormatNXmxDLS'>
  template: /Users/graeme/data/i04-1-run3-ins/Insulin_6_2.nxs:1:1800
  num images: 1800
  sequences:
    still:    0
    sweep:    1
  num stills: 0
--------------------------------------------------------------------------------
Writing experiments to imported.expt
  image_size: {3108,3262}

[graeme-winter]

This does not go through the usual dxtbx format mechanism

[graeme-winter]

        # dxtbx requires image size in the order fast, slow - which is the reverse of what
        # is stored in module.data_size. Additionally, data_size can have more than 2
        # dimensions, for multi-module detectors. So take the last two dimensions and reverse
        # them.  Examples:
        # [1,2,3]   --> (3, 2)
        # [1,2]     --> (2, 1)
        image_size = (int(module.data_size[-1]), int(module.data_size[-2]))

deep sustained sighing sounds

[graeme-winter]

To confirm: the dxtbx view of image size is indeed, fast, slow

Ethics-Gradient work :( [main] $ python3
Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:25:13) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dxtbx.model.experiment_list import ExperimentList
>>> el = ExperimentList.from_file("imported.expt")
>>> d = el[0].detector
>>> m = d[0]
>>> m.get_image_size()
(3108, 3262)

[graeme-winter]

OK, there are some definite sources of doubt and uncertainty surrounding the module shape - switch or not to switch? If commented out, we create CBF files which contain

X-Binary-Size-Fastest-Dimension: 3108
X-Binary-Size-Second-Dimension: 3262

Dectris say (W x H) 3,108 x 3,262 which is consistent

[graeme-winter]

With changes herein XDS is a very happy camper => need to understand why we are doing this.

@phyy-nx we (@benjaminhwilliams and I) note you have recently been exploring some of this terrain and may hold opinions. What we have at the moment is comprehensively wrong

[phyy-nx]

This nerdsniped me big time at the end of the day :) Working through my thought process:

• If someone were to take a photo of the detector, would it be taller than it is wide? In which case WxH is 3,108 x 3,262? Ima guess yes, based on how I've seen these.
• The NeXus field /entry/instrument/detector/module/data_size is 3108, 3262, that's defined as (slow, fast), or (W,H) if the above is true.
• Just to be sure, what is the shape of the data in /entry/data/data?
• So, now we move to dials.show, which is returning {3108,3262}. That should be (fast, slow), so that's backwards!
• The comment you cited is definitely right, sighing noises aside :)
• Is this line of code getting hit? Maybe it shouldn't be?
• If /entry/instrument/detector/module/data_size is right, then that cbf header would be backwards for sure

I'll stop here and await answers. Note, I have an active DLS account if you want to DM me a file path!