Merge remote-tracking branch 'upstream/master' into nodecode

.github/workflows/pytest.yml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8, 3.9, "3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12"]
 
     steps:
     - uses: actions/checkout@v2
@@ -26,7 +26,6 @@ jobs:
           ${{ runner.os }}-pip-${{ runner.os }}-
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
         pip install flake8 pytest
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         if [ -f optional_requirements.txt ]; then pip install -r optional_requirements.txt; fi

README.md

@@ -24,7 +24,7 @@ to make the experience better and these are recommended.
 Requirements
 ------------
 
-This requires `python` `v3.8.0` or higher. Unfortunately it is not
+This requires `python` `v3.10.0` or higher. Unfortunately it is not
 possible to support `swiftsimio` on versions of python lower than this.
 It is important that you upgrade if you are still a `python2` user.
 

docs/source/loading_data/index.rst

@@ -4,9 +4,9 @@ Loading Data
 The main purpose of :mod:`swiftsimio` is to load data. This section will tell
 you all about four main objects:
 
-+ :obj:`swiftsimio.reader.SWIFTUnits`, responsible for creating a correspondence between
++ :obj:`swiftsimio.metadata.objects.SWIFTUnits`, responsible for creating a correspondence between
   the SWIFT units and :mod:`unyt` objects.
-+ :obj:`swiftsimio.reader.SWIFTMetadata`, responsible for loading any required information
++ :obj:`swiftsimio.metadata.objects.SWIFTMetadata`, responsible for loading any required information
   from the SWIFT headers into python-readable data.
 + :obj:`swiftsimio.reader.SWIFTDataset`, responsible for holding all particle data, and
   keeping track of the above two objects.
@@ -47,8 +47,8 @@ notebook, and you will see that it contains several sub-objects:
   simulation.
 + ``data.dark_matter``, likewise containing information about the dark matter
   particles in the simulation.
-+ ``data.metadata``, an instance of :obj:`swiftsimio.reader.SWIFTMetadata`
-+ ``data.units``, an instance of :obj:`swiftsimio.reader.SWIFTUnits`
++ ``data.metadata``, an instance of :obj:`swiftsimio.metadata.objects.SWIFTSnapshotMetadata`
++ ``data.units``, an instance of :obj:`swiftsimio.metadata.objects.SWIFTUnits`
 
 Using metadata
 --------------
@@ -116,8 +116,49 @@ as a summary:
 Reading particle data
 ---------------------
 
-To find out what particle properties are present in our snapshot, we can use
-the instance of :obj:`swiftsimio.reader.SWIFTMetadata`, ``data.metadata``,
+To find out what particle properties are present in our snapshot, we can print
+the available particle types. For example:
+
+.. code-block:: python
+
+   data
+   
+prints the available particle types (or, more generally, groups):
+
+.. code-block:: python
+
+   SWIFT dataset at cosmo_volume_example.hdf5.
+   Available groups: gas, dark_matter, stars, black_holes
+
+The properties available for a particle type can be similarly printed:
+
+.. code-block:: python
+
+   data.dark_matter
+
+gives:
+
+.. code-block:: python
+
+   SWIFT dataset at cosmo_volume_example.hdf5.
+   Available fields: coordinates, masses, particle_ids, velocities
+   
+With compatible python interpreters, the available fields (and other attributes
+such as functions) can be seen using the tab completion feature, for example
+typing `>>> data.dark_matter.` at the command prompt and pressing tab twice
+gives:
+
+.. code-block:: python
+
+   data.dark_matter.coordinates                  data.dark_matter.masses
+   data.dark_matter.filename                     data.dark_matter.metadata
+   data.dark_matter.particle_ids                 data.dark_matter.generate_empty_properties()
+   data.dark_matter.group                        data.dark_matter.units
+   data.dark_matter.group_metadata               data.dark_matter.velocities
+   data.dark_matter.group_name                   
+
+The available fields can also be accessed programatically using the instance of
+:obj:`swiftsimio.reader.SWIFTMetadata`, ``data.metadata``,
 which contains several instances of
 :obj:`swiftsimio.reader.SWIFTParticleTypeMetadata` describing what kinds of
 fields are present in gas or dark matter:
@@ -268,3 +309,39 @@ in SWIFT will be automatically read.
    data = sw.load(
        "extra_test.hdf5",
    )
+
+
+Halo Catalogues
+---------------
+
+SWIFT-compatible halo catalogues, such as those written with SOAP, can be
+loaded entirely transparently with ``swiftsimio``. It is generally possible
+to use all of the functionality (masking, visualisation, etc.) that is used
+with snapshots with these files, assuming the files conform to the
+correct metadata standard.
+
+An example SOAP file is available at
+``http://virgodb.cosma.dur.ac.uk/swift-webstorage/IOExamples/soap_example.hdf5``
+
+You can load SOAP files as follows:
+
+.. code-block:: python
+
+   from swiftsimio import load
+
+   catalogue = load("soap_example.hdf5")
+
+   print(catalogue.spherical_overdensity_200_mean.total_mass)
+
+   # >>> [  591.      328.5     361.      553.      530.      507.      795.
+   #        574.      489.5     233.75      0.     1406.      367.5    2308.
+   #        ...
+   #        0.      534.        0.      191.75   1450.      600.      290.   ] 10000000000.0*Msun (Physical)
+
+What's going on here? Under the hood, ``swiftsimio`` has a discrimination function
+between different metadata types, based upon a property stored in the HDF5 file,
+``Header/OutputType``. If this is set to ``FullVolume``, we have a snapshot,
+and use the :obj:`swiftsimio.metadata.objects.SWIFTSnapshotMetadata`
+class. If it is ``SOAP``, we use
+:obj:`swiftsimio.metadata.objects.SWIFTSOAPMetadata`, which instructs
+``swiftsimio`` to read slightly different properties from the HDF5 file.

docs/source/masking/index.rst

@@ -123,6 +123,31 @@ ask for the temperature of particles, it will recieve an array containing
 temperatures of particles that lie in the region [0.2, 0.7] and have a
 density between 0.4 and 0.8 g/cm^3.
 
+Row Masking
+-----------
+
+For certian scenarios, in particular halo catalogues, all arrays are of the
+same length (you can check this through the ``metadata.homogeneous_arrays``
+attribute). Often, you are interested in a handful of, or a single, row,
+corresponding to the properties of a particular object. You can use the
+methods ``constrain_index`` and ``constrain_indices`` to do this, which
+return ``swiftsimio`` data objects containing arrays with only those
+rows.
+
+.. code-block:: python
+    
+    import swiftsimio as sw
+
+    mask = sw.mask(filename)
+
+    mask.constrain_indices([1, 99, 23421])
+
+    data = sw.load(filename, mask=mask)
+
+Here, the length of all the arrays will be 3. A quick performance note: if you
+are using many indices (over 1000), you will want to set ``spatial_only=False``
+to potentially benefit from range reading of overlapping rows in a single chunk.
+
 Writing subset of snapshot
 --------------------------
 In some cases it may be useful to write a subset of an existing snapshot to its

docs/source/visualisation/projection.rst

@@ -295,13 +295,13 @@ of correctly applied units to these projections soon.
 To use this feature for particle types that do not have smoothing lengths, you
 will need to generate them, as in the example below where we create a
 mass density map for dark matter. We provide a utility to do this through
-:meth:`swiftsimio.visualisation.smoothing_length_generation.generate_smoothing_lengths`.
+:meth:`swiftsimio.visualisation.smoothing_length.generate_smoothing_lengths`.
 
 .. code-block:: python
 
    from swiftsimio import load
    from swiftsimio.visualisation.projection import project_pixel_grid
-   from swiftsimio.visualisation.smoothing_length_generation import generate_smoothing_lengths
+   from swiftsimio.visualisation.smoothing_length import generate_smoothing_lengths
 
    data = load("cosmo_volume_example.hdf5")
 

docs/source/visualisation/volume_render.rst

@@ -155,6 +155,123 @@ the above example is shown below.
    # A 256 x 256 x 256 cube with dimensions of temperature
    temp_cube = mass_weighted_temp_cube / mass_cube
 
+
+Rendering
+---------
+
+We provide a volume rendering function that can be used to make images highlighting
+specific density contours. The notable function here is
+:meth:``swiftsimio.visualisation.volume_render.visualise_render``. This takes
+in your volume rendering, along with a colour map and centers, to create
+these highlights. The example below shows how to use this.
+
+.. code-block:: python
+
+   import matplotlib.pyplot as plt
+   import numpy as np
+   from matplotlib.colors import LogNorm
+   
+   from swiftsimio import load
+   from swiftsimio.visualisation import volume_render
+   
+   # Load the data
+   data = load("test_data/eagle_6.hdf5")
+   
+   # Rough location of an interesting galaxy in the volume.
+   region = [
+       0.225 * data.metadata.boxsize[0],
+       0.275 * data.metadata.boxsize[0],
+       0.12 * data.metadata.boxsize[1],
+       0.17 * data.metadata.boxsize[1],
+       0.45 * data.metadata.boxsize[2],
+       0.5 * data.metadata.boxsize[2],
+   ]
+   
+   # Render the volume (note 1024 is reasonably high resolution so this won't complete
+   # immediately; you should consider using 256, etc. for testing).
+   rendered = volume_render.render_gas(data, resolution=1024, region=region, parallel=True)
+   
+   # Quick view! By projecting along the final axis you can get
+   # the projected density from the rendered image.
+   plt.imsave("volume_render_quick_view.png", LogNorm()(rendered.sum(-1)))
+
+Here we can see the quick view of this image. It's just a regular density projection:
+
+.. image:: volume_render_quick_view.png
+
+.. code-block:: python
+   
+   # Now we will move onto the real volume rendering. Let's use the log of the density;
+   # using the real density leads to low contrast images.
+   log_rendered = np.log10(rendered)
+   
+   # The volume rendering function expects centers of 'bins' and widths. These
+   # bins actually represent gaussian functions around a specific density (or other
+   # visualization quantity). The brightest pixel value is at center. We will
+   # visualise this later!
+   width = 0.1
+   std = np.std(log_rendered)
+   mean = np.mean(log_rendered)
+   
+   # It's helpful to choose the centers relative to the data you have. When making
+   # a movie, you will obviously want to choose the centers to be the same for each
+   # frame.
+   centers = [mean + x * std for x in [1.0, 3.0, 5.0, 7.0]]
+   
+   # This will visualize your render options. The centers are shown as gaussians and
+   # vertical lines.
+   fig, ax = volume_render.visualise_render_options(
+       centers=centers, widths=width, cmap="viridis"
+   )
+   
+   histogram, edges = np.histogram(
+       log_rendered.flat,
+       bins=128,
+       range=(min(centers) - 5.0 * width, max(centers) + 5.0 * width),
+   )
+   bc = (edges[:-1] + edges[1:]) / 2.0
+   
+   # The normalization here is the height of a gaussian!
+   ax.plot(bc, histogram / (np.max(histogram) * np.sqrt(2.0 * np.pi) * width))
+   ax.semilogy()
+   ax.set_xlabel("$\\log_{10}(\\rho)$")
+   
+   plt.savefig("volume_render_options.png")
+
+This function :meth:`swiftsimio.visualisation.volume_render.visualise_render_options` allows
+you to see what densities your rendering is picking out:
+
+.. image:: volume_render_options.png
+
+.. code-block:: python   
+   
+   # Now we can really visualize the rendering.
+   img, norms = volume_render.visualise_render(
+       log_rendered,
+       centers,
+       widths=width,
+       cmap="viridis",
+   )
+   
+   # Sometimes, these images can be a bit dark. You can increase the brightness using
+   # tools like PIL or in your favourite image editor.
+   from PIL import Image, ImageEnhance
+   
+   pilimg = Image.fromarray((img * 255.0).astype(np.uint8))
+   enhanced = ImageEnhance.Contrast(ImageEnhance.Brightness(pilimg).enhance(2.0)).enhance(
+       1.2
+   )
+   
+   enhanced.save("volume_render_example.png")
+
+Which produces the image:
+
+.. image:: volume_render_example.png
+
+Once you have this base image, you can always use your photo editor to tweak it further.
+In particular, open the 'levels' panel and play around with the sliders!
+
+
 Lower-level API
 ---------------
 

pyproject.toml

@@ -12,34 +12,34 @@ packages = [
     "swiftsimio.metadata.particle",
     "swiftsimio.metadata.unit",
     "swiftsimio.metadata.writer",
+    "swiftsimio.metadata.soap",
     "swiftsimio.visualisation",
     "swiftsimio.visualisation.projection_backends",
     "swiftsimio.visualisation.slice_backends",
     "swiftsimio.visualisation.tools",
-    "swiftsimio.visualisation.smoothing_length"
+    "swiftsimio.visualisation.smoothing_length",
 ]
 
 [project]
 name = "swiftsimio"
-version="8.0.1"
+version="9.0.2"
 authors = [
     { name="Josh Borrow", email="[email protected]" },
 ]
-description="SWIFTsim (swift.dur.ac.uk) i/o routines for python."
+description="SWIFTsim (swiftsim.com) i/o routines for python."
 readme = "README.md"
-requires-python = ">3.8.0"
+requires-python = ">3.10.0"
 classifiers = [
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
     "License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)",
     "Operating System :: OS Independent",
 ]
 dependencies = [
     "numpy",
     "h5py",
-    "unyt>=2.9.0",
+    "unyt>=3.0.2",
     "numba>=0.50.0",
 ]
 

swiftsimio/__init__.py

@@ -1,5 +1,5 @@
 from .reader import *
-from .writer import SWIFTWriterDataset
+from .snapshot_writer import SWIFTSnapshotWriter
 from .masks import SWIFTMask
 from .statistics import SWIFTStatisticsFile
 from .__version__ import __version__
@@ -75,7 +75,7 @@ def mask(filename, spatial_only=True) -> SWIFTMask:
     """
 
     units = SWIFTUnits(filename)
-    metadata = SWIFTMetadata(filename, units)
+    metadata = metadata_discriminator(filename, units)
 
     return SWIFTMask(metadata=metadata, spatial_only=spatial_only)
 
@@ -109,5 +109,4 @@ def load_statistics(filename) -> SWIFTStatisticsFile:
     return SWIFTStatisticsFile(filename=filename)
 
 
-# Rename this object to something simpler.
-Writer = SWIFTWriterDataset
+Writer = SWIFTSnapshotWriter

swiftsimio/__version__.py

@@ -1 +1,3 @@
-__version__ = "8.0.1"
+import importlib.metadata
+
+__version__ = importlib.metadata.version("swiftsimio")