Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Draft PR #3355

Draft
wants to merge 65 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
6049d9a
Create audified cube and use with spectrum at spaxel tool
javerbukh Jul 9, 2024
627a34f
Add Sonify Data plugin and connect to spectrum per spaxel tool
javerbukh Jul 10, 2024
9e98b32
Fix errors
javerbukh Jul 10, 2024
e6d52b4
Try moving code to mixin
javerbukh Jul 10, 2024
0080b2f
Move code to viewers.py
javerbukh Jul 10, 2024
4b01304
Remove print statements
javerbukh Jul 10, 2024
505f73f
Patches:
Jul 11, 2024
89d6fff
fix np import
Jul 31, 2024
5c67736
add audio frequency range choice and equal loudness normalisation opt…
Aug 15, 2024
d8937c5
Create dropdown to select output sound device
javerbukh Oct 9, 2024
75b7ded
Various updates and QOL improvements
javerbukh Oct 11, 2024
9d6844d
Connect volume level in viewer to sonify plugin
javerbukh Oct 15, 2024
1d1417c
add volume attenuation functionality
Oct 16, 2024
6421ee5
add sound device switching
Oct 17, 2024
352eff9
feed ELN flag to Image Viewer
Oct 17, 2024
a7d501e
Merge pull request #18 from james-trayford/sonify-plugin-updates-jt
javerbukh Oct 17, 2024
ceafbb0
Enable start stop stream and strauss soft dependency
javerbukh Oct 18, 2024
a4811e4
Add note to plugin when strauss is not downloaded
javerbukh Oct 23, 2024
2a270ec
Add strauss as soft dependency
javerbukh Oct 25, 2024
13f37a8
Get build devices method working on windows
javerbukh Oct 31, 2024
4a1d628
ensure sound generation always uses the current spectrum-at-spaxel wl…
Oct 25, 2024
a05937d
post rebase clean-up (remove prints and rogue spaces)
Nov 3, 2024
0f0eb87
this syntax seems to work to install strauss on our specific git bran…
Nov 3, 2024
a56b26e
Merge pull request #19 from james-trayford/sonify-plugin-updates-jt
javerbukh Nov 6, 2024
18634ea
Update code to be PEP8
javerbukh Nov 6, 2024
cc51183
PEP8 fixes
javerbukh Nov 7, 2024
5d308ea
Merge branch 'main' into sonify-plugin-updates
javerbukh Nov 7, 2024
c6de013
Remove old code
javerbukh Nov 7, 2024
eaae60f
Fix test
javerbukh Nov 7, 2024
9682321
Fix test 2
javerbukh Nov 7, 2024
630be77
Update docs link in plugin
javerbukh Nov 7, 2024
51a4154
Use spectral subset for range and move advanced options to accordion
javerbukh Nov 8, 2024
cd1963e
fix volume bug (doesn't crash on vol=0)
Nov 8, 2024
de530d9
Rearrange order in plugin
javerbukh Nov 8, 2024
f3e65df
Merge pull request #20 from james-trayford/sonify-plugin-updates-jt
javerbukh Nov 8, 2024
15a8671
Fix code style
javerbukh Nov 8, 2024
a5ef389
Remove unused import
javerbukh Nov 8, 2024
a66beef
Add documentation and a test
javerbukh Nov 14, 2024
b4b64e0
Add install instructions to warning when package is not present
javerbukh Nov 14, 2024
99d04e8
Fix test failure
javerbukh Nov 14, 2024
9747650
Add plugin description
javerbukh Nov 14, 2024
b1d5806
Add plugin description
javerbukh Nov 15, 2024
2d0c5a4
Finish test and add standalone hook
javerbukh Nov 15, 2024
ae98601
Grey out start/stop stream and fix code style
javerbukh Nov 18, 2024
91ed211
Update docs and change disable message for plugin without strauss ins…
javerbukh Dec 3, 2024
30ddca8
Update jdaviz/configs/cubeviz/plugins/sonify_data/sonify_data.py
javerbukh Dec 5, 2024
1ab90f6
Merge branch 'main' into sonify-plugin-updates
javerbukh Dec 5, 2024
434a42a
Add cron job for strauss
javerbukh Dec 11, 2024
bf4fa3f
Add strauss deps to tox.ini
javerbukh Dec 11, 2024
6249a8c
Add sounddevice as dependency
javerbukh Dec 11, 2024
14c1602
Add port audio dep
javerbukh Dec 11, 2024
2dc155b
Try manually adding install for libportaudio2
javerbukh Dec 11, 2024
89dfcd2
Try different order and Python version
pllim Dec 12, 2024
8172cdd
not a Python package
pllim Dec 12, 2024
b552edc
Catch case with no sound devices and set plugin to disabled
javerbukh Dec 12, 2024
30e8ad2
catch no sound device case without errors in other cases, also remove…
Dec 12, 2024
4494ba5
Fix check for sound devices
javerbukh Dec 12, 2024
92982bd
i needed these quotes for the pip suggestion to work
Dec 12, 2024
17fd0e5
remove debug print
Dec 12, 2024
dc478ce
revert change
Dec 12, 2024
e25022f
Merge pull request #23 from james-trayford/jt_listener_mergechanges
javerbukh Dec 12, 2024
cc81d63
Fix codestyle and stop sonify plugin test running on CI
javerbukh Dec 13, 2024
fdef225
Merge branch 'main' into sonify-plugin-updates
javerbukh Dec 13, 2024
41b51f6
Add test that can run on CI
javerbukh Dec 13, 2024
a12b687
add test to run with CI
javerbukh Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move code to viewers.py
  • Loading branch information
javerbukh authored and James Trayford committed Jul 31, 2024
commit 0080b2fd288f29d2b113f539dc07470251a616a8
46 changes: 3 additions & 43 deletions jdaviz/configs/cubeviz/plugins/sonify_data/sonify_data.py
Original file line number Diff line number Diff line change
@@ -1,39 +1,13 @@
import os
from pathlib import Path

import numpy as np
import specutils
from astropy import units as u
from astropy.nddata import CCDData
from astropy.utils import minversion
from traitlets import Bool, List, Unicode, observe
from specutils import manipulation, analysis, Spectrum1D

from jdaviz.core.custom_traitlets import IntHandleEmpty, FloatHandleEmpty
from jdaviz.core.events import SnackbarMessage, GlobalDisplayUnitChanged
from jdaviz.core.registries import tray_registry
from jdaviz.core.template_mixin import (PluginTemplateMixin,
DatasetSelect, DatasetSelectMixin,
SpectralSubsetSelectMixin,
AddResultsMixin,
SelectPluginComponent,
SpectralContinuumMixin,
skip_if_no_updates_since_last_active,
with_spinner, SonifiedCubeMixin)
from jdaviz.core.validunits import check_if_unit_is_per_solid_angle
from jdaviz.core.user_api import PluginUserApi
from jdaviz.utils import flux_conversion

from jdaviz.configs.cubeviz.plugins.cube_listener import CubeListenerData
import sounddevice as sd

from jdaviz.core.template_mixin import PluginTemplateMixin, DatasetSelectMixin

__all__ = ['SonifyData']


@tray_registry('cubeviz-sonify-data', label="Sonify Data",
viewer_requirements=['spectrum', 'image'])
class SonifyData(PluginTemplateMixin, DatasetSelectMixin, SonifiedCubeMixin):
class SonifyData(PluginTemplateMixin, DatasetSelectMixin):
template_file = __file__, "sonify_data.vue"

sample_rate = IntHandleEmpty(44100).tag(sync=True)
@@ -47,19 +21,5 @@ def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)

def vue_sonify_cube(self, *args):
self.get_sonified_cube()

@with_spinner()
def get_sonified_cube(self):
viewer = self.app.get_viewer('flux-viewer')
spectrum = viewer.active_image_layer.layer.get_object(statistic=None)

clipped_arr = np.clip(spectrum.flux.value.T, 0, np.inf)
# arr = spectrum[wavemin:wavemax].flux.value.T
self.audified_cube = CubeListenerData(clipped_arr ** self.assidx, spectrum.wavelength.value, duration=0.8,
samplerate=self.sample_rate, buffsize=self.buffer_size)
self.audified_cube.audify_cube()
self.audified_cube.sigcube = (self.audified_cube.sigcube * pow(clipped_arr.sum(0) / clipped_arr.sum(0).max(), self.ssvidx)).astype('int16')
self.stream = sd.OutputStream(samplerate=self.sample_rate, blocksize=self.buffer_size, channels=1, dtype='int16', latency='low',
callback=self.audified_cube.player_callback)
self.audified_cube.cbuff = True
viewer.get_sonified_cube(self.sample_rate, self.buffer_size, self.assidx, self.ssvidx)
37 changes: 6 additions & 31 deletions jdaviz/configs/cubeviz/plugins/tools.py
Original file line number Diff line number Diff line change
@@ -11,8 +11,6 @@
from jdaviz.core.events import SliceToolStateMessage, SliceSelectSliceMessage
from jdaviz.core.tools import PanZoom, BoxZoom, SinglePixelRegion, _MatchedZoomMixin
from jdaviz.core.marks import PluginLine
from jdaviz.core.template_mixin import SonifiedCubeMixin



__all__ = []
@@ -84,7 +82,7 @@ def on_mouse_event(self, data):


@viewer_tool
class SpectrumPerSpaxel(SinglePixelRegion, SonifiedCubeMixin):
class SpectrumPerSpaxel(SinglePixelRegion):

icon = os.path.join(ICON_DIR, 'pixelspectra.svg')
tool_id = 'jdaviz:spectrumperspaxel'
@@ -98,8 +96,6 @@ def __init__(self, *args, **kwargs):
self._mark = None
self._data = None

# self.sonify_data_plg = None

def _reset_spectrum_viewer_bounds(self):
sv_state = self._spectrum_viewer.state
sv_state.x_min = self._previous_bounds[0]
@@ -122,21 +118,19 @@ def activate(self):
sv_state = self._spectrum_viewer.state
self._previous_bounds = [sv_state.x_min, sv_state.x_max, sv_state.y_min, sv_state.y_max]

# Retrieve sonified cube if present
self.start_stream()
super().activate()

def deactivate(self):
self.viewer.remove_event_callback(self.on_mouse_move)
self._reset_spectrum_viewer_bounds()
self.stop_stream()
self.viewer.stop_stream()
super().deactivate()

def on_mouse_move(self, data):
if data['event'] == 'mouseleave':
self._mark.visible = False
self._reset_spectrum_viewer_bounds()
self.stop_stream()
self.viewer.stop_stream()
return

x = int(np.round(data['domain']['x']))
@@ -176,7 +170,7 @@ def on_mouse_move(self, data):
if x >= spectrum.flux.shape[0] or x < 0 or y >= spectrum.flux.shape[1] or y < 0:
self._reset_spectrum_viewer_bounds()
self._mark.visible = False
self.stop_stream()
self.viewer.stop_stream()
else:
y_values = spectrum.flux[x, y, :]
if np.all(np.isnan(y_values)):
@@ -187,24 +181,5 @@ def on_mouse_move(self, data):
self._spectrum_viewer.state.y_max = np.nanmax(y_values.value) * 1.2
self._spectrum_viewer.state.y_min = np.nanmin(y_values.value) * 0.8

self.start_stream()
self.update_cube(x, y)

# def get_sonified_cube(self):
# spectrum = self.viewer.active_image_layer.layer.get_object(statistic=None)
# srate = 44100
# bsize = 2048
# assidx = 2.5
# ssvidx = 0.65
# wavemin = 15800
# wavemax = 16000
#
# clipped_arr = np.clip(spectrum.flux.value.T, 0, np.inf)
# arr = spectrum[wavemin:wavemax].flux.value.T
# self.audified_cube = CubeListenerData(clipped_arr ** assidx, spectrum.wavelength.value, duration=0.8,
# samplerate=srate, buffsize=bsize)
# self.audified_cube.audify_cube()
# self.audified_cube.sigcube = (self.audified_cube.sigcube * pow(clipped_arr.sum(0) / clipped_arr.sum(0).max(), ssvidx)).astype('int16')
# self.stream = sd.OutputStream(samplerate=srate, blocksize=bsize, channels=1, dtype='int16', latency='low',
# callback=self.audified_cube.player_callback)
# self.audified_cube.cbuff = True
self.viewer.start_stream()
self.viewer.update_cube(x, y)
181 changes: 180 additions & 1 deletion jdaviz/configs/cubeviz/plugins/viewers.py
Original file line number Diff line number Diff line change
@@ -6,8 +6,152 @@
from jdaviz.configs.default.plugins.viewers import JdavizViewerMixin
from jdaviz.configs.specviz.plugins.viewers import SpecvizProfileView
from jdaviz.core.freezable_state import FreezableBqplotImageViewerState
from jdaviz.configs.cubeviz.plugins.cube_listener import CubeListenerData
import sounddevice as sd

__all__ = ['CubevizImageView', 'CubevizProfileView']
__all__ = ['CubevizImageView', 'CubevizProfileView',
'WithSliceIndicator', 'WithSliceSelection']


class WithSliceIndicator:
@property
def slice_component_label(self):
return str(self.state.x_att)

@property
def slice_display_unit_name(self):
return 'spectral'

@cached_property
def slice_indicator(self):
# SliceIndicatorMarks does not yet exist
slice_indicator = SliceIndicatorMarks(self)
self.figure.marks = self.figure.marks + slice_indicator.marks
return slice_indicator

@property
def slice_values(self):
# NOTE: these are cached at the slice-plugin level
# Retrieve display units
slice_display_units = self.jdaviz_app._get_display_unit(
self.slice_display_unit_name
)

def _get_component(layer):
try:
# Retrieve layer data and units
data_comp = layer.layer.data.get_component(self.slice_component_label)
except (AttributeError, KeyError):
# layer either does not have get_component (because its a subset)
# or slice_component_label is not a component in this layer
# either way, return an empty array and skip this layer
return np.array([])

# Convert axis if display units are set and are different
data_units = getattr(data_comp, 'units', None)
if slice_display_units and data_units and slice_display_units != data_units:
data = np.asarray(data_comp.data, dtype=float) * u.Unit(data_units)
return data.to_value(slice_display_units,
equivalencies=u.spectral())
else:
return data_comp.data
try:
return np.asarray(np.unique(np.concatenate([_get_component(layer) for layer in self.layers])), # noqa
dtype=float)
except ValueError:
# NOTE: this will result in caching an empty list
return np.array([])

def _set_slice_indicator_value(self, value):
# this is a separate method so that viewers can override and map value if necessary
# NOTE: on first call, this will initialize the indicator itself
self.slice_indicator.value = value


class WithSliceSelection:
@property
def slice_index(self):
# index in state.slices corresponding to the slice axis
return 2

@property
def slice_component_label(self):
slice_plg = self.jdaviz_helper.plugins.get('Slice', None)
if slice_plg is None: # pragma: no cover
raise ValueError("slice plugin must be activated to access slice_component_label")
return slice_plg._obj.slice_indicator_viewers[0].slice_component_label

@property
def slice_display_unit_name(self):
return 'spectral'

@property
def slice_values(self):
# NOTE: these are cached at the slice-plugin level
# TODO: add support for multiple cubes (but then slice selection needs to be more complex)
# if slice_index is 0, then we want the equivalent of [:, 0, 0]
# if slice_index is 1, then we want the equivalent of [0, :, 0]
# if slice_index is 2, then we want the equivalent of [0, 0, :]
take_inds = [2, 1, 0]
take_inds.remove(self.slice_index)
converted_axis = np.array([])
for layer in self.layers:
world_comp_ids = layer.layer.data.world_component_ids
if self.slice_index >= len(world_comp_ids):
# Case where 2D image is loaded in image viewer
continue

# Retrieve display units
slice_display_units = self.jdaviz_app._get_display_unit(
self.slice_display_unit_name
)

try:
# Retrieve layer data and units using the slice index of the world components ids
data_comp = layer.layer.data.get_component(world_comp_ids[self.slice_index])
except (AttributeError, KeyError):
continue

data = np.asarray(data_comp.data.take(0, take_inds[0]).take(0, take_inds[1]), # noqa
dtype=float)

# Convert to display units if applicable
data_units = getattr(data_comp, 'units', None)
if slice_display_units and data_units and slice_display_units != data_units:
converted_axis = (data * u.Unit(data_units)).to_value(
slice_display_units,
equivalencies=u.spectral() + u.pixel_scale(1*u.pix)
)
else:
converted_axis = data

return converted_axis

@property
def slice(self):
return self.state.slices[self.slice_index]

@slice.setter
def slice(self, slice):
# NOTE: not intended for user-access - this should be controlled through the slice plugin
# in order to sync with all other viewers/slice indicators
slices = [0, 0, 0]
slices[self.slice_index] = slice
self.state.slices = tuple(slices)

@property
def slice_value(self):
return self.slice_values[self.slice]

@slice_value.setter
def slice_value(self, slice_value):
# NOTE: not intended for user-access - this should be controlled through the slice plugin
# in order to sync with all other viewers/slice indicators
# find the slice nearest slice_value
slice_values = self.slice_values
if not len(slice_values):
return
self.slice = np.argmin(abs(slice_values - slice_value))


@viewer_registry("cubeviz-image-viewer", label="Image 2D (Cubeviz)")
@@ -39,6 +183,9 @@ def __init__(self, *args, **kwargs):
# Hide axes by default
self.state.show_axes = False

self.audified_cube = None
self.stream = None

@property
def _default_spectrum_viewer_reference_name(self):
return self.jdaviz_helper._default_spectrum_viewer_reference_name
@@ -80,6 +227,38 @@ def data(self, cls=None):
if hasattr(layer_state, 'layer') and
isinstance(layer_state.layer, BaseData)]

def start_stream(self):
if hasattr(self, 'stream') and self.stream:
self.stream.start()
else:
print("unable to start stream")

def stop_stream(self):
if hasattr(self, 'stream') and self.stream:
self.stream.stop()
else:
print("unable to stop stream")

def update_cube(self, x, y):
if not hasattr(self, 'audified_cube') or not self.audified_cube or not hasattr(self.audified_cube, 'newsig') or not hasattr(self.audified_cube, 'sigcube'):
print("cube not initialized")
return
self.audified_cube.newsig = self.audified_cube.sigcube[:, x, y]
self.audified_cube.cbuff = True

def get_sonified_cube(self, sample_rate, buffer_size, assidx, ssvidx):
spectrum = self.active_image_layer.layer.get_object(statistic=None)

clipped_arr = np.clip(spectrum.flux.value.T, 0, np.inf)
# arr = spectrum[wavemin:wavemax].flux.value.T
self.audified_cube = CubeListenerData(clipped_arr ** assidx, spectrum.wavelength.value, duration=0.8,
samplerate=sample_rate, buffsize=buffer_size)
self.audified_cube.audify_cube()
self.audified_cube.sigcube = (self.audified_cube.sigcube * pow(clipped_arr.sum(0) / clipped_arr.sum(0).max(), ssvidx)).astype('int16')
self.stream = sd.OutputStream(samplerate=sample_rate, blocksize=buffer_size, channels=1, dtype='int16', latency='low',
callback=self.audified_cube.player_callback)
self.audified_cube.cbuff = True


@viewer_registry("cubeviz-profile-viewer", label="Profile 1D (Cubeviz)")
class CubevizProfileView(SpecvizProfileView, WithSliceIndicator):
25 changes: 0 additions & 25 deletions jdaviz/core/template_mixin.py
Original file line number Diff line number Diff line change
@@ -4972,28 +4972,3 @@ def clear_plot(self):
Clear all data from the current plot.
"""
self.plot.clear_plot()


class SonifiedCubeMixin:
def __init__(self, *args, **kwargs):
self.audified_cube = None
self.stream = None

def start_stream(self):
if hasattr(self, 'stream') and self.stream:
self.stream.start()
else:
print("unable to start stream")

def stop_stream(self):
if hasattr(self, 'stream') and self.stream:
self.stream.stop()
else:
print("unable to stop stream")

def update_cube(self, x, y):
if not hasattr(self, 'audified_cube') or not self.audified_cube or not hasattr(self.audified_cube, 'newsig') or not hasattr(self.audified_cube, 'sigcube'):
print("cube not initialized")
return
self.audified_cube.newsig = self.audified_cube.sigcube[:, x, y]
self.audified_cube.cbuff = True