Undefined device_type in class created by streams_maker

[lochhh]

While creating a local DJ database from scratch and ingesting the sample data following the guide in PR SainsburyWellcomeCentre/aeon_docs#84, aeon_ingest streams_worker throws the following error:

Traceback (most recent call last):
File "...\aeon\dj_pipeline\populate\process.py", line 82, in run
worker.run()
File "...\miniconda3\envs\aeon\Lib\site-packages\datajoint_utilities\dj_worker\worker.py", line 215, in run
self.register_worker()
File "...\miniconda3\envs\aeon\Lib\site-packages\datajoint_utilities\dj_worker\worker.py", line 118, in register_worker
"key_source_sql": process.key_source.proj().make_sql()
^^^^^^^^^^^^^^^^^^
File "...\miniconda3\envs\aeon\Lib\site-packages\datajoint\user_tables.py", line 55, in getattribute
cls().getattribute(name)
File "...\aeon\dj_pipeline\streams.py", line 856, in key_source
device_type_name = dj.utils.from_camel_case(device_type)
^^^^^^^^^^^
NameError: name 'device_type' is not defined

This error stems from the following class added by streams_maker.py to aeon.dj_pipeline.streams.py:

@schema 
class SpinnakerVideoSourcePosition(dj.Imported):
        definition = """ # Raw per-chunk Position from SpinnakerVideoSource(auto-generated with vunknown)
    -> SpinnakerVideoSource
    -> acquisition.Chunk
    ---
    sample_count: int      # number of data points acquired from this stream for a given chunk
    timestamps: longblob   # (datetime) timestamps of Position data
    x: longblob
    y: longblob
    angle: longblob
    major: longblob
    minor: longblob
    area: longblob
    id: longblob
    """

        @property
        def key_source(self):
            docstring = f"""Only the combination of Chunk and SpinnakerVideoSource with overlapping time.

            + Chunk(s) started after SpinnakerVideoSource install time & ended before SpinnakerVideoSource remove time
            + Chunk(s) started after SpinnakerVideoSource install time for SpinnakerVideoSource and not yet removed
            """
            self.__doc__ = docstring
            device_type_name = dj.utils.from_camel_case(device_type)
            return (
                acquisition.Chunk * SpinnakerVideoSource.join(SpinnakerVideoSource.RemovalTime, left=True)
                & 'chunk_start >= spinnaker_video_source_install_time'
                & f'chunk_start < IFNULL({device_type_name}_removal_time,"2200-01-01")'
            )

        def make(self, key):
            """Load and insert the data for the SpinnakerVideoSourcePosition table."""
            chunk_start, chunk_end = (acquisition.Chunk & key).fetch1("chunk_start", "chunk_end")
            data_dirs = acquisition.Experiment.get_data_directories(key)

            device_name = (SpinnakerVideoSource & key).fetch1(f"{dj.utils.from_camel_case(device_type)}_name")

            devices_schema = getattr(
                aeon_schemas,
                (acquisition.Experiment.DevicesSchema & {"experiment_name": key["experiment_name"]}).fetch1(
                    "devices_schema_name"
                ),
            )
            stream_reader = getattr(getattr(devices_schema, device_name), "Position")

            stream_data = io_api.load(
                root=data_dirs,
                reader=stream_reader,
                start=pd.Timestamp(chunk_start),
                end=pd.Timestamp(chunk_end),
            )

            self.insert1(
                {
                    **key,
                    "sample_count": len(stream_data),
                    "timestamps": stream_data.index.values,
                    **{
                        re.sub(r"\([^)]*\)", "", c): stream_data[c].values
                        for c in stream_reader.columns
                        if not c.startswith("_")
                    },
                },
                ignore_extra_fields=True,
            )

[ttngu207]

@lochhh thanks for catching this!

This bug is caused by some unintended consequences during our ruff fixes that I missed. What appears to be stylistic changes actually do have negative implication to the code logic, due to the complex "code-generation" logic in streams_maker.

Particularly these 2 commits

• 8c8317a
• 6be26d1

I will revert these 2 commits, find another way to pass ruff check (or just ignore that ruff rule), and retest

[lochhh]

Thanks for checking @ttngu207 ! Note also that the table definitions do not need to be f-strings as the replacement dict will replace the placeholders enclosed within curly braces, e.g. {aeon.__version__}

[ttngu207]

I've tested your PR #466 , it's working well, I will merge soon!

[ttngu207]

Fixed via #466