From 99c20143bce0bfb1d602cb1c7a4a68d00167bf9a Mon Sep 17 00:00:00 2001 From: "Karl N. Kappler" Date: Fri, 25 Oct 2024 14:59:57 -0700 Subject: [PATCH] remove unused Please enter the commit message for your changes. Lines starting --- .../2024/processing_configuration.ipynb | 1433 ----------------- 1 file changed, 1433 deletions(-) delete mode 100644 notebooks/aurora/2024/processing_configuration.ipynb diff --git a/notebooks/aurora/2024/processing_configuration.ipynb b/notebooks/aurora/2024/processing_configuration.ipynb deleted file mode 100644 index b5318c1..0000000 --- a/notebooks/aurora/2024/processing_configuration.ipynb +++ /dev/null @@ -1,1433 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "8148a4bd-32e9-4551-b4bc-f45e27a51b31", - "metadata": {}, - "source": [ - "## Processing Configuration" - ] - }, - { - "cell_type": "markdown", - "id": "226f7d10-dd15-4061-adc6-c68b448d60c0", - "metadata": {}, - "source": [ - "The processing\\_config (\"config\") together with the kernel dataset represent the key inputs to the processing pipeline. The processing config is based on the `mt_metadata.base.Base` class. This means it is a container that has a JSON or dictionary representation.\n", - "\n", - "It is the purpose of the config to encapsulate all the parameters required for processing. There are many parameters and effort has been given to selecting some \"reasonable\" default values so that users do not need to worry about all of these parameters if they don't want to.\n", - "\n", - "The ProcessingConfig is expected to evolve with aurora as new functionalities become available. This is one reason why such a genreic data srtuctrue was selected \n", - "In this tutorial, we will use the synthetic dataset example to show some of the features of the config object.\n", - "\n", - "Hopefully, it will be fairly easy to add other parameters to the config, such as:\n", - "- coherencey sorting \n", - "- polarization sorting\n", - "- ARMA prewhitening \n", - "- Other tools from the community\n", - "\n", - "\n", - "\n", - "There are two main ways one would normally build the object:\n", - "1. Use the ConfigCreator class\n", - "2. Edit existing config.json files\n", - "\n", - "But one can also initialize a Processing object, which is what is done inside ConfigCreator\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "fb9a2cb3-662b-4d5f-86cc-db54e98466f4", - "metadata": {}, - "outputs": [], - "source": [ - "from mt_metadata.transfer_functions.processing.aurora import Processing" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "777182c2-b77f-44b8-a663-1c7afe00796e", - "metadata": {}, - "outputs": [], - "source": [ - "p = Processing()" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "d9069a6b-b13e-4afb-8f11-12546eb2d035", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{\n", - " \"processing\": {\n", - " \"channel_nomenclature.ex\": \"ex\",\n", - " \"channel_nomenclature.ey\": \"ey\",\n", - " \"channel_nomenclature.hx\": \"hx\",\n", - " \"channel_nomenclature.hy\": \"hy\",\n", - " \"channel_nomenclature.hz\": \"hz\",\n", - " \"decimations\": [],\n", - " \"id\": null,\n", - " \"stations.local.id\": null,\n", - " \"stations.local.mth5_path\": null,\n", - " \"stations.local.remote\": false,\n", - " \"stations.local.runs\": [],\n", - " \"stations.remote\": []\n", - " }\n", - "}" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "p" - ] - }, - { - "cell_type": "markdown", - "id": "54ce7d72-1ed7-4f77-bacd-ccab442440bc", - "metadata": {}, - "source": [ - "### Using ConfigCreator" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "02880258-bac4-44e3-b602-cc8529f66a01", - "metadata": {}, - "outputs": [], - "source": [ - "from aurora.config import BANDS_DEFAULT_FILE\n", - "from aurora.config.config_creator import ConfigCreator\n" - ] - }, - { - "cell_type": "markdown", - "id": "b7ba7136-21f0-4f57-8eb5-d74071e5fdab", - "metadata": {}, - "source": [ - "The config creator does not need any arguments to initialize, but it uses a KernelDataset to generate a processing object with station and run information, and default settings are applied to the processing parameters." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "82dfb0da-fb31-48cb-a951-c8614a8f0ea1", - "metadata": {}, - "outputs": [], - "source": [ - "cc = ConfigCreator()" - ] - }, - { - "cell_type": "markdown", - "id": "6d072d95-c0e4-4c01-9bf2-e1e1d0397b3e", - "metadata": {}, - "source": [ - "The ConfigCreator class generates a processing config with default arguments if it is provided with a KernelDataset" - ] - }, - { - "cell_type": "markdown", - "id": "5d8359f0-6a5d-461f-88d4-bdc3a49e228a", - "metadata": {}, - "source": [ - "## Example of making a KernelDataset from an mth5" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "8e3a5ef1-b00d-4263-890a-cfe028a712b9", - "metadata": {}, - "outputs": [], - "source": [ - "from aurora.test_utils.synthetic.paths import SyntheticTestPaths\n", - "from mtpy.processing import RunSummary, KernelDataset\n", - "\n", - "synthetic_test_paths = SyntheticTestPaths()\n", - "MTH5_PATH = synthetic_test_paths.mth5_path" - ] - }, - { - "cell_type": "markdown", - "id": "7c013dff-14bf-402b-90d5-07151eabc075", - "metadata": {}, - "source": [ - "Make an example file to work with" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "3c7a4efa-8059-4d12-9421-1c1d905f38a1", - "metadata": {}, - "outputs": [], - "source": [ - "mth5_path = MTH5_PATH.joinpath(\"test12rr.h5\")\n", - "if mth5_path.exists():\n", - " pass\n", - "else:\n", - " synthetic_test_paths.mkdirs()\n", - " from mth5.data.make_mth5_from_asc import create_test12rr_h5\n", - " create_test12rr_h5(target_folder=MTH5_PATH)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "3bfaafa2-af89-4d08-ad76-874f18b66c14", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1m24:08:28T15:59:26 | INFO | line:761 |mth5.mth5 | close_mth5 | Flushing and closing /home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5\u001b[0m\n" - ] - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
channel_scale_factorsdurationendhas_datainput_channelsmth5_pathn_samplesoutput_channelsrunsample_ratestartstationsurveyrun_hdf5_referencestation_hdf5_reference
0{'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '...39999.01980-01-01 11:06:39+00:00True[hx, hy]/home/kkappler/software/irismt/aurora/data/syn...40000[ex, ey, hz]0011.01980-01-01 00:00:00+00:00test1EMTF Synthetic<HDF5 object reference><HDF5 object reference>
1{'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '...39999.01980-01-01 11:06:39+00:00True[hx, hy]/home/kkappler/software/irismt/aurora/data/syn...40000[ex, ey, hz]0011.01980-01-01 00:00:00+00:00test2EMTF Synthetic<HDF5 object reference><HDF5 object reference>
\n", - "
" - ], - "text/plain": [ - " channel_scale_factors duration \\\n", - "0 {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... 39999.0 \n", - "1 {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... 39999.0 \n", - "\n", - " end has_data input_channels \\\n", - "0 1980-01-01 11:06:39+00:00 True [hx, hy] \n", - "1 1980-01-01 11:06:39+00:00 True [hx, hy] \n", - "\n", - " mth5_path n_samples \\\n", - "0 /home/kkappler/software/irismt/aurora/data/syn... 40000 \n", - "1 /home/kkappler/software/irismt/aurora/data/syn... 40000 \n", - "\n", - " output_channels run sample_rate start station \\\n", - "0 [ex, ey, hz] 001 1.0 1980-01-01 00:00:00+00:00 test1 \n", - "1 [ex, ey, hz] 001 1.0 1980-01-01 00:00:00+00:00 test2 \n", - "\n", - " survey run_hdf5_reference station_hdf5_reference \n", - "0 EMTF Synthetic \n", - "1 EMTF Synthetic " - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "run_summary = RunSummary()\n", - "run_summary.from_mth5s([mth5_path,])\n", - "run_summary.df" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "f5ddde68-45a8-4d5c-9df3-ca64f810931d", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1m24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column fc, adding and setting dtype to .\u001b[0m\n", - "\u001b[1m24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column remote, adding and setting dtype to .\u001b[0m\n", - "\u001b[1m24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column run_dataarray, adding and setting dtype to .\u001b[0m\n", - "\u001b[1m24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column stft, adding and setting dtype to .\u001b[0m\n", - "\u001b[1m24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column mth5_obj, adding and setting dtype to .\u001b[0m\n" - ] - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
channel_scale_factorsdurationendhas_datainput_channelsmth5_pathn_samplesoutput_channelsrunsample_ratestartstationsurveyrun_hdf5_referencestation_hdf5_referencefcremoterun_dataarraystftmth5_obj
0{'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '...39999.01980-01-01 11:06:39+00:00True[hx, hy]/home/kkappler/software/irismt/aurora/data/syn...40000[ex, ey, hz]0011.01980-01-01 00:00:00+00:00test1EMTF Synthetic<HDF5 object reference><HDF5 object reference>FalseFalseNoneNoneNone
1{'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '...39999.01980-01-01 11:06:39+00:00True[hx, hy]/home/kkappler/software/irismt/aurora/data/syn...40000[ex, ey, hz]0011.01980-01-01 00:00:00+00:00test2EMTF Synthetic<HDF5 object reference><HDF5 object reference>FalseTrueNoneNoneNone
\n", - "
" - ], - "text/plain": [ - " channel_scale_factors duration \\\n", - "0 {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... 39999.0 \n", - "1 {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... 39999.0 \n", - "\n", - " end has_data input_channels \\\n", - "0 1980-01-01 11:06:39+00:00 True [hx, hy] \n", - "1 1980-01-01 11:06:39+00:00 True [hx, hy] \n", - "\n", - " mth5_path n_samples \\\n", - "0 /home/kkappler/software/irismt/aurora/data/syn... 40000 \n", - "1 /home/kkappler/software/irismt/aurora/data/syn... 40000 \n", - "\n", - " output_channels run sample_rate start station \\\n", - "0 [ex, ey, hz] 001 1.0 1980-01-01 00:00:00+00:00 test1 \n", - "1 [ex, ey, hz] 001 1.0 1980-01-01 00:00:00+00:00 test2 \n", - "\n", - " survey run_hdf5_reference station_hdf5_reference fc \\\n", - "0 EMTF Synthetic False \n", - "1 EMTF Synthetic False \n", - "\n", - " remote run_dataarray stft mth5_obj \n", - "0 False None None None \n", - "1 True None None None " - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "kernel_dataset = KernelDataset()\n", - "kernel_dataset.from_run_summary(run_summary, \"test1\", \"test2\")\n", - "kernel_dataset.df" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "4c200570-fb3f-46d0-a98c-37dbdbd57a29", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
surveystationrunstartendduration
0EMTF Synthetictest10011980-01-01 00:00:00+00:001980-01-01 11:06:39+00:0039999.0
1EMTF Synthetictest20011980-01-01 00:00:00+00:001980-01-01 11:06:39+00:0039999.0
\n", - "
" - ], - "text/plain": [ - " survey station run start \\\n", - "0 EMTF Synthetic test1 001 1980-01-01 00:00:00+00:00 \n", - "1 EMTF Synthetic test2 001 1980-01-01 00:00:00+00:00 \n", - "\n", - " end duration \n", - "0 1980-01-01 11:06:39+00:00 39999.0 \n", - "1 1980-01-01 11:06:39+00:00 39999.0 " - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "kernel_dataset.mini_summary" - ] - }, - { - "cell_type": "markdown", - "id": "18b9e827-38d1-4af7-905a-26babfac8cd5", - "metadata": {}, - "source": [ - "### Create Config from KernelDataset" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "4e543c6f-13ff-41c6-8d65-069961af57e0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[1m24:08:28T15:59:26 | INFO | line:108 |aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE\u001b[0m\n" - ] - } - ], - "source": [ - "config = cc.create_from_kernel_dataset(kernel_dataset)" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "8fe824ac-455b-43b6-a18b-6b147f1ac6fa", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/plain": [ - "{\n", - " \"processing\": {\n", - " \"band_setup_file\": \"/home/kkappler/software/irismt/aurora/aurora/config/emtf_band_setup/bs_test.cfg\",\n", - " \"band_specification_style\": \"EMTF\",\n", - " \"channel_nomenclature.ex\": \"ex\",\n", - " \"channel_nomenclature.ey\": \"ey\",\n", - " \"channel_nomenclature.hx\": \"hx\",\n", - " \"channel_nomenclature.hy\": \"hy\",\n", - " \"channel_nomenclature.hz\": \"hz\",\n", - " \"decimations\": [\n", - " {\n", - " \"decimation_level\": {\n", - " \"anti_alias_filter\": \"default\",\n", - " \"bands\": [\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.23828125,\n", - " \"frequency_min\": 0.19140625,\n", - " \"index_max\": 30,\n", - " \"index_min\": 25\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.19140625,\n", - " \"frequency_min\": 0.15234375,\n", - " \"index_max\": 24,\n", - " \"index_min\": 20\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.15234375,\n", - " \"frequency_min\": 0.12109375,\n", - " \"index_max\": 19,\n", - " \"index_min\": 16\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.12109375,\n", - " \"frequency_min\": 0.09765625,\n", - " \"index_max\": 15,\n", - " \"index_min\": 13\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.09765625,\n", - " \"frequency_min\": 0.07421875,\n", - " \"index_max\": 12,\n", - " \"index_min\": 10\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.07421875,\n", - " \"frequency_min\": 0.05859375,\n", - " \"index_max\": 9,\n", - " \"index_min\": 8\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.05859375,\n", - " \"frequency_min\": 0.04296875,\n", - " \"index_max\": 7,\n", - " \"index_min\": 6\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 0,\n", - " \"frequency_max\": 0.04296875,\n", - " \"frequency_min\": 0.03515625,\n", - " \"index_max\": 5,\n", - " \"index_min\": 5\n", - " }\n", - " }\n", - " ],\n", - " \"decimation.factor\": 1.0,\n", - " \"decimation.level\": 0,\n", - " \"decimation.method\": \"default\",\n", - " \"decimation.sample_rate\": 1.0,\n", - " \"estimator.engine\": \"RME_RR\",\n", - " \"estimator.estimate_per_channel\": true,\n", - " \"extra_pre_fft_detrend_type\": \"linear\",\n", - " \"input_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"method\": \"fft\",\n", - " \"min_num_stft_windows\": 2,\n", - " \"output_channels\": [\n", - " \"ex\",\n", - " \"ey\",\n", - " \"hz\"\n", - " ],\n", - " \"pre_fft_detrend_type\": \"linear\",\n", - " \"prewhitening_type\": \"first difference\",\n", - " \"recoloring\": true,\n", - " \"reference_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"regression.max_iterations\": 10,\n", - " \"regression.max_redescending_iterations\": 2,\n", - " \"regression.minimum_cycles\": 10,\n", - " \"regression.r0\": 1.5,\n", - " \"regression.tolerance\": 0.005,\n", - " \"regression.u0\": 2.8,\n", - " \"regression.verbosity\": 0,\n", - " \"save_fcs\": false,\n", - " \"window.clock_zero_type\": \"ignore\",\n", - " \"window.num_samples\": 128,\n", - " \"window.overlap\": 32,\n", - " \"window.type\": \"boxcar\"\n", - " }\n", - " },\n", - " {\n", - " \"decimation_level\": {\n", - " \"anti_alias_filter\": \"default\",\n", - " \"bands\": [\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 1,\n", - " \"frequency_max\": 0.0341796875,\n", - " \"frequency_min\": 0.0263671875,\n", - " \"index_max\": 17,\n", - " \"index_min\": 14\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 1,\n", - " \"frequency_max\": 0.0263671875,\n", - " \"frequency_min\": 0.0205078125,\n", - " \"index_max\": 13,\n", - " \"index_min\": 11\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 1,\n", - " \"frequency_max\": 0.0205078125,\n", - " \"frequency_min\": 0.0166015625,\n", - " \"index_max\": 10,\n", - " \"index_min\": 9\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 1,\n", - " \"frequency_max\": 0.0166015625,\n", - " \"frequency_min\": 0.0126953125,\n", - " \"index_max\": 8,\n", - " \"index_min\": 7\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 1,\n", - " \"frequency_max\": 0.0126953125,\n", - " \"frequency_min\": 0.0107421875,\n", - " \"index_max\": 6,\n", - " \"index_min\": 6\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 1,\n", - " \"frequency_max\": 0.0107421875,\n", - " \"frequency_min\": 0.0087890625,\n", - " \"index_max\": 5,\n", - " \"index_min\": 5\n", - " }\n", - " }\n", - " ],\n", - " \"decimation.factor\": 4.0,\n", - " \"decimation.level\": 1,\n", - " \"decimation.method\": \"default\",\n", - " \"decimation.sample_rate\": 0.25,\n", - " \"estimator.engine\": \"RME_RR\",\n", - " \"estimator.estimate_per_channel\": true,\n", - " \"extra_pre_fft_detrend_type\": \"linear\",\n", - " \"input_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"method\": \"fft\",\n", - " \"min_num_stft_windows\": 2,\n", - " \"output_channels\": [\n", - " \"ex\",\n", - " \"ey\",\n", - " \"hz\"\n", - " ],\n", - " \"pre_fft_detrend_type\": \"linear\",\n", - " \"prewhitening_type\": \"first difference\",\n", - " \"recoloring\": true,\n", - " \"reference_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"regression.max_iterations\": 10,\n", - " \"regression.max_redescending_iterations\": 2,\n", - " \"regression.minimum_cycles\": 10,\n", - " \"regression.r0\": 1.5,\n", - " \"regression.tolerance\": 0.005,\n", - " \"regression.u0\": 2.8,\n", - " \"regression.verbosity\": 0,\n", - " \"save_fcs\": false,\n", - " \"window.clock_zero_type\": \"ignore\",\n", - " \"window.num_samples\": 128,\n", - " \"window.overlap\": 32,\n", - " \"window.type\": \"boxcar\"\n", - " }\n", - " },\n", - " {\n", - " \"decimation_level\": {\n", - " \"anti_alias_filter\": \"default\",\n", - " \"bands\": [\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 2,\n", - " \"frequency_max\": 0.008544921875,\n", - " \"frequency_min\": 0.006591796875,\n", - " \"index_max\": 17,\n", - " \"index_min\": 14\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 2,\n", - " \"frequency_max\": 0.006591796875,\n", - " \"frequency_min\": 0.005126953125,\n", - " \"index_max\": 13,\n", - " \"index_min\": 11\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 2,\n", - " \"frequency_max\": 0.005126953125,\n", - " \"frequency_min\": 0.004150390625,\n", - " \"index_max\": 10,\n", - " \"index_min\": 9\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 2,\n", - " \"frequency_max\": 0.004150390625,\n", - " \"frequency_min\": 0.003173828125,\n", - " \"index_max\": 8,\n", - " \"index_min\": 7\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 2,\n", - " \"frequency_max\": 0.003173828125,\n", - " \"frequency_min\": 0.002685546875,\n", - " \"index_max\": 6,\n", - " \"index_min\": 6\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 2,\n", - " \"frequency_max\": 0.002685546875,\n", - " \"frequency_min\": 0.002197265625,\n", - " \"index_max\": 5,\n", - " \"index_min\": 5\n", - " }\n", - " }\n", - " ],\n", - " \"decimation.factor\": 4.0,\n", - " \"decimation.level\": 2,\n", - " \"decimation.method\": \"default\",\n", - " \"decimation.sample_rate\": 0.0625,\n", - " \"estimator.engine\": \"RME_RR\",\n", - " \"estimator.estimate_per_channel\": true,\n", - " \"extra_pre_fft_detrend_type\": \"linear\",\n", - " \"input_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"method\": \"fft\",\n", - " \"min_num_stft_windows\": 2,\n", - " \"output_channels\": [\n", - " \"ex\",\n", - " \"ey\",\n", - " \"hz\"\n", - " ],\n", - " \"pre_fft_detrend_type\": \"linear\",\n", - " \"prewhitening_type\": \"first difference\",\n", - " \"recoloring\": true,\n", - " \"reference_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"regression.max_iterations\": 10,\n", - " \"regression.max_redescending_iterations\": 2,\n", - " \"regression.minimum_cycles\": 10,\n", - " \"regression.r0\": 1.5,\n", - " \"regression.tolerance\": 0.005,\n", - " \"regression.u0\": 2.8,\n", - " \"regression.verbosity\": 0,\n", - " \"save_fcs\": false,\n", - " \"window.clock_zero_type\": \"ignore\",\n", - " \"window.num_samples\": 128,\n", - " \"window.overlap\": 32,\n", - " \"window.type\": \"boxcar\"\n", - " }\n", - " },\n", - " {\n", - " \"decimation_level\": {\n", - " \"anti_alias_filter\": \"default\",\n", - " \"bands\": [\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 3,\n", - " \"frequency_max\": 0.00274658203125,\n", - " \"frequency_min\": 0.00213623046875,\n", - " \"index_max\": 22,\n", - " \"index_min\": 18\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 3,\n", - " \"frequency_max\": 0.00213623046875,\n", - " \"frequency_min\": 0.00164794921875,\n", - " \"index_max\": 17,\n", - " \"index_min\": 14\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 3,\n", - " \"frequency_max\": 0.00164794921875,\n", - " \"frequency_min\": 0.00115966796875,\n", - " \"index_max\": 13,\n", - " \"index_min\": 10\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 3,\n", - " \"frequency_max\": 0.00115966796875,\n", - " \"frequency_min\": 0.00079345703125,\n", - " \"index_max\": 9,\n", - " \"index_min\": 7\n", - " }\n", - " },\n", - " {\n", - " \"band\": {\n", - " \"center_averaging_type\": \"geometric\",\n", - " \"closed\": \"left\",\n", - " \"decimation_level\": 3,\n", - " \"frequency_max\": 0.00079345703125,\n", - " \"frequency_min\": 0.00054931640625,\n", - " \"index_max\": 6,\n", - " \"index_min\": 5\n", - " }\n", - " }\n", - " ],\n", - " \"decimation.factor\": 4.0,\n", - " \"decimation.level\": 3,\n", - " \"decimation.method\": \"default\",\n", - " \"decimation.sample_rate\": 0.015625,\n", - " \"estimator.engine\": \"RME_RR\",\n", - " \"estimator.estimate_per_channel\": true,\n", - " \"extra_pre_fft_detrend_type\": \"linear\",\n", - " \"input_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"method\": \"fft\",\n", - " \"min_num_stft_windows\": 2,\n", - " \"output_channels\": [\n", - " \"ex\",\n", - " \"ey\",\n", - " \"hz\"\n", - " ],\n", - " \"pre_fft_detrend_type\": \"linear\",\n", - " \"prewhitening_type\": \"first difference\",\n", - " \"recoloring\": true,\n", - " \"reference_channels\": [\n", - " \"hx\",\n", - " \"hy\"\n", - " ],\n", - " \"regression.max_iterations\": 10,\n", - " \"regression.max_redescending_iterations\": 2,\n", - " \"regression.minimum_cycles\": 10,\n", - " \"regression.r0\": 1.5,\n", - " \"regression.tolerance\": 0.005,\n", - " \"regression.u0\": 2.8,\n", - " \"regression.verbosity\": 0,\n", - " \"save_fcs\": false,\n", - " \"window.clock_zero_type\": \"ignore\",\n", - " \"window.num_samples\": 128,\n", - " \"window.overlap\": 32,\n", - " \"window.type\": \"boxcar\"\n", - " }\n", - " }\n", - " ],\n", - " \"id\": \"test1-rr_test2_sr1\",\n", - " \"stations.local.id\": \"test1\",\n", - " \"stations.local.mth5_path\": \"/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5\",\n", - " \"stations.local.remote\": false,\n", - " \"stations.local.runs\": [\n", - " {\n", - " \"run\": {\n", - " \"id\": \"001\",\n", - " \"input_channels\": [\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"hx\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " },\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"hy\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " }\n", - " ],\n", - " \"output_channels\": [\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"ex\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " },\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"ey\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " },\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"hz\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " }\n", - " ],\n", - " \"sample_rate\": 1.0,\n", - " \"time_periods\": [\n", - " {\n", - " \"time_period\": {\n", - " \"end\": \"1980-01-01T11:06:39+00:00\",\n", - " \"start\": \"1980-01-01T00:00:00+00:00\"\n", - " }\n", - " }\n", - " ]\n", - " }\n", - " }\n", - " ],\n", - " \"stations.remote\": [\n", - " {\n", - " \"station\": {\n", - " \"id\": \"test2\",\n", - " \"mth5_path\": \"/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5\",\n", - " \"remote\": true,\n", - " \"runs\": [\n", - " {\n", - " \"run\": {\n", - " \"id\": \"001\",\n", - " \"input_channels\": [\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"hx\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " },\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"hy\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " }\n", - " ],\n", - " \"output_channels\": [\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"ex\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " },\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"ey\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " },\n", - " {\n", - " \"channel\": {\n", - " \"id\": \"hz\",\n", - " \"scale_factor\": 1.0\n", - " }\n", - " }\n", - " ],\n", - " \"sample_rate\": 1.0,\n", - " \"time_periods\": [\n", - " {\n", - " \"time_period\": {\n", - " \"end\": \"1980-01-01T11:06:39+00:00\",\n", - " \"start\": \"1980-01-01T00:00:00+00:00\"\n", - " }\n", - " }\n", - " ]\n", - " }\n", - " }\n", - " ]\n", - " }\n", - " }\n", - " ]\n", - " }\n", - "}" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "config" - ] - }, - { - "cell_type": "markdown", - "id": "9c820da4-1d30-4807-9cba-0f640f22ae40", - "metadata": {}, - "source": [ - "You can see the entire config by executing the cell below, or you can cut and paste the json code into a json editor (e.g. https://jsoneditoronline.org) and then you can look at it heirarchically." - ] - }, - { - "cell_type": "markdown", - "id": "1a2f65cb-fe61-43b9-bbc6-fce025869ec2", - "metadata": {}, - "source": [ - "![CONFIG_0](figures/config_0.png)" - ] - }, - { - "cell_type": "markdown", - "id": "bbe450e6-7031-44e2-b6d1-7ddd81577f61", - "metadata": {}, - "source": [ - "You can also transform the processing object to a json string" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "580e77cb-94d1-4d5c-bcf8-516a5557ce6c", - "metadata": {}, - "outputs": [], - "source": [ - "json_string = config.to_json()" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "e50cc515-a135-49a2-851e-03807358b109", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/plain": [ - "'{\\n \"processing\": {\\n \"band_setup_file\": \"/home/kkappler/software/irismt/aurora/aurora/config/emtf_band_setup/bs_test.cfg\",\\n \"band_specification_style\": \"EMTF\",\\n \"channel_nomenclature.ex\": \"ex\",\\n \"channel_nomenclature.ey\": \"ey\",\\n \"channel_nomenclature.hx\": \"hx\",\\n \"channel_nomenclature.hy\": \"hy\",\\n \"channel_nomenclature.hz\": \"hz\",\\n \"decimations\": [\\n {\\n \"decimation_level\": {\\n \"anti_alias_filter\": \"default\",\\n \"bands\": [\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.23828125,\\n \"frequency_min\": 0.19140625,\\n \"index_max\": 30,\\n \"index_min\": 25\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.19140625,\\n \"frequency_min\": 0.15234375,\\n \"index_max\": 24,\\n \"index_min\": 20\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.15234375,\\n \"frequency_min\": 0.12109375,\\n \"index_max\": 19,\\n \"index_min\": 16\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.12109375,\\n \"frequency_min\": 0.09765625,\\n \"index_max\": 15,\\n \"index_min\": 13\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.09765625,\\n \"frequency_min\": 0.07421875,\\n \"index_max\": 12,\\n \"index_min\": 10\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.07421875,\\n \"frequency_min\": 0.05859375,\\n \"index_max\": 9,\\n \"index_min\": 8\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.05859375,\\n \"frequency_min\": 0.04296875,\\n \"index_max\": 7,\\n \"index_min\": 6\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 0,\\n \"frequency_max\": 0.04296875,\\n \"frequency_min\": 0.03515625,\\n \"index_max\": 5,\\n \"index_min\": 5\\n }\\n }\\n ],\\n \"decimation.factor\": 1.0,\\n \"decimation.level\": 0,\\n \"decimation.method\": \"default\",\\n \"decimation.sample_rate\": 1.0,\\n \"estimator.engine\": \"RME_RR\",\\n \"estimator.estimate_per_channel\": true,\\n \"extra_pre_fft_detrend_type\": \"linear\",\\n \"input_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"method\": \"fft\",\\n \"min_num_stft_windows\": 2,\\n \"output_channels\": [\\n \"ex\",\\n \"ey\",\\n \"hz\"\\n ],\\n \"pre_fft_detrend_type\": \"linear\",\\n \"prewhitening_type\": \"first difference\",\\n \"recoloring\": true,\\n \"reference_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"regression.max_iterations\": 10,\\n \"regression.max_redescending_iterations\": 2,\\n \"regression.minimum_cycles\": 10,\\n \"regression.r0\": 1.5,\\n \"regression.tolerance\": 0.005,\\n \"regression.u0\": 2.8,\\n \"regression.verbosity\": 0,\\n \"save_fcs\": false,\\n \"window.clock_zero_type\": \"ignore\",\\n \"window.num_samples\": 128,\\n \"window.overlap\": 32,\\n \"window.type\": \"boxcar\"\\n }\\n },\\n {\\n \"decimation_level\": {\\n \"anti_alias_filter\": \"default\",\\n \"bands\": [\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 1,\\n \"frequency_max\": 0.0341796875,\\n \"frequency_min\": 0.0263671875,\\n \"index_max\": 17,\\n \"index_min\": 14\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 1,\\n \"frequency_max\": 0.0263671875,\\n \"frequency_min\": 0.0205078125,\\n \"index_max\": 13,\\n \"index_min\": 11\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 1,\\n \"frequency_max\": 0.0205078125,\\n \"frequency_min\": 0.0166015625,\\n \"index_max\": 10,\\n \"index_min\": 9\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 1,\\n \"frequency_max\": 0.0166015625,\\n \"frequency_min\": 0.0126953125,\\n \"index_max\": 8,\\n \"index_min\": 7\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 1,\\n \"frequency_max\": 0.0126953125,\\n \"frequency_min\": 0.0107421875,\\n \"index_max\": 6,\\n \"index_min\": 6\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 1,\\n \"frequency_max\": 0.0107421875,\\n \"frequency_min\": 0.0087890625,\\n \"index_max\": 5,\\n \"index_min\": 5\\n }\\n }\\n ],\\n \"decimation.factor\": 4.0,\\n \"decimation.level\": 1,\\n \"decimation.method\": \"default\",\\n \"decimation.sample_rate\": 0.25,\\n \"estimator.engine\": \"RME_RR\",\\n \"estimator.estimate_per_channel\": true,\\n \"extra_pre_fft_detrend_type\": \"linear\",\\n \"input_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"method\": \"fft\",\\n \"min_num_stft_windows\": 2,\\n \"output_channels\": [\\n \"ex\",\\n \"ey\",\\n \"hz\"\\n ],\\n \"pre_fft_detrend_type\": \"linear\",\\n \"prewhitening_type\": \"first difference\",\\n \"recoloring\": true,\\n \"reference_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"regression.max_iterations\": 10,\\n \"regression.max_redescending_iterations\": 2,\\n \"regression.minimum_cycles\": 10,\\n \"regression.r0\": 1.5,\\n \"regression.tolerance\": 0.005,\\n \"regression.u0\": 2.8,\\n \"regression.verbosity\": 0,\\n \"save_fcs\": false,\\n \"window.clock_zero_type\": \"ignore\",\\n \"window.num_samples\": 128,\\n \"window.overlap\": 32,\\n \"window.type\": \"boxcar\"\\n }\\n },\\n {\\n \"decimation_level\": {\\n \"anti_alias_filter\": \"default\",\\n \"bands\": [\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 2,\\n \"frequency_max\": 0.008544921875,\\n \"frequency_min\": 0.006591796875,\\n \"index_max\": 17,\\n \"index_min\": 14\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 2,\\n \"frequency_max\": 0.006591796875,\\n \"frequency_min\": 0.005126953125,\\n \"index_max\": 13,\\n \"index_min\": 11\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 2,\\n \"frequency_max\": 0.005126953125,\\n \"frequency_min\": 0.004150390625,\\n \"index_max\": 10,\\n \"index_min\": 9\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 2,\\n \"frequency_max\": 0.004150390625,\\n \"frequency_min\": 0.003173828125,\\n \"index_max\": 8,\\n \"index_min\": 7\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 2,\\n \"frequency_max\": 0.003173828125,\\n \"frequency_min\": 0.002685546875,\\n \"index_max\": 6,\\n \"index_min\": 6\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 2,\\n \"frequency_max\": 0.002685546875,\\n \"frequency_min\": 0.002197265625,\\n \"index_max\": 5,\\n \"index_min\": 5\\n }\\n }\\n ],\\n \"decimation.factor\": 4.0,\\n \"decimation.level\": 2,\\n \"decimation.method\": \"default\",\\n \"decimation.sample_rate\": 0.0625,\\n \"estimator.engine\": \"RME_RR\",\\n \"estimator.estimate_per_channel\": true,\\n \"extra_pre_fft_detrend_type\": \"linear\",\\n \"input_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"method\": \"fft\",\\n \"min_num_stft_windows\": 2,\\n \"output_channels\": [\\n \"ex\",\\n \"ey\",\\n \"hz\"\\n ],\\n \"pre_fft_detrend_type\": \"linear\",\\n \"prewhitening_type\": \"first difference\",\\n \"recoloring\": true,\\n \"reference_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"regression.max_iterations\": 10,\\n \"regression.max_redescending_iterations\": 2,\\n \"regression.minimum_cycles\": 10,\\n \"regression.r0\": 1.5,\\n \"regression.tolerance\": 0.005,\\n \"regression.u0\": 2.8,\\n \"regression.verbosity\": 0,\\n \"save_fcs\": false,\\n \"window.clock_zero_type\": \"ignore\",\\n \"window.num_samples\": 128,\\n \"window.overlap\": 32,\\n \"window.type\": \"boxcar\"\\n }\\n },\\n {\\n \"decimation_level\": {\\n \"anti_alias_filter\": \"default\",\\n \"bands\": [\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 3,\\n \"frequency_max\": 0.00274658203125,\\n \"frequency_min\": 0.00213623046875,\\n \"index_max\": 22,\\n \"index_min\": 18\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 3,\\n \"frequency_max\": 0.00213623046875,\\n \"frequency_min\": 0.00164794921875,\\n \"index_max\": 17,\\n \"index_min\": 14\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 3,\\n \"frequency_max\": 0.00164794921875,\\n \"frequency_min\": 0.00115966796875,\\n \"index_max\": 13,\\n \"index_min\": 10\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 3,\\n \"frequency_max\": 0.00115966796875,\\n \"frequency_min\": 0.00079345703125,\\n \"index_max\": 9,\\n \"index_min\": 7\\n }\\n },\\n {\\n \"band\": {\\n \"center_averaging_type\": \"geometric\",\\n \"closed\": \"left\",\\n \"decimation_level\": 3,\\n \"frequency_max\": 0.00079345703125,\\n \"frequency_min\": 0.00054931640625,\\n \"index_max\": 6,\\n \"index_min\": 5\\n }\\n }\\n ],\\n \"decimation.factor\": 4.0,\\n \"decimation.level\": 3,\\n \"decimation.method\": \"default\",\\n \"decimation.sample_rate\": 0.015625,\\n \"estimator.engine\": \"RME_RR\",\\n \"estimator.estimate_per_channel\": true,\\n \"extra_pre_fft_detrend_type\": \"linear\",\\n \"input_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"method\": \"fft\",\\n \"min_num_stft_windows\": 2,\\n \"output_channels\": [\\n \"ex\",\\n \"ey\",\\n \"hz\"\\n ],\\n \"pre_fft_detrend_type\": \"linear\",\\n \"prewhitening_type\": \"first difference\",\\n \"recoloring\": true,\\n \"reference_channels\": [\\n \"hx\",\\n \"hy\"\\n ],\\n \"regression.max_iterations\": 10,\\n \"regression.max_redescending_iterations\": 2,\\n \"regression.minimum_cycles\": 10,\\n \"regression.r0\": 1.5,\\n \"regression.tolerance\": 0.005,\\n \"regression.u0\": 2.8,\\n \"regression.verbosity\": 0,\\n \"save_fcs\": false,\\n \"window.clock_zero_type\": \"ignore\",\\n \"window.num_samples\": 128,\\n \"window.overlap\": 32,\\n \"window.type\": \"boxcar\"\\n }\\n }\\n ],\\n \"id\": \"test1-rr_test2_sr1\",\\n \"stations.local.id\": \"test1\",\\n \"stations.local.mth5_path\": \"/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5\",\\n \"stations.local.remote\": false,\\n \"stations.local.runs\": [\\n {\\n \"run\": {\\n \"id\": \"001\",\\n \"input_channels\": [\\n {\\n \"channel\": {\\n \"id\": \"hx\",\\n \"scale_factor\": 1.0\\n }\\n },\\n {\\n \"channel\": {\\n \"id\": \"hy\",\\n \"scale_factor\": 1.0\\n }\\n }\\n ],\\n \"output_channels\": [\\n {\\n \"channel\": {\\n \"id\": \"ex\",\\n \"scale_factor\": 1.0\\n }\\n },\\n {\\n \"channel\": {\\n \"id\": \"ey\",\\n \"scale_factor\": 1.0\\n }\\n },\\n {\\n \"channel\": {\\n \"id\": \"hz\",\\n \"scale_factor\": 1.0\\n }\\n }\\n ],\\n \"sample_rate\": 1.0,\\n \"time_periods\": [\\n {\\n \"time_period\": {\\n \"end\": \"1980-01-01T11:06:39+00:00\",\\n \"start\": \"1980-01-01T00:00:00+00:00\"\\n }\\n }\\n ]\\n }\\n }\\n ],\\n \"stations.remote\": [\\n {\\n \"station\": {\\n \"id\": \"test2\",\\n \"mth5_path\": \"/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5\",\\n \"remote\": true,\\n \"runs\": [\\n {\\n \"run\": {\\n \"id\": \"001\",\\n \"input_channels\": [\\n {\\n \"channel\": {\\n \"id\": \"hx\",\\n \"scale_factor\": 1.0\\n }\\n },\\n {\\n \"channel\": {\\n \"id\": \"hy\",\\n \"scale_factor\": 1.0\\n }\\n }\\n ],\\n \"output_channels\": [\\n {\\n \"channel\": {\\n \"id\": \"ex\",\\n \"scale_factor\": 1.0\\n }\\n },\\n {\\n \"channel\": {\\n \"id\": \"ey\",\\n \"scale_factor\": 1.0\\n }\\n },\\n {\\n \"channel\": {\\n \"id\": \"hz\",\\n \"scale_factor\": 1.0\\n }\\n }\\n ],\\n \"sample_rate\": 1.0,\\n \"time_periods\": [\\n {\\n \"time_period\": {\\n \"end\": \"1980-01-01T11:06:39+00:00\",\\n \"start\": \"1980-01-01T00:00:00+00:00\"\\n }\\n }\\n ]\\n }\\n }\\n ]\\n }\\n }\\n ]\\n }\\n}'" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "json_string" - ] - }, - { - "cell_type": "markdown", - "id": "0a55ea07-9ad5-4e20-a5ef-9944b5127e6c", - "metadata": {}, - "source": [ - "Which can be saved:" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "dbd8c6dd-cd94-43e0-bf64-9a2d26aa0f76", - "metadata": {}, - "outputs": [], - "source": [ - "with open(\"config.json\", \"w\") as fid:\n", - " fid.write(json_string)\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "id": "a511491b-1f6e-4b19-866f-c749dcb5435a", - "metadata": {}, - "source": [ - "### Default Parameters\n", - "\n", - "The default config parameters are listed below:" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "8292dd7b-08f8-401f-af4e-f3712f4a4d1b", - "metadata": {}, - "outputs": [], - "source": [ - "input_channels = [\"hx\", \"hy\"],\n", - "output_channels = [\"hz\", \"ex\", \"ey\"],\n", - "estimator = None,\n", - "emtf_band_file = BANDS_DEFAULT_FILE," - ] - }, - { - "cell_type": "markdown", - "id": "b0cf2e44-4f42-4988-8eb4-10db0e059de8", - "metadata": { - "tags": [] - }, - "source": [ - "# What can user change with the Config?" - ] - }, - { - "cell_type": "markdown", - "id": "9a7d6eb5-afce-4a04-91a2-4b26e077b6bd", - "metadata": { - "tags": [] - }, - "source": [ - "- Channel Nomenclature\n", - " - for example ex,ey maybe called e1, e2 in the mth5\n", - " - This is handled by passing a channel_nomenclature keyword argument.\n", - " - Examples of systems that might need this are LEMI and Phoenix\n", - "- Windowing Parameters\n", - " - Window shape (family)\n", - " - Window length\n", - " - Sliding window overlap\n", - " - Clock-Zero (optional)\n", - "- Choice of Stations\n", - " - This is currently done via KernelDataset\n", - "- Scale Factors for Individual Channels\n", - " - allows for correcting, or trying to add simple, frequency independent gain corrections\n", - "- Frequency Bands \n", - " - Group the Fourier coefficients into bands to be processed together and averaged for a TF estimate\n", - " - currently via an emtf-style band_setup file\n", - "- Number of Decimation Levels\n", - " - currently via an emtf-style band_setup file\n", - "- Regression Estimator Engine\n", - " - Currently only choices are RME, RME_RR (Regression M-estimate) and Remote reference regresson M-estimate\n", - "- Regression Parameters\n", - " - Maximum number of iterations\n", - " - Maximum number of redecending iterations\n", - " - Minimum number of frequecny cycles\n" - ] - }, - { - "cell_type": "markdown", - "id": "a36eb382-e23c-40d4-aff3-8fbb214e617b", - "metadata": { - "tags": [] - }, - "source": [ - "## EMTF Band Setup File" - ] - }, - { - "cell_type": "markdown", - "id": "9fb3ae51-2d71-41aa-95de-6dc618584d00", - "metadata": { - "tags": [] - }, - "source": [ - "The frequency bands will eventually be setup in a variety of ways, but currently aurora supports only specification of bands either by explicit construction or via EMTF \"band setup\" files. " - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "5f666cfb-4128-494b-bc21-fba2845afd93", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "PosixPath('/home/kkappler/software/irismt/aurora/aurora/config/emtf_band_setup/bs_test.cfg')" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "BANDS_DEFAULT_FILE" - ] - }, - { - "cell_type": "markdown", - "id": "f45d1579-d3b3-41c7-a4c1-a6fee19ce617", - "metadata": {}, - "source": [ - "Here is the content of a typical EMTF band setup file:" - ] - }, - { - "cell_type": "raw", - "id": "aa41ccd5-c154-45f3-b41f-3174cf5e7093", - "metadata": {}, - "source": [ - "25\n", - "1 25 30\n", - "1 20 24\n", - "1 16 19\n", - "1 13 15\n", - "1 10 12\n", - "1 8 9\n", - "1 6 7\n", - "1 5 5\n", - "2 14 17\n", - "2 11 13\n", - "2 9 10\n", - "2 7 8\n", - "2 6 6\n", - "2 5 5\n", - "3 14 17\n", - "3 11 13\n", - "3 9 10\n", - "3 7 8\n", - "3 6 6\n", - "3 5 5\n", - "4 18 22\n", - "4 14 17\n", - "4 10 13\n", - "4 7 9\n", - "4 5 6" - ] - }, - { - "cell_type": "markdown", - "id": "99ca1338-cbf5-49c8-b5ca-5a1a92d4d22e", - "metadata": {}, - "source": [ - "These legacy files have the following significance;\n", - "The first line, 25 indicates the number of bands, and there are 25 lines following, one line per frequency band.\n", - "\n", - "Each line comprises three numbers:\n", - "\n", - "`decimation_level, first_FC_index, last_FC_index`\n", - "\n", - "where \"FC\" stands for Fourier coefficient" - ] - }, - { - "cell_type": "markdown", - "id": "03f08493-956f-4b3a-81d5-b8739edfbd46", - "metadata": {}, - "source": [ - "The decimation factor applied at each level was controlled in EMTF by a sepearate file, called `decset.cfg`. In the old EMTF codes, this controlled the window length, overlap, the decimation factor, and the corners of the anti-alias filter applied before downsampling.\n", - "\n", - "The decimation factor in EMTF was almost always 4, and the default behaviour of the ConfigCreator is to assume a decimation factor of 4 at each level, but this can be changed manually. " - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "aurora-test", - "language": "python", - "name": "aurora-test" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -}