Skip to content

Commit

Permalink
Exercise for spectrograms
Browse files Browse the repository at this point in the history
  • Loading branch information
khliland committed Oct 23, 2024
1 parent 3cc4a58 commit 3d18913
Show file tree
Hide file tree
Showing 185 changed files with 19,708 additions and 5,928 deletions.
1,576 changes: 803 additions & 773 deletions D2Dbook/3_Data_sources/3_APIs/downloads/forecast.json

Large diffs are not rendered by default.

32 changes: 16 additions & 16 deletions D2Dbook/3_Data_sources/3_APIs/downloads/weather.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,36 +8,36 @@
"id": 804,
"main": "Clouds",
"description": "overcast clouds",
"icon": "04d"
"icon": "04n"
}
],
"base": "stations",
"main": {
"temp": 276.3,
"feels_like": 275.22,
"temp_min": 275.5,
"temp_max": 277.64,
"pressure": 1011,
"humidity": 100,
"sea_level": 1011,
"grnd_level": 994
"temp": 283.55,
"feels_like": 282.94,
"temp_min": 282.16,
"temp_max": 284.53,
"pressure": 1021,
"humidity": 88,
"sea_level": 1021,
"grnd_level": 1004
},
"visibility": 10000,
"wind": {
"speed": 1.34,
"deg": 357,
"gust": 2.51
"speed": 4.36,
"deg": 210,
"gust": 11
},
"clouds": {
"all": 95
"all": 100
},
"dt": 1728888498,
"dt": 1729708149,
"sys": {
"type": 2,
"id": 2006772,
"country": "NO",
"sunrise": 1728885203,
"sunset": 1728922287
"sunrise": 1729664132,
"sunset": 1729698363
},
"timezone": 7200,
"id": 3139081,
Expand Down
954 changes: 48 additions & 906 deletions D2Dbook/4_Data_quality/1_Signal/3_Decomposition.ipynb

Large diffs are not rendered by default.

53 changes: 47 additions & 6 deletions D2Dbook/4_Data_quality/2_Outliers/1_Outliers_and_anomalies.ipynb

Large diffs are not rendered by default.

277 changes: 235 additions & 42 deletions D2Dbook/4_Data_quality/2_Outliers/2_Outlier_statistics.ipynb

Large diffs are not rendered by default.

3,214 changes: 3,185 additions & 29 deletions D2Dbook/4_Data_quality/2_Outliers/3_Anomaly_detection.ipynb

Large diffs are not rendered by default.

2,182 changes: 2,101 additions & 81 deletions D2Dbook/4_Data_quality/3_Preprocessing/2_Imputation.ipynb

Large diffs are not rendered by default.

46 changes: 27 additions & 19 deletions D2Dbook/4_Data_quality/3_Preprocessing/3_Time_formats.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,23 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2024-08-17T11:09:26.933980Z",
"iopub.status.busy": "2024-08-17T11:09:26.933628Z",
"iopub.status.idle": "2024-08-17T11:09:26.940994Z",
"shell.execute_reply": "2024-08-17T11:09:26.940411Z"
},
"slideshow": {
"slide_type": "slide"
}
},
"tags": []
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Date and time is: 2024-10-21 11:54:01.076687\n",
"Timestamp is: 1729504441.076687\n"
]
}
],
"source": [
"from datetime import datetime\n",
"\n",
Expand Down Expand Up @@ -98,19 +102,23 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2024-08-17T11:09:26.974670Z",
"iopub.status.busy": "2024-08-17T11:09:26.974535Z",
"iopub.status.idle": "2024-08-17T11:09:26.977022Z",
"shell.execute_reply": "2024-08-17T11:09:26.976746Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"tags": []
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"An arbitrary format: 21/10-2024 11:57:40\n",
"Reformatted to the standard: 2024-10-21 11:57:40\n"
]
}
],
"source": [
"now = datetime.now()\n",
"formatted = now.strftime(\"%d/%m-%Y %H:%M:%S\")\n",
Expand Down Expand Up @@ -175,5 +183,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@
" - This coresponds to the different uses of filters in the [Noise reduction section](../1_Signal/2_Noise_reduction.ipynb).\n",
" - ... which also means that it is possible to echange simple averages with other smoothers.\n",
"\n",
"<img src=\"../../images/Accumulation.png\" width=\"700px\">"
"<img src=\"https://github.com/khliland/IND320/blob/main/D2Dbook/images/Accumulation.png?raw=TRUE\" width=\"700px\" />"
]
},
{
Expand Down
21 changes: 16 additions & 5 deletions D2Dbook/4_Data_quality/3_Preprocessing/5_Spectrograms.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,8 @@
"outputs": [],
"source": [
"# Produce a spectrogram using scipy.signal.stft (short-time Fourier transform)\n",
"from scipy.signal import stft\n",
"f_stft, t_stft, Zxx = stft(y_approx, fs, nperseg=576)"
"from scipy.signal import stft # Legacy function stft. Anyone want to update this to ShortTimeFFT?\n",
"f_stft, t_stft, Zxx = stft(y_approx, fs, nperseg=1152)"
]
},
{
Expand Down Expand Up @@ -127,7 +127,7 @@
"source": [
"# Plot the spectrogram without pre-computed frequency and time axes\n",
"import matplotlib.pyplot as plt\n",
"plt.specgram(y_approx, Fs=fs, NFFT=576, noverlap=400, scale='linear')\n",
"plt.specgram(y_approx, Fs=fs, NFFT=1152, noverlap=400, scale='linear')\n",
"plt.xlabel('Time (s)')\n",
"plt.ylabel('Frequency (Hz)')\n",
"plt.colorbar()\n",
Expand Down Expand Up @@ -186,7 +186,7 @@
"outputs": [],
"source": [
"# Produce a spectrogram using scipy.signal.stft (short-time Fourier transform)\n",
"f, t, Zxx = stft(samples, song.frame_rate, nperseg=576)"
"f, t, Zxx = stft(samples, song.frame_rate, nperseg=1152)"
]
},
{
Expand Down Expand Up @@ -261,6 +261,17 @@
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Exercise\n",
"- Revisit the sunspot data from the STL decomposition.\n",
"- Create a spectrogram.\n",
"- See how changing parameters alters the output.\n",
"- Try to get an intuition for the role of the sampling rate (test values)."
]
},
{
"cell_type": "markdown",
"metadata": {
Expand All @@ -280,7 +291,7 @@
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"display_name": "IND320_2024",
"language": "python",
"name": "python3"
},
Expand Down
Binary file not shown.
Binary file not shown.
Binary file modified D2Dbook/_build/.doctrees/3_Data_sources/2_Databases/4_Spark.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified D2Dbook/_build/.doctrees/environment.pickle
Binary file not shown.

Large diffs are not rendered by default.

40 changes: 20 additions & 20 deletions D2Dbook/_build/html/3_Data_sources/2_Databases/3_Cassandra.html

Large diffs are not rendered by default.

Loading

0 comments on commit 3d18913

Please sign in to comment.