Solved issue - For all people that want to try this tutorial as it was uploaded in 2021.

[Darius-Puzzler]

For all people that want to try this tutorial as it was uploaded in 2021.
Here is the way back maschine:

Windows 10 Version:

Create a Python Virtual Environment with Python 3.10.1 with CMD.exe.
-> install Python 3.10.1
-> "py -3.10 -m venv myenv"
-> "myenv\Scripts\activate"
-> to deactivate it simply type "deactivate" at cmd.exe prompt.

Activate your python virtual environment for Python 3.10.1.
The cmd prompt will look differently.
You can check all installed packages by typing "pip freeze".
(or use the specific command for conda etc.)
You will see, it is empty.
Now install the packages with the stated version numbers listed below.
I suggest installing stable-baselines3[extra] last.
After each installation of a package, check the below listed
packages for their version again!
For example, if you install stable-baselines3[extra] last, it will automatically
"update" your gymnasium or numpy etc. to a higher version which will cause issues!

In this case uninstall the unwantedly updated package and reinstall the version listed below again.

Python 3.10.1
gym==0.23.0
gymnasium==0.26.1
gym-super-mario-bros==7.3.0
nes_py==8.2.1
numpy==1.26.4
Shimmy==0.1.0