NOOS image build tools ?

[jeanparpaillon]

Hi,
I would like to integrate openplotter tools in another distribution. Are the tools/scripts for building NOOBS image available ?

[sailoog]

Hi,
We do not have a script to build openplotter, lot of things have to de done manually this is the needed packages:

sudo apt-get install gettext gpsd gpsd-clients python-w1thermsensor python-wxgtk3.0 hostapd dnsmasq mpg123 python-gammu gammu mosquitto crudini whois libusb-1.0-0-dev libfftw3-dev qt5-qmake libasound2-dev libpulse-dev autoconf automake python-dev python-matplotlib bridge-utils libqt4-dev gnuradio libvolk1-bin libusb-1.0-0 gr-iqbal qt5-default libqt5svg5 libportaudio2 make gcc xsltproc curl git build-essential libtool librtlsdr-dev rtl-sdr i2c-tools cmake zygrib libnss-mdns avahi-utils libavahi-compat-libdnssd-dev swig python-numpy python-scipy python-serial python-gps python-pil python-opengl python-flask bsdtar

and python packages:
sudo pip install --upgrade paho-mqtt pyudev pyrtlsdr pynmea2 twython websocket-client spidev requests requests_oauthlib PyMata pyglet pywavefront ujson Flask flask-socketio

Then you need to install signal k and node-red.

Compile these packages:

cd $HOME
mkdir compiling

cd $HOME/compiling
git clone https://github.com/sailoog/kalibrate-rtl
cd kalibrate-rtl
./bootstrap && CXXFLAGS='-W -Wall -O3'
./configure
make
sudo make install

cd $HOME/compiling
git clone https://github.com/sailoog/rtl_433
cd rtl_433/
mkdir build
cd build
cmake ../
make
sudo make install

cd $HOME/compiling
pkill aisdecoder
git clone https://github.com/sailoog/aisdecoder
cd aisdecoder
cmake -DCMAKE_BUILD_TYPE=release
make
sudo cp aisdecoder /usr/local/bin

cd $HOME/compiling
git clone https://github.com/sailoog/kplex
cd kplex
make
sudo make install

cd $HOME/compiling
git clone https://github.com/sailoog/canboat
cd canboat
make
sudo make install

cd $HOME/compiling
git clone https://github.com/sailoog/geomag
cd geomag/geomag
python setup.py build
sudo python setup.py install

cd $HOME/compiling
git clone https://github.com/sailoog/RTIMULib2
cd RTIMULib2/Linux
mkdir build
cd build
cmake ..
make -j4
sudo make install
sudo ldconfig
cd ..
cd RTIMULibDrive
make -j4
sudo make install
cd ..
cd RTIMULibDrive10
make -j4
sudo make install
cd ..
cd python
python setup.py build
sudo python setup.py install

cd $HOME/compiling
git clone https://github.com/sailoog/pypilot
git clone https://github.com/sailoog/pypilot_data
cp -rv pypilot_data/* pypilot
cd pypilot
python setup.py build
sudo python setup.py install

if [ ! -d ~/.pypilot ]; then
	mkdir ~/.pypilot
fi

echo '{"host": "localhost"}' > ~/.pypilot/signalk.conf

cd
sudo rm -rf $HOME/compiling/

and openplotter:

cd
cd .config
rm -rf openplotter/
git clone https://github.com/sailoog/openplotter
cd openplotter
git checkout beta
sudo chmod 775 /home/pi/.config/openplotter/openplotter
sudo chmod 775 /home/pi/.config/openplotter/keyword
sudo chmod 775 /home/pi/.config/openplotter/startup
sudo bash -c 'echo -e "\nexport PATH=$PATH:/home/pi/.config/openplotter\n" >> /home/pi/.profile'

The rest are dozens of notes, sorry.

[tkurki]

Take a look at https://github.com/RPi-Distro/pi-gen - on a quick read everything you listed looks automatable to me.

[jeanparpaillon]

Thank you very much ! Let's see how we can automate this

[FredericGuilbault]

Ill give it a try.
From what I reading on pi-gen, openplotter stuff should go on stage 2 and stage 3. stage4 & 5 should be skipped as openplotter seem to remove some of the raspbian default packages. like thunderbird, open-office and others.

[sailoog]

I went into pi-gen some time ago but I realized that it was not possible compile github packages or install them from pip. If I am not wrong this tool just install arm compiled packages from raspbian repository and edit files to config the final system. I we want to include all the required packages we should precompile and install them "manually" usin pi-gen tool, same for pip packages. Too much work. Am I misunderstanding anything?

We only generate a new image when Raspbian is upgraded, in this case from jessie to strech, perhaps once a year/2 years. We use separate sh scripts and notes to make some changes manually. It is not an automatized process but It takes only 2 hours in getting an img file. Then we convert this image in a NOOBS installation following the official steps. It would be great to have all this in just one step with pi-gen but if we have to edit pi-gen stages to manually install all required and not compiled packages I do not think we win something. It could be better gathering all sh scripts in just one and automatize the manual steps in that unique sh script. We could publish that script to make easier porting to different systems and allow people to improve the procedure.

I could have misunderstood pi-gen I will give it a try again this afternoon. If you have any tip I am listening. thanks.

[FredericGuilbault]

Here in stage 3
https://github.com/FredericGuilbault/pi-gen/blob/master/stage3/00-install-packages/01-run.sh
it seem to mount the target OS and chroot, so pretty much anything can be done at this point.

So Im expecting to be able to do something like this :
https://github.com/FredericGuilbault/pi-gen/blob/OpenPlotter/stage3/02-openplotter-install-packages/01-run.sh

But im not reaching stage 3 yet....

[sailoog]

Ok, let me know if it works please!

[sailoog]

After some test it seems that this could work. I will report...

[FredericGuilbault]

I wasen't able to build a functional image cuz pi-gen was not on root partition...
Once this problem fixed I have been able to make 2 test, On both case logs have shown a successful run of my custom changes (add of node reop + node install + signalk-server install via npm). But in both case, when I have run the resulting image on my rasp. My custom changes wasen't there :(.

pi-gen also seem to take some parameters form the host (Locale, /dev/, /proc/ ) OS. Witch make the build : Host dependent...

So that's what ill remember from my experience with pi-gen. This said, it would probably worth debugging why customs changes appear in the build log and not in the ISO but I don't have enough time to do it.

[tkurki]

Could you share how u tried it? If somebody else continues with this. Just the description how it failed will not help much if at all.

[sailoog]

Look at my fork: https://github.com/sailoog/openplotter-gen
It generates full working NOOBS and img files. I have not added all the scripts and manual steps yet but all I have tested works. Opencpn installation, pip packages...
Packages compilation and signal k installation not tested yet.
I get some warnings about locales but do not affect the final images.

[FredericGuilbault]

It was just a test of pi-gen, I did not tried to build OpenPlotter so there is not much I did,

My repo is here :
https://github.com/FredericGuilbault/pi-gen/tree/OpenPlotter

My custom changes are here :
https://github.com/FredericGuilbault/pi-gen/commit/3094159c5a0233aa5e4ba39a5fdfb128ecaed4aa

And then I followed instructions found on pi-gen README witch is pretty much just : sudo ./build.sh

Once it's done (over 2Hours ) The image can be sound somewhere in ./work/

[FredericGuilbault]

@sailoog

Comiting on the master branch of pi-gen, even if it's a fork. It will be problematic when pi-gen will push new commits and you will want to integrate them ( as rebase or as merge )

You could create a branch for OpenPlotter-master where upstream and features could be merged/rebased.

An other way would be to have your own repo of pi-gen and pull from upstream in a side branch , then merge them into your master. But IDK if github allow this workflow of git.

...Other approch might exist.

[sailoog]

Yes, this was just a try, I am even editing directly on github :)
after testing critical parts I will think about the best way to maintain the repo to allow people to push.

[FredericGuilbault]

if @jeanparpaillon agree, I think we can close this issue.

[tkurki]

Is this done & available someplace?

[sailoog]

Not yet. I would keep thi open until we have something. I will report here.