diff --git a/src/content/blog/announcements/2021-06-02-applications-closed.md b/src/content/blog/announcements/2021-06-02-applications-closed.md new file mode 100644 index 0000000000..f5e0ee13e2 --- /dev/null +++ b/src/content/blog/announcements/2021-06-02-applications-closed.md @@ -0,0 +1,10 @@ +--- +title: Applications are closed +date: 2021-06-02 +--- + +Thanks a lot for your interest in the Summer of Nix program! +We've done more than a hundred interviews in two months, it's been thrilling +to see so much excitement over this program. + +We are currently in the process of selecting participants and making the teams: stay tuned! diff --git a/src/content/blog/announcements/2021-07-20-hiring-event.md b/src/content/blog/announcements/2021-07-20-hiring-event.md new file mode 100644 index 0000000000..eaaf8aabe7 --- /dev/null +++ b/src/content/blog/announcements/2021-07-20-hiring-event.md @@ -0,0 +1,10 @@ +--- +title: Hiring event on October 27th 2021 +date: 2021-07-20 +--- + +Rumor was already out but now we announce it officially: we are organizing a +big hiring event on October 27th where Summer of Nix participants and +applicants can meet companies that use it. Participation is free but places are +limited for organizational reasons to Summer of Nix participants and applicants +for now. If you like to know more, send us a mail at summer@nixos.org. diff --git a/src/content/blog/announcements/2022-02-23-summer-of-nix-2021-report.md b/src/content/blog/announcements/2022-02-23-summer-of-nix-2021-report.md new file mode 100644 index 0000000000..2cb1e0384d --- /dev/null +++ b/src/content/blog/announcements/2022-02-23-summer-of-nix-2021-report.md @@ -0,0 +1,9 @@ +--- +title: Summer of Nix 2021 report is out +date: 2021-02-23 +--- + +Happy to announce that the [Summer of Nix 2021 report is +out](/assets/report-2021.pdf). This report explains in detail the motivation +behind Summer of Nix, and how it was organized in 2022. If you have comments or +questions, don't hesitate to reach out to summer@nixos.org. diff --git a/src/content/blog/announcements/2022-04-04-summer-of-nix-2022.md b/src/content/blog/announcements/2022-04-04-summer-of-nix-2022.md new file mode 100644 index 0000000000..4984147a3f --- /dev/null +++ b/src/content/blog/announcements/2022-04-04-summer-of-nix-2022.md @@ -0,0 +1,70 @@ +--- +title: Summer of Nix 2022 Announcement! +date: 2022-04-04 +--- + +Summer of Nix 2022 will happen **from July 18th to September 30th**! + +[Applications are open here from now until May 15th!](https://cryptpad.fr/form/#/2/form/view/HyAQLTXDtHy0JNGvRIwWXa6Swmxb31LuWsOoPRcmD-8/). + +Following the excitement of Summer of Nix 2021 (more in [last year’s +report](https://summer.nixos.org/assets/report-2021.pdf), I am happy to +announce that the European Commission, NLNet and the NixOS Foundation have +granted enough budget to organize another community event this year. After some +brainstorming sessions I am excited to share how we (an organization committee +of ~8 people now) would like to organize this event. + +Summer of Nix is **a paid summer program** where you can **learn, meet, and +work** with the Nix community. It targets experienced Nixers and Nix newcomers +alike to come together and work on a range of different topics. At the same +time, there will be talks about Nix and presentations by Nix-using companies +with the goal of fostering a strong sense of community and pushing the +ecosystem forward together. + +What’s new this year is that we offer … + +- a **communication stream** to write quality documentation, deployment + stories, tutorials, and produce video content and podcast episodes; +- an **ecosystem stream** to work directly on Nix tooling that improves the + overall packaging and package user experience; +- a **packaging stream** to package new free and open source software either + within nixpkgs or as independent flakes. + +All streams focus on improving Nix support for the **free and open source +(FOSS) ecosystem** from different angles, as last year with emphasis on the +packages sponsored by the European Commission via the NGI initiative and +NLNet. + +You can participate in Summer of Nix in several ways: + +- **As a participant** you will work full time on various projects together + with other participants within the 3 work streams for about 8 weeks (with + some flexibility, e.g. to take vacation). This is the main role in Summer of + Nix and great for those who work in a team to share and improve their Nix + skills, meet nice people while contributing to the community. +- **As a team coordinator** you will lead a small team, arrange communication + and mentoring, and guide technical discussions and decisions. You will also + be involved in the overall organization of the program with other + coordinators. Technical work in this role includes reviewing PRs and + answering technical questions. Hours can vary, but on average the time + commitment is about 12 hours a week. Experienced Nix tinkerers with a social + knack are encouraged to apply for this role. If you’re interested in Nix and + also enjoy team organization, this role is for you! +- **As a tech wiz** you will pair program with participants, answer their + questions, review their PRs, and show them how to get along with Nix. This + is a great way to be involved if you are an experienced Nixer and enjoy + helping others in a flexible way. +- **As a project lead** you offer a project within the three work streams. You + would work together with participants on this project, and provide them with + guidance and technical expertise. This is great for anyone who is looking + for support on their own Nix project. If you have an idea, please apply, but + understand that projects have to fall within the three work streams and we + will need to evaluate whether they can fit within the scope of the European + Commission’s grant. + +The compensation depends a bit on each role but will roughly be oriented on +Google’s Summer of Code stipend for the participant roles. + +Please write to summer@nixos.org if you have questions. In addition to this, we +will also have **a separate sponsor track** for companies who want to connect +with the community in this program this year. But more about this later… diff --git a/src/content/blog/stories/PerlDivingWithNix.md b/src/content/blog/stories/PerlDivingWithNix.md new file mode 100644 index 0000000000..9534f006cf --- /dev/null +++ b/src/content/blog/stories/PerlDivingWithNix.md @@ -0,0 +1,514 @@ +--- +title: Perl Diving with Nix +date: 2021-10-01 # estimated +extra: + author: Thomas Sean Kelly +--- + +## Introduction + +This is a record of my time in the Summer of Nix holding my breath and diving into the depths of Nix to gain some Perls of wisdom. + +Going through the issues assigned to my team I came across the the package for [OpenFoodFacts](https://world.openfoodfacts.org/discover) a collaborative database that collects and provides open data on 1 million food products from around the world and counting! + +Not knowing what we are eating seems like a strange thing. +But really, how much do we know what goes into those vacuum packed bags that we open and consume daily? + +OpenFoodFacts, I see your [quote](https://world.openfoodfacts.org/discover) and raise you a Newman! + +> We are living in a world today where lemonade is made from artificial flavors and furniture polish is made from real lemons. +> +> — Alfred E. Newman + +Anyway, enough preamble… + +This project represents a deviation from the common usage of Nix to create [derivations](https://nixos.org/manual/nix/unstable/expressions/derivations.html) that package binaries. +Instead, its main application [Product Opener](https://github.com/openfoodfacts/openfoodfacts-server) is a large app that requires many languages and dependencies which end up as a web service. + +The back-end is served using Apache with the Perl Module [`mod_perl`](https://en.wikipedia.org/wiki/Mod_perl) enabled (modules being how Apache is extended). +This embeds a Perl interpreter into the Apache server, which handles the production of dynamic content. + +As you can imagine there is a lot of Perl which is dynamically loaded by the Apache config file. + +A word to the wise: there is a lot that goes into an Apache config file and the wonder of Nix is that it abstracts most of it away including the loading of the above-mentioned [Apache modules](https://www.ibm.com/docs/en/i/7.2?topic=functionality-apache-modules). +Unless you really know what you're doing, it's best to use the `extraConfig` option, which appends configuration to the Nix auto-generated one, as it has many twists and turns. +It is possible to override the configuration file completely with the `configFile` option, but make sure you understand the auto-generated one first. + +To have a look for your self run + +```console +nix build nixpkgs#apacheHttpd && cd result/conf +``` + +A minimal ApacheHttpd configuration in Nix: + +```nix +services.httpd = { + enable = true; + enablePerl = true; + adminAddr = "test@test.com"; + extraConfig = builtins.readFile "${src}/apache.conf"; +}; +``` + +The `enablePerl` option is what loads up `mod_perl` for us. :hands-up: + +Slight detour into Perl: [CPAN][cpan] stands for **Comprehensive Perl Archive Network** and serves as the central repository for Perl modules. One way of defining the dependencies of a Perl project is writing a plain text file called `cpanfile` that requires all the modules to be loaded from CPAN. + +[cpan]: https://www.cpan.org/ + +So before even looking into the code contained in the project my first step was looking at all the Perl dependencies that were cataloged in the `cpanfile` – of which there were 65. + +``` +requires 'CGI', '>= 4.51, < 5.0'; # libcgi-pm-perl +requires 'Tie::IxHash'; # libtie-ixhash-perl +requires 'LWP::Authen::Digest'; # libwww-perl +requires 'LWP::Simple'; # libwww-perl +... +``` + +In the Nix community there is a ever growing list of `*2nix` tools that _translates_ one package manager's lock files into something that fits within the Nix ecosystem of immutable declarative packaging. +It was at this point that I started wondering if there were any tools like that for Perl. + +The short answer is no. +But there is some great support for adding something from [`meta::cpan`](https://metacpan.org/) (a repository containing what seems to be the entire Perl universe) to `nixpkgs`. +It takes the form of the function `nix-generate-from-cpan`, which is also exposed as a utility in `nixpkgs`. + +```console +nix run nixpkgs#nix-generate-from-cpan +``` + +for the win! + +So a few `nix run nixpkgs#nix-generate-from-cpan `'s later I had 40 odd shiny new Perl packages. :) + +This utility was great, but perhaps the reason there isn't a `*2nix` tool for the Perl world is that it is not fool proof. +In my experience it _just worked™_ 70% of the time, with the other 30% of the time needing minor fixes: either adding packages to `propagatedBuildInputs` or running `nix-generate-from-cpan` for a Perl package that was needed as a dependency. + +This might be an issue with the Perl ecosystem itself, as a `cpanfile` is not a lock file. +The above example demonstrates it can be very vague (`>=4.51, < 5.0`), and vagueness is the enemy of reproducible and thus the enemy of Nix. + +However in one case it failed drastically, and truth be told I was stuck on which way to go for a few days. + +The intransigent bugger was [`Barcode::Zbar`](https://metacpan.org/release/SPADIX/Barcode-ZBar-0.04/view/ZBar.pm), a module that provides a Perl interface to the [ZBar Barcode Reader](https://github.com/mchehab/zbar). +OpenFoodFacts has the rather excellent feature where you can just scan a bar code as a discovery mechanism. + +The aforementioned `nix-generate-from-cpan` kindly provided + +```nix + BarcodeZBar = buildPerlPackage { + pname = "Barcode-ZBar"; + version = "0.04"; + src = fetchurl { + url = "mirror://cpan/authors/id/S/SP/SPADIX/Barcode-ZBar-0.04.tar.gz"; + sha256 = "d57e1ad471b6a29fa4134650e6eec9eb834d42cbe8bf8f0608c67d6dd0f8f431"; + }; + meta = { + }; + }; +``` + +Hmmm, rather bare. It didn't even include the Zbar binary as part of its `propagatedBuildInputs`. + +**Side note**: `propagatedBuildInputs` here means anything that is a run-time dependency, whereas `buildInputs` are for dependencies that are exclusively build-time dependencies (e.g. tests and `Makefile` generators) - see [Perl Packaging in Nix](https://nixos.org/manual/nixpkgs/stable/#ssec-perl-packaging) + +After fixing the inputs it was time to give it a try. + +```nix +buildInputs = [ TestMore ExtUtilsMakeMaker ]; +propagatedBuildInputs = [ zbar PerlMagick ]; +``` + +Where upon it failed with the message: + +``` +perl5.34.0-Barcode-ZBar> ZBar.xs: In function 'XS_Barcode__ZBar_version': +perl5.34.0-Barcode-ZBar> ZBar.xs:202:9: error: too few arguments to function 'zbar_version' +``` + +On investigation it seems that this module was last updated in 2009, and since then the `zbar` project has been forked and continued developing. +The ZBar project now uses [semantic versioning][semver] while the module is stuck with `major.minor` versioning. + +[semver]: https://semver.org/ + +`nixpkgs` topping repology's list for [Projects up to date](https://repology.org/repositories/statistics/newest) obviously meant that it wasn't slouching when it came to `zbar` and had zoomed ahead of the Perl module to version `0.23.90`. (It was never clear what version the module was expecting, but 2009 put it at either `0.9` or `0.10`). + +It appeared to me that I had two options: + +1. Naively patch this function so it takes 3 arguments in the Perl module and hope that works. +2. More realistically, create an overlay for `zbar` for a version that was compatible with the Perl module. + +## Option 1 - The Patch + +Summer of Nix is all about learning, so I figured it was worth a shot. +After a quick watch of the excellent [How to create a patch for any package](https://www.youtube.com/watch?v=5K_2RSjbdXc) by Jon Ringer (go check it out) and a quick fiddle in `git` I had a patch file. + +``` +From e51b51a77eab1251babc58929a4d2107172a041f Mon Sep 17 00:00:00 2001 +From: Thomas Sean Dominic Kelly +Date: Fri, 6 Aug 2021 12:35:06 +0100 +Subject: [PATCH] version patch + +--- + ZBar.xs | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +diff --git a/ZBar.xs b/ZBar.xs +index ad6fc56..97bd2c0 100644 +--- a/ZBar.xs ++++ b/ZBar.xs +@@ -198,9 +198,10 @@ zbar_version() + PREINIT: + unsigned major; + unsigned minor; ++ unsigned patch; + CODE: +- zbar_version(&major, &minor); +- RETVAL = newSVpvf("%u.%u", major, minor); ++ zbar_version(&major, &minor, &patch); ++ RETVAL = newSVpvf("%u.%u.%u", major, minor, patch); + OUTPUT: + RETVAL + +-- +2.32.0 +``` + +Applying patches in Nix is the simplest thing in the world: +just add it to the [patch phase](https://nixos.org/manual/nixpkgs/stable/#ssec-patch-phase) and your golden. + +The manual is a bit sparse in this regard. +The code looks something like: + +```nix +patchPhase = [ ./version.patch ]; +``` + +This is possible as `buildPerlPackage` is built on top of [`stdenv`](https://nixos.org/manual/nixpkgs/stable/#idm140737320528768), allowing us to customise everything in the usual `nixpkgs` way. + +Aha – something is happening! It seems to successfully compile, but fails the tests. + +The full logs are below, but tl;dr the salient line seems to be: + +``` +#Error: Can't load '/build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so' for module Barcode::ZBar: /build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so: undefined symbol: zbar_scanner_reset at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/DynaLoader.pm line 193. +``` + +Ooof, `undefined symbol`… OK, so the module is looking for a symbol and not finding it in the `zbar` shared object. + +There goes my naivety. + +#### Logs + +
+ Click to see full logs + +``` +this derivation will be built: + /nix/store/gww59146rs399rjc3fnawrjng4pqf6dl-perl5.32.1-Barcode-ZBar-0.04.drv +building '/nix/store/gww59146rs399rjc3fnawrjng4pqf6dl-perl5.32.1-Barcode-ZBar-0.04.drv'... +unpacking sources +unpacking source archive /nix/store/g5kazmm00923w6rgcf5h6rzrlp7b1nhj-Barcode-ZBar-0.04.tar.gz +source root is Barcode-ZBar-0.04 +setting SOURCE_DATE_EPOCH to timestamp 1256327204 of file Barcode-ZBar-0.04/META.yml +patching sources +applying patch /nix/store/3ix6dz6lmifqrmbs24jbjh9z07wbscbi-0001-version-patch.patch +patching file ZBar.xs +configuring +patching ./examples/processor.pl... +patching ./examples/scan_image.pl... +patching ./examples/read_one.pl... +patching ./examples/paginate.pl... +Checking if your kit is complete... +Looks good +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 162. +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 166. +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 171. +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 173. +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 181. +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 183. +Use of uninitialized value $thispth in concatenation (.) or string at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/Liblist/Kid.pm line 187. +Warning (mostly harmless): No library found for -lzbar +Generating a Unix-style Makefile +Writing Makefile for Barcode::ZBar +Invalid LICENSE value 'lgpl' ignored +Writing MYMETA.yml and MYMETA.json +no configure script, doing nothing +building +build flags: SHELL=/nix/store/xvvgw9sb8wk6d2c0j3ybn7sll67s3s4z-bash-4.4-p23/bin/bash +cp ZBar.pm blib/lib/Barcode/ZBar.pm +cp ZBar/Processor.pod blib/lib/Barcode/ZBar/Processor.pod +cp ZBar/ImageScanner.pod blib/lib/Barcode/ZBar/ImageScanner.pod +cp ZBar/Image.pod blib/lib/Barcode/ZBar/Image.pod +cp ZBar/Symbol.pod blib/lib/Barcode/ZBar/Symbol.pod +Running Mkbootstrap for ZBar () +chmod 644 "ZBar.bs" +"/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- ZBar.bs blib/arch/auto/Barcode/ZBar/ZBar.bs 644 +"/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/bin/perl" "/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/xsubpp" -typemap '/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/ExtUtils/typemap' -typemap '/build/Barcode-ZBar-0.04/typemap' ZBar.xs > ZBar.xsc +mv ZBar.xsc ZBar.c +cc -c -D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/no-such-path/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -DVERSION=\"0.04\" -DXS_VERSION=\"0.04\" -fPIC "-I/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/CORE" ZBar.c +rm -f blib/arch/auto/Barcode/ZBar/ZBar.so +cc -shared -O2 -L/nix/store/gk42f59363p82rg2wv2mfy71jn5w4q4c-glibc-2.32-48/lib -fstack-protector-strong ZBar.o -o blib/arch/auto/Barcode/ZBar/ZBar.so \ + \ + +chmod 755 blib/arch/auto/Barcode/ZBar/ZBar.so +Manifying 5 pod documents +running tests +check flags: SHELL=/nix/store/xvvgw9sb8wk6d2c0j3ybn7sll67s3s4z-bash-4.4-p23/bin/bash VERBOSE=y test +"/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- ZBar.bs blib/arch/auto/Barcode/ZBar/ZBar.bs 644 +PERL_DL_NONLAZY=1 "/nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t +t/Decoder.t ....... 1/13 +# Failed test 'use Barcode::ZBar;' +# at t/Decoder.t line 10. +# Tried to use 'Barcode::ZBar'. +# Error: Can't load '/build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so' for module Barcode::ZBar: /build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so: undefined symbol: zbar_scanner_reset at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/DynaLoader.pm line 193. +# at t/Decoder.t line 10. +# Compilation failed in require at t/Decoder.t line 10. +# BEGIN failed--compilation aborted at t/Decoder.t line 10. +Bareword "Barcode::ZBar::Symbol::PARTIAL" not allowed while "strict subs" in use at t/Decoder.t line 48. +Bareword "Barcode::ZBar::Symbol::NONE" not allowed while "strict subs" in use at t/Decoder.t line 27. +Bareword "Barcode::ZBar::SPACE" not allowed while "strict subs" in use at t/Decoder.t line 57. +Bareword "Barcode::ZBar::Symbol::QRCODE" not allowed while "strict subs" in use at t/Decoder.t line 61. +Bareword "Barcode::ZBar::Config::ENABLE" not allowed while "strict subs" in use at t/Decoder.t line 61. +Bareword "Barcode::ZBar::Symbol::PARTIAL" not allowed while "strict subs" in use at t/Decoder.t line 69. +Bareword "Barcode::ZBar::Symbol::NONE" not allowed while "strict subs" in use at t/Decoder.t line 69. +Bareword "Barcode::ZBar::Symbol::EAN13" not allowed while "strict subs" in use at t/Decoder.t line 73. +Bareword "Barcode::ZBar::BAR" not allowed while "strict subs" in use at t/Decoder.t line 81. +Bareword "Barcode::ZBar::Symbol::EAN13" not allowed while "strict subs" in use at t/Decoder.t line 85. +Execution of t/Decoder.t aborted due to compilation errors. +# Looks like your test exited with 255 just after 1. +t/Decoder.t ....... Dubious, test returned 255 (wstat 65280, 0xff00) +Failed 13/13 subtests +t/Image.t ......... 1/22 +# Failed test 'use Barcode::ZBar;' +# at t/Image.t line 10. +# Tried to use 'Barcode::ZBar'. +# Error: Can't load '/build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so' for module Barcode::ZBar: /build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so: undefined symbol: zbar_scanner_reset at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/DynaLoader.pm line 193. +# at t/Image.t line 10. +# Compilation failed in require at t/Image.t line 10. +# BEGIN failed--compilation aborted at t/Image.t line 10. +Bareword "Barcode::ZBar::Symbol::EAN13" not allowed while "strict subs" in use at t/Image.t line 101. +Execution of t/Image.t aborted due to compilation errors. +# Looks like your test exited with 255 just after 1. +t/Image.t ......... Dubious, test returned 255 (wstat 65280, 0xff00) +Failed 22/22 subtests +t/pod-coverage.t .. skipped: Test::Pod::Coverage required for testing pod coverage +t/pod.t ........... skipped: Test::Pod 1.00 required for testing POD +t/Processor.t ..... 1/20 +# Failed test 'use Barcode::ZBar;' +# at t/Processor.t line 10. +# Tried to use 'Barcode::ZBar'. +# Error: Can't load '/build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so' for module Barcode::ZBar: /build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so: undefined symbol: zbar_scanner_reset at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/DynaLoader.pm line 193. +# at t/Processor.t line 10. +# Compilation failed in require at t/Processor.t line 10. +# BEGIN failed--compilation aborted at t/Processor.t line 10. +Bareword "Barcode::ZBar::Symbol::EAN13" not allowed while "strict subs" in use at t/Processor.t line 58. +Execution of t/Processor.t aborted due to compilation errors. +# Looks like your test exited with 255 just after 1. +t/Processor.t ..... Dubious, test returned 255 (wstat 65280, 0xff00) +Failed 20/20 subtests +t/Scanner.t ....... 1/3 +# Failed test 'use Barcode::ZBar;' +# at t/Scanner.t line 10. +# Tried to use 'Barcode::ZBar'. +# Error: Can't load '/build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so' for module Barcode::ZBar: /build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so: undefined symbol: zbar_scanner_reset at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/DynaLoader.pm line 193. +# at t/Scanner.t line 10. +# Compilation failed in require at t/Scanner.t line 10. +# BEGIN failed--compilation aborted at t/Scanner.t line 10. +Can't locate object method "new" via package "Barcode::ZBar::Scanner" (perhaps you forgot to load "Barcode::ZBar::Scanner"?) at t/Scanner.t line 14. +# Looks like your test exited with 255 just after 1. +t/Scanner.t ....... Dubious, test returned 255 (wstat 65280, 0xff00) +Failed 3/3 subtests +t/ZBar.t .......... 1/3 +# Failed test 'use Barcode::ZBar;' +# at t/ZBar.t line 10. +# Tried to use 'Barcode::ZBar'. +# Error: Can't load '/build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so' for module Barcode::ZBar: /build/Barcode-ZBar-0.04/blib/arch/auto/Barcode/ZBar/ZBar.so: undefined symbol: zbar_scanner_reset at /nix/store/n7hbdyp3bsmdxy2lcxivaxnq4nv8ndv3-perl-5.32.1/lib/perl5/5.32.1/x86_64-linux-thread-multi/DynaLoader.pm line 193. +# at t/ZBar.t line 10. +# Compilation failed in require at t/ZBar.t line 10. +# BEGIN failed--compilation aborted at t/ZBar.t line 10. +Undefined subroutine &Barcode::ZBar::version called at t/ZBar.t line 14. +# Looks like your test exited with 255 just after 1. +t/ZBar.t .......... Dubious, test returned 255 (wstat 65280, 0xff00) +Failed 3/3 subtests + +Test Summary Report +------------------- +t/Decoder.t (Wstat: 65280 Tests: 1 Failed: 1) + Failed test: 1 + Non-zero exit status: 255 + Parse errors: Bad plan. You planned 13 tests but ran 1. +t/Image.t (Wstat: 65280 Tests: 1 Failed: 1) + Failed test: 1 + Non-zero exit status: 255 + Parse errors: Bad plan. You planned 22 tests but ran 1. +t/Processor.t (Wstat: 65280 Tests: 1 Failed: 1) + Failed test: 1 + Non-zero exit status: 255 + Parse errors: Bad plan. You planned 20 tests but ran 1. +t/Scanner.t (Wstat: 65280 Tests: 1 Failed: 1) + Failed test: 1 + Non-zero exit status: 255 + Parse errors: Bad plan. You planned 3 tests but ran 1. +t/ZBar.t (Wstat: 65280 Tests: 1 Failed: 1) + Failed test: 1 + Non-zero exit status: 255 + Parse errors: Bad plan. You planned 3 tests but ran 1. +Files=7, Tests=5, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.20 cusr 0.03 csys = 0.25 CPU) +Result: FAIL +Failed 5/7 test programs. 5/5 subtests failed. +make: *** [Makefile:1040: test_dynamic] Error 255 +error: builder for '/nix/store/gww59146rs399rjc3fnawrjng4pqf6dl-perl5.32.1-Barcode-ZBar-0.04.drv' failed with exit code 2; + last 10 log lines: + > Non-zero exit status: 255 + > Parse errors: Bad plan. You planned 3 tests but ran 1. + > t/ZBar.t (Wstat: 65280 Tests: 1 Failed: 1) + > Failed test: 1 + > Non-zero exit status: 255 + > Parse errors: Bad plan. You planned 3 tests but ran 1. + > Files=7, Tests=5, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.20 cusr 0.03 csys = 0.25 CPU) + > Result: FAIL + > Failed 5/7 test programs. 5/5 subtests failed. + > make: *** [Makefile:1040: test_dynamic] Error 255 + For full logs, run 'nix log /nix/store/gww59146rs399rjc3fnawrjng4pqf6dl-perl5.32.1-Barcode-ZBar-0.04.drv'. +``` + +
+ +## Option 2: The Overlay + +Let's just overlay the source code with `0.10`, [a version from 2009](https://github.com/mchehab/zbar/releases/tag/0.10). + +This wasn't very successful as it seems the steps to build ZBar have [changed significantly](https://github.com/NixOS/nixpkgs/commit/57ffe86efa988788b6c58a1e3b682ee8f80c74a3#diff-2793faccce1027ffbb3d7e8658912e32422bdb8907db45e295f97c2768c526c2) between versions. + +However, I chose `0.10` not only for the fact that it was from 2009. It was the [oldest version](https://github.com/NixOS/nixpkgs/blob/7147ef8e80ae9f5d7f13b0c29bbf7a4d27982d3d/pkgs/tools/graphics/zbar/default.nix) in `nixpkgs`. + +So let's substitute the current package with this old one. + +A handy tool I found along the way was [Nix package versions](https://lazamar.co.uk/nix-versions/) which gives a nice web interface for finding older versions of packages and giving you the revision that they were in. + +Armed with a really hacky `zbar` overlay, let's try this again. + +```nix +zbar = final: prev: { + zbar = (import (builtins.fetchGit { + url = "https://github.com/NixOS/nixpkgs/"; + ref = "refs/heads/nixpkgs-unstable"; + rev = "12408341763b8f2f0f0a88001d9650313f6371d5"; + }) { system = "x86_64-linux"; }).zbar; +}; +``` + +Sidenote: A more Nix'y way of doing this would be to import this ancient version of `nixpkgs` as an input into your flake: + +```nix +inputs.nixpkgs-ancient = { + url = "github:NixOS/nixpkgs?rev=12408341763b8f2f0f0a88001d9650313f6371d5"; + flake = false; +} +``` + +and then use it via: + +```nix +zbar = final.callPackage ./zbar.nix { pkgs = final; pkgsAncient = import nixpkgs-ancient { system = final.system; }; }; +``` + +where `zbar.nix` is: + +```nix +{ pkgs, pkgsAncient }: + +let zbar = pkgsAncient.zbar; in + +with pkgs; +with perlPackages; + +buildPerlPackage { + pname = "Barcode-ZBar"; + version = "0.04"; + src = fetchurl { + url = "mirror://cpan/authors/id/S/SP/SPADIX/Barcode-ZBar-0.04.tar.gz"; + sha256 = "d57e1ad471b6a29fa4134650e6eec9eb834d42cbe8bf8f0608c67d6dd0f8f431"; + }; + doCheck = true; + buildInputs = [ pkg-config TestMore ExtUtilsMakeMaker TestHarness ]; + propagatedBuildInputs = [ zbar PerlMagick ]; + meta = { + homepage = "https://github.com/mchehab/zbar"; + description = "Perl interface to the ZBar Barcode Reader"; + license = with lib.licenses; [ gpl2Plus ]; + }; +} +``` + +Thanks to [Las](https://las.rs) for this Flake-ier way of doing things. + +Sadly this still fails with the `undefined symbol` error. +It seems we need an even more ancient version of Zbar + +## Where things float + +It seems that we have reached the point of diminishing returns and that it does not seem worthwhile to figure out how to build ever older versions of ZBar in the hope that this _might_ someday work. + +So what next? + +Perhaps we can convince upstream to take on the maintenance of this Perl module and bring it (and their own project) kicking and screaming into the decade of the 2020s. ([Sometimes your brain just needs some downtime](https://www.scientificamerican.com/article/mental-downtime/)) + +A few days after putting this project on the back burner I woke up with an inexplicable renewed enthusiasm for tackling this issue. + +Looking back through all the resources I noticed that the maintained version of `ZBar` has a series of folders named after various programming languages and platforms, including _Perl_. + +Interesting… + +It seems that when forking ZBar, @mchehab, the new maintainer, also made the sage decision to include all the myriad interfaces for the library and maintain them! + +Armed with this new Perl module that we **knew** worked with the latest ZBar library it was possible to construct a new `buildPerlPackage` that reused the `src` from the ZBar packaged in `nixpkgs`. + +```nix +BarcodeZBar = buildPerlPackage { + pname = "Barcode-ZBar"; + version = "0.04"; + src = "${zbar.src}/perl"; + postPatch = '' + substituteInPlace Makefile.PL --replace "-lzbar" "-L${zbar.lib}/lib -lzbar" + # This test requires network access + rm t/Processor.t + ''; + doCheck = true; + buildInputs = [ TestPodCoverage TestPod DevelChecklib TestMore ExtUtilsMakeMaker ]; + propagatedBuildInputs = [ zbar PerlMagick ]; + meta = { + homepage = "https://github.com/mchehab/zbar/tree/master/perl"; + description = "Perl interface to the ZBar Barcode Reader"; + license = with lib.licenses; [ gpl2Plus ]; + }; +}; +``` + +Et voilà, we have a Perl Module that interfaces with the current version of ZBar! + +## Addendum + +The keen-eyed among you will have noticed that there is a bit of trickery happening with the `postPatch` hook. + +Perl has the incredibly unhelpful error command: + +`Warning (mostly harmless): No library found for -lzbar` + +As you can imagine this is unquestionably harmful and causes the tests, and of lesser importance the final output 😛, to fail. + +This happens because for some reason our `propagatedBuildInputs` are not being added to the sandbox for the `Makefile` (which adds paths to `NIX_LDFLAGS`). Instead these had to be added manually by linking them directly in the `Makefile`. + +It is unclear to me why this is and perhaps a issue needs to be opened that addresses this. + +## A quick refresher on `NIX_LDFLAGS`. + +Nix differs drastically from other Linux distributions (and in difference is strength) and does not store header files and libraries in the traditional `/usr/include` where the compiler expects them. +Instead, everything is in the Nix Store. + +This means that somehow we need to make the compiler aware of the correct paths to these headers and libraries. +This is where `NIX_LDFLAGS` (and its partner in compile `NIX_CFLAGS_COMPILE`) come in. +The ALL CAPS should give you a clue that they are _environment variables_ which are used to furnish the compiler (by way of command line arguments) with these correct paths. +This is all done using shell scripts that wrap around the actual compiler. +For more information see the [C section](https://nixos.wiki/wiki/C) on the Nixos Wiki. diff --git a/src/content/blog/stories/deploying-simple-jitsi-meet-server.md b/src/content/blog/stories/deploying-simple-jitsi-meet-server.md new file mode 100644 index 0000000000..b14d0d65d7 --- /dev/null +++ b/src/content/blog/stories/deploying-simple-jitsi-meet-server.md @@ -0,0 +1,326 @@ +--- +title: Deploying a simple Jitsi Meet server with NixOS +date: 2022-07-28 +extra: + author: Simon Bruder +--- + +## Introduction + +During the COVID-19 pandemic, +video conferencing turned out to be an invaluable tool for online collaboration. +While many used proprietary tools for this, +there is one proven free and open source video conferencing tool +that also gained much traction and stayed popular: +[Jitsi Meet](https://jitsi.org/jitsi-meet/) + +Primarily known for its publicly available service [meet.jit.si](https://meet.jit.si), +it is also possible to self-host Jitsi Meet on your own server. + +### Reasons to self-host + +There are a variety of reasons +why you may consider hosting Jitsi Meet yourself +instead of relying on an already existing hosted instance: + +- You stay in control of your own data. + When relying on a third-party service, + you have to trust them, + and in case you are using the service professionally, + you may be legally required to sign a contract on data processing with the provider. +- You can have your own branding. + Not only does the service run on your own domain, + but you can also customise the logo, welcome page, etc. +- You can configure it to your liking. + Depending on the typical use case of the instance, + some optional features might be desirable + that are not commonly found on other instances. +- You can update the service when you want. + If you need the service to work, + the user interface totally changing during an important meeting may be problematic. + By hosting it yourself, you can decide when to update it. + +### Why NixOS + +So why deploy this on NixOS and not on Debian or with Docker? +NixOS has many advantages for running servers. +It allows you to configure your whole server declaratively, +so you can reuse the same configuration for multiple servers. +Also, because it is reproducible, +setting up a server with the same NixOS configuration +will ensure that it runs the exact same software with the same configuration. +Finally, if an update breaks your setup, +NixOS makes rollbacks as easy as running one command or rebooting, +so you can minimise downtime. + +## Requirements + +We will implement a small single-server deployment, +so you need one server (e.g., a VPS). +Its required specifications depend on how you want to use it. + +For just trying things out, one CPU core and 2 GB RAM should be enough, +though you will run into limitations pretty quickly. +The [official Jitsi Meet devops guide](https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-requirements) recommends 4 CPU cores and 8 GB RAM. + +If you don’t care about meeting recordings, +a 15 GiB disk could be enough, +though it doesn’t hurt to have more. +Recordings, however, will dramatically increase the resource consumption. +You will need at least 8 GB RAM, if possible even more, +and a CPU fast enough to keep up with handling the meeting while simultaneously encoding it. +Obviously, it also requires a larger disk to store the recordings on. + +You will also need a domain where Jitsi Meet should be available. + +If you already have a NixOS server set up +that has the required resources available, +you can skip to [Getting basic Jitsi Meet service running](#getting-basic-jitsi-meet-service-running). + +For configuring and deploying it, +[Nix](https://nixos.org) with flakes support is required to be installed on your local system. +If you have Nix version 3 or above, everything is already set up. +Otherwise, you need to [enable it explicitly](https://nixos.wiki/wiki/Flakes#Enable_flakes). + +## Setting up NixOS + +You can either set up a NixOS system [as described in the NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-installation) +or you can replace the default image of your provider with NixOS, +either manually by using [NixOS Lustrate](https://nixos.org/manual/nixos/stable/#sec-installing-from-other-distro) +or automatically with [nixos-infect](https://github.com/elitak/nixos-infect). + +If you use `nixos-infect`, +you must ensure that your public SSH key is already added to root’s `authorized_keys` before the script gets run, +otherwise you won’t be able to log in to the new NixOS system. +Consider reading [`nixos-infect`’s documentation](https://github.com/elitak/nixos-infect/blob/master/README.md) before using it +to avoid destroying important data. + +After the new NixOS system is up, +we will set up the configuration file used to deploy it. +For this we’ll use [deploy-rs](https://github.com/serokell/deploy-rs), +so you need that available. +You can either install it into your environment +or just enter a shell environment to make it available temporarily: + +``` +nix shell github:serokell/deploy-rs +``` + +You then need to create the file `flake.nix` in a new directory +that specifies the skeleton for the deployment: + +```nix +{ + inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixos-22.05"; # change this to your desired NixOS version + deploy-rs.url = "github:serokell/deploy-rs"; + }; + + outputs = { self, nixpkgs, deploy-rs }: { + nixosConfigurations.jitsi-meet = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; # if your target system isn’t x86_64, change this + modules = [ ./jitsi-meet/configuration.nix ]; + }; + + deploy.nodes.jitsi-meet = { + hostname = "meet.example.com"; # change this to your FQDN of the server + sshUser = "root"; + # When deploying for the first time, the server doesn’t yet have IPv6 connectivity. + # This forces ssh to connect over IPv4 and can be removed after the first successful deployment. + sshOpts = [ "-4" ]; + + profiles.system = { + path = deploy-rs.lib.x86_64-linux.activate.nixos self.nixosConfigurations.jitsi-meet; + }; + }; + + checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs.lib; + }; +} +``` + +Now you need to create the NixOS configuration in `jitsi-meet/configuration.nix`: + +```nix +{ modulesPath, ... }: + +{ + networking.hostName = "jitsi-meet"; + + imports = [ + # You can change this to match your target system + # (or remove it entirely if it doesn’t need special configuration). + # An introduction to NixOS profiles can be found in the manual: https://nixos.org/manual/nixos/stable/#ch-profiles + (modulesPath + "/profiles/qemu-guest.nix") + ]; + + networking = { + # replace those details with your network interface, the IPv6 gateway, address and prefix length + # IPv4 will automatically get configured over DHCP (if your provider supports it) + defaultGateway6 = { address = "fe80::1"; interface = "ens3"; }; + interfaces.ens3.ipv6.addresses = [ { address = "2a01:db8:abcd:1234::"; prefixLength = 64; } ]; + }; + + services.openssh.enable = true; + users.users.root.openssh.authorizedKeys.keys = [ + # don’t forget to add your SSH public key(s) here! + "ssh-ed25519 AAAA…" + ]; + + # if your disk device is called differently, you need to change this + boot.loader.grub.device = "/dev/sda"; + fileSystems."/" = { + device = "/dev/sda1"; + fsType = "ext4"; + }; + + system.stateVersion = "22.05"; # change this to the NixOS version you first installed the system with +} +``` + +Now, you can just run `deploy` and everything should be set up automatically. +After it worked successfully, +you can remove the `sshOpts = [ "-4" ];` line from the flake, +because it should now have IPv6 connectivity. + +## Getting basic Jitsi Meet service running + +What makes using Jitsi Meet with NixOS so nice +is that there is a module for it. +That means, you can enable a basic — but working — Jitsi Meet instance +just by adding the following to your system configuration: + +```nix +{ + services.jitsi-meet = { + enable = true; + hostName = "meet.example.com"; # change this to your domain + }; + + services.jitsi-videobridge.openFirewall = true; + + security.acme = { + acceptTerms = true; # ensure that you have read the subscriber agreement (https://letsencrypt.org/repository/) + defaults.email = "nospam@example.com"; # change this to your email address + }; + + networking.firewall.allowedTCPPorts = [ 80 443 ]; +} +``` + +After you deploy this configuration, +you should already have a working Jitsi Meet service. + +## Advanced configuration + +### Customisation + +For customising the features and look of the Jitsi Meet web application, +there are two options available. + +One is the interface configuration (`interface_config.js`), +which is deprecated but still required to configure some things. +It can be modified by setting the NixOS option `services.jitsi-meet.interfaceConfig`. +A commented list of all options and their default values can be found [in the Jitsi Meet source code](https://github.com/jitsi/jitsi-meet/blob/master/interface_config.js). + +An example configuration that hides the Jitsi watermark could look like this: + +```nix +{ + services.jitsi-meet.interfaceConfig = { + SHOW_JITSI_WATERMARK = false; + }; +} +``` + +In the future, all options from `interface_config.js` will be moved to `config.js`. +Many options are already configurable from it. +Like with the interface configuration, a commented list of all options and their default values can be found [in the Jitsi Meet source code](https://github.com/jitsi/jitsi-meet/blob/master/config.js). + +There is one option I recommend you to set, +`prejoinPageEnabled`, which enables a page to set up microphone and camera +and lets the user disable them _before_ joining a meeting. + +There are many more options available, +so it might be useful to look at the options and set up everything to your liking. + +An example configuration could look like this: + +```nix +{ + services.jitsi-meet.config = { + prejoinPageEnabled = true; + disableModeratorIndicator = true; + }; +} +``` + +### Recordings + +Using the Jitsi Meet component Jibri, +it is possible to record a meeting remotely — that means without running additional software on the participants’ clients. + +This currently is achieved by running a Chromium instance in a virtual X11 framebuffer, +which then is recorded by [ffmpeg](https://ffmpeg.org/). +Because of this, it is currently not possible to run multiple instances of Jibri on the same host +and therefore it is only possible to have one conference recording running at a time. +In theory it is possible to bypass these limits by running each Jibri in its own container, +but this is not officially supported by the Jibri developers and hasn’t yet been implemented for NixOS. +Moreover, the current NixOS module only allows them to run on the same host as all other Jitsi services, +which means they have to compete for resources. +It therefore is very important +that this server has enough resources to be able to handle this. + +To enable Jibri, add: + +```nix +{ + services.jitsi-meet.jibri.enable = true; +} +``` + +By default, it will record to `/tmp/recordings`. +This and more options can be configured by setting `services.jibri.config`, +for which a reference configuration file can be found [in the jibri source code](https://github.com/jitsi/jibri/blob/master/src/main/resources/reference.conf), +or `services.jibri.finalizeScript`. +The latter has a [useful example in the NixOS module documentation](https://search.nixos.org/options?channel=22.05&show=services.jibri.finalizeScript&from=0&size=50&sort=relevance&type=packages&query=services.jibri.). + +An example that changes the recording destination directory +and tunes the encoding options, +but does not upload the recordings to a different server, +could look like this: + +```nix +{ config, ... }: + +{ + services.jibri.config = { + recording = { + recordings-directory = "/var/lib/jitsi-meet-recordings"; + }; + ffmpeg = { + #framerate = 30; + #video-encode-preset = "veryfast"; # https://trac.ffmpeg.org/wiki/Encode/H.264#a2.Chooseapresetandtune + h264-constant-rate-factor = 21; # https://trac.ffmpeg.org/wiki/Encode/H.264#a1.ChooseaCRFvalue + }; + }; + + # This is required if the recordings directory can’t be created by Jibri itself + # e.g. due to missing permissions. + # If it is under /tmp/ (like the default), this is not needed. + systemd.tmpfiles.rules = [ + "d ${config.services.jibri.config.recording.recordings-directory} 0750 jibri jibri -" + ]; +} +``` + +## Outlook + +That concludes this blog post on how to set up a simple Jitsi Meet server on NixOS. +But it still only scratches the surface of what is possible with Jitsi Meet. +In the future, support for many of Jitsi Meet’s other features will be added to NixOS. +Some possible features include clustering, +authenticated participants, +and joining a meeting via telephone. + +Stay tuned! diff --git a/src/content/blog/stories/the-rise-of-special-project-infra.md b/src/content/blog/stories/the-rise-of-special-project-infra.md new file mode 100644 index 0000000000..016b8b6a75 --- /dev/null +++ b/src/content/blog/stories/the-rise-of-special-project-infra.md @@ -0,0 +1,190 @@ +--- +title: The Rise of Special Project "infra" +date: 2022-08-12 +extra: + author: Ctem +--- + +_72 hours out. A bead of sweat slides from your brow, falls to the marred chassis of your local build server, and sizzles into mist, leaving a scant salt stain to tell the tale. It’s the start of the hottest Summer of Nix on record, and you have three days to [research](#phase-i-research), [provision](#phase-ii-provision), [configure](#phase-iii-configure), and [deploy](#phase-iv-deploy). With or without you, this lecture is going live._ + +The deadline was 19 July, 2022; at five o’clock on that glistening Tuesday afternoon, one Eelco Dolstra – a living legend to those who understand – would take to the webcam from the verdant city of Utrecht to deliver a highly anticipated slice of fresh perspective on his now-19-year-old brainchild, [Nix](https://nixos.org/learn.html). It was the inaugural event in the premier Summer of Nix (SoN) Public Lecture Series. The hype was real, but so was our predicament: no self-hosted livestreaming infrastructure was yet in place. + +Would we simply fall back on the usual [gatekeeper](https://ec.europa.eu/info/strategy/priorities-2019-2024/europe-fit-digital-age/digital-markets-act-ensuring-fair-and-open-digital-markets_en) platforms, surrendering control of the narrative and feeding our own to corporate leviathans in a vacuum of moral agency? + +The ubiquity of these platforms suggests that most would.[^ubiquity] This, however, is SoN, and we aren’t most. Defying norms of defeatism and manufactured consent, we dare to declare a world of configurability. We celebrate that self-hosting empowers us to maintain ownership of both our content and its presentation, allowing us to introduce to our audience – our community – a way to engage with said content free from third-party influence. + +Note that our solidarity is no coincidence; SoN begins with a contractual agreement between each participant and Eelco himself to uphold both [NGI Zero](https://nlnet.nl/NGI0/)’s aim to contribute to an open internet and the [ACM Code of Ethics and Professional Conduct](https://www.acm.org/code-of-ethics), which emphasizes among other points the importance of respecting privacy. + +Needless to say, the prospect of streaming Eelco’s lecture exclusively to closed-source, centralized, and infamously privacy-disrespecting services was an irony too plain to ignore. Fortunately, infrastructural improvement is a fundamental objective of the program, and Nix is the definitive tool for the job. In short, the Public Lecture Series livestreaming infrastructure was a natural first target. Time was not on our side, but good people were: + +- The [NixOS Foundation](https://opencollective.com/nixos) was at the ready for server provisioning and DNS management. +- Personnel at [Tweag](https://www.tweag.io/) generously offered expertise in configuration and deployment in a pinch. +- A new SoN Special Project – tidily dubbed _infra_ – was launched to formalize the effort and attract participation. + +With this kind of support, the odds were decidedly stacked in our favor, _but could we deliver?_ We’d find out soon enough... + +### Phase I: Research + +Everybody needs a saga. Ours was: self-host a secure livestreaming server to accommodate a sizable audience participating from all around the world and at variable network speeds, _fast_. Oh, and add a chat room for synchronous Q&A. + +On the server side, we settled on the following stack: virtual private server (VPS) → NixOS → [Caddy](https://caddyserver.com/) → [Owncast](https://owncast.online/). + +The media would originate on our intrepid moderator’s local [OBS Studio](https://obsproject.com/) and be streamed at archival quality over the Real-Time Messaging Protocol (RTMP) to our remote Owncast service. Owncast would transcode the video to multiple stream qualities (i.e., optimized variants of video bitrate and CPU usage) and present the selected variant to the user in an attractive, branded web UI with a convenient chat box. + +### Phase II: Provision + +Controlling the narrative has always called for a bit of hardware. For this exercise, we went with a single VPS equipped with dedicated vCPUs. The general provisioning procedure follows: + +1. Choose a (preferably reputable and carbon negative) VPS provider. + +1. Provision an instance appropriate to the task and estimated audience. + + - To minimize latency, select the available region that best approximates the location of the source stream as well as the mean location of the majority of expected participants where applicable. Note that _object storage_ and _CDN caching_ additionally may be leveraged to accommodate participants; object storage enables the streaming service to serve media files when the concurrent user count exceeds available bandwidth, and CDN caching enables users to download these assets from the nearest available server. + - Select a server with enough (preferably dedicated) vCPU cores to handle video transcoding, which is a highly processor-intensive operation. Note that this should correspond to the stream variants selected in the [streaming service configuration](#streaming-service-configuration). + + The instance we provisioned for the first stress test included the following specs: + + | vCPU | RAM | Local storage | Location | + | ---: | ----: | ------------: | -------- | + | 8 | 32 GB | 240 GB | Germany | + +1. Reserve a public IPv4 address and IPv6 subnet and associate with the provisioned instance. + +1. Assuming registration of the desired domain name, use a/the DNS registrar-provided interface to update the (or create an) A record for the target subdomain (`live.nixos.org` in our case) with the IP address associated with the provisioned instance. Note that propagation may take up to 48 hours. + +### Phase III: Configure + +Perhaps unintuitively, this was the easiest part. Thanks to the [`nixpkgs`](https://github.com/NixOS/nixpkgs) community contributions of high-quality modules[^module] for production-ready services[^service], configuring a fully self-hosted livestreaming service safely behind a reverse proxy was scarcely more involved than toggling `enable`. Consider the following Nix code: + +```nix +{ + networking.firewall.allowedTCPPorts = [ 80 443 ] + ++ [ config.services.owncast.rtmp-port ]; + + security.acme.defaults.email = "admin@example.com"; + + services.owncast.enable = true; + + services.caddy = { + enable = true; + email = config.security.acme.defaults.email; + virtualHosts = { + "live.nixos.org".extraConfig = let + owncastWebService = "http://${config.services.owncast.listen}:${ + toString config.services.owncast.port + }"; + in '' + encode gzip + reverse_proxy ${owncastWebService} + ''; + }; + }; +} +``` + +Demonstrating the seamless UX of a well-written NixOS module, the Owncast configuration is completely out-of-the-box; using [all defaults](https://search.nixos.org/options?channel=unstable&from=0&size=50&sort=relevance&type=options&query=owncast), one line installs and activates the service, initializes all necessary state (including a SQLite database `owncast.db` that stores important service configuration such as administrative credentials) in `/var/lib/owncast`, and binds the web server to `127.0.0.1:8080`. + +The Caddy module is similarly loaded (with goodies). The only customization necessary here was to specify the domain we prepared (in the last step of the provisioning phase above) as a virtual host from which Caddy can forward client requests to the backend Owncast web server. For good measure, we also specified an email address for Caddy to use when setting up SSL on our behalf. This is the email address to use for account creation and correspondence from the SSL certificate authority, which is [Let's Encrypt](https://letsencrypt.org/) by default. + +The remaining configuration simply opens ports in the system firewall for HTTP, HTTPS, and RTMP. Note that the RTMP port is also an Owncast module-specified default. + +That’s the entirety of the NixOS configuration specific to our use case; all additional configuration (e.g., importing generated hardware configuration, enabling SSH and adding appropriate keys, defining Nix garbage collection rules, etc.) is simply boilerplate. + +### Phase IV: Deploy + +With the instance provisioned and the configuration prepared, it was about time to introduce the two, marking the beautiful beginning of a successful working relationship. This is generally a two-step process: + +1. Install NixOS on the instance. A provider may make this very easy (by providing a template image), moderately easy (by allowing [a custom ISO](https://github.com/nix-community/nixos-generators) to be uploaded), or hit-or-miss (by providing an image for a non-NixOS distribution from which NixOS may be converted with [NixOS-Infect](https://github.com/elitak/nixos-infect/) or [`NIXOS_LUSTRATE`](https://nixos.org/manual/nixos/stable/#sec-installing-from-other-distro)). + + Our provider didn’t do us too many favors in this regard. They were known to play nice with `nixos-infect`ed installations, however, so we went with that. + +1. Deploy the configuration. Note that several tools exist to automate this process: [deploy-rs](https://github.com/serokell/deploy-rs) is wildly popular for flake-based configurations, and [Cachix Deploy](https://blog.cachix.org/posts/2022-07-29-cachix-deploy-public-beta/) is a promising new offering (just to name two). + + We were quite frankly in a hurry, however, and found it perfectly acceptable to do it the old-fashioned way: + + 1. Access the host over SSH. + 1. Clone the configuration. + 1. Run `nixos-rebuild switch` to build and activate the configuration (and make it the boot default). + +With this, we had a fully operational production server. For the finishing touches, we would make just a few program-specific tweaks. + +#### Streaming service configuration + +As previously mentioned, much of the Owncast configuration is stored as mutable state, enabling the service to be modified on-the-fly from the administrative web UI. A suggested procedure follows: + +1. Log in to the administrative web UI: + + - URL: `https:///admin/` (e.g., `https://live.nixos.org/admin/`) + - User name: `admin` + - Password: `` (`abc123` by default) + +1. In server settings, set a new Stream Key. + + Note that this is also the administrative web UI password. + +1. In general settings, modify instance details, tags, and social handles as appropriate. + +1. In video settings: + + - Add stream variants to enable adaptive bitrate streaming to accommodate users on various-quality networks. + + The following table may be used as a guideline: + + | Encoder | Name | Resolution (WxH) | Bitrate | Framerate | + | ------- | ---- | ---------------- | -------------: | ------------ | + | x264 | SD | 854x480 | 800-1200 kbps | 25 or 30 fps | + | x264 | HD | 1280x720 | 1200-1900 kbps | 25 or 30 fps | + | x264 | FHD | 1920x1080 | 1900-4500 kbps | 25 or 30 fps | + + Note that some services recommend a higher bitrate for HD: + + | Encoder | Name | Resolution (WxH) | Bitrate | Framerate | + | ------- | ---- | ---------------- | --------: | ------------ | + | x264 | HD | 1280x720 | 3000 kbps | 25 or 30 fps | + | x264 | FHD | 1920x1080 | 4500 kbps | 25 or 30 fps | + + Our stress tests showed that the use of standard variants with our VPS were reportedly adequate for most users but problematic for some in Japan, Thailand, and the UK. Based on these results, object storage and CDN caching are recommended. + + Additional notes: + + - For interactive live streams (e.g., lectures with Q&A sessions), consider decreasing the latency buffer to Low. + - For further tuning, consider referring to a bitrate calculator (e.g., [Bitrate Calc](https://bitratecalc.com/)). + +#### Streaming client configuration + +With the server side all set, it was time to get streaming! The basic procedure follows: + +1. On a local workstation, install and launch [OBS Studio](https://obsproject.com/) ([obs-studio](https://search.nixos.org/packages?channel=22.05&show=obs-studio&from=0&size=50&sort=relevance&type=options&query=obs-studio) in `nixpkgs`). + +1. In stream settings, set the following values: + + - Service: `Custom...` + - Server: `rtmp://` (e.g., `rtmp://live.nixos.org`) + - Stream key: `abc123` (by default) + +1. Go live. + +1. Confirm the stream at the designated URL (`https://live.nixos.org` in our case). + +_It worked?_ It worked. Crisis averted with time to spare. 🍹 + +## Next Steps + +Following a successful first lecture, the SoN infra team is concerned not only with maintenance and optimization of the infrastructure we’ve already deployed but also with innovation on a larger scale. The ultimate goal is to contribute _a scalable, fully self-hosted, documented, NixOS-based, general-purpose, FLOSS[^floss] conference suite_. In short, we’re working toward a turnkey solution that can accommodate use cases ranging from the SoN Public Lecture Series all the way to a full conference experience – namely [NixCon](https://nixcon.org/). + +In addition to main conference components (e.g., administrative tools, breakout rooms, etc.) and general convenience features (e.g., collaboration tools such as shared whiteboards), a notable priority is the improvement of our accessibility story, e.g., through integration of the [Vosk speech recognition toolkit](https://alphacephei.com/vosk/) for livestream closed captioning/transcription/subtitling (in multiple languages). + +To this end, we’re working together with other stalwart teams such as the SoN Special Project for [Jitsi](https://jitsi.org/) and have received guidance and insight from the gracious maintainers of the [NixCon 2020 livestreaming infrastructure](https://github.com/nixcon/nixcon-video-infra), the crucial project of which our efforts will be a continuation. + +[^ubiquity]: In the context of this article, _ubiquity_ is scoped to societies who enjoy the free global exchange of information (i.e., unfiltered cross-border Internet traffic). Likewise, _most_ refers to the majority of members of such a society when faced with choosing a method for applicable content publishing. + +[^module]: + Service module quality, while service context-dependent, is generally evaluated with respect to such criteria as: + + - sensibility of included defaults (i.e., such that the service can be started without requiring excessive configuration for common use cases) + - exposure of a balanced set of options (i.e., such that the service is sufficiently configurable but not at the expense of module maintainability) + - inclusion of a balanced set of [integration tests](https://nixos.org/guides/integration-testing-using-virtual-machines.html) (i.e., that is sufficiently comprehensive for stable operation but not restrictively opinionated) + +[^service]: A _production-ready service_ is characterized here as a service that is appropriately licensed and sufficiently stable, secure, performant, and featureful for a given use case. Note that in the context of FLOSS[^floss], active maintainership is additionally preferred. + +[^floss]: While the term [FLOSS](https://www.gnu.org/philosophy/floss-and-foss.en.html) indicates a politically neutral position, this project prefers software that is licensed to protect the four essential freedoms of users as defined in the [Free Software Definition](https://www.gnu.org/philosophy/free-sw.en.html). As a bare minimum, the term _open source_ is used here to describe software that fulfills the conditions of the [Open Source Definition](https://opensource.org/osd).