Operating System Bias in Next Generation Internet and NLnet

Source

In Grants for Operating Systems I discussed my journey through the grant application writing business since beginning of last year. To keep things light and somewhat focused, I left out a topic, that I would like to write about in more detail in the following sections.

It's about selection bias in grants provided by Next Generation Internet (NGI), that can be applied for directly or through NLnet.

📝 Before going into further detail, I would like to point out several things:

  • I believe funding programs such as NGI and NLnet are a very important pillar of a free and decentralized computing world
  • I do not wish any harm upon any of the involved organizations or any of their employees
  • This article is here to document my personal experiences and findings based on publicly available data, in the hopes of addressing what appears to be a selection bias
  • Although none of the Arch Linux related funding requests discussed in my previous article have been approved, I have been involved with Operating System agnostic projects, funded through NLnet, but applied for by other people, in the past

Round two

With my latest NLnet application for funding work on ALPM projects, I managed to reach round two.

To give a bit of preliminary background on what those projects are about (a more detailed article on them will follow):

During the work on repod I noticed, that many of the metadata files used in Arch Linux's packaging do not have a specification and no common parsers and validators. This required writing a lot of additional code for the consumption and validation of those files and I noticed that several efforts across other languages and projects existed. I realized, that it would be best to provide central specifications, parsers and writers, which could be used all across the stack. As such, the ALPM projects are a very Arch Linux packaging specific attempt at improving the tooling of this community-driven distribution.

Given the above, it felt rather strange to read loaded questions in my set of round two questions, that implied users seeking more guarantees should just move to distribution agnostic functional package managers, such as Nix or Guix, as those could be combined with Arch Linux without effort.

I attempted to reply by outlining why funding for the packaging subsystem of community-driven distributions is important and that "just replace everything with Nix" is not the answer for Arch Linux and its users.

Using a package management system other than pacman on Arch Linux is not supported and is in fact likely to break the system. Users that wish to use nix or guix can use dedicated distributions such as NixOS.

Arch Linux and its maintainers have spent decades building up expertise in integrating their custom package management system with various init systems, dedicated tooling and languages. Arch Linux follows central principles (https://wiki.archlinux.org/title/Arch_Linux#Principles), that encourage a simple, rolling release, “follow upstream” approach. As such it differs from other distributions.

Arch Linux’s thousands of users are familiar with its package management system. A radical change, such as replacing the package management subsystem is nothing that can be done on a whim or is even realistic given the steep learning curve of learning one.

Working on the Arch Linux Package Management framework ensures the diversity in a small set of original, community driven Linux distributions (such as Debian, Arch Linux, NixOS), for which public funding is essential.

Other questions revolved around topics such as comparison with other standardization and file format efforts, allotted time for the work (which was deemed too high), future changes to file formats and generalizing parsers using structured format.

As mentioned in my previous article, my application was rejected after another six weeks after my reply. As the rejection message was generic, I do not know whether any of the answers (or all of them) were dissatisfactory.

Investigating Bias

The package manager related questions in my second round review struck me as very odd and led me to do some deeper investigation into the NLnet and NGI funding setup and to ask a few people, that received OS-agnostic funding about their experiences.

People's experience with the review process seems to be largely identical to mine. However, in at least one occasion an applicant got an actual specific (non-generic) reason for their rejected NLnet application towards an NGI Zero grant (which was surprising even to them). In at least one other case, a person was asked to make their application run on NixOS after receiving their final payment.

I started to look further into the NixOS story in this context and discovered several connections between NLnet and NixOS Foundation.

For the unaware: Nix is the system package manager used on NixOS and NixOS Foundation has the mission to "[..] support the Nix ecosystem's infrastructure, and projects implementing the purely functional deployment model." 1 I use Nix and NixOS interchangeably throughout the following subsections, as the package manager is tightly coupled with the Operating System.

Through membership in NGI Zero, NixOS Foundation is part of NGI Zero Core, NGI Zero Entrust, NGI Zero PET and NGI Zero Review (see background info on NGI Zero Core, background info on NGI Zero Entrust, background info on NGI Zero PET and background info on NGI Zero Review, respectively).

In an attempt at giving an overview of Nix and Guix vs. other Operating System and system package manager projects funded by NLnet, I went through several program pages to collect affiliated projects. This proved to be not so easy, as the website follows varying style approaches and does not offer a search functionality, which hinders filtering by keyword. So please take the following numbers with a grain of salt!

I manually searched through overview pages linked to in the following sections, using nix and guix as search term to correlate projects, that stand in some relation to NixOS Foundation/ NixOS or Guix. I did the same using the OS, operating system and package search terms to sieve through projects, that have some relation to other specific general-purpose Operating Systems and do not provide generic features (e.g. VPN or firewall stack on Linux, etc.), or concern themselves with other system package managers. The percentages for occurrences are rounded up to the 2nd position after decimal point.

Do note, that the NLnet projects largely appear to be not tied to specific Operating Systems!

NGI Assure

There are 145 ongoing NGI Assure projects.

Eight Nix related (5.52%):

Seven Guix related (4.83%):

Three other OSes, mostly special purpose or mobile (2.07%):

I was not able to find any other system package manager related projects.

NGI Zero Core

There are 21 ongoing NGI Zero Core projects.

I was not able to find any Nix related projects.

One Guix related (4.76%):

Four other Operating System related projects, mostly special purpose or mobile (19.05%):

I was not able to find any other system package manager related projects.

NGI Zero Entrust

There are 150 ongoing NGI Zero Entrust projects.

Six Nix related (4%) projects:

I was not able to find any Guix related projects.

Six other Operating System related projects, mostly special purpose or Android (4%):

I was not able to find any other system package manager related projects.

NGI Zero PET

There are 144 ongoing NGI Zero PET projects.

Three Nix related (2.1%):

Three Guix related (2.1%):

Eight other Operating System related projects, mostly special purpose or Android (5.56%):

I was not able to find any other system package manager related projects.

Internet Hardening Fund

There are 24 projects of the Internet Hardening Fund.

One Nix related (4.17%):

I was neither able to find any Guix related projects, nor any related to other Operating Systems or other system package managers.

User-Operated Internet Fund

There are eleven projects of the User-Operated Internet Fund.

One is an Operating System specific project (9.10%).

I was neither able to find any Nix or Guix related projects, nor any related to other system package managers.

NGI Zero Review

There are no publicly associated projects with the NGI Zero Review program, but the program itself promotes the use of Nix for "[b]est practices on packaging and reproducible builds" 2 (see this pull request towards the NixOS homepage to replace this problematic terminology) for projects, that are mentored by it. It is unclear whether NixOS Foundation is compensated for this mentoring role.

Further data on NixOS Foundation and NGI Zero

According to a Summer of Nix 2022 interview, "the European Commission through DG CNECT has partnerships with NLnet and the NixOS Foundation", funding several projects. Furthermore, the European Commission appears to be facilitating and encouraging the use of NixOS internally, trying to replace other operating systems. The platform code.europa.eu is mentioned as a place for development of software and services related to European Union institutions.

When looking at the NixOS Foundation's Financial Summary for 2022, it shows an influx of 140.000€ of "[f]unds from NLnet Foundation for the specific programs (i.e. Summer of Nix)". These are potentially for some of the above mentioned projects in the various NLnet related programs, for which NixOS Foundation may be handling payments to individuals (although NLnet grants are usually given to individuals). However, from reading the statement alone, it is unclear whether this is tied to individual grants, compensation for work on NGI Zero Review, or even other things.

The above data points at a direct or at least indirect monetary conflict of interest for NixOS Foundation in the context of NGI Zero, which in my opinion ultimately serves as a bias for any decision making process done in the context of that coalition. My believe is, that the decision making process is therefore intrinsically skewed, because NGI Zero appears to be set up to promote one specific Operating System (NixOS) and package manager (Nix). Looking at the numbers, also NGI Assure appears to be affected by this.

Conclusion

Considering the previous sections provides a rather depressing outlook for non-NixOS general-purpose Linux distributions, as well as for non-Nix/Guix package management systems, when it comes to funding opportunities through NLnet or NGI.

Of the 495 NLnet projects I sieved through (mostly superficially), 18 (3.64%) appear to be Nix related, eleven (2.22%) Guix related, while 22 (4.44%) account for the entire rest of Operating System specific projects. I was not able to find any other system package manager related project.

While I am convinced, that people related to NixOS write great applications, for me it is hard to believe that there are so few (good) applications by developers working on other packaging ecosystems, that not a single one was ever able to receive funding. Relatedly, I neither believe, that it is right to ask those developers to replace their ecosystem with Nix, nor be judged on the base of working on something that is not Nix. As such I believe that the above list of projects does not provide a balanced funding reality.

Therefore I would like to extend my points on funding organizations in my previous article:

  • Technological bias in funding organizations claiming to "[..] reflect the openness, diversity and the inclusion that are at the core of European values" 3 should be circumvented, or otherwise clearly stated.

Analogous, I would like to extend the advice for people trying to get their work funded by:

  • If you are working on an Operating System related topic and you are not working on NixOS, consider whether your time will be well spent on applying for NGI Zero related funds or probably even NLnet programs in general, as - given the publicly available data - there appears to be a bias on the European Commission level at play, that will very likely lead to your project not being selected or it getting very hard to be selected.
  • If you are working on a system package manager other than Nix or Guix, there is currently no data supporting the assumption, that this work would be funded when applying with NLnet.

As I am not sure how well received this article will be with some of the organizations, I have neither mentioned people, that have helped review or write my applications, nor those, that I have asked about their experiences.

While I hope, that there will be no backlash towards people I interacted with, I realize, that this article may make some people uncomfortable and even undermine my chances of ever getting funding in the future. Either way, I believe it was the right thing to do and I arrive at the following conclusions and questions for myself:

  • Given the above data points, why does NGI (and NLnet by extension) not more clearly state, that they are (seemingly) not interested in funding work on other package manager ecosystems? While I do not know how many others have applied for similar projects to mine, nor do I claim to speak for this unknown number of people, I would have loved to not waste my time on an application, that seems to have little chance of ever being accepted.
  • The reasoning behind enforcing one specific package manager and Operating System in NGI is intransparent to outsiders. It is unclear to me when this decision was made, and under what circumstances the European Commission decided on it. The previously mentioned Summer of Nix 2022 interview seems to indicate, that the European Commission wants to switch to NixOS for its own services. This begs the question: Why do they not just contract with one of the consulting companies available for professional support? Biased funding on the other hand will have a huge impact on the ecosystem at large (and in my opinion not for the better).
  • There is a lot of work to be done in all community-driven Linux distributions and this work has merit. Focusing only on one distribution will achieve two things: Destroying diversity and invalidating the work thousands of people have been doing in their free time for decades (often even trying to make a living by providing services around those Operating Systems).

There are many open questions and my hopes are, that the European Commission, NGI and NLnet reevaluate their focus on seemingly funding only one packaging ecosystem. I would be happy to receive feedback from people related to other Linux distributions, that interacted with NGI and NLnet, as well as from officials involved with the decision making process in the European Commission, NGI and NLnet and will update the post accordingly. Writing this, I am still hopeful, that my post can be a first step towards improving the current situation and that funds directed at critical infrastructure projects will be distributed more evenly amongst widely used Operating System projects (big and small, well marketed and quiet).

  1. https://nixos.org/community/index.html ↩

  2. https://nlnet.nl/NGI0/review ↩

  3. https://digital-strategy.ec.europa.eu/en/library/internet-humans-how-we-would-internet-future-be ↩