This blog post has been floating around as a draft for several years. It eventually split off into a presentation at GUADEC 2022, titled Offline learning with GNOME and Kolibri (YouTube). In that presentation, Manuel Quiñones and I explained how Endless OS reaches a unique audience by providing Internet-optional learning resources, and we provided an overview of our work with Kolibri. This post goes into more detail about the technical implementation of the Kolibri desktop app for GNOME, and in particular how it integrates with Endless OS.

Integrating a flatpak app with an immutable OS

In Endless OS, way back with Endless OS 4 in 2021, we added Kolibri, an app created by Learning Equality, as a new way to discover educational content. Kolibri has a rich library of video lessons, games, documents, e-books, and more; as well as tools for guided learning – both for classrooms, and for families learning at home. The curation means it is safe and comfortable to freely explore. And all of this works offline, with everything stored on your device.

Making this all come together was an interesting challenge, but looking back on it with Endless OS 6 alive and well, I can say that it worked out nicely.

The Kolibri app for GNOME

Learning Equality designed Kolibri with offline, distributed learning in mind. While an organization can run a single large Kolibri instance that everyone reaches with a web browser, it is equally possible for a group of people to use many small instances of Kolibri, where those instances connect with each other intermittently to exchange information. The developers are deeply interested in sneaker net-style use cases, and indeed Kolibri’s resilience has allowed it to thrive in many challenging situations.

Despite using Django and CherryPy at its heart, Kolibri often presents itself as a desktop app which expects to run on end user devices. Behind the scenes, the existing Windows and MacOS apps each bundle a Kolibri server, running it in the background for as long as the desktop app is running.

We worked with Learning Equality to create a new Kolibri app for GNOME. It uses modern GTK with WebKitGTK to show Kolibri itself. It also includes a desktop search provider, so you can search for Kolibri content from anywhere.

The Kolibri GNOME app is distributed as a flatpak, so its dependencies are neatly organized, it runs in a well-defined sandbox, and it is easy to install it from Flathub. For Endless OS, using flatpak means it is trivial to update Kolibri independent from Endless OS’s immutable base system.

Kolibri Daemon

But Endless OS doesn’t just include Kolibri. One of my favourite parts of Endless OS is it provides useful content out of the box, which is great for people with limited internet access. So in addition to Kolibri itself, we want a rich library of Kolibri content pre-installed. And with so much already there, ready to be used, we want it to be easy for people to search for that content right away and start Kolibri for the first time.

This becomes both a technical challenge and a philosophical challenge. Normally, each desktop user has their own instance of Kolibri, with its own hidden directory full of content. Because it is a flatpak, it normally doesn’t see the rest of the system unless we explicitly give it permission to, and every time we do that we need to think carefully about what it means. Should we really grant a WebView the ability to read and write /run/media? We try to avoid it.

At the same time, we want a way to create new apps which use content from Kolibri, so that library of pre-installed content is visible up front, from the apps grid. But it would be expensive if each of these apps ran its own instance of Kolibri. And whatever solution we employ, we don’t want to diverge significantly from the Kolibri people are using outside of Endless OS.

To solve these problems, we split the code which starts and stops the Kolibri service into a separate component, kolibri-daemon. The desktop app (kolibri-gnome) and the search provider each communicate with kolibri-daemon using D-Bus.

This design is exactly what happens when you start the Kolibri app from Flathub. And with the components neatly separated, on Endless OS we add eos-kolibri, which takes it a step further: it adds a kolibri system user and a service which runs kolibri-daemon on the D-Bus system bus. The resulting changes turn out to be straightforward, because D-Bus provides most of what we need for free.

With this in place, every user on the system shares the same Kolibri content, and it is installed to a single well-known location: /var/lib/kolibri. Now, pre-installing Kolibri content is a problem we can solve at the system level, and in the Endless OS image builder. Independent from the app itself.

Channel apps

Now that we have solved the problem of Kolibri content being duplicated, we can come back to having multiple apps share the same Kolibri service. In Endless OS, we want users to easily see the content they have installed, and we do this by adding launchers to the apps grid.

First, we need to create those apps. If someone has installed content from a Kolibri channel like TED-Ed Lessons or Blockly Games, we want Kolibri to generate a launcher for that channel.

But remember, Kolibri on Endless OS is an unprivileged system service. It can’t talk to the DynamicLauncher portal. That belongs to the user’s session, and we want these launchers to be visible before a user ever starts Kolibri in their own session. Kolibri also can’t be creating files in /usr/share/applications. That would be far too much responsibility.

Instead, we add a Kolibri plugin to generate desktop entries for channels. The desktop entries refer to the Kolibri app using a custom URI scheme, a layer of indirection because Kolibri (potentially inside a flatpak) is unaware of how the host system launches it. The URI scheme provides enough information to start in a channel-specific app mode, instead of in its default configuration.

Finally, instead of placing the desktop entry files in one of the usual places, we place them in a well-known location inside Kolibri’s data directory. That way the channel apps are available, but not visible by default.

In Endless OS, the channel launchers end up in /var/lib/kolibri/data/content/xdg, so in our system configuration we add that directory to XDG_DATA_DIRS. This turns out to be a good choice, because it is trivial to start generating search providers for those apps, as well.

Search providers

To make sure people can find everything we’ve included in Endless OS, we add as many desktop search providers as we can think of, and we encourage users to explore them. The search bar in Endless OS is not just for apps.

That means we need a search provider for Kolibri. It’s a simple enough problem. We extended kolibri-daemon‘s D-Bus interface with its own equivalents for the GNOME Shell search provider interface. It is capable of reading directly from Kolibri’s database, so we can avoid starting an HTTP server. But we also want to avoid dealing with kolibri-daemon as much as possible. It is a Python process, heavy with web server stuff and complicated multiprocessing code. And, besides, the daemon could be connecting to the D-Bus system bus, and the shell only talks to search providers on the session bus. That’s why the search provider itself is a separate proxy application, written in C.

But in Endless OS, we don’t just need one search provider, either. We want one for each of those channel apps we generated. So, I mentioned that our Kolibri plugin generates a search provider to go with each desktop file. Of course, loading and searching through Kolibri’s sqlite database is already expensive once, so it would be absurd to do it for every channel that is installed. That’s a lot of processes!

Fortunately, those search providers are all the same D-Bus service, with a different object path for each Kolibri channel. That one D-Bus service receives a lot of identical search queries for a lot of different object paths, but at least the system is only starting one process for it all. In the search provider code, I added a bespoke task multiplexer, which allows the service to run a single search in kolibri-daemon for a given query, then group the results and return them to different invocations from the shell.

It is a complicated workaround, but it means search results appear in distinct buckets with meaningful names and icons. For our purpose in Endless OS, it was definitely worth the trouble.

User accounts

There was one last wrinkle here: Kolibri kept asking people to set it up, make a user account (with a password!), and sign in. It is, after all, a standalone learning app with a big complicated database that keeps track of learning progress and understands how to sync content between devices. But this isn’t a great experience if you’re just here to watch that lecture about cats.

What we want is for Kolibri to already know who is accessing it. They’re already signed in as a desktop user. And most of the time, we want to blaze right through that initial “set up your device” step, or at least make it as smooth as possible.

To do that, we added an interface in kolibri-daemon so the desktop app can get an authentication token to use over HTTP. On the other side, kolibri-daemon privately communicates with Kolibri itself to verify an authentication token, and it communicates with logind to build a profile for the authenticating user.

It was ugly at first, with a custom kolibri-desktop-auth-plugin which sat on top of Kolibri’s authentication system. But after some iteration, upstream Kolibri now has its own understanding of desktop users. On the surface, it uses Kolibri’s app interface plugin for platform integration. With the newest version of Kolibri we have been able to solve authentication in a way that I am properly happy with.

My favourite part of the feature has been seeing it come together with Kolibri’s first run wizard. Given a working authentication token, Kolibri knows to skip creating an initial user account, leaving only some simple questions about how the user is planning to use Kolibri; independently or connecting to an existing classroom.

That’s it!

It has been great to work on the Kolibri desktop app, and I expect to take some of the approaches and lessons here over to other projects. It is the first big new Python desktop app I have worked with, and it was interesting using some modern Python tools in tandem with the GNOME ways of doing things. The resulting codebase has some fun details:

• The source repository includes a Flatpak manifest, so it builds and runs out of the box in GNOME Builder. As soon as that was working, I used Builder for everything.
• Meson is truly indispensable for this kind of thing. We’re sharing build configuration between a bunch of Python modules, all sorts of configuration and data files, and a pair of C projects – one of which is imported by a Python module using GObject introspection. This all works (in a mere 577 lines of meson.build, if you’re counting) because the build system is language-agnostic, and I love it for that. I know that isn’t a lot to ask, but the go-to for Python is decidedly not language-agnostic, and I do not love it.
• We added pre-commit to automatically clean up source files and run quick tests against them. It doesn’t actually require you have a Python codebase, but it is written in Python and I think people are afraid of how Pythony it looks? It’s really convenient, and it does a good job taking care of the usual nightmare of setting up a virtual environment to run all its tools. I often don’t bother with the actual git hook part, and instead I remember to run the thing manually, and we use the pre-commit github action to be sure.
• At some point, I added Python type hinting to every part of the project. This tremendously improved the development experience with Builder, and it allowed me to add a mypy pre-commit hook to catch mistakes.
• I got annoyed at the problem of needing to write release notes in the appdata file before knowing what the next release is called, so I devised a fun scheme where we add notes under "{current_version} next", and then bump-my-version (another tool that looks very Pythony but everyone should use it) knows to mark that release entry as released, setting the date and version appropriately. I wish it didn’t involve regex, but as a concept it has been nice to use. I was tempted to write a pre-commit hook which actually insists on an up to date “next release” entry in appdata, but I should find another project to try it with.
• With that said, a better workflow probably involves appstream-util news-to-appdata.
• Managing history in WebKit can be tricky because the BackForwardList is read-only. That was an issue with the Kolibri app because we (with our UI consisting almost entirely of a WebView) need to communicate about Kolibri’s state before its HTTP server is running. Kolibri upstream provides a static HTML loading screen for this purpose, which is fine, but now we have this file in our WebView’s back / forward list. I solved it by swapping between different WebViews, and later showing one in a dialog just for Kolibri’s setup wizard. At first, that was all to keep the history stack organized, but at the same time I found it made the app feel a little less like a web browser in a trench coat. We can switch from the loading WebView to the real thing with a nice crossfade, and only when the UI is actually for real finished loading.
• This whole project uses a lot of GObject throughout. At some point I finally read the pygobject manual and found myself happily doing property binding, signals and async functions and all those good things from Python. It was a much better experience than earlier in the project’s life where there was a type of angry mishmash between vanilla Python and GObject. (The thing that really freed this up was when I moved a lot of D-Bus code over to a C helper library with gdbus-codegen, which allowed me to delete the equivalent duplicative Python code, and also introduced a bunch more GObject). It’s easy to see why GObject works best with a language that doesn’t carry its own big standard library, but I was happy with how productive I could be in Python once I started actively preferring GObject, especially with the various magic helpers provided by PyGObject. In a future starting-from-scratch project, I would be tempted to make that a rule when adding imports and writing new classes.
• I made many commits here because I am obsessive about silly things, but this all works thanks to the genius and hard work of the folks at Learning Equality, as well as everyone at Endless, including Dan Nicholson, Daniel Garcia Moreno, Georges Stavracas, Jian-Hong Pan, Manuel Quiñones, and Will Thompson.

I have to admit I got carried away with certain aspects of this. In the end there is a certain discontent to be had spending creative energy on what is, from many angles, a glorified web browser. It’s frustrating when the web stack leads us to treat an application as a black box behind an HTTP interface, which makes integration difficult: boot it up (in its own complex runtime environment which is heroically not a Docker container); wait until it is ready (Kolibri is good at this, but sometimes you’re just watching a file or polling some well-known port); authenticate; ask it (over HTTP) some trivial question that amounts to a single SQL command; return None. But look at that nice framework we’re using!

At the same time, it isn’t lost on me that a software stack like Kolibri’s simply is a popular choice for a cross-platform app. It’s worth understanding how to work with it in a way that still does the best we can to be useful, efficient, and comfortable to use.

Beyond all the tech stuff, I want to emphasize that Kolibri is an exceptionally cool project. I truly admire what Learning Equality are doing with it, and if you’re interested in offline-first content, data sovereignty, or just open source learning in general, I highly recommend checking it out – either our app on Flathub, or at learningequality.org/kolibri.

I attended GUADEC 2024 last month in Denver, Colorado. I thought I’d write about some of the highlights for me.

It was definitely the smallest GUADEC I’ve been to, and it was unusual in some other ways too, such as having several hybrid presentations, with remote and in-person presenters sharing the stage. That took some adjusting, but it worked well, even if I missed some of the energy of past events. (I shared some thoughts about hybrid GUADEC on a Discourse thread).

I felt this GUADEC was really defined by the keynotes. They were great!

First, we had Ryan Stipes from Thunderbird telling us all about Thunderbird’s journey from a somewhat neglected but well-loved side project and on to a thriving self-funded community project: Thunderbird, The Death and Rebirth of an OSS Project (YouTube). He had a lot to say about the value of metrics to measure the impact of certain features and target platforms, which really resonated with people. (It is interesting to note, for instance, there appear to be more Thunderbird users using Windows 8.1 than Linux). He also had a lot to say about the success Thunderbird had just being direct and asking users for money.

Much of this success comes from Thunderbird doing a good job telling its own story. People clearly understand what Thunderbird is doing for them. And there was plenty of talk for the next few days: what does it mean for GNOME to own its story?

I also really enjoyed Stephanie Taylor’s keynote, all about Google Summer of Code (which started 20 years ago now!): Google Summer of Code 20 years of OSS Mentorship (YouTube). It just made me super happy as a GSoC alumni (one of thousands!) to see that program continuing to do so much good, and how much mentorship in open source has grown over the years.

Scott Jenson’s presentation, How can GNOME explore bigger concepts? (YouTube), is another really important watch. Scott’s advice about breaking free from traps like constraint thinking really resonated with me, especially his suggestion to, at first, treat the software like it is magic and see where that leads.

That approach reminds me of how software improves in implementation, as well. It is natural for a codebase to start off with a whole bunch of “do magic” stub functions, then slowly morph into a chaotic mess until finally it turns into something that actually might just work. And getting to that last step usually involves deleting a lot of code, after it turns out you never needed all that much magic. But you have to be patient with the chaos to get there. You have to believe in it.

Speaking of magic, there is so much around GNOME that is exciting right now, so I spent some time just being excited about things.

• I am always grateful for presentations about GTK and font rendering, which was satisfied on Friday with GTK’s new renderers – a deep dive (YouTube), and later with Advances in Font Technology (YouTube) – all great to watch if you’re interested in rendering stuff, or just fun Unicode trivia.
• Eitan Isaacson talked about Spiel, a modern speech synthesis system: The Whole Spiel – A New Speech Synthesis API (YouTube). I loved his examples showing how it is important to satisfy several very different use cases for speech synthesis. While one user may value the precision of eSpeak at chipmunk speed, another user would prefer their computer talks like a human. And if we can get speech synthesis working well for non-accessibility reasons, there’s a real curb cut effect that should benefit everyone, including people who are just starting to use accessibility tools.
• I went to the newest edition of Jonathan Blandford and Federico Mena Quintero’s presentation about Crosswords, GNOME Crosswords, Year Three (YouTube). It was abridged due to the format, but I especially enjoyed learning about the MVC-like data model for the application. It would be neat to see more GNOME apps using the same pattern.
• There was a lot to learn about GNOME OS and OpenQA testing. The process for a new developer to get into hacking on a GNOME system component tends to be really awkward – particularly if that developer doesn’t want to mess up their host system. So You’re always breaking GNOME (YouTube) got me pretty excited about what’s coming with GNOME OS and sysext, as well as for testing in general. The OpenQA workshop on Monday was also well attended. Some people were unclear what openqa.gnome.org was doing, or what it can do for them. Just stepping through some conveniently broken tests and fixing them together was an excellent way to demystify the thing.
• Also the lightning talks are good. Always watch the lightning talks.

Much of this work is being helped along with the Sovereign Tech Fund. This is the GUADEC where a lot of that is up for show, and I think it’s amazing to see so many quiet but high impact projects finally getting the attention (and funding) they deserve.

Outside of the event, it was great hanging out around Denver with all sorts of GNOME folks. I loved how many restaurants were perfectly happy to accommodate giant mobs of people. We saw huge mountains, the Colorado Rockies winning a baseball game, surprisingly good karaoke, and some very unique bars. On the last day, a large contingent of us headed to Meow Wolf, which was just a ridiculously fun way to spend a few hours. It reminded me of a point and click adventure game in the style of Myst and Riven, in all the best ways.

I was also suitably impressed by the 35 minute walk from where I was staying, around Empower Field, over the South Platte River, under some giant highway … which was actually entirely pleasant, for North America. This part of Denver has plenty of pedestrian bridges, which are both nice to walk along and really helpful to guide pedestrians through certain areas, so for me the obvious walking routes were completely different from (and as efficient as) the obvious driving routes.

The GUADEC dinner was, for me, the ideal GUADEC dinner. It was right there at the venue, at the same brewery people had been going to every day – but this time with free tacos! I truly appreciated the consistency there, for Denver has good beer and good tacos. I also appreciated that we were set up both inside and outside, at nice big tables with plenty of room for people to sit. It helped me to feel comfortable, and it was great for people who were there with families (which meant I got to meet said families!). It reminded me of the GUADEC 2022 taco party. An event like this really shines when people are moving around, and there was a lot of it here.

It turns out I didn’t take many pictures this year, but the official ones are better anyway. I did, however, take far too many pictures from the train ride home: I rode Amtrak, mostly for fun, on California Zephyr from Denver to Sacramento; then Coast Starlight from Sacramento to Seattle; and the smaller Cascades train from Seattle to Vancouver. It was beautiful, and seriously I think everyone should have the opportunity to try an overnight roomette on the Zephyr. My favourite part was sitting in the spacious observation car watching the world go by, getting only the tiniest amount of work done. I found tons of fun people to talk to, which I don’t usually do, but something about that space made it oddly comfortable. Everyone there was happy and sociable and relaxed. And I guess I was still in conference mode.

I returned home refreshed and excited for where GNOME is heading, especially with new progress around accessibility and developer tools. And with plenty of ideas for little projects I can work on this year.

Thanks to all the awesome people who make GUADEC happen, as well as my employer, Endless OS Foundation, for giving me the opportunity to spend several work days meeting people from around the GNOME community and wandering around Denver.