New responsibilities

As part of the same process outlined in Matthias Clasen's "LibreOffice packages" email, my management chain has made the decision to stop all upstream and downstream work on desktop Bluetooth, multimedia applications (namely totem, rhythmbox and sound-juicer) and libfprint/fprintd. The rest of my upstream and downstream work will be reassigned depending on Red Hat's own priorities (see below), as I am transferred to another team that deals with one of a list of Red Hat’s priority projects.

I'm very disappointed, because those particular projects were already starved for resources: I spent less than 10% of my work time on them in the past year, with other projects and responsibilities taking most of my time.

This means that, in the medium-term at least, all those GNOME projects will go without a maintainer, reviewer, or triager:

- gnome-bluetooth (including Settings panel and gnome-shell integration)

- totem, totem-pl-parser, gom

- libgnome-volume-control

- libgudev

- geocode-glib

- gvfs AFC backend

Those freedesktop projects will be archived until further notice:

- power-profiles-daemon

- switcheroo-control

- iio-sensor-proxy

- low-memory-monitor

I will not be available for reviewing libfprint/fprintd, upower, grilo/grilo-plugins, gnome-desktop thumbnailer sandboxing patches, or any work related to XDG specifications.

Kernel work, reviews and maintenance, including recent work on SteelSeries headset and Logitech devices kernel drivers, USB revoke for Flatpak Portal support, or core USB is suspended until further notice.

All my Fedora packages were orphaned about a month and a half ago, it's likely that there are still some that are orphaned, if there are takers. RHEL packages were unassigned about 3 weeks ago, they've been reassigned since then, so I cannot point to the new maintainer(s).

If you are a partner, or a customer, I would recommend that you get in touch with your Red Hat contacts to figure out what the plan is going forward for the projects you might be involved with.

If you are a colleague that will take on all or part of the 90% of the work that's not being stopped, or a community member that was relying on my work to further advance your own projects, get in touch, I'll do my best to accommodate your queries, time permitting.

I'll try to make sure to update this post, or create a new one if and when any of the above changes.

Speeding up the kernel testing loop

When I create kernel contributions, I usually rely on a specific hardware, which makes using a system on which I need to deploy kernels too complicated or time-consuming to be worth it. Yes, I'm an idiot that hacks the kernel directly on their main machine, though in my defense, I usually just need to compile drivers rather than full kernels.

But sometimes I work on a part of the kernel that can't be easily swapped out, like the USB sub-system. In which case I need to test out full kernels.

I usually prefer compiling full kernels as RPMs, on my Fedora system as it makes it easier to remove old test versions and clearly tag more information in the changelog or version numbers if I need to.

Step one, build as non-root

First, if you haven't already done so, create an ~/.rpmmacros file (I know...), and add a few lines so you don't need to be root, or write stuff in /usr to create RPMs.

$ cat ~/.rpmmacros
%_topdir        /home/hadess/Projects/packages
%_tmppath        %{_topdir}/tmp

Easy enough. Now we can use fedpkg or rpmbuild to create RPMs. Don't forget to run those under “powerprofilesctl launch” to speed things up a bit.

Step two, build less

We're hacking the kernel, so let's try and build from upstream. Instead of the aforementioned fedpkg, we'll use “make binrpm-pkg” in the upstream kernel, which builds the kernel locally, as it normally would, and then packages just the binaries into an RPM. This means that you can't really redistribute the results of this command, but it's fine for our use.

 If you choose to build a source RPM using “make rpm-pkg”, know that this one will build the kernel inside rpmbuild, this will be important later.

 Now that we're building from the kernel sources, that's our time to activate the cheat code. Run “make localmodconfig”. It will generate a .config file containing just the currently loaded modules. Don't forget to modify it to include your new driver, or driver for a device you'll need for testing.

Step three, build faster

If running “make rpm-pkg” is the same as running “make ; make modules” and then packaging up the results, does that mean that the “%{?_smp_mflags}” RPM macro is ignored, I make you ask rhetorically. The answer is yes. “make -j16 rpm-pkg”. Boom. Faster.

Step four, build fasterer

As we're building in the kernel tree locally before creating a binary package, already compiled modules and binaries are kept, and shouldn't need to be recompiled. This last trick can however be used to speed up compilation significantly if you use multiple kernel trees, or need to clean the build tree for whatever reason. In my tests, it made things slightly slower for a single tree compilation.

$ sudo dnf install -y ccache
$ make CC="ccache gcc" -j16 binrpm-pkg

Easy.

And if you want to speed up the rpm-pkg build:

$ cat ~/.rpmmacros
[...]
%__cc            ccache gcc
%__cxx            ccache g++

More information is available in Speeding Up Linux Kernel Builds With Ccache.

Step five, package faster

Now, if you've implemented all this, you'll see that the compilation still stops for a significant amount of time just before writing “Wrote kernel...rpm”. A quick look at top will show a single CPU core pegged to 100% CPU. It's rpmbuild compressing the package that you will just install and forget about.

$ cat ~/.rpmmacros
[...]
%_binary_payload    w2T16.xzdio

More information is available in Accelerating Ceph RPM Packaging: Using Multithreaded Compression.

TL;DR and further work

All those changes sped up the kernel compilation part of my development from around 20 minutes to less than 2 minutes on my desktop machine.

$ cat ~/.rpmmacros
%_topdir        /home/hadess/Projects/packages
%_tmppath        %{_topdir}/tmp
%__cc            ccache gcc
%__cxx            ccache g++
%_binary_payload    w2T16.xzdio

$ powerprofilesctl launch make CC="ccache gcc" -j16 binrpm-pkg

I believe there's still significant speed ups that could be done, in the kernel, by parallelising some of the symbols manipulation, caching the BTF parsing for modules, switching the single-threaded vmlinux bzip2 compression, and not generating a headers RPM (note: tested this last one, saves 4 seconds :)

 

The results of my tests. YMMV, etc.

Command	Time spent	Notes
koji build --scratch --arch-override=x86_64 f36 kernel.src.rpm	129 minutes	It's usually quicker, but that day must have been particularly busy
fedpkg local	70 minutes	No rpmmacros changes except setting the workdir in $HOME
powerprofilesctl launch fedpkg local	25 minutes
localmodconfig / bin-rpmpkg	19 minutes	Defaults to "-j2"
localmodconfig -j16 / bin-rpmpkg	1:48 minutes
powerprofilesctl launch localmodconfig ccache -j16 / bin-rpmpkg	7 minutes	Cold cache
powerprofilesctl launch localmodconfig ccache -j16 / bin-rpmpkg	1:45 minutes	Hot cache
powerprofilesctl launch localmodconfig xzdio -j16 / bin-rpmpkg	1:20 minutes

GMemoryMonitor (low-memory-monitor, 2nd phase)

TL;DR

Use GMemoryMonitor in glib 2.63.3 and newer in your applications to lower overall memory usage, and detect low memory conditions.

low-memory-monitor

To start with, let's come back to low-memory-monitor, announced at the end of August.

It's not really a “low memory monitor”. I know, the name is deceiving, but it actually monitors memory pressure stalls, and how hard it is for the kernel to allocate memory when applications need it. The longer it takes to allocate memory, the longer the kernel takes to allocate it, usually because it needs to move memory around to make room for a big allocation, when an application starts up for example, or prepares an in-memory buffer for saving.

It is not a daemon that will kill programs on low memory. It's not a user-space out-of-memory killer, and does not take those policy decisions. It can however be configured to ask the kernel to do that. The kernel doesn't really know what it's doing though, and user-space isn't helping either, so best disable that for now...

As listed in low-memory-monitor's README (and in the announcement post), there were a number of similar projects around, but none that would offer everything we needed, eg.:

• Has a D-Bus interface to propagate low memory conditions
• Requires Linux 5.2's kernel memory pressure stalls information (Android's lowmemorykiller daemon has loads of code to get the same information from the kernel for older versions, and it really is quite a lot of code)
• Written in a compiled language to save on startup/memory usage costs (around 500 lines of C code, as counted by sloccount)
• Built-in policy, based upon values used in Android and Endless OS

 GMemoryMonitor

Next up, in our effort to limit memory usage, we'll need some help from applications. That's where GMemoryMonitor comes in. It's simple enough, listen to the low-memory-warning signal and free some image thumbnails, index caches, or dump some data to disk, when you receive a signal.

The signal also gives you a “warning level”, with 255 being when low-memory-monitor would trigger the kernel's OOM killer, and lower values different levels of “try to be a good citizen”.

The more astute amongst you will have noticed that low-memory-monitor runs as root, on the system bus, and wonder how those new fangled (5 years old today!) sandboxed applications would receive those signals. Fear not! Support for a portal version of GMemoryMonitor landed in xdg-desktop-portal on the same day as in glib. Everything tied together with installed tests that use the real xdg-desktop-portal to test the portal and unsandboxed versions.

How about an OOM killer?

By using memory pressure stall information, we receive information about the state of the kernel before getting into swapping that'd cause the machine to become unusable. This also means that, as our threshold for keeping everything ticking is low, if we were to kill high memory consumers, we'd get a butter smooth desktop, but, based on my personal experience, your browser and your mail client would take it in turns disappearing from your desktop in a way that you wouldn't even notice.

We'll definitely need to think about our next step in application state management, and changing our running applications paradigm.

Distributions should definitely disable the OOM killer for now, and possibly try their hands at upstream some systemd OOMPolicy and OOMScoreAdjust options for system daemons.

Conclusion

Creating low-memory-monitor was easy enough, getting everything else in place was decidedly more complicated. In addition to requiring changes to glib, xdg-desktop-portal and python-dbusmock, it also required a lot of work on the glib CI to save me from having to write integration tests in C that would have required a lot of scaffolding. So thanks to all involved in particular Philip Withnall for his patience reviewing my changes.

low-memory-monitor: new project announcement

I'll soon be flying to Greece for GUADEC but wanted to mention one of the things I worked on the past couple of weeks: the low-memory-monitor project is off the ground, though not production-ready.

low-memory-monitor, as its name implies, monitors the amount of free physical memory on the system and will shoot off signals to interested user-space applications, usually session managers, or sandboxing helpers, when that memory runs low, making it possible for applications to shrink their memory footprints before it's too late either to recover a usable system, or avoid taking a performance hit.

It's similar to Android's lowmemorykiller daemon, Facebook's oomd, Endless' psi-monitor, amongst others

Finally a GLib helper and a Flatpak portal are planned to make it easier for applications to use, with an API similar to iOS' or Android's.

Combined with work in Fedora to use zswap and remove the use of disk-backed swap, this should make most workstation uses more responsive and enjoyable.

Gaming hardware support

While my colleagues are working on mice that shine in all kinds of different colours, I went towards the old school.

For around 10 units of currency, you should be able to find the uDraw tablet for the PlayStation 3, the drawing tablet that brought down a company.

The device contains a large touchpad which can report one or two touches, for right-clicking (as long as the fingers aren't too close), a pen interface which will make the cheapest of the cheapest Wacom tablets feel like a professional tool from 30 years in the future, a 4-button joypad (plus Start/Select/PS) with the controls either side of the device, and an accelerometer to play Marble Madness with.

The driver landed in kernel 4.10. Note that it only supports the PlayStation 3 version of the tablet, as the Wii and XBox 360 versions require receivers that aren't part of the package. Here, a USB dongle should be provided.

Recommended for: point'n'click adventure games, set-top box menu navigation.

The second driver landed in kernel 4.12, and is a primer for more work to be done. This driver adds support for the Retrode 2's joypad adapters.

The Retrode is a USB console cartridge reader which makes Sega Mega Drive (aka Genesis) and Super Nintendo (aka Super Famicom) cartridges show up as files on a mass storage devices in your computer.

It also has 4 connectors for original joypads which the aforementioned driver now splits up and labels, so you know which is which, as well as making the mouse work out of the box. I'd still recommend picking up the newer optical model of that mouse, from Hyperkin. Moving a mouse with a ball in it is like weighing a mobile phone from that same era.

I will let you inspect the add-ons for the device, like support for additional Nintendo 64 pads and cartridges, and Game Boy/GB Color/GB Advance, and Sega Master System adapters.

Recommended for: cartridge-based retro games, obviously.

Integrated firmware updates, and better integration with Games is in the plans.

I'll leave you with this video, which shows how you could combine GNOME Games, a Retrode, this driver, a SNES mouse, and a cartridge of Mario Paint. Let's get creative :)

Hardware support news

Trackballs

I dusted off (literally) my Logitech Marble trackball to replace the Intuos tablet + mouse combination that I was using to cut down on the lateral movement of my right arm which led to back pains.

Not that you care about that one bit, but that meant that I needed a way to get a scroll wheel working with this scroll-wheel less trackball. That's now implemented in gnome-settings-daemon for GNOME 3.16. You'd run:

gsettings set org.gnome.settings-daemon.peripherals.trackball scroll-wheel-emulation-button 8

With "8" being the mouse button number to use to make the trackball ball into a wheel. We plan to add an interface to configure this in the Settings.

Touchscreens

Touchscreens are now switched off when the screensaver is on. This means you'll usually need to use one of the hardware buttons on tablets, or a mouse or keyboard on laptops to turn the screen back on.

Note that you'll need a kernel patch to avoid surprises when the touchscreen is re-enabled.

More touchscreens

The driver for the Goodix touchscreen found in the Onda v975w is now upstream as well.

A GNOME Kernel wishlist

GNOME has long had relationships with Linux kernel development, in that we would have some developers do our bidding, helping us solve hard problems. Features like inotify, memfd and kdbus were all originally driven by the desktop.

I've posted a wishlist of kernel features we'd like to see implemented on the GNOME Wiki, and referenced it on the kernel mailing-list.

I hope it sparks healthy discussions about alternative (and possibly existing) features, allowing us to make instant progress.

And now for some hardware (Onda v975w)

Prodded by Adam Williamson's fedlet work, and by my inability to getting an Android phone to display anything, I bought an x86 tablet.

At first, I was more interested in buying a brand-name one, such as the Dell Venue 8 Pro Adam has, or the Lenovo Miix 2 that Benjamin Tissoires doesn't seem to get enough time to hack on. But all those tablets are around 300€ at most retailers around, and have a smaller 7 or 8-inch screen.

So I bought a "not exported out of China" tablet, the 10" Onda v975w. The prospect of getting a no-name tablet scared me a little. Would it be as "good" (read bad) as a PadMini or an Action Pad?

Vrrrroooom.

Well, the hardware's pretty decent, and feels rather solid. There's a small amount of light leakage on the side of the touchscreen, but not something too noticeable. I wish it had a button on the bezel to mimick the Windows button on some other tablets, but the edge gestures should replace it nicely.

The screen is pretty gorgeous and its high DPI triggers the eponymous mode in GNOME.

With help of various folks (Larry Finger, and the aforementioned Benjamin and Adam), I got the tablet to a state where I could use it to replace my force-obsoleted iPad 1 to read comic books.

I've put up a wiki page with the status of hardware/kernel support. It's doesn't contain all my notes just yet (sound is working, touchscreen will work very very soon, and various "basic" features are being worked on).

I'll be putting up the fixed-up Wi-Fi driver and more instructions about installation on the Wiki page.

And if you want to make the jump, the tablets are available at $150 plus postage from Aliexpress.

Update: On Google+ and in comments of this blog, it was pointed out that the seller on Aliexpress was trying to scam people. All my apologies, I just selected the cheapest from this website. I personally bought it on Amazon.fr using NewTec24 FR as the vendor.

GUADEC[1] Hardware giveaway

I'm cleaning up my hardware chest, and giving away some hardware to a good home. I intend on travelling with those that I found a new home for at GUADEC. All of them are in good working condition. If there is a charger or power supply, it will be a UK one.

Drop me a mail with your intended usage (preferably GNOME or kernel related), or need some more info about the devices.

Up for grabs

Palm Pilot Tungsten E2

The predecessor to all your new-fangled smartphones. This one could even do Bluetooth syncing using gnome-pilot, all those years ago. Might be nice as a remote control of some sort, or legacy support for Pilots.

D-Link DIR-615 Wi-Fi N router

Works with DD-WRT. Would be great to work with DD-WRT or associated on a way to configure those through a GNOME UI à-la Airport base stations.

HP iPaq 914

Euro plug. Apparently this can't run Linux... Yet!

DXR3 card

Offload your MPEG2 decoding to this PCI card.

iPod Touch 2G

Too old to run any recent iOS, but good enough to show off your web apps skills, or work on Notes sync with IMAP servers.

Broadcom Crystal HD mini-PCIE 70015 and 70012

2 video decoder cards usable with Linux. You'd need to port the GStreamer plugin to GStreamer 1.0 to get those (or one of those at least).

Plantronics and Motorola Bluetooth headsets

Not the newest devices, but they work.

Red Hat branded power adapter

USB to Nokia/Motorola with this retractable extension lead.

On their way to a good home [2]

Logitech MX 5000 pack and diNovo keyboard

Space-age mouse and keyboard set. Benjamin Tissoires will be getting the (now not so much space age as grubby and outdated) pack to hopefully implement HID++ 1.0 in the kernel.

The diNovo keyboard is a nice little Bluetooth keyboard for a media PC or the likes, even has the tiniest of trackpads.

Logitech 9000, PS3 Eye and Creative OV511-based webcams

Hans de Goede will get those to make them work out-of-the-box in Fedora, and upstream, trying to clean up some hacks he gave me for those a long time ago.

No-name USB GPS dongle and Tom-Tom Bluetooth GPS

For Zeeshan, just in case he gets bored implementing geoclue2.

Nokia N82 and Palm Centro

For Dan Williams. ModemManager's testing gear is growing.

Belkin G and Dell 1450 Wi-Fi USB dongles

Giovanni and Jasper will enjoy those Wi-Fi dongles that will create bugs in gnome-shell's network menu and the new aggregate menu.

[1]: Or near enough for some of the items :)

[2]: I made my pre-selection based on the possible uses for the hardware.

Notes on Apple IR remotes reverse-engineering

In 2009, I wrote a driver to make the infra-red remote on my original MacBookAir work out-of-the-box on Linux. The driver was rejected upstream on the basis that the device would soon be supported through more generic means. In the meanwhile, it lived in Fedora's kernel tree, and I took some notes about implementing pairing, so that only your remote would work with your computer.

I'm posting this now because I wanted to poke at a MacOS X application today, and couldn't for the life of me remember the name of the program to monitor disk-activity. Hope this finds its way to a search engine near you.

• Launched the System Preferences, Security, and unlock the panel.
• In a terminal: sudo fs_usage -f filesys -w and check the output when enabling/disabling the remote.
• We can see the modified file is /Library/Preferences/com.apple.driver.AppleIRController.plist
• Installed PlistEditPro and opened the file up.
• Now try to pair a remote (menu and next together)
• You can see the UID value changing in the file. I named the remotes I had available to me:
• New remote: UID = 145 
• Old clean remote: UID = 24
• Old dirty remote: UID = 227 
• After adding some debug to the aforementioned appleir driver, in Linux, I got:
• New remote: appleir: received (5 bytes) 25 87 e0 91 02
• Old clean remote: appleir: received (5 bytes) 25 87 e0 18 03
• Old dirty remote: appleir: received (5 bytes) 25 87 e0 e3 02
• So the 4th byte is the remote's UID.

Now one could implement remote pairing using a sysfs attribute, a udev helper to apply the pairing across reboots, and PolicyKit helper to set and save the paired UID.

This will be left as an exercise to the reader :)

infra-red remotes in GNOME 3

gnome-lirc-properties has served its purpose. It will probably carry on working on GNOME 3 desktops, but you won't be happy when it drags in GTK+ 2.x Python bindings, HAL or doesn't integrate into the new control-center.

But things have changed since gnome-lirc-properties was first written, and the way to handle IR remotes has changed as well:

• A large majority of receivers are now supported in the kernel using rc-core (né ir-core).
• Some receivers aren't supported (iguanaIR amongst others), and some need porting from pure input drivers to rc-core. Some functionality for the ATI Remote Wonder remotes is also not supported by the new drivers. If you're interested in working on this, drop a mail to the LIRC list.
• Mauro Chehab is making progress at propagating the key events from the kernel up to the stack to X11 applications. There's some patches in that direction on the Red Hat Bugzilla.

This means that:

• Event delivery would still need a broker in the session, to get to unfocused applications. gnome-settings-daemon can step in that role (and step out of the way when the application is focused, so the app can bring context dependent behaviour). gnome-settings-daemon already handle some of the more common multimedia keys in its media-keys plugin.
• The only configuration one would need to do is selecting the type of remote for the receiver, eventually tweaking the keymap for that remote.

So to write a replacement for gnome-lirc-properties that would fit into GNOME 3, one would need:

• A way to enumerate receivers on the machine
• A way to change the remote configuration (changing the keymap) for that receiver
• Eventually a way to tweak the keymap

This could all be handled through a D-Bus version of ir-keytable. If Mauro's patches reach X.org mainstream, then a kernel/GNOME summer of code project could be had for this work. Best to start writing some kernel patches, or laying some code if you want to get a headstart.

PS: For completeness' sake, there are also "pure" input devices that are remotes that wouldn't be handled through this. Those would need to be blacklisted in the input layer, and handled through rc-core instead.

I'm upstream!

Or at least, my Wacom Bluetooth tablet driver is. I was wondering in which tree it was lost. You'll still need a patch to bluetoothd though.

Thanks kernel people

Whoever is responsible for making the rt73usb driver work great out of the box: THANK YOU. I tried it without success when I installed Fedora 8 gold, and now it works brilliantly (and out-of-the-box) on Fedora 9.

Current hacking includes: GPRS/3G support via Bluetooth in NetworkManager, fprintd hacking, and gnome-lirc-properties integration into Fedora (Debian and Ubuntu people, upstream your bleeding patches, kthx).

And for nanobob and Borkis on FIFA: you really didn't need to quit the game when I scored those 2nd goals. Losing against a guy full of margaritas must hurt.