free software

Waiting for a Linux system to be online

by
Designed by Freepik

What is an “online” system?

Networking is a complex topic, and there is lots of confusion around the definition of an “online” system. Sometimes the boot process gets delayed up to two minutes, because the system still waits for one or more network interfaces to be ready. Systemd provides the network-online.target that other service units can rely on, if they are deemed to require network connectivity. But what does “online” actually mean in this context, is a link-local IP address enough, do we need a routable gateway and how about DNS name resolution?

The requirements for an “online” network interface depend very much on the services using an interface. For some services it might be good enough to reach their local network segment (e.g. to announce Zeroconf services), while others need to reach domain names (e.g. to mount a NFS share) or reach the global internet to run a web server. On the other hand, the implementation of network-online.target varies, depending on which networking daemon is in use, e.g. systemd-networkd-wait-online.service or NetworkManager-wait-online.service. For Ubuntu, we created a specification that describes what we as a distro expect an “online” system to be. Having a definition in place, we are able to tackle the network-online-ordering issues that got reported over the years and can work out solutions to avoid delayed boot times on Ubuntu systems.

In essence, we want systems to reach the following networking state to be considered online:

  1. Do not wait for “optional” interfaces to receive network configuration
  2. Have IPv6 and/or IPv4 “link-local” addresses on every network interface
  3. Have at least one interface with a globally routable connection
  4. Have functional domain name resolution on any routable interface

A common implementation

NetworkManager and systemd-networkd are two very common networking daemons used on modern Linux systems. But they originate from different contexts and therefore show different behaviours in certain scenarios, such as wait-online. Luckily, on Ubuntu we already have Netplan as a unification layer on top of those networking daemons, that allows for common network configuration, and can also be used to tweak the wait-online logic.

With the recent release of Netplan v1.1 we introduced initial functionality to tweak the behaviour of the systemd-networkd-wait-online.service, as used on Ubuntu Server systems. When Netplan is used to drive the systemd-networkd backend, it will emit an override configuration file in /run/systemd/system/systemd-networkd-wait-online.service.d/10-netplan.conf, listing the specific non-optional interfaces that should receive link-local IP configuration. In parallel to that, it defines a list of network interfaces that Netplan detected to be potential global connections, and waits for any of those interfaces to reach a globally routable state.

Such override config file might look like this:

[Unit]
ConditionPathIsSymbolicLink=/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service

[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-networkd-wait-online -i eth99.43:carrier -i lo:carrier -i eth99.42:carrier -i eth99.44:degraded -i bond0:degraded
ExecStart=/lib/systemd/systemd-networkd-wait-online --any -o routable -i eth99.43 -i eth99.45 -i bond0

In addition to the new features implemented in Netplan, we reached out to upstream systemd, proposing an enhancement to the systemd-networkd-wait-online service, integrating it with systemd-resolved to check for the availability of DNS name resolution. Once this is implemented upstream, we’re able to fully control the systemd-networkd backend on Ubuntu Server systems, to behave consistently and according to the definition of an “online” system that was lined out above.

Future work

The story doesn’t end there, because Ubuntu Desktop systems are using NetworkManager as their networking backend. This daemon provides its very own nm-online utility, utilized by the NetworkManager-wait-online systemd service. It implements a much higher-level approach, looking at the networking daemon in general instead of the individual network interfaces. By default, it considers a system to be online once every “autoconnect” profile got activated (or failed to activate), meaning that either a IPv4 or IPv6 address got assigned.

There are considerable enhancements to be implemented to this tool, for it to be controllable in a fine-granular way similar to systemd-networkd-wait-online, so that it can be instructed to wait for specific networking states on selected interfaces.

A note of caution

Making a service depend on network-online.target is considered an antipattern in most cases. This is because networking on Linux systems is very dynamic and the systemd target can only ever reflect the networking state at a single point in time. It cannot guarantee this state to be remained over the uptime of your system and has the potentially to delay the boot process considerably. Cables can be unplugged, wireless connectivity can drop, or remote routers can go down at any time, affecting the connectivity state of your local system. Therefore, “instead of wondering what to do about network.target, please just fix your program to be friendly to dynamically changing network configuration.” [source].

Netplan v1.1 released

by

I’m happy to announce that Netplan version 1.1 is now available on GitHub and is soon to be deployed into a Debian and/or Ubuntu installation near you! Six months and 120 commits after the previous version (including one patch release v1.0.1), this release is brought to you by 17 free software contributors from around the globe. 🚀

Kudos to everybody involved! ❤️

Highlights

  • Custom systemd-networkd-wait-online logic override to wait for link-local and routable interfaces. (#456, #482)
  • Modification of the embedded-switch-mode setting without virtual-function (VF) definitions on SR-IOV devices (#454)
  • Parser flag to ignore individual, broken configurations, instead of not generating any backend configuration (#412)
  • Fixes for @ProtonVPN (#495) and @microsoft Azure Linux (#445), contributed by those companies

Releasing v1.1

Documentation

Bug fixes

New Contributors

Full Changelog: 1.0…1.1

Creating a Netplan enabled system through Debian-Installer

by

With the work that has been done in the debian-installer/netcfg merge-proposal !9 it is possible to install a standard Debian system, using the normal Debian-Installer (d-i) mini.iso images, that will come pre-installed with Netplan and all network configuration structured in /etc/netplan/.

In this write-up, I’d like to run you through a list of commands for experiencing the Netplan enabled installation process first-hand. Let’s start with preparing a working directory and installing the software dependencies for our virtualized Debian system:

$ mkdir d-i_tmp && cd d-i_tmp
$ apt install ovmf qemu-utils qemu-system-x86

Now let’s download the official (daily) mini.iso, linux kernel image and initrd.gz containing the Netplan enablement changes:

$ wget https://d-i.debian.org/daily-images/amd64/daily/netboot/gtk/mini.iso
$ wget https://d-i.debian.org/daily-images/amd64/daily/netboot/gtk/debian-installer/amd64/initrd.gz
$ wget https://d-i.debian.org/daily-images/amd64/daily/netboot/gtk/debian-installer/amd64/linux

Next we’ll prepare a VM, by copying the EFI firmware files, preparing some persistent EFIVARs file, to boot from FS0:\EFI\debian\grubx64.efi, and create a virtual disk for our machine:

$ cp /usr/share/OVMF/OVMF_CODE_4M.fd .
$ cp /usr/share/OVMF/OVMF_VARS_4M.fd .
$ qemu-img create -f qcow2 ./data.qcow2 20G

Finally, let’s launch the debian-installer using a preseed.cfg file, that will automatically install Netplan (netplan-generator) for us in the target system. A minimal preseed file could look like this:

# Install minimal Netplan generator binary
d-i preseed/late_command string in-target apt-get -y install netplan-generator

For this demo, we’re installing the full netplan.io package (incl. the interactive Python CLI), as well as the netplan-generator package and systemd-resolved, to show the full Netplan experience. You can choose the preseed file from a set of different variants to test the different configurations:

We’re using the linux kernel and initrd.gz here to be able to pass the preseed URL as a parameter to the kernel’s cmdline directly. Launching this VM should bring up the official debian-installer in its netboot/gtk form:

$ export U=https://people.ubuntu.com/~slyon/d-i/netplan-preseed+full.cfg
$ qemu-system-x86_64 \
	-M q35 -enable-kvm -cpu host -smp 4 -m 2G \
	-drive if=pflash,format=raw,unit=0,file=OVMF_CODE_4M.fd,readonly=on \
	-drive if=pflash,format=raw,unit=1,file=OVMF_VARS_4M.fd,readonly=off \
	-device qemu-xhci -device usb-kbd -device usb-mouse \
	-vga none -device virtio-gpu-pci \
	-net nic,model=virtio -net user \
	-kernel ./linux -initrd ./initrd.gz -append "url=$U" \
	-hda ./data.qcow2 -cdrom ./mini.iso;

Now you can click through the normal Debian-Installer process, using mostly default settings. Optionally, you could play around with the networking settings, to see how those get translated to /etc/netplan/ in the target system.

After you confirmed your partitioning changes, the base system gets installed. I suggest not to select any additional components, like desktop environments, to speed up the process.

During the final step of the installation (finish-install.d/55netcfg-copy-config) d-i will detect that Netplan was installed in the target system (due to the preseed file provided) and opt to write its network configuration to /etc/netplan/ instead of /etc/network/interfaces or /etc/NetworkManager/system-connections/.

Done! After the installation finished, you can reboot into your virgin Debian Sid/Trixie system.

To do that, quit the current Qemu process, by pressing Ctrl+C and make sure to copy over the EFIVARS.fd file that was modified by grub during the installation, so Qemu can find the new system. Then reboot into the new system, not using the mini.iso image any more:

$ cp ./OVMF_VARS_4M.fd ./EFIVARS.fd
$ qemu-system-x86_64 \
        -M q35 -enable-kvm -cpu host -smp 4 -m 2G \
        -drive if=pflash,format=raw,unit=0,file=OVMF_CODE_4M.fd,readonly=on \
        -drive if=pflash,format=raw,unit=1,file=EFIVARS.fd,readonly=off \
        -device qemu-xhci -device usb-kbd -device usb-mouse \
        -vga none -device virtio-gpu-pci \
        -net nic,model=virtio -net user \
        -drive file=./data.qcow2,if=none,format=qcow2,id=disk0 \
        -device virtio-blk-pci,drive=disk0,bootindex=1
        -serial mon:stdio

Finally, you can play around with your Netplan enabled Debian system! As you will find, /etc/network/interfaces exists but is empty, it could still be used (optionally/additionally). Netplan was configured in /etc/netplan/ according to the settings given during the d-i installation process.

In our case, we also installed the Netplan CLI, so we can play around with some of its features, like netplan status:

Thank you for following along the Netplan enabled Debian installation process and happy hacking! If you want to learn more, find us at GitHub:netplan.

Netplan v1.0 paves the way to stable, declarative network management

by

New “netplan status –diff” subcommand, finding differences between configuration and system state

As the maintainer and lead developer for Netplan, I’m proud to announce the general availability of Netplan v1.0 after more than 7 years of development efforts. Over the years, we’ve so far had about 80 individual contributors from around the globe. This includes many contributions from our Netplan core-team at Canonical, but also from other big corporations such as Microsoft or Deutsche Telekom. Those contributions, along with the many we receive from our community of individual contributors, solidify Netplan as a healthy and trusted open source project. In an effort to make Netplan even more dependable, we started shipping upstream patch releases, such as 0.106.1 and 0.107.1, which make it easier to integrate fixes into our users’ custom workflows.

With the release of version 1.0 we primarily focused on stability. However, being a major version upgrade, it allowed us to drop some long-standing legacy code from the libnetplan1 library. Removing this technical debt increases the maintainability of Netplan’s codebase going forward. The upcoming Ubuntu 24.04 LTS and Debian 13 releases will ship Netplan v1.0 to millions of users worldwide.

Highlights of version 1.0

In addition to stability and maintainability improvements, it’s worth looking at some of the new features that were included in the latest release:

  • Simultaneous WPA2 & WPA3 support.
  • Introduction of a stable libnetplan1 API.
  • Mellanox VF-LAG support for high performance SR-IOV networking.
  • New hairpin and port-mac-learning settings, useful for VXLAN tunnels with FRRouting.
  • New netplan status –diff subcommand, finding differences between configuration and system state.

Besides those highlights of the v1.0 release, I’d also like to shed some light on new functionality that was integrated within the past two years for those upgrading from the previous Ubuntu 22.04 LTS which used Netplan v0.104:

  • We added support for the management of new network interface types, such as veth, dummy, VXLAN, VRF or InfiniBand (IPoIB). 
  • Wireless functionality was improved by integrating Netplan with NetworkManager on desktop systems, adding support for WPA3 and adding the notion of a regulatory-domain, to choose proper frequencies for specific regions. 
  • To improve maintainability, we moved to Meson as Netplan’s buildsystem, added upstream CI coverage for multiple Linux distributions and integrations (such as Debian testing, NetworkManager, snapd or cloud-init), checks for ABI compatibility, and automatic memory leak detection. 
  • We increased consistency between the supported backend renderers (systemd-networkd and NetworkManager), by matching physical network interfaces on permanent MAC address, when the match.macaddress setting is being used, and added new hardware offloading functionality for high performance networking, such as Single-Root IO Virtualisation virtual function link-aggregation (SR-IOV VF-LAG).

The much improved Netplan documentation, that is now hosted on “Read the Docs”, and new command line subcommands, such as netplan status, make Netplan a well vested tool for declarative network management and troubleshooting.

Integrations

Those changes pave the way to integrate Netplan in 3rd party projects, such as system installers or cloud deployment methods. By shipping the new python3-netplan Python bindings to libnetplan, it is now easier than ever to access Netplan functionality and network validation from other projects. We are proud that the Debian Cloud Team chose Netplan to be the default network management tool in their official cloud-images for Debian Bookworm and beyond. Ubuntu’s NetworkManager package now uses Netplan as it’s default backend on Ubuntu 23.10 Desktop systems and beyond. Further integrations happened with cloud-init and the Calamares installer.

Please check out the Netplan version 1.0 release on GitHub! If you want to learn more, follow our activities on Netplan.io, GitHub, Launchpad, IRC or our Netplan Developer Diaries blog on discourse.

Netplan v0.105 is now available

by

I’m happy to announce that Netplan version 0.105 is now available on GitHub and is soon to be deployed into an Ubuntu/Debian installation near you! Six month and exactly 100 commits after the previous version, this release is brought to you by 7 free software contributors from around the globe.

Changelog

  • Add support for VXLAN tunnels (#288), LP#1764716
  • Add support for VRF devices (#285), LP#1773522
  • Add support for InfiniBand (IPoIB) (#283), LP#1848471
  • Allow key configuration for GRE tunnels (#274), LP#1966476
  • Allow setting the regulatory domain (#281), LP#1951586
  • Documentation improvements & restructuring (#287)
  • Add meson build system (#268)
  • Add abigail ABI compatibility checker (#269)
  • Update of Fedora RPM spec (#264)
  • CI improvements (#265, #282)
  • Netplan set uses the consolidated libnetplan YAML parser (#254)
  • Refactor ConfigManager to use the libnetplan YAML parser (#255)
  • New netplan_netdef_get_filepath API (#275)
  • Improve NetworkManager device management logic (#276), LP#1951653

Bug fixes

Netplan v0.104 is now available

by

I’m happy to announce that netplan.io version 0.104 is now available on Github and has been uploaded into the next Ubuntu LTS release, code name “Jammy”: netplan.io 0.104-0ubuntu1! This is a big release that has been brought to you by 10 contributors around the globe.

Changelog:

  • Enable embedded-switch-mode setting on SmartNICs (#253)
  • Permit multiple patterns for the driver globs in match (#202), LP#1918421
  • Improve routing capabilities (#248), LP#1892272, LP#1805038
  • Support additional link offload options for networkd (#225) (#242), LP#1771740
    • receive-checksum-offload, transmit-checksum-offload, tcp-segmentation-offload,
      tcp6-segmentation-offload, generic-segmentation-offload, generic-receive-offload,
      large-receive-offload
  • Consolidate enum-to-string arrays (#230)
  • Handle differing ip6-privacy default value for NetworkManager (#263)
  • YAML state tracking (--state rootdir) for DBus API and netplan try (#231), LP#1943120
  • Support ConfigureWithoutCarrier (ignore-carrier) for networkd (#215)
  • Move primary git branch master to main
  • Documentation improvements (#226)
  • Compatibility for glib-2.70 (#235)
  • Cleanup Makefile, install only public headers
  • Improve test reliability & enable integration testing CI for autopkgtests
  • Netplan get to use the libnetplan parser (#252)
  • libnetplan:
    • introduce the notion of NetplanState (#232)
    • use an explicit parser context (#233)
    • expose coherent generator APIs (#239)
    • improve overall error handling (#234)
    • consolidation of YAML parsing into the library (#241, #249, #250, #251)
  • Restrict the symbol export to a determined public API (#227)
    • WARNING: We dropped some internal symbols from the API that we know
      have no external consumers (that we are aware of)
    • 0.103: _serialize_yaml, contains_netdef_type, tmp, validate_default_route_consistency
    • 0.102: cur_filename, netplan_netdef_new
    • 0.100: address_option_handlers, is_hostname, validate_ovs_target, wireguard_peer_handlers
    • 0.99: current_file, is_ip4_address, is_ip6_address, missing_id,
      missing_ids_found, parser_error, validate_backend_rules, validate_netdef_grammar,
      yaml_error

Bug fixes:

  • Fix removal of defunct OVS vlan interfaces (#256), LP#1959147
  • Make ConfigManager cleanup on destruction (#259), LP#1959729
  • Do not write unvalidated YAML from keyfile (#247), LP#1952967
  • Disable temporary address generation for real with NetworkManager (#244), LP#1948027
    • this is a slight change in behavior for NM, but is in line with the docs
      and implementation of the networkd backend renderer
  • Ignore empty YAML hints and delete files on set network=null (#246), LP#1946957
  • Wait for ‘netplan try’ to be ready in DBus API (#245), LP#1949893
  • Initialize self.state in apply (#243), LP#1949104
  • Driver fallback to nl80211 and/or wext for wpa_supplicant (#240), LP#1814012
  • Handle missing gateway in keyfile routes, keep dns-search fallback (#238)
  • Make it possible to unset a whole devtype subtree (#236), LP#1942930
  • Fix normalization of multiple keys on a single dict in tests (#229)
  • Add default-routes feature flag
  • Fix memory leaks, dangling pointers & overall cleanup of API data (#228)
  • Small whitespace and formatting fixes & shipping EditorConfig (#224)