Raspberry Pi OS moves to Network Manager

There has been a major change on network configuration coming on Raspberry Pi OS for quite some time now. Along with the introduction of Debian Bookworm, the old dhcpcd has been replaced with Network Manager. This changes the way how I connect back to my parents that requires new configuration. Some of the configuration is similar to that of Inovato's Quadra that I used during the Raspberry Pi shortage.

To demonstrate the configuration here I used my 4GB Raspberry Pi 4 for which Imager 1.8.3 recommends Raspberry Pi OS 64-bit based on Debian Bookworm now. While this image includes desktop, I still prefer to use text based user interface to configure Network Manager. Please note the elegant way 5GHz is used hence higher speed and less interference.

jordana@pi4g:~ $ sudo nmtui
Edit a connection
<Add>
Device wlan0
SSID broadcom
Mode <Access Point>
Channel <A (5 GHz)>
Security <WPA & WPA2 Personal>
Password ********************
IPv4 CONFIGURATION <Shared>
<Show>
Addresses 192.168.3.1/24
<OK>
<Back>
Quit

This configuration automatically sets up DHCP on the access point with dnsmasq built into Network Manager hence separate dnsmasq should not be installed from apt package repository. However we will need to use Google DNS server as DHCP option hence we need to add a configuration file for that.

jordana@pi4g:~ $ sudo nano /etc/NetworkManager/dnsmasq-shared.d/server.conf
dhcp-option=6,8.8.8.8

Text based user interface does not allow setting the necessary IPv4 routing rules so for that I need to use command line interface of Network Manager. First we need to check UUID for the connection created above on wlan0 device.

jordana@pi4g:~ $ nmcli connection
NAME                UUID           TYPE      DEVICE     
Wi-Fi connection 1  6a55d164-xxxx  wifi      wlan0      
jordana@pi4g:~ $ sudo nmcli connection modify 6a55d164-xxxx ipv4.routing-rules "priority 1 from 192.168.3.0/24 table 1"

Next we need to install zerotier and set up its IP address and authorization on Zerotier Central web console to get the interface and IP address we want.

jordana@pi4g:~ $ ip a
4: zt5u4vfg76: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    inet 192.168.193.3/24 brd 192.168.193.255 scope global zt5u4vfg76

Finally, after setting up Managed Routes 192.168.3.0/24 via 192.168.193.3 on Zerotier Central web console we need to dispatch default route towards my parents Raspberry Pi.

jordana@pi4g:~ $ sudo nano /etc/NetworkManager/dispatcher.d/10-zt5u4vfg76-up
#!/bin/bash
if [ "$1" = "zt5u4vfg76" ] && [ "$2" = "up" ]; then
  sudo /sbin/ip route add default via 192.168.193.4 table 1
fi
jordana@pi4g:~ $ sudo chmod +x /etc/NetworkManager/dispatcher.d/10-zt5u4vfg76-up

My friend struggled with the automatic choice of TKIP by Network Manager and needed the following configuration to make it work with iPad:

sudo nmcli con modify "Wi-Fi Hot" 802-11-wireless-security.proto rsn

sudo nmcli con modify "Wi-Fi Hot" 802-11-wireless-security.pairwise ccmp

UPDATE

Even though Network Manager takes care about MASQUERADE I realised that I still need this iptables rule to ensure zerotier-cli can access zerotier service locally.

quadra@inovato:~$ sudo iptables -t nat -I POSTROUTING -o lo -j ACCEPT
quadra@inovato:~$ sudo iptables-save -f /etc/iptables.rules
quadra@inovato:~$ sudo nano /etc/rc.local
iptables-restore /etc/iptables.rules

Enabling TRIM with USB SSD

From time to time I had hiccups with my USB 3 SSD when run Raspberry Pi diagnostics (agnostics). Occasionally it did not reach the 160MB/s advertised write speed. They were a bargain on amazon so I thought I have to live with this. 64GB SSD drive was 10 GBP while the USB 3 enclosure was 20 GBP.

64 GB SSD drive

USB 3 enclosure

This article from Jeff Geerling explains how to configure TRIM on SSD:

Enabling TRIM on an external SSD on a Raspberry Pi

Initially I observed that TRIM was not configured on my boot drive:

jordana@pi4g:~ $ sudo fstrim -v /
fstrim: /: the discard operation is not supported
jordana@pi4g:~ $ lsblk -D
NAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda           0      0B       0B         0
├─sda1        0      0B       0B         0
└─sda2        0      0B       0B         0

I installed the SCSI tools based on the article and confirmed that my drive supports TRIM:

jordana@pi4g:~ $ sudo apt-get install -y sg3-utils lsscsi
jordana@pi4g:~ $ sudo sg_vpd -p bl /dev/sda
Block limits VPD page (SBC):
  Maximum unmap LBA count: 4194240
  Maximum unmap block descriptor count: 8
jordana@pi4g:~ $ sudo sg_vpd -p lbpv /dev/sda
Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 1

I checked that the provisioning mode is full and changed it to unmap:

jordana@pi4g:~ $ sudo find /sys/ -name provisioning_mode -exec grep -H . {} + | sort
/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:full
jordana@pi4g:~ $ sudo su
root@pi4g:/home/jordana# echo unmap > /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode

This enabled TRIM and made my write speed consistent:

jordana@pi4g:~ $ sudo find /sys/ -name provisioning_mode -exec grep -H . {} + | sort
/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1/2-1:1.0/host0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:unmap
jordana@pi4g:~ $ lsblk -D
NAME   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda           0      512B       4G         0
├─sda1        0      512B       4G         0
└─sda2        0      512B       4G         0
jordana@pi4g:~ $ sudo fstrim -v /
/: 42.3 GiB (45465530368 bytes) trimmed
jordana@pi4g:~ $ sh /usr/share/agnostics/sdtest.sh
Run 1
prepare-file;0;0;164663;321
seq-write;0;0;163840;320
rand-4k-write;0;0;60179;15044
rand-4k-read;53108;13277;0;0
Sequential write speed 163840 KB/sec (target 10000) - PASS
Random write speed 15044 IOPS (target 500) - PASS
Random read speed 13277 IOPS (target 1500) - PASS
jordana@pi4g:~ $ systemd-analyze
Startup finished in 2.275s (kernel) + 14.610s (userspace) = 16.885s 
graphical.target reached after 10.126s in userspace

To make this setting persistent I had to find out my idVendor:idProduct with lsusb and configure udev rule as follows:

jordana@pi4g:~ $ lsusb -tv

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M

    ID 1d6b:0003 Linux Foundation 3.0 root hub

    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M

        ID 2109:0715 VIA Labs, Inc. VL817 SATA Adaptor

jordana@pi4g:~ $ sudo nano /etc/udev/rules.d/10-trim.rules

ACTION=="add|change", ATTRS{idVendor}=="2109", ATTRS{idProduct}=="0715", SUBSYSTEM=="scsi_disk", ATTR{provisioning_mode}="unmap"

Install zerotier and use it with zeronsd on WSL

This is a guide how to install zerotier and use it with zeronsd on Windows Subsystem for Linux (WSL).

To install WSL run Windows Powershell as Administrator. As zerotier requires systemd I installed Debian:

PS C:\Windows\system32> wsl --install -d Debian
PS C:\Windows\system32> wsl --set-version Debian 2
PS C:\Windows\system32> wsl -l -v
  NAME      STATE           VERSION
* Debian    Stopped         2

Once installed I started Debian, updated its packages and added configuration to enable systemd:

jordana@dad-pc:~$ sudo apt update
jordana@dad-pc:~$ sudo apt upgrade
jordana@dad-pc:~$ sudo nano /etc/wsl.conf
[boot]
systemd=true

After reboot of Debian I had to install openssh server and reboot again to see systemd starting properly. 

jordana@dad-pc:~$ sudo apt install openssh-server

After this I have seen systemd starting properly so I could install zerotier with curl:

jordana@dad-pc:~$ systemctl list-unit-files --type=service
jordana@dad-pc:~$ sudo apt install curl
jordana@dad-pc:~$ curl -s https://install.zerotier.com | sudo bash
jordana@dad-pc:~$ sudo zerotier-cli join xxxxxxxxxxxxxxx
200 join OK

In order to configure local domain name resolution is routed to zeronsd, I had to disable that WSL generates /etc/resolv.conf in /etc/wsl.conf and add localhost to /etc/resolv.conf and reboot again.

jordana@dad-pc:~$ sudo nano /etc/wsl.conf
[network]
generateResolvConf = false
jordana@dad-pc:~$ sudo nano /etc/resolv.conf
nameserver localhost

Finally I could install dnsmasq and configure zeronsd local domain routed towards zeronsd server:

jordana@dad-pc:~$ sudo apt install dnsmasq
jordana@dad-pc:~$ sudo nano /etc/dnsmasq.conf
server=/zeronsd/192.168.193.1
server=8.8.8.8
jordana@dad-pc:~$ sudo systemctl restart dnsmasq

In order to let ansible to keep all apt packages latest I had to install python3 and configure sudo without password:

jordana@dad-pc:~$ sudo apt install python3
jordana@dad-pc:~$ sudo nano /etc/sudoers.d/nopasswd
jordana ALL=(ALL) NOPASSWD: ALL

Another workaround for ssh login issue over zerotier

 From a hotel room during a business trip I have experienced again that ssh over zerotier stuck at some initialization phase:

jordana@penguin:~$ ssh -v ubuntu@oracle.zeronsd
OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n  15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
...
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

I suspect this has something to do with dropped packets again due to MTU limitations over zerotier tunnel as ssh into public IP directly works OK. To workaround this time I changed this line in ssh config:

jordana@penguin:~$ sudo nano /etc/ssh/ssh_config
   MACs hmac-sha2-256

While this made ssh work from terminal directly but ansible still could not connect.

Running ansible apt via ZeroTier from penguin on Chromebook aka Crostini

As more traditional daily home work kicked in, I recently started to use a Chromebook and a Windows desktop so the Raspberry Pi 4 has lost its place on my desk. In order to keep all my Linux hosts updated with ansible apt even on the go, I installed ZeroTier on all my Linux hosts. To use hostnames instead of IP address in my ZeroTier network I maintain local DNS entries in .zt domain on my pi.hole running on my good old Raspberry Pi 1b. In order to use this DNS server on Crostini for .zt domain I installed dnsmasq on it and configured the IP address of Raspberry Pi 1b on the ZeroTier network as follows:

jordana@penguin:~$ sudo nano /etc/dnsmasq.conf
server=/zt/192.168.193.1 

To make sure this is used every time Crostini boots I uncommented this line from dhclient:

jordana@penguin:~$ sudo nano /etc/dhcp/dhclient.conf
prepend domain-name-servers 127.0.0.1;

To access admin interface on pi.hole I installed Firefox ESR from apt on Crostini as the integration into the desktop is amazing. Installing RealVNC Viewer was also reasonably straightforward on Chromebook. After downloading the deb package an Install with Linux has popped up for right-click in the Files application. This makes signing into RealVNC with the servers running on all my Raspberry Pi somewhat redundant and the limit of 5 computers coming with RealVNC Home license easily expandable.

UPDATE

Finally I moved away from double administration of names by using zeronsd based on this quick guide:

ZeroNSD Quickstart

As I have only found x86/amd64 releases in the official repo I have used a t2.micro on AWS for this service as it is eligible in the Free Tier.

Please note that in order for dnsmasq and zeronsd to coexist on this AWS machine we need the following configurations:

admin@ip-172-31-8-100:~$ sudo nano /etc/dhcp/dhclient.conf
prepend domain-name-servers 127.0.0.1;
admin@ip-172-31-8-100:~$ sudo nano /etc/dnsmasq.conf
server=/zeronsd/192.168.193.10
interface=lo
bind-interfaces

Please note that such configuration achieves per interface DNS setting. This is what zerotier-systemd-manager from the above ZeroNSD quick install suppose to do but usually fails due to most hosts are not using systemd-networkd and resolvconf for all other interfaces. Hence the installation of zerotier-systemd-manager can be skipped from the above quick guide with such configuration.

UPDATE 2

It turned out that after one year AWS charges even for a t2.micro so I moved zeronsd to my chromebook as I only use it for ansible. While this is not ideal as my chromebook is sleeping most of the time, I could not compile zeronsd on my good old Raspberry Pi 1b whatever I tried. I was so desparate that I tried tailscale instead and with that Magic DNS works out of the platform without the need to install extra components. This made me thinking hard about migrating to tailscale which BTW available on my Synology NAS Package Center and installed on latest quadra image by default.

UPDATE 3

Just for the record it realised that tailscale free plan limits the number of subnets to one while Zerotier poses no limit on subnets only on 25 nodes in its free plan so I stick to Zerotier.

UPDATE 4

On Coronation Day 6 May 2023, I have successfully built latest zeronsd 0.5.2 from git on Raspberry Pi 4 running Raspberry Pi OS 64 bit using the latest cargo 1.69.0 with these 3 commands:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt install librust-openssl-dev
cargo install --git https://github.com/zerotier/zeronsd --branch main

Unfortunately with default config zeronsd service could start successfully at first attempt because could not reach my.zerotier.com. I had to add a longer restart timeout to come up properly.

jordana@pi4g:~ $ sudo nano /lib/systemd/system/zeronsd-xxxxxxxxxxxxxxxxx.service
[Service]
RestartSec=10

UPDATE 5

Finally I could successfully build on Raspberry Pi 1 Model B with 512M RAM by increasing swap memory to 512M based on this article:

https://pimylifeup.com/raspberry-pi-swap-file/

Not sure if this is needed but I also added install option to run only 1 job at a time:

cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1

Please note the install takes nearly 10 hours on an otherwise quite system so plan carefully ahead.

As it turned out there is no need for changing the swap size in case Raspberry Pi OS lite is used.

UPDATE 6

In order to run pi-hole with zeronsd on the same host you have to limit pi-hole to bind on physical interface only. 

UPDATE 7

As this post is originally started off due to me moving to ARM based Chromebook let me keep here a link for Raspberry Pi Imager that can be installed on it.