This guide will show you how to run the newly GA’ed bits of MicroShift on a Raspberry Pi 4 using Fedora IoT 38 or 39. It can announce routes via mDNS so hosting applications in an mDNS aware LAN is a breeze.
THIS IS COMPLETELY UNSUPPORTED. DON’T EVEN THINK OF RED HAT SUPPORT WHEN YOU FOLLOW THESE INSTRUCTIONS.
“Completely” means neither the Fedora community nor the Red Hat Support organisation will be able to help you in a structured way if you run into problems. This is because mixing binaries from both sources yanks away any basis for understanding what exactly is happening there, invalidates all knowledge bases, etc…
You might be wondering why Red Hat doesn’t support its Enterprise Linux on Raspberry Pis (they are 64 Bits capable and offer enough CPU/RAM after all). A requisite for support is a certification against a strict standard (the stricter the standard, the less surprises down the road). Currently, the minimum level that Red Hat requires is an ARM SystemReady IR v2.0+ compliant firmware – which the Raspberry Pis do not provide.
Motivation
My home automation setup is based on an older install of MicroShift on a Raspberry Pi 4. This hardware is close to ideal because it combines a small form factor which makes it fit my breaker box with GPIO ports that allow me to integrate additional sensors.
Recently I managed to fry the SDCard through repeated short circuits (yes, I was testing something). This is where the idea of a declarative configuration shone: after setting up MicroShift again based on the old instructions, I was able to deploy all applications & their configurations from my repository. I was also able to mount the old SDCard filesystem in a read-only fashion and copy all data over, effectively re-creating the complete state of my setup.
At the same time it felt silly to continue to run old versions when a GA’ed version of MicroShift is now available. This guide shows how to set up this more modern version on Fedora IoT 38. The same approach also works on Fedora IoT 39.
Prerequisites
You have a Raspberry Pi 4 with >2GB of RAM and an SDCard with >10 GiB for the root partition and additional space as storage for PVs (this guide uses a 128GB SDCard).
Note that the SDCard Interface on a Raspberry Pi is comparably slow, if you are interested in faster IO speeds, use a storage device attached via USB or similar. For this use case, I valued simplicity over performance and have everything running from the SDCard slot.
You have an access.redhat.com account, an OpenShift or Red Hat Device Edge Subscription and can access your OpenShift pull secret.
Provision Fedora IOT 38
Create an SDCard with a bootable Fedora IOT 38 image. There are multiple ways to do this – on MacOS, I used the Raspberry Pi Imager application to write the image to the SDCard, and configured the SSH key(s) on first boot using the zezere service provided by the Fedora project.
Once the device is fully booted, you should be able to log in as root via ssh.
Download Microshift packages
Download the following RPMs from https://access.redhat.com/downloads/content/package-browser . You can do this on the Raspberry Pi by copying the download links and using curl -O.
- cri-o
- cri-tools
- crun
- jansson
- microshift
- microshift-greenboot
- microshift-networking
- microshift-selinux
- openshift-clients
- openvswitch-selinux-extra-policy
- openvswitch3.1
- NetworkManager
- NetworkManager-libnm
- NetworkManager-ovs
- NetworkManager-wwan
- NetworkManager-wifi
The microshift-*, cri*, crun, openvswitch* packages make up MicroShift. They pull in NetworkManager* as dependency, which in turn requires the jansson library. openshift-clients provides the CLI tools to interface with MicroShift. Make sure you download the RHEL9 version of all packages (with el9 in the name) – some are available with a higher version number for RHEL8 which then fail to install.
Prepare Fedora IoT and install MicroShift rpms
We will need to make some adjustments to be able to install MicroShift, such as preparing the filesystem layout, configuring hostname and time zone and updating the system.
Grow the root partition
In order to be able to host the additional rpms and have some room to breathe, we’ll extend the root partition to 32 GiB.
# parted /dev/mmcblk0 resizepart 3 32GiB
Warning: Partition /dev/mmcblk0p3 is being used. Are you sure you want to continue?
Yes/No? yes
Information: You may need to update /etc/fstab.
# mount /sysroot -o remount,rw
mount: (hint) your fstab has been modified, but systemd still uses
the old version; use 'systemctl daemon-reload' to reload.
# resize2fs /dev/mmcblk0p3
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/mmcblk0p3 is mounted on /sysroot; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 4
The filesystem on /dev/mmcblk0p3 is now 7997952 (4k) blocks long.Code language: PHP (php)Prepare RHEL volume group
MicroShift assumes an LVM volume group named “rhel” to host persistent volumes using the topolvm storage provider. This volume group will be hosted on an extended partition.
# parted /dev/mmcblk0 mkpart extended 32GiB 128GB
Information: You may need to update /etc/fstab.
# parted /dev/mmcblk0 mkpart logical 33GiB 97GiB
Information: You may need to update /etc/fstab.
# parted /dev/mmcblk0 set 5 lvm on
Information: You may need to update /etc/fstab.
# pvcreate /dev/mmcblk0p5
Physical volume "/dev/mmcblk0p5" successfully created.
Creating devices file /etc/lvm/devices/system.devices
# vgcreate rhel /dev/mmcblk0p5
Volume group "rhel" successfully createdCode language: PHP (php)Update and Configure System & install mDNS
rpm-ostree is the image-based deployment mechanism used by Fedora IoT. It requires you to reboot the system to make package installations visible. Two ostrees are maintained by default at the same time. This allows you to roll back your device to a previously known good configuration in case a new setup is broken.
# rpm-ostree update
⠈ Receiving objects; 99% (11112/11125) 5.3 MB/s 465.1 MB 1925 metadata, 9252 content objects fetched; 457581 KiB transferred in 93 seconds; 1.2 GB content written
Receiving objects; 99% (11112/11125) 5.3 MB/s 465.1 MB... done
Staging deployment... done
Freed: 205.8 MB (pkgcache branches: 1)
Upgraded:
[...]
# rpm-ostree install nss-mdns avahi
[...]
# timedatectl set-timezone Europe/Berlin
# hostnamectl set-hostname microshift-new.local
# systemctl rebootCode language: PHP (php)Enable mDNS
After reboot, log back in and activate the mMDNS service.
# systemctl enable --now avahi-daemon.serviceCode language: CSS (css)Install MicroShift packages
We will install the MicroShift packages in two steps – first we overlay existing packages from the Fedora IoT ostree with the ones downloaded earlier. Then we will install the remaining packages.
At this point it is a good idea to mention again that the resulting setup will be completely unsupported.
# ls
NetworkManager-1.44.0-3.el9.aarch64.rpm
NetworkManager-libnm-1.44.0-3.el9.aarch64.rpm
NetworkManager-ovs-1.44.0-3.el9.aarch64.rpm
NetworkManager-wifi-1.44.0-3.el9.aarch64.rpm
NetworkManager-wwan-1.44.0-3.el9.aarch64.rpm
cri-o-1.27.1-11.1.rhaos4.14.git9b9c375.el9.aarch64.rpm
cri-tools-1.27.0-2.1.el9.aarch64.rpm
crun-1.11-1.rhaos4.14.el9.aarch64.rpm
jansson-2.14-1.el9.aarch64.rpm
microshift-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.aarch64.rpm
microshift-greenboot-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.noarch.rpm
microshift-networking-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.aarch64.rpm
microshift-selinux-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.noarch.rpm
openshift-clients-4.14.0-202310191146.p0.g0c63f9d.assembly.stream.el9.aarch64.rpm
openvswitch-selinux-extra-policy-1.0-34.el9fdp.noarch.rpm
openvswitch3.1-3.1.0-40.el9fdp.aarch64.rpm
# rpm-ostree override replace NetworkManager-{,libnm-,wwan-,wifi-}1* crun* jansson*
[...]
# rpm-ostree install ./cri-* ./openvswitch* ./microshift* ./openshift-clients* ./NetworkManager-ovs*
[...]
# systemctl reboot Code language: PHP (php)Once the system comes back up, log in again and verify the ostree:
# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
Version: 38.20231101.0 (2023-11-01T14:02:01Z)
BaseCommit: e1b38ff646cab74c4d58d535d4ba70736e52ae5cfc95b88374d9cca93c4cc3c7
GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
LocalOverrides: crun 1.11-1.fc38 -> 1.11-1.rhaos4.14.el9 jansson 2.13.1-6.fc38 -> 2.14-1.el9 NetworkManager-wwan NetworkManager-libnm NetworkManager NetworkManager-wifi 1:1.42.8-1.fc38 -> 1:1.44.0-3.el9
LayeredPackages: avahi nss-mdns
LocalPackages: cri-o-1.27.1-11.1.rhaos4.14.git9b9c375.el9.aarch64 cri-tools-1.27.0-2.1.el9.aarch64 microshift-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.aarch64
microshift-greenboot-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.noarch microshift-networking-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.aarch64
microshift-selinux-4.14.1-202310271350.p0.g1586504.assembly.4.14.1.el9.noarch NetworkManager-ovs-1:1.44.0-3.el9.aarch64 openshift-clients-4.14.0-202310191146.p0.g0c63f9d.assembly.stream.el9.aarch64
openvswitch-selinux-extra-policy-1.0-34.el9fdp.noarch openvswitch3.1-3.1.0-40.el9fdp.aarch64
[...]Code language: PHP (php)Configure and start MicroShift
At this point, we pretty much just have to follow the MicroShift installation instructions from step 3 onwards. There’s a small caveat as we have to add a compatibility fix for Fedora’s systemd-resolved.
Download OpenShift Pull Secret & configure cri-o
Download your pull secret from https://console.redhat.com/openshift/install/pull-secret and store it as ~/openshift-pull-secret
# ls ~/openshift-pull-secret
/root/openshift-pull-secret
# cp $HOME/openshift-pull-secret /etc/crio/openshift-pull-secret
# chown root:root /etc/crio/openshift-pull-secret
# chmod 600 /etc/crio/openshift-pull-secretCode language: PHP (php)Configure firewall
Open the mandatory firewall ports.
# firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16
success
# firewall-cmd --permanent --zone=trusted --add-source=169.254.169.1
success
# firewall-cmd --reload
successCode language: PHP (php)Enable MicroShift
Now you can enable (but not yet start) MicroShift. This will also cause the greenboot health checks to run at system startup. In case the microshift service fails, disable MicroShift before rebooting – otherwise the greenboot health checks will identify a problem on the next reboot, and will roll back to the previously known good configuration.
# systemctl enable microshift.serviceCode language: CSS (css)Add and enable systemd-resolved fix
On start, MicroShift copies the resolv.conf DNS configuration file location into the CoreDNS configmap. On a system like Fedora IoT 38 this causes the wrong resolv.conf location to be present in the CoreDNS config map, causing the pod to crashloop. To work around this, we will create systemd unit that patches the configmap after microshift has started.
# cat >/etc/systemd/system/microshift-on-fedora.service <<EOF
[Unit]
Description=MicroShift 4.14 with systemd-resolved-fix
After=microshift.service
BindsTo=microshift.service
[Service]
ExecStart=/bin/bash -c "kubectl --kubeconfig /var/lib/microshift/resources/kubeadmin/kubeconfig get cm dns-default -n openshift-dns -o yaml | sed 's_/run/systemd/resolve/resolv.conf_/etc/resolv.conf_' | kubectl --kubeconfig /var/lib/microshift/resources/kubeadmin/kubeconfig apply -f -"
Type=oneshot
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
EOF
# systemctl enable microshift-on-fedora.serviceCode language: PHP (php)Start MicroShift
Now you can start MicroShift (with the fix). Please use the microshift-on-fedora unit to start and stop MicroShift, it will in turn start and stop the underlying MicroShift service. You can query the status of microshift using the regular microshift.service unit. Give it some time to pull all required pods & start up properly. If you receive an error message, it might be MicroShift not having started up quickly enough on the first try. Wait one or two minutes and try again.
# systemctl start microshift-on-fedora
# systemctl status microshift
● microshift.service - MicroShift
Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: active (running) since Thu 2023-11-02 22:22:40 CET; 10h ago
Main PID: 1525 (microshift)
Tasks: 16 (limit: 4160)
Memory: 580.1M
CPU: 1h 54min 41.851s
CGroup: /system.slice/microshift.service
└─1525 microshift run
Nov 03 09:12:28 microshift-new.local microshift[1525]: kubelet I1103 09:12:28.694006 1525 kubelet.go:2457] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qcvpv" event={"ID":"e86bfc5a-7fb9-425e-ab9b-1ecef66f6996","Type":"ContainerDied","Data":"6514c6574172b44b036011f8a97a37d932066def9d98075a1b30d6b8c8e7977b"}
Nov 03 09:12:28 microshift-new.local microshift[1525]: kubelet I1103 09:12:28.694223 1525 scope.go:115] "RemoveContainer" containerID="080aa0f9ff8c69165d7d464abb13afd7a6236bd52aedaf3eef61cc5e3b6ef158"
[...]Code language: PHP (php)Copy access credentials
Upon start, microshift creates a kubeconfig configuration file that allows administrative access. Copy it into your home directory to use via the CLI.
# mkdir -p ~/.kube/
# cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config
# chmod go-r ~/.kube/configCode language: PHP (php)Test client access
oc status and oc project will throw an error, since MicroShift does not know the project API objects. This is expected. You can test the CLI access as follows:
# oc get all -A
[...]
# kubectl get all -A
[...]Code language: PHP (php)Test the Setup using a sample Application
We will use Node Red as a sample application to test the MicroShift platform. It starts on an unprivileged port (doesn’t have to run as root) and can use a persisten volume to store its flows.
Deploy sample application
This sample application deploys Node-Red and creates a route that is advertised via mDNS.
# kubectl create namespace node-red
namespace/node-red created
# kubectl -n node-red apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: node-red-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
name: node-red
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: node-red
strategy:
type: Recreate
template:
metadata:
creationTimestamp: null
labels:
app: node-red
spec:
containers:
- image: nodered/node-red
imagePullPolicy: Always
name: node-red
ports:
- containerPort: 1880
name: ui
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- name: data
mountPath: /data
livenessProbe:
httpGet:
path: /
port: ui
readinessProbe:
httpGet:
path: /
port: ui
startupProbe:
httpGet:
path: /
port: ui
failureThreshold: 30
periodSeconds: 10
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: data
persistentVolumeClaim:
claimName: node-red-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: node-red
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- port: 1880
protocol: TCP
targetPort: ui
name: ui
selector:
app: node-red
sessionAffinity: None
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: node-red
spec:
host: nodered.local
port:
targetPort: ui
to:
kind: Service
name: node-red
weight: 100
wildcardPolicy: None
EOF
persistentvolumeclaim/node-red-pv-claim created
deployment.apps/node-red created
service/node-red created
route.route.openshift.io/node-red createdOnce the application is deployed, you should be able to access it from the CLI or your mDNS aware browser.
# curl http://nodered.local/
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
[...]Code language: HTML, XML (xml)Review storage artifacts
The topolvm storage provider creates LVM logical volumes for each Kubernetes PV that is created.
# kubectl describe pvc node-red-pv-claim -n node-red
Name: node-red-pv-claim
Namespace: node-red
StorageClass: topolvm-provisioner
Status: Bound
Volume: pvc-745b7227-76b0-4018-8474-f77238609f41
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: topolvm.io
volume.kubernetes.io/selected-node: microshift-new.local
volume.kubernetes.io/storage-provisioner: topolvm.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: node-red-6b87898896-hv5mv
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 10m persistentvolume-controller waiting for first consumer to be created before binding
Normal Provisioning 10m topolvm.io_topolvm-controller-9b4ff4fb6-98jbf_6c22d071-31a6-47a4-9423-f0fbe9f177d2 External provisioner is provisioning volume for claim "node-red/node-red-pv-claim"
Normal ExternalProvisioning 10m persistentvolume-controller waiting for a volume to be created, either by external provisioner "topolvm.io" or manually created by system administrator
Normal ProvisioningSucceeded 10m topolvm.io_topolvm-controller-9b4ff4fb6-98jbf_6c22d071-31a6-47a4-9423-f0fbe9f177d2 Successfully provisioned volume pvc-745b7227-76b0-4018-8474-f77238609f41
# kubectl get pv -o jsonpath='{.items[*].metadata.name} {.items[*].spec.csi.volumeHandle}{"\n"}'
pvc-745b7227-76b0-4018-8474-f77238609f41 d2268877-8d1d-4fb0-b08d-bef055496bc5
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
d2268877-8d1d-4fb0-b08d-bef055496bc5 rhel -wi-ao---- 3.00g
#Code language: HTML, XML (xml)Remove sample application
To remove the sample application, simply delete the namespace. This will also clean up the LVM logical volumes.
# kubectl delete namespace node-red
namespace "node-red" deleted
# lvs
#Code language: PHP (php)Summary
This guide showed how to set up MicroShift 4.14 (and beyond?) on a Raspberry Pi 4 using Fedora IoT 38 – a totally unsupported setup. This is mostly achieved by replacing some Fedora packages with those coming from Red Hat Enterprise Linux and adding the remaining packages to the image-based Fedora deployment and following the MicroShift installation instructions.
Fedora 38 uses systemd-resolved while Red Hat Enterprise Linux doesn’t – this triggers a code path in MicroShift which causes a wrong configuration to be stored for the CoreDNS pod. This is why we need a workaround systemd unit that reverses the configuration after MicroShift startup.