February 28, 2025 23:42

The setting…

I recently started adopting WireGuard VPN into more and more of my server infrastructure. Consequently, I was wondering if the Linux-based JetKVM could also be integrated into these networks - especially useful since I do not want to use their cloud solution, but still want to deploy these onto remote locations as needed. So, getting WireGuard to run there was both harder and easier than expected - here is a write-up on how to do it. If you just came here to quickly copy-paste some commands, there you go:

The quick route

…if you do not care what is going on, here is a quick list of commands you need to run. If they break at some point, I fear you still have to understand what you are doing. Good luck!

# starting off on your host system, hopefully running Ubuntu 22.04 or such...
sudo apt install gcc g++ gperf bison flex texinfo help2man make libncurses5-dev python3-dev autoconf automake libtool libtool-bin gawk wget bzip2 xz-utils unzip patch libstdc++6 rsync git meson ninja-build # -> https://github.com/crosstool-ng/crosstool-ng/blob/master/testing/docker/ubuntu22.04/Dockerfile

# let's get the cross-toolchain setup
git clone https://github.com/crosstool-ng/crosstool-ng -b crosstool-ng-1.27.0 # -> https://crosstool-ng.github.io/docs/install/#clone
pushd crosstool-ng
./bootstrap
./configure --enable-local
make
./ct-ng arm-unknown-linux-uclibcgnueabihf
export CT_PREFIX=$(pwd)/dist
./ct-ng build
popd

# now get the wireguard-tools compiled
git clone https://git.zx2c4.com/wireguard-tools # -> https://www.wireguard.com/compilation/#step-4-compile-and-install-the-wg8-tool
pushd wireguard-tools
CFLAGS="-static -Os" CC="$CT_PREFIX/arm-unknown-linux-uclibcgnueabihf/bin/arm-unknown-linux-uclibcgnueabihf-gcc" make -C src -j$(nproc)
file src/wg # should report some arm binary
python3 -m http.server # start a local webserver to download the binary from

# jump over onto your JetKVM (enable developer mode and login using SSH as root)
cd /userdata # place stuff here to prevent it getting lost during upgrades
wget http://YOUR_HOST_IP_ADDRESS:8000/src/wg # download the binary

# setup wireguard once (after https://www.wireguard.com/quickstart/#command-line-interface)
modprobe wireguard
ip link add dev wg0 type wireguard
ip address add dev wg0 192.168.2.1/24 # use the value from Interface.Address inside your config
./wg setconf wg0 myconfig.conf # WARNING: "wg" IS NOT "wg-quick" -> see note below
ip link set up dev wg0
./wg show # check if connection is established, as usual

# ...for boot-persistance, you have to write a script, place it in /userdata and link it in /etc/init.d - see below

Careful: wg is NOT equals to wg-quick. The latter is a command line script (needing /bin/bash, not available here), which is also able to perform the ip address add based on the Interface.Address key inside your config. As you only have wg available, you need to remove this key from the config file and perform the ip-assignment manually.

Diving deeper

Okay. I’ve just got my JetKVM. I want WireGuard running. I just discovered the developer setting inside the UI, but I have no idea what architecture is there, how I could cross-compile and where to even start from. This chapter will now dive deeper into the process of getting WireGuard running and all the things I discovered along the way…

Initially, I found some reference material for deploying Tailscale onto the JetKVM, but it is using a not-yet-merged plugin system extension. Also, I just want to get WireGuard itself running, without relying on a third-party service or a go-userspace implementation of WireGuard.

Jumping onto the JetKVM for the first time, I discovered that the kernel was shipped with the wireguard-module available (using modprobe wireguard and confirming with lsmod). This was a good sign, as it meant that I could use the wg-tool to configure the VPN connection… Only problem: the wg-tool was not available on the system. So, I had to compile it myself.

Building `wg` - the wrong way…

So, my first attempt was to just clone the wireguard-tools repository and to compile it using my host machines cross-compiler. I just installed gcc-arm-linux-gnueabihf, which allows me to execute an ARM compiler. Experienced developers will already notice my mistake here, as I’ve selected the gnueabihf-variant (…). After taking a look into the src/Makefile of the tool-repository, I was able to determine that I could tell it to use my cross-compiler just by setting the CC-variable. While I was at it, I also set the V-variable to enable debugging (so I could see which commands are being used to compile) and the CFLAGS-variable to enable static linking and to optimize the binary for size. I also found some WITH_*-variables, which control the behavior of make install - which I cannot use, as the binary should not be installed on my host-machine, but onto the JetKVM later on. So, my initial compilation attempt looked like this:

V=1 CFLAGS="-static -Os" CC=arm-linux-gnueabihf-gcc make -C src -j1

…which ran fine - first try 😎

To upload the binary onto the JetKVM, I used an old trick of starting an HTTP server and then grabbing the file on the target machine using wget, as it is available there. Of course, netcat is also often used for these transfer operations, but at that point I thought it would be a quick play…

# on my host
python3 -m http.server
# on the JetKVM
cd /userdata
wget http://YOUR_HOST_IP_ADDRESS:8000/src/wg

…and now the trouble began. Executing the binary did not yield in the expected output:

./wg genkey
/bin/sh: ./wg: not found

…wait? The binary was placed into the directory, marked as executable, but the shell still reported that it was not found? Also, confirming a successful transfer with md5sum just proved that the file was fine… Yeah - the error message is indeed not helpful, as it tries to communicate another issue: The dynamic linker used for the binary was incorrect.

What is a dynamic linker? Every ELF binary on your Linux system may link dynamically against your system libraries. This may include libraries to access your sound system (libasound), network devices (libcurl) or even just basic file operations (libc). If you run ldd /bin/ls on your host machine, you can see…

ldd /bin/ls
        linux-vdso.so.1 (0x00007ffc58f3f000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x0000766b60df9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000766b60a00000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x0000766b60d5f000)
        /lib64/ld-linux-x86-64.so.2 (0x0000766b60e77000)

…that multiple libraries are required for this binary to run. Most interesting is the dynamic linker, which is the ld-linux-*-entry. Also shown via file /bin/ls…

file /bin/ls
/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3eca7e3905b37d48cf0a88b576faa7b95cc3097b, for GNU/Linux 3.2.0, stripped

…this interpreter will parse the binary and load the required libraries into the memory of the process. If any of the binaries or the interpreter itself are missing, your program will not run - the error message will be not found. On most machines, you can directly call the glibc-linker and ask it to run the binary or to report the linked libraries (same output as ldd, use --help to discover even more)…

/lib64/ld-linux-x86-64.so.2 --list /bin/ls
	linux-vdso.so.1 (0x00007fff1af9a000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007c7158546000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007c7158200000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007c71584ac000)
	/lib64/ld-linux-x86-64.so.2 (0x00007c71585c4000)

So, what went wrong when I compiled the wg binary for the JetKVM? Well, most of the diagnostic tools are missing on that machine…

file ./wg
/bin/sh: file: not found

ldd ./wg
/bin/sh: ldd: not found

/lib/ld-uClibc.so.0 ./wg
Standalone execution is not enabled

…but I was able to find the documentation for the linker, which seem to implement some environment variables regardless. Throwing this knowledge onto some existing system binary, revealed the underlying problem:

LD_WARN= LD_TRACE_PRELINKING=1 LD_TRACE_LOADED_OBJECTS=1 /bin/busybox 
        libc.so.0 => /lib/libc.so.0 (0xa6e71000)
        ld-uClibc.so.1 => /lib/ld-uClibc.so.0 (0xa6f00000)

Turns out, the JetKVM is using a smaller, alternative libc implementation, which is called uClibc-ng. My host-machine is based on glibc, so any binary compiled by my hosts native cross-compiler would link against that. Whoops, what do we do now?

Building `wg` - the right way…

There is a difference between just using the hosts cross-compiler and using a whole cross-compiler toolchain. The latter not only includes a compiler for the target architecture, but also allows to link against alternative libc implementations, other Linux kernel versions or even whole other operating systems. While multiple tools for such job exist, I want to mention the two most popular ones:

Buildroot is not only a tool to build the target toolchain, but also includes a packaging system to create an entire Linux from scratch. In my case, this would be overkill, as I just need to get some binary working and do not want to port the whole JetKVM build-infrastructure to Buildroot.
Crosstool-NG allows to build a cross-compiler toolchain for a specific target architecture. This is exactly what I need, as I just want to compile a single binary for the JetKVM.

Setting up Crosstool-NG

Accessing the JetKVM sources, they also published a repository containing the build-scripts to create the base system firmware image. Digging deeper, it also contains a copy of a pre-compiled toolchain - and looking at the config, it is even based on a slightly older version of Crosstool-NG. But, there is a problem: Not all necessary source-code packages were pushed and also the config seems to be outdated, as it references Linux 5.10.66, but on-device is already Linux 5.10.120 running. While I directly opened an issue about that, Rockchip will likely take a while to provide those file - if ever. So, I decided to just build the toolchain myself - having never done that before, this was quite an adventure.

# prepare your host system -> https://crosstool-ng.github.io/docs/install/ -> https://github.com/crosstool-ng/crosstool-ng/blob/master/testing/docker/ubuntu22.04/Dockerfile
sudo apt install gcc g++ gperf bison flex texinfo help2man make libncurses5-dev python3-dev autoconf automake libtool libtool-bin gawk wget bzip2 xz-utils unzip patch libstdc++6 rsync git meson ninja-build

# obtain the toolchain tool -> https://crosstool-ng.github.io/docs/install/#clone
git clone https://github.com/crosstool-ng/crosstool-ng -b crosstool-ng-1.27.0
pushd crosstool-ng

# build the toolchain infrastructure, required to build the target toolchain itself
./bootstrap
./configure --enable-local # just do it in-place, as I do not want to install it globally
make

Now that we have the infrastructure to build the toolchain, we need a target configuration. A common tool to configure such stuff is Kconfig, which is also used here. Luckly, the command ./ct-ng list-samples revealed a configuration, closely matching the one used by the JetKVM: arm-unknown-linux-uclibcgnueabihf - quickly examining the target triplet (well, here more like a quadlet)…

arm - the target architecture
unknown - the vendor (not specified)
linux - the operating system
uclibcgnueabihf - the libc implementation based on uClibc, the compiler and the ABI

So, let’s select that configuration and start building (which took quite a while)…

./ct-ng arm-unknown-linux-uclibcgnueabihf
export CT_PREFIX=$(pwd)/dist # otherwise the toolchain will be installed into ~/x-tools/..., what I do not want, so I can easilier cleanup everything
./ct-ng build

Compiling `wg` with the own toolchain

This is now a quite easy task, as we just need to replace the compiler used:

make -C src clean
CFLAGS="-static -Os" CC="$CT_PREFIX/arm-unknown-linux-uclibcgnueabihf/bin/arm-unknown-linux-uclibcgnueabihf-gcc" make -C src -j$(nproc)

Once again, uploading the binary as described before - but now…

# on the JetKVM
./wg genkey
oC0loAlxS+GErei4pbQkd9tuwGRpQxFTBak+35lOhVM=

…it works! 🎉

Setting up WireGuard

…for one-time use. Basically, you can now follow the instructions on the official page to setup your WireGuard interface wg0. I was extra lazy and used the peer configuration generator of my OPNsense firewall, stored e.g. as myconfig.conf…

modprobe wireguard
ip link add dev wg0 type wireguard
ip address add dev wg0 192.168.2.1/24 # use the value from Interface.Address inside your config
./wg setconf wg0 myconfig.conf # WARNING: "wg" IS NOT "wg-quick" -> see note below
ip link set up dev wg0
./wg show # check if connection is established, as usual

As described before, the wg-tool is not the same as the wg-quick-tool, so you have to modify your configuration file to not include the Interface.Address-key.

Making it reboot-persistent

…is a bit more tricky. The JetKVM is using a custom init-system, which is not based on systemd or openrc, but on a custom shell-script based system. The scripts are located in /etc/init.d and are executed in alphabetical order. So, to make your WireGuard connection persistent, you have to write a script, place it in /userdata (to prevent it from getting lost during upgrades) and then link it into /etc/init.d. Here is an example script (stolen from StackExchange), store it using vi as /userdata/wg-starter:

#!/bin/sh
set -x
exec > /tmp/wg-starter-log.txt 2>&1

start() {
    /sbin/modprobe wireguard
    /bin/sleep 30 # WireGuard will attempt to resolve the DNS name before the network is up
    /sbin/ip link add dev wg0 type wireguard
    /sbin/ip address add dev wg0 192.168.2.1/24
    /userdata/wg setconf wg0 /userdata/myconfig.conf
    /sbin/ip link set up dev wg0
}
   
stop() {
    /sbin/ip link delete dev wg0
}
   
case "$1" in 
    start)
       start
       ;;
    stop)
       stop
       ;;
    restart)
       stop
       start
       ;;
    *)
       echo "Usage: $0 {start|stop|restart}"
esac

exit 0

Then link it into the init-system:

ln -s /userdata/wg-starter /etc/init.d/S99wg-starter

Also, for a better user-experience I recommend linking the wg-binary into /usr/bin:

ln -s /userdata/wg /usr/bin/wg

Done. 🚀

Closing words

First off, this solution will not survive system updates: As you have to place the link for the wg-starter into the system_* partition, upon update it will be overridden, hence deleting the link again. Even worse, due to the JetKVMs Web-RTC implementation, you’ll may run into a couple of issues, where the screen does not connect properly. On the Tailscale side, this is also known and they are investigating potential fixes.

Overall I’m happy with the solution, as I learned again some stuff about embedded development. For now, you’ll just have to wait until their Web-RTC implementation is fixed, so you can use the VPN connection properly.

Appendix

Notes to rebuild their toolchain

put https://github.com/jetkvm/rv1106-system/blob/dev/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/arm-rockchip830-linux-uclibcgnueabihf_defconfig to .config
yes "" | ./ct-ng oldconfig
export build=$(pwd)
these “developers” are using 1:1 an Android kernel from a custom location (likely with custom patches), so the toolchain needs to access it (a more recent copy is included in their repository)
1. mkdir src
2. ln -sf $(pwd)/../rv1106-system/sysdrv/source/kernel.tar.xz ./src/linux-5.10.66.tar.xz
3. mkdir -p .build/arm-rockchip830-linux-gnueabihf/src/linux-5.10.66
4. args - this does not work and the ISL sources are also completely missing… so I stopped at that point
???

Some discoveries

The board is using an own, closed-source bootloader-binary (likely in ROM), which then dispatches U-Boot from one of the boot_*-partitions (aka slots)
- Its configuration is matching the Android AVB configuration structs, stored in /dev/block/by-name/misc
- Take a look into the sources of the rk_ota tool, which is used for OTA updates
  - rk_ota --misc=display will show the current slot and other useful information
They are using 1:1 an Android Kernel, likely provided directly by Rockchip
- The /proc/cmdline likely stems NOT from U-Boot, but from the kernel itself
- The Android kernel seems to initialize the /dev on boot on its own, allowing the root-cmdline parameter to be used, which references the correct system_*-slot
- Some kernel modules are only available as binaries - which are closed-source 🙄
The boot is using the Busybox init-system with custom scripts
- S10splash just dumps a pre-defined binary into the framebuffer upon boot, the built-in display is available under /dev/fb0
- S20linkmount
  - may creates the /dev/block/by-name-links, if not existing
  - it does NOT mount the system_*-partitions, as these calls are parametrized using IGNORE?!
  - it does mount the /userdata
- /etc/fstab does not match the mount-outputs, so it is likely not used at all
I pulled down some partition images and binaries - if you have to analyze them on your host, use these commands:
- JetKVM: cat /path/to/file | nc YOUR_HOST_IP_ADDRESS 1234
- Host: nc -l -p 1234 -q 1 > file

The setting…

The quick route

Diving deeper

Building `wg` - the wrong way…

Building `wg` - the right way…

Setting up Crosstool-NG

Compiling `wg` with the own toolchain

Setting up WireGuard

Making it reboot-persistent

Closing words

Appendix

Notes to rebuild their toolchain

Some discoveries

Content

Search

Metadata

Matrix

Categories

Tags

WireGuard VPN on a JetKVM

The setting…

The quick route

Diving deeper

Building wg - the wrong way…

Building wg - the right way…

Setting up Crosstool-NG

Compiling wg with the own toolchain

Setting up WireGuard

Making it reboot-persistent

Closing words

Appendix

Notes to rebuild their toolchain

Some discoveries

Content

Search

Metadata

Matrix

Categories

Tags

Building `wg` - the wrong way…

Building `wg` - the right way…

Compiling `wg` with the own toolchain