What is going on?

Hi! Something lovely today. Starting this year I picked up my reMarkable 2 e-ink tablet again and wanted to use it. A while ago I experimented with running Entware on it, but have not re-enabled it, I thought… However, I already observed before, if the device was turned of for “a while”, it would need two attempts to start: First would get stuck in the bootscreen, the second then load to the normal PIN-entry. It did bother me a little, but as the software was working otherwise fine… I did not address it.

Well, enter the faithful 2026-02-21, on which I realized that now the reMarkable cloudservice would not synchronize anymore. While I generally do not care about the sync-state of files with the cloud, I still use it to send myself documents in a PDF format for archival or further processing. So this was finally the issue, which triggered my will to investigate.

Broken DNS Resolution

First off, I attempted to perform a system update - maybe reMarkable broke it in my older software version (I always keep auto-updates disabled for this device) and already addressed my issue. Sadly, the device also refused to download any firmware update as their server could not be reached - annoying! Luckily, reMarkable offers SSH-connections to their device using a randomized device-password, so one can directly interact with the Linux under the hood.

In there I first queried systemctl status for dead units and immediately landed a hit: The systemd-resolved unit was not started! It is responsible for DNS resolution and required by both the cloud-sync and firmware-update functions. This is what I copy-pasted from my terminal:

root@reMarkable:~# systemctl status systemd-resolved
○ systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found ordering cycle on systemd-resolved.service/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on systemd-tmpfiles-setup.service/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on local-fs.target/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on opt.mount/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on home.mount/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on xochitl.service/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on dbus.socket/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Found dependency on sysinit.target/start
Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Job systemd-resolved.service/start deleted to break ordering cycle starting with sysinit.target/start
root@reMarkable:~#

The usual systemctl start systemd-resolved did not help, so I needed to search the web a bit. Turns out, that this unit requires write-access to the /var/tmp path to start - but…

root@reMarkable:~# ls -la /var/tmp 
lrwxrwxrwx    1 root     root            12 May  6 10:09 /var/tmp -> volatile/tmp
root@reMarkable:~# ls /var/volatile/
root@reMarkable:~# 

…reMarkable placed a symlink there instead and its target directory was missing. Specifically it got disappeared at some point, as I recall the device working fine a few months ago. To fix this one quickly, I executed:

root@reMarkable:~# mkdir /var/volatile/tmp
root@reMarkable:~# systemctl enable --now systemd-resolved

…and the cloud sync immediately started working again and flooding my inbox with queued delayed emails. Success!

Broken Firmware Updates

I wanted to apply a software update to the tablet regardless, as I was already connected via SSH… So, I went into the settings, enabled updates and performed a search - yay it found one! But, after installing the update, the usual “reboot to apply software version xxx…” button did not show up and even a manual reboot did not change the software version. Attaching via SSH showed another issue in the journal of their fakeupdateengine_service:

Jul 11 09:43:30 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 69 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:31 reMarkable kernel: EXT4-fs (mmcblk2p2): mounted filesystem with ordered data mode. Opts: (null)
Jul 11 09:43:31 reMarkable kernel: ext4 filesystem being mounted at /mnt supports timestamps until 2038 (0x7fffffff)
Jul 11 09:43:31 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 70 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:31 reMarkable swupdate[466]: FAILURE ERROR : Error opening lock file /var/lock/fw_printenv.lock
Jul 11 09:43:31 reMarkable swupdate[466]: FAILURE ERROR : execute postinstall scripts failed
Jul 11 09:43:31 reMarkable swupdate[466]: FATAL_FAILURE Installation failed !
Jul 11 09:43:31 reMarkable swupdate[466]: IDLE Waiting for requests...
Jul 11 09:43:31 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 71 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:31 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 72 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:31 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 73 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:32 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 74 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:32 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 75 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:32 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 76 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:32 reMarkable swupdate[466]: FAILURE ERROR : No suitable .swu image found
Jul 11 09:43:32 reMarkable swupdate[466]: FAILURE ERROR : Error processing update chunk named '3.27.3.0', version 3.27.3.0, part default
Jul 11 09:43:32 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 77 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}
Jul 11 09:43:33 reMarkable fakeupdateengine_service[765]: PROGRESS 100 99353088 78 "3.27.3.0" 2 2 waveforms-3.27.3.0-rm2-public.tar.gz rawfile 2 0 {"0": {"VERSION" : "3.27.3.0"}}

Hmmm, so their used uboot-tools environment utility refused to change the bootloader configuration, as /var/lock/ is also damaged. Turns out reMarkable did something similar there as well:

root@reMarkable:~# ls /var/lock/
ls: /var/lock/: No such file or directory
root@reMarkable:~# ls -la /var/lock
lrwxrwxrwx    1 root     root            11 May  6 10:09 /var/lock -> ../run/lock
root@reMarkable:~# ls /run/lock
ls: /run/lock: No such file or directory
root@reMarkable:~# mkdir /run/lock
root@reMarkable:~# 

One additional reboot later and the updates also started working fine again.

…and all the rest

As eluded earlier, the tablet needs multiple attempts to boot after not being used for a while. This fixes itself after at least one partial boot attempt, so I assume it also has something to do with some automatic kind of folder cleanup… Wait, SystemD has something for that!

The root-cause?

Well, reMarkable uses the systemd-tmpfiles utility (via systemd 255 (255.21^)) to create, maintain and cleanup these folders. If you run:

root@reMarkable:~# systemd-tmpfiles --cat-config
# /usr/lib/tmpfiles.d/00-create-volatile.conf
#This goes hand-in-hand with the base-files of OE-Core. The file must
# be sorted before 'systemd.conf' becuase this attempts to create a file
# inside /var/log.


d		/run/lock		1777	-	-	-
d		/var/volatile/log		-	-	-	-
d		/var/volatile/tmp		1777	-	-
L		/var/tmp		-	-	-	-	/var/volatile/tmp
<...>
# /usr/lib/tmpfiles.d/tmp.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d

…you can see that somebody (with a typo at “becuase”) added these folders into its configuration. They used the d mode, so they get created and only the contents are going to be deleted. This does not work out as initially thought of, as these target-paths are in reality tmpfs instances (not persisted to disk):

root@reMarkable:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root               261787    220729     23138  91% /
devtmpfs                349480         0    349480   0% /dev
tmpfs                   513832         0    513832   0% /dev/shm
tmpfs                   205536       660    204876   0% /run
tmpfs                     4096         0      4096   0% /sys/fs/cgroup
tmpfs                   513832        12    513820   0% /tmp
/dev/mmcblk2p1           20422       544     19878   3% /var/lib/uboot
/dev/mmcblk2p4         6722700    584200   5777284   9% /home
/dev/mmcblk2p4         6722700    584200   5777284   9% /opt
tmpfs                   513832         0    513832   0% /var/volatile

This means after (re-)boot these paths are missing, until systemd-tmpfiles-setup.service invokes systemd-tmpfiles to re-create these paths. Note, there is a minor error logged by it:

root@reMarkable:~# systemd-tmpfiles --create
/usr/lib/tmpfiles.d/tmp.conf:12: Duplicate line for path "/var/tmp", ignoring.

But in reality the tmpfs should already been mounted at that point, so changes by systemd-tmpfiles-setup.service should not be lost. So, I assume my issues have something to do with the original error logged by systemd-resolved:

Jul 11 09:36:53 reMarkable systemd[1]: sysinit.target: Job systemd-resolved.service/start deleted to break ordering cycle starting with sysinit.target/start

…Oh no…

The root-cause: It was me.

Turns out, during my initial experiments with Entware I had installed a hook into the root user, which partially restored Entware upon login - silently. This lead to an unit opt.mount being placed in /etc/systemd/system/, which triggers the ordering cycle issue from before. More interestingly, this cycle gets created by a specific line in the /etc/fstab authored by reMarkable:

/dev/unknown         /home                ext4       defaults,nofail,noauto,x-systemd.after=xochitl.service,x-systemd.device-timeout=0 0  3

Hence, the /home path is to be mounted, after the xochitl.service unit gets started, which in turn is part of multi-user.target, which itself depends (…) on local-fs.target to be ready… Then opt.mount being the one to depending on the /home mount (part of the local-fs.target) and itself being needed to succeed local-fs.target. Yay!

Hence, there is a new bug for Entware which I accidentally ran into. Ooops, how embarrassing.

Why post this now?

I collected all my logs, instructions for a workaround and reported them to the customer support of reMarkable on the same day. They replied back on the 2026-02-22 and queried for a lot of additional information like my proof of purchase and address details, which I refused to provide - after all, this was entirely out of scope for the initially reported bug. Their final reply was then, referring to my own successful “repair”:

It is great to know that the issue has already been fixed. I would also like to thank you for your efforts in finding the root cause of the issue and providing an immediate resolution that would help resolve similar issues if they were encountered in the current firmware.

Here at reMarkable, we value all your feedback on how we can improve our products and services. We’ll pass this on to our headquarters to be considered as we contemplate future enhancements.

So, everything is good then - right?

Picky readers may have noticed, that I upgraded from version 3.27.1.0 to version 3.27.3.0 in the logs above, which does not match this timeline. These firmware versions are much more recent than what was available as 2026-02. This is because I ran into the exact same set of issues today (2026-07-11), multiple software versions later as when I reported the bug the first time. Hence, I re-raised the issue with the reMarkable support and they did a remarkable job of highlighting the actual issue at 2026-07-30:

Based on our analysis, it appears that the device’s filesystem and startup configuration differ from the standard reMarkable software configuration. Specifically, the logs indicate a non-standard /opt mount associated with /home, which results in a startup dependency cycle and is consistent with the behavior you’ve reported.

So, I reevaluated my assumption that I had disabled (not removed) changes previously made for Entware and then realized that I was the issue. As the first version of this post was written way more accusatory of reMarkable for containing a software bug (they never had in the first place), I chose to patch this post instead and keep it online for your enjoyment.

I hope this helps you out getting your device back running and not making the same mistake as I did: Document your own non-standard changes, so you know of what hacks you did, which may come around later to bite you…