ladas552.me
Site for my Nix docs
nix.ladas552.me
1@document.meta
2title: Impermanence on NixOS with ZFS and tmpfs
3description: Guide for my impermanence setup
4authors: [
5 ladas552
6]
7categories: [
8 Impermanence
9 ZFS
10 Guide
11]
12created: 2026-01-02
13draft: false
14layout: post
15version: 1.1.1
16@end
17
18** What is Impermanence
19 ___
20 It wipes your `/root` on reboot and your startup is a blank canvas, but you can persist mounts and bind mount directories from it in your normal root to save stuff like cache and tokens. So you wipe all the junk and save actually useful stuff.
21
22 For example you can install full KDE Plasma session, run it, and if you get bored. Just disable it and no KDE junk left.
23
24 *Important to note*: That impermanence of my setup uses tmpfs, so it writes `/root` to RAM, so nothing actually gets erased on the Disk. Meaning no continuous I/O rewrites wearing out your Drive(/not that it would mater in practice/). But the state isn't saved between reboots, as with anything on RAM
25
26 Also when I refer to `/root`, it's actually the whole `/`, not just root user directory.
27*** Why did you set it up
28 ___
29 I was bored. I don't find benefits of impermanence so crucial to completely overhaul how your system behaves and I don't trust myself to maintain it.
30
31 But there are some benefits to it:
32 - I only backup important files, no cache, no states, only files and media;
33 - I always know what's on my system because it's declared in the config;
34 - It opens up possibilities to experiment more with my system, because if I could setup impermanence and not loose all my files, I am unstoppable;
35*** What's the meaning of writing this page?
36 ___
37 It's not that hard to setup impermanence, but to requires reading a lot of stuff, and if you don't use ZFS or BTRFS even full reinstall for rearranging partitions. I have read several articles, watched videos, and stole code from many GitHub repos.
38
39 Plus most guides just go to the wipe stage right away, without saying how to persist, or how it practically works for the user to not loose their files. I will try to compete in these aspects.
40
41*** What is your current setup?
42 ___
43
44 I got ZFS with tmpfs, with 2 persistence datasets. `/cache` and `/persist`. And a plaint vFAT `/boot` partition, where GRUB or Systemd-boot will put generation images.
45
46 Most people assume you gotta have 64Gb or RAM for tmpfs to be convenient, but actually you just need a well structured disk layout. Then the tmpfs usually only takes 50MB to 100MB of RAM.
47
48 Template structure of ZFS datasets that will be essential:
49
50 - `/cache` is for rust targets, everything in `~/.cache`, .local states, etc.
51
52 - `/persist` is for Media, browser profiles, Projects, etc. This is the only datasets that get's backed up by `sanoid`.
53
54 - `/nix` for /nix/store. NixOS won't boot without it. All the files that aren't persisted, but appear on my system are symlinked from `/nix`. That includes config files and services.
55
56 - `/tmp` for /tmp. yeah, anyways it's to not overload tmpfs when downloading something on browser. with `boot.tmp.cleanOnBoot = true;` it is cleared on boot anyways.
57
58 tmpfs is erased on reboot, so `/` and everything below it, including `/home` is gone, unless put into `/cache` or `/persist` datasets.
59 tmpfs is on RAM, so it can overload if exceeds certain size, to prevent that I got several more zfs datasets, that aren't persisted, meaning they don't have connection to files in other datasets, but aren't erased by default.
60
61** What we need?
62*** A ZFS setup
63 This isn't a *ZFS guide* so, unless you have *ZFS setup* on your *NixOS*, you can kiss this Guide goodbye. Go read OpenZFS documentation. But here is my installation script in case you'd like more guiding. But seriously, go read docs, real engineers know a lot more than I do.
64
65 {https://github.com/Ladas552/Flake-Ocean/blob/e460837d18f37723510f9b74c46636fd2b5b4f25/install/impermanence.norg}[permalink in the github repo, please insure the branch is up to date before checking in], it contains actual descriptions to each line as a Norg file format that you can tangle. Like bible in org mode.
66 @code sh
67 sudo zpool create -f \
68 -o ashift=12 \
69 -o autotrim=on \
70 -O compression=zstd \
71 -O acltype=posixacl \
72 -O atime=off \
73 -O xattr=sa \
74 -O normalization=formD \
75 -O mountpoint=none \
76 zroot "/dev/sda2"
77
78 sudo zfs create -o mountpoint=legacy zroot/root
79 sudo mount -t zfs zroot/root /mnt
80
81 sudo mount --mkdir "$BOOTDISK" /mnt/boot
82 # All the stuff below will be explained later
83 sudo zfs create -o mountpoint=legacy zroot/nix
84 sudo mount --mkdir -t zfs zroot/nix /mnt/nix
85
86 sudo zfs create -o mountpoint=legacy zroot/tmp
87 sudo mount --mkdir -t zfs zroot/tmp /mnt/tmp
88
89 sudo zfs create -o mountpoint=legacy zroot/cache
90 sudo mount --mkdir -t zfs zroot/cache /mnt/cache
91
92 sudo zfs create -o mountpoint=legacy zroot/persist
93 sudo zfs snapshot zroot/persist@blank
94 sudo mount --mkdir -t zfs zroot/persist /mnt/persist
95 # All the stuff above will be explained later
96 sudo nixos-install --no-root-password --flake "github:Ladas552/Nix-Is-Unbreakable#NixVM"
97 @end
98
99*** Partitions
100 ___
101 A new way to manage your system. NixOS.
102
103 Tho you probably already use NixOS if you are reading this, if you don't then get out while you can.
104
105 On a more serious note, you need ZFS setup, with 2 particular datasets.
106 @code nix
107 fileSystems = {
108 "/nix" = {
109 device = "zroot/nix";
110 fsType = "zfs";
111 };
112 "/tmp" = {
113 device = "zroot/tmp";
114 fsType = "zfs";
115 };
116 };
117 @end
118
119 If you don't have them, but have ZFS installed, just create them using commands
120 @code sh
121 sudo zfs create -o mountpoint=legacy zroot/tmp
122 sudo zfs create -o mountpoint=legacy zroot/nix
123 @end
124
125 This will insure that you won't delete your `/nix/store` and it stays intact between reboots. And for this particular setup the `tmp` dataset will be used so our `tmpfs` *root* will insure that it won't randomly overload.
126
127*** Impermanence module
128 ___
129 The [Impermanence module]{https://github.com/nix-community/impermanence} is a NixOS flake that creates `mount binds`. The main purpose of it is to just put stuff in special `/persist` dataset, and still be able to access it from `/root` and `/home`. More about technicality of bind mounts later
130
131 Basically you define certain directories names in it, and it creates them, then binds them to specific relevant locations, like `".config/nvim"` will be located in `~/.config/nvim`. And if you put your Neovim config there, neovim will still follow the config, but it will be located on different dataset, and won't be wiped on boot.
132
133 Neat right? Not really, because if directory already exists, Impermanence will override that old directory with new empty one. *Don't panic*. Data isn't lost, it was just reallocated, you can delete the directory from impermanence module and it will comeback.
134
135 That's the main reason why most people reinstall their OS if they want to use Impermanence, because it's a pain in the glands to move the files from directories before persisting it and moving things back. There are projects that circumvent that, but I didn't use them. For example: [Persist-retro]{https://github.com/Geometer1729/persist-retro}.
136
137 Also to persist an individual file, you need to move the file, and manually copy it to persist directory. Otherwise it complains about the original file being in the way of a mount bind.
138
139**** You forgot to tell installation instructions
140 ___
141 It's nix so here is just a snippet of code. Works for flakes.
142 @code nix
143 #flake.nix
144 {
145 inputs.impermanence.url = "github:nix-community/impermanence";
146 }
147 @end
148 And then just import the module, like:
149 @code nix
150 imports = [
151 inputs.impermanence.nixosModules.impermanence
152 ];
153 @end
154
155 We will only use the `nixosModule` because I don't have standalone Home-Manager and not planning to adopt impermanence for distros outside of NixOS.
156
157 I have seen people use impermanence module on non flake setups, but I am not so interested in them to find and link a good one.
158*** Immutable users
159 ___
160 As we delete everything in `/root`, it means passwords for users, and most importantly `root` user will be deleted.
161
162 So just make them immutable. You can store the password file in sops, or just provide raw path from `/persist` directory.
163
164 @code nix
165 {
166 # setup immutable users for impermanence
167
168 # silence warning about setting multiple user password options
169 # https://github.com/NixOS/nixpkgs/pull/287506#issuecomment-1950958990
170 # Stolen from Iynaix https://github.com/iynaix/dotfiles/blob/4880969e7797451f4adc3475cf33f33cc3ceb86e/nixos/users.nix#L18-L24
171 options = {
172 warnings = lib.mkOption {
173 apply = lib.filter (
174 w: !(lib.hasInfix "If multiple of these password options are set at the same time" w)
175 );
176 };
177 };
178
179 config = {
180 # disabling user mutability
181 users.mutableUsers = false;
182
183 # defining regular user, ME!
184 users.users.ladas552 = {
185 isNormalUser = true;
186 description = "Ladas552";
187 extraGroups = [
188 "networkmanager"
189 "wheel"
190 ];
191 initialPassword = "pass";
192 # Use a path or your encryption method here
193 hashedPasswordFile = config.sops.secrets."mystuff/host_pwd".path;
194 };
195
196 nix.settings.trusted-users = [ "ladas552" ];
197
198 # Setting root user
199 users.users.root = {
200 initialPassword = "pass";
201 hashedPasswordFile = config.sops.secrets."mystuff/host_pwd".path;
202 };
203 };
204 }
205 @end
206
207 Other features for immutable users:
208 - Can use `--no-root-password` flag in `nixos-install` command. Meaning you don't ever have to monitor it, it will install password automatically.
209 - Can't use `passwd <user>` command. So if you mess up your password path the first time, you have to reboot to previous generation to set it correctly.
210
211 The `initialPassword` is set as plain text because it suppose to be a backup if sops decryption failed, so you won't leave with useless system state. Otherwise, it's unused and won't have security implications for your host.
212
213*** When do we start deleting stuff?
214 ___
215 Not so fast bakaru, we first need to save our stuff.
216
217 So you need to create persisted directories
218 @code sh
219 sudo zfs create -o mountpoint=legacy zroot/persist
220 sudo zfs create -o mountpoint=legacy zroot/cache
221 @end
222
223 And now we add them to be mounted on boot
224 @code nix
225 # persist mount
226 fileSystems."/persist" = {
227 device = "zroot/persist";
228 fsType = "zfs";
229 # so it's required to boot, and you won't reboot into empty desktop
230 neededForBoot = true;
231 };
232
233 # cache are files that should be persisted, but not to snapshot
234 # e.g. npm, cargo cache etc, that could always be redownload
235 "/cache" = {
236 device = "zroot/cache";
237 fsType = "zfs";
238 neededForBoot = true;
239 };
240 @end
241
242 I also recommend setting up backups with sanoid if you didn't already.
243
244 @code nix
245 services.sanoid = {
246 enable = true;
247 # if you have sanoid options somewhere else, lib.mkForce
248 # will override anything, so you only have snapshots that matter
249 datasets = lib.mkForce {
250 "zroot/persist" = {
251 hourly = 50;
252 daily = 15;
253 weekly = 3;
254 monthly = 1;
255 };
256 };
257 };
258 @end
259
260 Now we have basic datasets that will store out stuff, Impermanence can wait now, we need to assign bind mounts for directories and files!
261
262 Don't know what bind mounts are? Well simply put. They are mounts that make some directory to appear in normal location, but actually it's in a different dataset all together.
263
264 So for example: `/home/alice225/Downloads` will be deleted on boot. But, not it's content. On the next boot, the content of `Downloads` that is in `/persist` dataset will remount itself to `/home/alice225/Downloads` path.
265
266 It will appear seamless to other applications and to yourself. But now you can access the same files in both `/home/alice225/Downloads` and in `/persist/home/alice225/Downloads`. And remember, it is *not a symlink*. Symlinks fool programs, while bind mounts genuinely make files accessible in several locations. But be careful, because they share permissions, and can also be deleted.
267
268*** Okay, we have locations, let's move some files into them
269 ___
270 So, with Impermanence module, there are some options available. Simplest approach would be to use it directly
271
272 @code nix
273 environment.persistence = {
274 # one of out datasets
275 "/persist" = {
276 # useful option
277 hideMounts = true;
278 directories = [
279 # absolute path to directories in string values
280 "/var/log"
281 "/var/lib/nixos"
282 "/etc/NetworkManager/"
283 ];
284 };
285 };
286 @end
287
288 Now you are able to just define your paths as normal and save them. But this is just normal impermanence, This blog post is about *My* setup specifically, so how about we add some abstractions to the vanilla scheme.
289
290**** New options
291 ___
292 `directories` and `files` in impermanence module are just a list of string, so we can make pseudo options to add more strings to the list and `++` them with actual persistence option, or just reference our list.
293
294 It will make it easier to define directories under different scopes. For example, setting files in `home.nix` file, but they will be used in `configuration.nix` file.
295
296 @code nix
297 {lib,...}:
298 {
299 options = {
300 # options to put directories in, persistence but shortened
301 # stolen from @iynaix
302 root = {
303 directories = lib.mkOption {
304 type = lib.types.listOf lib.types.str;
305 default = [ ];
306 description = "Directories to persist in root filesystem";
307 };
308 files = lib.mkOption {
309 type = lib.types.listOf lib.types.str;
310 default = [ ];
311 description = "Files to persist in root filesystem";
312 };
313 cache = {
314 directories = lib.mkOption {
315 type = lib.types.listOf lib.types.str;
316 default = [ ];
317 description = "Directories to persist, but not to snapshot";
318 };
319 files = lib.mkOption {
320 type = lib.types.listOf lib.types.str;
321 default = [ ];
322 description = "Files to persist, but not to snapshot";
323 };
324 };
325 };
326 home = {
327 directories = lib.mkOption {
328 type = lib.types.listOf lib.types.str;
329 default = [ ];
330 description = "Directories to persist in home directory";
331 };
332 files = lib.mkOption {
333 type = lib.types.listOf lib.types.str;
334 default = [ ];
335 description = "Files to persist in home directory";
336 };
337 cache = {
338 directories = lib.mkOption {
339 type = lib.types.listOf lib.types.str;
340 default = [ ];
341 description = "Directories to persist, but not to snapshot";
342 };
343 files = lib.mkOption {
344 type = lib.types.listOf lib.types.str;
345 default = [ ];
346 description = "Files to persist, but not to snapshot";
347 };
348 };
349 };
350 };
351 }
352 # Holy cow nix is indented to all suns
353 @end
354
355 You can add more options if need be. In this we only define lists for `/persist` and `/cache`, but they are separated into `root` and `home`. So add `root` and `home` options to `nixocConfiguration` and add only `home` to home-manager for example.
356
357 `home` is just a simple way to separate directories in `/home/ladas552` from just `/`.
358
359 But, you know, I use flake-parts and I don't need to add these options to different files and scope. I can just inherit them all in one file!
360
361 @code nix
362 { lib, ... }:
363 # link to snippet in my config
364 # https://github.com/Ladas552/Flake-Ocean/blob/85ee207aa2e5e0d2e44aad0a0818a533ceca72cf/modules/nixosModules/Impermanence/imp-options.nix
365 {
366 flake.modules =
367 let
368 # options to put directories in, persistence but shortened
369 # stolen from @iynaix
370
371 root = {};
372 home = {};
373 # Same thing as above
374 in
375 {
376 nixos.options.options.custom.imp = { inherit root home; };
377 hjem.options.options.custom.imp = { inherit home; };
378 homeManager.options.options.custom.imp = { inherit home; };
379 };
380 }
381 @end
382
383 This code block is from my Dendrithic config with several module classes. So each `nixos`, `hjem` and `homeManager` classes have their own `options` module that inherit the same type of options in each module scope.
384
385 You probably didn't get any of that, but you don't need to tbh. My setup is My setup, do whatever you want.
386
387**** Persist in action
388 ___
389 Now we can define some important directories and files to persist
390 @code nix
391 custom.imp = {
392 root = {
393 directories = [
394 "/etc/NetworkManager/"
395 "/var/lib/NetworkManager"
396 "/var/lib/iwd"
397 ];
398 };
399 home = {
400 directories = [
401 ".librewolf"
402 ];
403 cache = {
404 files = [ ".local/share/com.jeffser.Alpaca/alpaca.db" ];
405 directories = [
406 ".local/share/nvim"
407 ".local/state/nvim"
408 ".config/libreoffice"
409 ".cache/librewolf"
410 ".cache/keepassxc"
411 ".config/keepassxc"
412 ".cache/nix"
413 ".cache/nix-index"
414 ];
415 };
416 };
417 };
418 @end
419
420 But if you set this thing up, and rebuild it wouldn't do a thing. Remember, these options and list are just place holders. Meant to be easy to write and read. Now we gotta add them to an actual Impermanence module options.
421
422 @code nix
423 { lib, config, ... }:
424 let
425 cfg = config.custom.imp;
426 # config.custom.meta.user is just my placeholder for `username`
427 # just hard code the value, or replace it with your own solution to select a user
428 cfghm = config.home-manager.users."${config.custom.meta.user}".custom.imp;
429 cfghj = config.hjem.users."${config.custom.meta.user}".custom.imp;
430 in
431 {
432 environment.persistence = {
433 "/persist" = {
434 hideMounts = true;
435 # referencing files via our abstraction option
436 files = lib.unique cfg.root.files;
437 directories = lib.unique (
438 # here you can define directories normally
439 [
440 "/var/log"
441 "/var/lib/nixos"
442 ]
443 # and concatenate too!
444 ++ cfg.root.directories
445 );
446 # add persists to `/home/user` path
447 users."${config.custom.meta.user}" = {
448 files = lib.unique ([ ] ++ cfghm.home.files ++ cfghj.home.files);
449 directories = lib.unique (
450 [ ] ++ cfg.home.directories ++ cfghm.home.directories ++ cfghj.home.directories
451 );
452 };
453 };
454 # same as above
455 "/cache" = {
456 hideMounts = true;
457 files = lib.unique cfg.root.cache.files;
458 directories = lib.unique cfg.root.cache.directories;
459 users."${config.custom.meta.user}" = {
460 files = lib.unique (cfg.home.cache.files ++ cfghm.home.cache.files ++ cfghj.home.cache.files);
461 directories = lib.unique (
462 cfg.home.cache.directories ++ cfghm.home.cache.directories ++ cfghj.home.cache.directories
463 );
464 };
465 };
466 };
467 }
468 @end
469
470 You will need to adjust this code snippets to your own config. For example, if you don't use `hjem`. And replace `config.custom.meta.user` with your own username.
471
472 I am not stating this approach is the best, but it just how I ended up using Impermanence module, and it might be useful for you to know.
473
474 Now upon rebuild Impermanence module should add all the interesting directories to datasets and establish bind mounts.
475
476 Wait, did we forget something? Uhhh...
477
478*** DELETE ERASE REDUCTED
479 ___
480 @code nix
481 # replace the root mount with tmpfs
482 # wipes everything if you don't have proper persists, be warned
483 fileSystems."/" = lib.mkForce {
484 device = "tmpfs";
485 fsType = "tmpfs";
486 neededForBoot = true;
487 options = [
488 "defaults"
489 # whatever size feels comfortable, smaller is better
490 "size=1G"
491 "mode=755"
492 ];
493 };
494 @end
495
496 This is all you need. So simple in comparison to the whole page above, right?
497
498 The main reasons for that are:
499 - It's harder to keep what you have gained, but trivial to loose everything you ever had;
500 - Also the `size=1G` wouldn't make it possible to use tmpfs as a main without persists and bind mounts. Bind mount only makes files accessible in 2 locations, but only stored in ZFS dataset.
501
502 This should be all you need to start using tmpfs for impermanence on ZFS.
503
504 There are some niceties I want to share tho, to make your life easier.
505
506*** Some sprinkles to your epitome of agony
507**** Snippets
508 ___
509 Set this so you aren't lectured by `you know what you are doing` lecture from sudo every boot
510 @code nix
511 security.sudo.extraConfig = "Defaults lecture=never";
512 @end
513
514 If you use {https://github.com/Mic92/sops-nix}[sops-nix], set ssh paths to `/persist` because otherwise `nixos-install` won't find the keys.
515 @code nix
516 sops.age.sshKeyPaths = [
517 "/persist/home/vimjoyer/.ssh/ssh-key"
518 ];
519 sops.age.keyFile = lib.mkDefault "/persist/home/vimoyer/.config/sops/age/keys.txt";
520 @end
521
522**** Persist everything
523 ___
524 Persist every bit that might be useful to you, tokens, cookies and all that if they matter to you. Depending on the application, you might wanna persist only one file. But for something like steam, it compiles shader cache, which is persistable with this snippet
525 @code nix
526 # persist steam
527 custom.imp.home = {
528 cache.directories = [
529 ".local/share/Steam"
530 ".cache/mesa_shader_cache"
531 ".cache/mesa_shader_cache_db"
532 ".cache/radv_builtin_shaders"
533 ];
534 };
535 @end
536
537 But how would you know what to persist? Well, first you might want to look at others people config files. Because the best way to avoid pit falls is following the walked road.
538
539 Some pretty extended persist list can be found in following configurations:
540 - {https://github.com/search?q=repo%3Aiynaix%2Fdotfiles+custom.persist&type=code}[Iynaix dotfiles];
541 - {https://github.com/saygo-png/nixos}[Saygo's config];
542 - {https://github.com/xarvex/dotfyls}[Xarvex dotfyls];
543
544 If they don't use the same modules as you, figure it out on your own. Most of the time programs follow xdg conventions and store files in `.config .cache .local/state .local/share`. Or instead of persisting the settings, symlink raw files.
545
546**** Credits and suggestions
547 ___
548 You can suggest anything you'd like to add to {*** Some sprinkles to your epitome of agony}. Cool configs using impermanence, tips, snippets and so on. Just ping me on Discord or write an issue on github. Your username will be added below as a privilege for being so awesome!
549
550 List of awesome people:
551 - *Iynaix*'s impermanence abstraction structure and ZFS setup
552 - *Vimjoyer*'s Impermanence video introducing basic concept to me
553 - *talyz* for making Impermanence module and extensive readme for the project
554 - *Graham* for {https://grahamc.com/blog/erase-your-darlings/}
555 - *Willbush* for {https://willbush.dev/blog/impermanent-nixos/}
556 - *Elis Hirwing* for {https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/}
557 - {https://github.com/fliplus}[Flipus] and *snohater* for providing feedback on readability of the thing.
558 - *You* for reading this all, good luck with your NixOS config. Hope this article helped you the same way all the people mentioned above helped me. Without such a vast community, I wouldn't be able to figure all these things out. And I didn't reinvent a wheel, it's all the work of Open Source contributors. So, hopefully you also share your knowledge in the future. Improve on what's old or reduce it to atoms, it's up to you. It's your setup, and only for you to decide, whether you really want to setup it up.