Linux snapshots?

Back in the day (mid 2000’s) ReiserFS was my go to Linux filesystem, it was fast & reliable. But then after its creator went to jail, which coincided with release of ext4 in 2008. I made a switch to ext4 and never even thought about switching to any other filesystem on my laptops/workstations.

That is until I read that Ubuntu 24.04 has an option to use OpenZFS, which has a key feature that ext4 lacks: the ability to create snapshots. Snapshots are point-in-time copies of your file system that can be invaluable for backup, testing, or recovery. This is especially useful if something goes wrong during a system update, or when you’re developing apps/tools like auto-cpufreq. Allowing you to undo multiple system changes without manually backing up files or remembering the original configuration.

OpenZFS snapshots basics

This being a relatively new thing for Linux/Ubuntu, there wasn’t a lot of material for what I wanted to do and I initially came across “Using ZFS Snapshots and Clones” Ubuntu tutorial.

Which helped me do basic things, like create individual pool snapshots of my i.e home directory. However, it didn’t explain how to take a snapshot of all pools at once, nor did it warn against restoring a snapshot on a live running system. Snapshots should be restored in recovery mode, when none of the filesystems are mounted, to avoid potential issues.

To start of, you can list ZFS pools by running: zfs list and below is an example run:

zfs list            
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
bpool                                             97.9M  1.65G    96K  /boot
bpool/BOOT                                        96.9M  1.65G    96K  none
bpool/BOOT/ubuntu_tkvutm                          96.8M  1.65G  96.8M  /boot
rpool                                              192G   262G   192K  /
rpool/ROOT                                        17.9G   262G   192K  none
rpool/ROOT/ubuntu_tkvutm                          17.9G   262G  4.56G  /
rpool/ROOT/ubuntu_tkvutm/srv                       192K   262G   192K  /srv
rpool/ROOT/ubuntu_tkvutm/usr                       972K   262G   192K  /usr
rpool/ROOT/ubuntu_tkvutm/usr/local                 780K   262G   700K  /usr/local
rpool/ROOT/ubuntu_tkvutm/var                      13.2G   262G   192K  /var
rpool/ROOT/ubuntu_tkvutm/var/games                 192K   262G   192K  /var/games
rpool/ROOT/ubuntu_tkvutm/var/lib                  6.35G   262G  6.20G  /var/lib
rpool/ROOT/ubuntu_tkvutm/var/lib/AccountsService   584K   262G   584K  /var/lib/AccountsService
rpool/ROOT/ubuntu_tkvutm/var/lib/NetworkManager    600K   262G   360K  /var/lib/NetworkManager
rpool/ROOT/ubuntu_tkvutm/var/lib/apt              76.5M   262G  75.9M  /var/lib/apt
rpool/ROOT/ubuntu_tkvutm/var/lib/dpkg             71.4M   262G  59.0M  /var/lib/dpkg
rpool/ROOT/ubuntu_tkvutm/var/log                   764M   262G   759M  /var/log
rpool/ROOT/ubuntu_tkvutm/var/mail                  192K   262G   192K  /var/mail
rpool/ROOT/ubuntu_tkvutm/var/snap                 6.10G   262G  6.10G  /var/snap
rpool/ROOT/ubuntu_tkvutm/var/spool                 416K   262G   252K  /var/spool
rpool/ROOT/ubuntu_tkvutm/var/www                   192K   262G   192K  /var/www
rpool/USERDATA                                     174G   262G   192K  none
rpool/USERDATA/home_rwv34p                         174G   262G   165G  /home
rpool/USERDATA/root_rwv34p                        15.4M   262G  15.0M  /root
rpool/keystore                                    39.0M   262G  16.5M  -

Each of these pools represents a different mount point, main difference being:

  • bpool: This pool contains the boot-related datasets (e.g., /boot).
  • rpool: This pool contains the root filesystem and user data.

Hence, to take a snapshot of my home directory, I would run:

sudo zfs snapshot rpool@rpool/USERDATA/home_rwv34p@home-snap-sep22

“rpool@rpool/USERDATA/home_rwv34p” being the pool name and “@home-snap-sep22” resulting in a “home-snap-sep22” snapshot name.

Which I can see and verify if I run:
zfs list -t snapshot

To restore this individual/single pool snapshot I would run:
zfs rollback rpool/home-snap-sep22

How to create full system backups with OpenZFS snapshots?

To create a recursive snapshot of all pools including the dataset and all its child datasets run:

sudo zfs snapshot -r rpool@snap-full-$(date +"%b-%d") && sudo zfs snapshot -r bpool@snap-full-$(date +"%b-%d")

Result will be a snapshot called i.e: “snap-full-Sep-22”, to verify/list the snapshot, run: zfs list -t snapshot and you’ll see output like this:

zfs list -t snapshot 
NAME                                                                USED  AVAIL  REFER  MOUNTPOINT
bpool@snap-full-Sep-22                                                0B      -    96K  -
bpool/BOOT@snap-full-Sep-22                                           0B      -    96K  -
bpool/BOOT/ubuntu_tkvutm@snap-full-Sep-22                             0B      -  96.8M  -
rpool@snap-full-Sep-22                                                0B      -   192K  -
rpool/ROOT@snap-full-Sep-22                                           0B      -   192K  -
rpool/ROOT/ubuntu_tkvutm@snap-full-Sep-22                             0B      -  4.56G  -
rpool/ROOT/ubuntu_tkvutm/srv@snap-full-Sep-22                         0B      -   192K  -
rpool/ROOT/ubuntu_tkvutm/usr@snap-full-Sep-22                         0B      -   192K  -
rpool/ROOT/ubuntu_tkvutm/usr/local@snap-full-Sep-22                   0B      -   700K  -
rpool/ROOT/ubuntu_tkvutm/var@snap-full-Sep-22                         0B      -   192K  -
rpool/ROOT/ubuntu_tkvutm/var/games@snap-full-Sep-22                   0B      -   192K  -
rpool/ROOT/ubuntu_tkvutm/var/lib@snap-full-Sep-22                     0B      -  6.20G  -
rpool/ROOT/ubuntu_tkvutm/var/lib/AccountsService@snap-full-Sep-22     0B      -   584K  -
rpool/ROOT/ubuntu_tkvutm/var/lib/NetworkManager@snap-full-Sep-22      0B      -   360K  -
rpool/ROOT/ubuntu_tkvutm/var/lib/apt@snap-full-Sep-22                 0B      -  75.9M  -
rpool/ROOT/ubuntu_tkvutm/var/lib/dpkg@snap-full-Sep-22                0B      -  59.0M  -
rpool/ROOT/ubuntu_tkvutm/var/log@snap-full-Sep-22                     0B      -   759M  -
rpool/ROOT/ubuntu_tkvutm/var/mail@snap-full-Sep-22                    0B      -   192K  -
rpool/ROOT/ubuntu_tkvutm/var/snap@snap-full-Sep-22                    0B      -  6.10G  -
rpool/ROOT/ubuntu_tkvutm/var/spool@snap-full-Sep-22                   0B      -   252K  -
rpool/ROOT/ubuntu_tkvutm/var/www@snap-full-Sep-22                     0B      -   192K  -
rpool/USERDATA@snap-full-Sep-22                                       0B      -   192K  -
rpool/USERDATA/home_rwv34p@snap-full-Sep-22                           0B      -   165G  -
rpool/USERDATA/root_rwv34p@snap-full-Sep-22                           0B      -  15.0M  -
rpool/keystore@snap-full-Sep-22                                       0B      -  16.5M  -

Notice that immediately after creating a snapshot, the USED column shows 0B.

This is a good thing! It happens because no changes have been made to any files or datasets since the snapshot was taken. As you modify files, the USED value will grow. This increase reflects the difference between the snapshot and the current state of your system. The snapshot essentially captures a moment in time, and as changes occur, the snapshot tracks how the filesystem diverges from that point.

How to recover/rollback full system backups with OpenZFS snapshots?

For best results in terms of data consistency and results, restore snapshots by booting into (single user) recovery mode. I.e during system boot in GRUB select the (recovery mode) image: “Ubuntu 24.04.1 LTS, with Linux 6.8.9-45-generic (recovery mode)”

Followed by selecting “Advanced options for Ubuntu 24.04.1 LTS”.

When “Recovery Menu (filesystem state: read only)” pops up:
select “root – Drop to root shell prompt” and hit “Enter” during:
“Press Enter for maintenance
(or press Control-D” to continue)”.

To restore above taken rpool@snap-full-Sep-22 & bpool@snap-full-Sep-22 snapshots, all its pools including the dataset and all its child datasets run:

for snap in $(zfs list -t snapshot -H -o name | grep '^rpool|^bpool'); do zfs rollback -r $snap; done

Verity the results by running zfs list -t snapshot and as during the process of taking snapshots, right after the rollback the USED column should shows 0B for all pools and datasets.

That’s it! Your system and all of its files are back to the state when snapshot was taken.

In this process, after taking snapshots I deliberately deleted important private & system config files and wrecked havoc on rest of the setup. However, after snapshot was restored, it was all good as new.

Other useful things to know about OpenZFS

You can’t rename snapshots

Hence the best way to do is delete (destroy) old ones, i.e

For single pool:

sudo zfs destroy bpool@snapshot-name
sudo zfs destroy rpool@snapshot-name

For all pools, datasets and their dataset children:

sudo zfs destroy -r bpool@snap-full-Sep-22
sudo zfs destroy -r rpool@snap-full-Sep-22

OpenZFS is using all my RAM memory on Ubuntu 24.04!

Initially, I thought there was something wrong with my Ubuntu 24.04 installation using OpenZFS and LUKS for encryption. Even after a fresh boot and with no applications running, my used RAM consistently hovered around the 90% mark.

Turns out this is a ZFS feature because it’ll cache all system data & metadata. When new apps/processes are run that require a lot of pages, cached data will be evicted and allocated back to the apps that need it.

To see how much resources are exactly used by ZFS ARC (adaptive replacement cache) run: arc_summary for full overview or: arc_summary | grep "Min size" for an exact number.

Alternatively simply use htop which gives you a more relevant RAM usage with OpenZFS, as it will show RAM usage with help of following colors:

  • Green = Used memory
  • Blue = Buffers
  • Yellow/Orange = Cache

Happy hacking!

, , , , , , , , , , , , ,
Show Comments