A passthrough FUSE filesystem that intelligently moves files between storage tiers based on frequency of use, file age, and tier fullness.
autotier
is a tiered FUSE filesystem which acts as a merging passthrough to any number of underlying filesystems. These underlying filesystems can be of any type. Behind the scenes, autotier
moves files around such that the most often accessed files are kept in the highest tier. autotier
fills each defined tier up to their configuration-defined quota, starting at the fastest tier with the highest priority files. If you do a lot of writing, set a lower quota for the highest tier to allow for more room. If you do mostly reading, set a higher watermark to allow for as much use as possible out of your available top tier storage.
- Install
- Configure
- If data is already on any tiers, run
autotier-init-dirs /path/to/tier1 /path/to/tier2 [ /path/to/tier3 ... ]
to clone directory structures across all tiers. (Note on conflicting file paths between tiers) - Mount Filesystem
- Get deb:
$ wget https://github.com/45Drives/autotier/releases/download/v1.2.0/autotier_1.2.0-1focal_amd64.deb
- Install deb:
# apt install ./autotier_1.2.0-1focal_amd64.deb
- Edit configuration file.
- Mount filesystem.
- Optionally add user to
autotier
group to allow non-root users to run CLI commands:# usermod -aG autotier <user>
(takes effect on next login)
- Get deb:
$ wget https://github.com/45Drives/autotier/releases/download/v1.2.0/autotier_1.2.0-1bullseye_amd64.deb
- Install deb:
# apt install ./autotier_1.2.0-1bullseye_amd64.deb
- Edit configuration file.
- Mount filesystem.
- Optionally add user to
autotier
group to allow non-root users to run CLI commands:# usermod -aG autotier <user>
(takes effect on next login)
- Install rpm:
# dnf install https://github.com/45Drives/autotier/releases/download/v1.2.0/autotier-1.2.0-1.el8.x86_64.rpm
- Edit configuration file.
- Mount filesystem.
- Optionally add user to
autotier
group to allow non-root users to run CLI commands:# usermod -aG autotier <user>
(takes effect on next login)
- Install dependencies:
# apt install libfuse3-dev libboost-system-dev libboost-filesystem-dev libboost-serialization-dev librocksdb-dev libtbb-dev
$ git clone https://github.com/45drives/autotier
$ cd autotier
$ git checkout <version>
(v1.2.0 is the latest tag)$ make -j8
(ormake -j8 no-par-sort
to use c 11 instead of c 17)# make install
- Edit configuration file.
- Mount filesystem.
- Optionally add user to
autotier
group to allow non-root users to run CLI commands:# usermod -aG autotier <user>
(takes effect on next login)
Ubuntu: # apt remove autotier
or # apt purge autotier
EL8: # dnf remove autotier
From source: # make uninstall
from root of cloned repo
See man autotier
after installing for full configuration details.
Default configuration file location: /etc/autotier.conf
.
For global configuration of autotier
, options are placed below the [Global]
header. Example:
[Global] # global settings
Log Level = 1 # 0 = none, 1 = normal, 2 = debug
Tier Period = 100 # number of seconds between file move batches
Copy Buffer Size = 1 MiB # size of buffer for moving files between tiers
The global config section can be placed before, after, or between tier definitions.
The layout of a single tier's configuration entry is as follows:
[Tier 1] # tier name (can be anything)
Path = # full path to tier storage pool
Quota = # absolute or % usage to keep tier under
# Quota format: x (%|B|MB|MiB|KB|KiB|MB|MiB|...)
# Example: Quota = 5.3 TiB
As many tiers as desired can be defined in the configuration, however they must be in order of fastest to slowest. The tier's name can be whatever you want but it cannot be global
or Global
. Tier names are only used for config diagnostics and file pinning.
Below is a complete example of a configuration file:
# autotier config
[Global] # global settings
Log Level = 1 # 0 = none, 1 = normal, 2 = debug
Tier Period = 1000 # number of seconds between file move batches
Copy Buffer Size = 1 MiB # size of buffer for moving files between tiers
[Tier 1] # tier name (can be anything)
Path = /mnt/ssd_tier # full path to tier storage pool
Quota = 5 TiB # absolute or % usage to keep tier under
[Tier 2]
Path = /mnt/ssd_tier
Quota = 90 %
[Tier 3]
Path = /mnt/cold_storage
Quota = 100 %
See man autotier
after installing for full usage details.
Usage:
autotier [<flags>] <command> [<arg1 arg2 ...>]
Commands:
config - display current configuration values
help - display this message
list-pins - show all pinned files
list-popularity
- print list of all tier files sorted by frequency of use
oneshot - execute tiering only once
pin <"tier name"> <"path/to/file" "path/to/file" ...>
- pin file(s) to tier using tier name in config file
status - list info about defined tiers
unpin <"path/to/file" "path/to/file" ...>
- remove pin from file(s)
which-tier <"path/to/file" "path/to/file" ...>
- list which tier each argument is in
Flags:
-c, --config <path/to/config>
- override configuration file path (default /etc/autotier.conf)
-h, --help - display this message and cancel current command
-q, --quiet - set log level to 0 (no output)
-v, --verbose
- set log level to 2 (debug output)
-V, --version
- print version and exit
- if log level >= 1, logo will also print
- combine with -q to mute logo output
Examples:
Trigger tiering of files immediately:
autotier oneshot
Show status of configured tiers:
autotier status
Pin a file to a tier with <Tier Name>:
autotier pin "<Tier Name>" /path/to/file
Pin multiple files:
autotier pin "<Tier Name>" /path/to/file1 /path/to/dir/* /bash/expansion/**/*
find path/* -type f -print | xargs autotier pin "<Tier Name>"
Remove pins:
autotier unpin path/to/file
find path/* -type f -print | xargs autotier unpin
List pinned files:
autotier list-pins
Usage:
autotierfs [<flags>] <mountpoint> [-o <fuse,options,...>]
Flags:
-c, --config <path/to/config>
- override configuration file path (default /etc/autotier.conf)
-h, --help - display this message and cancel current command
-o, --fuse-options <comma,separated,list>
- mount options to pass to fuse (see man mount.fuse)
-q, --quiet - set log level to 0 (no output)
-v, --verbose
- set log level to 2 (debug output)
-V, --version
- print version and exit
- if log level >= 1, logo will also print
- combine with -q to mute logo output
- Ensure a mount point exists, e.g.
sudo mkdir /mnt/autotier
- Edit
/etc/fstab
as root and add the following line:/usr/bin/autotierfs /mnt/autotier fuse allow_other,default_permissions 0 0
- Apply with
sudo mount -a
- Ensure a mount point exists, e.g.
sudo mkdir /mnt/autotier
- Mount filesystem with
sudo autotierfs /mnt/autotier -o allow_other,default_permissions
To have cron
schedule file tiering, first disable automatic tiering by setting Tier Period = -1
in /etc/autotier.conf
.
Then in the cron entry, call autotier oneshot
.
If autotier is set up with tiers that already contain files, there is a chance that you could have more than one file with the same path relative to the tier root. In the event of a collision during file movement while tiering, the file that is already in the tier is left untouched and the incoming file has .autotier_conflict.<original tier name>
appended to the file name. autotier status
will list any file conflicts found.
Development documentation (generated by Doxygen) can be found here
┓
└─ ┃ ├─
└─ ┣ ├─
└─ ┃ └─
┛