HTTPDirFS - HTTP Directory Filesystem with a permanent cache, and Airsonic / Subsonic server support
Have you ever wanted to mount those HTTP directory listings as if it was a partition? Look no further, this is your solution. HTTPDirFS stands for Hyper Text Transfer Protocol Directory Filesystem.
The performance of the program is excellent. HTTP connections are reused through curl-multi interface. The FUSE component runs in the multithreaded mode.
There is a permanent cache system which can cache all the file segments you have
downloaded, so you don't need to these segments again if you access them later.
This feature is triggered by the --cache
flag. This is similar to the
--vfs-cache-mode full
feature of
rclone mount
There is support for Airsonic / Subsonic server. This allows you to mount a remote music collection locally.
If you only want to access a single file, there is also a simplified Single File Mode. This can be especially useful if the web server does not present a HTTP directory listing.
Please note if you install HTTDirFS from a repository, it can be outdated.
HTTPDirFS is available as a package in Debian 12 "Bookworm", If you are on
Debian Bookworm, you can simply run the following
command as root
:
apt install httpdirfs
For more information on the status of HTTDirFS in Debian, please refer to Debian package tracker
HTTPDirFS is available in the Arch User Repository.
HTTPDirFS is available in the FreeBSD Ports Collection.
For important development related documentation, please refer src/README.md.
Under Debian 12 "Bookworm" and newer versions, you need the following dependencies:
libgumbo-dev libfuse3-dev libssl-dev libcurl4-openssl-dev uuid-dev help2man
libexpat1-dev pkg-config meson
You can then compile the program similar to how you compile a typical program that uses the Meson build system:
meson setup builddir
cd builddir
meson compile
To install the program, do the following:
sudo meson install
To uninstall the program, do the following:
sudo ninja uninstall
To clean the build directory, run:
ninja clean
For more information, please refer to this tutorial.
I don't have the resources to test out compilation for Linux distributions other than Debian. I also do not have the resources to test out compilation for FreeBSD or macOS. Thereforce I have removed the instruction on how to compile for these operating systems in the README for now. Please feel free to send me a pull request to add them back in.
./httpdirfs -f --cache $URL $MOUNT_POINT
An example URL would be
Debian CD Image Server. The -f
flag
keeps the program in the foreground, which is useful for monitoring which URL
the filesystem is visiting.
For more usage related help, run
./httpdirfs --help
or
man httpdirfs
Please note that the man page only works if you have installed HTTPDirFS properly.
The Airsonic / Subsonic server support is dedicated the my Debian package maintainer Jerome Charaoui.You can mount the music collection on your Airsonic / Subsonic server (*sonic), and browse them using your favourite file browser.
You simply have to supply both --sonic-username
and --sonic-password
to
trigger the *sonic server mode. For example:
./httpdirfs -f --cache --sonic-username $USERNAME --sonic-password $PASSWORD $URL $MOUNT_POINT
You definitely want to enable the cache for this one, otherwise it is painfully slow.
There are two ways of mounting your *sonic server
- the index mode
- and the ID3 mode.
In the index mode, the filesystem is presented based on the listing on the
Index
link in your *sonic's home page.
In ID3 mode, the filesystem is presented using the following hierarchy: 0. Root
- Alphabetical indices of the artists' names
- The arists' names
- All of the albums by a single artist
- All the songs in an album.
By default, *sonic server is mounted in the index mode. If you want to mount in
ID3 mode, please use the --sonic-id3
flag.
Please note that the cache feature is unaffected by how you mount your *sonic server. If you mounted your server in index mode, the cache is still valid in ID3 mode, and vice versa.
HTTPDirFS is also known to work with the following applications, which implement some or all of Subsonic API:
- Funkwhale (requires
--sonic-id3
and--no-range-check
, more information in issue #45) - LMS (requires
--sonic-insecure
and--no-range-check
, more information in issue #46. To mount the demo instance, you might also need--insecure-tls
) - Navidrome, more information in issue #51.
If you just want to access a single file, you can specify
--single-file-mode
. This effectively creates a virtual directory that
contains one single file. This operating mode is similar to the unmaintained
httpfs.
e.g.
./httpdirfs -f --cache --single-file-mode https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.0.0-amd64-netinst.iso mnt
This can be useful if the web server does not present a HTTP directory listing. This feature was implemented due to Github issue #86
You can cache the files you have accessed permanently on your hard drive by
using the --cache
flag. The file it caches persist across sessions.
By default, the cache files are stored under ${XDG_CACHE_HOME}/httpdirfs
,
which by default is ${HOME}/.cache/httpdirfs
. Each HTTP directory gets its
own cache folder, they are named using the escaped URL of the HTTP directory.
Once a segment of the file has been downloaded once, it won't be downloaded again.
Please note that due to the way the permanent cache system is implemented. The maximum download speed is around 15MiB/s, as measured using my localhost as the web server. However after you have accessed a file once, accessing it again will be the same speed as accessing your hard drive.
If you have any patches to make the initial download go faster, please submit a pull request.
The permanent cache system relies on sparse allocation. Please make sure your filesystem supports it. Otherwise your hard drive / SSD will get heavy I/O from cache file creation. For a list of filesystem that supports sparse allocation, please refer to Wikipedia.
This program has basic support for using a configuration file. By default, the
configuration file which the program reads is
${XDG_CONFIG_HOME}/httpdirfs/config
, which by
default is at ${HOME}/.config/httpdirfs/config
. You will have to create the
sub-directory and the configuration file yourself. In the configuration file,
please supply one option per line. For example:
--username test
--password test
-f
Alternatively, you can specify your own configuration file by using the
--config
option.
You can control how much log HTTPDirFS outputs by setting the
HTTPDIRFS_LOG_LEVEL
environmental variable. For details of the different
types of log that are supported, please refer to
log.h and
log.c.
For the normal HTTP directories, this program downloads the HTML web pages/files using libcurl, then parses the listing pages using Gumbo, and presents them using libfuse.
For *sonic servers, rather than using the Gumbo parser, this program parse *sonic servers' XML responses using expat.
The cache system stores the metadata and the downloaded file into two
separate directories. It uses uint8_t
arrays to record which segments of the
file had been downloaded.
Note that HTTPDirFS requires the server to support HTTP Range Request, some
servers support this features, but does not present "Accept-Ranges: bytes
in
the header responses. HTTPDirFS by default checks for this header field. You can
disable this check by using the --no-range-check
flag.
- Linux Format - Issue 264, July 2020
- First of all, I would like to thank Jerome Charaoui for being the Debian Maintainer for this piece of software. Thank you so much for packaging it!
- I would like to thank Cosmin Gorgovan for the technical and moral support. Your wisdom is much appreciated!
- I would like to thank Edenist for providing FreeBSD compatibility patches.
- I would like to thank hiliev for providing macOS compatibility patches.
- I would like to thank Jonathan Kamens for providing a whole bunch of code improvements and the improved build system.
- I would like to thank -Archivist for not providing FTP or WebDAV access to his server. This piece of software was written in direct response to his appalling behaviour.