For the past few months, I've been really into archivers and compressers. This is not something that happened out of the blue, as I have been in my current position for almost 3 years now and my personal storage have been constantly growing, and so did my collages'. As a consequence, the STaaS that hosts these personal folders is filling up slowly but steadly. To alleviate the problem, I started researching on how do tools like 7z, zip, tar work on the background. This really draw my attention and encouraged me to develop my own personal tool to compress all my stuff. This way, I would know what happens in each step by heart, and could implement improvements if I wanted to.

Archiver vs. Compressor

In Linux, a compressor and an archiver serve related byt distinct purposes that are often confused because many tools blend the two, as the one I impleemented myself and is discussed in this post. An archiver's job is to bundle different files and dirs, along with their metadata, into a single file. This task does not reduce its size. tar is the classing example for this. A compressor, on the other hand, focuses purely on reducing the size of a data buffer using a very complex algorithm to eliminate redundancy. This typically works on a single file or data stream rather than a collection of files. gzip is a good example for this, designed to compress one file at a time, without any awareness of dir structures. Because of this division of labor, the two are frequently chaided together: archive → compress (tar.gz extension); and its inverse, decompress → unarchive. Some other tools combine both functions natively, like zip or 7z. With this difference in mind, what I proposed myself to develop is a tool that combines both, being the archiver procedure designed purely by me to be compressed using zstd afterwards.

Development journey

At first, the main objective was to have a working alternative to the tools mentioned above. This objective was acomplished rather quickly, as I implemented so little optimizations and employed so much bytes for the headers of each file inside the final archive. However, the more I programmed, the more I enjoyed the project, so I did not stop there. At the end, two months later, I ended up with a tool that operates its own archive format, which can be explored and modified without unarchiving, native zstd compression and support for multithreading and sparse files.

Why a new archive format

At the final weeks of the development the feature I cared about most was sparse file support. VM disk images, database files and thin-provisioned volumes are full of zero-filled holes that don't need to be read, stored or transferred. spare uses SEEK_HOLE/SEEK_DATA to detect those holes during packing and fallocate(PUNCH_HOLE) to recreate them on unpack, so a 4 GB image with 20% real data stays a ~820 MB archive instead of a 4 GB one, with a single -S flag, not a mode I have to remember exists.

Every block also carries its own xxh64 checksum over the header, filename, hole map and data, so corruption in any single file is caught without hashing the whole archive.

Built for pipelines

The other thing I wanted was for the archiver to behave like a proper Unix tool. Passing - as the archive path lets spare read from stdin or write to stdout, so it can sit in the middle of a pipeline instead of always producing a file on disk:

spare p  - file1 file2 | ssh user@host "spare u -"        # copy files over SSH
spare pz - file1 file2 | aws s3 cp - s3://bucket/b.szt   # stream to object storage
gpg -d secrets.szt   | spare u - -z                      # decrypt and unpack

Compression is zstd, applied over the whole archive rather than per file, and multithreading (-jN) speeds up both packing and compression on multi-core machines.

Does it hold up against tar?

Once I had something that worked, I didn't just want proof it wasn't broken, I wanted proof it could push further than tar, so I put it side by side and kept squeezing until the numbers actually meant something. On ~75,000 small files (a Linux kernel tree, ~1.5 GB), single-threaded pack time came out roughly on par with tar, and once I let spare -j4 loose across cores it packed and compressed in under half the time. The sparse VM image mattered most to me, though, since that ended up being the feature I cared about above everything else by the end: spare -S matched tar --sparse on both archive size and speed, confirming the hole-detection logic I'd spent so much time on was pulling its weight. Unpacking is where I still give a little ground, spare runs a bit slower than tar in most cases, because verifying every block's checksum and rebuilding the hole map isn't free. That's a trade-off I'm happy to keep for now; I'd rather pay a few extra seconds on unpack than discover years from now that a backup silently rotted, and it's the next thing on my list to optimize away.

Where this leaves me

spare was never meant to dethrone tar for the rest of the world, but on my own machines it already has: tar, zip and 7z don't get touched anymore, everything I archive goes through spare now, sparse-aware packing, pipeline-friendly streaming, and per-file integrity checks bundled into one tool instead of three. There's a less comfortable truth hiding in that, though: as far as I know I'm the only person on the planet running it, so its survival depends entirely on whether I keep reaching for it myself. The day I stop using it is the day it's dead, which is exactly why I don't plan to.