A git-annex-centric benchmark of file systems

Disclaimer!

Presented here benchmarking was done in 2015!!!
Lots of development has happened in filesystems space/kernel since then, so results might no longer hold, and hopefully the differences between various file systems has reduced drastically. We would encourage readers to perform benchmarking on their hardware and target perspective setups before finalizing the deployment having made an informed decision. The scripts we used and shared for our benchmarking could serve as a starting point.

Problem statement

For the DataLad project we will establish a large number of git-annex repositories. Majority of those git-annex repositories will not contain any data, but rather a lot of empty files and/or broken symlinks. Some data sharing projects which we are going to cover contain thousands and even millions of files (e.g. HCP500 release S3 bucket contained over 5 million files). Originally, we had a very nice experience with ZoL (ZFS on Linux) on computational clusters we administer/use. Its volume management, data integrity checking, and various features (snapshotting, etc) are very appealing and served us well before. So ZFS was a logical choice to deploy for our DataLad development/storage server(s). But unfortunately, during development we ran into complete system stalls (which thanks to ZoL developers were promptly resolved), and overall very slow performance (even with SSD L2ARC caches and relatively large RAM on those boxes). So it became important to investigate if we were indeed hitting the limits of contemporary conventional file-system designs while dealing with large number of files, or just getting bound by ZoL.

System setup

All the benchmarks were ran on a recently purchased server

• CPU: Intel(R) Xeon(R) CPU E5-1607 v2 @ 3.00GHz (quad core)
• RAM: 64GB (registered)
• OS: Debian Jessie (8.1) amd64 build, with NeuroDebian and ZoL repositories enabled
• Kernel: 3.16.7-ckt9-3~deb8u1 (Debian: 3.16.0-4-amd64)
• ZoL/ZFS module: 0.6.4-15-7fec46
• Btrfs tools: originally 3.17 (as in Jessie) but then tested a single setup initiated with 4.0 tools (backport coming from NeuroDebian repository)
• Drives: 6 Seagate Constellation ES.3 (ST4000NM0033-9ZM170, fw SN04) on an LSI SAS3008 HBA

Scripts

test_fs.py script was created to initiate a range of prototypical file-system setups, with software raid/MD (redundancy), LVM (volume management) and various file systems (EXT4, XFS, ZFS, ReiserFS, Btrfs) in various configurations. We tested typical operations in the life-time of a git/annex repository — creation (only for larger repositories test), du, chmod, git clone, git annex get, git annex (in)direct, tar etc — full list is available in results tables below.

Results

Below we present results of the analysis of collected timings, which in its entirety is available as IPython notebook with all the necessary data in the repository if you would like to have an alternative analysis or visualization done since presentation here is quite messy.

Overall verdict is: we are going for Software Raid (RAID6) + LVM (scalability) + Btrfs (performance) as our setup with robust volume management (snapper tool providing snapshots management) and good performance (with tolerable impact from enabled compression). Btrfs exhibited robust performance across a wide range of meta-information utilization, while ReiserFS — the next contender — failed to scale (although it may have behaved better with some tune up/options — we didn't explore). ZoL/ZFS quickly became way too slow to be even properly tested and thus was not considered for our case. Additional features of Btrfs, such as COW (copy-on-write) can already be utilized by git-annex so providing Btrfs additional bonus points in our decision. The choice of compression (lzo vs zlib) for Btrfs did not have a clear winner, and impact from compression was not completely detrimental (although went up to ~30% on git clone operation overall impact due to compression).

Disclaimer YMMV. We have tested file-systems on quite an obscure setup, which is very heavy on meta-information without much of actual data being stored — lots of tiny files with as many symlinks and directories (under .git/annex/objects). Though, majority of the benchmarked commands (e.g., chmod, du) were meta-information access heavy, so even if large data files were stored we would expect similar performance in those cases.

Small repositories test

Initial test consisted of creating relatively small repositories each containing 100 files in 20 directories added to git-annex. Multiple rounds (without removing previously created test directories/repositories) were ran, thus slowly growing impact on the file-system. ZFS file-system setups ran only 10 rounds of such tests, while 100 times for other file-systems.

Following results present timings on the first 10 rounds across file systems on "cold" runs of the commands. Ratings/timing table presents overall rating (0 — is the best, estimated as a median among ratings across all benchmark commands) with mean timing (across rounds) per each command.

  ports_to_dataframe(load_reports('simpleannex_ndirs=20_nfiles=100-', equal_length=True))
  summary_plots(df, 'cold', gross_violins=True)
  Got 19 reports
  # of reports per each FS: [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
  Got 2850 entries total

Gross-rating/timing for each file-system/per each command

Gross time per each file-system

Execution time per each command across file-systems and across runs

and "warm" (re-running the same command thus utilizing possible caching of meta-information and data) runs:

  summary_plots(df, 'warm')
  
  

Gross-rating/timing for each file-system/per each command

Execution time per each command across file-systems and across runs

In both of the above reports (cold and warm) on small repositories we can see that ZFS performs quite poorly (and also becomes slower with higher FS utilization on many commands) but still on the same order as XFS and EXT4, while Btrfs and ReiserFS perform much smoother with ReiserFS being the most efficient.

Following plots shows all 100 rounds:

  df = reports_to_dataframe(load_reports('simpleannex_ndirs=20_nfiles=100-', equal_length=False))
  summary_plots(df, 'cold')
  Got 19 reports
  # of reports per each FS: [10, 100, 100, 100, 100, 100, 100, 100, 100, 10, 10, 100, 100, 100, 100, 100, 100, 100, 100]
  Got 24450 entries total

Gross-rating/timing for each file-system/per each command

Execution time per each command across file-systems and across runs

Middle-sized repositories test

For this test we have decided to get closer to typical middle size repositories with 1000 files in each of the 100 directories (thus 100,000 original files per repository) and run 10 rounds. Unfortunately ZFS runs were taking way to long (sample single round took more than 24h) so they were interrupted.

Surprisingly ReiserFS performance detremented quite significantly, bringing Btrfs into the winners. Effect from the choice of compression (lzo vs zlib) wasn't clearly pronounced so we decided to go with lzo.

  df = reports_to_dataframe(load_reports('simpleannex-big_ndirs=100_nfiles=1000-'))
  summary_plots(df, 'cold')
  Got 7 reports
  # of reports per each FS: [10, 10, 10, 10, 10, 10, 10]
  Got 1540 entries total

Gross-rating/timing for each file-system/per each command

Execution time per each command across file-systems and across runs

  summary_plots(df, 'warm')

Gross-rating/timing for each file-system/per each command

Execution time per each command across file-systems and across runs