<a name="benchmarks"></a> Benchmarks ------------------------- The reference system uses an Intel i7-9700K cpu, and runs Ubuntu x64 20.04. The [open source benchmark program] is compiled with `clang` v10.0 using `-O3` flag. | Hash Name | Width | Bandwidth (GB/s) | Small Data Velocity | Quality | Comment | | --------- | ----- | ---------------- | ----- | --- | --- | | __XXH3__ (SSE2) | 64 | 31.5 GB/s | 133.1 | 10 | | __XXH128__ (SSE2) | 128 | 29.6 GB/s | 118.1 | 10 | | _memcpy_ | N/A | 28.0 GB/s | N/A | N/A | _for reference_ | City64 | 64 | 22.0 GB/s | 76.6 | 10 | | T1ha2 | 64 | 22.0 GB/s | 99.0 | 9 | Slightly worse [collisions] | City128 | 128 | 21.7 GB/s | 57.7 | 10 | | __XXH64__ | 64 | 19.4 GB/s | 71.0 | 10 | | SpookyHash | 64 | 19.3 GB/s | 53.2 | 10 | | Mum | 64 | 18.0 GB/s | 67.0 | 9 | Slightly worse [collisions] | __XXH32__ | 32 | 9.7 GB/s | 71.9 | 10 | | City32 | 32 | 9.1 GB/s | 66.0 | 10 | | Murmur3 | 32 | 3.9 GB/s | 56.1 | 10 | | SipHash | 64 | 3.0 GB/s | 43.2 | 10 | | FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties | Blake2 | 256 | 1.1 GB/s | 5.1 | 10 | Cryptographic | SHA1 | 160 | 0.8 GB/s | 5.6 | 10 | Cryptographic but broken | MD5 | 128 | 0.6 GB/s | 7.8 | 10 | Cryptographic but broken [open source benchmark program]: https://github.com/Cyan4973/xxHash/tree/release/tests/bench [collisions]: https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study note 1: Small data velocity is a _rough_ evaluation of algorithm's efficiency on small data. For more detailed analysis, please refer to [the wiki](https://github.com/Cyan4973/xxHash/wiki/Performance-comparison#benchmarks-concentrating-on-small-data-). note 2: On 32-bit applications, the ranking between hashes differs drastically, strongly favoring 32-bit hashes, See [this section of the wiki](https://github.com/Cyan4973/xxHash/wiki/Performance-comparison#bandwidth-on-x86-with-various-data-size-of-len-2n-32-bit-friendliness) for illustration. note 3: some algorithms feature _faster than RAM_ speed. In which case, they can only reach their full speed when input data is already in CPU cache (L3 or better). Otherwise, they max out at RAM speed limit. <br/> <a name="other-languages"></a> ## Multiple languages The following versions produce xxHash-compatible results in different languages. |Language |Author |URL |-- |-- |-- |__Python__ |Yue Du |https://pypi.python.org/pypi/xxhash/ |__Java__ |Adrien Grand |https://github.com/lz4/lz4-java/tree/master/src/java/net/jpountz/xxhash |__Java__ (XXH3, XXH128) |James Z.M. Gao |https://github.com/OpenHFT/Zero-Allocation-Hashing/blob/master/src/main/java/net/openhft/hashing/XXH3.java |__C++__ (simple) |Stefan Brumme |https://create.stephan-brumme.com/xxhash/ |__C++ 17__ |Red Gavin |https://github.com/RedSpah/xxhash_cpp |__C++__ constexpr (XXH64) |Daniel Kirchner |https://github.com/ekpyron/xxhashct |__C++__ constexpr (XXH32) |Takayuki Matsuoka |https://github.com/Cyan4973/xxHash/issues/496 |__C++__ constexpr (XXH3) |chys87 |https://github.com/chys87/constexpr-xxh3 |__C#__ (port) |Melnik Alexander |https://github.com/uranium62/xxHash |__C#__ (7.0) |Otaku |https://github.com/differentrain/YYProject.XXHash |__C#__ (.net std 2.0) |Sedat Kapanoğlu |https://github.com/ssg/HashDepot#xxhash |__JavaScript__ (WebAssembly)|Michael Jungo |https://www.npmjs.com/package/xxhash-wasm |__JavaScript__ (port) |Pierre Curto |https://npmjs.org/package/xxhashjs |__JavaScript__ (nodeJS) |Brian White |https://npmjs.org/package/xxhash |__JavaScript__ (nodeJS, xxh3)|Nhan Khong |https://github.com/ktrongnhan/xxhash-addon |__JSX__ (static JavaScript)|Yoshiki Shibukawa |https://www.npmjs.org/package/xxhash.jsx |__PHP8__ |Anatol Belski |https://php.watch/versions/8.1/xxHash |__PHP7__ |Craig R Megasaxon |https://github.com/Megasaxon/php-xxhash |__PHP__ |Nir Heimann |https://github.com/nheimann1/php-xxhash |__PHP__ (port) |Scott Dutton |https://github.com/exussum12/xxhash |__R__ |Dirk Eddelbuettel |https://github.com/eddelbuettel/digest |__R__ (XXH3) |mikefc |https://github.com/coolbutuseless/xxhashlite |__Perl__ |Sanko Robinson |https://metacpan.org/module/Digest::xxHash |__Perl__ (streaming) |Bela Bodecs |https://github.com/DoubleBB/digest-xxhash64 |__Go__ (XXH64) |Ahmed Waheed |https://github.com/OneOfOne/xxhash |__Go + ASM__ |Caleb Spare |https://github.com/cespare/xxhash |__Go__ (XXH3) |Jeff Wendling |https://github.com/zeebo/xxh3 |__Swift__ |Daisuke T |https://github.com/daisuke-t-jp/xxHash-Swift |__Ruby__ (port) |Justin W Smith |http://rubygems.org/gems/ruby-xxHash |__Ruby__ (wrapper) |Vasiliy Ermolovich |https://rubygems.org/gems/xxhash |__Ruby__ (digest::class) |konsolebox |https://rubygems.org/gems/digest-xxhash |__Pascal__ |Vojtěch Čihák |http://sourceforge.net/projects/xxhashfpc |__Rust__ |Jake Goulding |https://libraries.io/cargo/twox-hash |__Rust__ (const xxh3) |Arthur Martirosyan |https://crates.io/crates/xxhash-rust |__Kotlin__ |Matthew Dolan |https://github.com/appmattus/crypto/tree/main/cryptohash |__Julia__ |Hanan Rosemarin |https://github.com/hros/XXhash.jl |__Scala__ |Desmond Yeung |https://github.com/desmondyeung/scala-hashing |__D__ |Masahiro Nakagawa |https://github.com/repeatedly/xxhash-d |__Lua__ (binding) |Masatoshi Teruya |https://github.com/mah0x211/lua-xxhash |__Lua__ (jit, XXH32) |szensk |https://github.com/szensk/luaxxhash |__Lua__ (jit, XXH64) |Soojin Nam |https://github.com/sjnam/luajit-xxHash |__OCaml__ |Pieter Goetschalckx|http://opam.ocaml.org/packages/xxhash/ |__Haskell__ |Christian Marie |http://hackage.haskell.org/package/xxhash |__Dart__ (XXH3) |SamJakob |https://pub.dev/packages/xxh3 |__Erlang__ |Pierre Matri |https://github.com/pierresforge/erlang-xxhash |__Erlang__ (XXH3) |Ali Farhadi |https://github.com/farhadi/xxh3 |__PicoLisp__ |mpech |https://git.envs.net/mpech/xxhash-picolisp |__Elixir__ (nif) |Ali Farhadi |https://github.com/farhadi/xxh3 |__Elixir__ (port) |Mykola Konyk |https://github.com/ttvd/elixir-xxhash |__Tcl__ (XXH32, port) |D. Bohdan |https://wiki.tcl-lang.org/48790 |__Bash__ (port) |Devin Hussey |https://github.com/easyaspi314/xxbash |__sh__ (port) |Jan Chren |https://gitlab.com/rindeal/xxHashish |__Crystal__ |Lucjan Suski |https://github.com/methyl/xxhash |__Zig__ |clownpriest |https://github.com/clownpriest/xxhash <br/> <a name="references"></a> ## xxHash is used by #### Databases <div id="xxhashusedby_databases" class="container"> <ul class="list-inline"> <li><a href="http://prestodb.io/" ><img src="images/logo50/prestoDB.png" /><span> PrestoDB</span></a></li> <li><a href="https://rocksdb.org/" ><img src="images/logo50/rocksdb.png" /><span> RocksDB</span></a></li> <li><a href="https://www.mysql.com/" ><img src="images/logo50/mysql.png" /><span> MySQL</span></a></li> <li><a href="https://clickhouse.com/" ><img src="images/logo50/clickhouse.png" /><span> ClickHouse</span></a></li> <li><a href="https://www.arangodb.org/" ><img src="images/logo50/arangoDB.png" /><span> ArangoDB</span></a></li> <li><a href="https://pgroonga.github.io/" ><img src="images/logo50/pgroonga.png" /><span> PGroonga</span></a></li> <li><a href="http://spark.apache.org/" ><img src="images/logo50/spark.png" /><span> Spark</span></a></li> <li><a href="https://mariadb.org/" ><img src="images/logo50/mariadb.png" /><span> MariaDB</span></a></li> <li><a href="https://groonga.org/docs/news.html#release-10-0-8"><img src="images/logo50/groonga.png"/><span> Groonga</span></a></li> </ul> </div> #### Games <div id="xxhashusedby_games" class="container"> <ul class="list-inline"> <li><a href="https://youtu.be/DkspHgt27Io?t=717"><img src="images/logo50/CoD_BO_CW.png"/><span> CoD Black Ops</span></a></li> <li><a href="https://mafiagame.fandom.com/wiki/Mafia:_Definitive_Edition_Credits"><img src="images/logo50/mafiade.png"/><span> Mafia</span></a></li> <li><a href="https://www.minecraft.net" ><img src="images/logo50/minecraft.png" /><span> Minecraft</span></a></li> <li><a href="https://ppsspp.org/" ><img src="images/logo50/ppsspp.png" /><span> PPSSPP</span></a></li> <li><a href="https://dolphin-emu.org/" ><img src="images/logo50/dolphin.png" /><span> Dolphin</span></a></li> <li><a href="https://cxbx-reloaded.co.uk"><img src="images/logo50/cxbx.png" /><span> Cxbx-reloaded</span></a></li> <li><a href="http://www.cocos2d.org/" ><img src="images/logo50/cocos2D.png" /><span> Cocos2D</span></a></li> <li><a href="https://www.lwjgl.org/" ><img src="images/logo50/lwjgl.png" /><span> LWJGL</span></a></li> <li><a href="http://fc-solve.shlomifish.org/"><span> Freecell Solver</span></a></li> </ul> </div> #### Filters <div id="xxhashusedby_filters" class="container"> <ul class="list-inline"> <li><a href="https://www.haproxy.org/" ><img src="images/logo50/haproxy.png" /><span> HAProxy</span></a></li> <li><a href="https://rspamd.com/" ><img src="images/logo50/rspamd.png" /><span> Rspamd</span></a></li> <li><a href="https://www.pfsense.org/" ><img src="images/logo50/pfsense.png" /><span> pfSense</span></a></li> <li><a href="http://freecode.com/projects/fio/"><img src="images/logo50/placeholder.png"/><span> fio</span></a></li> <li><a href="https://npmjs.org/package/bloomxx/"><img src="images/logo50/placeholder.png"/><span> bloomxx</span></a></li> <li><a href="http://devisedbydavid.com/open_source/bloom_filter"><img src="images/logo50/placeholder.png"/><span> C & Python Bloom Filter</span></a></li> <li><a href="https://github.com/mozilla-services/lua_bloom_filter"><img src="images/logo50/mozilla.png"/><span> LUA Bloom Filter</span></a></li> </ul> </div> #### File Transfer <div class="container"> <ul class="list-inline"> <li><a href="https://partnerhelp.netflixstudios.com/hc/en-us/articles/360000581207-Production-Assets-Data-Management"><img src="images/logo50/netflix.png"/><span> Netflix</span></a></li> <li><a href="https://rsync.samba.org/" ><img src="images/logo50/rsync.png" /><span> rsync</span></a></li> <li><a href="http://www.lz4.org/" ><img src="images/logo50/lz4.png" /><span> LZ4</span></a></li> <li><a href="https://pomfort.com/silverstack/" ><img src="images/logo50/silverstack.png" /><span> Silverstack</span></a></li> <li><a href="http://www.lespace.co.jp/file_bl/rapidcopy/rapidcopy.html"><img src="images/logo50/rapidcopy.png"/><span> Rapidcopy</span></a></li> <li><a href="https://www.hedgeformac.com/" ><img src="images/logo50/syncFactory.png" /><span> Hedge</span></a></li> <li><a href="https://ipmsg.org/tools/fastcopy.html.en"><img src="images/logo50/fastcopy.png" /><span> fastcopy</span></a></li> <li><a href="https://codesector.com/teracopy" ><img src="images/logo50/teracopy.png" /><span> TeraCopy</span></a></li> <li><a href="https://www.dragonflybsd.org/hammer/" ><img src="images/logo50/hammer.png" /><span> Hammer 2</span></a></li> <li><a href="https://compressme.net/" ><img src="images/logo50/chk48.png" /><span> CHK</span></a></li> </ul> </div> #### Other <div id="xxhashusedby_other" class="container"> <ul class="list-inline"> <li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=59e1a2f4bf83744e748636415fde7d1e9f557e05"><img src="images/logo50/linux.png"/><span> Linux</span></a></li> <li><a href="https://docs.microsoft.com/en-us/azure/kusto/query/hashfunction"><img src="images/logo50/azure50.png"/><span> Microsoft Azure</span></a></li> <li><a href="https://devblogs.microsoft.com/cppblog/linker-throughput-improvement-in-visual-studio-2019/"><img src="images/logo50/microsoft.png"/><span> Visual Studio</span></a></li> <li><a href="https://docs.nvidia.com/nsight-compute/CopyrightAndLicenses/index.html"><img src="images/logo50/nvidia.png"/><span> NSight compute</span></a></li> <li><a href="https://forum.huawei.com/enterprise/en/how-to-understand-inline-deduplication-and-compression/thread/604642-891"><img src="images/logo50/huawei.png"/><span> Dorado</span></a></li> <li><a href="https://wiki.freebsd.org/pkgng" ><img src="images/logo50/freebsd.png" /><span> PKG</span></a></li> <li><a href="https://www.qemu.org/" ><img src="images/logo50/qemu.png" /><span> Qemu</span></a></li> <li><a href="https://btrfs.wiki.kernel.org/index.php/Main_Page"><img src="images/logo50/btrfs.png"/><span> btrfs</span></a></li> <li><a href="https://www.xpra.org/" ><img src="images/logo50/xpra.png" /><span> Xpra</span></a></li> <li><a href="http://www.teamviewer.com/" ><img src="images/logo50/teamviewer.png" /><span> TeamViewer</span></a></li> <li><a href="http://factorcode.org/" ><img src="images/logo50/factor.png" /><span> Factor</span></a></li> <li><a href="http://nvlabs.github.io/nvbio/index.html"><img src="images/logo50/nvbio.png" /><span> nVBio</span></a></li> <li><a href="http://dvisvgm.de/"> <img src="images/logo50/placeholder.png" /><span> dvisvgm</span></a></li> <li><a href="http://www.fastbuild.org/"> <img src="images/logo50/fastbuild.png" /><span> FastBuild</span></a></li> <li><a href="http://keypirinha.com/"> <img src="images/logo50/keypirinha.png" /><span> Keypirinha</span></a></li> <li><a href="http://quickhash-gui.org/"> <img src="images/logo50/quickhash.png" /><span> QuickHash</span></a></li> </ul> </div> ### Special Thanks to - [Takayuki Matsuoka](https://github.com/t-mat/) for creating `xxhsum -c` and the canvas of this webpage. - [Mathias Westerdahl](https://github.com/JCash) for introducing the first version of XXH64. - [Devin Hussey](https://github.com/easyaspi314) for the excellent low-level optimizations on XXH3 and XXH128. ---