Benchmarking the best way to store long, Object value pairs in a map in a threadsafe way.

In order to run this set of benchmarks:

1. Clone the repository
2. Run gradlew jmh
3. Look at the results in the build/reports/jmh/results.csv file

Note: This test is expected to run for ~8 hours.

• ConcurrentHashMap<Long, Object> (Built-in JDK collections)
• HashMap<Long, Object> (Built-in JDK collections)
• TLongObjectHashMap<Object> (Trove 3.0.3)
• SynchronizedTLongObjectHashMap<Object> (Trove 3.03)
• LongObjectHashMap<Object> (Eclipse Collections 8.0.0)
• SynchronizedLongObjectHashMap<Object> (Eclipse Collections 8.0.0)
• HashLongObjMap<Object> (Koloboke 1.0.0)
• HashLongObjectMap<Object> (HPPCRT 0.7.4)
• Long2ObjectArrayMap<Object> (Fastutil 7.1.0)
• Long2ObjectOpenHashMap<Object> (Fastutil 7.1.0)
• Long2ObjectHashMap<Object> (Agrona 0.9.3)

Each map undergoes 4 seperate operations tested. Copy, Get, Remove, and Update. And for each operation, the throughput, average time, sampled time and single-shot time is measured.

Note: For non-concurrent implementations, a manual synchronized block is used after the declaration of the map using the map as the lock.

For each test type, the map being tested is populated with 1 thousand key value pairs. Each pair's key is incremented by 1, starting at 0 and each value is a random object which can either be: Integer, Long, Boolean, Float, Double, byte[] (of variable size between 0-1000 random bytes), String (of variable length from 0-250 random chars), and ArbitraryPOJO which holds 2 objects in seperate fields which are randomly generated objects ranging from any of the aforementioned types.

The map tested is copied using a provided method when available, otherwise it is manually constructed.

The map tested attempts to do (map.size/2) successful data retrievals and (map.size/2) unsuccessful data retrievals.

The map tested is manually iterated through and has one key,value pair cleared per iteration.

The map tesed is manually iterated through and each key has a new value assigned to it.

As with any statistics, nothing should be taken definitively. So here are potential issues with these particular benchmarks:

• Agrona's "Get" test failed, this is likely due to it not failing silently with an incorrect key
• This test does not test concurrent access ability. It is designed with the intention of the maps being used in a multithreaded environment with few actually simultaneous method invocations.
• There is not enough data to do a true statistical analysis so while general trends can be seen, the concrete numbers are not nescessarily accurate or precise.
• This was done in Java 8, using JMH version 1.12 (which is slightly outdated). We were forced to use 1.12 because of incompatibility issues with the gradle plugin at versions greater than 1.12.

What this suggests: Fastutil's Long2ObjectArrayMap is the worst performing with the fewest amount of ops/ms and ConcurrentHashMap doesn't do much better. However, Koloboke's HashLongObjMap and Fastutil's Long2ObjectOpenHashMap are significantly more performant than other maps. Additionally, Eclipse's two map implementations perform about equally and are close runner-ups to the most performant in this test.

What this suggests: Most map implementations have similar copy times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap are considerably slower than the rest.

What this suggests: Most map implementations have similar copy times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap are considerably slower than the rest.

What this suggests: Most map implementations have similar copy times, however ConcurrentHashMap is considerably slower. This means that most maps perform similarly on "cold" runs.

What this suggests: Fastutil's Long2ObjectArrayMap is the worst performing with the fewest amount of ops/ms and ConcurrentHashMap doesn't do much better. However, HashMap, Koloboke's HashLongObjMap and Fastutil's Long2ObjectOpenHashMap are significantly more performant than other maps for retrieving data. Additionally, Trove's two map implementations perform about equally and are close runner-ups to the most performant in this test.

What this suggests: Most map implementations have similar data retrieval times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap are considerably slower than the rest.

What this suggests: Most map implementations have similar data retrieval times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap are considerably slower than the rest.

What this suggests: Most map implementations have similar data retrieval times, however ConcurrentHashMap and (and especially) Fastutil's Long2ObjectArrayMap are considerably slower. This means that these maps perform poorly on "cold" runs.

What this suggests: Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMap are the worst performing maps with the fewest amount of ops/ms and ConcurrentHashMap doesn't do much better. However, HashMap, Koloboke's HashLongObjMap and Eclipse's LongObjectHashMap are significantly more performant than other maps for removing data.

What this suggests: Most map implementations have similar data removal times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps are considerably slower than the rest.

What this suggests: Most map implementations have similar data removal times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps are considerably slower than the rest.

What this suggests: Most map implementations have similar data removal times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps are considerably slower. This means that these maps perform poorly on "cold" runs.

What this suggests: Fastutil's Long2ObjectArrayMap is the worst performing with the fewest amount of ops/ms and ConcurrentHashMap doesn't do much better. However, Koloboke's HashLongObjMap, Agrona's Long2ObjectHashMap and Fastutil's Long2ObjectOpenHashMap are significantly more performant than other maps for updating data.

What this suggests: Most map implementations have similar data update times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps are considerably slower than the rest.

What this suggests: Most map implementations have similar data update times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps are considerably slower than the rest.

What this suggests: Most map implementations have similar data update times, however ConcurrentHashMap (and especially) Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps are considerably slower than the rest.

• While not always the most performant, Koloboke's HashLongObjMap consistently one of the fastest collections measured. This makes Koloboke's collection the most favorable for most general cases.
• Fastutil's Long2ObjectOpenHashMap is a close runner-up for general use-cases, but it slows down slightly on remove operations.
• Fastutil's Long2ObjectArrayMap and Agrona's Long2ObjectHashMaps a consistently on the bottom of performance tests.
• For this specific use case, manually synchronized HashMaps are preferred to ConcurrentHashMaps.