Sometimes you need to store key-value data (i.e., a dictionary, associative array or HashMap) in your application. Using a full blown RDMS can do it but is perhaps overkill. What you are looking for is instead a library that can persist HashMaps and support ACID semantics (transactions, concurrency and durability). There are 7 candidate libraries I tested for this in Java:

MapDB

Advantages: written in Java and Kotlin (I find this an advantage since my Java app doesn’t have to interop with native code). Disadvantages: not backed by any big company. 40 contributors (really only 1) vs. 700+ contributors (RocksDB). development is slower – in fact there is not much development activity at all looking at the commit history. documentation (and code samples) is not as good. consumes 2x more storage than RocksDB in my testing. AFAICT, there is no transaction isolation with this DB (v. 3.0.10) [1]. There is only a single connection to the db ever. You can choose to multiplex that connection across multiple threads but it comes with all the associated caveats if you do so. Put in other words, there is only a single global transaction and you are working with READ UNCOMMITTED isolation level. Only one writer can open the db at a time (multiple readers are okay) [2].

LMDB

This was my top choice when I was starting my evaluation and I really wanted to use it as old is gold but it requires you to declare the size of the database in advance [1]. This is a no-go for me. Internally it uses a B-Tree for indexing plus a memory map. The memory map creates a virtual address space that allows us to access data on disk as if it was in memory. Refer mmap for details.

RocksDB

This library is written in C++ but has Java bindings. It has much more bells and whistles than MapDB as it is backed by a big corporation and is a mature offering with 700+ contributors, decent documentation and code samples. But as it turns out, while testing it failed with an exception (all sizzle, no steak). RocksDB is based on a Log Structured Merge Tree (LSMT) which has emerged as the other popular alternative to B-Tree for write-heavy workloads.

Speedb

This was suggested to me in one of the forums. It is a drop-in replacement for RocksDB. When I tried it, I got exact same result as RocksDB down to the same exception. (translation: it is a copy of RocksDB with maybe a few changes to internal methods none of which made any material impact in my testing)

BDB

Originally it wasn’t in my list of candidates but I decided to add it as a bonus later on based on discussion about it on this page. There is a Java edition written entirely in Java. One user wrote “BDB JE is scalable to Terabytes of data and I use it in production systems all the time. It’s a wonderful piece of tech.” However the testing echoed another user’s experience: “Once Bdb reaches the limits of what can be cached in memory i’m finding that it slows down unacceptably. This generally happens after about 1mm inserts.”

Xodus

This one is broken [1] and disqualified.

H2 MVStore

from the docs:

The MVStore is somewhat similar to the Berkeley DB Java Edition because it is also written in Java, and is also a log structured storage, 

The API of the MVStore is similar to MapDB (previously known as JDBM) from Jan Kotek, and some code is shared between MVStore and MapDB. However, unlike MapDB, the MVStore uses is a log structured storage. The MVStore does not have a record size limit.

Who Would Win?

I tested the libraries on a HashMap of String -> String and benchmarked the time to insert 10M records. The key is a String of length 16 and the value is another String of length 100. With UTF-8 encoding each record takes up 116 bytes uncompressed. Total size is thus about 1G uncompressed. Here are the results (I don’t add a column for Speedb as it turned out to be a clone of RocksDB):

LMDB was a clear loser in my tests (also somewhat surprising because the next upcoming v4 version of MapDB seems to be going in direction of LMDB [1]). Whereas MapDB and RocksDB took about a minute to insert 10M records, I had to abort LMDB after 1.5 hours! Maybe there was something wrong in my code although I couldn’t find it. Compare the performance to this article and this. Lesson: never trust what you read online (including my article :-). This is esp. true regarding claims made by author(s) of a library. Do your own testing.

With BDB I found that the time it took to insert a record increased as the database became larger and larger. E.g., inserting the first 200k records was pretty fast and then it became slower and slower. LMDB also shows this same pattern which indicates they have overlapping internals. Inserting the first 1M records in LMDB took 70s but later on it became so slow that inserting 10M records couldn’t finish even after 90 minutes and I had to abort.

Have you tried any of the libraries above? Which one would you suggest?