AlloyDB is a new HTAP database that Google claims to give 4x better OLTP performance than Postgres. Here, we describe a performance test to compare its performance to Postgres as I wanted to see it for myself. TL;DR: For the tests we did, we find the performance is the same with little to no difference at all. This is not to challenge Google’s claim as the results vary depending on how the databases are configured, what data is stored and what queries are run etc.

The Setup

For our Postgres server we use a VM with 8 vCPU, 32 GB RAM and 250 GB SSD. This costs $447 / month. Refer this.

For our AlloyDB server we use a VM with 4 vCPU, 32 GB RAM and 250 GB SSD. This costs 48.24*4+8.18*32+0.3*250 = $530 / month. Refer this. Although a VM with 2 vCPU would be closer in cost to $447, the GCloud console did not provide that configuration as an option.

shared_buffers is an important setting and is found to be set to 10704 MB for Postgres and 25690 MB for AlloyDB by GCloud – we did not customize the settings.

We downloaded and installed sysbench 1.0.20 on Debian. Log file. sysbench test scripts are located under /usr/share/sysbench.

Preparing Test Table

A test table with 1B rows was created in both Postgres and AlloyDB using following command as example:

sysbench \
	--db-driver=pgsql \
--table_size=1000000000 \
--tables=1 \
--threads=10 \
--pgsql-host=xxx \
--pgsql-port=5432 \
--pgsql-user=sysbench \
--pgsql-db=sysbench \
--pgsql-password=xxx \
oltp_common \
prepare

The time to create this table was 638m53.148s for Postgres and 563m30.938s for AlloyDB. Its similar.

The table schema is simply:

sysbench=> \d sbtest1
                                Table "public.sbtest1"
 Column |      Type      | Collation | Nullable |               Default
--------+----------------+-----------+----------+-------------------------------------
 id     | integer        |           | not null | nextval('sbtest1_id_seq'::regclass)
 k      | integer        |           | not null | 0
 c      | character(120) |           | not null | ''::bpchar
 pad    | character(60)  |           | not null | ''::bpchar
Indexes:
    "sbtest1_pkey" PRIMARY KEY, btree (id)
    "k_1" btree (k)

Thus each row is 188 bytes in size and 1B rows gives us 188 GB of data. The actual size was measured using

sysbench=> SELECT ROUND(pg_total_relation_size (oid) / ( 1024.0 * 1024.0 ), 2) AS
       "Size in MB"
FROM   pg_class c
WHERE  relname = 'sbtest1';

and found to be 242,437.01 MB for Postgres and 242,650.06 MB for AlloyDB. They are practically the same.

A huge table was created to make sure the table size is more than the available RAM. This will drive the performance in the red zone.

Running the tests

After this we ran all the test scripts under /usr/share/sysbench with 100 threads.

sysbench \
	--db-driver=pgsql \
--threads=100 \
--pgsql-host=xxx \
--pgsql-port=5432 \
--pgsql-user=sysbench \
--pgsql-db=sysbench \
--pgsql-password=xxx \
$1 \
run

This is what we found (tx/s = transactions per second):

Conclusion

The results are practically the same with little to no difference. What about MySQL? We did perform the same analysis on MySQL 8.0 (using similar VM as Postgres). Here are the results:

Running the sysbench tests however gave this error: Commands out of sync; you can't run this command now and prevented further testing. This is a bug in sysbench fixed in this commit but not released at time of this writing. I am actually surprised how people use sysbench to test MySQL when its broken out of the box.

Further Reading

• Creating Custom Sysbench Scripts