Performance

This document captures various performance metrics, comparing RelStorage to other ZODB storages at given points in time. The numbers given here cannot be compared to each other outside their own test.

RelStorage 2.1a1 vs RelStorage 2.0

This section compares the performance improvements in RelStorage 2.1a1 with RelStorage 2.0. (Hot and warm results are omitted because they had essentially no change in these tests.)

All RelStorage schemas were history free and did not use memcache. ZODB was version 5.1.1, MySQL was version 5.7.17, and PostgreSQL was version 9.6.1. All databases and cache settings were at the default. The database drivers were mysqlclient-1.3.9 and psycopg2-2.6.2.

The test suite was zodbshootout 0.6 running on CPython 2.7.13 on a 2.5Ghz Intel Core i7 (MacBookPro11,5) under macOS 10.12.2.

First, the default test scenario for zodbshootout (1000 objects per transaction, each with a size of 128), running in two concurrent processes. We can see substantial gains for PostgreSQL on all tests (30-40%), while MySQL shows modest gains for adding and reading objects (10%).

_images/perf-rs21v20-c2.png

That test is useful for assessing raw throughput, but it is not very representative of most real-world uses. Studies of some production databases show that most transactions consist of 100 or fewer objects that are often around 256 bytes in size. (Naturally these numbers can vary quite a bit depending on application.) Also, many applications use in-process concurrency, whether threads or gevent.

This test reflects that, using 6 threads each working on 100 256-byte objects. We can again see substantial gains for PostgreSQL on adding and updating objects (20% and 60%, respectively), and modest gains for MySQL on both those tasks (10% and 7%, respectively).

_images/perf-rs21v20-c6-n100-s256.png

RelStorage 2.0

All RelStorage schemas were history free and did not use a memcache instance. ZEO was version 5.0.4 and used a FileStorage backend from ZODB 5.1.1 MySQL was version 5.5.53 and PostgreSQL was version 9.6.1. The database drivers were mysqlclient-1.3.9 and psycopg2 2.6.2. All RelStorage cache settings were at their default values (MySQL configuration had been somewhat modified from its defaults but PostgreSQL was at its defaults).

The test suite was zodbshootout 0.6 with a concurrency level of 2 (and all other options the default), running on CPython 2.7.12 on a 2.5Ghz Intel Core i7 (MacBookPro11,5) under macOS 10.12.2.

Transaction PostgreSQL MySQL ZEO
Add 1000 Objects 19534 23184 8152
Update 1000 Objects 17140 23790 7122
Read 1000 Warm Objects 14232 16649 4796
Read 1000 Cold Objects 15595 18070 4860
Read 1000 Hot Objects 86326 88422 82600
_images/perf-rs20-no-mem.png

Memcache

This is exactly the same configuration as above, but shows the effect of introducing a local memcache instance. The “warm” test performs better, but the other tests perform worse.

Transaction PostgreSQL MySQL ZEO
Add 1000 Objects 14789 17378 8220
Update 1000 Objects 13834 17711 7021
Read 1000 Warm Objects 27394 27978 4662
Read 1000 Cold Objects 7621 8415 4344
Read 1000 Hot Objects 86165 83438 76540
_images/perf-20.png

PyPy

This is the same as the second configuration, except we’re using PyPy 5.6.0 for everything (including running the ZEO server). The database drivers were PyMySQL-0.7.9 and psycopg2cffi-2.7.5.

Transaction PostgreSQL MySQL ZEO
Add 1000 Objects 15235 13849 4681
Update 1000 Objects 18522 25939 4853
Read 1000 Warm Object 15138 964041 23843
Read 1000 Cold Object 11974 9876 3360
Read 1000 Hot Objects 99241 81685 25552
_images/perf-rs20pypy.png

Under PyPy, the benchmark was run in shared threads mode (--threads shared) instead of separate processes, to allow PyPy’s JIT to warm up. Under CPython, shared threads for concurrency two generally reduces performance (with the exception of one test), but under PyPy it substantially improves performance.

For comparison, here’s the shared thread results for CPython:

Transaction PostgreSQL MySQL ZEO
Add 1000 Objects 13409 14081 4852
Update 1000 Objects 11541 14310 4496
Read 1000 Warm Objects 42646 43699 123079
Read 1000 Cold Objects 10253 7712 3497
Read 1000 Hot Objects 24973 26129 25049

RelStorage 1.6.1 vs RelStorage 2.0

These tests are run using the same configuration (databases, database drivers, CPython version and same machine) as above, to show any performance differences between RelStorage 2.0 and RelStorage 1.6.1 (running with ZODB 4.4.4 and ZEO 4.3.1).

_images/perf-rs20-rs16.png

RelStorage 1.4.0b1

All RelStorage schemas were history free and used a memcache instance. ZEO was version 3.9.3 and used a FileStorage backend.

The test suite was zodbshootout 0.2 with a concurrency level of 2, running on Python 2.6 on a 2.1Ghz Intel Core 2 Duo (T8100)

Transaction PostgreSQL MySQL ZEO
Add 1000 Objects 9177 12711 5044
Update 1000 Objects 10040 9431 4017
Read 1000 Warm Objects 20897 22310 1924
Read 1000 Cold Objects 6591 5853 1932
Read 1000 Hot Objects 36847 37950 36545
_images/perf-rs14.png