This document captures various performance metrics, comparing RelStorage to other ZODB storages at given points in time. The numbers given here cannot be compared to each other outside their own test.
RelStorage 2.1a1 vs RelStorage 2.0¶
This section compares the performance improvements in RelStorage 2.1a1 with RelStorage 2.0. (Hot and warm results are omitted because they had essentially no change in these tests.)
All RelStorage schemas were history free and did not use memcache. ZODB was version 5.1.1, MySQL was version 5.7.17, and PostgreSQL was version 9.6.1. All databases and cache settings were at the default. The database drivers were mysqlclient-1.3.9 and psycopg2-2.6.2.
The test suite was zodbshootout 0.6 running on CPython 2.7.13 on a 2.5Ghz Intel Core i7 (MacBookPro11,5) under macOS 10.12.2.
First, the default test scenario for zodbshootout (1000 objects per transaction, each with a size of 128), running in two concurrent processes. We can see substantial gains for PostgreSQL on all tests (30-40%), while MySQL shows modest gains for adding and reading objects (10%).
That test is useful for assessing raw throughput, but it is not very representative of most real-world uses. Studies of some production databases show that most transactions consist of 100 or fewer objects that are often around 256 bytes in size. (Naturally these numbers can vary quite a bit depending on application.) Also, many applications use in-process concurrency, whether threads or gevent.
This test reflects that, using 6 threads each working on 100 256-byte objects. We can again see substantial gains for PostgreSQL on adding and updating objects (20% and 60%, respectively), and modest gains for MySQL on both those tasks (10% and 7%, respectively).
All RelStorage schemas were history free and did not use a memcache instance. ZEO was version 5.0.4 and used a FileStorage backend from ZODB 5.1.1 MySQL was version 5.5.53 and PostgreSQL was version 9.6.1. The database drivers were mysqlclient-1.3.9 and psycopg2 2.6.2. All RelStorage cache settings were at their default values (MySQL configuration had been somewhat modified from its defaults but PostgreSQL was at its defaults).
The test suite was zodbshootout 0.6 with a concurrency level of 2 (and all other options the default), running on CPython 2.7.12 on a 2.5Ghz Intel Core i7 (MacBookPro11,5) under macOS 10.12.2.
|Add 1000 Objects||19534||23184||8152|
|Update 1000 Objects||17140||23790||7122|
|Read 1000 Warm Objects||14232||16649||4796|
|Read 1000 Cold Objects||15595||18070||4860|
|Read 1000 Hot Objects||86326||88422||82600|
This is exactly the same configuration as above, but shows the effect of introducing a local memcache instance. The “warm” test performs better, but the other tests perform worse.
|Add 1000 Objects||14789||17378||8220|
|Update 1000 Objects||13834||17711||7021|
|Read 1000 Warm Objects||27394||27978||4662|
|Read 1000 Cold Objects||7621||8415||4344|
|Read 1000 Hot Objects||86165||83438||76540|
This is the same as the second configuration, except we’re using PyPy 5.6.0 for everything (including running the ZEO server). The database drivers were PyMySQL-0.7.9 and psycopg2cffi-2.7.5.
|Add 1000 Objects||15235||13849||4681|
|Update 1000 Objects||18522||25939||4853|
|Read 1000 Warm Object||15138||964041||23843|
|Read 1000 Cold Object||11974||9876||3360|
|Read 1000 Hot Objects||99241||81685||25552|
Under PyPy, the benchmark was run in shared threads mode (
shared) instead of separate processes, to allow PyPy’s JIT to warm
up. Under CPython, shared threads for concurrency two generally
reduces performance (with the exception of one test), but under PyPy
it substantially improves performance.
For comparison, here’s the shared thread results for CPython:
|Add 1000 Objects||13409||14081||4852|
|Update 1000 Objects||11541||14310||4496|
|Read 1000 Warm Objects||42646||43699||123079|
|Read 1000 Cold Objects||10253||7712||3497|
|Read 1000 Hot Objects||24973||26129||25049|
RelStorage 1.6.1 vs RelStorage 2.0¶
These tests are run using the same configuration (databases, database drivers, CPython version and same machine) as above, to show any performance differences between RelStorage 2.0 and RelStorage 1.6.1 (running with ZODB 4.4.4 and ZEO 4.3.1).
All RelStorage schemas were history free and used a memcache instance. ZEO was version 3.9.3 and used a FileStorage backend.
The test suite was zodbshootout 0.2 with a concurrency level of 2, running on Python 2.6 on a 2.1Ghz Intel Core 2 Duo (T8100)
|Add 1000 Objects||9177||12711||5044|
|Update 1000 Objects||10040||9431||4017|
|Read 1000 Warm Objects||20897||22310||1924|
|Read 1000 Cold Objects||6591||5853||1932|
|Read 1000 Hot Objects||36847||37950||36545|