Tips and Other Useful Information#

While RelStorage is essentially a drop-in replacement for FileStorage or ZEO, there are a few differences. This document will highlight some of those. Much of this information is applicable for all storages, though.

The Importance of Managing Transaction Lifetimes#

ZODB, and many other Python libraries, use transaction to represent application-level transactions.

When using RelStorage, an application-level transaction is also backed by a RDBMS transaction. In order to present a consistent view of the data in the database, RDBMS transactions typically need to reserve certain resources. In some database implementations, if these resources are reserved for long enough, problems can develop. For example, under PostgreSQL, problems with the auto-vacuum processes might emerge, manifesting as reduced performance and increased disk usage.

For this reason, it is important to be aware of, and manage, the lifetime of application-level transactions.

A secondary reason to manage application-level transaction lifetimes is that after transaction boundaries (starting a new transaction) is the only time a ZODB Connection can see new object states, including new objects. (Technically there are some ways around this, but they’re not recommended and can lead to consistency errors.)

Use Explicit Transaction Managers#

The best way to ensure transaction lifetimes are managed correctly is to be explicit about transaction lifetimes. That means making a call to begin(), followed by a call to commit() or abort() after the application has done its work (in a timely fashion).

Unfortunately, for historical reasons, the transaction library does not require this by default. In that mode, any use of a ZODB Connection or persistent object may automatically start a new transaction. Version 3 of the transaction library provides a new explicit attribute; when set to True, the library will enforce proper lifetime management. This has the added benefit of being much more efficient.

Applications that use the global transaction manager can set this attribute (in each thread) before opening a database connection:

import transaction
transaction.manager.explicit = True
transaction.begin()
try:
    ...
except:
    transaction.abort()
    raise
else:
    transaction.commit()

For new applications, and carefully written existing applications, this is usually an easy change.

It’s also possible to use a non-global transaction manager just with the ZODB. This manager can be put into explicit mode:

import transaction
txm = transaction.TransactionManager(explicit=True)
db = ZODB.config.databaseFromFile('path/to/config')
conn = db.open(txm)
txm.begin()
try:
    ...
except:
    transaction.abort()
    raise
else:
    transaction.commit()

Caution

This won’t work with other libraries that expect to use the global transaction manager.

Using a transaction runner that’s aware of explicit transactions, such as nti.transactions.loop.TransactionLoop is another option.

The Importance of Closing The Database#

Once you’ve opened a ZODB database, it’s important to always call close on that database when you are finished with it.

Important

This includes at program exit time if a script just “falls off the end.”

FileStorage writes its index out when the database is closed, and RelStorage writes its persistent cache files when the database is closed. If the database is never closed, this work doesn’t happen.

The way in which this is done is application specific. Some frameworks or application servers (such as gunicorn) have hooks or events that are called or fired which make good times to close the database.

A simple way to do this can be with atexit handlers. To decouple the handler from the database, the application might want to maintain a registry of open databases. Here’s an example using zope.component:

import atexit
from zope import component
from ZODB.interfaces import IDatabase

def close_database():
   db = component.getGlobalSiteManager().queryUtility(IDatabase)
   if db is None:
       return # already closed
   db.close()
   component.getGlobalSiteManager().unregisterUtility(db, IDatabase)

atexit.register(close_database)
db = ZODB.config.databaseFromFile('path/to/config')
component.getGlobalSiteManager().registerUtility(db, IDatabase)
try:
    serve_forever()
except:
     close_database()
     raise

It might also be necessary to register a signal handler to perform the same operation in the event of unclean shutdowns. See issue #183 for more discussion.

Packing A Database May Increase Disk Usage#

After packing a RelStorage, whether history-free or history-preserving, whether through the ZODB APIs or the command line tool Packing Or Reference Checking A ZODB Storage: zodbpack, you may find that the database disk usage has actually increased, sometimes by a substantial fraction of the main database size.

RelStorage deliberately stores information in the database about the object references it discovered during packing. The next time a pack is run, this information is used for objects that haven’t changed, making it unnecessary for RelStorage to discover it again. In turn, this makes subsequent packs substantially faster when there are many objects that haven’t changed (which is typically the case for many applications.)

A future version of RelStorage might provide the option to remove this data when a pack finishes.

You can also use the multi-zodb-gc script provided by the zc.zodbdgc project to pack a RelStorage. It does not store this persistent data, but it may be substantially slower than the native packing capabilities, especially on large databases.

Use `readCurrent(ob)` Judiciously#

At least on PostgreSQL, this involves disk I/O. See PostgreSQL for more.

Tips and Other Useful Information#

The Importance of Managing Transaction Lifetimes#

Use Explicit Transaction Managers#

The Importance of Closing The Database#

Packing A Database May Increase Disk Usage#

Use readCurrent(ob) Judiciously#

Use `readCurrent(ob)` Judiciously#