How to solve ActiveRecord::PreparedStatementCacheExpired errors on deploy06 May 2017
Occasionally when deploying a Rails app on Postgres you may see an
ActiveRecord::PreparedStatementCacheExpired error. This will only happen if you have run a migration in the deploy.
This happens because Rails makes use of Postgres’ cached prepared statements feature for performance. You can disable that feature to avoid these errors (not recommended) but there is a better way to handle it safely if you want zero-downtime deploys.
First, some background. In Postgres the prepared statement cache becomes invalidated if the schema changes in a way that it affects the returned result.
- adding or removing a column then doing a
- removing the foo column then doing a
My work here ensures that in case this happens in a Rails transaction, we correctly deallocate the outdated prepared statement cache and raise
ActiveRecord::PreparedStatementCacheExpired. It is up to the application developer to decide what to do with this.
The developer may choose to catch this error and retry the transaction. We can expect the transaction to succeed on the second attempt, since Rails clears the prepared statement cache after the transaction fails.
Here’s how you can transparently rescue and retry transactions.
# Make all transactions for all records automatically retriable in the event of # cache failure class ApplicationRecord class << self # Retry automatically on ActiveRecord::PreparedStatementCacheExpired. # # Do not use this for transactions with side-effects unless it is acceptable # for these side-effects to occasionally happen twice def transaction(*args, &block) retried ||= false super rescue ActiveRecord::PreparedStatementCacheExpired if retried raise else retried = true retry end end end end
You can now call a retriable transaction like this:
# Automatically retries in the event of ActiveRecord::PreparedStatementCacheExpired ApplicationRecord.transaction do # ... end
# Automatically retries in the event of ActiveRecord::PreparedStatementCacheExpired MyModel.transaction do # ... end
That should clear up any prepared statement cache errors you’re seeing on deploy, and make it completely invisible to your end users.
IMPORTANT NOTE: if you are sending emails, POSTing to an API or doing other such things that interact with the outside world inside your transactions, this could result in some of those things occasionally happening twice.
NB. This is why retrying is not automatically performed by Rails, and instead we leave this up to the application developer.
If you have a transaction with side-effects that cannot be avoided and would prefer the original behaviour of raising rather than retrying in the event of this error, you can call the original like this:
# Raises instead of retries on ActiveRecord::PreparedStatementCacheExpired ActiveRecord::Base.transaction do # ... post_to_some_api send_some_email # ... end
Avoiding side-effects in transactions
There’s a potential trip-up here, since you might have implemented these side-effect methods as a model
after_save callback or similar.
Ideally you should structure your application so that there are no side-effects in any of the model callbacks. You should instead move these methods outside of transactions completely, since it makes your transactions easier to reason about as an atomic unit and in any case it’s bad practice to hold a database transaction open unnecessarily.
One way to do this is to use a Service Object approach. Let’s say you have a User model that looks like this:
class User after_create :send_registration_email private def send_registration_email UserMailer.registered(self).deliver_now end end
With our new auto-retry transaction, if we create a user inside the transaction we run the risk of sending the registration email twice.
User.transaction do # this might get retried and send the email twice # ... user.create!(params) # ... end
One way to resolve the problem is to remove the callback from the model and create a service object to encapsulate this logic instead.
class UserCreator def initialize(user) @user = user end def create(params) User.transaction do # this is safe to retry since we send the email outside # of the transaction # ... user.create!(params) # ... end send_registration_email end private def send_registration_email UserMailer.registered(self).deliver_now end end
Use it like this:
By using a service object, not only have you made your transaction side-effect free, you have also made your model thinner and easier to manage at the same time.