How to solve ActiveRecord::PreparedStatementCacheExpired errors on deploy

Occasionally when deploying a Rails app on Postgres you may see an ActiveRecord::PreparedStatementCacheExpired error. This will only happen if you have run a migration in the deploy.

This happens because Rails makes use of Postgres’ cached prepared statements feature for performance. You can disable that feature to avoid these errors (not recommended) but there is a better way to handle it safely if you want zero-downtime deploys.

First, some background. In Postgres the prepared statement cache becomes invalidated if the schema changes in a way that it affects the returned result.

Examples:

My work here ensures that in case this happens in a Rails transaction, we correctly deallocate the outdated prepared statement cache and raise ActiveRecord::PreparedStatementCacheExpired. It is up to the application developer to decide what to do with this.

The developer may choose to catch this error and retry the transaction. We can expect the transaction to succeed on the second attempt, since Rails clears the prepared statement cache after the transaction fails.

Here’s how you can transparently rescue and retry transactions.

# Make all transactions for all records automatically retriable in the event of
# cache failure
class ApplicationRecord
  class << self
    # Retry automatically on ActiveRecord::PreparedStatementCacheExpired.
    #
    # Do not use this for transactions with side-effects unless it is acceptable
    # for these side-effects to occasionally happen twice
    def transaction(*args, &block)
      retried ||= false
      super
    rescue ActiveRecord::PreparedStatementCacheExpired
      if retried
        raise
      else
        retried = true
        retry
      end
    end
  end
end

You can now call a retriable transaction like this:

# Automatically retries in the event of ActiveRecord::PreparedStatementCacheExpired
ApplicationRecord.transaction do
  # ...
end

or

# Automatically retries in the event of ActiveRecord::PreparedStatementCacheExpired
MyModel.transaction do
  # ...
end

That should clear up any prepared statement cache errors you’re seeing on deploy, and make it completely invisible to your end users.

IMPORTANT NOTE: if you are sending emails, POSTing to an API or doing other such things that interact with the outside world inside your transactions, this could result in some of those things occasionally happening twice.

NB. This is why retrying is not automatically performed by Rails, and instead we leave this up to the application developer.

If you have a transaction with side-effects that cannot be avoided and would prefer the original behaviour of raising rather than retrying in the event of this error, you can call the original like this:

# Raises instead of retries on ActiveRecord::PreparedStatementCacheExpired
ActiveRecord::Base.transaction do
  # ...
  post_to_some_api
  send_some_email
  # ...
end

Avoiding side-effects in transactions

There’s a potential trip-up here, since you might have implemented these side-effect methods as a model after_save callback or similar.

Ideally you should structure your application so that there are no side-effects in any of the model callbacks. You should instead move these methods outside of transactions completely, since it makes your transactions easier to reason about as an atomic unit and in any case it’s bad practice to hold a database transaction open unnecessarily.

One way to do this is to use a Service Object approach. Let’s say you have a User model that looks like this:

class User
  after_create :send_registration_email

  private

  def send_registration_email
    UserMailer.registered(self).deliver_now
  end
end

With our new auto-retry transaction, if we create a user inside the transaction we run the risk of sending the registration email twice.

User.transaction do
  # this might get retried and send the email twice
  # ...
  user.create!(params)
  # ...
end

One way to resolve the problem is to remove the callback from the model and create a service object to encapsulate this logic instead.

class UserCreator
  def initialize(user)
    @user = user
  end

  def create(params)
    User.transaction do
      # this is safe to retry since we send the email outside
      # of the transaction
      # ...
      user.create!(params)
      # ...
    end
    send_registration_email
  end

  private

  def send_registration_email
    UserMailer.registered(self).deliver_now
  end
end

Use it like this:

UserCreator.new(user).create(params)

By using a service object, not only have you made your transaction side-effect free, you have also made your model thinner and easier to manage at the same time.