How to load a file into IEx

I’m making this little post because I’ve googled for this at least 5 times now and none of the results are ever what I’m looking for.

If you want to LOAD a file into IEx, do it like this:

iex> import_file("relative/path/to/file.ex")

If you want to RELOAD a module after it’s already been loaded from a file, do it like this:

iex> r MyModule

That’s it!

Will Elixir/Phoenix make Ruby on Rails obsolete?

In this article I want to compare Ruby vs. Elixir to see how they stack up, and explore how this new language might affect the Ruby ecosystem. But to start with, a little background on Ruby and why it’s so great.

In 2005 DHH released a video entitled “How to build a blog engine in 15 minutes”.

It showed off a little-known framework called Rails, which allowed for unheard-of speed in developing the type of CRUD web applications common throughout the 2000s.

Rails was almost magical in the how it could do so much with such a tiny amount of code, and even today Rails is still probably the best choice for fast iteration in web development.

Most of this magic was built on the back of Ruby’s extremely powerful metaprogramming abstractions. Ruby was my first love and will always occupy a special place in my heart. It is one of the most metaprogrammable languages ever designed and that shines through as it’s core strength.

This metaprogramming power allows DSLs to be constructed in a way that complex operations can be written in a way that reads almost like english:

# RSpec
login_as learner
visit course_path(course)
find('.join').click
expect(page).to have_content('Thanks for joining!')

Even in 2017 Ruby is still one of the best options as a general purpose scripting language and for many types of web application. There are a lot of developers around who are fluent in Ruby and they tend to be of high quality.

However, Ruby does have a number of shortcomings when applied to modern web developent:

Runtime speed - The Ruby runtime has never been a speed demon (although it’s slowly getting better). There are design choices in the language that make high performance difficult (ahem mutable strings) and as heroic as the efforts of Matz and his team are, they simply don’t have the developer muscle that goes into optimising something like the JVM or GoLang.

The practical upshot of this is a floor of around ~100ms for a typical web request using Rails, and test suites that can become painfully slow for large applications.

Boot speed/memory consumption - Primarily a Rails problem, as the number of gems goes up the boot speed of a Rails app gets slower and slower. It is not uncommon to see a single Rails process using 300-400MB of memory, notwithstanding any leaks - which become more and more likely as more third party gems are included in a project.

Concurrency (not parallelism!) - Ruby is pretty bad at being concurrent. The native tool for handling concurrency is the Ruby thread, which is one of the hardest concurrency primitives to reason about and requires painful messing about with mutexes and semaphores to handle safely.

Scalability (parallelism) - While it does have threads, native Ruby code is always limited by the GIL so that true parallelism within a process is not possible. The rule is, if it’s Ruby code then only one line can execute at a time within a given process. The way to scale Ruby is to throw a lot of processes at it, and a lot of servers. But processes are costly and this is an expensive way to scale.

Despite these issues, Ruby has remained popular for the primary reason that it is beautiful to work with, and is so malleable that it allows for fast iteration especially for startups who don’t quite know what they are building yet.

There are many languages that solve the performance problems listed above, but none so far have matched the beauty and expressiveness of Ruby. Elixir promises to change that.

Elixir promises to deliver the elegance and power of Ruby, built on the reliability, speed and scalability of the Erlang virtual machine. The stated design goals are compatibility (with Erlang), productivity and extensibility.

As it’s designers are keen to point out, Elixir is not Ruby for the EVM. The syntax may appear to have a passing familiarity, but many core concepts in the language are quite different.

At its core, Elixir is secretly a LISP pretending to be something else. This allows for a particularly powerful form of metaprogramming via macros, that allows you to rewrite the AST before it is evaluated. It is also a functional language with no mutable state. This promises to greatly improve long term maintainability as compared to OO languages like Ruby where data structures are mutable by default.

Many of Elixir’s features are not strictly part of the language, but rather are built using the same macro system that is also available to you as a developer. It is highly extensible in a similar way to Clojure.

Elixir’s greatest strength is the Erlang Virtual Machine, which along with Erlang’s OTP framework provides a base for building systems that run forever, self-heal and scale as large as you need them.

Among other things the Erlang platform also promises:

And I suppose this article wouldn’t be complete without some arbitrary benchmark numbers comparing Elixir/Phoenix to Ruby/Rails:

Phoenix - 43,063 req/s, 2.82ms avg latency Rails - 3,274 req/s, 17.25ms avg latency

That’s a 13.5x performance increase, which means you need 13.5x LESS machines to do the same job, and everybody gets faster page loading to boot.

So will Elixir make Ruby obsolete?

Yes, and no.

Ruby isn’t going anywhere and I will continue to use it myself for many client projects. I worked on a small ecommerce store recently handling a couple of hundred users a day. Ruby/Rails was a great choice for this, the established ecosystem allowed me to leverage existing libraries to get up and running really fast. It will never need to run on more than one server and will be easy to maintain for other developers.

Ruby is great for projects that don’t need to scale or handle realtime updates and for hacking on prototypes to get something up really fast. It has a vibrant community and a fantastic ecosystem of tools. Even the UK government is using Ruby for it’s projects which just goes to show how mainstream it has become.

However, for apps requiring high scalability and reliability as well as malleability that allows for fast iteration, I think we will start to see many successful startups and some of the more adventurous Rubyists switching to Elixir as their secret weapon.

Perhaps a better question would be, what is the best framework for a scalable app allowing for fast iteration, receiving high traffic and sending realtime updates?

As of 2017, that looks like Elixir/Phoenix.

A Simple, Tiny Flashcard App In Swift

I had to read out my postcode to someone on the phone recently and the conversation went something like this:

Me: FO0 8AR

Her: Was that F for ferret or S for sierra?

Me: Ferret

Her: OK, so that’s FO0 8ER

Me: No, that’s AR

Her: AR0 ?

Me: facepalm

So I decided that it might be a good idea to finally learn the NATO phonetic alphabet. That’s Alfa Bravo Charlie all that jazz.

Instead of actually learning it like a normal person, I thought hang on maybe I can turn this into an opportunity. Maybe I can write a flashcard app for the iPhone to learn it.

And that longwinded story is how I build my first Swift app:

Flashcard App screenshot

I’ve had my eye on Swift for some time. I like it mostly because it means I don’t have to write Objective C, which is so ugly it makes me throw up a little bit in my mouth every time I have to look at it. Those square brackets… shudders.

Syntax jokes aside, I’m also a big fan of anything that allows me to write more code and less boilerplate, and Swift seems to fit the bill.

This app turned out to be super simple, the source code is here if you feel like checking it out.

It can even read the correct pronunciation out for you. And it worked - next time I have to read out a postcode or email address over the phone I’ll be prepared.

How to implement token-based authentication between Ember.js and Rails using Devise backend

This post is more as a helpful reminder to myself, because this is the third time around I’ve had to implement this.

Here is the Rails code necessary in order to get seamless authentication working using ember-simple-auth’s Devise backend.

RAILS

# app/controllers/api_controller.rb
class ApiController < ActionController::Base
  attr_reader :current_user
  respond_to :json
  before_action :authenticate_user_from_token!
  protect_from_forgery with: :null_session

  private

  def authenticate_user_from_token!
    authenticated = authenticate_with_http_token do |user_token, options|
      user_email = options[:email].presence
      user       = user_email && User.find_by_email(user_email)

      if user && Devise.secure_compare(user.authentication_token, user_token)
        @current_user = user
        sign_in user, store: false
      else
        render json: { errors: ['Invalid authorization.'] }, status: :unauthorized
      end
    end

    unless authenticated
      render json: { errors: ['No authorization provided.'] }, status: :unauthorized
    end
  end
end
# app/controllers/api/sessions_controller.rb
class Api::SessionsController < Devise::SessionsController
  respond_to :json

  # POST /api/users/sign_in
  def create
    respond_to do |format|
      format.json do
        self.resource = warden.authenticate!(auth_options)
        data = {
          user_id: resource.id,
          token: resource.authentication_token,
          email: resource.email
        }
        render json: data, status: :created
      end
    end
  end
end
# app/models/user.rb
class User < ActiveRecord::Base
  before_save :ensure_authentication_token

  devise :database_authenticatable, :recoverable, :trackable, :validatable

  # Generate a token for this user if one does not already exist
  def ensure_authentication_token
    if authentication_token.blank?
      self.authentication_token = generate_authentication_token
    end
  end

  # Identical to above except it saves the user
  def ensure_authentication_token!
    ensure_authentication_token
    save
  end

  # Forces a new authentication token to be generated for this user and saves it
  # to the database
  def reset_authentication_token!
    self.authentication_token = generate_authentication_token
    save
  end

  private

  def generate_authentication_token
    loop do
      token = Devise.friendly_token
      break token unless User.find_by(authentication_token: token)
    end
  end
end

# config/routes.rb
Rails.application.routes.draw do
  namespace 'api', constraints: { format: 'json' } do
    devise_for :users, singular: 'user', module: 'api', controllers: { sessions: 'api/sessions' }
  end
end

And here is the Ember code counterpart:

EMBER

// app/adapters/application.js (if using Active Model Adapter)
import ActiveModelAdapter from 'active-model-adapter';
import DataAdapterMixin from 'ember-simple-auth/mixins/data-adapter-mixin';

export default ActiveModelAdapter.extend(DataAdapterMixin, {
  namespace: 'api',
  authorizer: 'authorizer:devise'
});

// app/authenticators/devise.js
import DeviseAuthenticator from 'ember-simple-auth/authenticators/devise';

export default DeviseAuthenticator.extend({
  serverTokenEndpoint: '/api/users/sign_in'
});
// app/authorizors/devise.js
import DeviseAuthorizer from 'ember-simple-auth/authorizers/devise';

export default DeviseAuthorizer.extend();

// app/controllers/application.js
import Ember from 'ember';

export default Ember.Controller.extend({
  session: Ember.inject.service('session'),

  actions: {
    invalidateSession() {
      this.get('session').invalidate();
    }
  }
});

// app/controllers/login.js
import Ember from 'ember';

export default Ember.Controller.extend({
  session: Ember.inject.service('session'),

  actions: {
    authenticate() {
      this.get('session').authenticate('authenticator:devise', EMAIL, PASSWORD).catch((reason) => {
        this.set('errorMessage', reason.error);
      });
    }
  }
});
// app/routes/some_authenticated_route
import Ember from 'ember';
import AuthenticatedRouteMixin from 'ember-simple-auth/mixins/authenticated-route-mixin';

export default Ember.Route.extend(AuthenticatedRouteMixin);
// app/routes/some_unauthenticated_route
import Ember from 'ember';
import UnauthenticatedRouteMixin from 'ember-simple-auth/mixins/unauthenticated-route-mixin';

export default Ember.Route.extend(UnauthenticatedRouteMixin);
// config/environment.js
module.exports = function(environment) {
  var ENV = {};

  ENV['ember-simple-auth'] = {
    authorizer: 'authorizer:devise',
    crossOriginWhitelist: ['*']
  }

  return ENV;
};

Chasing the unicorn - zero-downtime Rails deploys

Background

At Fosubo we use Amazon Opsworks to host our Rails servers. Our configuration consists of two c3.large servers running 16 Rails workers each, both connected to a shared load balancer.

Code merged into master is automatically deployed using the default Opsworks deploy command.

This worked fine in the early days when we had a low volume of requests, however since then our traffic volume has increased dramatically and we have started to see exceptions on deploys. This means that while deploys are running, some users are seeing 500 pages. When you deploy multiple times a day, this can result in quite a few unhappy users.

Our goal: To eliminate exceptions on deploys and be able to deploy multiple times a day with no downtime.

After digging into this, it turns out that zero-downtime deploys with Rails are quite difficult and there are three major issues:

  1. Multiple Rails migration processes executing simultanously are subject to a race condition.
  2. Rails’ prepared statement cache is out of date during the window of time between schema changes and restarting the rails servers which results in SQL exceptions.
  3. Code that references removed columns will also fail during this window.

Here’s how I addressed the first concern via pull request to Rails core.

Part 1 - Fixing concurrent migrations in Rails

The default Opsworks deploy works like this:

  • Run deploy command on all instances simultaneously
    • Copy the new code
    • Migrate database if the option is checked
    • Symlink the ‘current’ directory to the new code
    • Restart all the server processes

It turns out that running migrations on all deploy targets simultaneously is actually quite dangerous.

Lets simulate what happens when we run a simple migration (in this case adding an index) in multiple simultaneous processes:

class FirstTestDummy < ActiveRecord::Migration
  # Add a hash index to a large table (500,000 rows)
  def up
    add_index :review_requests, :url_code, using: :hash
  end
end
for run in {1..3}
do
bundle exec rake db:migrate &
done

One migration succeeds, but the other two fail:

PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "pg_class_relname_nsp_index"
DETAIL:  Key (relname, relnamespace)=(hash_index_review_requests_on_url_code, 5605052) already exists.

Result: disaster

After reproducing this I dug into the Rails source and was surprised to find that Rails migrations were basically running with scissors and performed absolutely no locking whatsoever. That is to say, if you run them in parallel they will execute concurrently and tread on each other’s toes with undefined and potentially dangerous results.

Now, this bug may not be apparent in quick migrations because its probable that one machine will be slightly faster than the other and finish before the others even start. However if you have a migration that takes a long time to complete (or if you get unlucky) you might see a race condition as multiple instances simultaneously try to make the same changes.

For the above case the result is some failed deploys - annoying but not catastrophic. However, lets look at another example:

class SecondTestDummy < ActiveRecord::Migration
  # John has too much money in his account because of the bug we recently fixed, make sure to reset it.
  def up
    john = User.find(1)
    john.money -= 100
    john.save!
  end
end

The expectation for migrations is that they get run only once. However, if we repeated our ‘deploy’ with the migration above, it is possible that John could end up with -200 or -300 money, rather than -100 as intended.

Result: catastrophe

As developers we would like to have confidence that a migration will be run once and exactly once, regardless of the deployment method.

Thankfully I managed to get this problem fixed in upstream Rails - I filed an issue on the Rails project page about it, and submitted a patch that uses advisory locking to enforce that only one migration can run at a time.

Kudos to sgriff, tenderlove, and the other Rails maintainers for reviewing my code and helping to get this PR merged in so quickly. A two-day turnaround is pretty fast for a project of this size.

Now if you try to run multiple migrations in parallel, only one will get the lock and run the migration - the others will take no actions and instead fail with a ConcurrentMigrationError which could be rescued allowing for a clean exit.

This patch won’t be in production until at least the next Rails release, so for the time being I wrote a gem called opsworks_interactor that works around the issue by using rolling deploys.

Result: disaster averted