Getting Friendly with Fixtures

Factories ease the pain early on, but can hurt as an app grows.
Fixtures hurt a little early, but shine with complexity.
Both make testing easier in their own ways.
Neither are an antipattern.

There’s a divide in the Rails world. Well, there are many divides in the Rails world, but this one is over how to write tests. Ok. Fine. There are many divides in the Rails world, especially over how to write tests. I’ll just get to the point. Rails developers don’t agree on how to set up their test data.

There are two sides to this debate. Fixtures come with Rails by default. They are the method of choice for the Rails Core team, and DHH swears by them. In the other corner, we’ve got folks swapping out fixtures with factory gems like Factory Girl and Fabrication. The two strategies have enough differences and proponents to keep the discussion alive after so many years. Recently, a factory advocate went so far as to claim that fixtures were an antipattern, which got me thinking about how fixtures and factories fit into the testing book.

I’ve been using Fabrication as my test-data-ifier of choice for the past few months. Before that I was using Factory Girl after giving up on Fixtures pretty early on. Up until last week, I felt like I was in a pretty good place with test object generation. I had been using Fabricators actively in the testing book, and included them in some testing training. Last week, I was thrown back into an old client project that was backed by fixtures. My first thought was to rewrite the tests to use fabricators, but there just wasn’t time for it. Reluctantly, I was back to using fixtures for my tests.

The week started out slow as I tried to fit new fixtures into my fabricator workflow, but after a surprisingly short adjustment period, I was actually enjoying myself. I wasn’t preoccupied with putting together the right package of objects for complex tests. Everything I needed was just there, waiting to be tested. I expected to make it through the week, but I didn’t think I’d consider switching back to using fixtures regularly. It made me think back to the fixture/factory debate in a new light.

I wanted to take another look at the differences between fixtures and factories with a fresh set of eyes.

Definitions

Fixtures are defined in yaml files. They generally look like this:

bond:
  first_name: James
  last_name: Bond
  email: jamesbond007@mi6.gov.uk

bourne:
  first_name: Jason
  last_name: Bourne
  email: jbourne@cia.gov

Factories (Fabricators in this case) are defined like this:

Fabricator :agent do
  first_name  { Faker::Name.first_name }
  last_name  { Faker::Name.last_name }
  email { Faker::Internet.email }
end

The fixture way is to write multiple definitions for the same type of object. Factories are effectively templates for objects, so they end up looking a lot more like the table definitions in db/schema.rb. Fixtures encourage repetition and work best with some creative and memorable test data. Factories don’t make you think or repeat yourself.

Writing fixtures seems like work when you look at them side-by-side like this, especially when you’re using a data generator gem, like Faker, with factories. Defining data for single types of models is a clear win for factories.

Creating objects

Factories create new objects on demand in tests. Tests using factories start with a clean slate every time, waiting for you to ask for a specific type of object. Fixtures have a different approach. When Rails boots up the testing environment, one of the first things it does is read all the fixture definitions and insert them directly into the database, indiscriminately. Fixtures skip the entire object creation process, bypassing validations and callbacks.

Because of how they are loaded, fixtures are ridiculously fast, but can’t be changed. If an object with a certain type of test data is needed for a test, it needs to defined in a fixture file (or created with Model.new). This is especially frustrating early in development as the data and app requirements solidify.

Because of how factories are created, they have the advantage of flexibility. If you get to a point in a test where you realize you need a new kind of object, you can easily override a factory definition to suit the needs of the test. This is especially helpful in a new application where we are still figuring out how our test data needs to be structured.

test 'on_assignment collects agents on active missions' do
  agent_on_assignment = Fabricate(:agent, mission: Fabricate(:active_mission))
  agent_not_on_assignment = Fabricate :agent
  assert_includes Agent.on_assignment, agent_on_assignment
  refute_includes Agent.on_assignment, agent_not_on_assignment
end

In this test, we only need a predefined Mission and Agent factory, and handle the details in the test. We didn't need to jump to another file to set up a definition. If we need an agent with a mission again, then we can extract this into it’s own :agent_with_active_mission factory. This kind of flexibility pays off early on.

Unfortunately, there is a performance hit with factories. Factories need to create an entire object and possibly save it to the database to make it available to the test. This stacks up as you need more objects in tests, but as a side effect encourages testing in isolation.

That same test using fixtures would look something like this:

test 'on_assignment collects agents on active missions' do
  agents(:bourne).missions.clear
  assert_includes Agent.on_assignment, agents(:bond)
  refute_includes Agent.on_assignment, agents(:bourne)
end

We need to make sure that the :bond agent fixture is associated with a mission. To do that, we have to hop over to the fixture file for agents and missions. We also have to assume that the :bourne agent fixture might have missions associated with it, so we have to clear that association.

When it comes to flexibility and obviousness, factories have a clear advantage with their ability to dynamically create objects as you need them.

The Downside of Factories

A lot of the comparisons I’ve seen between fixtures and factories use examples from an application that’s pretty early in development or in single tests. Factories address the problem of not knowing what test data will look like. They let you make decisions about test data when you are most informed about what you need: in the test.

This advantage early on slowly turns into a disadvantage as the application grows. Factories create objects, encouraging developers to approach tests with as few objects and as little data as possible. There is a tax associated with adding new test data and objects. Tests take longer, and new data variants need to be added. Factories start building objects with associations and dependencies to test increasingly complicating scenarios. The simplicity that factories afforded us early on can end up fatiguing a more complex application.

It’s similar to the last mile problem with broadband internet. Writing those first tests are much easier with factories, but the amount of effort needed to cover complex tests goes up over time. It’s not a deal-breaker, but it is a bit irritating.

Fixtures after the honeymoon

We already looked at the downsides of using fixtures. You end up experiencing some pain early on by having to come up with good, redundant test data without a generator. You have to jump back and forth between tests and fixtures often when dealing with changing requirements. Maintaining that kind of overhead through the course of the project is intimidating.

But here’s the thing about fixtures: They get easier over time. Getting started with fixtures is a little painful, but can be pretty damned enjoyable once the data requirements settle down. I’ve found that fixtures can be a lot better if you go in with these things in mind:

You’ll write at least three fixtures for each model

Yes. You will have to write your own test data. As developers, this sucks because we know we don’t have to. It just feels wrong.

But, if you manage to push through those feelings of suckiness and frustration, a funny thing happens. Your test data starts to form a narrative. A story takes shape around your models, logically tying objects together as they would in your application. With factories, you’re building molds to be filled with nondescript plastic, but with fixtures, you end up building something by hand with more detail.

Fixtures give you the chance to stop thinking about your application as a bunch of classes and imagine it in the real world. It really helps.

Embrace loading everything up front

Three fixtures for each model seems like overkill when you start, but it quickly feels insufficient when you need to represent different types of data.

It’s not.

When you’re working with factories, you end up defining many different objects to handle different situations. This is a side effect of factories starting each test with a clean slate. If you often need a user object that has multiple purchase objects, you need to define a unique factory that builds those.

If you can embrace the way Rails loads all the fixtures before testing, you will have access the the most complex relationships with minimal effort. Factories require that you start with small and isolated test data and add more complex factories incrementally. Fixtures, on the other hand, want to be as complex as possible, as early as possible. You can do this by simply making sure associated objects are connected.

Yeah, you’ll have to keep that data up-to-date.

When you’ve got the same type of thing defined at least three times, you’ll need to implement changes to the data at least three times. Again, as developers, it feels like we shouldn’t have to do this. It just feels like a waste of time.

We also get a concentrated dose of this early on in the life of an application as our models take shape around shifting requirements.

If you work past that initial pain, it isn’t nearly as bad as you think. Migrations and data changes happen a lot early on, but quickly taper off as an application grows.

Don’t get stuck on naming

Factories are designed to be templates for different representations of model data. Factories end up having a lot of names to recall a lot of types of data. Need a user who has multiple orders? You’ll likely end up with a :user_with_multiple_orders factory to get you just what you need.

Nope. Not with fixtures. Your limited number of fixture objects will fulfill many different roles, so we won’t have the luxury of semantic naming. Just give those objects short, unique names that are easy to recall and call it a day.

Don’t forget about Object.new

Fixtures skip the entire object creation lifecycle. When you’re testing validations or callbacks in Rails, just create a new model object with the data from a fixture, like so:

test 'should require a unique code_name' do
  duplicate_mission = Mission.new(missions(:grand_slam).attributes)
  assert_invalid Mission.new, code_name: "can't be blank"
  assert_invalid duplicate_mission, code_name: 'has already been taken'
end

(assert_invalid is a custom assertion I set up in test_helper.rb)

Creating new Model objects is also a great way to handle one-off variants of your data that you will inevitably need.

So, Fixtures or Factories?

I don’t think there’s going to be a resolution to the Fixtures vs. Factories discussion. They are both incredibly useful and have their own pros and cons. Knowing the limitations of each will help you chose the best tool for the job.

I’m moving back to fixtures after giving them another chance, but would be perfectly happy on a project with factories. It’s nice to have options.

Do you want to learn how to test your Rails apps?
I'm writing a book about it! Sign up for updates.

Avatar

Hi. I'm Eric.

I've been building things with Rails for three years, and trying to figure out how to test for almost as long. By day, I'm a Rails product consultant, and by night, well I guess I'm still a Rails product consultant, only it's darker outside. You should follow me @genericsteele.