Chef Workflows at Chef NYC With Etsy, SecondMarket, and Paperless Post


Daniel Schauenberg

The Etsy Way

  • every node runs chef client every 10 minutes
  • github enterprise
  • production env, dev env, testing env (runs every new version of cookbook on chef server)
  • knife-spork for workflow
  • run shef on dev vm to detect syntax errors, debugging
  • using a tool called Review (in house Etsy) to automatically create branch and pull request

The Etsy Workflow

$ knife spork check apache
$ knife spork bump apache
$ git commit
$ git push
$ knife spork upload apache

Changes should be in github before pushing to chef server so no one can ever override someone else’s changes on chef server.

knife spork upload is basically an alias for knife cookbook upload --freeze


production deploy

$ knife node flip testing
$ knife role flip test_role testing

$ knife spork promote apache
$ git commit
$ git push
$ knife spork promote  apapche --remote (production deploy)

other tools Etsy uses

  • chef-handlers - push failed run stack trace to gist, push to chat
  • knife-lastrun

  • github enterprise

  • dev vms
  • shef as dev env
  • chef server and knife spork as dev system
  • monitoring, notifications, graphs


Julian Dunn

how not to trample others when making changes

The SecondMarket Way

  • break cookbooks into individual repos
  • chef-cookbooks/{repo}.git
  • allows for independent development, tracking and merging from upstream and contirbuting changes
  • roles contain only data, roles do not contain run-lists
  • use a roles cookbook with all data versioned within cookbook, point to this for run list (?)

cookbook development

  • vagrant and virtualbox
  • unit tests with chefspec
  • integration tests: test kitchen with minitests
  • missing databags and search
  • develop against a chef server
  • chefspec can mock
  • cheftest environment
  • plugins:


  • spork, change versions per env, notifications to hipchat (hipchat handler runs on all nodes)
  • semantic versioning
  • auto updated along with jira

the future

  • more people touching chef
  • move toward more modular cookbooks: easier to change, one change doesn’t update entire infrastructure repo
  • improve unit test and integration tests
  • cookbooks in CI
  • code reviews

Paperless Post

Aaron Quint

The Paperless Post Setup

  • 5 staging environments (1 per team, 1 pre-prod)
  • 4 full time ops
  • 16 devs who can make changes
  • many different apps and technologies managed by chef
  • private vmware/vsphere backed cloud
  • hosted chef

$ /bin/knife kill me --now

the agony

  • managimg multiple versions of cookbooks for different devs in each environment
  • wanting to test small changes on different environments
  • lack of visibility when deploy changes
  • impossible to correlate changes in git with state in chef-server

The Paperless Post Workflow

  • chckout feature
  • name branch
  • do work
  • commit
  • pull request for code review
  • merge branch into staging
  • deploys to environment
  • sends alerts via email and campfire
  • runs deploy in viewable jenkins environment

The Paperless Post Deploy

$ pp chef deploy earth organization/repo

  • check out locally
  • merge
  • git diff to check all cookbooks changed in that diff
  • check latest versions of cookbooks in chef server
  • bump version in metadata and environment.json
  • commit and push to env
  • send deploy to jenkins
  • jenkins uploads cookbooks and environment file
  • send notifications to campfire/email

the minor discomforts

  • dealing with conflicts is hard, especially in metadata
  • $ pp chef boom resets one environment to match another environment version
  • version number becomes meaningless
  • doesn’t handle change to roles in that you cant scope a role to an environment or new cookbook very well
  • chef server consistency is problematic

to improve

  • more code review as part of process
  • simple roles that map to recipes so roles can be managed and tested at environment level
  • using secondary store to keep track of versions, perhaps with zookeeper
  • chef versions could be tied to SHA instead of SemVer