For the last few years, all the projects I’ve been working on have made extensive use of a configuration management tool called Chef. It helps you define the configuration of the servers that you manage and enables you to rebuild a server easily. If you’re using Chef, you don’t change server configuration manually, everything gets defined in Ruby code.

In contrast to my work projects, my private virtual server was set up manually and over the years has grown organically. A lot of the software was working, but was set up in a non-standard way. Two weeks ago I ran some updates on my webserver hosting this blog and a few other websites and finally, after several years without much trouble, this update caused my webserver to stop working. I had to make the decision to either try to repair the existing setup, or start fresh and this time do it right. Obviously, I chose the latter. Since I’m hosting some stuff for friends, I was in a bit of a hurry to be back online for at least those websites. As a result, I had to take an iterative approach where my first goal was to get the important stuff running quick. Afterwards I then improved on the existing configuration, generalizing common functionality and adding more to it. As of today, my blog is back online and only a few minor things are left to install.

As I said, I have some practical knowledge modifying and managing configuration from work and I also had a two-day training, some weeks back. However, until now I never created a new project from scratch. I’ve learned a lot and I know that some of the things I’m doing could probably be done very different.

Tools

Normally, you use Chef in a Client/Server mode, where the server hosts the configuration and a client pulls it from there. For a single server this seems like too much effort. Since I’m familiar with it, I decided to use the serverless Chef-Solo to provision my server. Chef-Solo isn’t promoted much anymore, instead the way to go would be using Chef-Zero. I plan to migrate to this new approach in the future but at the moment I didn’t want to spend more time on this.

Chef structures configuration into so-called cookbooks. These can have dependencies on other cookbooks and in order to manage dependencies a tool called Berkshelf exists. It’s installed automatically if you install Chef. I’m using VirtualBox to test-run my configuration against a VM before I actually run it against my server. The VM has some initial setup (ssh keys) which is marked by a snapshot, this way I can always roll back and fully recreate an environment without having to reinstall the whole system. In the future I want to introduce Vagrant to my setup.

When you’re using Berkshelf, you manage your cookbooks in seperate repositories. I’m using Bitbucket for version control, as it offers an unlimited amount of private Git repositories.

Chef-Solo

The Chef Solo related files have their own repository, the files:

cookbooks/
data_bags/
Berksfile
deploy.sh
install.sh
metadata.rb
solo.json
solo-test.json
solo.rb
vendor.sh

Here’s a short explanation of what the seperate files do. I’ve taken some inspiration from this blog post.

deploy.sh
Run a deployment against a server. It copies the local directory to your server and then runs install.sh

install.sh
This script is invoked on the server, it installs everything Chef related and then brings the server to the state described in the configuration (“converge”). This file shouldn’t be invoked locally on your development machine!

solo.rb
This sets up where chef-solo should look for the cookbooks it will use to provision a machine.

solo.json / solo-test.json
When you execute chef-solo, it needs to know what to execute. This is done in a so-called run list. I have two run lists, one for my server, on for the testing environment. The later includes some attributes that override default configuration values.

vendor.sh
I’m not that familiar with Berkshelf, yet. This script uses Berkshelf to pull in all the cookbooks that need to be present in the cookbooks/ folder for the chef solo run. It also copies some files that I don’t want to have in version control (like the TLS server key).

metadata.rb / Berksfile
I’m managing the chef-solo repository as its own cookbook, so that Berkshelf can pull in it’s dependencies. These two files define which cookbooks have to be pulled in by Berkshelf. I’ve set up the dependencies in the Berksfile to use the git repositories as a source. That’s probably not the best way to go, but for now it works.

Cookbooks

I’ve mentioned before that Chef structures configuration into cookbooks. These cookbooks contain so-called recipes plus a number of other files for attributes, files, templates, and some other things. You can write your own cookbooks, but you can also use the many available commmunity cookbooks. The default go-to resource for these is supermarket.chef.io. You can reference these cookbooks in your Berksfile and Berkshelf will pull them in automatically.

My cookbooks

Here’s a list of the cookbooks I’m using currently:

bigwhoop-defaults
bigwhoop-git
bigwhoop-apache2
bigwhoop-mysql
bigwhoop-bitbucket
bigwhoop-pamoroth
bigwhoop-reinhard.codes

You can see that I have created a cookbook for every component that I want to have installed on my server. Additionally there’s the defaults cookbook and the chef-solo-repo cookbook that brings all of this together. The cookbooks/ folder in the chef-repo contains 25 cookbooks currently, most of which are transient community dependencies that Berkshelf pulls in automatically.

A cookbook example

attributes/
attributes/default.rb
recipes/
recipes/default.rb
recipes/test.rb
templates/
templates/auto-update.sh.erb
Berksfile
metadata.rb

The above shows the directory structure of the cookbook defining some default settings for my server. It has attribute definitions, two recipes, a bash script template, and meta information for Chef and Berkshelf

attributes/default.rb
In this file I define some attributes that can be used in recipes. It’s important to note that these attributes are set on the node (=server) that is being configured, so you can access them from other recipes, as long as they belong to the same run list. It’s good practices to use the cookbook name that defines the attributes as a namespace.

recipes/default.rb
This is the default recipe of the cookbook. If you don’t specify a specific recipe in your run list, the default recipe will be used.
Here I’m configuring that the packages “vim” and “ntp” should be installed. Also I’m creating the scripts directory and a log directory for cron jobs. Note that I’m using the attribute values defined in the cookbook’s attributes.
After that, the auto-update.sh.erb template is used to create the file auto-update.sh in the scripts directory and a cron job is created. It’s noteworthy, that this works because I have a dependency on the cron community cookbook defined in my cookbooks metadata.

recipes/test.rb
This test recipe is included in the run list that is used to setup the test environment. It is used to override values like the ip address of the server.

templates/auto-update.sh.erb
This is the template file that is used to create a script file in the default recipe.

Berksfile / metadata.rb
Again, these files define some meta data and dependencies for the cookbook.

Website deployment

The main thing I do with my server is website hosting. I have to cookbooks that deal with webserver and database configuration. While the generic bigwhoop-apache2 cookbook sets up the default webserver configuration, the website specifics are contained in these cookbooks. They also create the folders where the websites are served from and pull the project from their respective git repositories.

As an example, take the recipe for the website pamoroth.de:

directory "/var/www/pamoroth" do
  owner "www-data"
  group "www-data"
  mode 00755
end

template "/etc/apache2/sites-available/pamoroth.conf" do
  source "apache-site-config.erb"
  owner "root"
  group "root"
  mode 00755
end

link "/etc/apache2/sites-enabled/pamoroth.conf" do
  to "/etc/apache2/sites-available/pamoroth.conf"
  owner "root"
  group "root"
  action :create
  mode 00755
end

git "/var/www/pamoroth" do
  repository "git@bitbucket.org:(username)/pamoroth.de.git"
  action :sync
  user "www-data"
  ssh_wrapper node["bigwhoop-bitbucket"]["ssh-wrapper"]
end

template "/tmp/initalize-pamoroth-db.sql" do
  source "initialize-db.sql.erb"
  user "root"
  group "root"
  action :create
  mode 00755
end

execute "init-database" do
    command "mysql < /tmp/initalize-pamoroth-db.sql -S /var/run/mysql-" + node["name"] + "/mysqld.sock -u  -p" + node["bigwhoop-mysql"]["password"]
    only_if { File.exist?("/tmp/initalize-pamoroth-db.sql") }
end

The webserver configuration template apache-site-config.erb contains the virtual host definitions for this website. When Chef Solo runs, it creates the respective configuration in the sites-available/ folder of Apache and then creates a link to this file in sites-enabled/, in order to enable the configuration. After that, the website’s code is pulled from its repository at Bitbucket. The ssh_wrapper contains information regarding the authentication with Bitbucket. It’s defined in the corresponding cookbook, as you can see from the attribute namespace.

Then, finally, the database is initialized by creating the file initialize-db.sql in the tmp/ folder and then executing it. The sql file is written in an idempotent way, so you can run it multiple times. The state of the database will alway be the same after it has been run. It’s worth mentioning that I only create the database structure in this script. The content of the database is then populated manually.

What’s next?

I still have some functionality to add to the server so for the next week or two I will be busy with that. Afterwards I want to have a closer look at Berkshelf and see what I can improve in terms of dependency handling. Switching to Chef Zero and introducing Vagrant into my tooling will also be topics that I want to look at.

Anyway, even though what I have now is far from perfect, it’s quite neat to be able to set up a small server in less then 5 minutes.