Amtal

Erlang Project Power Levels

Erlang projects can take a wide range of forms, from the happy green meadows of putting everything in one directory, to the jagged peaks of abstraction provided by OTP’s release handling.

There’s little documented convention on which approach is best for what, so most newcomers to the language tend to start simple and gradually upgrade to more structured layouts as their code grows. It’s a non-trivial learning process, so I’d like to shape it a bit by describing a direction of growth.

Level 0: Monad

Everything is in one folder. Source code, compiled code, include headers, everything. Compilation is done by running erl -make, sometimes with an Emakefile for additional flags. Sometimes there’s a Makefile.

Dependencies are installed globally by hand: the project isn’t aware of them. (When everything you need came with OTP, this is fine.)

Running your app is trivial, since erl sees everything in the directory you start it.

Hot code reloading is refreshingly easy: compiling and loading a module is one shell command:
c(Module).

Level 0 is characterized by custom-built solutions on top of a trivial namespace. It works well when you’re just starting to write code, but rapidly starts to reinvent wheels. If you find yourself needing to write scripts to generate documentation or run unit tests, or adding path parameters to erl on startup, it’s time to level up to…

Level 1: Application

An application is a collection1 of related modules. It can be started and stopped, and has well known conventions about its structure.

Rather than drown in the manuals on the topic, just use rebar. It’s a script that understands the directory structure conventions, and has commands for compiling, unit testing, and typechecking your application. It also generates minimal application templates: try rebar create-app appid=foo.

Rebar’s behavior can be customized by placing a rebar.config file in application root. Documentation is almost non-existant, but the sample should get you started and there’s a wide variety of projects to use as examples. In the worst case, rebar is written in Erlang and the source is very clear.

The huge advantage is dependency management. The ‘deps’ setting in rebar.config lets you pull projects from git/hg/svn repositories, and rebar will take care of compiling them. This is a huge advantage over manually installing dependencies, espcially if you’re sharing your code with others.

For running your code, you’ll need to add dependency ebin folders to the code path; if starting from application root, erl -pa ebin deps/*/ebin should do. Hot code reloading is still easy, though the compile step must now be done in an OS terminal and is a little longer.

This level will get you through a lot – it has support for all kinds of unit testing, compiling C drivers, documentation generation, and static code analysis tools! If you’re working on a library, you won’t need to go further.

However, if you start seriously running whatever you wrote, you’ll see the number of parameters passed to erl grow. You’ll start to think about installing the right version of ERTS on remote machines. You’ll realize that sometimes, connecting a remote shell using ^G isn’t enough and you need access to the actual stdout console.

This is a sign of growth. Time for…

Level 2: Release

A release is a folder containing everything your applications needs to run2. You can tar it up, move it to another machine with the same architecture, and run it with zero dependencies!

The release rebar generates contains all your application dependencies, the Erlang runtime, and some support scripts. The start script boots up Erlang for you, with all the parameters you specify, and automatically starts your applications. It even takes care of detaching and attaching to stdout for you with run_erl – something you’ll appreciate when you decide to use rb.

Generating the release is done using ‘rebar generate’, after you create a node configuration. Rebar has a template for those, and you can look at projects like riak_search for examples. (Riak itself uses a technique I like, of separating the release generation from the applications. Note that the repo contains no Erlang code: it’s concerned purely with packaging!)

You can use releases for development as well, running your code with the same VM arguments as it will in production. The trick is to give -pa arguments to the release start script, such that your dev ebin folders come first on the path. Etorrent uses this trick, in addition to various others.

Once you’ve got rebar generating releases, using them to deploy and test code is pleasant and painless. They do, however, have a major downfall: how do you do hot code upgrades? Those things are one of Erlang’s big features, and while the development trick can be used it adds significant dependencies to what’s supposed to be a self-contained package.

Now, there’s no need for hot code upgrades if your system is sitting idle or not holding state: many projects can make do with full restarts. However, you can guess where this is going…

Level 3: Releases and Upgrades

Thus far, I haven’t talked about versions. To do hot code upgrades, a system needs to accurately track what version of code it’s running and what versions it can down or upgrade to.

The application versions included in a release all play well together – releases are stable code points, snapshots of your tangled GIT repository where everything worked and nothing was inconsistent.

A specific release has a unique version, and consists of a bunch of specific application versions. If you check its lib folder2, you’ll see them all in app-vsn format.

A specific application can know how to upgrade from, and downgrade to, some previous application versions. This is described in an .appup file3.

Two different releases, then, may differ only in one or two applications. If those applications know how to upgrade from the old ones, a “release upgrade” can be generated from their .appups. This release upgrade is a tar file containing the new application directories, and instructions on what modules to reload/processes to tweak.

Installing an upgrade is then a three step process, done from the shell of a currently running release:

  1. Unpack the upgrade. This places the new app-vsn application folders into the lib directory, and reads upgrade information.
  2. Install the upgrade, and spend some time confirming things work correctly.
  3. Once confident that everything’s okay, you can make the release permanent for future restarts.

The end result, after a release has had several upgrades applied to it, is the lib folder will contain multiple entries with different versions for some applications. You’ll be able to hop between them, by moving between different release versions: the version changes will be done live, based on the procedures defined in the .appup files.


Release upgrade generation is a bit tricky. Or, perhaps, it’s very simple – but so few people truly need it that it’s a bit obscure. Thus far we’ve been using rebar to wrap the boring details in a single command, but the rebar support for release-upgrade generation is rather new and not widely used. If you’re this far into a project, you probably have the experience to figure something out yourself while a user-friendly process for rebar-generated upgrades is worked out. The recent addition of support for using GIT tags to specify application version in rebar, in particular, should significantly change how rebar and releases are used.

If you want to play with rebar-generated upgrades, there is a promising tutorial on the topic, with some great “try as you go” examples. Otherwise, just keep in mind how the upgrades work and keep an eye out for new developments!

4 Summary

Erlang has support for a wide range of project types, but always works towards the high goal it was built for: managing a large complex system with minimal downtime and maximum safety.

While few projects are as massive as what Erlang was built to deal with, the techniques can be applied to anything – and the community is steadily making it easier and easier to put them into practice, by developing easy to use tools like rebar.

Soon, live release upgrades may become the norm on all standalone Erlang projects.

Appendix

Here are some examples of the different project types. I recommend getting a copy of the source, and exploring on your own.

1

A simple application with some embedded C code:

amtal@arrakis:~/code/erl$ git clone git://github.com/klaar/quoted.erl ; cd quoted.erl
Initialized empty Git repository in /home/amtal/code/erl/quoted.erl/.git/
Receiving objects:  22% (39/174)   
Receiving objects: 100% (174/174), 28.33 KiB, done.
Resolving deltas: 100% (69/69), done.
amtal@arrakis:~/code/erl/quoted.erl$ ls
c_src  LICENSE  README.md  rebar.config  src  test
amtal@arrakis:~/code/erl/quoted.erl$ rebar compile
==> quoted.erl (compile)
Compiled src/quoted.erl
Compiling c_src/quoted_nif.c
amtal@arrakis:~/code/erl/quoted.erl$ ls *
LICENSE  README.md  rebar.config
c_src:
quoted_nif.c  quoted_nif.o
ebin:
quoted.app  quoted.beam
priv:
quoted.so
src:
quoted.app.src  quoted.erl
test:
quoted_tests.erl
amtal@arrakis:~/code/erl/quoted.erl$ 

2

Riak is a good example of a complete, complex release. To play around with it, do:

git clone git://github.com/basho/riak.git
./rebar get-deps
./rebar compile
./rebar generate

You’ll see an extensive set of application dependencies fetched to deps/, and a standalone release bundle in rel/.

3

The biggest set of open source example upgrades, is the OTP standard library. Clone https://github.com/erlang/otp then look at different lib/*/ebin/*.appup files. Most of them are empty, check wc -l lib/*/ebin/*.appup for good ones.

Files of interest: