Amtal

Disclaimer: this post is only relevant to Erlang developers. If you aren’t one, you’re better off reading something else.

Adding Live Updates to Your Erlang Library

Often advertised, rarely used. Most uses are internal to large companies, as if it’s a hard thing to do.

Here’s a simple procedure to live-update-ize your Erlang application.

1.0 Motivation

People using your stuff can upgrade it live. This makes you look like a ninja.

Procedure ensures you’re following good versioning practice. (Few people do; you’ll stand out…)

Live updates work only on applications that follow good OTP practice. The capability is proof that your code is well designed and robust!

2.0 Stuff You Should Already Do

(And if you don’t, you’re a bad person and should feel bad.)

2.1 Version Control

GIT is the current community choice, so I’ll focus on it. Others should work.

2.2 Meaningful Application Versions

Regularly and frequently, set meaningful versions in your application’s ebin/<yourapp>.app file (src/<yourapp>.app.src with Rebar.) If you don’t already do this, start using a simple convention like Semantic Versioning. It takes 5 minutes to learn.

2.3 Proper OTP Structure

3.0 Live Update Overview

Erlang can keep up to two copies of code loaded at once. Each version of your application is packaged separately, and stored in a version-tagged folder. (Check a lib directory of a release or your local Erlang installation.)

External m:f(args) calls always go to the latest copy. When doing an upgrade, the release handler:

  1. Takes a system running an application with version A.
  2. Loads version B of that application. All external calls now go to new code.
  3. Stops or starts any removed or added supervision tree processes.
  4. Sometimes, transparently updates the state of running processes.

This is all done according to a simple set of instructions in the ebin/<myapp>.appup file.

{"v0.0.1",
  [{"v0.0.0", % upgrade instructions:
    [{load_module,  myapp_user_prot}    % pure function changes
    ,{add_module,   myapp_user_commands}% new module
    ,{remove_module,myapp_user_cmds}    % no longer used module
    % process restarts (when a live upgrade isn't worth the effort)
    ,{apply, {supervisor,terminate_child,[myapp_svc_sup,myapp_hash_cache]}}
    ,{apply, {supervisor,restart_child,  [myapp_svc_sup,myapp_hash_cache]}}
    % live upgrades (when a restart would lose useful state)
    ,{update,myapp_user_count,{advanced,["v0.0.0","v0.0.1"]}}
    ]}],
  [{"v0.0.0", % downgrade instructions:
    % trivial inverse of the upgrade
    [{load_module,  myapp_user_prot}
    ,{remove_module,myapp_user_commands}
    ,{add_module,   myapp_user_cmds}
    ,{apply, {supervisor,terminate_child,[myapp_svc_sup,myapp_hash_cache]}}
    ,{apply, {supervisor,restart_child,  [myapp_svc_sup,myapp_hash_cache]}}
    ,{update,myapp_user_count,{advanced,["v0.0.1","v0.0.0"]}}
    ]}]}.

For most updates, these instructions are all you need. Hell, most updates only use load_module.

3.1 Update Chain

Suppose that for every version of your application (2.1), you write an ebin/<myapp>.appup file. It contains two things:

  1. Upgrade instruction from the previous version, to this one.
  2. Downgrade instruction. (Trivial inverse of the upgrade.)

This means that any running version of your application, can be updated to any other version, through repeated updates.

More practically, it lets you do frequent upgrades on a running system, while retaining the capability to roll several versions back if needed. All live.

4.0 Method of Procedure

The first time you set an application version number (v0.0.0), do nothing.

Every time you bump the version number, do the following.

4.1 Version Branch

Branch your code, with the new version as the name.

From this point forward, the code must not change.

What may change are upgrade instructions. Any typos or mistakes discovered in the instructions, are easy to correct. 1

4.2 Relup File

Do a git diff --stat v0.0.1..v0.0.0 | grep .erl. For each module changed, use git diff to figure out whether the change was:

Once upgrade instructions have been written, copy and invert them into downgrade instructions.

5.0 Deployment

Let the people running your code worry about that. Relups are generated from adjacent releases, and manipulated via the release_handler module (and usually some in-house scripts to manage file distribution and logging) but those details don’t matter.

All that matters is your beautiful .relup files make live upgrades possible.

1 This is why a branch is used instead of a tag. Tags are still recommended for tagging specific distributions of a version, or for replacing old version branches whose upgrades have been tested to work.