Mnesia and Distillery

At Versus, we decided to move to Mnesia from a combination of ETS and GenServers, to more easily handle multiple nodes per environment. This article details some of the roadblocks we encountered during this transition and how we solved them.

Add Mnesia to extra applications

Although Mnesia can be started and used from an iex session when first experimenting with it, Distillery won't make the application available from a release without an explicit instruction. Adding Mnesia to the list of extra applications in our mix file met this requirement.

  def application do
      extra_applications: [:logger, :mnesia, :ecto_sql],

Mnesia directory must be a charlist

When configuring a directory for Mnesia in disc_copies mode (persisted to memory and disk), the config value must be a charlist. When configuring via mix, this might look like:

config :mnesia, :dir, 'mnesia/data'
when configuration is done via a Distillery config provider this can get a bit trickier as few of the underlying config languages have multiple string-like types that can be mapped differently. As we use json for runtime configuration, support was added to that library to turn special objects into charlists during processing.

Node name matters

To re-emphasize, being careful and consistent with the node names across a cluster is critical to Mnesia behaving as expected. Although this may seem fairly obvious, where we ran into a problem was when running migrations, as they operate with a small subset of applications running. To solve this we used one of the variables Distillery provides for custom commands, '$NAME', resulting in something like this:

#!/usr/bin/env bash
release_ctl eval --mfa "Elixir.SomeApplication.ReleaseCommands.migrate/1" $NAME
which allows the node to be started with the proper name in the migrate function:

  def migrate(node_name) do

    IO.puts("Starting dependencies...")

    |> Enum.each(&is_ok/1)

    IO.puts("Starting repos...")

    |> Application.get_env(:ecto_repos, [])
    |> &1.start_link([]))
    |> Enum.each(&is_ok/1)

    IO.puts("Running migrations...")

Our current load is consistent and predictable enough to make dynamically adding or removing nodes unnecessary. Nodes are added by hand as new partners come on board and require additional capacity. When setting up tools to make this easy, we ran into another issue with adding nodes to the Mnesia cluster. We had originally configured Mnesia to start on boot, but because that requires knowledge of the network topology not yet available, each node was coming up un-synced, and there is no way to add a disc_copy node to another once both have been established. To resolve this issue, we first use Distillery's 'console_clean' command to bootstrap additional nodes (all after the first). Only after all nodes have the proper schema saved in Mnesia do we start the application on the new node for the first time.

Additional notes

While working through these issues, many of the usual sources of information and debugging had little to offer. We found a lot solutions and advice in RabbitMQ forums. This post on setting up Mnesia with Erlang was also incredibly useful.