Directory Based Configuration

May 13, 2010

While working on POM, a project metadata tool, I kept returning to an unusual question: why put configuration into a file when the filesystem already gives us named values and a hierarchy?

The attraction is easiest to explain with one command:

printf '%s\n' '10' > config/max

There. The setting is changed. No configuration editor, no parser, no load-modify-serialize cycle. Any tool that can write a text file can update that setting without understanding the rest of the configuration.

That is the big advantage of directory-based configuration. The interesting question is how far we can take it before the conventions we add become more troublesome than the file format we removed.

A configuration you can walk through

Consider a small YAML document:

name: POM
summary: Project Object Model for Ruby
authors:
  - Trans
sites:
  homepage: https://example.org/pom
  development: https://example.org/pom/source

We could represent the same information with this directory tree:

pom/
  name
  summary
  authors/
    0001
  sites/
    homepage
    development

The name file contains POM, the summary file contains the description, and authors/0001 contains Trans. The two files under sites contain the URLs. The application knows that authors is an ordered list and sites is a mapping. That distinction is part of our configuration convention; the filesystem doesn’t supply it for us.

Now the data is accessible through ordinary tools:

cat pom/name
cat pom/sites/homepage
printf '%s\n' 'A different project description' > pom/summary

A shell script can change the description without knowing anything about the authors or sites. A Ruby program can read just the homepage. A tool can create a new setting by creating a file. The files remain readable by hand, and updating one value leaves the other files alone.

For simple settings, that is a very low barrier to entry.

What becomes easier

The strongest benefit is selective access. A consumer need only know the path of the setting it wants. It doesn’t have to understand the whole document or even have a library for its format.

Selective writes are just as useful. A program can update pom/summary without rewriting the surrounding configuration. With a YAML file, the usual parse-and-serialize approach can change comments, quoting, or layout unless the editing tool preserves those details. Directory storage keeps an unrelated setting out of the write operation altogether.

The filesystem also provides familiar building blocks: copying, moving, diffing, searching, and permissions. We can give a private setting different permissions from a public one, subject to the directory permissions needed to reach it. With one ordinary YAML file, read access generally exposes the whole document.

These advantages matter most when several small tools need to inspect or change a few independent values. The format fits the way those tools already work.

What still needs a convention

A directory removes the document parser. It doesn’t remove the need to interpret data.

Does 10 mean an integer or a string? Is an empty file an empty string, a missing value, or an instruction to use a default? Is the final newline part of the value? How do we distinguish a list from a mapping, represent an empty collection, or store a key containing a slash?

For a small, fixed set of settings, the application can answer these questions directly: max is an integer, summary is text, and the final newline is a storage convention. That may be all we need. For a general configuration format, those rules must be specified and shared by every reader and writer.

YAML already provides a model for mappings, sequences, and scalar values, with rules for resolving types. It still takes application validation to establish that max must be a positive integer, for example. A directory layout needs both the application rules and a representation convention. The YAML specification describes that data model.

The more elaborate our values become, the more attractive an established serialization format becomes.

A write is not a transaction

The simplicity of printf ... > config/max is real. So is its behavior: shell redirection truncates the file before writing the new value. Another process could read it in between those operations. Writing a YAML file in place has the same problem; YAML doesn’t make file writes safe.

For a single file, one common solution is to write a temporary replacement in the same directory and rename it over the destination. On a filesystem with the usual POSIX rename guarantees, that replacement is atomic: a reader opening the path sees the old file or the new one. This works for one directory setting just as it does for a whole YAML document. Linux’s rename documentation explains the replacement behavior. Crash durability requires additional care; atomic replacement alone doesn’t establish it.

The distinction appears when settings must change together. Suppose a server address and its credentials form a pair. Two separate file replacements can expose a mixture of old and new values. One replacement YAML file can publish both together, provided readers open it once and writers don’t modify that published version in place.

A directory system can coordinate updates with locks, or publish immutable generations and have each reader select one generation before reading. Those are workable designs, but they add a protocol. Neither representation prevents competing writers from overwriting one another’s changes without coordination.

A small CRUD utility could make these operations more convenient: read a setting, validate a new value, replace its file atomically, or remove it. The files would remain accessible to ordinary tools. For changes spanning several settings, the utility could also manage locking or generations, provided participating readers and writers follow the same protocol. Direct shell writes would remain useful for simple edits, with their simpler guarantees.

For an occasionally edited project description, that machinery may be unnecessary. For a live service with interdependent settings, it matters.

Space, tools, and the editing experience

Many tiny files require directory entries and file metadata, and reading several settings means opening several files. Allocated space depends on the filesystem and its features; there is no universal rule that every one-byte file consumes exactly four kilobytes. Some filesystems support small inline contents, including ext4 inline data. The cost should be measured on the intended system rather than waved away or treated as an automatic disqualification.

There is a human cost too. One YAML document makes it easy to review a group of settings in context, add explanatory comments, and copy a complete configuration. A directory makes individual settings easy to manipulate, but inspecting the whole can mean opening many files. The numbered author entries in our example already show how a tidy list can become awkward.

Git can track either representation. Separate files may reduce conflicts between unrelated settings, but a collection spread across many files can also make a change harder to review. Neither layout automatically wins.

Which work should the format make easy?

For a modest set of independent text values shared among shell scripts and small programs, I find the directory approach compelling. Being able to change a setting with an ordinary write is a practical advantage, not just a curiosity.

For nested configuration that people normally edit as a whole, or settings that must be published as one coherent version, a YAML document is often the simpler choice. A hybrid can make sense too: separate files for independent settings or secrets, and a structured document for values that belong together.

What I like about directory-based configuration is that it makes a small operation stay small. Whether it makes the whole system simpler depends on how often the rest of the configuration needs to come along for the ride.

← Selected writing · Older writing