ETS-Backed Registries: Sub-Millisecond Access in Elixir

The Problem with Database-Backed Registries

When your platform manages 552 AI agents, 157 OSINT tool adapters, and hundreds of

blog articles, every read matters. Querying PostgreSQL for metadata that rarely

changes introduces unnecessary latency and database load. The solution is

Erlang Term Storage (ETS), a built-in key-value store that lives in process

memory and supports concurrent reads without locking.

ETS Fundamentals for Registry Design

ETS tables are created by a process and destroyed when that process terminates.

For registries, we use :named_table and :public access so any process can

read without message passing:

:ets.new(:agent_registry, [
:set,
:named_table,
:public,
read_concurrency: true
])

The read_concurrency: true option optimizes for workloads where reads vastly

outnumber writes, which is exactly the pattern for registries.

Compile-Time Loading Pattern

The platform populates ETS tables at application startup by scanning the filesystem.

The agent registry, for example, walks the .aiad/agents/ directory and parses

each .agent.md file:

defp load_agents do
Path.wildcard("#{root}/.aiad/agents/*.agent.md")
|> Enum.map(&parse_agent_file/1)
|> Enum.each(fn agent ->
:ets.insert(:agent_registry, {agent.slug, agent})
end)
end

This pattern means the registry is always consistent with the filesystem. Adding

a new agent file automatically makes it discoverable at next startup.

The Ensure-Table Pattern

A critical pattern for ETS registries is idempotent table creation. Multiple

processes may attempt to access the registry before it is initialized:

def ensure_table do
case :ets.info(@table) do
:undefined ->
:ets.new(@table, [:set, :named_table, :public, read_concurrency: true])
load_data()
:ok
_ ->
:ok
end
end

Every public function calls ensure_table/0 before accessing data. This

eliminates race conditions during startup without requiring a supervision tree

dependency.

Query Patterns

ETS supports pattern matching through :ets.match_object/2 and match

specifications via :ets.select/2. For the OSINT tool registry, we filter

by category:

def tools_by_category(category) do
ensure_table()
:ets.select(@table, [
{{:"$1", :"$2"},
[{:==, {:map_get, :category, :"$2"}, category}],
[:"$2"]}
])
end

For simpler lookups, :ets.lookup/2 returns results in constant time:

def get_by_slug(slug) do
ensure_table()
case :ets.lookup(@table, slug) do
[{^slug, article}] -> {:ok, article}
[] -> {:error, :not_found}
end
end

Performance Characteristics

ETS reads are measured in microseconds, not milliseconds. In benchmarks on

the Prismatic Platform:

OperationLatencyNotes

|-----------|---------|-------|

Single lookup~0.5 usConstant time Full scan (552 agents)~45 usLinear but fast Pattern match (category filter)~12 usDepends on selectivity Write (single insert)~1 usConcurrent reads unaffected

Compare this to PostgreSQL queries that typically take 1-5ms even with connection

pooling. For metadata registries, ETS provides a 1000x improvement.

Table Lifecycle Management

ETS tables are owned by the process that creates them. If that process crashes,

the table is destroyed. The platform uses two strategies to handle this:

Supervisor-owned tables: A dedicated GenServer creates and owns the table.

The supervisor restarts it on crash, and ensure_table/0 repopulates data.

2. Heir tables: Using the :heir option, table ownership transfers to another

process on crash, preserving data across restarts.

For registries with cheap reload costs (filesystem scanning), strategy 1 is

simpler and preferred. For registries with expensive initialization, strategy 2

avoids data loss.

When Not to Use ETS

ETS is not a database replacement. It lacks transactions, persistence across

restarts, and distributed replication. Use ETS for:

Read-heavy metadata registries

Caching layers with known invalidation patterns

Counters and rate limiting

Lookup tables loaded at startup

Use PostgreSQL (via Ecto) for data that must survive restarts, requires ACID

transactions, or needs complex querying beyond key-value patterns.