We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
OSINT at Scale: 157 Self-Registering Intelligence Adapters
How metaprogramming and ETS power a dynamic intelligence toolkit
Prismatic Engineering
Prismatic Platform
The Challenge of Intelligence at Scale
Modern open-source intelligence gathering requires querying dozens of data sources
simultaneously. Business registries, sanctions lists, domain WHOIS databases, social
media platforms, and government records each have their own APIs, authentication
requirements, and data formats. Building and maintaining individual integrations
is unsustainable as the number of sources grows.
The Prismatic OSINT framework solves this with a self-registering adapter architecture.
Each adapter is a standalone Elixir module that declares its capabilities at compile
time and automatically appears in the platform's UI and API.
Self-Registration via @after_compile
Every OSINT adapter uses the PrismaticOsintSources.Adapter behaviour, which injects
an @after_compile hook into the module. When the module compiles, it registers itself
in an ETS-backed registry with its metadata:
defmodule PrismaticOsintSources.Adapters.CzechAres do
use PrismaticOsintSources.Adapter,
name: "Czech ARES",
slug: "czech-ares",
category: :czech,
description: "Czech Administrative Registry of Economic Subjects",
auth_required: false,
input_fields: [
%{name: :query, type: :text, label: "Company name or ICO", required: true}
]
@impl true
def search(query, opts) do
# Implementation fetches from ARES XML API
end
end
The use macro injects the @after_compile callback that calls
PrismaticOsintSources.Registry.register/2 with the module and its metadata.
No manual registration step is needed.
Adapter Categories
The 157 adapters are organized into six categories, each serving a distinct
intelligence domain:
|----------|-------|---------|
The search/2 and run/2 Interfaces
Each adapter implements two primary functions. The search/2 function performs
a quick lookup and returns structured results suitable for entity matching.
The run/2 function executes a full investigation with detailed output
including raw data, parsed entities, and confidence scores.
The distinction matters for performance. During entity resolution, search/2 is
called across dozens of adapters concurrently using Task.async_stream/3 with a
configurable concurrency limit. The run/2 function is reserved for deep dives
on confirmed entities where the full data payload is needed.
ETS Registry for Sub-Millisecond Access
The adapter registry uses a named ETS table with read_concurrency: true for
fast lookups. At runtime, the UI queries the registry to render the tool catalog,
generate dynamic forms based on each adapter's input_fields, and display
execution results. Because ETS lookups are O(1), the entire catalog renders
in under 1ms regardless of how many adapters are registered.
Dynamic UI Generation
The LiveView toolbox at /hub/osint/tools reads adapter metadata from the
registry and generates the interface dynamically. Each adapter's input_fields
configuration drives form rendering: text inputs, email fields, domain selectors,
and IP address inputs are all generated from the adapter's declared schema.
When a new adapter is compiled, it appears in the UI automatically on the next
page load without any template changes.
Lessons Learned
Three key insights emerged from building this system. First, compile-time
registration eliminates an entire class of wiring bugs. Second, ETS provides
the performance characteristics needed for real-time UI rendering. Third,
standardizing the adapter interface across 157 modules enabled bulk operations
like concurrent search and unified result formatting that would be impossible
with ad-hoc integrations.