Distributing a Symbolic NLP Engine: Brainhat as a Knowledge Service

This one is personal. My father has been writing a program called Brainhat since 1990. It is a symbolic natural-language engine — you type English at it, it turns the English into a graph of concepts, and it can reason over that graph well enough to answer questions, resolve pronouns, and fire rules you taught it a few sentences earlier. No neural network, no training run, no GPU. One C binary and a dictionary, in continuous development for thirty-six years and still moving: the release I worked from is dated this summer.

He wrote a whitepaper this year proposing that the engine could run as a distributed service: stateless engine replicas that share every conversation, plus long-lived "knowledge nodes" that hold different facts and can hand what they know to each other. I offered to find out whether the engine would actually do that. This post is what I learned. The short version is that the design works, the engine did most of what the whitepaper asked of it straight out of the tarball, and the places where it pushed back are the ones a security engineer looks at first, which is why they get most of the words below.

What I had to work with

A release tarball with a Linux binary and its data files, and no source, on purpose. I wanted to meet the engine the way anyone deploying it would, from the outside, so for everything below it is a black box. His development notes were still being written when I started; they are on their way, and a second pass with them in hand is part of the plan.

The engine has three properties the whole design leans on. It can run as a daemon (brainhat -d) that forks a fresh child per TCP connection, so every connection starts from the same clean state. It can save and restore its entire context — every concept, every rule, the discourse so far — as a small binary image. And it has a debug console, reachable mid-conversation by typing break, with two commands that matter: receive PORT, which listens for an image and merges it into the running process, and send IP:PORT, which streams the process's context out to whoever is listening. Those two commands were built for engines talking to each other. I used them for everything.

The design

Five containers on one private Docker network. A small Python service I called the shim (about six hundred lines, standard library only) is the only thing with a port on the host, and that port is bound to loopback. Behind it sit two engine replicas and two knowledge nodes, all running the unmodified binary as an unprivileged user.

you ──http 127.0.0.1:8080──> shim (Python)
                               ├── engine-a   brainhat -d +repeat            } stateless replicas
                               ├── engine-b   brainhat -d +repeat            }
                               ├── node-a     brainhat -d +repeat -i a.init  } long-lived
                               └── node-b     brainhat -d +repeat -i b.init  } knowledge nodes

The engine replicas remember nothing between turns. A conversation's memory is a context image, one to five kilobytes, that the shim keeps on disk. For every turn the shim picks the next engine round-robin, connects to the daemon (which forks a child), pushes the image into that child through the console, sends the user's sentence, pulls the updated image back out, and closes. One turn on the wire looks like this:

>> break
debug> receive 10599            <- shim streams the session image in; "Context received"
debug> exit
>> who is near her?             <- the actual turn
>> break
debug> send 172.18.0.6:41234    <- child streams its context back to the shim
debug> exit

Turn one lands on engine-a, turn two lands on engine-b, and engine-b resolves "her" from a sentence it never saw, because the referent came in with the image. A rule taught on engine-a fires on engine-b. Stop engine-a in the middle of a conversation and the conversation continues on engine-b without noticing; start it again and it takes the next turn. That was the whitepaper's first exit criterion, and it passes from a cold start.

Knowledge nodes are the same daemon started with an init file of plain-English facts and held on one long-lived connection each. A transfer between nodes is the engine's own peer mechanism, driven from outside: the receiver runs receive 10000, the sender runs send <receiver-ip>:10000. Afterwards node-b answers a fact that was only ever taught to node-a, and still answers its own, because a receive is a merge rather than a replacement. That was the second exit criterion I was after, and it passes too.

Why I ran the whole thing as a black box

The daemon is a network service written in C that parses untrusted text. That alone decides how to treat it, and it has nothing to do with whose code it is: I would put the same walls around anything I cannot audit, including my own. Every engine port — 4144 for the text daemon, 4145 for its web interface, 10000 and the 10500 range for context transfer — stays on the compose network. The only published port is the shim's, on 127.0.0.1, and one of the test suites does nothing except assert that this is still true. The engine ships with a ChatGPT integration flag and a credentials file; both are left alone, and the daemon never gets the flag. None of this is paranoia, it is just the default posture for any binary I cannot audit, and it cost nothing.

Running as root turned out to be wrong for a reason I did not expect. As root the daemon does the responsible thing and drops each child into a chroot jail as uid 65534 — and the jail has no /tmp, so every context transfer fails on its temp file. As an ordinary user the chroot is refused and everything works. The engine is doing the right thing on both counts; the jail and the console transfer were simply written years apart and had never been run together, and someone else's deployment is usually where that first happens. It is one line in the list I sent him, and probably one line in his notes.

Where the engine pushed back

Everything below has a reproduction in the repository and went to my father as a numbered list with a question attached to each. None of it is a verdict. Each one is either a corner I hit from outside without the documentation, or a behaviour whose intent I cannot know until I have his notes. The ones I think are interesting:

• Restoring from disk and restoring into a live process differ. The restore flag segfaults on an image that contains a fired inference, before it reads a byte of input, while the same bytes pushed into a live child through the console restore fine, inference and all. Whether the disk path is meant to re-run inference on load is his call to make. The service uses the console path, which is the one he built for engines talking to each other in the first place.
• Every child writes its saveset to the same temp file. The path is /tmp/bh%x.mbin, and the hex is not the PID. Watching /tmp during ninety-six concurrent transfers, one name appeared a hundred and one times. It behaves like rand() seeded once and inherited across fork(). Two conversations transferring at the same moment truncate each other's file and lose facts. Inside a single-user container that shows up as lost context under load, and the shim works around it by serializing transfers per daemon. On a shared multi-user host a predictable name in a world-writable directory is the classic symlink pattern, which is the reflex that made me look at the file names at all. A mkstemp() name would retire the workaround, and it is the first item on the list.
• A newline is not a sentence boundary. An init file with one fact per line teaches the engine the first line only. End every statement with a period and it learns all of them, which is consistent with how the engine reads everything else. I found this when twenty-four nodes each recalled exactly one of their five facts, and the fix was on my side.
• Merging has a ceiling I have not mapped. A node that receives context from many other nodes gives out on the sixteenth merge if the contexts contain rules, and with plain facts it carries on but keeps only the most recent merges. Eight merges of five facts each are fine. Whether that is a size, the cost of rules re-firing on every receive, or something his notes explain in a sentence is the question I most want answered, because the answer decides how the service should split knowledge across nodes.
• Small things. The console's send wants a dotted IP and rejects host names even where DNS works. A rule whose consequence is "scared" never fires, while the same rule with "tired" or "wet" does. The engine speaks synonyms back at you: tell it about the refrigerator and it will answer about the fridge, which is correct and still broke my test assertions until I let any dictionary label through.

Twenty-four nodes, a hundred and twenty secrets

The exit criteria are two engines and two nodes, which proves the mechanism and nothing about scale. So the last scenario generates twenty-four more node containers, each told five things of five kinds — a location, a nearness, a color, a possession, and a rule — with every noun drawn from the engine's own dictionary so that each word belongs to exactly one node. Then every node is asked its own five, its neighbor's five, and one question is fanned out to all twenty-six nodes at once.

• Recall: 119 of 120. The one miss is the "scared" rule above.
• Isolation: 120 questions about a neighbor's secrets, 120 answers of "I do not know." No leaks.
• Discovery: all 24 fan-out questions named exactly the owning node, in about 65 ms across 26 engines.
• Gathering: pulling all 24 contexts into one hub node is where the merge ceiling shows up. Independent nodes with distinct knowledge work at this scale; one node that knows everything does not, yet.

What it costs

Numbers from one run of the benchmark suite on the live stack, on a 28-core machine. Engine-side is what the shim measured around the engine; client-side includes about 20 ms of HTTP and interpreter start-up.

For a symbolic engine that is doing real parsing and inference on every turn, on one core, with a full context restore and capture on either side, I find those numbers hard to complain about. Every load level kept every conversation's context intact. The thing limiting throughput is my lock, not the engine.

Testing it like a product

The repository ends up with eleven suites that run against the live stack from one script: 124 unit tests for the shim, the two exit criteria, session isolation, a sixteen-turn conversation alternating engines, twelve concurrent sessions, an engine killed mid-conversation, a node restarted, the benchmark, the secrets scenario at eight nodes, and the topology check that only loopback is published. About two minutes end to end, one line per suite. The whole thing runs on any Linux with Docker and the Compose plugin; there is a preflight script that checks the machine and says what to install, because the person I most want to run this uses a plain Linux desktop and has better things to do than debug my container setup.

That matters more than it sounds. A proof of concept that only runs on the laptop it was built on has proved a concept to one person. One that a skeptical author can unpack, run, and break in an afternoon has proved it to the person whose opinion decides what happens next.

Who actually wrote this

Worth being clear about, because it changes how much weight the findings deserve. The engine, the ideas in it, and thirty-six years of C are my father's. I am not a C programmer anywhere near his level, and I did not touch a line of the engine; the source was not part of this exercise, and it would not have been mine to change if it were.

The shim, the transport experiments, the 124 unit tests, the eleven suites, the generator for the secrets scenario, and most of the documentation were built with an AI coding agent — Claude Code — working under my direction. I set the design from the whitepaper, the security posture, and what counted as passing; the agent wrote and ran the code, and every finding above came out of a reproduction it ran and I checked, usually more than once, because the first explanation for a failure was wrong about half the time. The judgement in this post is mine. The typing largely was not. A reader deciding how far to trust the numbers should know that, and I would rather say it plainly than let the repository imply I wrote six hundred lines of Python and twelve spikes over a weekend by hand.

Where this goes next

What went back to him is twelve findings with reproductions and nine questions, three of which would change the design if answered. Is the disk restore meant to re-run inference on load, or is the console receive the intended path? Would a mkstemp() saveset name be acceptable, since it is the only reason the service serializes transfers? And is there a bound on merged context, even a rough one, so that knowledge can be split across nodes deliberately? This is the first round of a collaboration, not a report card. He has promised his development notes, and the second pass is already shaped by them: revisit every finding with the intent known, retire the workarounds that were guesses, and see which of the corners above have already moved in a newer build, because the engine is not standing still. The whitepaper's "motives" layer waits on the map format, which the notes should cover, and the LLM front end comes after that. Two phases pass cold today; the plan is four, built together rather than reverse-engineered.

Addendum, 2026-09-08: his notes arrived the next day. What changed because of them, with the diagrams.