AI Infrastructure · Systems Engineering · LLM Inference
GATEWAYS · RUNTIMES · EVALUATION · OBSERVABILITY · DISTRIBUTED SYSTEMS
Building and debugging systems around LLM gateways, inference runtimes,
structured generation, evaluation, observability, and reliability.
|
PRODUCTION BIFROST Go · Gateway · Governance |
RESEARCH INFERENCE Sampling · Grammars · Decoding |
EVALUATION STRUCTTRACE Rust · Regression · Provenance |
OPEN SOURCE SYSTEMS Reliability · Compilers · Observability |
ENGINEERING SURFACE
Bifrost · Inference Runtimes · Constrained Decoding · StructTrace · OpenObserve01 / SELECTED SYSTEM
Regression infrastructure for structured LLM pipelines
Rust · Evaluation · Structured Outputs · CI
TEST CASES
↓
BASELINE + CANDIDATE
↓
VALIDITY · CORRECTNESS · PROVENANCE
↓
REGRESSION VERDICT
↓
CI RELEASE GATE
|
EXECUTION PAIRED baseline vs candidate |
EVIDENCE REPLAYABLE durable run artifacts |
TRACEABILITY PROVENANCE reproducible decisions |
DELIVERY CI GATES stable exit behavior |
Instead of treating valid output as equivalent to correct output,
StructTrace separates structural validity, task correctness, regression behavior, and execution evidence.
$ structtrace --explain verdict
A regression is not simply a lower aggregate score.
StructTrace keeps baseline and candidate outputs together per test case, allowing failures to be attributed to concrete transitions such as valid-and-correct becoming valid-but-wrong, or previously valid output becoming structurally invalid.
The CI gate evaluates those changes directly.
02 / INFERENCE LAB
What changes when a grammar touches the token distribution?
Qwen2.5 · XGrammar · Outlines · Sampling
MODEL LOGITS
↓
GRAMMAR MASK
↓
LEGAL TOKEN SET
↓
RENORMALIZED DISTRIBUTION
↓
SAMPLE
|
PROBABILITY MASS REMOVED which tokens disappear? |
SAMPLING NUCLEUS CHANGED does the top-p set move? |
BEHAVIOR SEMANTICS SHIFT does the answer itself change? |
Artifact-backed experiments studying the effect of grammar-constrained decoding
beyond the binary question of whether the final output validates.
Ran Qwen2.5-7B on GSM8K across Outlines and XGrammar backends.
Studying schema-withheld vs schema-in-prompt behavior,
temperature, nucleus sampling, probability redistribution, and semantic choice.
"Grammars are written in characters. Models emit tokens."
03 / SPECIFICATION SYSTEM
Turn specification text into executable evidence
TypeScript · RFCs · ABNF · Validator Conformance
RFC SPECIFICATION
↓
ABNF GRAMMAR
↓
GENERATED TEST CASES
↓
AJV · python-jsonschema · jsonschema-rs
↓
DIVERGENCE MATRIX
|
SOURCE RFC-DERIVED not hand-written examples |
EXECUTION MULTI-VALIDATOR same case, multiple engines |
EVIDENCE TRACEABLE failures point back to spec |
A deterministic CLI that parses RFC ABNF grammars, generates RFC-cited cases,
executes them across validators, and makes implementation disagreement reproducible.
04 / PRODUCTION ENGINEERING
Member of Technical Staff Intern
Go · AI Gateway · Governance · Reliability
REQUEST → AUTHENTICATE → APPLY POLICY → ROUTE → PROVIDER
|
AUTH VIRTUAL KEYS identity · expiry |
GOVERNANCE POLICY model restrictions |
ROUTING PROVIDERS multi-provider behavior |
OPERATIONS RELIABILITY storage · timeouts · logs |
Worked on Bifrost, Maxim's open-source AI gateway
unifying access to 20+ LLM providers through a single API.
My work touched the Go backend, governance, virtual-key policy, provider behavior,
configuration, request lifecycle, and embedded management UI.
05 / OPEN SOURCE
Reproduce first. Trace the path. Find the invariant. Change the minimum surface.
REPRODUCE → TRACE → ISOLATE → FIX → VERIFY
|
OPENOBSERVE Observability |
Scheduler drift · startup ordering · trace identity · search reliability |
|
BAML LLM Compiler |
Parser ambiguity · type resolution · TypeVar collisions · code generation |
|
SOURCEMETA JSON Schema · C++ |
Added forbid_empty_enum linting with automatic fixes and multi-draft coverage
|
|
SCHEMASTORE Schema Registry |
Schema lint and formatting consistency fixes |
|
CORRECTNESS INVARIANT What must remain true? |
RESILIENCE FAILURE What breaks partially? |
STATE AUTHORITY Where is truth stored? |
EXECUTION RETRY Is repetition safe? |
|
DISTRIBUTION ORDERING What arrives first? |
LIFECYCLE READINESS When is it actually usable? |
DEBUGGING EVIDENCE Can I reproduce it? |
OPERATIONS OBSERVE Can I explain the failure? |
The questions I care about most usually reduce to
state · failure · ordering · ownership · observability · invariants
|
LLM SYSTEMS INFERENCE RUNTIMES structured generation |
SYSTEMS DISTRIBUTED SYSTEMS concurrency |
LOWER LEVEL C++ · CUDA memory |
APPLICATION
↓
AGENT EXECUTION
↓
LLM GATEWAY
↓
INFERENCE RUNTIME
↓
SCHEDULING · KV CACHE · BATCHING · SAMPLING
↓
MODEL EXECUTION
↓
GPU KERNELS
↓
CUDA
|
DESKTOP AI
A cross-platform AI screen-teaching desktop companion built and shipped across product, backend, and distribution.
|
AGENT ORCHESTRATION
A local-first desktop system for orchestrating persistent AI coding workers across multiple execution backends.
|
09 / TOOLCHAIN
PRIMARY
Go · Rust · Python · TypeScript
SYSTEMS
Linux · Docker · PostgreSQL · SQLite · gRPC · WebSockets
GOING DEEPER
C++ · CUDA · inference internals
10 / DIRECTION
|
AI inference runtime infrastructure observability developer infrastructure |
LLM gateways evaluation systems distributed systems production reliability |
I want to work close to systems where
correctness, performance, reliability, and architecture actually matter.
UNDERSTAND THE FAILURE MODE.
MEASURE THE BEHAVIOR.