FORMYSECRMEP foundation systems lab

Research 05

LLM Performance Improvement And Self-Developed Inference Engine

Local LLM serving is treated as infrastructure: routing, cache reuse, prefill, streaming, and web delivery.

FieldLLM inference infrastructure and Web AI execution
View evidence Back to research index
Measured or implemented

What it is

This research is about developing the AI execution layer itself, not merely calling an external AI service.

How it is researched

The lab builds local serving paths, route choice, cache reuse, shared-prefix prefill, streaming behavior, and web delivery so the model runtime becomes controllable infrastructure.

Why it matters

It turns AI from a prompt-only tool into a developed system layer that can be connected to security modules, web products, and evidence workflows.

Evidence

  • Same-GGUF local serving: 487.92 tok/s sustained vs Ollama 72.16 tok/s, 6.76x.
  • Peak local serving: 639 tok/s, 8.86x vs the same Ollama baseline.
  • Shared-prefix prefill, 7B: 13,284.85 tok/s, 109.7x vs sequential prefill profile.
External reading

Difference

The AI story is development of an execution layer, not simply use of an external AI model.

Boundary

vLLM, TensorRT-LLM, SGLang, and other latest serving stacks need matched same-hardware comparisons.