Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Options

• Trade-offs between hosted APIs and self-hosting.
• Patterns for on-premises, edge, and air-gapped environments.
• Single model versus model portfolio approaches.
• Comparison of cost per decision.

Hardware Sizing and Capacity

• Memory footprint based on parameter count.
• CPU-only versus GPU acceleration.
• Planning for concurrency and throughput.
• Allowing headroom for task length growth.

Serving Stacks

• Options including vLLM and SGLang.
• Execution via llama.cpp and GGUF.
• Using Ollama for local development.
• Selecting stacks based on decision workload characteristics.

Quantisation Without Losing Calibration

• Precision formats and associated memory savings.
• Measuring quality and calibration loss.
• Per-tensor and importance-aware quantisation variants.
• Conducting acceptance testing after quantisation.

Latency Engineering

• Distinguishing median versus tail latency.
• Latency growth relative to context length.
• Implementing warm starts and model caching.
• Batching multiple decision questions.

Edge and Constrained Environments

• Running small decision models on CPUs.
• Deployment on Apple Silicon and embedded systems.
• Offline operation and update strategies.
• Distributing weights in air-gapped settings.

Containerisation and Scaling

• Designing images for model serving.
• Managing volumes and caches.
• Implementing horizontal scaling and routing.
• Setting resource limits and sharing GPUs.

Observability

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Tracking decision outcomes and escalations.
• Alerting on calibration drift.

Versioning and Rollout

• Versioning models and adapters.
• Deploying via shadow and canary strategies.
• Maintaining pinned model files for audit purposes.
• Executing rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Managing network isolation and egress control.
• Maintaining audit logs of decisions.