Get in Touch
 Duration 35 hours

Course Outline

Introduction and Diagnostic Foundations

  • An overview of failure modes in LLM systems and common Ollama-specific challenges
  • Establishing reproducible experiments and controlled environments
  • The debugging toolkit: local logs, request/response captures, and sandboxing

Reproducing and Isolating Failures

  • Techniques for creating minimal failing examples and seeds
  • Distinguishing stateful vs stateless interactions to isolate context-related bugs
  • Managing determinism, randomness, and controlling nondeterministic behaviour

Behavioural Evaluation and Metrics

  • Quantitative metrics: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies
  • Qualitative evaluations: human-in-the-loop scoring and rubric design
  • Task-specific fidelity checks and acceptance criteria

Automated Testing and Regression

  • Unit tests for prompts and components, along with scenario and end-to-end tests
  • Developing regression suites and golden example baselines
  • Integrating with CI/CD for Ollama model updates and automated validation gates

Observability and Monitoring

  • Structured logging, distributed traces, and correlation IDs
  • Key operational metrics: latency, token usage, error rates, and quality signals
  • Alerting, dashboards, and SLIs/SLOs for model-backed services

Advanced Root Cause Analysis

  • Tracing through graphed prompts, tool calls, and multi-turn flows
  • Comparative A/B diagnosis and ablation studies
  • Data provenance, dataset debugging, and addressing dataset-induced failures

Safety, Robustness, and Remediation Strategies

  • Mitigations: filtering, grounding, retrieval augmentation, and prompt scaffolding
  • Rollback, canary, and phased rollout patterns for model updates
  • Post-mortems, lessons learned, and continuous improvement loops

Summary and Next Steps

Requirements

  • Substantial experience in building and deploying LLM applications.
  • A working knowledge of Ollama workflows and model hosting.
  • Proficiency with Python, Docker, and foundational observability tools.

Target Audience

  • AI Engineers
  • ML Ops Professionals
  • QA Teams responsible for maintaining production LLM systems

Number of participants


Price per participant

Provisional Upcoming Courses (Require 5+ participants)

Related Categories