# Build a Local Router with Nemotron Lightning

Prompt Engineering channel: why agentic systems need a **router**, NVIDIA Switchyard's four strategies, and a local hybrid of Nemotron 3.5 Lightning (DGX Spark) + Kimi K3 — companion demo to the Level1Techs Switchyard ingest.

Tags: youtube-inbox, source, nvidia, switchyard, nemotron, local-ai, model-routing

## Summary

Frontier-for-everything is expensive (API) and wasteful (subscriptions). A routing layer chooses capable vs efficient models for cost, speed, specialization, and privacy — each with different requirements. Switchyard (OSS on RouteLLM) is the plumbing: profiles, targets, traces. Lightning (~30B MoE, latent MoE, hybrid Mamba+transformer, 1M context, speculative decoding) is the fast local workhorse; Kimi K3 is the capable API tier.

## Key points

- **Why route (four reasons, different constraints)**: cost (enterprises "token maxing"); speed (hundreds of agent steps; GPT-5.6-on-Cerebras called out for latency); specialization (fine-tunes beat the best-average model); privacy (local for sensitive data, hybrid otherwise). OpenRouter-style single endpoint is *access* routing, not task routing.
- **Options**: proprietary (OpenRouter, "Dev Fusion"); OSS classifier RouteLLM; **NVIDIA Switchyard** — classifier + proxy, multiple strategies, cross-session conversation state, action traces. Built on RouteLLM.
- **Nemotron 3.5 Lightning**: ~30B MoE / ~30 experts; latent MoE (project tokens to smaller latent space then experts) — "better accuracy per byte"; hybrid Mamba + transformer to hold 1M context; speculative decoding (MTP; other decoder names in captions are ASR-uncertain). On **Artificial Analysis** index: on par with GPT-OSS at much higher throughput.
- **Local numbers (speaker, NVIDIA early access, DGX Spark)**: ~**71 tok/s** single stream; MTP accepting ~**70%** of guesses. 1400-token reasoning task: Lightning ~**20s** local vs Kimi K3 API ~**50s**.
- **Switchyard config primitives**: profile (stage vs escalation router); targets (`capable` vs `efficient` + provider, including local Spark). App points at the profile.
- **Four strategies (pick one per route)**: **random** (fixed ratio — A/B / baseline, $0); **LLM classifier** (extra model call every query — content → tier); **stage** (uses existing conversation signals: tool failures, whether edits land — mostly free); **escalation** (start cheap, judge watches, latch upward on trouble — one judge call until latch). **Don't route per turn** — breaks caching; route at task start and stick.
- **Cost story**: Opus 4.8 ~80% on a dataset at ~$180 vs Lightning+Switchyard similar performance at much lower cost (slide-level claim; no independent table in transcript).
- **Demos**: (1) two-stage all-local digest — network alerts stay on Lightning; PII (hostnames, IPs, account numbers) stripped; only combined JSON digest to Kimi K3 for root-cause. (2) per-alert escalation of hard cases to the bigger model.
- **Repo**: speaker says application codebase + Lightning/Switchyard links are in the YouTube description (not copied into transcript).

## Quotes / memorable lines

- "Models are just one piece of the puzzle… harnesses are becoming a lot more important."
- "The best average model is not good for everything."
- "If you're doing routing it's actually very important that you don't do it per turn because that's going to be extremely expensive. It's going to break the caching."

## Sources

- YouTube: https://www.youtube.com/watch?v=ofpqWw0OFKo
- Raw ingest: `raw/youtube-inbox/ofpqWw0OFKo/`
- Channel: [Prompt Engineering](https://www.youtube.com/@engineerprompt)
- Related NVIDIA/Switchyard trail: see [[youtube-switchyard-nemotron-lightning]]

## Related pages

- [[youtube-switchyard-nemotron-lightning]]
- [[hermes-agent]] (model routing / multi-backend)
- [[grok-bot]] (no model picker; managed router)