NVIDIA NeMo Switchyard

NVIDIA NeMo Switchyard

Author: langgenius
Version: 0.0.1
Type: model

Description

Routes each LLM call between a capable model and an efficient model using
NVIDIA NeMo Switchyard. The routing library
() runs inside the plugin. No separate Switchyard service is needed.

The plugin uses Switchyard's stage router, the same algorithm as its preset. It reads the
tool results in the conversation:

• No tool history, or weak signals: the efficient model answers.
• A critical error (for example out of memory), the same error repeated, or a context-compaction
  marker: the capable model answers.
• After an escalation, the capable model keeps the next 2 turns. A clean test pass ends this early.

The routing gives the most benefit to agent workloads with tool calls. Plain chat always goes to
the default tier.

Setup

Add a model under NVIDIA NeMo Switchyard in Dify. The model name is the route name shown in Dify, for
example . Each route has two upstreams. Each upstream is any endpoint that speaks one of
these APIs:

Saving the route sends a short request to both upstreams. An error names the tier that failed.

Behavior

• Fallback: if the selected upstream fails before it sends any output, the plugin tries the
  other tier.
• Streaming: Dify's flag is passed to the upstream. A non-streaming call reads the
  whole upstream response before returning, so any upstream failure falls back. Saving a route
  always checks both upstreams with a streaming request.
• Which model answered: the field of each result chunk is the upstream model that
  served it. Dify's run logs currently record the route name instead.
• Sessions: Dify does not pass a conversation ID to model plugins. The plugin derives a
  session key from the end user and the first user message. The capable hold uses this key. Two
  conversations by one user that start with the same message share routing state.
• State is per plugin process. Routing state lives in memory and is lost when the plugin
  restarts.
• Reasoning: streamed reasoning is shown in tags. Chat Completions upstreams stream
  it when the model returns . Responses upstreams stream it only when the
  tier's reasoning summary is on. On Chat Completions upstreams, reasoning from earlier
  assistant tool-call turns is sent back as .
• Reasoning in workflows: with the LLM node's default reasoning format, the
  `
  [ENDTHINKFLAG]