OpenRouterChatGenerator

This component enables chat completion with any model hosted on OpenRouter.

Most common position in a pipeline	After a ChatPromptBuilder
Mandatory init variables	“api_key”: An OpenRouter API key. Can be set with OPENROUTER_API_KEY env variable or passed to init() method.
Mandatory run variables	“messages:” A list of ChatMessage objects
Output variables	“replies”: A list of ChatMessage objects
API reference	OpenRouter
GitHub link	https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openrouter

Overview

The OpenRouterChatGenerator enables you to use models from multiple providers (such as openai/gpt-4o, anthropic/claude-3.5-sonnet, and others) by making chat completion calls to the OpenRouter API.

This generator also supports OpenRouter-specific features such as:

• Provider routing and model fallback that are configurable with the generation_kwargs parameter during initialization or runtime.
• Custom HTTP headers that can be supplied using the extra_headers parameter.

This component uses the same ChatMessage format as other Haystack Chat Generators for structured input and output. For more information, see the ChatMessage documentation.

Tool Support

OpenRouterChatGenerator supports function calling through the tools parameter, which accepts flexible tool configurations:

• A list of Tool objects: Pass individual tools as a list
• A single Toolset: Pass an entire Toolset directly
• Mixed Tools and Toolsets: Combine multiple Toolsets with standalone tools in a single list

This allows you to organize related tools into logical groups while also including standalone tools as needed.

python

from haystack.tools import Tool, Toolset
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator

# Create individual tools
weather_tool = Tool(name="weather", description="Get weather info", ...)
news_tool = Tool(name="news", description="Get latest news", ...)

# Group related tools into a toolset
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])

# Pass mixed tools and toolsets to the generator
generator = OpenRouterChatGenerator(
    tools=[math_toolset, weather_tool, news_tool]  # Mix of Toolset and Tool objects
)

For more details on working with tools, see the Tool and Toolset documentation.

Initialization

To use this integration, you must have an active OpenRouter subscription with sufficient credits and an API key. You can provide it with the OPENROUTER_API_KEY environment variable or by using a Secret.

Then, install the openrouter-haystack integration:

shell

pip install openrouter-haystack

Streaming

OpenRouterChatGenerator supports streaming responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the streaming_callback parameter during initialization.

Usage

On its own

python

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator

client = OpenRouterChatGenerator()
response = client.run(
    [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]
)
print(response["replies"][0].text)

With streaming and model routing:

python

from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator

client = OpenRouterChatGenerator(model="openrouter/auto",
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))

response = client.run(
    [ChatMessage.from_user("What are Agentic Pipelines? Be brief.")]
    )

## check the model used for the response
print("\n\n Model used: ", response["replies"][0].meta["model"])

In a pipeline

python

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator

prompt_builder = ChatPromptBuilder()
llm = OpenRouterChatGenerator(model="openai/gpt-4o-mini")

pipe = Pipeline()
pipe.add_component("builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("builder.prompt", "llm.messages")

messages = [
    ChatMessage.from_system("Give brief answers."),
    ChatMessage.from_user("Tell me about {{city}}")
]

response = pipe.run(
    data={"builder": {"template": messages,
                      "template_variables": {"city": "Berlin"}}}
)
print(response)