Skip to main content
Version: 2.19

OpenAIChatGenerator

OpenAIChatGenerator enables chat completion using OpenAI’s large language models (LLMs).

Most common position in a pipelineAfter a ChatPromptBuilder
Mandatory init variables"api_key": An OpenAI API key. Can be set with OPENAI_API_KEY env var.
Mandatory run variables“messages”: A list of ChatMessage objects representing the chat
Output variables“replies”: A list of alternative replies of the LLM to the input chat
API referenceGenerators
GitHub linkhttps://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/openai.py

Overview​

OpenAIChatGenerator supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on).

OpenAIChatGenerator needs an OpenAI key to work. It uses an OPENAI_API_KEY environment variable by default. Otherwise, you can pass an API key at initialization with api_key:

python
generator = OpenAIChatGenerator(model="gpt-4o-mini")

Then, the component needs a list of ChatMessage objects to operate. ChatMessage is a data class that contains a message, a role (who generated the message, such as user, assistant, system, function), and optional metadata. See the usage section for an example.

You can pass any chat completion parameters valid for the openai.ChatCompletion.create method directly to OpenAIChatGenerator using the generation_kwargs parameter, both at initialization and to run() method. For more details on the parameters supported by the OpenAI API, refer to the OpenAI documentation.

OpenAIChatGenerator can support custom deployments of your OpenAI models through the api_base_url init parameter.

Structured Output​

OpenAIChatGenerator supports structured output generation, allowing you to receive responses in a predictable format. You can use Pydantic models or JSON schemas to define the structure of the output through the response_format parameter in generation_kwargs.

This is useful when you need to extract structured data from text or generate responses that match a specific format.

python
from pydantic import BaseModel
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

class NobelPrizeInfo(BaseModel):
    recipient_name: str
    award_year: int
    category: str
    achievement_description: str
    nationality: str

client = OpenAIChatGenerator(
    model="gpt-4o-2024-08-06",
    generation_kwargs={"response_format": NobelPrizeInfo}
)

response = client.run(messages=[
    ChatMessage.from_user(
        "In 2021, American scientist David Julius received the Nobel Prize in"
        " Physiology or Medicine for his groundbreaking discoveries on how the human body"
        " senses temperature and touch."
    )
])
print(response["replies"][0].text)

Model Compatibility and Limitations
  • Pydantic models and JSON schemas are supported for latest models starting from gpt-4o-2024-08-06.
  • Older models only support basic JSON mode through {"type": "json_object"}. For details, see OpenAI JSON mode documentation.
  • Streaming limitation: When using streaming with structured outputs, you must provide a JSON schema instead of a Pydantic model for response_format.
  • For complete information, check the OpenAI Structured Outputs documentation.

Streaming​

You can stream output as it’s generated. Pass a callback to streaming_callback. Use the built-in print_streaming_chunk to print text tokens and tool events (tool calls and tool results).

python
from haystack.components.generators.utils import print_streaming_chunk

## Configure any `Generator` or `ChatGenerator` with a streaming callback
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)

## If this is a `ChatGenerator`, pass a list of messages:
## from haystack.dataclasses import ChatMessage
## component.run([ChatMessage.from_user("Your question here")])

## If this is a (non-chat) `Generator`, pass a prompt:
## component.run({"prompt": "Your prompt here"})
note

Streaming works only with a single response. If a provider supports multiple candidates, set n=1.

See our Streaming Support docs to learn more how StreamingChunk works and how to write a custom callback.

Give preference to print_streaming_chunk by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.

Usage​

On its own​

Basic usage:

python
from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator

client = OpenAIChatGenerator()
response = client.run(
	  [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
)
print(response)

With streaming:

python
from haystack.dataclasses import ChatMessage
from haystack.components.generators.chat import OpenAIChatGenerator

client = OpenAIChatGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))
response = client.run(
	  [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
)
print(response)

With multimodal inputs:

python
from haystack.dataclasses import ChatMessage, ImageContent
from haystack.components.generators.chat import OpenAIChatGenerator

llm = OpenAIChatGenerator(model="gpt-4o-mini")

image = ImageContent.from_file_path("apple.jpg", detail="low")
user_message = ChatMessage.from_user(content_parts=[
	"What does the image show? Max 5 words.",
	image
	])

response = llm.run([user_message])["replies"][0].text
print(response)

In a Pipeline​

python
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack import Pipeline
from haystack.utils import Secret

## no parameter init, we don't use any runtime template variables
prompt_builder = ChatPromptBuilder()
llm = OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY"), model="gpt-4o-mini")

pipe = Pipeline()
pipe.add_component("prompt_builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("prompt_builder.prompt", "llm.messages")
location = "Berlin"
messages = [ChatMessage.from_system("Always respond in German even if some input data is in other languages."),
            ChatMessage.from_user("Tell me about {{location}}")]
pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}})

Additional References​

📓 Tutorial: Building a Chat Application with Function Calling

:cook: Cookbook: Function Calling with OpenAIChatGenerator