Changed the message role of ReAct observation to tool

[jamesljlster]

Description

This pull request changed the message role of ReAct observation to tool.

I am developing a chatbot with the ReAct agent. Sometimes the chatbot gives strange responses to the user. After observing with tracer, I believe the problem is related to the inappropriate message role of tool message (observation), making the ReAct agent chat itself:

After setting the observation message role to tool, the stability got improved:

Related pull request: #17273 (the above test was made after #17273 was merged)
Test model: https://ollama.com/library/qwen2.5
Tracer: https://docs.arize.com/phoenix/tracing/integrations-tracing/llamaindex
Test code:

from llama_index.core import Settings
from llama_index.core.agent import ReActAgent
from llama_index.core.base.llms.types import ChatMessage, MessageRole
from llama_index.core.bridge.pydantic import Field
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai_like import OpenAILike

from phoenix.otel import register
from openinference.instrumentation.llama_index import LlamaIndexInstrumentor


# Setup LLM
Settings.llm = OpenAILike(  # type: ignore
    api_base="http://localhost:11434/v1",
    api_key="dummy",
    model="qwen2.5",
    is_chat_model=True,
    timeout=300,
)


# Define tools
def get_weather(
    location: str = Field(
        description="A city name and state, formatted like '<name>, <state>'"
    ),
) -> str:
    """Useful for getting the weather for a given location."""
    return "Sunny"


class Greeter:
    """Useful for handling the general messages from users."""

    __name__ = "greeter"
    _memory: ChatMemoryBuffer

    def __init__(self, memory: ChatMemoryBuffer):
        self._memory = memory

    def __call__(
        self, message: str = Field(description="The message from the user.")
    ) -> str:
        response = Settings.llm.chat(
            [
                ChatMessage(
                    role=MessageRole.SYSTEM,
                    content=(
                        "You are a helpful assistant responsible for answering user "
                        "about the weather associated questions. You should focus on "
                        "your responsibility, don't answer the irrelevent questions."
                    ),
                ),
                *self._memory.get(),
                ChatMessage(role=MessageRole.USER, content=message),
            ]
        )

        assert response.message.content is not None
        return response.message.content


# Setup ReAct agent
memory = ChatMemoryBuffer.from_defaults()
agent = ReActAgent.from_tools(
    [
        FunctionTool.from_defaults(get_weather),
        FunctionTool.from_defaults(Greeter(memory)),
    ],
    llm=Settings.llm,
    memory=memory,
    verbose=True,
)

# Setup tracer
tracer_provider = register(
    project_name="ReAct Agent Experiment",
    endpoint="http://localhost:6006/v1/traces",
)
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

# Chat with the agent
while True:
    response = agent.chat(input("User Input: "))
    print(response)

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[logan-markewich]

This assumes that the LLM you are using supports a tool role though? If your LLM supports a tool role already, you should be using FunctionCallingAgent instead of react anyways? 🤔

[jamesljlster]

I'm not sure if a model supports tool role, it supports function/tool calling too. Things are a bit complicated in my situation. I use Llama 3.1 and 3.3 hosted by vLLM in my company. The both supports the tool role, but I can't make vLLM produces the correct tool calling responses. The problem might be related to the tool parser plugin, and I currently have no idea how to fix it.

Anyway, I think it's better to set the message role to the appropriate one. If legacy models don't support the tool role, should I make it configurable?

[logan-markewich]

@jamesljlster the vllm issue sounds like the real issue. Assuming vLLM is launched in openai-compatible server mode, it should be straightforward

pip install llama-index-llms-opeani-like

from llama_index.llms.openai_like import OpenAILike
from llama_index.core.agent import FunctionCallingAgent

llm = OpenAILike(model="some model", api_key="fake", api_base="http://localhost:800/v1", is_chat_model=True, is_function_calling_model=True)

agent = FunctionCallingAgent.from_tools(tools, llm=llm)