中国科大认知全重实验室发布Agent-R1训练框架,支持自主思考与工具调用!

发布于 2025-4-9 06:29
浏览
0收藏

什么是 Agent-R1

中国科大认知全重实验室发布Agent-R1训练框架,支持自主思考与工具调用!-AI.x社区

Agent-R1是由中科大认知智能全国重点实验室开发的智能体强化学习训练框架,致力于推进强化学习智能体技术的融合发展。框架采用端到端强化学习方法,突破了依赖人工设计工作流的传统智能体开发瓶颈,让AI直接从与环境的交互中学习最优策略,实现自主决策与行动。开发者只需定义特定领域的工具和奖励函数,即可将Agent-R1扩展到各种应用场景,无需编写复杂的工作流程。

背景

中国科大认知全重实验室发布Agent-R1训练框架,支持自主思考与工具调用!-AI.x社区

随着大型语言模型(LLM)技术的快速发展,智能体(Agent)已成为人工智能领域的热门话题。然而,当前市场上大多数所谓的"智能体"实际上只是工作流系统——通过预定义的代码路径来编排LLM和工具的使用,而非真正的自主智能。正如Anthropic最新定义的那样,真正的LLM智能体应当是"系统中的LLM能够动态指导自己的处理过程和工具使用,保持对任务完成方式的控制权"。

传统工作流系统在实际应用中面临着严重的局限性:

• 无法规划,往往陷入死胡同

• 记忆能力有限,难以维持超过5-10分钟的任务

• 长期行动效率低下,行动序列常因累积误差而失败

这些痛点揭示了AI发展的"苦涩教训":通过硬编码知识和规则来构建智能体系统只能带来短期收益,长期来看必然会遇到瓶颈。而突破性进展最终将通过基于搜索和学习的计算扩展方法到来。

在这一背景下,OpenAI、Anthropic等前沿实验室已开始研发基于强化学习训练的真正智能体。OpenAI的DeepResearch和Claude Sonnet 3.7在搜索和编程任务中展现出了令人印象深刻的能力——它们能够规划搜索策略,交叉引用多个来源,并基于中间反馈完成复杂的多步骤任务。

这一技术突破表明,通过将强化学习与推理能力结合,LLM可以从与环境的直接交互中学习最优策略,无需依赖人工设计的提示词和工作流。这些智能体不仅能够思考,还能够自主行动、规划和调整策略,真正实现长期、复杂任务的自主完成。

Agent-R1正是在这一关键技术变革期应运而生的开源框架,致力于推进强化学习与智能体技术的融合发展,让这一前沿技术能够被更广泛的研究者和开发者所使用,共同推动AI真正迈向自主智能的新时代。

核心特点

多轮工具调用能力

Agent-R1采用端到端强化学习方法,训练智能体从完整交互轨迹中学习。这使模型能够进行连续多轮工具调用,理解长期因果关系,将当前行动与未来结果关联起来,形成真正的规划和迭代能力。

多工具协调机制

框架支持智能体学习如何协调使用多种工具共同解决复杂任务。智能体能够灵活选择最适合当前情境的工具组合,形成有效的工具组合策略,而非被限制在单一工具的使用模式中。

过程奖励系统

创新的过程奖励机制允许对每个工具调用的有效性进行单独评估,而非仅关注最终结果。通过奖励归一化技术,该方法平衡了过程奖励与结果奖励,确保智能体在追求最终目标的同时也注重解决问题的过程质量。

自定义工具和环境

框架与主流LLM工具调用格式完全兼容,开发者可以轻松扩展自己的工具和应用场景。只需定义特定领域的工具和奖励函数,即可将Agent-R1应用到任何专业领域,无需复杂的工作流设计。

多种强化学习算法

Agent-R1 支持多种先进的强化学习算法,包括PPOGRPOREINFORCE++,为不同需求和场景提供灵活选择。开发者可以根据具体应用选择最适合的算法,优化智能体的学习效果。

多模态支持

最新版本实现了全面的多模态支持,无缝集成视觉-语言模型(VLMs),使智能体能够处理和推理文本与视觉输入的组合。这一功能极大扩展了智能体的应用场景,能够在丰富的多模态环境中进行有效交互。

案例展示

在HotpotQA上的实验验证显示,即使是1.5B的小模型,在强化学习后也能自主推理并调用搜索引擎,解决复杂问题。

User

Question: The album that contains the song "Get Together" by Madonna was mainly recorded in whose home studio?
  • 1.

Assistant

<think>
To answer this question, we need to find out the song "Get Together" by Madonna and then determine in which home studio she recorded the album it belongs to. I will use the search tool to find information about the song "Get Together" by Madonna.
</think>
<tool_call>
{"name": "search", "arguments": {"query": "Get Together by Madonna"}}
</tool_call>
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.

Tool

<tool_response>
{"results": ["Get Together (Madonna song) \"Get Together\" is a song by American singer Madonna from her tenth studio album \"Confessions on a Dance Floor\" (2005). Produced by Madonna and Stuart Price, the song was released as the third single from the album by Warner Bros. Records on June 6, 2006. The decision was spurred by the fact that \"Get Together\" was the third most downloaded song from the album. It was also released to coincide with the start of Madonna's Confessions Tour. Inspired by Stardust's single \"Music Sounds Better with You\", \"Get Together\" portrays an anthem-like picture with its lyrics, about the different possibilities of finding love on the dance floor.", "Keep It Together (song) \"Keep It Together\" is a song by American singer Madonna from her fourth studio album \"Like a Prayer\" (1989). It was released on January 30, 1990 by Sire Records as the sixth and final single from the album in the United States, Canada and Japan. Written and produced by Madonna and Stephen Bray, the main inspiration behind \"Keep It Together\" was Madonna's relationship with her family\u2014whom she dearly missed after her divorce from actor Sean Penn. The song was dedicated to American funk and soul band Sly and the Family Stone. The lyrics deal with the realization of how important Madonna's family has been to her life. A pop and funk song consisting of an upbeat rhythm and groove, \"Keep It Together\" features percussion, banjo and a conga.", ...]}
</tool_response>
  • 1.
  • 2.
  • 3.

Assistant

<think>
Based on the search results, "Get Together" is indeed a song by Madonna from her album "Confessions on a Dance Floor" (2005). This provides the name of the album and its release details. 
Next, I need to find out where this album was mainly recorded.
I will now use the search tool to find information about the album "Confessions on a Dance Floor" by Madonna, specifically looking for details on its recording process.
</think>
<tool_call>
{"name": "search", "arguments": {"query": "Confessions on a Dance Floor (album)"} }
</tool_call>
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.

Tool

<tool_response>
{"results": ["Confessions on a Dance Floor Confessions on a Dance Floor is the tenth studio album by American singer and songwriter Madonna. It was released on November 9, 2005 by Warner Bros. Records. A complete departure from her previous studio album \"American Life\" (2003), the album includes influences of 1970s and 1980s disco, as well as modern-day club music. Initially, she began working with Mirwais Ahmadza\u00ef for the album, but later felt that their collaboration was not going in the direction she desired. Madonna took her collaboration with Stuart Price who was overviewing her documentary \"I'm Going to Tell You a Secret\". The album was mainly recorded at Price's home-studio where Madonna spent most of her time during the recordings.", "Confessions (Usher album) Confessions is the fourth studio album by American singer Usher. It was released on March 23, 2004, by Arista Records. Recording sessions for the album took place during 2003 to 2004, with its production handled by his longtime collaborator Jermaine Dupri, alongside with Jimmy Jam and Terry Lewis and Lil Jon, among others. Primarily an R&B album, \"Confessions\" showcases Usher as a crooner through a mixture of ballads and up-tempos, incorporating musical genres of dance-pop, hip hop and crunk. The album's themes generated controversy about Usher's personal relationships; however, the album's primary producer Jermaine Dupri claimed the album reflects his personal story.",...]}
</tool_response>
  • 1.
  • 2.
  • 3.

Assistant

<think>
From the search results, we learn that "Confessions on a Dance Floor" by Madonna was indeed released in 2005, but the primary information we seek is where this album was mainly recorded. The results indicate that it was "mainly recorded at Price's home-studio".
The album "Confessions on a Dance Floor" by Madonna was mainly recorded at Stuart Price's home studio.
Therefore, the answer to the question "The album that contains the song 'Get Together' by Madonna was mainly recorded in whose home studio?" is Stuart Price's home studio.
</think>
<answer>
Stuart Price's home studio
</answer>
  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.

训练过程分析

以下三组图片分别展示了使用三种主流的强化学习算法训练时模型性能工具调用轮数以及最终回答长度的变化过程。

PPO

中国科大认知全重实验室发布Agent-R1训练框架,支持自主思考与工具调用!-AI.x社区

REINFORCE++

中国科大认知全重实验室发布Agent-R1训练框架,支持自主思考与工具调用!-AI.x社区

GRPO

中国科大认知全重实验室发布Agent-R1训练框架,支持自主思考与工具调用!-AI.x社区

可以观察到,模型 (Qwen2.5-1.5B-Instruct) 在面对具有挑战性的多跳问题时,有效地学会了先思考再多轮调用工具,最终提高了精确匹配(EM)分数。虽然不同强化学习算法的效果各异,但总体趋势保持一致。

值得注意的是,实验揭示了一个显著的相关性:模型性能、工具调用轮数和最终回答长度在整个训练过程中呈现一致的变化趋势。这展示了 Scaling Law 的全新维度——一个与智能体-环境交互频率相关的维度。随着智能体学会通过多次工具调用更有效地与环境交互,性能也相应提升,这表明多轮环境交互能力对智能体性能的重要性可能与传统的扩展因素同等关键。

结语

我们正处于AI技术革命的前沿,Agent-R1不仅体现了一种技术框架,还彰显了对AI未来发展的长远愿景——从被动执行到主动思考,从固定路径到自主探索。Agent-R1的推出,旨在让更多的研究者和开发者能够接触到前沿的强化学习与智能体技术,推动技术的广泛应用。只有通过开放合作,才能真正推动AI从模仿到创造,从受限到自由的飞跃。

项目地址:​​​https://github.com/0russwest0/Agent-R1​

本文转载自​​NLP PaperWeekly​​,作者:NLP PaperWeekly

收藏
回复
举报


回复
相关推荐