Microsoft announces AutoGen Studio

🔍 Spotlight

Large language model (LLM) agents are the current hot topic in AI, and low-code frameworks are the latest wave in the agent field. Last Monday, Microsoft announced AutoGen Studio, a low-code interface for their popular agent framework AutoGen. The release cements AutoGen’s status as one of the leading—if not the leading—agent-building library.

AutoGen, released by Microsoft in September 2023, is a Python package providing built-in tools for easy construction of multi-agent workflows. Agents and teams of agents are instantiated as classes that abstract much of the boilerplate code, allowing a conversation between several agents to be instantiated in minutes. Support is provided for easy utilization of LLMs from various providers such as OpenAI and Anthropic, secure execution of code written by the agents, integration of tools like retrieval-augmented generation (RAG) databases, and more. Although AutoGen is the only such agent framework released by a major tech company so far, community and startup competitors have emerged, including crewAI, Langchain’s LangGraph, and Langroid.

As code-based frameworks for developing LLM-based applications have proliferated, so too have low-code options. Langflow and Flowise are two such examples, enabling users to visually represent language models, databases, tools, and more as building blocks that can be linked together, allowing developers with little or no coding experience to construct powerful agentic workflows.

AutoGen Studio brings these capabilities to Microsoft’s framework, enabling users to input the configuration, prompts, and rules for its agent and teams classes using a graphical interface rather than Python code. Once built, these workflows can be deployed as APIs to be called by other applications, saved to JSON files for later use in Python programs, or built within Docker containers. While Studio does not yet support the drag-and-drop interface enabled by Langflow and Flowise, its developers state that this feature is currently under development.

Coupled with Microsoft’s recent announcement of Copilot Studio agents, the release of AutoGen Studio bolsters its position as arguably the furthest-looking of the major tech companies in enabling agentic capabilities.

📰 News

Cohere API gets multi-step tool use

Cohere’s API now natively supports multi-step problem solving with integrated planning and tool use. A parser provided by Cohere automatically implements the planning step and an action → observation → reflection cycle, enabling users to build a simple agent workflow with just a few lines of code.

NEXA AI sponsors agent hackathon with $7,000 prizes

NEXA AI, a startup building custom LLMs for agent workflows, is hosting a hackathon for users to build agentic tools using their models. Though details are sparse, the company claims sponsorship by Huggingface and up to $7,000 in prizes.

🧪 Research

Improving Multi-Agent Debate with Sparse Communication Topology

Multi-agent systems allow for a multiplicity of different communication schemes, in which the agents are allowed to “speak” in various orders or to different combinations of their fellow agents. This publication by Google DeepMind researchers investigates optimal communication topologies, finding that thoughtfully designed sparse configurations can outperform more highly-connected ones at a fraction of the cost.

TAU-bench, Sierra’s new benchmark for agent performance

Sierra, a startup provider of AI agents for customer interaction, published TAU-bench, a new benchmark to measure agent performance. They found that most available models performed poorly, highlighting the need for more sophisticated LLMs built specifically for agentic workflows.

Security of AI Agents

The vulnerability of LLM-based agents to attack is a growing area of concern and study. This paper provides an overview of the security vulnerabilities inherent to agents, as well as possible defense mechanisms.

HUSKY: A Unified, Open-Source Language Agent for Multi-Step Reasoning

Most LLM agents utilize base, proprietary models without fine-tuning, relying on their native strengths in verbal reasoning. The creators of HUSKY fine-tune several 7-billion parameter models to act as the components of an agentic reasoning system, achieving superior performance compared with flagship out-of-the-box commercial models such as GPT-4.

Adversarial Attacks on Multimodal Agents

The authors of this paper identify two methods for attacking visual LLMs by manipulating images to produce captions that serve as prompt injections. The attacks respectively target white-box captioning models and proprietary LLMs powered by CLIP, achieving a worryingly high success rate in manipulating agent behavior.

AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents

AgentDojo is a framework for assessing the security of agent systems, allowing testers to subject agents to attack while engaged in a variety of common tasks.

🛠️ Useful stuff

Create voice agents with LangChain using Vocode

Vocode is an open-source library for voice interaction with LLMs, now integrated into LangChain.

Agent tools for TypeScript devs

A toolbox of commonly used LLM functions and tools such as web scrapers and APIs, packaged in a standard interface for TypeScript developers.

A new course on function calling with LLMs

Yet another agent course by Andrew Ng’s DeepLearning.AI educational platform, this one on facilitating data extraction by LLMs using function calling.

💡 Analysis

Interview with LangChain’s CEO on building agents

An interview with Harrison Chase, co-founder and CEO of LangChain, on building the orchestration layer for AI agents. Chase discusses the origin and current state of the agent ecosystem, as well as how LangChain can facilitate agent development.

X thread on essential tools in the Python agent stack

A helpful roundup of the most important Python tools for agent building, with plenty of new suggestions from respondents.