Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
Enterprises are bullish on agentic applications that can understand user instructions and intent to perform different tasks in digital environments. It’s the next wave in the age of generative AI, but many organizations still struggle with low throughputs with their models. Today, Katanemo, a startup building intelligent infrastructure for AI-native applications, took a step to solve this problem by open-sourcing Arch-Function. This is a collection of state-of-the-art large language models (LLMs) promising ultra-fast speeds at function-calling tasks critical to agentic workflows.
But, just how fast are we talking about here? According to Salman Paracha, the founder and CEO of Katanemo, the new open models are nearly 12 times faster than OpenAI’s GPT-4. It even outperforms offerings from Anthropic all while delivering significant cost savings at the same time.
The move can easily pave the way for super-responsive agents that could handle domain-specific use cases without burning a hole in the businesses’ pockets. According to Gartner, by 2028, 33% of enterprise software tools will use agentic AI, up from less than 1% at present, enabling 15% of day-to-day work decisions to be made autonomously.
What exactly does Arch-Function bring to the table?
A week ago, Katanemo open-sourced Arch, an intelligent prompt gateway that uses specialized (sub-billion) LLMs to handle all critical tasks related to the handling and processing of prompts. This includes detecting and rejecting jailbreak attempts, intelligently calling “backend” APIs to fulfill the user’s request and managing the observability of prompts and LLM interactions in a centralized way.
The offering allows developers to build fast, secure and personalized gen AI apps at any scale. Now, as the next step in this work, the company has open-sourced some of the “intelligence” behind the gateway in the form of Arch-Function LLMs.
As the founder puts it, these new LLMs – built on top of Qwen 2.5 with 3B and 7B parameters – are designed to handle function calls, which essentially allows them to interact with external tools and systems for performing digital tasks and accessing up-to-date information.
Using a given set of natural language prompts, the Arch-Function models can understand complex function signatures, identify required parameters and produce accurate function call outputs. This allows it to execute any required task, be it an API interaction or an automated backend workflow. This, in turn, can enable enterprises to develop agentic applications.
“In simple terms, Arch-Function helps you personalize your LLM apps by calling application-specific operations triggered via user prompts. With Arch-Function, you can build fast ‘agentic’ workflows tailored to domain-specific use cases – from updating insurance claims to creating ad campaigns via prompts. Arch-Function analyzes prompts, extracts critical information from them, engages in lightweight conversations to gather missing parameters from the user, and makes API calls so that you can focus on writing business logic,” Paracha explained.
Speed and cost are the biggest highlights
While function calling is not a new capability (many models support it), how effectively Arch-Function LLMs handle is the highlight. According to details shared by Paracha on X, the models beat or match frontier models, including those from OpenAI and Anthropic, in terms of quality but deliver significant benefits in terms of speed and cost savings.
For instance, compared to GPT-4, Arch-Function-3B delivers approximately 12x throughput improvement and massive 44x cost savings. Similar results were also seen against GPT-4o and Claude 3.5 Sonnet. The company has yet to share full benchmarks, but Paracha did note that the throughput and cost savings were seen when an L40S Nvidia GPU was used to host the 3B parameter model.
“The standard is using the V100 or A100 to run/benchmark LLMS, and the L40S is a cheaper instance than both. Of course, this is our quantized version, with similar quality performance,” he noted.
Another exciting day here Katanemo as we open source some of the "intelligence" behind Arch (https://t.co/9nwakOGPp0). Meet Katanemo Arch-Function, a collection of state-of-the-art (SOTA) LLMs designed for function calling tasks – that meet/beat frontier LLM performance, but… pic.twitter.com/IajF8w3syz
— Salman Paracha (Building Intelligent Infra) (@salman_paracha) October 15, 2024
With this work, enterprises can have a faster and more affordable family of function-calling LLMs to power their agentic applications. The company has yet to share case studies of how these models are being utilized, but high-throughput performance with low costs makes an ideal combo for real-time, production use cases such as processing incoming data for campaign optimization or sending emails to clients.
According to Markets and Markets, globally, the market for AI agents is expected to grow with a CAGR of nearly 45% to become a $47 billion opportunity by 2030.
VB Daily
Stay in the know! Get the latest news in your inbox daily
Thanks for subscribing. Check out more VB newsletters here.
An error occured.
