Google Cloud Open-Source Tools Cut Costs of Idle AI Agents on K8s

Kubernetes has become the go-to platform for distributed training and inference, but what happens to compute bills when it runs AI agents that spend most of their time waiting? When an agent's container or virtual machine remains provisioned, customers pay for idle time as well as work time.

To help customers control these costs and make agent workloads more predictable, Google LLC has donated three projects to the Cloud Native Computing Foundation, said Allan Naim, director of AI infrastructure product management at Google Cloud. For Kubernetes itself, the new challenge is releasing compute resources during activity gaps while letting each agent resume from where it left off.

"Act two was a few years ago; Kubernetes became the de facto standard for inference and training," Naim said. "Now what's really interesting is we're opening up this third act... where the typical patterns are quite different from how Kubernetes was originally designed."

Naim and Gari Singh, a Google Cloud product manager, spoke with Rebecca Knight of theCUBE, SiliconANGLE Media's livestreaming studio, during an exclusive broadcast at the KubeCon + CloudNativeCon NA conference. They discussed how Google Cloud is using open-source tools to cut compute costs when AI agents are idle.

Open building blocks for distributed training and inference meet agentic AI
Naim explained that cost is a serious issue in agent workloads, with agents idle 90% to 95% of the time, waiting for models, test runs, or human approval. Google developed Agent Substrate, an open-source runtime that hosts AI agents in isolated environments on Kubernetes. When an agent is suspended, the runtime saves a snapshot of its memory and local files and releases its compute resources for other agents. When work resumes, it restores the agent from that snapshot in an available sandbox.

"From an economic standpoint, this is helpful," Naim said. "We've seen a 10x cost reduction running Agent Substrate on Kubernetes versus running these agents directly on dedicated virtual machines and containers."

Security is another concern. In response, Google open-sourced gVisor in 2018, which sits atop the host kernel to block or audit every system call made by an agent, according to Singh.

"Agents and agent frameworks are designed to solve problems — and try to solve problems — until they can't," Singh said. "They'll find clever workarounds. They know everything around them. You want to make sure it's not making certain low-level changes to your low-level security, not making outbound network calls, not trying to open files it shouldn't open."

Singh noted that Google is also working with Anyscale Inc. on KubeRay, a tool that runs Anyscale's Ray platform natively on Kubernetes, enabling platform teams to provide distributed training and inference to many research groups. Naim added that OpenRL brings this approach to reinforcement learning, which provides guardrails for post-trained models.

"In the past, to do reinforcement learning, you needed a dozen PhD researchers," Naim said. "Our goal is, 'How do we bring reinforcement learning into platform engineering so you can run reinforcement learning in a Kubernetes environment and use it to continuously improve your agent workloads?'"

The following is the full video interview, part of SiliconANGLE and theCUBE's coverage of the KubeCon + CloudNativeCon NA conference:

(* Disclosure: Google Cloud sponsored this portion of theCUBE. Google Cloud and other sponsors have no editorial control over content on theCUBE or SiliconANGLE.)