CNCG and AAIF Chandigarh Meetup

Network with like minded people.

AI agents are unlike anything Kubernetes was designed to run: they generate and execute their own code, so every session must be treated as untrusted; they're singleton and stateful, so Deployments and ReplicaSets fit badly; and they're bursty and interactive, so cold-start latency is user-facing.

Agent Sandbox (kubernetes-sigs/agent-sandbox), launched as a SIG Apps subproject at KubeCon NA 2025, aims to make that a native primitive: a Sandbox CRD purpose-built for agent workloads, with pluggable isolation backends (gVisor, Kata) selectable per security/performance trade-off.

This talk introduces the project from a practitioner's seat - we built and operate a gVisor-based agent-sandbox platform and have felt every gap the project is trying to close. We'll cover: the Sandbox API and how its isolation backends compare in practice, and the hard problems still open upstream - session-state snapshot/restore, warm pools and cold-start latency. You'll leave knowing whether to adopt, contribute, or wait.

Modern AI/ML workloads demand efficient scheduling and resource sharing, but running large-scale batch jobs on Kubernetes isn't always straightforward. Kueue, a CNCF project, addresses these challenges by providing a Kubernetes-native job queueing system for AI/ML, HPC, and batch workloads.

In this session, we'll explore how Kueue helps organizations efficiently schedule AI/ML workloads. Using a real contribution to Kueue as a case study, I'll demonstrate how its end-to-end CI pipeline was optimized by me and team by introducing feature-based test labeling and improved test sharding, enabling faster parallel execution and reducing developer feedback time.

Beyond understanding Kueue's architecture and purpose, attendees will gain practical insights into CI optimization strategies for Kubernetes projects, learn how engineering decisions are made in a large CNCF open source project, and see how contributors can make meaningful performance improvements to the developer experience.

Ever wonder what actually happens when an AI agent's tool call goes somewhere it shouldn't? We'll start with the fundamentals of sandboxing, why AI agents and MCP servers need isolation, and what the tradeoffs look like.

Then we'll get hands-on with Docker Sandboxes (sbx), comparing Locked Down and Balanced modes and using sbx policy log to watch an out-of-policy call get caught in real time.

Title Did My Prompt Break Production? Golden dataset LLM regression testing on Kubernetes with Argo

Abstract Your AI app's behavior lives in one place: the prompt. Change a few words in it, and the app can start giving different answers to your users. But unlike code, prompt changes go to production with no tests and no review. One small edit to a refund bot's prompt, and it can quietly start approving refunds it should have declined. You find out weeks later, when the money is already gone.

This talk shows a simple fix: test your prompts the same way you test your code. We keep a golden dataset, a fixed set of real support tickets where we already know the right answer (full refund, partial, or none). Every time someone changes the prompt, CI runs the new prompt against this dataset. If it gets answers wrong that the old prompt got right, the pull request fails, and you see exactly which cases broke, before anything reaches users.

We will build the whole flow live on Kubernetes: prompts versioned in Git, promptfoo evals gating every pull request, and Argo CD shipping approved prompts to the cluster, so rollback is just a git revert. As a finale, we upgrade the model version without touching the prompt, and watch the golden dataset catch behavior drift that nobody committed.

You will leave with a ready-to-use open-source repo and a clear answer to the question in the title.

Target Audience: Platform engineers, DevOps and SRE practitioners, AI/ML developers

Topics Covered: MLOps, CI/CD for AI, GitOps pipelines, golden dataset evaluation, prompt versioning, Argo CD, promptfoo, Kubernetes

AI agents are usually one-off scripts kagent, a CNCF Sandbox project, turns them into Kubernetes resources you deploy with kubectl apply. In this beginner-friendly demo, an SRE agent investigates a crashing deployment, finds the root cause, and proposes a fix with human approval. Leave ready to deploy your first agent.

Abstract Most RAG systems are just jupyter notebook POCs: embed, retrieve, generate, all in a few cells that can hardly support 10 concurrent queries. But with the real traffic, Vector search fails on high concurrency, the GPU generation step becomes a bottleneck because you never tuned it, and one slow re-rank call takes the whole request down with it.

This talk covers how to split a RAG pipeline into separate services on Kubernetes: embedding, vector retrieval, re-ranking, orchestration, LLM generation and evaluation. Each of these has a different scaling profile and a different way of failing, so each needs to be deployed and scaled on its own terms. I'll walk through running vector databases as sharded, stateful services, autoscaling GPU inference with KEDA instead of relying on CPU-based HPA, where caching actually pays off, and how to keep one failing component from taking down the whole system.

Attendees will have a working reference architecture for a scalable RAG pipeline that serve thousands of concurrent queries, along with a clear picture of which Kubernetes building blocks (StatefulSets, operators, KEDA, GPU scheduling) map to which part of the pipeline. Along with that, a bonus architecture pipeline for performing evaluation at scale.

Key takeaways

• A reference breakdown of RAG into separately scalable Kubernetes services
• How to run vector databases as sharded stateful services
• Autoscaling for GPU inference with KEDA
• How to design for scale and a robust architecture

Audience and level Intermediate to advanced. Platform, ML infra, and backend engineers already running production Kubernetes who are now putting RAG or agentic AI systems into production. Kubernetes fundamentals assumed; no prior RAG experience required.