Jijesh Kalliyat

When a user runs kubectl exec, the API server authenticates the caller and records their identity in the audit log. That identity is never forwarded to the kubelet or the container runtime. Any security tooling operating below the API server boundary works blind to who initiated the session. KEP-6035 (Merged) proposes threading the audit request ID through the full exec path from the API server to the kubelet to the CRI runtime, via a new envs field on ExecRequest and a corresponding Env addition to PodExecOptions. The result: runtime security agents can correlate process-level events with the originating Kubernetes user for the first time. This talk covers the identity gap in concrete terms, the design decisions behind the KEP, and the eBPF proof-of-concept built to validate tamper-proof identity anchoring at the container boundary using bpf_task_storage. Audience: engineers working on Kubernetes security, runtime observability, or eBPF-based tooling.

References:

• https://blog.sigtrapd.dev/posts/k8s-exec-identity-gap/
• https://blog.sigtrapd.dev/posts/exploring-the-exec-identity-gap/
• https://blog.sigtrapd.dev/posts/bridging-k8s-exec-identity-gap/
• https://www.youtube.com/watch?v=bl8OX2nlEfM&t=900s - Presented in the SIG Node weekly meeting

Abstract Modern AI workloads place new demands on Kubernetes scheduling. Distributed training jobs, inference services, interactive development environments, and batch workloads often compete for the same GPU resources, leading to resource fragmentation, long queue times, inefficient placement decisions, and underutilized infrastructure.

In this session, we will explore the scheduling challenges unique to AI platforms and examine how KAI Scheduler extends Kubernetes scheduling capabilities to better support large-scale GPU environments. We will cover key concepts such as gang scheduling, topology-aware placement, hierarchical queues, workload prioritization, fair-share scheduling, resource reservations, and GPU resource optimization.

We will discuss how these scheduling strategies help platform teams improve utilization, reduce contention, and operate multi-tenant AI infrastructure more efficiently. Attendees will gain practical insights into the trade-offs involved in scheduling AI workloads and how advanced scheduling techniques can improve both cluster efficiency and workload performance.

The session will conclude with a live demonstration showcasing advanced scheduling capabilities in Kubernetes, followed by an interactive Q&A discussing real-world deployment considerations and operational best practices.

Key Takeaways

• Understand why AI workloads introduce new scheduling challenges for Kubernetes.
• Learn the core architecture and capabilities of KAI Scheduler.
• Explore gang scheduling, topology-aware scheduling, and hierarchical queue management.
• Understand how advanced scheduling strategies improve utilization and fairness.
• Learn practical approaches for operating large-scale AI infrastructure on Kubernetes.

Audience Kubernetes Platform Engineers Platform Architects Site Reliability Engineers AI Infrastructure Engineers MLOps Engineers

Tags Kubernetes, Scheduling, Cloud Native, AI Infrastructure, Platform Engineering, GPU, Resource Management, Multi-Tenancy

Sudheesh Sudhakaran

Every Kubernetes debugging guide stops at the same place:

kubectl describe pod checking the events looking at the logs

That's it. But what happens when the pod is stuck in ContainerCreating for twenty minutes, the events say nothing useful, and the logs don't exist because the container never actually started?

This talk goes below the abstraction most engineers never cross, into the Container Runtime Interface (CRI), containerd, the OCI runtime spec, and the shim processes that actually run your workloads. I'll walk you through the real path a pod takes from kubelet to a running Linux process, and build a battle tested debugging playbook along the way. Image pull mismatches between docker and kubelet, sandbox container failures, OCI hook errors (including the one that hit me everytime while running GPU workload on Kubernetes), and cgroup limits that don't match what you configured.

You'll leave with a working mental model of the CRI layer, a crictl / ctr / runc cheat sheet for production incidents, and the confidence to debug the next ContainerCreating hang without guessing.

Takeaways:

1. A layer-by-layer map of what runs between kubelet and your container
2. A symptom to tool debugging workflow you can apply immediately
3. Real production war stories, including GPU-specific CRI failure modes on H100 nodes

Imagine running distributed, highly available databases like etcd in production Kubernetes clusters, and as the backing store for hosted Kubernetes control planes managing highly critical and sensitive data. The stakes are high, and the consequences of failure are severe, like data corruption, quorum loss, member failures, and storage bloat. These issues are complex to manage manually, error-prone, and harder to scale. And since these etcd clusters back the Kubernetes clusters themselves, any disruption to the etcd database can be catastrophic to every workload running on top of them.

What if there was a way to provision & operate etcd clusters at scale with close to zero manual intervention, ensuring that your clusters are self-healing and resilient to failures? In this talk, we will share the lessons we learned from 8+ years of operating production-grade etcd clusters at Gardener and how we built open-source solutions like etcd-druid and etcd-backup-restore to simplify provisioning and managing HA etcd clusters, reliably automating tasks such as scheduled backups, corruption detection, quorum loss recovery, member replacement, scheduled defragmentation, and on-demand operations on running clusters through opsTask and more, with zero recorded data loss across thousands of clusters.

Actively used in production by SAP Gardener, Akamai Cloud, StackIT, and Scaleway, etcd-druid & etcd-backup-restore are Linux Foundation projects under the NeoNephos Foundation, with an active maintainer community. We'll show what it takes to run etcd clusters reliably at scale, in kubernetes, for kubernetes, and make them truly self-healing.

GitOps is great, until your deployment needs more than “sync this manifest.” Argo CD and Flux made GitOps mainstream, but progressive delivery, multi-platform targets, and connecting CI to CD often require extra tooling or custom workflows, especially once you move beyond K8s-only deployments.

PipeCD is a CNCF Sandbox project that approaches this differently. Instead of bolting canary and blue-green support onto a sync engine, PipeCD treats deployment pipelines as first-class citizens: defined in Git, versioned, and fully auditable, across K8s, Terraform, Cloud Run, Lambda, and ECS through a single consistent workflow.

Drawing from active contributions to the PipeCD community, this talk explores the system from its architecture and internals, not just user-facing workflows.

By the end of this talk, attendees will be able to evaluate whether PipeCD fits their delivery stack, understand the architectural tradeoffs compared to Argo CD and Flux, and reason about a progressive delivery pipeline in PipeCD from scratch.

We will cover:

• why GitOps workflows sometimes need progressive delivery primitives?

• how PipeCD’s agent-based architecture works under the hood (the firewall-friendly, outbound-only Piped agent and the Control Plane)?

• how the two-repository model and EventWatcher bridge CI and CD for any artifact, not just container images?

• how canary, automated analysis, and automatic rollback fit into a deployment pipeline?

• how PipeCD’s new v1 plugin architecture makes it extensible to any platform?

This is not a “PipeCD is better” talk. It is about understanding the tradeoffs between different GitOps approaches so you can make better infrastructure decisions.