Stateful AI, Resilient Jobs & Data Pipelines

This cloud native data pipelines platforms proof of concept (POC) on GitHub delivers three data pipeline architectures to deliver high-quality data at required levels of urgency. Data quality and data urgency are becoming more critical to keep pace with the speed of decisions and resulting actions by humans or AI agents. Data sovereignty, data locality, legacy data assets, lock-in avoidance, cost savings are driving organizations to evaluate open-source data pipelines to tap insights and decisions from their operational databases.

The data pipelines use proven modern infrastructures: Airflow for task orchestration, Spark for large datasets processing, dbt for transformations, Kafka for messaging, and MCP server for natural language queries.

One operational source Postgres database feeds three independent, parallel pipelines: (1) Micro-batch for agentic AI, (2) Iceberg lakehouse, and (3) Streaming CDC mirror.

This talk will walk through the pipelines with reasons for building them on open-source and cloud-native infrastructures including Kubernetes, PostgreSQL, ClickHouse, Apache Iceberg, MinIO S3, MCP, Spark, Debezium CDC, Strimzi, KRaft, Prometheus, and Grafana. The pipelines are being functionally tested on Kubernetes cluster.

We invite interested parties to try the POC in a sandbox, contribute, or provide feedback for improvements https://github.com/kalluripradeep/modern-etl-stack. Our objective is to keep building on the platform and continue testing more intensely.

Paper: https://medium.com/@raghu.v.shankar/data-pipelines-platform-for-analytics-and-ai-proof-of-concept-poc-bring-your-own-use-case-6bfbb8e9aa2e

We were moving our Airflow platform onto a new set of Kubernetes clusters. The plan read as routine: drain the old clusters, let the workloads come back up on the new ones. Then someone asked the question that turns a migration into an incident. What happens to the three-hour Spark job that is halfway done when we drain the node it is running on?

Kubernetes treats pods as fungible and reschedulable by design. The control plane can evict, preempt, or relocate a pod at any time, which is fine as long as the workload is stateless or cheap to restart. It breaks down when a single "task" in your platform holds a worker for its entire duration: a multi-hour Spark submit, a poke-mode sensor, an operator that submits work to an external system and blocks on polling it. Drain the node under that task and the orchestrator observes a dead worker, marks the task failed, and retries it. That default retry is the whole problem.

This is a field report on keeping Apache Airflow alive through Kubernetes disruptions, and in particular through a planned cross-cluster migration, where the disruptions are deliberate and fleet-wide instead of rare and random. Two failure modes surface as soon as you run this kind of work on a platform that can reschedule the underlying pod at will.

The first is retry-budget theft. Airflow's retries setting is meant for transient application failures: a timed-out API call, a deadlock. But an evicted pod's SIGTERM flows through the same failure path (handle_failure, then is_eligible_to_retry) and consumes one of those retries. During a migration you drain cluster after cluster, so a task rescheduled several times can exhaust its entire retry budget on infrastructure churn it had no part in. That budget is provisioned for application-level failures; infrastructure disruption drains it instead.

The second is discarded work. A retry restarts the task from the beginning. A Spark job most of the way through returns to zero even though the underlying job is still running on the cluster and still addressable by its job id. Applied to every long-running task in flight during a drain, a routine migration turns into duplicated compute and missed SLAs across the fleet at once.

The tempting fix is a heuristic: I received a SIGTERM, the task is still in RUNNING, it was not a timeout, therefore it was probably an eviction, so do not charge the retry. I will show why that inference is unsound, and why classifying disruptions by process of elimination eventually misclassifies a genuine failure as a disruption and resumes it indefinitely, which is worse than the status quo.

The version that holds up is two separate pieces, both public Airflow proposals I wrote (AIP-96 Resumable Operators and AIP-97 Infrastructure-Aware Task Execution), building on the accepted AIP-103 durable task state:

First, reconnect instead of resubmit. Persist the external job id to durable task state before the wait begins. On resume, look up that id and reattach to the still-running job rather than launching a second one. The restart becomes a reconnect.

Second, consume a positive disruption signal rather than inferring one. Have the executor read the signals Kubernetes already emits: pod eviction and delete events, the kubelet's termination reason, an annotation surfaced through the Downward API, or a drain protocol that marks in-flight tasks disrupted before the SIGTERM is delivered. When the signal confirms an infrastructure-initiated termination, the task resumes without decrementing the retry budget, using the same mechanism deferral and reschedule mode already use: try_number is left unincremented, so the resume counts as the same attempt.

Composed, the two make an eviction during a long external job free: no duplicate submission, no retry consumed. That is what makes a cluster drain or a fleet-wide migration survivable for long-running work. In-flight tasks reconnect to their running jobs afterward with their retry budgets intact.

None of this is specific to Airflow. The underlying idea is to extend disruption readiness up a layer, from the pod to the workload running on it. Kubernetes already provides the primitives to survive a disruption at the pod level: PodDisruptionBudgets to bound how much of the fleet moves at once, a termination grace period for a clean shutdown window, and the eviction API and Downward API to signal that a move is imminent. Stateful compute adds checkpointing on top. Most platforms configure those primitives and stop, leaving the infrastructure disruption-ready while the workload running on it is not. The orchestrator still translates "the node was reclaimed" into "your task failed." Closing that gap, making the workload disruption-ready the way the pod already is, is the subject of this talk. Any platform running stateful or long-lived work can close it and make disruptions free.

You will leave able to identify where your own platform conflates an infrastructure eviction with an application failure, wire a positive Kubernetes disruption signal into a workload's lifecycle, and make a cluster drain or a full cross-cluster migration survivable for long-running jobs.