Most teams that run Argo Workflows in production already have Kubernetes observability in place, with Prometheus scraping pod metrics, Grafana dashboards tracking cluster resource utilization, and possibly a commercial APM tool aggregating logs across namespaces. The reasonable assumption is that this stack gives you meaningful visibility into whether your workflows are healthy. In practice, it tells you whether your pods are healthy, which is a related but meaningfully different question, and the difference between the two is where a large category of Argo failures quietly lives.
What Pod-Level Metrics Can and Cannot See
When a Kubernetes monitoring tool detects a problem, it is almost always detecting a problem at the infrastructure layer: a pod OOMKilled, a node running out of disk, a container failing its liveness probe, a resource quota exceeded. These are infrastructure events, and Prometheus is well-suited to surface them. What Prometheus cannot tell you is what the Argo Workflow Controller concluded about that infrastructure event, because the controller's interpretation of a pod failure depends on how that step's template was configured, including its retry policy, its timeout settings, and its relationship to sibling steps in the DAG.
A step that hits its retry limit and exhausts all configured attempts does not produce a Kubernetes event that generic monitoring tools recognize as a pipeline failure. The workflow may continue to downstream steps if the failed step was not declared a dependency of those steps, or it may silently mark the step as failed while other parallel branches complete normally. Your Grafana dashboard shows healthy pod completion rates because the pods themselves completed; the workflow-level outcome is invisible at the infrastructure layer where your monitoring operates.
The Failure Modes That Generic Monitoring Misses
When our team ran a deep analysis of Fetch Analytics' workflow infrastructure, we found inefficiencies that their existing monitoring stack had not surfaced, precisely because the problems were Argo-specific rather than infrastructure-specific. The pattern we see most commonly in production Argo deployments involves four failure modes that standard Kubernetes observability cannot detect without purpose-built tooling.
Template retry storms occur when a step template is configured with an aggressive retry policy and the underlying issue causing step failures is not transient but structural, such as a misconfigured artifact location or a dependency that has changed its schema. Each retry creates a new pod, and a workflow with multiple parallel steps each retrying independently can produce hundreds of pod creations within a short window, consuming node resources and degrading overall cluster performance in a way that looks like a resource utilization spike rather than a workflow configuration problem.
Pending pod accumulation happens when pods are scheduled but cannot start because the cluster lacks the resources to place them, and the Argo Workflow Controller interprets these pods as in-progress steps rather than failing steps. Your monitoring stack sees healthy pods in pending state, which is a normal Kubernetes condition for pods waiting on scheduling. The Workflow Controller continues counting them as active steps and does not escalate them to failed status until a workflow-level timeout fires, which may be configured to hours rather than minutes. Your pipeline has effectively stalled, but nothing in your standard alert ruleset has fired.
Artifact pass/fail transparency is the third gap. When a step produces an artifact that a downstream step depends on, the artifact storage operation happens outside the normal pod lifecycle that Kubernetes monitors. A step container can exit with status code 0, satisfying every pod-level health check, while the artifact it was supposed to produce was written to the wrong location, was written with a size of zero bytes due to an upstream data issue, or was never written because the storage client timed out silently. The downstream step that depends on that artifact will fail when it attempts to read it, but the upstream step's pod metrics show a clean completion.
Workflow controller reconciliation lag is the fourth, and it becomes significant at scale. As we described in our architectural deep dive, the Workflow Controller reconciles observed state against desired state for every active workflow in the cluster, and under high workflow volumes the reconciliation loop slows down. Standard cluster monitoring does not expose reconciliation lag as a named metric; what you observe instead is that workflows take longer to progress between steps without any pod-level anomaly explaining the delay.
Distributed lock contention is the fifth failure mode, and it has become more relevant since Argo Workflows 3.7 introduced multi-controller locking with semaphores and mutexes across multiple controllers. When workflows compete for a shared semaphore and the lock acquisition queue builds up, the workflows waiting on the lock appear healthy from a Kubernetes perspective because their pods have not started yet. They are simply queued. Generic monitoring has no visibility into the lock queue depth or how long individual workflows have been waiting for a lock to release, which means contention that is effectively stalling your pipeline throughput looks identical to a healthy low-traffic period in your cluster dashboards.
What Argo-Aware Monitoring Actually Requires
Meaningful observability for Argo Workflows needs to track state at the workflow and template level, not just the pod level. The metrics that matter are workflow success rate broken down by template type, step retry rate per workflow and per template, pending pod duration at the step level rather than the cluster level, artifact operation success and failure rates, workflow controller reconciliation duration under varying concurrent workflow loads, and, for deployments using 3.7's multi-controller locking, semaphore queue depth and lock wait duration per workflow.
For Fetch Analytics, building custom alerting and dashboards specifically designed around Argo's behavior rather than treating Argo as a generic Kubernetes workload gave the team the workflow-level visibility their infrastructure monitoring had never provided. The result was that engineering time previously spent investigating unexplained pipeline failures could be redirected toward building the features that actually mattered for their customers.
If your team is relying on cluster-level metrics to understand Argo Workflows health, you are likely flying partially blind on a category of failures that your current tooling is not designed to surface. Adding Argo-aware monitoring layers does not replace your existing Kubernetes observability stack; it sits above it and fills the gap between what your infrastructure sees and what your workflows are actually doing.
Pipekit is a managed control plane for Argo Workflows, built and maintained by the team that contributes to the Argo Workflows project. Learn how Pipekit approaches observability.
