Skip to content

Adopt Cilium as a CNI in Welkin-apps

  • Status: Accepted
  • Deciders: Product Team
  • Date: 2026-01-14

Context and Problem Statement

Welkin currently relies on Calico for networking. However, we have identified several evolving requirements that are difficult to address with our current stack:

  • FQDN Network Policies: We need a robust way to define egress policies based on domain names rather than volatile IP ranges.
  • Network Observability: Platform administrators and customers require better tooling to debug network traffic and visualize L7 flows.
  • IPv6 Support: Customer demand for IPv6 is increasing, requiring a CNI with first-class dual-stack support.
  • Stable Outbound IP: In Cluster API (CAPI) environments, worker Nodes often lack public IP. We need a way to provide stable outbound IP (Egress Gateway) so external systems can allowlist traffic from the Cluster.
  • Future Traffic Management: We want to retain the option to adopt Gateway API or service mesh capabilities as needs evolve. This ADR does not decide whether or how to offer them to customers.

What technology should we use to fulfill networking requirements in Welkin?

Decision Drivers

  • Operational Efficiency: Simplify network debugging and FQDN management.
  • Strategic Alignment (ADR-0015): We have a strong preference for community-driven open-source projects. Compared to Calico, Cilium is a community-driven CNCF graduated project.
  • Scalability and Modern Standards: Requirement for scalable network management and a preference for technology that can support future networking standards.
  • Resource Cost: We want to minimize the CPU and memory "tax" imposed by the networking and mesh layers.
  • Feature Parity: Provide stable egress IP for CAPI-based Clusters.

Considered Options

  1. Option 1: Adopt Cilium as the primary CNI for Welkin-apps.
  2. Option 2: Stay with Calico and attempt to implement these features natively or through third-party workarounds.
  3. Option 3: Use Istio on top of Calico for additional service mesh capabilities.

Pros and Cons of the Options

  • Option 1: Adopt Cilium

    • Good, because it is a community-led project, avoiding the "open-core" pitfalls of competitors.
    • Good, because it offers network observability, scalable management, and optional service mesh capabilities.
    • Good, because it natively supports FQDN-based egress filtering and offers Gateway API support that we can evaluate separately.
    • Good, because it solves the "stable IP" problem in CAPI via the Egress Gateway.
    • Bad, because eBPF debugging requires specialized training for the team.

  • Option 2: Stay with Calico

    • Good, because the team is already familiar with its operational nuances.
    • Bad, because it follows an open-core strategy, which conflicts with ADR-0015.
    • Bad, because it lacks integrated L7 observability and service mesh capabilities comparable to Cilium.
  • Option 3: Use Istio on top of Calico
    • Good, because Istio is the most mature and feature-rich service mesh in the ecosystem, offering advanced traffic management (fault injection, circuit breaking) that is battle-tested in large-scale production.
    • Good, because Istio is a CNCF Graduated project.
    • Good, because Istio in ambient mode offers trades away some of the functionality that the sidecar proxy approach offered to have a smaller footprint.
    • Bad, in the case of Istio ambient mode because of increased architectural and operational complexity, where Istio's ambient mode adds a "CNI"-like plugin on top of the actual CNI of choice.
    • Bad, because adding yet another layer in the networking stack means more moving parts in the lowest layers, putting platform stability at risk.
    • Bad, because while Istio attracted significant interest around 2018 or so with the unique properties of a service mesh, the field seems to have moved on and eBPF-based CNI plugins such as Cilium offer the main security and observability features, without needing to introduce one more component. We do not, in 2026, see a compelling need for Istio.

Decision Outcome

Chosen option: Option 1, because it provides eBPF-native networking, aligns with our values of community-driven software, and offers Hubble observability and a native Egress Gateway solution. While migration was initially a concern, we have identified a technical path to mitigate risks. Adoption of Cilium's service mesh or Gateway API capabilities is outside the scope of this decision.

Positive Consequences

  • Open Source Integrity: In accordance with ADR-0015, we benefit from Cilium being a truly community-driven open-source project. This contrasts with Calico's open-core strategy, ensuring that critical features are not gated behind enterprise licenses.
  • Superior Observability: Cilium offers significantly better observability through Hubble, providing granular flow visibility and service dependency mapping.
  • Scalable Management: Cilium’s eBPF-based architecture provides more scalable network management and performance as Cluster sizes grow.
  • Future Traffic Management: Cilium offers service mesh and Gateway API capabilities. Whether to use these capabilities or offer them to customers requires a separate decision.
  • CAPI Compatibility: Cilium Egress Gateway allows us to provide stable IP even when worker Nodes are on private networks.
  • Unified Stack: Using one technology for CNI and FQDN policies reduces the "moving parts" in the platform.
  • Industry Standard: Alignment with the networking choice of major Infra Providers and the CNCF graduated ecosystem.

Negative Consequences

  • New Technology Stack: eBPF-based networking introduces a different mental model for troubleshooting compared to standard IPtables.
  • Implementation Effort: Requires updating automation and Deployment manifests to handle Cilium-specific CRDs.