Docs
kubernetes
cilium+gatewayAPI

cilium+gatewayAPI

cilium 简介

  1. 高性能 eBPF 体系的 CNI 网络面
  2. 生态非常丰富

例如阿里云 ack 的 terway 网络模式, 也就是 cilium 套壳;

生态丰富

cni 网络:

高性能 eBPF, 替代了 calico，也替代了 kube-proxy, 也不再需要 iptable 和 ipvs 这些传统网络模块, 网络路径更短，性能更高;

特性: 直连路由, 网络策略, 可观测性

l2-announcer:

本地 LoadBalancer 方案, 代替 MeatLB, OpenELB, kube-vip 这些方案

l2podAnnouncements:

将 pod 的地址在局域网宣告, 让 Pod IP 在二层网络(局域网)直接可达; 可以方便需要直连pod的程序;

gatewayAPI:

平滑对接 k8s 的 gateway 方案, 替代第三方 ingress 组件和gateway组件; 特性是 四层route 可以走 eBPF, 七层路由仍然是 Envoy

实例

cilium+l2-announcer+gatewayAPI 的组合

helm 安装cilium 

helm repo add cilium https://helm.cilium.io

helm repo update cilium

helm search repo cilium

# 准备helm的参数文件

# 查看 Cilium 1.19.2 版本的完整默认 values
helm show values cilium/cilium --version 1.19.2 > cilium-1.19.2-default-values.yaml

# 挑选需要修改的内容到自己的文件中
# values-home.yaml

helm install cilium cilium/cilium  -f values-home.yaml --namespace kube-system

我的values文件, 作为参考, 内容可以找AI解释

image:
  repository: "harbor.services.wait/quay.io/cilium/cilium"
  useDigest: false
envoy:
  image:
    repository: "harbor.services.wait/quay.io/cilium/cilium-envoy"
    useDigest: false
  # prometheus:
  #   enabled: false
operator:
  image:
    repository: "harbor.services.wait/quay.io/cilium/operator"
    useDigest: false
  # prometheus:
  #   enabled: true
routingMode: "native"
k8sServiceHost: "192.168.5.22"
k8sServicePort: "6443"
kubeProxyReplacement: true
localRedirectPolicy: true
ipam:
  mode: kubernetes
  operator:
    clusterPoolIPv4PodCIDRList: 10.12.0.0/16
    clusterPoolIPv4MaskSize: 24
ipv4NativeRoutingCIDR: 10.12.0.0/16
autoDirectNodeRoutes: true
bpf:
  lbExternalClusterIP: true
  masquerade: true

enableIPv4Masquerade: true
enableIPv6Masquerade: true
loadBalancer:
  mode: dsr
  algorithm: maglev
  acceleration: native
maglev:
  tableSize: 65521
  hashSeed: "6DXnnvXRI47wjvNt"
bandwidthManager:
  enabled: true
  bbr: true
hubble:
  relay:
    enabled: true
    image:
      repository: "harbor.services.wait/quay.io/cilium/hubble-relay"
      useDigest: false
  ui:
    enabled: true
    backend:
      image:
        repository: "harbor.services.wait/quay.io/cilium/hubble-ui-backend"
        useDigest: false
    frontend:
      image:
        repository: "harbor.services.wait/quay.io/cilium/hubble-ui"
        useDigest: false
  metrics:
    enabled: [dns,drop,tcp,flow,port-distribution,icmp,http]
prometheus:
  enabled: true
socketLB:
  enabled: true
  hostNamespaceOnly: true
l2announcements:
  enabled: true
  leaseDuration: 3s
  leaseRenewDeadline: 1s
  leaseRetryPeriod: 500ms
externalIPs:
  enabled: true
devices: enp+

# 开启 l2pod通告
l2podAnnouncements:
  enabled: true
  # interface: "enp6s18"
  # 新版本只支持 interfacePattern 字段语法
  interfacePattern: "enp6s18"

gatewayAPI:
  enabled: true

LoadBalancer 配置

简述

  1. 创建地址池
  2. 地址分配策略

地址池

apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
  name: "pool-30"
spec:
  blocks:
  - start: "192.168.5.30"
    stop: "192.168.5.39"

  # 自定义设置, 只有 service 带有这个标签的才能匹配
  serviceSelector:
    matchLabels:
      lbip: cwx

分配策略

apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: cwx-local
spec:
  serviceSelector:
    matchLabels:
      lbip: cwx

  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist

  # 广播的网卡
  interfaces:
    - enp6s18

  loadBalancerIPs: true
  externalIPs: true

gatewayAPI

默认已经有了一个 gatewayClass, 所以只需要创建 gateway 即可

wait@ub05:~$ kubectl get gatewayClass
NAME      CONTROLLER                      ACCEPTED   AGE
cilium    io.cilium/gateway-controller    True       3h23m
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: cilium-gateway
  namespace: cwx
spec:
  # 传递给服务, 使用前面地址池的地址
  infrastructure:
    labels:
      lbip: cwx
    annotations:
      io.cilium/lb-ipam-ips: "192.168.5.32"

  gatewayClassName: cilium
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: Same

  - name: https
    protocol: HTTPS
    port: 443
    hostname: "*.services.wait"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: key-services-wait
        group: ""

验证

# 这个是默认的 gatewayClass
wait@ub05:~$ kubectl get gatewayClass
NAME      CONTROLLER                      ACCEPTED   AGE
cilium    io.cilium/gateway-controller    True       3h23m


wait@ub05:~$ kubectl get gateway
NAME              CLASS     ADDRESS        PROGRAMMED   AGE
cilium-gateway    cilium    192.168.5.32   True         67m

# 确认有获取到 lb 地址池指定的ip地址
wait@ub05:~$ kubectl get svc
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                         AGE
cilium-gateway-cilium-gateway   LoadBalancer   10.11.231.169   192.168.5.32   80:32070/TCP,443:31454/TCP                      67m

配置http路由

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web6-route
  namespace: cwx
spec:
  parentRefs:
    - name: cilium-gateway
      # sectionName: https  # 不设置 sectionName 就同时支持 http 和 https
  hostnames:
    - "web6.services.wait"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: web5
          port: 80

域名解析, 将 web6.services.wait 指向 192.168.5.32

访问成功

# https
# curl -i http://web6.services.wait/v1/address?uid=1
curl -i https://web6.services.wait/v1/address?uid=1
HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
date: Mon, 30 Mar 2026 17:40:21 GMT
content-length: 68
x-envoy-upstream-service-time: 1
server: envoy

{"code":200,"message":"成功","data":{"uid":1,"address":"beijing"}}

一些环境检查

# 检查所有 HTTPRoute 状态
kubectl get httproute --all-namespaces

# show gateway
kubectl describe gateway cilium-gateway

# 数据面状态
kubectl -n kube-system exec pods/cilium-g7k6f -- cilium status


# 查看集群内 httproute 所配置的域名
kubectl get httproute --all-namespaces -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,HOST:.spec.hostnames[*]'

问题

  1. 跨命名空间时, 注意 referencegrant 设置资源访问权限

  2. Gateway 获得 IP失败, 检查 ciliuml2announcementpolicy 广播策略, 主要是那个网卡选择

最后更新于
版本升级k8s常见异常处理