Forem: opmlee

云原生 EgressGateway：适用于 Calico，Flannel，Weave 和 Spiderpool 等 CNI 的出口网关

opmlee — Fri, 29 Dec 2023 10:14:05 +0000

Calico 是一套开源的网络和网络安全解决方案，也是作为 Kubernetes 容器网络解决方案 (CNI: Container Network Interface) 的一种实现。

当前痛点

在多个集群互相访问，企业管理员希望规范化跨集群网络请求。如下图集群 B 中的 Pod 为一组应用，管理员希望右侧同一业务组的 Pod 出集群的流量都使用特定 IP。

EgressGateway 解决方式

👋 针对上述的需求，我们使用 https://github.com/spidernet-io/egressgateway 来解决这个问题。EgressGateway 是一个开源的云原生出口网关，兼容如下 Calico，Flannel，Weave 和 Spiderpool 的 CNI 网络方案。EgressGateway 有如下特性：

支持通过 EgressPolicy 配置不通租户的的 Egress 策略
支持通过 EgressClusterPolicy 集群级 Egress 策略；
支持 IPv4/IPv6 双栈；
拥有低内核兼容性；
自动检测集群内部流量；
支持设置租户/集群默认 EgressGateway。

如果你想了解更多详情，请访问 EgressGateway https://spidernet-io.github.io/egressgateway/dev/ 文档站。

实现原理

在数据面，我们通过实施了如下 datapath 来实现 EgressGateway 的功能。左侧节点被 EgressPolicy 策略命中的 Pod 流量，会通过 iptabels, route, ipset 规则并通过 vxlan 转发到网关节点，然后从网关节点将源 IP 转为 Egress IP 离开集群。

安装实施

将 Calico 的的 FELIX_CHAININSERTMODE 设置为 Append。

   kubectl set env daemonset -n kube-system calico-node FELIX_CHAININSERTMODE=Append

安装 EgressGateway。

   helm repo add egressgateway https://spidernet-io.github.io/egressgateway/
   helm repo update

   helm install egressgateway egressgateway/egressgateway \
        -n kube-system \
        --set feature.tunnelIpv4Subnet="192.200.0.1/16" \ # 选择一个没有被使用的虚拟子网
        --wait --debug

验证 EgressGateway 都已经启动并就绪。

   $ kubectl get pod -n kube-system | grep egressgateway
   egressgateway-agent-29lt5                  1/1     Running   0          9h
   egressgateway-agent-94n8k                  1/1     Running   0          9h
   egressgateway-agent-klkhf                  1/1     Running   0          9h
   egressgateway-controller-5754f6658-7pn4z   1/1     Running   0          9h

选择一个或者多个节点作为出口网关。我们通过给 node 打上特点的 label 来标识这些节点。

   kubectl get node -o wide
   kubectl label node $NODE_NAME egressgateway="true"

创建 EgressGateway 实例。在创建 EgressGateway 时，我们在 spec.nodeSelector 中将指定网关节点为上面打 label 的节点，并为这些网关分配了可用的 spec.ippools 的地址。

   cat <<EOF | kubectl apply -f -
   apiVersion: egressgateway.spidernet.io/v1beta1
   kind: EgressGateway
   metadata:
     name: default
   spec:
     ippools:
       ipv4:
       - "172.22.0.100-172.22.0.110"
     nodeSelector:
       selector:
         matchLabels:
           egressgateway: "true"
   EOF

执行下面命令检查刚才创建的网关，我们可以在 status 中看到已经选择的 2 个节点已经就绪。稍后我们可以通过创建 EgressPolicy CR 来选择可使用的 EgressGateway，并通过 label 匹配业务 Pod，使业务流量通过网关节点离开集群。

   $ kubectl get EgressGateway default -o yaml
   apiVersion: egressgateway.spidernet.io/v1beta1
   kind: EgressGateway
   metadata:
     name: default
     uid: 7ce835e2-2075-4d26-ba63-eacd841aadfe
   spec:
     ippools:
       ipv4:
       - 172.22.0.100-172.22.0.110
       ipv4DefaultEIP: 172.22.0.110
     nodeSelector:
       selector:
         matchLabels:
           egressgateway: "true"
   status:
     nodeList:
     - name: egressgateway-worker1
       status: Ready
     - name: egressgateway-worker2
       status: Ready

在开始之前我们先在集群外的任意机器，启动一个容器模拟被访问的外部服务。

   docker run -d --net=host ghcr.io/spidernet-io/egressgateway-nettools:latest /usr/bin/nettools-server -protocol web -webPort 8080

我们在集群内部创建一个 Deployment，用于模拟需要访问外部使用统一 Egress IP 的业务应用。

   kubectl create deployment visitor --image nginx

此时我们创建一个 EgressPolicy，来为其实施 Egress 策略。我们通过 spec.egressGatewayName 制定了上面创建的 EgressGateway 名称。

   cat <<EOF | kubectl apply -f -
   apiVersion: egressgateway.spidernet.io/v1beta1
   kind: EgressPolicy
   metadata:
    name: test
    namespace: default
   spec:
    egressGatewayName: default
    appliedTo:
     podSelector:
      matchLabels:
       app: "visitor"
   EOF

通过 kubectl exec 进我们创建的业务应用，执行测试。
```
kubectl exec -it visitor-6764bb48cc-29vq9 bash
$ curl 10.6.1.92:8080
Remote IP: 172.22.0.110
```
🎉 通过结果看到，我们的 Egress IP 已经变为了 EgressGateway 池中配置的 IP。当然也可以在创建 EgressPolicy 指定使用的 Egress IP。

总结

在本文中我们分析了 Calico 网络下的 Egress IP 网络需求，并通过 EgressGateway 来解决这个问题，同时对 EgressGateway 的网络原理做了阐述，最后我们介绍了 EgressGateway 的更多功能。

Kubernetes Egress Gateway

opmlee — Thu, 30 Nov 2023 09:35:24 +0000

Starting with 2021, we received some feedback as follows.

There are two clusters A and B. Cluster A is VMWare-based and runs mainly Database workloads, and Cluster B is a Kubernetes cluster. Some applications in Cluster B need to access the database in Cluster A, and the network administrator wants the cluster Pods to be managed through an egress gateway.

We have developed and open-sourced the EgressGateway project to address this issue.

Introduce

In a Kubernetes cluster, when Pods access external services, their Egress IP addresses are not fixed. In the Overlay network, the Egress IP address is determined by the node where the Pod resides. While in the Underlay network, Pods directly use their own IP addresses for external communication. Consequently, when Pods are rescheduled, regardless of the network mode, their IP addresses for external communication change. This instability poses a challenge for system administrators in managing IP addresses, especially as the cluster scales and during network fault diagnostics. Controlling egress traffic based on a Pod's original egress IP outside the cluster becomes difficult.

To solve this problem, EgressGateway has been introduced into the k8s cluster. It is an open-source EgressGateway designed to resolve egress egress IP address issues across various CNI network modes, such as Calico, Flannel, Weave, and Spiderpool. Through flexible configuration and management of egress policies, EgressGateway allows setting egress IP addresses for tenant-level or cluster-level workloads. When Pods need to access the external network, the system consistently uses the configured Egress IP as the egress address, providing a stable solution for egress traffic management.

Compatible with the following network solutions

Features

Solve IPv4 IPv6 dual-stack connectivity.
Solve the high availability of Egress Nodes.
Allow filtering Pods Egress Policy (Destination CIDR).
Allow filtering of egress Applications (Pods).
Can be used in low kernel version.
Support multiple egress gateways instance.
Support namespaced egress IP.
Supports automatic detection of cluster traffic for egress gateways policies.
Support namespace default egress instances.

Quick Start

Prerequisites

A self-managed Kubernetes cluster with a minimum of 2 nodes.
Helm has been installed in your cluster.
Let's take calico CNI as an example. For more you can ref to our Docs Website
1. If your cluster is using Calico as the CNI plugin, run the following command to ensure that EgressGateway's iptables rules are not overridden by Calico rules. Failure to do so may cause EgressGateway to malfunction.

  # set chainInsertMode
  $ kubectl patch felixconfigurations  default --type='merge' -p '{"spec":{"chainInsertMode":"Append"}}'

  # check status
  $ kubectl get FelixConfiguration default -o yaml
    apiVersion: crd.projectcalico.org/v1
    kind: FelixConfiguration
    metadata:
      generation: 2
      name: default
      resourceVersion: "873"
      uid: 0548a2a5-f771-455b-86f7-27e07fb8223d
      spec:
        chainInsertMode: Append
      ......

Regarding spec.chainInsertMode, refer to Calico docs for details

Install EgressGateway

Add EgressGateway Repo

helm repo add egressgateway https://spidernet-io.github.io/egressgateway/

helm repo update

Install EgressGateway

Quickly install EgressGateway through the following command:
```
helm install egressgateway egressgateway/egressgateway \
      -n kube-system \
        --set feature.tunnelIpv4Subnet="192.200.0.1/16" \
        --wait --debug
```
In the installation command, please consider the following points:
- Make sure to provide the IPv4 and IPv6 subnets for the EgressGateway tunnel nodes in the installation command. These subnets should not conflict with other addresses within the cluster.
- You can customize the network interface used for EgressGateway tunnels by using the --set feature.tunnelDetectMethod="interface=eth0" option. By default, it uses the network interface associated with the default route.
- If you want to enable IPv6 support, set the --set feature.enableIPv6=true option and also feature.tunnelIpv6Subnet.
- The EgressGateway Controller supports high availability and can be configured using --set controller.replicas=2.
- To enable return routing rules on the gateway nodes, use --set feature.enableGatewayReplyRoute=true. This option is required when using Spiderpool to work with underlay CNI.

Verify that all EgressGateway Pods are running properly.

$ kubectl get pod -n kube-system | grep egressgateway
egressgateway-agent-29lt5                  1/1     Running   0          9h
egressgateway-agent-94n8k                  1/1     Running   0          9h
egressgateway-agent-klkhf                  1/1     Running   0          9h
egressgateway-controller-5754f6658-7pn4z   1/1     Running   0          9h

Any feature configurations can be achieved by adjusting the Helm values of the EgressGateway application.

Create EgressGateway release

EgressGateway defines a group of nodes as the cluster's egress gateway, responsible for forwarding egress traffic out of the cluster. To define a group of EgressGateway, run the following command:
```
cat <<EOF | kubectl apply -f -
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressGateway
metadata:
  name: default
spec:
  ippools:
    ipv4:
    - "172.22.0.100-172.22.0.110"
  nodeSelector:
    selector:
      matchLabels:
        egressgateway: "true"
EOF
```
Descriptions:
- In the provided YAML example, adjust spec.ippools.ipv4 to define egress exit IP addresses based on your specific environment.
- Ensure that the CIDR of spec.ippools.ipv4 matches the subnet of the egress interface on the gateway nodes (usually the interface associated with the default route). Mismatched subnets can cause connectivity issues for egress traffic.
- Use spec.nodeSelector in the EgressGateway to select a group of nodes as the egress gateway. You can select multiple nodes to achieve high availability.
Label the egress gateway nodes by applying labels to them. For production environments, it is recommended to label at least 2 nodes. For POC environments, label 1 node.
```
kubectl get node
kubectl label node $NodeName egressgateway="true"
```

Check the status:

$ kubectl get EgressGateway default -o yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressGateway
metadata:
  name: default
  uid: 7ce835e2-2075-4d26-ba63-eacd841aadfe
spec:
  ippools:
    ipv4:
    - 172.22.0.100-172.22.0.110
    ipv4DefaultEIP: 172.22.0.110
  nodeSelector:
    selector:
      matchLabels:
        egressgateway: "true"
status:
  nodeList:
  - name: egressgateway-worker1
    status: Ready
  - name: egressgateway-worker2
    status: Ready

Descriptions:

The status.nodeList field indicates the nodes that match the spec.nodeSelector, along with the status of their corresponding EgressTunnel objects.
The spec.ippools.ipv4DefaultEIP field randomly selects one IP address from spec.ippools.ipv4 as the default VIP for this group of EgressGateways. This default VIP is used when creating EgressPolicy objects for applications that do not specify a VIP address.

Create Applications and Egress Policies

Create an application that will be used to test Pod access to external resources and apply labels to it.
```
kubectl create deployment visitor --image nginx
```
Create an EgressPolicy CR object for your application.

An EgressPolicy instance is used to define which Pods' egress traffic should be forwarded through EgressGateway nodes, along with other configuration details.
You can create an example as follows. When a matching Pod accesses any external address in the cluster (excluding Node IP, CNI Pod CIDR, ClusterIP), it will be forwarded through EgressGateway nodes.
Note that EgressPolicy objects are tenant-level, so they must be created under the tenant of the selected application.
```
cat <<EOF | kubectl apply -f -
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
 name: test
 namespace: default
spec:
 egressGatewayName: default
 appliedTo:
  podSelector:
   matchLabels:
    app: "visitor"
EOF
```
Descriptions:
- spec.egressGatewayName specifies the name of the EgressGateway group to use.
- spec.appliedTo.podSelector determines which Pods within the cluster this policy should apply to.
- There are two options for the source IP address of egress traffic in the cluster:
  - You can use the IP address of the gateway nodes. This is suitable for public clouds and traditional networks but has the downside of potential IP changes if a gateway node fails. You can enable this by setting spec.egressIP.useNodeIP=true.
  - You can use a dedicated VIP. EgressGateway uses ARP principles for VIP implementation, making it suitable for traditional networks rather than public clouds. The advantage is that the egress source IP remains fixed. If no settings are specified in the EgressPolicy, the default VIP of the egressGatewayName will be used, or you can manually specify spec.egressIP.ipv4 , which must match the IP pool configured in the EgressGateway.

Check the status of the EgressPolicy

$ kubectl get EgressPolicy -A
NAMESPACE   NAME   GATEWAY   IPV4           IPV6   EGRESSTUNNEL
default     test   default   172.22.0.110          egressgateway-worker2

$ kubectl get EgressPolicy test -o yaml
apiVersion: egressgateway.spidernet.io/v1beta1
kind: EgressPolicy
metadata:
  name: test
  namespace: default
spec:
  appliedTo:
    podSelector:
      matchLabels:
        app: visitor
  egressIP:
    allocatorPolicy: default
    useNodeIP: false
status:
  eip:
    ipv4: 172.22.0.110
  node: egressgateway-worker2

Descriptions:

status.eip displays the egress IP address used by the group of applications.
status.node shows which EgressGateway node is responsible for real-time egress traffic forwarding. EgressGateway nodes support high availability. When multiple EgressGateway nodes exist, all EgressPolicy instances will be evenly distributed among them.

Check the status of EgressEndpointSlices.

Each EgressPolicy object has a corresponding EgressEndpointSlices that stores the IP collection of Pods selected by the EgressPolicy. If your application is unable to access external resources, you can check if the IP addresses in this object are correct.

$ kubectl get egressendpointslices -A
NAMESPACE   NAME         AGE
default     test-kvlp6   18s

$ kubectl get egressendpointslices test-kvlp6 -o yaml
apiVersion: egressgateway.spidernet.io/v1beta1
endpoints:
- ipv4:
  - 172.40.14.195
  node: egressgateway-worker
  ns: default
  pod: visitor-6764bb48cc-29vq9
kind: EgressEndpointSlice
metadata:
  name: test-kvlp6
  namespace: default

Test Results

Deploy the nettools application outside the cluster to simulate an external service. nettools will return the requester's source IP address in the HTTP response.
```
docker run -d --net=host ghcr.io/spidernet-io/egressgateway-nettools:latest /usr/bin/nettools-server -protocol web -webPort 8080
```

Verify the effect of egress traffic in the visitor Pod within the cluster. You should observe that when the visitor accesses the external service, nettools returns a source IP matching the EgressPolicy .status.eip.

$ kubectl get pod
NAME                       READY   STATUS    RESTARTS   AGE
visitor-6764bb48cc-29vq9   1/1     Running   0          15m

$ kubectl exec -it visitor-6764bb48cc-29vq9 bash
$ curl 10.6.1.92:8080
Remote IP: 172.22.0.110

Community

If you have any egress network requirements, feel free to discuss them with us in our community. EgressGateway is an open-source project, and if you find value in it, we welcome you to star our project as a sign of support.