1
# Preface
2

3
I've actually wanted to play with a K8s cluster for a long time, but always felt that without sufficient knowledge, it would be too difficult to attempt.
4
Recently, I spent some time studying DN42 and routing protocols like BGP and OSPF, and realized that it no longer feels so difficult. So I decisively started with K3s (
5

6
The main reason for choosing K3s over K8s is its lightweight nature: low resource requirements, no need to pull a bunch of images for deployment, availability of domestic mirrors… In short, K3s suits my needs better.
7

8
I'm a beginner just starting to explore K3s, so please go easy on me if I make any mistakes~
9

10
# Analysis
11

12
## Choosing the CNI Component
13

14
My current network architecture looks like this:
15

16
```mermaid
17
graph TD
18
    subgraph ZeroTier Domestic
19
        subgraph WDS
20
            Gateway <--> VM1
21
            Gateway <--> VM2
22
        end
23
        NGB <--> Gateway
24
        HFE-NAS <--> Gateway
25
        NGB <--> HFE-NAS
26
    end
27

28
    subgraph IEPL
29
        Global-NIC <==OSPF==> CN-NIC
30
    end
31

32
    subgraph ZeroTier Global
33
        HKG02 <-->  HKG04
34
        TYO <--> HKG04
35
        TYO <--> HKG02
36
    end
37

38
    CN-NIC <--> NGB
39
    CN-NIC <--> HFE-NAS
40
    CN-NIC <--OSPF--> Gateway
41

42
    Global-NIC <--OSPF--> TYO
43
    Global-NIC <--OSPF--> HKG02
44
    Global-NIC <--OSPF--> HKG04
45

46
    %% Style definition: orange background, bold border to represent routers
47
    classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold;
48
    class Global-NIC,CN-NIC,Gateway router;

Among this, the WDS node is a Proxmox VE host with multiple VMs underneath. It advertises its VMs’ IPv4 prefixes via OSPF. When Hong Kong nodes need to access a VM under the WDS node, they can do so by joining the OSPF internal network to achieve multi-hop reachability. This keeps the encapsulation layer count to only one, so there’s no worry about MTU “disappearing act”.

I plan to create two new VMs under WDS to serve as the master and a node (temporarily called KubeMaster and KubeNode-WDS1). Then HKG04 (temporarily called KubeNode-HKG04) will also join the K3s cluster as a node.

The simplest approach would be to use K3s’s default Flannel as the CNI. However, Flannel is based on VXLAN, and adding another layer of my existing internal network would lead to the following MTU “disappearing act”:

1
Data packet -> Flannel VXLAN encapsulation -> ZeroTier encapsulation -> Physical link

The actual usable MTU for inter-container communication would likely be compressed to 1350 or even lower. Therefore, I tried to find a CNI solution that can work directly on top of this internal network, and then I found Calico. As I understand, Calico uses BGP as its underlying routing protocol, supports starting in no-encapsulation (No-Encap) mode, and hands packets directly to the upper routers for routing. Thus, I chose Calico as the CNI component.

Routing Design#

To ensure that intermediate routers know how to route Pod IPs, KubeMaster and KubeNode-WDS1 are under the Proxmox VE host. They need to establish BGP with HKG04 across the entire internal network. This means that every router at each intermediate level must learn the full BGP routes, so that the following routing path can be established:

1
graph LR
2

3
subgraph WDS
4
    KubeMaster
5
    KubeNode-WDS1
6
    Gateway
7
end
8

9
subgraph IEPL
10
    CN-Namespace
11
    Global-Namespace
12
end
13

14
KubeNode-WDS1 <--> Gateway
15
KubeMaster <--> Gateway <--> CN-Namespace <--> Global-Namespace <--> HKG04
16

17
%% Style definition: highlight nodes with routing capability
18
classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold;
19
class Gateway,CN-Namespace,Global-Namespace router;

Otherwise, any intermediate hop would drop packets because it doesn’t recognize the source/destination IP. Also, due to the property of iBGP that routes learned from a neighbor cannot be propagated to the next iBGP neighbor, all BGP sessions between Gateway, CN-Namespace, Global-Namespace and the nodes need to enable Route Reflector; otherwise, nodes cannot correctly learn routes from each other.

That said, this architecture would be more suitable for BGP Confederation, but my existing network is already quite complex, and adding BGP confederations would make later maintenance more troublesome. Moreover, my number of nodes is small, so the overhead of iBGP Full Mesh is acceptable. ~~It’s definitely not because I’m lazy (so~~

Thus, the final network routing structure is as follows:

1
graph TD
2
subgraph WDS
3
    VM1
4
    VM2
5
    Gateway
6
end
7

8
subgraph IEPL
9
    CN-Namespace
10
    Global-Namespace
11
end
12

13
VM1 <-.Calico iBGP Full Mesh.-> VM2
14
VM1 <--iBGP Route Reflector--> Gateway
15
VM2 <--iBGP Route Reflector--> Gateway <--iBGP--> CN-Namespace <--iBGP--> Global-Namespace <--iBGP Route Reflector--> HKG04
16
Gateway <--iBGP--> Global-Namespace
17
HKG04 <-.Calico iBGP Full Mesh.-> VM1
18
VM2 <-.Calico iBGP Full Mesh.-> HKG04
19

20
%% Style definition
21
classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold;
22
%% Mark nodes with routing/forwarding or RR functions as Router
23
class Gateway,CN-Namespace,Global-Namespace router;

The dashed-line BGP sessions are automatically created by Calico, while the solid-line parts need to be manually created by us.

Keeping Calico’s own iBGP Full Mesh is for future scalability, so that nodes can preferentially establish direct P2P connections via ZeroTier instead of taking a detour through the Route Reflector aggregation router.

Deployment#

After clarifying the structure, deployment becomes simple.

Enable Kernel Forwarding and Disable rp_filter#

Standard practice.

1
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
2
echo "net.ipv6.conf.default.forwarding=1" >> /etc/sysctl.conf
3
echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf
4
echo "net.ipv4.conf.default.rp_filter=0" >> /etc/sysctl.conf
5
echo "net.ipv4.conf.all.rp_filter=0" >> /etc/sysctl.conf
6
sysctl -p

Install K3s#

Master#

Because the KubeMaster control plane node is located inside China, it’s best to configure image acceleration:

1
mkdir -p /etc/rancher/k3s
2
cat <<EOF > /etc/rancher/k3s/registries.yaml
3
mirrors:
4
  docker.io:
5
    endpoint:
6
      - "https://docker.m.daocloud.io"
7
  quay.io:
8
    endpoint:
9
      - "https://quay.m.daocloud.io"
10
EOF

Install using the mirror:

1
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | \
2
  INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC=" \
3
  --flannel-backend=none \
4
  --disable-network-policy \
5
  --cluster-cidr=10.42.0.0/16" sh -

Note the need to specify --flannel-backend=none and --disable-network-policy to disable the default CNI component.
Use cat /var/lib/rancher/k3s/server/node-token to view the token and record it.

Worker Nodes#

For nodes inside China, configure image acceleration:

1
mkdir -p /etc/rancher/k3s
2
cat <<EOF > /etc/rancher/k3s/registries.yaml
3
mirrors:
4
  docker.io:
5
    endpoint:
6
      - "https://docker.m.daocloud.io"
7
  quay.io:
8
    endpoint:
9
      - "https://quay.m.daocloud.io"
10
EOF

Then install K3s using the mirror and join the cluster:

1
export INSTALL_K3S_MIRROR=cn
2
export K3S_URL=https://<master node IP>:6443  # Replace with your master node's actual IP
3
export K3S_TOKEN=K10...your token...::server:xxx # Replace with the full token obtained in the first step
4

5
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | sh -

At this point, the status of each node should be NotReady because the CNI component is missing.

Install Calico and Configure No-Encap Mode#

On the master, manually download https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml and install the Calico operator:

1
kubectl create -f tigera-operator.yaml

Configure a custom resource by creating a custom-resource.yaml file:

1
apiVersion: operator.tigera.io/v1
2
kind: Installation
3
metadata:
4
  name: default
5
spec:
6
  # Add image registry configuration
7
  registry: quay.m.daocloud.io
8
  calicoNetwork:
9
    ipPools:
10
    - blockSize: 26
11
      cidr: 10.42.0.0/16
12
      encapsulation: None
13
      natOutgoing: Enabled
14
      nodeSelector: all()

Here, specify encapsulation: None to enable No-Encap mode. You can also modify the IPv4 CIDR here if needed. Then:

1
kubectl apply -f custom-resource.yaml

to perform the installation. Use:

1
kubectl get pods -A -o wide

to check Pod status, waiting for each node to finish pulling images.

Configure BGP Topology#

Label Nodes#

Label nodes to specify that nodes under WDS connect to the Gateway’s BGP in the WDS node, and nodes outside China connect to the BGP of the Global Namespace:

1
kubectl label nodes kubemaster region=WDS
2
kubectl label nodes kubenode-wds-1 region=WDS
3
kubectl label nodes kubenode-hkg04 region=Global

Calico Configuration#

Create a YAML configuration file:

1
apiVersion: crd.projectcalico.org/v1
2
kind: BGPPeer
3
metadata:
4
  name: route-reflector-domestic
5
spec:
6
  nodeSelector: region == 'Domestic' # This part is not actually used; I originally designed a general aggregation router in the Domestic area
7
  peerIP: 100.64.0.108
8
  asNumber: 64512
9
---
10
apiVersion: crd.projectcalico.org/v1
11
kind: BGPPeer
12
metadata:
13
  name: route-reflector-wds
14
spec:
15
  nodeSelector: region == 'WDS'
16
  peerIP: 192.168.100.1
17
  asNumber: 64512
18
---
19
apiVersion: crd.projectcalico.org/v1
20
kind: BGPPeer
21
metadata:
22
  name: route-reflector-global
23
spec:
24
  nodeSelector: region == 'Global'
25
  peerIP: 100.64.1.106
26
  asNumber: 64512

This means:

All nodes with label region equal to Domestic will have a BGP session to 100.64.0.108 (the domestic aggregation router) using AS 64512
All nodes with label region equal to WDS will have a BGP session to 192.168.100.1 (the Gateway for all VMs under the WDS node) using AS 64512
All nodes with label region equal to Global will have a BGP session to 100.64.1.106 (the overseas aggregation router) using AS 64512

This achieves what is shown in the diagram: all VMs under the WDS node, including the master and KubeNode-WDS1, connect to the Gateway aggregation router of the WDS node, and all nodes in overseas areas connect to the overseas aggregation router.

Configure Aggregation Router iBGP#

This part is simply a matter of writing Bird configuration files (easy).
Here are a few examples:

k3s/ibgp.conf:

1
function is_insider_as(){
2
    if bgp_path.len > 0 && !(bgp_path ~ [= 64512 =]) then {
3
        return false;
4
    }
5

6
    if net ~ [ 10.42.0.0/16{16,32} ] then {
7
        return true;
8
    }
9

10
    return false;
11
}
12

13
template bgp k3sbackbone{
14
    local as K3S_AS;
15
    router id INTRA_ROUTER_ID;
16
    neighbor as K3S_AS;
17

18
    ipv4{
19
        table intra_table_v4;
20
        import filter{
21
            if is_insider_as() then accept;
22
            reject;
23
        };
24
        export filter{
25
            if is_insider_as() then accept;
26
            reject;
27
        };
28
        next hop self;
29
        extended next hop;
30
    };
31

32
    ipv6{
33
        table intra_table_v6;
34
        import filter{
35
            if is_insider_as() then accept;
36
            reject;
37
        };
38
        export filter{
39
            if is_insider_as() then accept;
40
            reject;
41
        };
42
        next hop self;
43
    };
44
};
45

46
template bgp k3speers{
47
    local as K3S_AS;
48
    neighbor as K3S_AS;
49
    router id INTRA_ROUTER_ID;
50
    rr client;
51
    rr cluster id INTRA_ROUTER_ID;
52

53
    ipv4{
54
        table intra_table_v4;
55
        import filter{
56
            if is_insider_as() then accept;
57
            reject;
58
        };
59
        export filter{
60
            if is_insider_as() then accept;
61
            reject;
62
        };
63
        next hop self;
64
    };
65

66
    ipv6{
67
        table intra_table_v6;
68
        import filter{
69
            if is_insider_as() then accept;
70
            reject;
71
        };
72
        export filter{
73
            if is_insider_as() then accept;
74
            reject;
75
        };
76
        next hop self;
77
    };
78
};
79

80
include "ibgpeers/*";

ibgpeers/backbone-cn.conf:

1
protocol bgp 'k3s_backbone_cn_v4' from k3sbackbone{
2
    neighbor fd18:3e15:61d0:cafe:f001::1;
3
};

ibgpeers/master.conf:

1
protocol bgp 'k3s_master_v4' from k3speers{
2
    neighbor 192.168.100.251;
3
};

Main points: it’s best not to enable Route Reflector between the aggregation routers, and remember to enable next hop self.

After everything is done, using kubectl get nodes should show all nodes as Ready:

1
NAME             STATUS   ROLES           AGE     VERSION
2
kubemaster       Ready    control-plane   2d23h   v1.34.5+k3s1
3
kubenode-hkg04   Ready    <none>          11h     v1.34.6+k3s1
4
kubenode-wds-1   Ready    <none>          2d7h    v1.34.5+k3s1

Use kubectl get pods -A -o wide to view Pods:

1
NAMESPACE         NAME                                       READY   STATUS      RESTARTS        AGE     IP                NODE             NOMINATED NODE   READINESS GATES
2
calico-system     calico-kube-controllers-64fc874957-6bdlz   1/1     Running     0               5h38m   10.42.253.136     kubenode-hkg04   <none>           <none>
3
calico-system     calico-node-2qz82                          1/1     Running     0               4h24m   10.2.5.7          kubenode-hkg04   <none>           <none>
4
calico-system     calico-node-dhl2c                          1/1     Running     0               4h24m   192.168.100.251   kubemaster       <none>           <none>
5
calico-system     calico-node-nbpkj                          1/1     Running     0               4h23m   192.168.100.252   kubenode-wds-1   <none>           <none>
6
calico-system     calico-typha-7bb5db4bdc-rfpwg              1/1     Running     0               5h38m   10.2.5.7          kubenode-hkg04   <none>           <none>
7
calico-system     calico-typha-7bb5db4bdc-rwwr5              1/1     Running     0               5h38m   192.168.100.251   kubemaster       <none>           <none>
8
calico-system     csi-node-driver-jglwp                      2/2     Running     0               5h38m   10.42.64.68       kubenode-wds-1   <none>           <none>
9
calico-system     csi-node-driver-jqjsc                      2/2     Running     0               5h38m   10.42.253.137     kubenode-hkg04   <none>           <none>
10
calico-system     csi-node-driver-vk26s                      2/2     Running     0               5h38m   10.42.141.16      kubemaster       <none>           <none>
11
kube-system       coredns-695cbbfcb9-8fx4p                   1/1     Running     1 (7h27m ago)   2d23h   10.42.141.14      kubemaster       <none>           <none>
12
kube-system       helm-install-traefik-crd-5bkwx             0/1     Completed   0               2d23h   <none>            kubemaster       <none>           <none>
13
kube-system       helm-install-traefik-m9fgj                 0/1     Completed   1               2d23h   <none>            kubemaster       <none>           <none>
14
kube-system       local-path-provisioner-546dfc6456-dmn4g    1/1     Running     1 (7h27m ago)   2d23h   10.42.141.15      kubemaster       <none>           <none>
15
kube-system       metrics-server-c8774f4f4-2wkwh             1/1     Running     1 (7h27m ago)   2d23h   10.42.141.12      kubemaster       <none>           <none>
16
kube-system       svclb-traefik-999cddce-hpmcm               2/2     Running     6 (7h26m ago)   11h     10.42.253.134     kubenode-hkg04   <none>           <none>
17
kube-system       svclb-traefik-999cddce-q4225               2/2     Running     2 (7h27m ago)   2d22h   10.42.141.9       kubemaster       <none>           <none>
18
kube-system       svclb-traefik-999cddce-xmd64               2/2     Running     2 (7h26m ago)   2d6h    10.42.64.66       kubenode-wds-1   <none>           <none>
19
kube-system       traefik-788bc4688c-vbbhj                   1/1     Running     1 (7h27m ago)   2d22h   10.42.141.13      kubemaster       <none>           <none>
20
tigera-operator   tigera-operator-6b95bbf4db-vl46l           1/1     Running     1 (7h27m ago)   2d23h   192.168.100.251   kubemaster       <none>           <none>

Use kubectl exec -it -n calico-system <calico-node-xxxx> -- birdcl s p to check the status of Bird:

1
root@KubeMaster:~/kube/calico# kubectl exec -it -n calico-system calico-node-2qz82 -- birdcl s p
2
Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init)
3
BIRD v0.3.3+birdv1.6.8 ready.
4
name     proto    table    state  since       info
5
static1  Static   master   up     08:58:17
6
kernel1  Kernel   master   up     08:58:17
7
device1  Device   master   up     08:58:17
8
direct1  Direct   master   up     08:58:17
9
Mesh_192_168_100_251 BGP      master   up     08:58:33    Established
10
Mesh_192_168_100_252 BGP      master   up     08:59:00    Established
11
Node_100_64_1_106 BGP      master   up     12:57:44    Established

ip r shows the system routing table:

1
root@KubeMaster:~/kube/calico# ip r
2
default via 192.168.100.1 dev eth0 proto static
3
10.42.64.64/26 proto bird
4
        nexthop via 192.168.100.1 dev eth0 weight 1
5
        nexthop via 192.168.100.252 dev eth0 weight 1
6
blackhole 10.42.141.0/26 proto bird
7
10.42.141.9 dev caliac6501d3794 scope link
8
10.42.141.12 dev calib07c23291bb scope link
9
10.42.141.13 dev caliab16e60bd19 scope link
10
10.42.141.14 dev calid5959219080 scope link
11
10.42.141.15 dev cali026d8f1ddb7 scope link
12
10.42.141.16 dev califa657ba417a scope link
13
10.42.253.128/26 via 192.168.100.1 dev eth0 proto bird
14
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.251

Ping a Pod’s IP – if everything is fine, it should work directly:

1
root@KubeMaster:~/kube/calico# ping 10.42.253.137
2
PING 10.42.253.137 (10.42.253.137) 56(84) bytes of data.
3
64 bytes from 10.42.253.137: icmp_seq=1 ttl=60 time=33.7 ms
4
64 bytes from 10.42.253.137: icmp_seq=2 ttl=60 time=33.5 ms
5
^C
6
--- 10.42.253.137 ping statistics ---
7
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
8
rtt min/avg/max/mdev = 33.546/33.632/33.718/0.086 ms

Tune MTU#

This step is actually for stability…?
Tests have shown that although my ZeroTier MTU is 1420, packets start to fragment around 1392 bytes (test with ping -M do -s <packet size> <Pod_IP>). Therefore, force the Pod MTU to 1370:

1
root@KubeMaster:~/kube/calico# cat patch-mtu.yaml
2
apiVersion: operator.tigera.io/v1
3
kind: Installation
4
metadata:
5
  name: default
6
spec:
7
  calicoNetwork:
8
    mtu: 1370
9
    nodeAddressAutodetectionV4:
10
      firstFound: true
11
root@KubeMaster:~/kube/calico# kubectl apply -f patch-mtu.yaml
12
installation.operator.tigera.io/default configured