DN42&OneManISP – Confederation, Automation, IPAM and Overlay Network This article does not go into implementation details in depth; it only records the internal network architecture design process and automation ideas. If you need reference code or configuration files, please see iYoRoy-Network/bird2-config: BIRD2 Configuration for iYoRoy Network (AS4242422024, AS205369). The repository contains a significant amount of AI-generated code. This article only borrows the logical layering concept of Underlay/Overlay (base layer/business layer), specifically referring to the separation of address roles, and does not involve tunneling encapsulation technologies such as VXLAN/GRE. TL;DR The entire internal network architecture was refactored. What was originally a horizontally split network based on “DN42 / IANA / WireGuard / BIRD config files” was transformed into a set of infrastructure organised by network intent. The final design can be summarised as: Use WireGuard to carry underlay transport between nodes; Use BGP Confederation instead of OSPF / full-mesh iBGP as the internal routing backbone; Separate node identity and service addresses with Underlay / Overlay; Automatically derive AS, loopback and link-local addresses from Tier / Region / Node ID / Token; Use BGP Large Community to express route origin, propagation scope and export policy; Use Ansible to compile high-level peer intent into WireGuard + BIRD + interface configurations and deploy them automatically. Background In some earlier articles from these two series, we successfully brought up a DN42 AS and an IANA AS, with the same infrastructure carrying traffic for both networks simultaneously. In a previous post, DN42&OneManISP - Troubleshooting OSPF Source Address in a Coexistence Environment - iYoRoy's Develop Diary, there was already some initial separation and isolation for the two networks. However, as the number of nodes grew, manually maintaining the WireGuard and BIRD configuration files for this whole thing became more and more complex. Moreover, public network nodes also needed to interconnect and provide mutual IP transit, so in the end I decided to do a thorough refactoring of this entire internal network. Analysis Problems with the Old Architecture Before the refactoring, DN42 routes between nodes were mutually propagated, using OSPF over WireGuard as the IGP, with full-mesh iBGP on top to carry full routing information between border routers. For IANA traffic, nodes were completely isolated from each other, with each node independently advertising its own /48 IANA IPv6 prefix. To sum up: WireGuard was responsible for tunnels between nodes; OSPF over WireGuard was responsible for the IGP; DN42 border routers ran full-mesh iBGP on top; IANA nodes basically each independently advertised their own public prefixes. This architecture worked well when there were few nodes, but problems started to become apparent as the number of nodes increased: WireGuard, OSPF, and BIRD configurations had to be maintained separately; Adding a new node required changes in multiple places, making it easy to miss something; Routing policies for DN42 and IANA influenced each other, yet the configuration was scattered; Cross-PoP scheduling of IANA prefixes was not natural; Some nodes were only transit nodes but were forced to have an overlay identity; Source address selection, route leaks, and exposure of internal addresses became increasingly difficult to control. Design Goals and BGP Confederation The main goals for this refactoring were: Adding nodes should be simple: ideally, only the node identity and peer relationships need to be described; DN42, IANA and the internal backbone should be able to share infrastructure while remaining policy-isolated from each other; IP addresses should be schedulable resources, not fixed identities of nodes; The internal topology should not leak to external BGP peers; Configurations should be generated from declarative data, not by handwriting lots of repetitive WireGuard/BIRD sessions; The deployment process should be automated. There were also several other, more detailed requirements: Some IANA nodes should be able to carry IP transit for other nodes, using an IANA PoP to advertise public IPs for other nodes; Currently, the IANA AS is maintained jointly by たのしい and myself; we need to coordinate, with some infrastructure needing mutual transit and route tuning; We only have one 44Net IPv4 /24 (sponsored by たのしい); if we want to advertise it, all nodes would need iBGP interconnection and an IGP, otherwise only a single PoP can use it; The IANA AS needs to accept downstreams; Route optimization is needed between IANA PoPs. The previous IANA IPv6 prefix assignment rule followed Continent 3 bits + Region 3 bits + Multi-node within region 2 bits, carving several /48s out of a complete /40. However, due to certain upstream limitations on the number of advertised prefixes, and because such fragmented prefixes are unfavourable for IPAM, we planned to eventually aggregate external advertisements only to the continent level, and then handle more detailed parts via internal iBGP routing. The internal network structure on たのしい's side is BGP Confederation, implemented with a manually simulated approach. Actually, after this analysis, defining the internal network architecture became quite straightforward: the most suitable solution was to switch the internal network to BGP Confederation as well, and then utilise the concept of "BGP as IGP" to advertise /32 and /128 routes within the confederation to handle internal routing. At the same time, because a confederation avoids the split-horizon characteristic of traditional iBGP, it doesn't require strict full-mesh, nor does it need separate iBGP instances, making it very decentralized xD. Furthermore, BGP Confederation can be easily merged with たのしい's infrastructure, requiring only that both sides treat the other's internal ASes as confederation ASes. This solution has another benefit, because the containment relationship of our infrastructure is actually like this: graph subgraph 4242423377 Infrastructure 3377_DN42_PoP[DN42 PoP] 3377_IANA_PoP[IANA PoP] end subgraph 4242422024 Infrastructure 2024_DN42_PoP[DN42 PoP] 2024_IANA_PoP[IANA PoP] end 3377_IANA_PoP <==Cross-Infra Confederation==> 2024_IANA_PoP 3377_DN42_PoP <--> DN42 <--> 2024_DN42_PoP 3377_IANA_PoP <--> IANA <--> 2024_IANA_PoP Our DN42 PoPs and transit are isolated, but the IANA PoPs and transit are interconnected. In the BGP Confederation scenario, we can use filters + BGP (Large) Community to isolate route origins while still exchanging IANA traffic engineering intents, achieving a single standard usable across both infrastructures. Overlay and IPAM In the old design, a node often simultaneously held two identities: It was a router in the network; It was also a service address holder reachable within DN42 / IANA. After the refactoring, I decided to separate these two identities: Underlay: the node's identity as a router, used for internal interconnection, next-hop, tunnels, and forwarding; Overlay: the DN42 / IANA address used when the node provides services to the outside world or accepts traffic. This way, a node can only participate in forwarding without having a DN42 IPv4 address; it can also temporarily announce a certain overlay /32 or /128 when needed. In practical application scenarios, my DN42 address space is not very abundant; I originally registered a /28, with only 16 usable addresses. Some nodes, like IEPL tunnel nodes or IX nodes, only perform forwarding roles, do not need to host services, and do not need to be connected from the outside. Therefore, theoretically they do not need to be assigned independent, reachable DN42/IANA addresses. Moreover, since we had already decided to switch the internal network to BGP Confederation, based on this configuration we could re-plan the originally horizontally split DN42 + IANA network structure into a vertical Underlay + Overlay division. DN42 and IANA addresses/traffic become the payload of the upper-layer Overlay Network, while the lower-layer Underlay Network serves as the infrastructure for underlying communication and forwarding between nodes. In this scenario, IP addresses become resources that can be easily scheduled within the internal network, rather than unique IDs bound to a node. The advantages are clear: Saves scarce address resources: transit nodes can have only an underlay identity, without needing DN42 IPv4 or public addresses; Anycast and address migration: Overlay addresses become resources schedulable via BGP, no longer tightly bound to a specific machine; Hides transit nodes: pure transit nodes do not need to expose globally reachable addresses, making it harder for outsiders to directly probe the internal topology; Easy new node onboarding: new nodes join the underlay first, and overlay addresses are assigned when they need to carry services; Elegant transit delivery: IANA PoPs can deliver customer prefixes or service addresses to other nodes via the internal confederation. Design In summary, the final design consisted of the following IPAM mechanism and automation workflow. Each node possesses some basic metadata: Tier: indicates the node's purpose, e.g., backbone, IX access, or intra-net node Region: the continent region where the node is located ID: the node's ID within that region Token: a random 16-bit string assigned to the node, used as a unique ID for the node under IPv6 For Tier and Region, several tables were created as data sources: Tier Desc 1 Backbone 2 IX 3 Backbone + IX Mixed 9 Node Region Desc 0 Reserved 1 Asia 2 Europe 3 Africa 4 North America 5 South America 6 Oceania 7 Antarctica Confederation AS: 4220240000 – 4220249999 Assignment rule: 422024{tier}{region}{node_id:02d} Where tier indicates the node's purpose, region indicates the area where the node is located, and the last two digits node_id distinguish different nodes within the same region. Underlay IPAM IPv4 Prefix: 100.64.0.0/16 Assignment rule: 100.64.{tier*10 + region}.{node_id}/32 IPv6 Prefix: fd18:3e15:61d0:ffff::/64 Assignment rule: fd18:3e15:61d0:ffff:{tier}:{region}::{node_token}/128 To simplify the automation workflow, I also intended to bring the link-local addresses between WireGuard tunnels under automated management, generating them using the aforementioned Tier, Region, and Token, with the automation scripts generating WireGuard configuration and automatically setting up the tunnel and confederation BGP session. The link-local assignment rule generally follows the normal IPv6 rule: fe80::2024:{tier}:{region}:{node_token}/64 Thus, each node only needs to be assigned its metadata to automatically generate the internal network confederation AS and addresses according to the rules above. Overlay IPAM The DN42 IPv4 part is assigned manually. This part has some historical legacy architecture, and changes need to be merged into the official registry, so I plan to assign it manually. The Underlay IPv6 is itself DN42-reachable; if other DN42 IPv6 addresses need to be allocated, they can simply be advertised on demand. For IANA IPv6, the current design format is to first calculate the continent /43 according to the initially proposed rule, e.g., for Asia: 2a14:7583:f220::/43 Then calculate the /46 based on the region, e.g., for Hong Kong region HKG: 2a14:7583:f224::/46 Next, bits from /46 to /112 are set to zero, and the last 16 bits are filled with the node Token to serve as the node's IANA IPv6 address. For example, one of my backbone nodes in Hong Kong: 2a14:7583:f224::7d89/128 Community Isolation Mechanism In this architecture, BGP Large Community effectively acts as the metadata and control information carried along with a route as it propagates between different modules and infrastructures. Based on this metadata, when we export routes to different external peers we can determine: Does it belong to DN42 or IANA? Is it locally originated, brought in by a downstream, learned from a peer, or learned from an upstream? Can it be exported to other ASes? Is it underlay-only? Does this route need AS prepending? As a concrete example, if the first identifier marks the route origin/destination as DN42 or IANA, all routes permitted to be sent to DN42 will be tagged with: (4242422024, 1, 1) Similarly, all routes destined for IANA will be tagged with: (205369, 1, 1) Within various protocols, this acts as a master switch, controlling whether export should occur. IANA upstream export filter: template bgp iana_upstream_v6 { ... ipv6 { ... import filter { if !iana_filter_default_check() then reject; # basic check remove_confederation_as(); # remove confederation as to prevent external peers from maliciously carrying internal as remove_private_community(); # remove private community to prevent external peers from maliciously manipulating internal routes iana_upstream_add_community(); # tag all routes from IANA with community permitting broadcast in IANA infrastructure accept; }; export filter { if !iana_filter_default_check() then reject; # basic check if !iana_upstream_check_community() then reject; # check community: is this a route destined for IANA / does it carry no-advertise/no-export etc. remove_confederation_as(); # remove confederation as remove_private_community(); # remove private community accept; }; ... }; ... } Where iana_filter_default_check() is used to check prefix length, ROA, whether it is a default route, and other miscellaneous content: function iana_filter_default_check(){ if net ~ [::/0] then return false; if net.len > 48 then return false; if bgp_large_community ~ [(IANA_OWNAS,1,1)] && is_self_iana_v6() then return true; if roa_check(iana_roa_v6, net, bgp_path.last) = ROA_INVALID then return false; return true; } remove_confederation_as(), as the name suggests, removes internal confederation ASes: function remove_confederation_as() {     bgp_path.delete([4220240000..4220249999]); # 4242422024 Infrastructure     bgp_path.delete([4233770000..4233779999]); # 4242423377 Infrastructure } remove_private_community(), as the name suggests, removes internal Communities. The current implementation is very rough and needs detailed refinement later, because some Communities should be open to downstreams, allowing them to use Communities to convey routing intents and perform optimizations, such as prepending AS path, which can be used for route tuning or traffic engineering: function remove_private_community(){     bgp_large_community.delete([(4242422024, *, *)]);     bgp_large_community.delete([(205369, *, *)]); } iana_upstream_add_community(), tags all routes from upstream with the Community allowing circulation in the IANA infrastructure, as well as a route source identifier: function iana_upstream_add_community(){     bgp_large_community.add((205369,1,1));     bgp_large_community.add((205369,2,102)); } Where (205369,2,102) identifies the route as coming from an upstream. Route sources are divided into three types: Upstream, Peer, Downstream/Customer/Self (treated as customer). For these three different types of BGP sessions, the exported routes usually differ: To upstream: we need the upstream to advertise our own prefixes and our downstreams' prefixes, so all routes originating from downstream should be exported to upstream. To downstream: we need to provide network services, so we need to export all routes we know, i.e., routes from upstream, peer, downstream. To peer: peering connections are solely for both parties to access each other's networks, so we must not export each other's upstreams or peers, otherwise it becomes free transit/tunneling. Therefore, only routes from downstream are exported. Correspondingly in the current internal network, routes from peers are tagged (205369,2,101), and routes from downstreams are tagged (205369,2,100). iana_upstream_check_community() checks the route's Community, and based on the principles above, judges whether it should be advertised to upstream: function iana_upstream_check_community(){ if !(bgp_large_community ~ [(205369,1,1)]) then return false; # not permitted for broadcast in IANA infrastructure, reject if bgp_large_community ~ [(205369,65535,65282)] then return false; # no-advertise if bgp_large_community ~ [(205369,65535,65281)] then return false; # no-export if bgp_community ~ [(65535,65281)] then reject; # no-advertise if bgp_community ~ [(65535,65282)] then reject; # no-export if bgp_large_community ~ [(205369,2,0)] then return false; # from internal network, this community is used to identify routes from the internal network that should not be advertised externally, reject here if bgp_large_community ~ [(205369,2,101)] then return false; # from peer, reject advertisement if bgp_large_community ~ [(205369,2,102)] then return false; # from upstream, reject advertisement return true; } Similarly, a similar check mechanism applies to DN42. Based on this mechanism, isolating the DN42 and IANA networks becomes very straightforward. The above is just a general explanation; finer community design and various export policies could be further elaborated, but that would turn into another article on routing policy details. This part actually draws heavily from たのしい’s BGP Communities; after all, both infrastructures need to interoperate, and many community specifications are basically copied. Thanks to たのしい for providing the ideas~ Route Lifecycle Phase 1: Ingress / Import When all external routes enter the system, they first pass through the import filter of the corresponding domain. IANA has three types of sources: upstream; peer / IX; downstream. DN42 also has two main types of sources: regular transit / eBGP peer; IX / route server. In addition, the local machine also originates some routes: underlay loopback; DN42 overlay address; IANA own / anycast address. These routes are tagged with Large Communities upon entering the confederation. All subsequent filters must be adjusted based on this. Phase 2: Core / Intra Confederation After entering the BIRD RIB, routes propagate between nodes via the internal Intra BGP Confederation. Here, Intra Confederation is not just about handling connectivity, but more importantly, allowing internal routes to carry policy information as well. The previous OSPF was great at solving: Where is this loopback? How to reach this next-hop? But it could not convey AS information and policy intents, for example: This route was learned from an IANA peer and must not be exported to another peer; This route is DN42 underlay-only and must never be leaked to eBGP; This route is a downstream customer prefix and can be exported to upstream. All of these are strengths of BGP policy. So moving the internal backbone from OSPF / full-mesh iBGP to confederation-style BGP is essentially upgrading the internal control plane to a "policy-carryable control plane". Phase 3: Egress / Export When a route is about to leave the infrastructure, it passes through the export filter again. Here, based on communities, systemic prevention of route leaks and macro-control like AS prepending can be performed. For example: DN42 underlay routes carry an underlay-only community and thus are not exported to DN42 eBGP; Routes learned from an IANA upstream are not exported to another upstream; Routes learned from an IANA peer are not exported to another peer; Routes learned from a DN42 IX are not exported back to IX; NO_EXPORT / NO_ADVERTISE are respected; Before external export, internal confederation ASes and private communities are removed. In summary, it can be illustrated by this diagram: flowchart LR classDef ext fill:#eef7ff,stroke:#5b8def,stroke-width:1px; classDef local fill:#f5f5f5,stroke:#888,stroke-width:1px; classDef filter fill:#fff3d6,stroke:#d19a00,stroke-width:1px; classDef core fill:#eaf8ea,stroke:#3c9b43,stroke-width:1px; classDef export fill:#fdecec,stroke:#d45a5a,stroke-width:1px; subgraph SRC["Route Source"] direction TB IU["IANA upstream<br/>transit / full routes"] IP["IANA peer / IX"] ID["IANA downstream<br/>customer prefixes"] DT["DN42 transit / eBGP peer"] DX["DN42 IX / route server"] LU["Local underlay loopback<br/>100.64.x.y / fd18:...:ffff"] LO["Local overlay address<br/>DN42 own / IANA own / anycast"] end subgraph INFRA["Bird2-Configuration Infrastructure"] direction LR subgraph IMPORT["Ingress / Import filters"] direction TB IUF["IANA upstream import<br/>tag: 205369:1:1<br/>tag: 205369:2:102"] IPF["IANA peer import<br/>tag: 205369:1:1<br/>tag: 205369:2:101"] IDF["IANA downstream import<br/>AS-SET / path check<br/>tag: 205369:2:100"] DTF["DN42 eBGP import<br/>prefix / ROA / self check<br/>tag: 4242422024:2:101/102"] DXF["DN42 IX import<br/>prefix / ROA check<br/>tag: 4242422024:2:101"] STF["Static origin<br/>underlay / overlay route tags"] end META["Large Community metadata layer<br/>domain + source + scope<br/>internal route API"] subgraph CORE["Internal control plane"] direction TB RIB["BIRD RIB"] CONFED["Intra BGP Confederation<br/>member AS: 422024xxxx<br/>policy: full / default / iana_full / no_iana"] WG["WireGuard underlay transport<br/>link-local next-hop<br/>fwmark policy routing"] UNDERLAY["net_underlay<br/>loopback + table 1142"] OVERLAY["net_overlay<br/>DN42 / IANA service addresses"] end subgraph EGRESS["Egress / Export filters"] direction TB DEXP["DN42 eBGP export<br/>reject underlay-only<br/>respect no-export/no-advertise"] DXEXP["DN42 IX export<br/>reject peer/transit learned routes"] IEXP["IANA upstream / peer export<br/>export local/downstream only<br/>reject peer/upstream learned"] DOWNEXP["IANA downstream export<br/>policy: default / own_only / reject"] KEXP["Kernel export<br/>krt_prefsrc by community"] CLEAN["cleanup before external export<br/>remove confed AS<br/>remove private communities"] end end subgraph DST["Route Destination"] direction TB OD["DN42 peers / transit"] OX["DN42 IX"] OI["IANA upstream / peer"] OC["IANA downstream"] KF["Linux kernel FIB<br/>actual packet forwarding"] end IU --> IUF IP --> IPF ID --> IDF DT --> DTF DX --> DXF LU --> STF LO --> STF IUF --> META IPF --> META IDF --> META DTF --> META DXF --> META STF --> META META --> RIB RIB <--> CONFED CONFED --- WG WG --- UNDERLAY RIB --- OVERLAY RIB --> DEXP RIB --> DXEXP RIB --> IEXP RIB --> DOWNEXP RIB --> KEXP DEXP --> CLEAN --> OD DXEXP --> CLEAN --> OX IEXP --> CLEAN --> OI DOWNEXP --> CLEAN --> OC KEXP --> KF class IU,IP,ID,DT,DX ext; class LU,LO local; class IUF,IPF,IDF,DTF,DXF,STF filter; class META,RIB,CONFED,WG,UNDERLAY,OVERLAY core; class DEXP,DXEXP,IEXP,DOWNEXP,KEXP,CLEAN export; class OD,OX,OI,OC,KF ext; Automation: Compiling Network Intent into Configuration All the above discussion on IPAM, Confederation AS, Underlay/Overlay, and BGP Communities would not truly solve the problem if it still ended up being maintained by hand-written configurations; it would just move the complexity from one form to another. Before this refactoring, adding a new node or peer often required simultaneous changes in many places: WireGuard configuration: interface name, listen port, peer public key, endpoint, allowed IPs BIRD configuration: BGP protocol name, neighbor address, neighbor interface, ASN, import/export filter IPAM-related configuration: loopback, link-local, router id, overlay address Deployment-related configuration: which nodes need rendering, which services need restarting Some special cases: nodes behind NAT, passive peers, special local AS, special filters Logically, these things describe the same thing: a connection belonging to a certain routing domain exists between two nodes. But in manual configuration, they end up scattered across WireGuard, BIRD, network interfaces, systemd/openrc, and other places. If just one field is missed or inconsistent, very strange problems can occur, such as WireGuard being up but BIRD neighbor pointing to the wrong interface; or the link-local being changed, but the peer session still having the old address. So in this refactoring, I didn't want Ansible to just be a tool that "fills YAML into Jinja2 templates". If we simply moved the original hand-written config into templates, we would essentially still be maintaining a pile of low-level configurations, just in a different file format. Since automation was introduced, the input should not be "I want to generate a configuration file that looks like this", but rather: Who is this node? Which tier / region does it belong to? Who does it interconnect with? Does this connection belong to DN42, IANA or Intra? What import/export policy should this connection use? Does it have special constraints like NAT, passive, endpoint override? As for the specific WireGuard interface wording, how the BIRD session is written, what the link-local address is – all these should, as much as possible, be generated by the automation system based on rules. Source of Truth: Node State, Not Config Files The input for this automation is mainly placed in Ansible's host_vars. The variables for each node are no longer just parameters needed for template rendering, but the description of the node's identity in the network. For example, a node will have at least: tier: node tier, e.g., backbone, IX, normal node region: region where the node is located node_id: node number within the same region node_token: a stable 16-bit token used to generate the IPv6 address suffix These fields are used for IPAM address management and identification information within the Confederation (as mentioned earlier, AS calculation and IP address derivation). The advantage of this approach is that when a new node joins, we just need to configure its basic metadata, and all its fundamental identities within the underlay are already determined. This significantly reduces subsequent maintenance effort. If link-local addresses were handwritten in every BIRD session, then any change to a node's token or address rule would require changes on all peers; but if the BIRD neighbor address is automatically derived from the peer node's metadata, it always stays consistent with the source of truth. Domain Intent: Declarative Peer Configuration The most important abstraction in this automation is Domain Intent, which treats the Session and its corresponding WireGuard tunnel binding as a single entity, then partitions them by the network type (DN42/IANA/Intra). Taking an Intra peer as an example, an internal connection fundamentally requires two types of configuration simultaneously: WireGuard interface, to provide the transport tunnel BIRD BGP session, to exchange routes over this tunnel If maintained manually, these two parts easily duplicate the same information: interface name peer node listen port endpoint neighbor interface BGP protocol name import/export policy So we abstract a regular Intra peer into a single peer intent. Simplified, it looks something like this: - node: tyo03-jp interface: intra_tyo03 wireguard: listen_port: 10234 passive: true bgp: protocol: intra_ibgp_tyo03 ipv4: import_policy: full export_policy: full ipv6: import_policy: full export_policy: full This declaration does not directly correspond to a single configuration file, but describes the peer intent: There is an internal connection named intra_tyo03 between the current node and tyo03-jp. This connection uses WireGuard for transport, and establishes an Intra BGP session over it, with BGP policy using full. Afterwards, Ansible automatically renders the corresponding WireGuard and BIRD configuration based on this. Thus, a peer relationship only needs to be described once, and the subsequent low-level configuration is generated automatically. For example, the BIRD neighbor's link-local address does not need to be handwritten; it can be automatically obtained from the peer node's tier / region / node_token. The WireGuard endpoint can also be automatically derived from the peer's ansible_host in the inventory and the peer's return peer listen_port. The biggest advantage of this is avoiding state inconsistency between multiple configuration layers. Automatic Endpoint Derivation and Passive Peers In my internal network architecture, not all nodes have perfectly symmetric public reachability. For example, some nodes / DN42 peers are behind NAT and can only initiate connections; some tunnels require a fixed endpoint as the listener, with the other end initiating. Therefore, the peer intent must support special cases like passive listening. If a peer is not passive, the automation can attempt to read ansible_host from the peer's inventory, combine it with the listen_port declared by the peer, and automatically generate the WireGuard endpoint: endpoint = peer_ansible_host + ":" + peer_return_listen_port If the peer is marked as passive, no endpoint is rendered, allowing the peer to actively initiate the connection. Essentially, the endpoint is determined by "how the peer is accessed" and "which port the peer listens on". These two pieces of information already exist in the inventory and peer intent, so there's no need to duplicate them. Of course, for extremely unusual links, the most basic manual override for the endpoint is retained. The goal of automation is not to eliminate all edge cases, but to make the normal cases require no special handling. Generic Scenarios & Exceptions This is essentially a fallback mechanism for special cases: Legacy tunnels Special internal peers carried over DN42 Nodes behind NAT Temporary workarounds Peers requiring strange BGP parameters Certain sessions needing special local AS Some routes only intended for the IANA domain, not the DN42 domain The final design kept: most regular Intra peers can generate WireGuard + BIRD via domain intent. DN42 peer WireGuard can also be generated from DN42 peer intents. However, some special BIRD sessions, such as those requiring separate local_as or special export policies, can still be explicitly declared using low-level configuration. Before final rendering, Ansible merges the generated configuration and legacy/manual overrides into the final configuration. The benefit is that automation covers 90% of the repetitive and error-prone parts, while still leaving room for the remaining 10% of special cases. Configuration Rendering The entire rendering process can be roughly understood as: flowchart TD A[host_vars: node metadata + domain intent] --> B[materialize domain intent] B --> C[effective_wireguard_interfaces] B --> D[effective_intra_ibgp_sessions] C --> E[render wg-quick configs] D --> F[render BIRD intra sessions] A --> G[render BIRD root / DN42 / IANA modules] A --> H[render underlay / overlay interfaces] E --> I[.rendered-wireguard] F --> J[.rendered] G --> J H --> K[.rendered-network] There are three types of final artifacts: .rendered/: BIRD configuration .rendered-wireguard/: WireGuard configuration .rendered-network/: underlay / overlay dummy interface configuration The BIRD configuration is further divided into several modules, written via Jinja2 templates; the actual overlay network separation is also handled here: root config: global defines, includes, kernel protocol DN42 module: DN42 filters, RPKI, eBGP peers, IX peers IANA module: upstream, downstream, peers, RPKI, static routes Intra module: confederation-style BGP sessions, static routes, internal filters Whether these modules are included is also decided automatically based on data as much as possible. For instance, if a node has no DN42 peers, there's no need to render and include ebgp.conf; no IX sessions, no need to include ix.conf. This reduces empty configurations and meaningless includes, and avoids some nodes failing template rendering due to missing related variables. Validation & Deployment The final deployment process also added some check and validation stages: render BIRD validate BIRD render WireGuard validate WireGuard deploy dummy interfaces deploy WireGuard deploy BIRD BIRD configuration is rendered and validated locally first to catch syntax errors as early as possible. WireGuard configuration is also rendered and validated first to avoid missing keys or generating obviously invalid wg-quick configs. Finally, after all validations pass, the underlying interfaces + WireGuard are deployed, and only then is BIRD uploaded and reloaded. Conclusion I'm exhausted; this article has drained me. Why is narrating this architecture in linear language so difficult? After sorting things out, most of the code was basically written by AI, and the results were decent. If you need a reference, see bird2-config/ansible at dev · iYoRoy-Network/bird2-config. The applicable scenarios for this solution are actually quite narrow; the DN42 + IANA BGP Player scene is really too niche xD. Moreover, a major reason this solution can function properly is that the IANA prefixes and DN42, along with the address ranges chosen for my own internal network, do not conflict at all. If there were address conflicts in the transit segments, one would probably still need to consider proper ISP isolation solutions like VRF/MPLS L3VPN. References: Configure BGP Confederation & Fake Confederation in Bird (Updated 2020-06-07) - Lan Tian @ Blog 浅谈 BGP 中的 Transit（中转）与 Peering（对等互联） | 网络蝙蝠侠部落

Home Network Upgrade – GPON Stick + 802.11k/v/r Roaming The beginning of the article explains the process of configuring a GPON stick and BE10000 PPPoE. If you only need to look up 802.11k/v/r related content, you can jump directly to the "Configuring 802.11 k/v/r" section. The reason this all started was that during the May Day holiday I picked up a Xiaomi BE10000 locally in Harbin. I wanted to try using a GPON stick to see if I could push my home gigabit connection past 1 Gbps (cc: 浙江移动烽火光猫获取超密 + G-010S-A 猫棒上网 – 米露小窝). Moreover, since I use mesh-style tools like EasyTier and need features like Wake-on-LAN, and my existing Xiaomi AX3600 at home is already flashed with ImmortalWRT, I was hoping to also flash a WRT-based system. Then perhaps I could use 802.11 k/v/r for automatic switching, achieving a sort of manual mesh effect. After searching around, it seemed only the BE10000 met my requirements: it has an SFP+ cage and can be flashed with QWRT. I brought it home over the summer vacation — time to tinker with it() And while I was at school I even fixed the only minor imperfection: the NFC tap-to-connect feature (ref: Adapting NFC Functionality for QWRT on Xiaomi BE10000 Router - iYoRoy's Develop Diary). This machine is now truly complete (nod) Terminology 1. Broadband & Optical Communication (Fiber & PON) Abbreviation Full Name Description / Notes FTTH Fiber To The Home Fiber to the home. PON Passive Optical Network Passive optical network. Mainstream technology for home broadband access. GPON Gigabit-Capable PON Gigabit-capable passive optical network. Our GPON stick is based on this standard. OLT Optical Line Terminal Optical line terminal. The carrier's central office equipment, responsible for distributing optical signals downstream. ONU Optical Network Unit Optical network unit. Customer premises equipment such as an ONT or GPON stick. UPC Ultra Physical Contact Ultra physical contact. A common fiber connector type (usually blue end face). APC Angled Physical Contact Angled physical contact. A common fiber connector type (usually green end face, with an 8-degree angle). LOID Logical ONU ID Logical ONU identifier. A string used by the carrier to authenticate the ONU. PLOAM Physical Layer OAM Physical Layer Operations, Administration and Maintenance. Also a password system used for ONU authentication. SN Serial Number Serial number. The unique factory number of the hardware device, often used for ONU registration. 2. Wi-Fi & Roaming Abbreviation Full Name Description / Notes AP Access Point Access point. The role of the main router or satellite router emitting Wi‑Fi signals. SSID Service Set Identifier Service set identifier. The Wi‑Fi name seen by users. BSS Basic Service Set Basic service set. The set consisting of a single AP and the devices connected within its coverage. BSSID Basic Service Set Identifier Basic service set identifier. Usually the MAC address of the AP's wireless interface. ESS Extended Service Set Extended service set. A unified network of multiple BSSs (same SSID), i.e., the roaming environment. RRM Radio Resource Management Radio Resource Management (802.11k). Used to obtain neighbor reports of surrounding APs. WNM Wireless Network Management Wireless Network Management (802.11v). Allows the AP to send roaming guidance suggestions to clients. FT Fast Transition Fast Transition (802.11r). Reduces the handshake and authentication time when a client switches APs. DS Distribution System Distribution system. The wired network backbone; ft_over_ds means exchanging roaming information via the wired backbone. NAS ID Network Access Server ID Network access server identifier. Used to uniquely identify a BSS node within a roaming domain. SAE Simultaneous Authentication of Equals Simultaneous Authentication of Equals. WPA3 key exchange protocol, more secure than WPA2. PSK Pre-Shared Key Pre-shared key. The most common password-based authentication method in home Wi‑Fi. 3. Network Protocols & System Settings Abbreviation Full Name Description / Notes PPPoE Point-to-Point Protocol over Ethernet Point-to-Point Protocol over Ethernet. The broadband dial-up protocol we commonly use. VLAN / PVID Virtual Local Area Network / Port VLAN ID Virtual LAN / Port VLAN ID. Used to isolate network traffic; required for ONU dial-up. SFP+ Enhanced Small Form-factor Pluggable Enhanced small form-factor pluggable. Supports transmission rates up to 10 Gbps. UCI Unified Configuration Interface Unified Configuration Interface. The underlying command-line configuration system of OpenWrt/QWRT. LuCI Lua Configuration Interface Lua Configuration Interface. The web-based graphical configuration interface of OpenWrt/QWRT. DHCP Dynamic Host Configuration Protocol Dynamic Host Configuration Protocol. Used to automatically assign IP addresses to devices on the local network. L2 Layer 2 Layer 2. In the article, "L2 segment" refers to a local network within the same broadcast domain. Materials Xiaomi BE10000 Router Xiaomi AX3600 Router G-010S-A NOKIA GPON Stick Heatsink SC/APC to SC/UPC fiber patch cable SC/UPC fiber optic adapter Flashing resources (you'll need to find these yourself; redistributing others' materials isn't great. You can refer to the firmware linked in the flashing tutorial below.) Regarding that SC/APC to SC/UPC fiber cable — these terms describe the fiber connector specifications. Normal home FTTH typically uses UPC, which has a blue connector, whereas the GPON sticks we buy generally use an APC connector, which is green. The main difference lies in the shape of the ferrule end face: UPC has a slightly convex end face, while APC has an angled end face. If the OLT downstream optical power is strong enough, you could directly connect a UPC connector to the GPON stick and it might still work, but you'll suffer about 3 dB of additional optical loss. Therefore, I decided to play it safe. Image source: Fiber Connector PC, UPC, APC Main Differences - Zhihu GPON sticks are notorious for running hot, so remember to attach a heatsink: After attaching it, the temperature I saw in the ONU's management interface was around 60 ℃. Flashing the BE10000 I won't go into detail here. Just follow 【小米万兆路由器刷OpenWrt】小米BE10000|SSH解锁固化|UBoot|iStore商店|多拨|-小米无线路由器及小米网络设备-恩山无线论坛 - Powered by Discuz!. The firmware has some known bugs, such as Wi‑Fi settings made through the frontend interface not being written correctly, which can prevent the entire networking subsystem from starting, leading to loss of connectivity and eventually triggering an automatic fallback. Therefore, some of the modifications to the main router described below are implemented using UCI commands. Obtaining the Carrier ONU Configuration There's no universal guide for this step... Go to Xianyu or Taobao and purchase a super admin password. Record the LOID, SN, LOID CheckCode (Password), PLOAM Password, and the VLAN ID from the Internet configuration section. In some regions you may also need to record the upstream MAC address. LOID and LOID CheckCode: PLOAM Password: VLAN ID: Generally speaking, carriers either use LOID, possibly together with a LOID CheckCode (Password), or they might use the PLOAM Password. Determine which applies based on your own situation. In my case, with China Unicom Anhui, I only needed to record the SN, VLAN ID, and LOID for it to work. (Because my Unicom ONU wouldn't let me log in, I used a China Mobile ONU for the screenshots xD) Sometimes you don't even need the super admin password to get this information; certain ONUs (e.g., China Unicom Heilongjiang, Skyworth ONUs) provide these details even with a regular user account. At the same time, record the PPPoE username and password: Typically, the username is public. On some ONUs you can use the F12 trick to remove the password attribute and reveal the actual password, but some ONUs will only send back meaningless placeholder characters. In that case you can call the carrier to reset the password. Configuring the GPON Stick Insert the GPON Stick into the SFP+ port. If the Ethernet port LED does not light up, try going to QWRT -> Network -> ECM Hardware Acceleration Settings and forcing the SFP1 and SFP2 interface speeds to Force 2.5Gbps. (I wasn't sure which one corresponds to the current SFP slot, so I changed both.) Then Save & Apply, and you may need to reboot the router. The default br-lan IP prefix should be 192.168.1.0/24. Keep it unchanged and access 192.168.1.10 to open the GPON stick's management console: Go to GPON ONU Settings and fill in the information we just recorded: LOID and SN And enable VLAN configuration: Check Interoperability Compatibility Mode, and fill in the VLAN ID under PVID. After saving you may need to reboot the GPON stick. Then go to the Status page. If you see the PON Authentication Status / PON Signal Status as O5, it means the GPON stick has completed registration and is working normally. Setting up PPPoE Dial-up On the BE10000, the default WAN port is eth4, while the SFP port is eth5. Therefore, you need to switch the WAN interface from eth4 to eth5. Go to Network -> Interfaces -> Devices, find br-lan and add eth4 to it, then remove eth5. Next, go to the Interfaces page, change the WAN network adapter to eth5, and enter the PPPoE username and password you recorded earlier: After saving, you should see that PPPoE has successfully dialed and you can access the internet normally: (I changed the entire br-lan subnet to 192.168.3.0/24; this step is not required.) Modifying Basic Wi‑Fi Settings Through testing I found that… modifying Wi‑Fi information directly via LuCI on the main router doesn't take effect properly (as mentioned above). Therefore, I modified the settings via UCI over SSH: uci set wireless.ath0.ssid='[CENSORED]' uci set wireless.ath0.encryption='psk2+ccmp' uci set wireless.ath0.key='[CENSORED]' uci set wireless.ath1.ssid='[CENSORED]' uci set wireless.ath1.encryption='psk2+ccmp' uci set wireless.ath1.key='[CENSORED]' uci commit wireless wifi reload Here, SSID is the Wi‑Fi name, and the encryption value psk2+ccmp stands for WPA2-PSK/AES, the commonly used WPA2-AES configuration. Although WPA3/SAE is more secure, in multi-AP + 802.11r roaming scenarios, WPA2-PSK + FT-PSK generally offers better compatibility. Some phones or IoT devices may experience connection failures, roaming failures, or slower switching when using combinations like WPA2/WPA3 Mixed, FT-SAE, PMF, etc. If maximum compatibility is the priority, it is recommended to use WPA2-PSK/AES for the primary SSID used for whole-home roaming, and ensure that the encryption method is consistent across both the main router and the satellite router. For the satellite router, since we want roaming, both Wi‑Fi networks must be on the same L2 segment. This requires disabling the satellite's DHCP server and configuring it as a device under the main router. Therefore, the following configuration steps apply to the satellite router: Remove all other interfaces under Network -> Interfaces -> Interfaces, keeping only br-lan: Under Network -> Interfaces -> Devices, add the wan port to the br-lan device: Assign an IP address to the br-lan device under Network -> Interfaces -> Devices: Make sure to include the subnet mask for the IPv4 address, and set the IPv4 gateway to the IP address of the main router. Under Network -> Firewall -> General Settings, adjust the firewall rules to allow traffic. (btw, because I didn't want to bother with detailed rules and figured internal devices shouldn't be at much risk, I simply allowed everything. Please do not follow this if you have specific security needs xD) Since the main router and satellite router are now on the same L2 broadcast domain, there should only be one DHCP server on the LAN: the main router. If the satellite's DHCP server is not turned off, clients might receive DHCP responses from both routers during connection or roaming, leading to incorrect gateways/DNS, or issues like DHCPNAK wrong server-ID. Symptoms of this problem may include: Wi‑Fi has already switched to another AP, but network recovery is slow; Brief disconnection after roaming; Client reacquires an IP address; Occasional gateway/DNS anomalies; It looks like 802.11r isn't working, but the real culprit is the DHCP layer. On the satellite router, run via SSH: uci set dhcp.lan.ignore='1' uci commit dhcp /etc/init.d/dnsmasq restart If the satellite router also has IPv6 RA / DHCPv6 enabled and you want the main router to handle all IPv6, you can also disable them: uci set dhcp.lan.ra='disabled' uci set dhcp.lan.dhcpv6='disabled' uci set dhcp.lan.ndp='disabled' uci commit dhcp /etc/init.d/odhcpd restart After disabling, you can verify with: uci show dhcp.lan You should see something like: dhcp.lan.ignore='1' That's it. Note that the dnsmasq process still running does not mean the satellite is still handing out DHCP leases. As long as the generated configuration contains something like: no-dhcp-interface=br-lan it means it will no longer provide DHCP service on the LAN / br-lan. After saving and applying, you should be able to connect through the main router and access the satellite via the IP address you assigned. (I couldn't find a clear option to disable the DHCP service in LuCI for this part, so I had GPT edit UCI directly via SSH.) In the satellite router's Network -> Wireless, configure the SSID interfaces to match the main router's settings: At this point, both Wi‑Fi networks should connect successfully (albeit with the same SSID). Configuring 802.11 k/v/r Main Router uci set wireless.ath0.ieee80211k='1' uci set wireless.ath0.rrm_neighbor_report='1' uci set wireless.ath0.rrm_beacon_report='1' uci set wireless.ath0.ieee80211v='1' uci set wireless.ath0.time_advertisement='0' uci set wireless.ath0.wnm_sleep_mode='0' uci set wireless.ath0.bss_transition='1' uci set wireless.ath0.ieee80211r='1' uci set wireless.ath0.nasid='Master_2_4G' uci set wireless.ath0.mobility_domain='cafe' uci set wireless.ath0.reassociation_deadline='1000' uci set wireless.ath0.ft_over_ds='0' uci set wireless.ath0.ft_psk_generate_local='1' uci set wireless.ath1.ieee80211k='1' uci set wireless.ath1.rrm_neighbor_report='1' uci set wireless.ath1.rrm_beacon_report='1' uci set wireless.ath1.ieee80211v='1' uci set wireless.ath1.time_advertisement='0' uci set wireless.ath1.wnm_sleep_mode='0' uci set wireless.ath1.bss_transition='1' uci set wireless.ath1.ieee80211r='1' uci set wireless.ath1.nasid='Master_5G' uci set wireless.ath1.mobility_domain='cafe' uci set wireless.ath1.reassociation_deadline='1000' uci set wireless.ath1.ft_over_ds='0' uci set wireless.ath1.ft_psk_generate_local='1' uci commit wireless wifi reload It's important to note that 802.11k/v/r does not mean the AP actively forces clients to roam. The final roaming decision still rests with the client. The AP merely provides neighbor information, roaming suggestions, and fast reassociation capabilities via these protocols. Support for these protocols varies across different phones, computers, and IoT devices. The configuration above covers two wireless interfaces: ath0: Main router 2.4G ath1: Main router 5G The configuration logic is identical for both interfaces; only the nasid differs. 1. 802.11k: Radio Resource Measurement / Neighbor Report Related configuration: uci set wireless.ath0.ieee80211k='1' uci set wireless.ath0.rrm_neighbor_report='1' uci set wireless.ath0.rrm_beacon_report='1' 1.1 ieee80211k='1' Enables 802.11k Radio Resource Management. The main purpose of 802.11k is to allow the AP to provide information about surrounding APs to clients. Clients don't need to blindly scan all channels; they can use the neighbor list provided by the AP to find a suitable roaming target faster. Simple understanding: 802.11k lets the client know "what other APs with the same SSID are nearby and available to switch to." Without 802.11k, clients usually need to scan channels themselves. Scanning takes time and can cause brief stutters. When enabled, clients that support 802.11k can obtain candidate AP information more quickly. 1.2 rrm_neighbor_report='1' Enables Neighbor Report. This is the most commonly used and critical capability within 802.11k. The AP provides the client with information about neighboring BSSs, such as: Neighbor AP's BSSID Channel PHY type Whether it belongs to the same ESS Supported roaming capabilities Simple understanding: This parameter allows the AP to tell the client: "Here are the nearby APs, and these are the channels they are on." This is very important for multi-AP roaming because the client doesn't have to scan from channel 1 to channel 165; it can prioritize scanning the candidate channels suggested by the AP. 1.3 rrm_beacon_report='1' Enables Beacon Report support. Beacon Report allows the AP to request that the client report information about the beacons it has scanned. In other words, the client can tell the AP: Which APs it has seen Their signal strength Which channels they are on An overview of the current wireless environment Simple understanding: Neighbor Report is the AP telling the client what's nearby; Beacon Report is the client telling the AP what it sees in return. In a typical home network, rrm_neighbor_report usually has a more immediate effect; rrm_beacon_report is a supplementary capability — just enable it. 2. 802.11v: BSS Transition / Roaming Guidance Related configuration: uci set wireless.ath0.ieee80211v='1' uci set wireless.ath0.time_advertisement='0' uci set wireless.ath0.wnm_sleep_mode='0' uci set wireless.ath0.bss_transition='1' 2.1 ieee80211v='1' Enables 802.11v Wireless Network Management. 802.11v covers a broad range of functions. For home Wi‑Fi roaming, the most relevant is BSS Transition Management. Simple understanding: 802.11v allows the AP to send a request to the client saying, "I suggest you switch to another AP." Note that this is only a suggestion, not a command. The client can accept or decline. For instance, a client might still be connected to the main router but has physically moved close to the satellite router. The AP can use 802.11v to inform it: Your signal to this AP is now relatively weak; you might consider switching to that nearby AP. Enabling 802.11v can improve the "sticky client" problem, but it does not guarantee that all devices will comply. 2.2 bss_transition='1' Enables BSS Transition Management. This is the most crucial roaming-related feature within 802.11v. Once enabled, the AP can send BSS Transition Management Requests to clients, typically containing a list of recommended target APs. Simple understanding: ieee80211v is the master switch for 802.11v; bss_transition enables the actual function used for roaming suggestions. If you enable ieee80211v without bss_transition, the roaming guidance effect may be incomplete. 2.3 time_advertisement='0' Disables Time Advertisement. 802.11v includes a Time Advertisement function where the AP can broadcast time information to clients. Time synchronization from the AP is generally not needed in a home Wi‑Fi roaming scenario, so this is set to 0. 2.4 wnm_sleep_mode='0' Disables WNM Sleep Mode. WNM Sleep Mode is part of 802.11v and is mainly used for client power saving. The client can enter a special sleep state while the AP retains some context for it. This feature is typically not relied upon in home routers or multi-AP roaming scenarios, and compatibility with some devices may be problematic, so it is disabled here. 3. 802.11r: Fast Transition / Fast Roaming Related configuration: uci set wireless.ath0.ieee80211r='1' uci set wireless.ath0.mobility_domain='cafe' uci set wireless.ath0.reassociation_deadline='1000' uci set wireless.ath0.ft_over_ds='0' uci set wireless.ath0.ft_psk_generate_local='1' 3.1 ieee80211r='1' Enables 802.11r Fast BSS Transition. The purpose of 802.11r is to shorten the authentication and reassociation time when a client switches from one AP to another. During a normal roam, the client may need to complete an entire authentication process again. 802.11r pre-derives part of the key material, allowing for a faster transition when the client decides to switch. It does not dictate when the client should roam, but when the client does decide to roam, it makes the handover process faster. Suitable for scenarios like: Moving a phone between rooms Voice calls Video conferencing Gaming Multi-AP same-SSID environments Note that: Most modern devices support 802.11r Some older or less compatible IoT devices may not like 802.11r If you encounter devices that fail to connect, suspect 802.11r compatibility first 3.2 mobility_domain='cafe' Sets the Mobility Domain. The Mobility Domain is the roaming domain identifier for 802.11r. Only APs within the same Mobility Domain will be considered by clients as part of the same fast roaming domain. For a given SSID, all APs participating in 802.11r fast roaming should use the same mobility_domain. Here we use: uci set wireless.ath0.mobility_domain='cafe' cafe is a 16-bit hexadecimal value (exactly 4 hex characters), akin to a magic number like DEADBEEF. This value can be customized, for example: mobility_domain='1234' mobility_domain='abcd' mobility_domain='beef' But note: It must be identical within the same roaming network It can differ between independent networks It must be a 4-character hexadecimal string 3.3 reassociation_deadline='1000' Sets the reassociation deadline. This parameter indicates the maximum time window allowed for the client to complete a Fast Transition reassociation. The unit is typically understood as TU (1 TU ≈ 1.024 ms). 1000 TU is approximately 1 second. Simple understanding: After the client initiates fast roaming, it needs to complete the reassociation within this time window. In a home network, 1000 is a common, lenient, and safe choice. Too short a deadline might prevent some devices from completing the switch in time; too long generally offers no obvious benefit. 3.4 ft_over_ds='0' Sets the Fast Transition method for 802.11r. 802.11r has two common modes: FT over the Air FT over DS Here we use FT over the Air: uci set wireless.ath0.ft_over_ds='0' (i.e., ft_over_ds is disabled, which enables FT over the Air). FT over the Air The client performs the fast transition directly with the target AP. This is the more common and intuitive method in non-enterprise environments. FT over DS The client communicates with the target AP through the currently connected AP over the distribution system. The networked device first contacts the new AP via the old AP, then completes the switch. In an actual home OpenWrt/QWRT multi-AP setup, it doesn't provide a significant advantage and may introduce device compatibility issues. 3.5 ft_psk_generate_local='1' Allows the local AP to generate the keys required for 802.11r based on the PSK. In home network scenarios using WPA-PSK / SAE Mixed, the AP can locally derive the necessary key material for Fast Transition using the Wi‑Fi password. Home networks typically do not have an enterprise authentication server, so this parameter can simply be turned on. Applicable scenarios: WPA2-PSK WPA2/WPA3 Mixed SAE mixed (depending on firmware support) Ordinary home networks without RADIUS 4. NAS ID: Unique Identity for Each BSS uci set wireless.ath0.nasid='Master_2_4G' uci set wireless.ath1.nasid='Master_5G' 4.1 nasid='Master_2_4G' nasid stands for NAS Identifier, which is the identity of the current BSS. In 802.11r scenarios, it is used to differentiate between different APs / BSSs. Every wireless interface participating in roaming should have a unique nasid. For example, on the main router: ath0 -> Master_2_4G ath1 -> Master_5G And correspondingly on the satellite router: 2.4G -> Slave_2_4G 5G -> Slave_5G 5. Why Configure the Same Parameters for All ath Interfaces? Because ath0, ath1, and ath2 are three distinct wireless BSSs. Even if they broadcast the same SSID, they remain independent wireless interfaces within the system. Therefore, the 802.11k/v/r parameters must be applied to each interface separately. This set of parameters can be summarized as: Parameter Protocol Function ieee80211k 802.11k Enable radio resource measurement rrm_neighbor_report 802.11k Allow AP to provide neighbor AP list rrm_beacon_report 802.11k Allow client to report scanned beacon information ieee80211v 802.11v Enable wireless network management capabilities bss_transition 802.11v Allow AP to send roaming suggestions to clients time_advertisement 802.11v Time advertisement; typically disabled in home roaming wnm_sleep_mode 802.11v WNM power saving mode; typically disabled in home roaming ieee80211r 802.11r Enable fast roaming nasid 802.11r / hostapd Identifies current BSS; recommended to be unique mobility_domain 802.11r Sets fast roaming domain; must be consistent across all APs reassociation_deadline 802.11r Sets the time window for fast reassociation ft_over_ds 802.11r Selects FT method; 0 means FT over the Air ft_psk_generate_local 802.11r Generate FT keys locally based on PSK Satellite Router You can directly enable the corresponding settings in LuCI. Refer to my configuration: Apply the same settings to all three wireless interfaces, ensuring the NAS ID is unique for each. Also, make sure the encryption settings are exactly the same on both ends. 802.11r only reduces the re-authentication time during roaming, but it requires that APs within the same roaming domain have consistent parameters: SSID, password, encryption method, Mobility Domain, etc. If the main router uses WPA2-PSK and the satellite uses WPA2/WPA3 Mixed, or if one enables SAE/FT-SAE while the other only enables FT-PSK, clients may not be able to use the same Fast Transition process during roaming. This can manifest as: Brief disconnection of the Wi‑Fi icon during roaming; Re-performing the 4-Way Handshake after switching; Stuttering in voice/video calls; Inability of some IoT devices to connect. Therefore, if the priority is stable whole-home roaming, it is recommended that all APs participating in the same SSID roaming use: WPA2-PSK / AES-CCMP / FT-PSK Which in OpenWrt/QWRT is commonly: encryption='psk2+ccmp' ieee80211r='1' ft_psk_generate_local='1' ft_over_ds='0' mobility_domain='cafe' WPA3/SAE can be put on a separate SSID for newer devices, and should not be mixed with the IoT / whole-home roaming main SSID. Frequency Band Analysis My home network environment is relatively complex. The main router is located in the center of the living room, which is openly connected to the kitchen and balcony, so the main router's signal can directly cover those areas. The satellite router is in the study, with bedrooms on both sides. The study and living room are connected by a short hallway, and Bedroom A is separated from the living room by the bathroom: The two asterisks mark the main router (living room) and the satellite router (study). The frequency band data collected in each room is as follows: Master Bedroom Study (the high red peak here is from the satellite router) Second Bedroom It can be observed that the channel in the master bedroom is fairly clean, the study has a strong signal thanks to the satellite router, but the second bedroom suffers from severe interference, coupled with a weaker signal. Additionally, our two APs (the two strongest red Wi‑Fi signals) are currently both crowded onto the same channels, resulting in actual speed tests of only about 80 Mbps. Therefore, we need to make some minor adjustments to the channels and power levels, and also configure 802.11 k/v/r. Tuning Power and Channels The main goal is to separate the channels and prevent clients from sticking to a specific access point. I can't fully explain all the reasoning myself; I also referenced a lot of configurations suggested by AI. Let's have AI explain it here too (escape) 1. Band Strategy (Channel): Fully Separated to Avoid Co-Channel Interference The main and satellite routers use completely non-overlapping channels on all bands, which is the most critical step in a multi-router setup. 2.4 GHz Band (Main 1 / Satellite 11): In the 2.4 GHz band, only channels 1, 6, and 11 are completely non-overlapping and interference-free. The main and satellite routers occupy channels 1 and 11 respectively, ensuring that the two devices do not "fight" each other (co-channel interference) when transmitting 2.4 GHz signals, thereby ensuring the stability of smart home devices that rely on 2.4 GHz. 5 GHz Band (Main 36 / Satellite 52): The main router uses the lower channel 36, and the satellite uses the DFS channel 52. These two channels are completely independent when using an 80 MHz bandwidth. Summary: This spatial channel isolation means that when a device moves between the main and satellite routers, the background noise floor is minimal, maximizing network throughput. 2. Bandwidth Strategy: Balancing Stability and Extreme Speed 2.4 GHz uses 20 MHz: This is a very sensible approach. Although 40 MHz is theoretically faster, in the extremely crowded 2.4 GHz band (microwave ovens, Bluetooth all operate here), enabling 40 MHz multiplies interference, often leading to frequent dropouts. Locking it to 20 MHz sacrifices peak speed but trades it for maximum wall-penetration stability and coverage, which is ideal for IoT smart devices that don't require high speeds. 5 GHz uses 80 MHz: 80 MHz is the current sweet-spot bandwidth for most phones and computers, capable of providing extremely high internal network transfer speeds and external download speeds. It is the primary internet access band. 3. Power Strategy (Tx Power): "Weak 2.4 GHz, Strong 5 GHz" This power configuration is the most brilliant aspect of this set of parameters, perfectly addressing the "sticky client" problem in multi-AP environments. 2.4 GHz Power Lowered (18 dBm / 20 dBm): 2.4 GHz signals have long wavelengths and strong wall-penetration ability. If the power isn't reduced, a phone moving around the house will stubbornly "cling" to a distant 2.4 GHz signal, resulting in extremely slow speeds. Lowering the 2.4 GHz power on both routers artificially shrinks the 2.4 GHz coverage circle, encouraging the device to disconnect sooner when the signal weakens and search for a better signal. 5 GHz Power Maximized (23 dBm / 24 dBm): 5 GHz signals have poor wall penetration and attenuate quickly. Keeping the power high helps compensate for this weakness, extending the high-speed 5 GHz coverage area. Summary: This power differential configuration creates a natural form of "band steering" at the physical layer. When a phone receives both 2.4 GHz and 5 GHz signals simultaneously, the high-powered 5 GHz signal's strength can easily surpass the low-powered 2.4 GHz signal, making the phone willingly prioritize connecting to the faster 5 GHz network. 4. Roaming Coordination With the physical layer parameters — band, bandwidth, and power — all optimally tuned as described above, the 802.11k/v/r protocols form a perfect roaming closed loop: 11k (Neighbor Report) + 11v (BSS Transition Management): The routers actively tell the phone "which nearby node has a better signal" and suggest it switch. Because the 2.4 GHz power is suppressed, the phone is more likely to trigger the 11v switching threshold when moving. 11r (Fast Transition): Combined with a unified SSID (PINer) and the same encryption method, this eliminates the several hundred milliseconds normally required for re-entering the password and re-authenticating when the phone switches routers, achieving a truly "seamless" experience (e.g., WeChat voice calls don't drop). The consistent mobility_domain and unique nasid are also standard requirements for 11r configuration. Ref: Gemini The final configuration for band, bandwidth, and power is: Main Router 192.168.3.1 2.4G: channel 1 / 20 MHz / 18 dBm 5G: channel 36 / 80 MHz / 24 dBm Satellite Router 192.168.3.2 2.4G: channel 11 / 20 MHz / 20 dBm 5G: channel 52 / 80 MHz / 23 dBm Results & Follow-up In the final testing, connections defaulted to the main router in the living room, balcony, kitchen, and bathroom. In the second bedroom, it automatically switches to the satellite router. Closing the door in the study also triggers a switch from the main router to the satellite. It takes about 5 seconds from signal degradation to the actual switch. The master bedroom was inconsistent; sometimes it switches, sometimes it doesn't, but it switched in 3 out of 5 tests. Speed test after plugging into the 5 Gbps port: The actual measured downstream speed reached 1336.31 Mbps, and the upstream speed reached 158.16 Mbps. I guess that counts as a success? However, the perceived improvement in daily use isn't huge because, at the time of testing, very few platforms could saturate such speeds. Combined with current carrier strategies like QoS, speeds are rarely maxed out. Still, multi-threaded download tools like IDM might be able to take better advantage of it, I suppose?

Leveraging systemd Features to Build a Multi-tenant CodeServer Analysis The motivation came from a university Python course – I didn't want to bring my heavy laptop to every class, so I tried to see if a tablet could do the job. After some searching, I found coder/code-server: VS Code in the browser which fit the requirements perfectly while also being cross-platform. I deployed it for myself, but later my friends also wanted access. That got me thinking about how to support multi-tenancy and security simultaneously. Initially I considered containerization, but because of the need to install new pip packages, containers would have been tricky – I would have had to persist Python site-packages and everything, and performance would suffer. So I decided to stick with a binary deployment. The requirements are simple, roughly summarized as follows: Each user can log into CodeServer using their own domain name. Each user can change their own login password. Each user's Python environment is completely isolated and can independently install pip packages. Users should not be able to access each other's files. Prevent users from accidentally deleting critical system components. Prevent users from accidentally writing infinite loops or similar that exhaust server resources. Prevent users from using the server for cryptocurrency mining or as a jump box for attacks. After analysis and discussions with AI, the final solution is roughly as follows: Use Linux's native multi-user mechanism – each user runs a CodeServer under their own home directory for basic isolation. This easily satisfies requirements 1, 4 and 5. Listening ports are calculated based on UID. Create a venv manually for each user and activate it automatically via .bashrc to satisfy requirement 3. Use systemd's restrictions to limit process behavior, achieving requirements 6–7 to a certain extent. Use systemd's path mechanism to monitor $HOME/.config/code-server/config.yaml and automatically restart the daemon, allowing users to change their login password by modifying the configuration file – a relatively elegant way to achieve requirement 2. For requirement 7, given the computer literacy level of first-year students, I'm not too worried (I hope?). I didn't cut off network access completely because some requirements need internet to download packages, or might involve web scraping. systemd's restrictions should already block most malicious operations. This article does not cover SELinux, because in my opinion SELinux would be over-engineering for these requirements, and I'm not very familiar with SELinux/SEPolicy (that's the main reason lol). I also wanted to avoid imposing restrictions that hinder legitimate usage, so SELinux is not used. Deployment Configuring systemd Download the CodeServer binary and upload it to the server – for example, I put it at /usr/bin/code-server. Then write the systemd service file, create /etc/systemd/system/code-server@.service: # /etc/systemd/system/code-server@.service [Unit] Description=Code-Server for %i After=network.target [Service] Type=simple User=%i Group=%i WorkingDirectory=/home/%i MemoryMax=1G CPUQuota=150% TasksMax=200 IOWeight=50 ProtectSystem=strict PrivateTmp=yes PrivateDevices=yes NoNewPrivileges=yes RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX # ExecStart=/bin/bash -c "PASSWORD=$(echo -n "%i" | md5sum | cut -d' ' -f1) /usr/bin/code-server --bind-addr 0.0.0.0:$((7000 + $(id -u %i))) --auth password" ExecStart=/bin/bash -c ' \ PORT=$((7000 + $(id -u %i) %% 10000)); \ /usr/bin/code-server --bind-addr 0.0.0.0:$PORT; \ ' Restart=always RestartSec=5 [Install] WantedBy=multi-user.target The filename is code-server@.service; the part after @ is substituted for %i. The design passes the username via %i, calculates the port, and starts automatically. The port calculation logic is 7000+uid%10000. Typically UIDs start from 1000, so ports start from 8000 in order of user creation. Note this part of the configuration: MemoryMax=1G CPUQuota=150% TasksMax=200 IOWeight=50 ProtectSystem=strict PrivateTmp=yes PrivateDevices=yes NoNewPrivileges=yes RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX (I'll let AI explain it) It mainly covers two aspects: Resource Control and Security & Sandboxing. I. Resource Control This uses Linux's cgroups mechanism to prevent the service from consuming excessive system resources and affecting other programs. MemoryMax=1G Effect: Limits the service to a maximum of 1GB of memory. Outcome: If the service tries to use more than 1GB, the system (OOM Killer) will forcibly kill the process to protect the system. CPUQuota=150% Effect: Limits the service to at most 150% CPU usage. Outcome: 100% represents one full CPU core. 150% means the service can saturate one core and use half of a second core (i.e., up to 1.5 CPU cores' worth of compute). TasksMax=200 Effect: Limits the service to a maximum of 200 tasks (processes or threads). Outcome: Prevents the service from creating unlimited child processes due to bugs or infinite loops (e.g., fork bomb), which could exhaust system PID resources and crash the system. IOWeight=50 Effect: Sets the disk I/O priority weight. The default is usually 100. Outcome: A value of 50 means when system I/O is busy, this service gets lower priority for disk read/write resources compared to default services, preventing it from choking the whole system when doing heavy file I/O. II. Security & Isolation This uses Linux namespaces and other kernel security mechanisms to "cage" the service, minimizing damage even if the service is compromised. ProtectSystem=strict Effect: Strictly protects system files. Outcome: The entire operating system's filesystem (the root / and everything under it, except special API directories like /dev, /proc, /sys) is made read-only for this service. The service cannot modify, overwrite, or delete any critical system files. PrivateTmp=yes Effect: Provides an independent temporary directory for the service. Outcome: The service sees its own private /tmp and /var/tmp directories. It cannot see or modify files placed in the global /tmp by other users/services, and vice versa. This effectively prevents symlink attacks or data leaks based on temporary files. PrivateDevices=yes Effect: Isolates physical devices. Outcome: Mounts a private /dev directory for the service, containing only pseudo-devices (e.g., /dev/null, /dev/zero, /dev/urandom). The service cannot see real physical hardware devices (like /dev/sda, USB devices), completely eliminating the possibility of directly reading/writing block devices. NoNewPrivileges=yes Effect: Prevents privilege escalation. Outcome: Ensures that the service and all its child processes, no matter what, cannot gain new system privileges. Even if the service invokes a program with the SUID bit (e.g., sudo or su), it cannot escalate to root privileges. RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX Effect: Restricts the socket address families the service can use. Outcome: The service can only use: AF_INET: IPv4 network communication AF_INET6: IPv6 network communication AF_UNIX: local UNIX domain sockets (for local IPC) Any other address families (e.g., low-level packet capture AF_PACKET, Bluetooth AF_BLUETOOTH, etc.) are blocked by the kernel. This greatly reduces the network attack surface. When creating a new user, just run: adduser <username> systemctl enable --now code-server@<username> to simultaneously create the corresponding CodeServer service. Configuring password change auto-restart Create /etc/systemd/system/code-server-restart@.path: # /etc/systemd/system/code-server-restart@.path [Unit] Description=Monitor code-server config change for %i After=network.target [Path] PathChanged=/home/%i/.config/code-server/config.yaml [Install] WantedBy=multi-user.target Create /etc/systemd/system/code-server-restart@.service: [Unit] Description=Triggered safe restart for %i [Service] Type=oneshot User=root ExecStart=/usr/local/bin/safe-restart-codeserver.sh %i [Install] WantedBy=multi-user.target The logic works like this: Step 1: Start monitoring (handled by the .path file) The system or administrator starts code-server-restart@yms.path. systemd parses the .path file, replacing %i with yms. PathChanged=/home/yms/.config/code-server/config.yaml: systemd starts quietly listening (using the inotify mechanism) for changes to this specific config.yaml file at the kernel level. Step 2: Detect change and trigger The user modifies settings in the CodeServer web interface, or edits config.yaml via the command line and saves. PathChanged detects that the file has been modified and closed (ensuring the write is complete and avoiding reading half-written dirty data). Implicit binding: Because the .path file does not explicitly specify a service to trigger using Unit=, systemd's default behavior is to trigger the .service file with the same name (without the suffix). systemd then automatically starts code-server-restart@yms.service. Step 3: Execute the action (handled by the .service file) systemd executes code-server-restart@yms.service. Again, %i is replaced with yms. Type=oneshot: Tells the system this is not a long-running daemon but a one-shot task. It exits when done. User=root: The restart operation requires higher privileges, so it's forced to run as root. ExecStart=/usr/local/bin/safe-restart-codeserver.sh <user>: This is the final step in the logic. The system runs the custom shell script as root, passing the username as an argument. We don't restart the main daemon directly inside the restart service because CodeServer auto-saves on every edit. There's a risk that during password changes, a half-edited file might trigger a restart, causing incomplete writes. So we write a script that waits until the file is stable before triggering the restart: /usr/local/bin/safe-restart-codeserver.sh #!/bin/bash USER_NAME=$1 CONFIG_FILE="/home/$USER_NAME/.config/code-server/config.yaml" STAMP_FILE="/tmp/code-server-restart-${USER_NAME}.stamp" # 1. Intercept: if stamp file exists and config file is older (or same age) than stamp, # this trigger is a leftover queued event from systemd – exit if [ -f "$STAMP_FILE" ] && [ "$CONFIG_FILE" -ot "$STAMP_FILE" ]; then echo "Config hasn't changed since last restart, exiting." exit 0 fi # 2. Debounce wait logic (unchanged) while true; do last_md5=$(md5sum "$CONFIG_FILE") sleep 5 current_md5=$(md5sum "$CONFIG_FILE") if [ "$last_md5" == "$current_md5" ]; then # File is stable, perform restart systemctl restart code-server@$USER_NAME # 3. After successful restart, update the stamp file's mtime touch "$STAMP_FILE" break else echo "Config file for $USER_NAME is still changing, waiting..." fi done The timestamp mechanism prevents multiple pointless restarts. If the config file's last edit time is earlier than the last restart time, the restart is rejected. After writing the files, enable the trigger: systemctl enable --now code-server-restart@<username>.path (note it's .path, not .service). Then try slowly editing the password in the config file via the web interface, and check the logs for something like: ○ code-server-restart@lyr.service - Triggered safe restart for lyr Loaded: loaded (/etc/systemd/system/code-server-restart@.service; disabled; preset: enabled) Active: inactive (dead) since Sat 2026-06-06 06:43:55 UTC; 29s ago Invocation: 0bd37fcb6ea44a58870c06b4fde5300c TriggeredBy: ● code-server-restart@lyr.path Process: 6997 ExecStart=/usr/local/bin/safe-restart-codeserver.sh lyr (code=exited, status=0/SUCCESS) Main PID: 6997 (code=exited, status=0/SUCCESS) Mem peak: 2.5M CPU: 43ms Jun 06 06:43:40 CodeServer systemd[1]: Starting code-server-restart@lyr.service - Triggered safe restart for lyr... Jun 06 06:43:45 CodeServer safe-restart-codeserver.sh[6997]: Config file for lyr is still changing, waiting... Jun 06 06:43:50 CodeServer safe-restart-codeserver.sh[6997]: Config file for lyr is still changing, waiting... Jun 06 06:43:55 CodeServer systemd[1]: code-server-restart@lyr.service: Deactivated successfully. Jun 06 06:43:55 CodeServer systemd[1]: Finished code-server-restart@lyr.service - Triggered safe restart for lyr. After editing stops for 5 seconds, CodeServer will restart, and the new password can be used to log in. Configuring Python venv First, install Python3 on the system (no need to elaborate – use the package manager). Also install python3-venv: apt install python3-venv Then, enter the user's home directory. I chose to create a folder named .venv for the virtual environment. Inside .venv, run python3 -m venv myvenv to create a virtual environment named myvenv, and append the following line to the user's .bashrc: fi fi +source ~/.venv/myvenv/bin/activate After that, the venv is automatically activated when the shell starts. Installing the Python extension in CodeServer and pointing it to the venv I won't go into detail here – just install the Python and debugger extensions in the CodeServer web interface, and set the Python interpreter to the one under the venv. Postscript I actually set this up a long time ago but never wrote about it. At that time, vulnerabilities like CopyFail and DirtyFrag hadn't been disclosed yet. When they were disclosed, I tested them immediately and found that this systemd configuration happened to block them all. A rough analysis follows: CopyFail uses a special socket type: AF_ALG. Our systemd configuration only allows AF_INET AF_INET6 AF_UNIX, so CopyFail cannot escalate privileges. Similarly for DirtyFrag – it uses AF_NETLINK, AF_RXRPC, and AF_ALG, all of which are blocked. Also, NoNewPrivileges=yes acts as a final line of defense. The ultimate goal of those exploits is to tamper with /usr/bin/su to inject malicious ELF shellcode, or modify /etc/passwd to clear the root password, then invoke the setuid su command to gain root privileges. With NoNewPrivileges set, the process and all its descendants can absolutely never gain higher privileges through the setuid or setgid flags of any file. Even if a future vulnerability finds a way to bypass the network restrictions and successfully replace /usr/bin/su with a malicious shell, executing su would still spawn a low-privilege shell – privilege escalation becomes impossible. This article does not cover SELinux, because the current solution is acceptably simplified for the scenario of "friends sharing, non-production critical", avoiding the risk of misconfiguration causing functional issues and operational complexity. In reality, systemd's sandboxing options are primarily based on namespaces and cgroups, and cannot fully replace mandatory access control (like SELinux or AppArmor). For higher security requirements for tenant isolation (e.g., preventing kernel escape vulnerabilities among untrusted users), SELinux/AppArmor is still the mainstream choice. Of course, this solution has other limitations. For example, when there are many users or special UID allocation requirements that exceed 10000, the port calculation logic may fail. For legitimate needs like scientific computing, the current restrictions may be too strict. This article is only intended as a discussion and reference. Random thoughts: In the age of containerization and cloud-native, traditional systemd still has many interesting features worth exploring that can accomplish a lot… Reference: systemd.git - A fork of systemd to make components more independant

Adapting NFC Functionality for QWRT on Xiaomi BE10000 Router English Translation Title: Adapting NFC Functionality for QWRT on Xiaomi BE10000 Router Analysis After flashing the Xiaomi BE10000 with QWRT, the device's network potential is indeed greatly unleashed. Advanced features such as 2.5G optical modules and SFP+ interfaces work perfectly. The only drawback is that the factory NFC "tap to connect to Wi-Fi" feature no longer works. After researching, the NFC tag is essentially an EEPROM chip mounted on the motherboard. Using i2cdetect for scanning: root@QWRT:~# i2cdetect -l i2c-1 i2c QUP I2C adapter I2C adapter i2c-2 i2c QUP I2C adapter I2C adapter i2c-0 i2c QUP I2C adapter I2C adapter root@QWRT:~# i2cdetect -y -r 0 0 1 2 3 4 5 6 7 8 9 a b c d e f 00: -- -- -- -- -- -- -- -- -- -- -- -- -- 10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 50: -- -- -- -- 54 -- -- -- -- -- -- -- -- -- -- -- 60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 70: -- -- -- -- -- -- -- -- The scan results quickly pinpoint the device attached to I2C bus 0 at physical address 0x54. NFC tap-to-connect-to-Wi-Fi uses standard NDEF format data (Ref: Wi-Fi Simple Configuration — ndeflib 0.3.2 documentation). Therefore, simply writing the data into the EEPROM according to the standard can restore the NFC tap-to-connect functionality. Implementation This article discusses only low-level hardware driver adaptation and NDEF standard protocol encapsulation for the OpenWrt/QWRT system. The hardware parameters mentioned are derived from public specification documents and generic I2C debugging tools. This project is for personal research interest and does not include or distribute any vendor-proprietary binary code. It is intended for technical exchange and learning only. Do not use for commercial purposes. Any risk of device damage resulting from attempts described herein shall be borne solely by the reader. Extract Data and Construct NDEF Payload To automatically update NFC data based on Wi-Fi credentials, we first need to obtain the current Wi-Fi SSID and password. On OpenWrt, these configurations are managed entirely by UCI (Unified Configuration Interface). Therefore, we only need to read the wireless configuration file from UCI. To ensure compatibility with modern smartphones, early devices typically used a Device Password Token to trigger WPS negotiation, but modern Android/iOS systems have restricted this behavior. For broader compatibility, we must follow the Wi-Fi Simple Configuration (WSC) specification and package the configuration as a WLAN Configuration Token (credential configuration token). (Ref: Wi-Fi Simple Configuration — ndeflib 0.3.2 documentation) Map OpenWrt's wireless encryption modes (e.g., WPA2, WPA3-SAE) precisely to the hex codes defined by the WSC specification: 0x1003: Authentication Type 0x100F: Encryption Type 0x1045: SSID 0x1027: Network Key (password) Through a script, we automatically traverse and select the primary AP bridged to lan (e.g., wifi0), convert its attributes to hex strings, and produce a standard NDEF payload. Write to NFC EEPROM When the NFC EEPROM receives a long string of NDEF data, writing too quickly or in excessively large blocks per write can easily cause the chip's I2C state machine to lock up. After testing, we chose to use the i2ctransfer tool for atomic fragmented writes. Two critical timing details: Due to communication limitations, each loop slices only 4 bytes, with auto-incrementing register addresses. Between each 4-byte block write, a 10 ms delay is enforced to allow sufficient internal erase/write time for the chip. Finally, any remaining data less than 4 bytes is padded with 0x00. Automatically Trigger Writes on Wi-Fi Configuration Changes To closely follow OpenWrt's architecture, we initially tried using hooks but found they often failed. Eventually, three fallback layers were added: LuCI frontend trigger: Register a hook under /etc/uci-defaults/ to bind the NFC sync script to the system's ucitrack mechanism. When a user modifies the Wi-Fi password in LuCI and clicks "Save & Apply", the system automatically updates the NFC data in the background. Hotplug layer: Add a hotplug event listener in /etc/hotplug.d/iface/70-nfc. When the router's lan or wifi interface changes to ifup state, the system automatically triggers the sync. Cron job: If none of the above triggers work, a cron job forces a check every 15 seconds to determine if an NFC update is needed. Additionally, considering that the NFC EEPROM has limited write endurance, if the network interface restarts even once and triggers a full rewrite, the chip would soon wear out. Therefore, a simple hash check mechanism is introduced in the underlying nfc-sync script: When the script is awakened, it first extracts the current wireless configuration and calculates its MD5 hash. It compares this hash with the old hash cached in /var/run/nfc-wireless.md5. Only when the MD5 value actually changes does it issue the I2C write command. Otherwise, the process terminates immediately. Combined with a concurrent file lock (/var/lock/nfc-sync.lock), this logic ensures that the NFC hardware's lifespan is absolutely protected against any network flapping or multiple concurrent events. After testing, automatic updating works as expected: ![[Pasted image 20260525230448.png]] Code repository: KaguraiYoRoy/be10000-qwrt-nfc: NFC Userland Implementation of QWRT for Xiaomi BE10000 (RC01) Router References: Wi-Fi Simple Configuration — ndeflib 0.3.2 documentation

Building a Cross-Region K3s Cluster from Scratch - Calico No-Encapsulation CNI # Preface I've actually wanted to play with a K8s cluster for a long time, but always felt that without sufficient knowledge, it would be too difficult to attempt. Recently, I spent some time studying DN42 and routing protocols like BGP and OSPF, and realized that it no longer feels so difficult. So I decisively started with K3s ( The main reason for choosing K3s over K8s is its lightweight nature: low resource requirements, no need to pull a bunch of images for deployment, availability of domestic mirrors… In short, K3s suits my needs better. I'm a beginner just starting to explore K3s, so please go easy on me if I make any mistakes~ # Analysis ## Choosing the CNI Component My current network architecture looks like this: ```mermaid graph TD subgraph ZeroTier Domestic subgraph WDS Gateway <--> VM1 Gateway <--> VM2 end NGB <--> Gateway HFE-NAS <--> Gateway NGB <--> HFE-NAS end subgraph IEPL Global-NIC <==OSPF==> CN-NIC end subgraph ZeroTier Global HKG02 <--> HKG04 TYO <--> HKG04 TYO <--> HKG02 end CN-NIC <--> NGB CN-NIC <--> HFE-NAS CN-NIC <--OSPF--> Gateway Global-NIC <--OSPF--> TYO Global-NIC <--OSPF--> HKG02 Global-NIC <--OSPF--> HKG04 %% Style definition: orange background, bold border to represent routers classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold; class Global-NIC,CN-NIC,Gateway router; Among this, the WDS node is a Proxmox VE host with multiple VMs underneath. It advertises its VMs' IPv4 prefixes via OSPF. When Hong Kong nodes need to access a VM under the WDS node, they can do so by joining the OSPF internal network to achieve multi-hop reachability. This keeps the encapsulation layer count to only one, so there's no worry about MTU "disappearing act". I plan to create two new VMs under WDS to serve as the master and a node (temporarily called KubeMaster and KubeNode-WDS1). Then HKG04 (temporarily called KubeNode-HKG04) will also join the K3s cluster as a node. The simplest approach would be to use K3s's default Flannel as the CNI. However, Flannel is based on VXLAN, and adding another layer of my existing internal network would lead to the following MTU "disappearing act": Data packet -> Flannel VXLAN encapsulation -> ZeroTier encapsulation -> Physical link The actual usable MTU for inter-container communication would likely be compressed to 1350 or even lower. Therefore, I tried to find a CNI solution that can work directly on top of this internal network, and then I found Calico. As I understand, Calico uses BGP as its underlying routing protocol, supports starting in no-encapsulation (No-Encap) mode, and hands packets directly to the upper routers for routing. Thus, I chose Calico as the CNI component. Routing Design To ensure that intermediate routers know how to route Pod IPs, KubeMaster and KubeNode-WDS1 are under the Proxmox VE host. They need to establish BGP with HKG04 across the entire internal network. This means that every router at each intermediate level must learn the full BGP routes, so that the following routing path can be established: graph LR subgraph WDS KubeMaster KubeNode-WDS1 Gateway end subgraph IEPL CN-Namespace Global-Namespace end KubeNode-WDS1 <--> Gateway KubeMaster <--> Gateway <--> CN-Namespace <--> Global-Namespace <--> HKG04 %% Style definition: highlight nodes with routing capability classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold; class Gateway,CN-Namespace,Global-Namespace router; Otherwise, any intermediate hop would drop packets because it doesn't recognize the source/destination IP. Also, due to the property of iBGP that routes learned from a neighbor cannot be propagated to the next iBGP neighbor, all BGP sessions between Gateway, CN-Namespace, Global-Namespace and the nodes need to enable Route Reflector; otherwise, nodes cannot correctly learn routes from each other. That said, this architecture would be more suitable for BGP Confederation, but my existing network is already quite complex, and adding BGP confederations would make later maintenance more troublesome. Moreover, my number of nodes is small, so the overhead of iBGP Full Mesh is acceptable. It's definitely not because I'm lazy (so Thus, the final network routing structure is as follows: graph TD subgraph WDS VM1 VM2 Gateway end subgraph IEPL CN-Namespace Global-Namespace end VM1 <-.Calico iBGP Full Mesh.-> VM2 VM1 <--iBGP Route Reflector--> Gateway VM2 <--iBGP Route Reflector--> Gateway <--iBGP--> CN-Namespace <--iBGP--> Global-Namespace <--iBGP Route Reflector--> HKG04 Gateway <--iBGP--> Global-Namespace HKG04 <-.Calico iBGP Full Mesh.-> VM1 VM2 <-.Calico iBGP Full Mesh.-> HKG04 %% Style definition classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold; %% Mark nodes with routing/forwarding or RR functions as Router class Gateway,CN-Namespace,Global-Namespace router; The dashed-line BGP sessions are automatically created by Calico, while the solid-line parts need to be manually created by us. Keeping Calico's own iBGP Full Mesh is for future scalability, so that nodes can preferentially establish direct P2P connections via ZeroTier instead of taking a detour through the Route Reflector aggregation router. Deployment After clarifying the structure, deployment becomes simple. Enable Kernel Forwarding and Disable rp_filter Standard practice. echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf echo "net.ipv6.conf.default.forwarding=1" >> /etc/sysctl.conf echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf echo "net.ipv4.conf.default.rp_filter=0" >> /etc/sysctl.conf echo "net.ipv4.conf.all.rp_filter=0" >> /etc/sysctl.conf sysctl -p Install K3s Master Because the KubeMaster control plane node is located inside China, it's best to configure image acceleration: mkdir -p /etc/rancher/k3s cat <<EOF > /etc/rancher/k3s/registries.yaml mirrors: docker.io: endpoint: - "https://docker.m.daocloud.io" quay.io: endpoint: - "https://quay.m.daocloud.io" EOF Install using the mirror: curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | \ INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC=" \ --flannel-backend=none \ --disable-network-policy \ --cluster-cidr=10.42.0.0/16" sh - Note the need to specify --flannel-backend=none and --disable-network-policy to disable the default CNI component. Use cat /var/lib/rancher/k3s/server/node-token to view the token and record it. Worker Nodes For nodes inside China, configure image acceleration: mkdir -p /etc/rancher/k3s cat <<EOF > /etc/rancher/k3s/registries.yaml mirrors: docker.io: endpoint: - "https://docker.m.daocloud.io" quay.io: endpoint: - "https://quay.m.daocloud.io" EOF Then install K3s using the mirror and join the cluster: export INSTALL_K3S_MIRROR=cn export K3S_URL=https://<master node IP>:6443 # Replace with your master node's actual IP export K3S_TOKEN=K10...your token...::server:xxx # Replace with the full token obtained in the first step curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | sh - At this point, the status of each node should be NotReady because the CNI component is missing. Install Calico and Configure No-Encap Mode On the master, manually download https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml and install the Calico operator: kubectl create -f tigera-operator.yaml Configure a custom resource by creating a custom-resource.yaml file: apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Add image registry configuration registry: quay.m.daocloud.io calicoNetwork: ipPools: - blockSize: 26 cidr: 10.42.0.0/16 encapsulation: None natOutgoing: Enabled nodeSelector: all() Here, specify encapsulation: None to enable No-Encap mode. You can also modify the IPv4 CIDR here if needed. Then: kubectl apply -f custom-resource.yaml to perform the installation. Use: kubectl get pods -A -o wide to check Pod status, waiting for each node to finish pulling images. Configure BGP Topology Label Nodes Label nodes to specify that nodes under WDS connect to the Gateway's BGP in the WDS node, and nodes outside China connect to the BGP of the Global Namespace: kubectl label nodes kubemaster region=WDS kubectl label nodes kubenode-wds-1 region=WDS kubectl label nodes kubenode-hkg04 region=Global Calico Configuration Create a YAML configuration file: apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: route-reflector-domestic spec: nodeSelector: region == 'Domestic' # This part is not actually used; I originally designed a general aggregation router in the Domestic area peerIP: 100.64.0.108 asNumber: 64512 --- apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: route-reflector-wds spec: nodeSelector: region == 'WDS' peerIP: 192.168.100.1 asNumber: 64512 --- apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: route-reflector-global spec: nodeSelector: region == 'Global' peerIP: 100.64.1.106 asNumber: 64512 This means: All nodes with label region equal to Domestic will have a BGP session to 100.64.0.108 (the domestic aggregation router) using AS 64512 All nodes with label region equal to WDS will have a BGP session to 192.168.100.1 (the Gateway for all VMs under the WDS node) using AS 64512 All nodes with label region equal to Global will have a BGP session to 100.64.1.106 (the overseas aggregation router) using AS 64512 This achieves what is shown in the diagram: all VMs under the WDS node, including the master and KubeNode-WDS1, connect to the Gateway aggregation router of the WDS node, and all nodes in overseas areas connect to the overseas aggregation router. Configure Aggregation Router iBGP This part is simply a matter of writing Bird configuration files (easy). Here are a few examples: k3s/ibgp.conf: function is_insider_as(){ if bgp_path.len > 0 && !(bgp_path ~ [= 64512 =]) then { return false; } if net ~ [ 10.42.0.0/16{16,32} ] then { return true; } return false; } template bgp k3sbackbone{ local as K3S_AS; router id INTRA_ROUTER_ID; neighbor as K3S_AS; ipv4{ table intra_table_v4; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; extended next hop; }; ipv6{ table intra_table_v6; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; }; }; template bgp k3speers{ local as K3S_AS; neighbor as K3S_AS; router id INTRA_ROUTER_ID; rr client; rr cluster id INTRA_ROUTER_ID; ipv4{ table intra_table_v4; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; }; ipv6{ table intra_table_v6; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; }; }; include "ibgpeers/*"; ibgpeers/backbone-cn.conf: protocol bgp 'k3s_backbone_cn_v4' from k3sbackbone{ neighbor fd18:3e15:61d0:cafe:f001::1; }; ibgpeers/master.conf: protocol bgp 'k3s_master_v4' from k3speers{ neighbor 192.168.100.251; }; Main points: it's best not to enable Route Reflector between the aggregation routers, and remember to enable next hop self. After everything is done, using kubectl get nodes should show all nodes as Ready: NAME STATUS ROLES AGE VERSION kubemaster Ready control-plane 2d23h v1.34.5+k3s1 kubenode-hkg04 Ready <none> 11h v1.34.6+k3s1 kubenode-wds-1 Ready <none> 2d7h v1.34.5+k3s1 Use kubectl get pods -A -o wide to view Pods: NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-system calico-kube-controllers-64fc874957-6bdlz 1/1 Running 0 5h38m 10.42.253.136 kubenode-hkg04 <none> <none> calico-system calico-node-2qz82 1/1 Running 0 4h24m 10.2.5.7 kubenode-hkg04 <none> <none> calico-system calico-node-dhl2c 1/1 Running 0 4h24m 192.168.100.251 kubemaster <none> <none> calico-system calico-node-nbpkj 1/1 Running 0 4h23m 192.168.100.252 kubenode-wds-1 <none> <none> calico-system calico-typha-7bb5db4bdc-rfpwg 1/1 Running 0 5h38m 10.2.5.7 kubenode-hkg04 <none> <none> calico-system calico-typha-7bb5db4bdc-rwwr5 1/1 Running 0 5h38m 192.168.100.251 kubemaster <none> <none> calico-system csi-node-driver-jglwp 2/2 Running 0 5h38m 10.42.64.68 kubenode-wds-1 <none> <none> calico-system csi-node-driver-jqjsc 2/2 Running 0 5h38m 10.42.253.137 kubenode-hkg04 <none> <none> calico-system csi-node-driver-vk26s 2/2 Running 0 5h38m 10.42.141.16 kubemaster <none> <none> kube-system coredns-695cbbfcb9-8fx4p 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.14 kubemaster <none> <none> kube-system helm-install-traefik-crd-5bkwx 0/1 Completed 0 2d23h <none> kubemaster <none> <none> kube-system helm-install-traefik-m9fgj 0/1 Completed 1 2d23h <none> kubemaster <none> <none> kube-system local-path-provisioner-546dfc6456-dmn4g 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.15 kubemaster <none> <none> kube-system metrics-server-c8774f4f4-2wkwh 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.12 kubemaster <none> <none> kube-system svclb-traefik-999cddce-hpmcm 2/2 Running 6 (7h26m ago) 11h 10.42.253.134 kubenode-hkg04 <none> <none> kube-system svclb-traefik-999cddce-q4225 2/2 Running 2 (7h27m ago) 2d22h 10.42.141.9 kubemaster <none> <none> kube-system svclb-traefik-999cddce-xmd64 2/2 Running 2 (7h26m ago) 2d6h 10.42.64.66 kubenode-wds-1 <none> <none> kube-system traefik-788bc4688c-vbbhj 1/1 Running 1 (7h27m ago) 2d22h 10.42.141.13 kubemaster <none> <none> tigera-operator tigera-operator-6b95bbf4db-vl46l 1/1 Running 1 (7h27m ago) 2d23h 192.168.100.251 kubemaster <none> <none> Use kubectl exec -it -n calico-system <calico-node-xxxx> -- birdcl s p to check the status of Bird: root@KubeMaster:~/kube/calico# kubectl exec -it -n calico-system calico-node-2qz82 -- birdcl s p Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init) BIRD v0.3.3+birdv1.6.8 ready. name proto table state since info static1 Static master up 08:58:17 kernel1 Kernel master up 08:58:17 device1 Device master up 08:58:17 direct1 Direct master up 08:58:17 Mesh_192_168_100_251 BGP master up 08:58:33 Established Mesh_192_168_100_252 BGP master up 08:59:00 Established Node_100_64_1_106 BGP master up 12:57:44 Established ip r shows the system routing table: root@KubeMaster:~/kube/calico# ip r default via 192.168.100.1 dev eth0 proto static 10.42.64.64/26 proto bird nexthop via 192.168.100.1 dev eth0 weight 1 nexthop via 192.168.100.252 dev eth0 weight 1 blackhole 10.42.141.0/26 proto bird 10.42.141.9 dev caliac6501d3794 scope link 10.42.141.12 dev calib07c23291bb scope link 10.42.141.13 dev caliab16e60bd19 scope link 10.42.141.14 dev calid5959219080 scope link 10.42.141.15 dev cali026d8f1ddb7 scope link 10.42.141.16 dev califa657ba417a scope link 10.42.253.128/26 via 192.168.100.1 dev eth0 proto bird 192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.251 Ping a Pod's IP – if everything is fine, it should work directly: root@KubeMaster:~/kube/calico# ping 10.42.253.137 PING 10.42.253.137 (10.42.253.137) 56(84) bytes of data. 64 bytes from 10.42.253.137: icmp_seq=1 ttl=60 time=33.7 ms 64 bytes from 10.42.253.137: icmp_seq=2 ttl=60 time=33.5 ms ^C --- 10.42.253.137 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 33.546/33.632/33.718/0.086 ms Tune MTU This step is actually for stability…? Tests have shown that although my ZeroTier MTU is 1420, packets start to fragment around 1392 bytes (test with ping -M do -s <packet size> <Pod_IP>). Therefore, force the Pod MTU to 1370: root@KubeMaster:~/kube/calico# cat patch-mtu.yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: mtu: 1370 nodeAddressAutodetectionV4: firstFound: true root@KubeMaster:~/kube/calico# kubectl apply -f patch-mtu.yaml installation.operator.tigera.io/default configured