Home Network Upgrade – GPON Stick + 802.11k/v/r Roaming
The beginning of the article explains the process of configuring a GPON stick and BE10000 PPPoE. If you only need information about 802.11k/v/r, you can jump to the "Configuring 802.11k/v/r" section. The reason for this project goes back to the Labour Day holiday, when I picked up a Xiaomi BE10000 locally in Harbin. I wanted to try a GPON stick to see if I could push my home gigabit broadband beyond 1 Gbps (ref: Zhejiang Mobile FiberHome GPON super admin password + G-010S-A GPON stick internet access – Milu’s Blog (Chinese)). Because I use mesh-like tools such as EasyTier and need features like Wake-on-LAN, and my existing Xiaomi AX3600 already runs ImmortalWRT, I hoped to flash a WRT-based system and then use 802.11k/v/r to achieve automatic handover, essentially building a manual mesh. After searching, it seemed only the BE10000 met my requirements: it has an SFP+ cage and can be flashed with QWRT. I brought it home during the summer break – time to tinker! I already fixed its one imperfection – the NFC tap-to-connect – while I was at university (ref: Adapting NFC for Xiaomi BE10000 on QWRT – Yousheng's Dev Diary (Chinese)), so the device is now complete (nod) Glossary 1. Broadband & Optical Communication (Fiber & PON) Abbreviation Full Name Explanation / Notes FTTH Fiber To The Home Fiber optic connection directly to the home. PON Passive Optical Network Mainstream technology for residential broadband access. GPON Gigabit-Capable PON Gigabit Passive Optical Network. The GPON Stick you use is based on this standard. OLT Optical Line Terminal The ISP’s central office equipment that distributes optical signals downstream. ONU Optical Network Unit Customer premises equipment, such as an optical modem or GPON stick. UPC Ultra Physical Contact A common fiber connector type (usually a blue end-face). APC Angled Physical Contact Another common fiber connector type (usually a green end-face with an 8-degree angle). LOID Logical ONU ID A string of characters the ISP uses to authenticate the ONU. PLOAM Physical Layer OAM Physical Layer Operations, Administration and Maintenance; also a password system used for ONU authentication. SN Serial Number The hardware’s unique factory serial number, often used for ONU registration. 2. Wireless LAN & Roaming (Wi-Fi & Roaming) Abbreviation Full Name Explanation / Notes AP Access Point The role a primary or secondary router plays when emitting Wi-Fi signals. SSID Service Set Identifier The Wi-Fi network name you see. BSS Basic Service Set A single AP and all devices connected within its coverage area. BSSID Basic Service Set Identifier Usually the MAC address of the AP’s wireless interface. ESS Extended Service Set Multiple BSSs forming a unified network (same SSID) – i.e., a roaming environment. RRM Radio Resource Management 802.11k – used to obtain neighbor reports about surrounding APs. WNM Wireless Network Management 802.11v – allows an AP to send roaming guidance suggestions to clients. FT Fast Transition 802.11r – reduces handshake and authentication time when a client switches APs. DS Distribution System The wired network backbone; ft_over_ds means roaming information is exchanged over the wired backbone. NAS ID Network Access Server Identifier Uniquely identifies a BSS node within a roaming domain. SAE Simultaneous Authentication of Equals The key exchange protocol for WPA3, more secure than WPA2. PSK Pre-Shared Key The most common home Wi-Fi authentication method, where you enter a password. 3. Network Protocols & System Settings Abbreviation Full Name Explanation / Notes PPPoE Point-to-Point Protocol over Ethernet The commonly used broadband dial-up protocol. VLAN / PVID Virtual Local Area Network / Port VLAN ID Used to isolate network traffic; essential for GPON stick dial-up. SFP+ Enhanced Small Form-factor Pluggable An enhanced hot-pluggable optical module interface supporting up to 10 Gbps. UCI Unified Configuration Interface The underlying command-line configuration system for OpenWrt/QWRT. LuCI Lua Configuration Interface The web-based graphical configuration interface for OpenWrt/QWRT. DHCP Dynamic Host Configuration Protocol Protocol for automatically assigning IP addresses to devices on the LAN. L2 Layer 2 The data link layer. “L2 segment” in this article refers to a LAN within the same broadcast domain. Preparation Xiaomi BE10000 Router Xiaomi AX3600 Router G-010S-A NOKIA GPON Stick Heatsinks SC/APC to SC/UPC fiber patch cable SC/UPC fiber adapter Flashing resources (find them yourself – redistributing others’ work isn’t ideal. You can refer to the firmware mentioned in the flashing tutorial below.) Regarding that SC/APC to SC/UPC fiber cable, it describes the connector specifications. Typical residential FTTH usually uses UPC (blue connector), whereas the GPON sticks we buy almost always have an APC (green) connector. The main difference is the end-face shape: UPC is slightly domed outwards, APC is cut at an angle. If the ISP's OLT downstream optical power is strong enough, you could connect UPC directly to the stick, but you’d get about 3 dB of extra loss. So I decided to play it safe. Image source: Differences between PC, UPC and APC fiber connectors – Zhihu (Chinese) GPON sticks are notorious for running hot, so remember to attach heatsinks: After attaching them, I saw temperatures of around 60 °C in the stick’s management console. Flashing the BE10000 No need to elaborate here – just follow Flashing OpenWrt on Xiaomi 10G Router | Xiaomi BE10000 | SSH Unlock | UBoot | iStore | Multi-WAN – Right.com.cn Forum (Chinese). The firmware has some known bugs. For example, Wi-Fi settings configured through the front-end may fail to write correctly, causing the entire network subsystem to crash, the router to become unreachable, and eventually triggering an automatic fallback. That’s why some of the modifications below are done using UCI commands. Obtaining ISP ONU Configuration There is no universal guide for this step… Buy a super admin password from Xianyu or Taobao, and record the LOID, SN, LOID CheckCode (Password), PLOAM Password, and the VLAN ID of the Internet connection. Some regions may also require you to record the upstream MAC address. LOID and LOID CheckCode: PLOAM Password: VLAN ID: Generally, ISPs use LOID together with a possible LOID CheckCode (Password), or they might use PLOAM Password. Decide based on your situation. In my case (Anhui Unicom), I only needed to record SN, VLAN ID, and LOID and everything worked. (I couldn’t log into the Unicom ONT, so I used a Mobile one for these screenshots, lol.) Sometimes you don’t even need the super admin password; some ONTs (e.g., Skyworth models for Heilongjiang Unicom) display this information even for a regular user. Also, record your PPPoE username and password: Generally the username is visible. On some ONTs you can reveal the password by using F12 to remove the password attribute; others send back meaningless placeholders. In that case, call your ISP to reset the password. Configuring the GPON Stick Insert the GPON Stick into the SFP+ port. If the port LED doesn’t light up, try going to QWRT → Network → ECM Hardware Acceleration Settings and force the SFP1 and SFP2 interface speeds to Force 2.5Gbps. (I couldn’t be sure which one corresponds to the active SFP port, so I changed both.) Save & apply, and you may need to reboot the router. The default br-lan IP prefix should be 192.168.1.0/24. Keep it unchanged and access 192.168.1.10 to open the stick’s management console: Find GPON ONU Settings and fill in the information you recorded earlier: LOID and SN Also enable VLAN configuration: Tick Interoperability Compatible Mode, and fill in the VLAN ID under PVID. You may need to reboot the stick after saving. Then go to the status page. If you see the PON authentication status / signal status as O5, the stick has successfully registered and is working. Configuring Dial-Up (PPPoE) On the BE10000, the default WAN port is eth4, while the SFP port is eth5. You need to switch the WAN from eth4 to eth5. Go to Network → Interfaces → Devices, locate br-lan, and add eth4 to it while removing eth5. Then go to the Interfaces page, change the WAN device to eth5, and enter your PPPoE username and password: After saving, you should see that PPPoE has dialled successfully and you can access the internet: (I changed the entire br-lan subnet to 192.168.3.0/24 – this step is not mandatory.) Modifying Basic Wi-Fi Settings As I discovered, modifying Wi-Fi settings directly through LuCI on the main router fails to write correctly (as mentioned above). Therefore, modify them via SSH using UCI: # Set main router Wi-Fi info uci set wireless.ath0.ssid='[CENSORED]' uci set wireless.ath0.encryption='psk2+ccmp' uci set wireless.ath0.sae='1' uci set wireless.ath0.key='[CENSORED]' uci set wireless.ath1.ssid='[CENSORED]' uci set wireless.ath1.encryption='psk2+ccmp' uci set wireless.ath1.sae='1' uci set wireless.ath1.key='[CENSORED]' uci set wireless.ath2.ssid='[CENSORED]' uci set wireless.ath2.encryption='psk2+ccmp' uci set wireless.ath2.sae='1' uci set wireless.ath2.key='[CENSORED]' uci commit wireless wifi reload Here SSID is the Wi-Fi name, and encryption value psk2+ccmp represents WPA2-PSK/WPA3-SAE Mixed Mode. For the secondary router, because we are setting up roaming, both Wi-Fi networks must be on the same L2 segment. Thus, you must disable its DHCP server and configure the secondary router as a device under the main router. The steps for the secondary router are: Delete all other interfaces under Network → Interfaces → Interfaces, keeping only br-lan: Under Network → Interfaces → Devices, add the wan port to the br-lan device: Assign an IP address to br-lan under Network → Interfaces → Devices: Remember to include the netmask for the IPv4 address, and set the IPv4 gateway to the main router’s IP. Under Network → Firewall → General Settings, adjust the firewall rules accordingly. (btw, I was too lazy to configure detailed rules and assumed the internal network devices aren’t a huge risk, so I allowed everything. Don’t copy this if you have specific security needs, lol.) After saving and applying, you should be able to connect to the main router and access the secondary router using the IP you just set. On the secondary router, go to Network → Wireless and configure each SSID interface to match the main router’s settings: At this point, both Wi-Fi radios should be working (even with the same SSID) and they will be on the same channel. Configuring 802.11k/v/r Main Router # Main router 802.11k/v/r configuration # ath0 (2.4 GHz) uci set wireless.ath0.ieee80211k='1' uci set wireless.ath0.rrm_neighbor_report='1' uci set wireless.ath0.rrm_beacon_report='1' uci set wireless.ath0.ieee80211v='1' uci set wireless.ath0.time_advertisement='0' uci set wireless.ath0.wnm_sleep_mode='0' uci set wireless.ath0.bss_transition='1' uci set wireless.ath0.ieee80211r='1' uci set wireless.ath0.nasid='Master_2_4G' uci set wireless.ath0.mobility_domain='cafe' uci set wireless.ath0.reassociation_deadline='1000' uci set wireless.ath0.ft_over_ds='0' uci set wireless.ath0.ft_psk_generate_local='1' # ath1 (5G-1) uci set wireless.ath1.ieee80211k='1' uci set wireless.ath1.rrm_neighbor_report='1' uci set wireless.ath1.rrm_beacon_report='1' uci set wireless.ath1.ieee80211v='1' uci set wireless.ath1.time_advertisement='0' uci set wireless.ath1.wnm_sleep_mode='0' uci set wireless.ath1.bss_transition='1' uci set wireless.ath1.ieee80211r='1' uci set wireless.ath1.nasid='Master_5G' uci set wireless.ath1.mobility_domain='cafe' uci set wireless.ath1.reassociation_deadline='1000' uci set wireless.ath1.ft_over_ds='0' uci set wireless.ath1.ft_psk_generate_local='1' # ath2 (5G-2) uci set wireless.ath2.ieee80211k='1' uci set wireless.ath2.rrm_neighbor_report='1' uci set wireless.ath2.rrm_beacon_report='1' uci set wireless.ath2.ieee80211v='1' uci set wireless.ath2.time_advertisement='0' uci set wireless.ath2.wnm_sleep_mode='0' uci set wireless.ath2.bss_transition='1' uci set wireless.ath2.ieee80211r='1' uci set wireless.ath2.nasid='Master_5G2' uci set wireless.ath2.mobility_domain='cafe' uci set wireless.ath2.reassociation_deadline='1000' uci set wireless.ath2.ft_over_ds='0' uci set wireless.ath2.ft_psk_generate_local='1' uci commit wireless wifi reload Note: 802.11k/v/r does not force clients to roam proactively; the client still makes the final decision. The AP only provides neighbor information, roaming suggestions, and fast re-association capabilities using these protocols. Support varies across different phones, PCs, and IoT devices. The configuration above covers three wireless interfaces: ath0: Main router 2.4 GHz ath1: Main router 5 GHz-1 ath2: Main router 5 GHz-2 All three follow the same logic, differing only in nasid. 1. 802.11k: Radio Resource Measurement / Neighbor Report Relevant settings: uci set wireless.ath0.ieee80211k='1' uci set wireless.ath0.rrm_neighbor_report='1' uci set wireless.ath0.rrm_beacon_report='1' 1.1 ieee80211k='1' Enables 802.11k Radio Resource Management. 802.11k allows the AP to provide clients with information about neighboring APs, so they don't need to blindly scan all channels. Clients can use the neighbor list to quickly find a suitable target for roaming. In short: 802.11k lets the client know "which APs with the same SSID are nearby and available to switch to". Without 802.11k, a client typically scans channels by itself, which takes time and can cause brief lag. When enabled, compatible clients can obtain candidate AP information much faster. 1.2 rrm_neighbor_report='1' Enables Neighbor Report. This is the most common and critical capability of 802.11k. The AP provides the client with details about neighboring BSSs, such as: BSSID of the neighboring AP Operating channel PHY type Whether it belongs to the current ESS Supported roaming capabilities In short: This parameter lets the AP tell the client: "Here are the other APs nearby, and they are on these channels." This is crucial for multi-AP roaming because the client can scan only the suggested channels instead of sweeping from channel 1 to 165. 1.3 rrm_beacon_report='1' Enables Beacon Report support. Beacon Report allows the AP to request that the client report back the beacons it has observed. In other words, the client can tell the AP: Which APs it sees Their signal strengths Their channels A rough picture of the current wireless environment In short: Neighbor Report is the AP telling the client what’s nearby; Beacon Report is the client telling the AP what it sees. In a typical home network, rrm_neighbor_report is more directly useful; rrm_beacon_report is a supplementary capability – just enable it. 2. 802.11v: BSS Transition / Roaming Guidance Relevant settings: uci set wireless.ath0.ieee80211v='1' uci set wireless.ath0.time_advertisement='0' uci set wireless.ath0.wnm_sleep_mode='0' uci set wireless.ath0.bss_transition='1' 2.1 ieee80211v='1' Enables 802.11v Wireless Network Management. The most relevant part for home Wi-Fi roaming is BSS Transition Management. In short: 802.11v lets the AP suggest to a client: "I recommend you switch to another AP." This is only a suggestion, not an order. The client can accept or refuse. For example, if the client is still attached to the main router but is already physically near the secondary router, the AP can use 802.11v to inform it: "Your signal to this AP is now mediocre; consider moving to the other one." Enabling 802.11v helps with the "sticky client" problem, but there’s no guarantee all devices will obey. 2.2 bss_transition='1' Enables BSS Transition Management. This is the key roaming-related feature of 802.11v. When enabled, the AP can send a BSS Transition Management Request to the client, typically containing a list of recommended target APs. In short: ieee80211v is the master switch for 802.11v; bss_transition turns on the actual roaming suggestion function. If you enable ieee80211v but not bss_transition, the roaming guidance effect may be incomplete. 2.3 time_advertisement='0' Disables Time Advertisement. 802.11v includes a Time Advertisement feature where the AP can broadcast time information. For home roaming, you generally don't need the AP to provide time sync, so set it to 0. 2.4 wnm_sleep_mode='0' Disables WNM Sleep Mode. WNM Sleep Mode is part of 802.11v, mainly used for client power saving. The client can enter a special sleep state while the AP retains some context. Home routers and multi-AP roaming generally don’t rely on this, and some devices have compatibility issues, so it’s turned off. 3. 802.11r: Fast Transition / Fast Roaming Relevant settings: uci set wireless.ath0.ieee80211r='1' uci set wireless.ath0.mobility_domain='cafe' uci set wireless.ath0.reassociation_deadline='1000' uci set wireless.ath0.ft_over_ds='0' uci set wireless.ath0.ft_psk_generate_local='1' 3.1 ieee80211r='1' Enables 802.11r Fast BSS Transition. 802.11r shortens the authentication and re-association time when a client moves from one AP to another. Without it, the client may need to complete a full authentication cycle. 802.11r pre-derives some key material so the handover is faster. It does not decide when to roam, but once the client decides to roam, it makes the process quicker. Suitable for: Moving between rooms with a phone Voice calls Video conferences Gaming Multi-AP environments with the same SSID Be aware: Most new devices support 802.11r Some older or less compatible IoT devices may dislike 802.11r If a device fails to connect, suspect 802.11r compatibility first 3.2 mobility_domain='cafe' Sets the Mobility Domain, an 802.11r roaming domain identifier. Only APs sharing the same Mobility Domain are considered part of the same fast-roaming group by clients. All APs participating in 802.11r fast roaming under the same SSID must use the same mobility_domain. Here we use: uci set wireless.ath0.mobility_domain='cafe' cafe is a 16-bit hexadecimal value (4 hex characters), similar to magic numbers like DEADBEEF. You can customize it, for example: mobility_domain='1234' mobility_domain='abcd' mobility_domain='beef' But remember: Must be consistent within the same roaming network Different independent networks can differ Must be exactly 4 hex characters 3.3 reassociation_deadline='1000' Sets the Reassociation Deadline. This parameter indicates the maximum time window allowed for a client to complete Fast Transition re-association. The unit is usually TU (1 TU ≈ 1.024 ms), so 1000 is roughly 1 second. In short: After a client initiates fast roaming, it must complete re-association within this window. For home networks, 1000 is a common, generous, and safe choice. Too short may prevent some devices from finishing the switch; too long generally has no real benefit. 3.4 ft_over_ds='0' Sets the 802.11r Fast Transition method. There are two common approaches: FT over the Air FT over DS Here we use FT over the Air by setting it to 0 (disabling ft_over_ds). FT over the Air: The client performs the fast handover directly with the target AP. This is more common and intuitive in non-enterprise environments. FT over DS: The client communicates with the target AP through the currently connected AP via the distribution system. The device contacts the new AP via the old one before switching. In real home OpenWrt/QWRT multi-AP setups, this doesn’t bring a clear advantage and can sometimes cause device compatibility issues. 3.5 ft_psk_generate_local='1' Lets the local AP generate 802.11r keys from the PSK. In home networks using WPA-PSK / SAE Mixed mode, the AP can locally derive the key material needed for Fast Transition from the Wi-Fi password. Home networks usually lack an enterprise authentication server, so just enable this. Suitable for: WPA2-PSK WPA2/WPA3 Mixed SAE mixed (depends on firmware support) Typical home networks without RADIUS 4. NAS ID: Unique Identity for Each BSS uci set wireless.ath0.nasid='Master_2_4G' uci set wireless.ath1.nasid='Master_5G' uci set wireless.ath2.nasid='Master_5G2' 4.1 nasid='Master_2_4G' nasid is the NAS Identifier, the identity of the current BSS. In 802.11r scenarios, it’s used to distinguish different APs / BSSs. Every wireless interface participating in roaming should have a unique nasid. For the main router: ath0 -> Master_2_4G ath1 -> Master_5G ath2 -> Master_5G2 For the secondary router, set correspondingly: 2.4G -> Slave_2_4G 5G-1 -> Slave_5G 5G-2 -> Slave_5G2 5. Why configure the same set of parameters for all three ath interfaces? Because ath0, ath1, and ath2 are three different wireless BSSs. Even if they broadcast the same SSID, they remain independent wireless interfaces at the system level, so 802.11k/v/r parameters must be written to each interface individually. This part can be summarized as: Parameter Protocol Function ieee80211k 802.11k Enable radio resource measurement rrm_neighbor_report 802.11k Allow AP to provide neighbor AP list rrm_beacon_report 802.11k Allow client to report scanned beacon info ieee80211v 802.11v Enable wireless network management bss_transition 802.11v Allow AP to send roaming suggestions time_advertisement 802.11v Time advertisement; usually disabled for home roaming wnm_sleep_mode 802.11v WNM power save; usually disabled for home roaming ieee80211r 802.11r Enable fast roaming nasid 802.11r / hostapd Identifies current BSS; should be unique mobility_domain 802.11r Set fast roaming domain; must be identical across all APs reassociation_deadline 802.11r Set fast re-association time window ft_over_ds 802.11r Select FT method; 0 = FT over the Air ft_psk_generate_local 802.11r Generate FT keys locally from PSK Secondary Router Just enable the corresponding settings directly in LuCI. You can refer to my configuration: Apply the same settings to all three wireless interfaces, while ensuring the NAS ID is different. Band Analysis My home network layout is somewhat complex. The main router sits in the center of the living room, which opens directly onto the kitchen and balcony, so its signal covers those areas well. The secondary router is in the study, flanked by two bedrooms. The study and living room are linked by a short corridor, and Bedroom A is separated from the living room by a bathroom: The two stars mark the main router (living room) and the secondary router (study). Signal collection results in each room: Master Bedroom Study (the tall red peak is from the secondary router) Second Bedroom It's clear: the master bedroom’s channels are relatively clean; the study has a strong signal because the secondary router is right there; but the second bedroom suffers from heavy interference, a weaker signal, and both our APs (the two strongest red Wi-Fi signals) are crowded on the same channel. As a result, actual speed tests in the second bedroom only reached about 80 Mbps. So, we need to adjust channels and power, and also configure 802.11k/v/r. Tuning Power and Channels The main goal is to separate the channels and prevent clients from sticking to one specific access point. Honestly, I can’t fully explain the “why” behind all of this; I also consulted AI for many settings. I’ll let the AI explain here too (lol). 1. Channel Strategy: Completely non-overlapping, avoiding co-channel interference The main and secondary routers use non-overlapping channels across all bands – the most critical step in multi-AP setups. 2.4 GHz band (Main 1 / Secondary 11): In 2.4 GHz, only channels 1, 6, and 11 are fully non-overlapping. Assigning 1 and 11 ensures the two devices don’t “collide” (co-channel interference), which guarantees stability for IoT devices that rely on 2.4 GHz. 5 GHz-1 band (Main 36 / Secondary 52): The main router uses the low channel 36; the secondary uses DFS channel 52. These are completely independent at 80 MHz bandwidth. 5 GHz-2 band (Main 149 / Secondary 157): The two routers’ high-band 5 GHz channels are also separated. Summary: This spatial channel isolation minimizes background noise and maximizes network throughput as devices move between the routers. 2. Bandwidth Strategy: Balancing stability and peak speed 2.4 GHz set to 20 MHz: A very wise move. Although 40 MHz is theoretically faster, the extremely crowded 2.4 GHz band (microwaves, Bluetooth) would experience multiplied interference with 40 MHz, causing frequent dropouts. Locking to 20 MHz sacrifices peak speed but maximizes wall-penetration stability and coverage, ideal for speed-insensitive IoT devices. 5 GHz-1 set to 80 MHz: 80 MHz is the mainstream sweet spot for most phones and laptops, offering very high LAN throughput and WAN download speeds – the primary high-speed band. 5 GHz-2 set to 40 MHz: An interesting strategy. Limiting the second 5 GHz radio to 40 MHz saves valuable wireless spectrum (reducing interference to neighbors) and serves as a high-stability backup high-speed network, suitable for older devices that don’t support 80 MHz or for isolating specific devices. 3. Power Strategy: “Weak 2.4 GHz, Strong 5 GHz” This power configuration is the most brilliant part; it perfectly solves the “sticky client” problem in multi-AP environments. 2.4 GHz power lowered (18 dBm / 20 dBm): 2.4 GHz signals have long wavelengths and penetrate walls extremely well. Without reducing power, a phone moving around the house will stubbornly “cling” to a distant 2.4 GHz signal, resulting in terrible speeds. Lowering the 2.4 GHz power on both routers artificially shrinks the 2.4 GHz coverage circles, encouraging devices to disconnect earlier when the signal weakens and search for a better one. 5 GHz power maxed out (23 dBm / 24 dBm): 5 GHz signals penetrate poorly and attenuate quickly. Keeping high power compensates for this weakness and expands the high-speed 5 GHz coverage area. Summary: This power differential creates a natural band steering effect at the physical layer. When a phone sees both 2.4 GHz and 5 GHz signals, the strong 5 GHz signal can easily exceed the weak 2.4 GHz signal, making the phone “happily” prefer the faster 5 GHz network. 4. Roaming Coordination With the physical-layer channels, bandwidth, and power properly tuned, the 802.11k/v/r protocols complete the seamless roaming loop: 11k (Neighbor Report) + 11v (BSS Transition Management): The router actively tells the phone “which nearby node has a better signal” and suggests a switch. Because the 2.4 GHz power is suppressed, the phone easily triggers the 11v threshold when moving. 11r (Fast Transition): Combined with a unified SSID (PINer) and matching encryption, this eliminates the hundreds of milliseconds otherwise needed to re-authenticate when switching APs, achieving a truly “seamless” experience (e.g., WeChat voice calls don’t drop). The identical mobility_domain and unique nasid are also standard 11r requirements. Ref: Gemini The final channel, bandwidth, and power configuration: Main Router 192.168.3.1 2.4G: channel 1 / 20 MHz / 18 dBm 5G-1: channel 36 / 80 MHz / 24 dBm 5G-2: channel 149 / 40 MHz / 24 dBm Secondary Router 192.168.3.2 2.4G: channel 11 / 20 MHz / 20 dBm 5G-1: channel 52 / 80 MHz / 23 dBm 5G-2: channel 157 / 40 MHz / 24 dBm Results & Follow-up Testing shows: in the living room, balcony, kitchen, and bathroom, devices default to the main router. In the second bedroom, they automatically switch to the secondary router. In the study, closing the door also triggers a switch from main to secondary. From signal degradation to actual handover takes about 5 seconds. The master bedroom is less deterministic – sometimes it switches, sometimes not; in 5 tests, it switched 3 times. Speed test after connecting to the 5 Gbps port: The actual downstream speed reached 1336.31 Mbps, and upstream 158.16 Mbps. Looks like a success? However, the real-world experience isn’t dramatically different, because few servers can saturate such speeds, plus ISP policies like QoS make it hard to reach the limit. But a multi-threaded download tool like IDM might make better use of it?
Building a Cross-Region K3s Cluster from Scratch - Calico No-Encapsulation CNI
# Preface I've actually wanted to play with a K8s cluster for a long time, but always felt that without sufficient knowledge, it would be too difficult to attempt. Recently, I spent some time studying DN42 and routing protocols like BGP and OSPF, and realized that it no longer feels so difficult. So I decisively started with K3s ( The main reason for choosing K3s over K8s is its lightweight nature: low resource requirements, no need to pull a bunch of images for deployment, availability of domestic mirrors… In short, K3s suits my needs better. I'm a beginner just starting to explore K3s, so please go easy on me if I make any mistakes~ # Analysis ## Choosing the CNI Component My current network architecture looks like this: ```mermaid graph TD subgraph ZeroTier Domestic subgraph WDS Gateway <--> VM1 Gateway <--> VM2 end NGB <--> Gateway HFE-NAS <--> Gateway NGB <--> HFE-NAS end subgraph IEPL Global-NIC <==OSPF==> CN-NIC end subgraph ZeroTier Global HKG02 <--> HKG04 TYO <--> HKG04 TYO <--> HKG02 end CN-NIC <--> NGB CN-NIC <--> HFE-NAS CN-NIC <--OSPF--> Gateway Global-NIC <--OSPF--> TYO Global-NIC <--OSPF--> HKG02 Global-NIC <--OSPF--> HKG04 %% Style definition: orange background, bold border to represent routers classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold; class Global-NIC,CN-NIC,Gateway router; Among this, the WDS node is a Proxmox VE host with multiple VMs underneath. It advertises its VMs' IPv4 prefixes via OSPF. When Hong Kong nodes need to access a VM under the WDS node, they can do so by joining the OSPF internal network to achieve multi-hop reachability. This keeps the encapsulation layer count to only one, so there's no worry about MTU "disappearing act". I plan to create two new VMs under WDS to serve as the master and a node (temporarily called KubeMaster and KubeNode-WDS1). Then HKG04 (temporarily called KubeNode-HKG04) will also join the K3s cluster as a node. The simplest approach would be to use K3s's default Flannel as the CNI. However, Flannel is based on VXLAN, and adding another layer of my existing internal network would lead to the following MTU "disappearing act": Data packet -> Flannel VXLAN encapsulation -> ZeroTier encapsulation -> Physical link The actual usable MTU for inter-container communication would likely be compressed to 1350 or even lower. Therefore, I tried to find a CNI solution that can work directly on top of this internal network, and then I found Calico. As I understand, Calico uses BGP as its underlying routing protocol, supports starting in no-encapsulation (No-Encap) mode, and hands packets directly to the upper routers for routing. Thus, I chose Calico as the CNI component. Routing Design To ensure that intermediate routers know how to route Pod IPs, KubeMaster and KubeNode-WDS1 are under the Proxmox VE host. They need to establish BGP with HKG04 across the entire internal network. This means that every router at each intermediate level must learn the full BGP routes, so that the following routing path can be established: graph LR subgraph WDS KubeMaster KubeNode-WDS1 Gateway end subgraph IEPL CN-Namespace Global-Namespace end KubeNode-WDS1 <--> Gateway KubeMaster <--> Gateway <--> CN-Namespace <--> Global-Namespace <--> HKG04 %% Style definition: highlight nodes with routing capability classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold; class Gateway,CN-Namespace,Global-Namespace router; Otherwise, any intermediate hop would drop packets because it doesn't recognize the source/destination IP. Also, due to the property of iBGP that routes learned from a neighbor cannot be propagated to the next iBGP neighbor, all BGP sessions between Gateway, CN-Namespace, Global-Namespace and the nodes need to enable Route Reflector; otherwise, nodes cannot correctly learn routes from each other. That said, this architecture would be more suitable for BGP Confederation, but my existing network is already quite complex, and adding BGP confederations would make later maintenance more troublesome. Moreover, my number of nodes is small, so the overhead of iBGP Full Mesh is acceptable. It's definitely not because I'm lazy (so Thus, the final network routing structure is as follows: graph TD subgraph WDS VM1 VM2 Gateway end subgraph IEPL CN-Namespace Global-Namespace end VM1 <-.Calico iBGP Full Mesh.-> VM2 VM1 <--iBGP Route Reflector--> Gateway VM2 <--iBGP Route Reflector--> Gateway <--iBGP--> CN-Namespace <--iBGP--> Global-Namespace <--iBGP Route Reflector--> HKG04 Gateway <--iBGP--> Global-Namespace HKG04 <-.Calico iBGP Full Mesh.-> VM1 VM2 <-.Calico iBGP Full Mesh.-> HKG04 %% Style definition classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold; %% Mark nodes with routing/forwarding or RR functions as Router class Gateway,CN-Namespace,Global-Namespace router; The dashed-line BGP sessions are automatically created by Calico, while the solid-line parts need to be manually created by us. Keeping Calico's own iBGP Full Mesh is for future scalability, so that nodes can preferentially establish direct P2P connections via ZeroTier instead of taking a detour through the Route Reflector aggregation router. Deployment After clarifying the structure, deployment becomes simple. Enable Kernel Forwarding and Disable rp_filter Standard practice. echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf echo "net.ipv6.conf.default.forwarding=1" >> /etc/sysctl.conf echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf echo "net.ipv4.conf.default.rp_filter=0" >> /etc/sysctl.conf echo "net.ipv4.conf.all.rp_filter=0" >> /etc/sysctl.conf sysctl -p Install K3s Master Because the KubeMaster control plane node is located inside China, it's best to configure image acceleration: mkdir -p /etc/rancher/k3s cat <<EOF > /etc/rancher/k3s/registries.yaml mirrors: docker.io: endpoint: - "https://docker.m.daocloud.io" quay.io: endpoint: - "https://quay.m.daocloud.io" EOF Install using the mirror: curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | \ INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC=" \ --flannel-backend=none \ --disable-network-policy \ --cluster-cidr=10.42.0.0/16" sh - Note the need to specify --flannel-backend=none and --disable-network-policy to disable the default CNI component. Use cat /var/lib/rancher/k3s/server/node-token to view the token and record it. Worker Nodes For nodes inside China, configure image acceleration: mkdir -p /etc/rancher/k3s cat <<EOF > /etc/rancher/k3s/registries.yaml mirrors: docker.io: endpoint: - "https://docker.m.daocloud.io" quay.io: endpoint: - "https://quay.m.daocloud.io" EOF Then install K3s using the mirror and join the cluster: export INSTALL_K3S_MIRROR=cn export K3S_URL=https://<master node IP>:6443 # Replace with your master node's actual IP export K3S_TOKEN=K10...your token...::server:xxx # Replace with the full token obtained in the first step curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | sh - At this point, the status of each node should be NotReady because the CNI component is missing. Install Calico and Configure No-Encap Mode On the master, manually download https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml and install the Calico operator: kubectl create -f tigera-operator.yaml Configure a custom resource by creating a custom-resource.yaml file: apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Add image registry configuration registry: quay.m.daocloud.io calicoNetwork: ipPools: - blockSize: 26 cidr: 10.42.0.0/16 encapsulation: None natOutgoing: Enabled nodeSelector: all() Here, specify encapsulation: None to enable No-Encap mode. You can also modify the IPv4 CIDR here if needed. Then: kubectl apply -f custom-resource.yaml to perform the installation. Use: kubectl get pods -A -o wide to check Pod status, waiting for each node to finish pulling images. Configure BGP Topology Label Nodes Label nodes to specify that nodes under WDS connect to the Gateway's BGP in the WDS node, and nodes outside China connect to the BGP of the Global Namespace: kubectl label nodes kubemaster region=WDS kubectl label nodes kubenode-wds-1 region=WDS kubectl label nodes kubenode-hkg04 region=Global Calico Configuration Create a YAML configuration file: apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: route-reflector-domestic spec: nodeSelector: region == 'Domestic' # This part is not actually used; I originally designed a general aggregation router in the Domestic area peerIP: 100.64.0.108 asNumber: 64512 --- apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: route-reflector-wds spec: nodeSelector: region == 'WDS' peerIP: 192.168.100.1 asNumber: 64512 --- apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: route-reflector-global spec: nodeSelector: region == 'Global' peerIP: 100.64.1.106 asNumber: 64512 This means: All nodes with label region equal to Domestic will have a BGP session to 100.64.0.108 (the domestic aggregation router) using AS 64512 All nodes with label region equal to WDS will have a BGP session to 192.168.100.1 (the Gateway for all VMs under the WDS node) using AS 64512 All nodes with label region equal to Global will have a BGP session to 100.64.1.106 (the overseas aggregation router) using AS 64512 This achieves what is shown in the diagram: all VMs under the WDS node, including the master and KubeNode-WDS1, connect to the Gateway aggregation router of the WDS node, and all nodes in overseas areas connect to the overseas aggregation router. Configure Aggregation Router iBGP This part is simply a matter of writing Bird configuration files (easy). Here are a few examples: k3s/ibgp.conf: function is_insider_as(){ if bgp_path.len > 0 && !(bgp_path ~ [= 64512 =]) then { return false; } if net ~ [ 10.42.0.0/16{16,32} ] then { return true; } return false; } template bgp k3sbackbone{ local as K3S_AS; router id INTRA_ROUTER_ID; neighbor as K3S_AS; ipv4{ table intra_table_v4; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; extended next hop; }; ipv6{ table intra_table_v6; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; }; }; template bgp k3speers{ local as K3S_AS; neighbor as K3S_AS; router id INTRA_ROUTER_ID; rr client; rr cluster id INTRA_ROUTER_ID; ipv4{ table intra_table_v4; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; }; ipv6{ table intra_table_v6; import filter{ if is_insider_as() then accept; reject; }; export filter{ if is_insider_as() then accept; reject; }; next hop self; }; }; include "ibgpeers/*"; ibgpeers/backbone-cn.conf: protocol bgp 'k3s_backbone_cn_v4' from k3sbackbone{ neighbor fd18:3e15:61d0:cafe:f001::1; }; ibgpeers/master.conf: protocol bgp 'k3s_master_v4' from k3speers{ neighbor 192.168.100.251; }; Main points: it's best not to enable Route Reflector between the aggregation routers, and remember to enable next hop self. After everything is done, using kubectl get nodes should show all nodes as Ready: NAME STATUS ROLES AGE VERSION kubemaster Ready control-plane 2d23h v1.34.5+k3s1 kubenode-hkg04 Ready <none> 11h v1.34.6+k3s1 kubenode-wds-1 Ready <none> 2d7h v1.34.5+k3s1 Use kubectl get pods -A -o wide to view Pods: NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-system calico-kube-controllers-64fc874957-6bdlz 1/1 Running 0 5h38m 10.42.253.136 kubenode-hkg04 <none> <none> calico-system calico-node-2qz82 1/1 Running 0 4h24m 10.2.5.7 kubenode-hkg04 <none> <none> calico-system calico-node-dhl2c 1/1 Running 0 4h24m 192.168.100.251 kubemaster <none> <none> calico-system calico-node-nbpkj 1/1 Running 0 4h23m 192.168.100.252 kubenode-wds-1 <none> <none> calico-system calico-typha-7bb5db4bdc-rfpwg 1/1 Running 0 5h38m 10.2.5.7 kubenode-hkg04 <none> <none> calico-system calico-typha-7bb5db4bdc-rwwr5 1/1 Running 0 5h38m 192.168.100.251 kubemaster <none> <none> calico-system csi-node-driver-jglwp 2/2 Running 0 5h38m 10.42.64.68 kubenode-wds-1 <none> <none> calico-system csi-node-driver-jqjsc 2/2 Running 0 5h38m 10.42.253.137 kubenode-hkg04 <none> <none> calico-system csi-node-driver-vk26s 2/2 Running 0 5h38m 10.42.141.16 kubemaster <none> <none> kube-system coredns-695cbbfcb9-8fx4p 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.14 kubemaster <none> <none> kube-system helm-install-traefik-crd-5bkwx 0/1 Completed 0 2d23h <none> kubemaster <none> <none> kube-system helm-install-traefik-m9fgj 0/1 Completed 1 2d23h <none> kubemaster <none> <none> kube-system local-path-provisioner-546dfc6456-dmn4g 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.15 kubemaster <none> <none> kube-system metrics-server-c8774f4f4-2wkwh 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.12 kubemaster <none> <none> kube-system svclb-traefik-999cddce-hpmcm 2/2 Running 6 (7h26m ago) 11h 10.42.253.134 kubenode-hkg04 <none> <none> kube-system svclb-traefik-999cddce-q4225 2/2 Running 2 (7h27m ago) 2d22h 10.42.141.9 kubemaster <none> <none> kube-system svclb-traefik-999cddce-xmd64 2/2 Running 2 (7h26m ago) 2d6h 10.42.64.66 kubenode-wds-1 <none> <none> kube-system traefik-788bc4688c-vbbhj 1/1 Running 1 (7h27m ago) 2d22h 10.42.141.13 kubemaster <none> <none> tigera-operator tigera-operator-6b95bbf4db-vl46l 1/1 Running 1 (7h27m ago) 2d23h 192.168.100.251 kubemaster <none> <none> Use kubectl exec -it -n calico-system <calico-node-xxxx> -- birdcl s p to check the status of Bird: root@KubeMaster:~/kube/calico# kubectl exec -it -n calico-system calico-node-2qz82 -- birdcl s p Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init) BIRD v0.3.3+birdv1.6.8 ready. name proto table state since info static1 Static master up 08:58:17 kernel1 Kernel master up 08:58:17 device1 Device master up 08:58:17 direct1 Direct master up 08:58:17 Mesh_192_168_100_251 BGP master up 08:58:33 Established Mesh_192_168_100_252 BGP master up 08:59:00 Established Node_100_64_1_106 BGP master up 12:57:44 Established ip r shows the system routing table: root@KubeMaster:~/kube/calico# ip r default via 192.168.100.1 dev eth0 proto static 10.42.64.64/26 proto bird nexthop via 192.168.100.1 dev eth0 weight 1 nexthop via 192.168.100.252 dev eth0 weight 1 blackhole 10.42.141.0/26 proto bird 10.42.141.9 dev caliac6501d3794 scope link 10.42.141.12 dev calib07c23291bb scope link 10.42.141.13 dev caliab16e60bd19 scope link 10.42.141.14 dev calid5959219080 scope link 10.42.141.15 dev cali026d8f1ddb7 scope link 10.42.141.16 dev califa657ba417a scope link 10.42.253.128/26 via 192.168.100.1 dev eth0 proto bird 192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.251 Ping a Pod's IP – if everything is fine, it should work directly: root@KubeMaster:~/kube/calico# ping 10.42.253.137 PING 10.42.253.137 (10.42.253.137) 56(84) bytes of data. 64 bytes from 10.42.253.137: icmp_seq=1 ttl=60 time=33.7 ms 64 bytes from 10.42.253.137: icmp_seq=2 ttl=60 time=33.5 ms ^C --- 10.42.253.137 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 33.546/33.632/33.718/0.086 ms Tune MTU This step is actually for stability…? Tests have shown that although my ZeroTier MTU is 1420, packets start to fragment around 1392 bytes (test with ping -M do -s <packet size> <Pod_IP>). Therefore, force the Pod MTU to 1370: root@KubeMaster:~/kube/calico# cat patch-mtu.yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: mtu: 1370 nodeAddressAutodetectionV4: firstFound: true root@KubeMaster:~/kube/calico# kubectl apply -f patch-mtu.yaml installation.operator.tigera.io/default configured
[Fun Experiment] A LAN Spanning 20km: Seamlessly Merging Remote Networks on OpenWrt Using ZeroTier + OSPF
Background I was originally setting up my own ZeroTier "big internal network". Because the network structure is relatively complex, I decided to use OSPF instead of static routes to configure internal routing. I had tried to configure ZeroTier on my home OpenWrt before but never succeeded. Recently, I took it out again to work on it and discovered it was a configuration issue with OpenWrt. After fixing it, I was chatting with a good friend and had an idea: Kagura iYoRoy: 02-10 14:49:05 Hey... Kagura iYoRoy: 02-10 14:49:06 Then... Kagura iYoRoy: 02-10 14:49:20 If you also set up OSPF on your router... Kagura iYoRoy: 02-10 14:49:27 Our two home networks would be directly interconnected, huh? ( Let's do it! Basic Information Local Side Router OS: OpenWrt, X-WRT 26.04_b202601250827 LAN IPv4 Prefix: 192.168.3.0/24 ISP: Hefei China Unicom NAT Environment: NAT1 Remote Side Router OS: OpenWrt, X-WRT 25.04_b202510240128 LAN IPv4 Prefix: 192.168.1.0/24 ISP: Hefei China Mobile NAT Environment: NAT1 Installing ZeroTier and Using a Self-Hosted Planet I used ZTNet as the self-hosted Controller. The setup process won't be elaborated here as you can find it online. The OpenWrt version I'm using has started using apk instead of opkg as the package manager. Use apk to install zerotier-one directly: apk add zerotier After completion, open /etc/config/zerotier to find the default configuration file. config zerotier 'global' # Sets whether ZeroTier is enabled or not option enabled 0 # Sets the ZeroTier listening port (default 9993; set to 0 for random) #option port '9993' # Client secret (leave blank to generate a secret on first run) option secret '' # Path of the optional file local.conf (see documentation at # https://docs.zerotier.com/config#local-configuration-options) #option local_conf_path '/etc/zerotier.conf' # Persistent configuration directory (to perform other configurations such # as controller mode or moons, etc.) #option config_path '/etc/zerotier' # Copy the contents of the persistent configuration directory to memory # instead of linking it, this avoids writing to flash #option copy_config_path '1' # Network configuration, you can have as many configurations as networks you # want to join (the network name is optional) config network 'earth' # Identifier of the network you wish to join option id '8056c2e21c000001' # Network configuration parameters (all are optional, if not indicated the # default values are set, see documentation at # https://docs.zerotier.com/config/#network-specific-configuration) option allow_managed '1' option allow_global '0' option allow_default '0' option allow_dns '0' # Example of a second network (unnamed as it is optional) #config network # option id '1234567890123456' # option allow_managed '1' # option allow_global '0' # option allow_default '0' # option allow_dns '0' Modify it according to your needs: config zerotier 'global' option enabled '1' # Enable ZeroTier client service option config_path '/etc/zerotier' # Persistent directory: for storing identity secret, Moon node definitions, and network settings option secret '' # Leave secret blank: identity will be auto-generated on first run and saved to identity.secret option copy_config_path '1' # Flash protection policy: copy config to memory on startup. If set to 0, read/write directly to Flash config network 'earth' option id '<network ID>' # 16-digit ZeroTier Network ID option allow_managed '1' # Allow receiving controller-assigned IPs, routes, and tags option allow_global '1' # Allow receiving globally routable IPv6 unicast addresses (GUA) via ZeroTier option allow_default '0' # Allow ZeroTier to take over the default gateway (similar to a global proxy) option allow_dns '1' # Allow receiving and using DNS servers configured in the ZeroTier control panel Regarding copy_config_path '1' Because the ZeroTier working directory /var/lib/zerotier-one is part of tmpfs in OpenWrt, its contents are cleared on reboot. Therefore, configurations like planet, identity, and network files need to be stored in the router's Flash storage, i.e., the path set in config_path. The default logic is to create a soft link from the configured config_path to /var/lib/zerotier-one on startup to achieve persistence. All read/write operations in /var/lib/zerotier-one are then written to Flash. However, frequent ZeroTier read/writes can significantly reduce Flash lifespan. Enabling copy_config_path '1' specifies that on ZeroTier startup, the configurations from config_path are copied directly into /var/lib/zerotier-one. This greatly extends the internal Flash lifespan, but the downside is that modifications made via zerotier-cli are not automatically synced back to Flash by default, making this option less suitable for scenarios requiring frequent configuration adjustments. After making changes, use: /etc/init.d/zerotier start /etc/init.d/zerotier enable to start ZeroTier and enable auto-start on boot. On first startup, if the secret field was left empty, it will be auto-generated. After startup, copy all files from /var/lib/zerotier-one to /etc/zerotier. Download the Planet file to the config_path set above, i.e., /etc/zerotier. After completion, restart ZeroTier: /etc/init.d/zerotier restart That's it. Then, go to your ZeroTier Controller console, and you should see the new device has joined. Next, you may need to allow ZeroTier traffic through the firewall. This step can be referenced from other online tutorials. I chose to allow all traffic; it shouldn't be a big issue under NAT1. Installing and Configuring Bird2 I didn't expect the Bird2 version in the apk repository to be very recent. As of this writing on 2026-02-10, the Bird2 version in apk is 2.18 Use the following command to install: apk add bird2 # bird daemon itself apk add bird2c # birdc command Because OpenWrt's default bird configuration file is located at /etc/bird.conf, and I prefer modular referencing by placing different configurations in separate folders based on function, I chose to move the default config file to /etc/bird/bird.conf and store various config files within that folder. Open /etc/init.d/bird: #!/bin/sh /etc/rc.common # Copyright (C) 2010-2017 OpenWrt.org USE_PROCD=1 START=70 STOP=10 BIRD_BIN="/usr/sbin/bird" BIRD_CONF="/etc/bird.conf" BIRD_PID_FILE="/var/run/bird.pid" start_service() { mkdir -p /var/run procd_open_instance procd_set_param command $BIRD_BIN -f -c $BIRD_CONF -P $BIRD_PID_FILE procd_set_param file "$BIRD_CONF" procd_set_param stdout 1 procd_set_param stderr 1 procd_set_param respawn procd_close_instance } reload_service() { procd_send_signal bird } Change the BIRD_CONF value to /etc/bird/bird.conf: - BIRD_CONF="/etc/bird.conf" + BIRD_CONF="/etc/bird/bird.conf" Then create the /etc/bird folder. All subsequent OSPF configuration files will be placed here. Configuring OSPF My configuration file structure follows these rules: /etc/bird/bird.conf serves as the sole entry point, defining basic configurations like Router ID, filter prefixes, and then including other sub-configurations. Configurations for different networks are placed in separate folders, e.g., public internet parts in /etc/bird/inet/, DN42 parts in /etc/bird/dn42/, and my own internal network parts in /etc/bird/intra/. Each network has a defs.conf handling common functions (similar to utils in Golang development?). Thus, the final configuration file structure is: /etc/bird/bird.conf: Configuration entry point define INTRA_ROUTER_ID = 100.64.0.100; define INTRA_PREFIX_V4 = [ 100.64.0.0/16+, 192.168.0.0/16+ ]; # IPv4 prefixes allowed to be advertised via OSPF define INTRA_PREFIX_V6 = [ fd18:3e15:61d0::/48+ ]; # IPv6 prefixes allowed to be advertised via OSPF protocol device { scan time 10; }; ipv4 table intra_table_v4; # Define internal routing IPv4 table ipv6 table intra_table_v6; # Define internal routing IPv6 table include "intra/defs.conf"; include "intra/kernel.conf"; include "intra/ospf.conf"; The RouterID here is directly taken from the node's IPv4 address within the ZeroTier internal network. Separate tables are used for future safety, e.g., if connecting this node to DN42. /etc/bird/intra/defs.conf: Functions for filters function is_intra_net4() { return net ~ INTRA_PREFIX_V4; } function is_intra_net6(){ return net ~ INTRA_PREFIX_V6; } function is_intra_dn42_net4(){ return net ~ [ 172.20.0.0/14+ ]; } function is_intra_dn42_net6(){ return net ~ [ fd00::/8+ ]; } /etc/bird/intra/kernel.conf: Write routes learned by OSPF into the system routing table protocol kernel intra_kernel_v4 { kernel table 254; scan time 20; ipv4 { table intra_table_v4; import none; export filter { if source = RTS_STATIC then reject; accept; }; }; }; protocol kernel intra_kernel_v6 { kernel table 254; scan time 20; ipv6 { table intra_table_v6; import none; export filter { if source = RTS_STATIC then reject; accept; }; }; }; /etc/bird/intra/ospf.conf: OSPF module protocol ospf v3 intra_ospf_v4 { router id INTRA_ROUTER_ID; # Specify RouterID ipv4 { table intra_table_v4; # Specify routing table import where is_intra_dn42_net4() || is_intra_net4() && source != RTS_BGP; export where is_intra_dn42_net4() || is_intra_net4() && source != RTS_BGP; }; include "ospf/*"; }; protocol ospf v3 intra_ospf_v6 { router id INTRA_ROUTER_ID; # Specify RouterID ipv6 { table intra_table_v6; # Specify routing table import where is_intra_dn42_net6() || is_intra_net6() && source != RTS_BGP; export where is_intra_dn42_net6() || is_intra_net6() && source != RTS_BGP; }; include "ospf/*"; }; /etc/bird/intra/ospf/backbone.conf: OSPF Area Configuration area 0.0.0.0 { interface "br-lan" { stub; }; # Local LAN interface interface "zta7oqfzy6" { # ZeroTier interface type broadcast; cost 100; hello 20; }; }; After completion, use: /etc/init.d/bird start /etc/init.d/bird enable to start Bird and enable auto-start on boot. If everything is fine, you can use birdc s p to check Bird's status. If all goes well, after the other side is configured, you should see the OSPF state as Running: root@X-WRT:/etc/bird# birdc s p BIRD 2.18 ready. Name Proto Table State Since Info device1 Device --- up 14:28:02.410 intra_kernel_v4 Kernel intra_table_v4 up 14:28:02.410 intra_kernel_v6 Kernel intra_table_v6 up 14:28:02.410 intra_ospf_v4 OSPF intra_table_v4 up 14:28:02.410 Running intra_ospf_v6 OSPF intra_table_v6 up 14:31:38.389 Running Have your friend follow the same process. Once both sides show Running status, you can use birdc s r protocol intra_ospf_v4 to view the routes learned by OSPF. You'll find that routes to the other side via ZeroTier are being learned normally: root@X-WRT:/etc/bird# birdc s r protocol intra_ospf_v4 BIRD 2.18 ready. Table intra_table_v4: ... 192.168.1.0/24 unicast [intra_ospf_v4 23:20:21.398] * I (150/110) [100.64.0.163] via 100.64.0.163 on zta7oqfzy6 ... 192.168.3.0/24 unicast [intra_ospf_v4 14:28:02.511] * I (150/10) [100.64.0.100] dev br-lan You can also ping your friend's server from your PC: iyoroy@iYoRoy-PC:~$ ping 192.168.1.103 PING 192.168.1.103 (192.168.1.103) 56(84) bytes of data. 64 bytes from 192.168.1.103: icmp_seq=1 ttl=63 time=54.3 ms 64 bytes from 192.168.1.103: icmp_seq=2 ttl=63 time=10.7 ms 64 bytes from 192.168.1.103: icmp_seq=3 ttl=63 time=15.2 ms ^C --- 192.168.1.103 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 10.678/26.717/54.279/19.576 ms iyoroy@iYoRoy-PC:~$ traceroute 192.168.1.103 traceroute to 192.168.1.103 (192.168.1.103), 30 hops max, 60 byte packets 1 100.64.0.163 (100.64.0.163) 10.445 ms 9.981 ms 9.892 ms 2 192.168.1.103 (192.168.1.103) 11.621 ms 10.994 ms 10.948 ms Web browsing and speed tests work normally: Summary This series of operations essentially implements the following network structure: flowchart TB %% === Style Definitions === classDef phyNet fill:#e3f2fd,stroke:#1565c0,stroke-width:2px classDef virNet fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,stroke-dasharray: 5 5 classDef router fill:#333,stroke:#000,stroke-width:2px,color:#fff classDef ztCard fill:#f57c00,stroke:#e65100,stroke-width:2px,color:#fff,shape:rect classDef bird fill:#a5d6a7,stroke:#2e7d32,stroke-width:1px,color:#000 classDef invisibleContainer fill:none,stroke:none,color:none %% === Physical Layer Containers === subgraph Top_Physical_Layer [" "] direction LR subgraph Left_Side ["My Home (Node A)"] direction TB L_Router[X-WRT Router A]:::router L_LAN[LAN: 192.168.3.0/24] L_LAN <--> L_Router end subgraph Right_Side ["Friend's Home (Node B)"] direction TB R_Router[X-WRT Router B]:::router R_LAN[LAN: 192.168.1.0/24] R_LAN <--> R_Router end end %% === Virtual Layer Container === subgraph Middle_Side [ZeroTier Virtual L2 Network] direction LR subgraph ZT_Stack_A [My Home ZT Access] direction TB L_NIC(zt0: 100.64.0.x):::ztCard L_Bird(Bird OSPF):::bird L_NIC <-.- L_Bird end subgraph ZT_Stack_B [Friend's Home ZT Access] direction TB R_NIC(zt0: 100.64.0.y):::ztCard R_Bird(Bird OSPF):::bird R_NIC <-.- R_Bird end L_NIC <==P2P Tunnel==> R_NIC end %% === Cross-Layer Connections === L_Router === L_NIC R_Router === R_NIC %% === Style Application === class Left_Side,Right_Side phyNet class Middle_Side virNet class Top_Physical_Layer invisibleContainer The underlying P2P network is still powered by ZeroTier. However, using OSPF for internal routing allows both sides to directly route to devices on each other's network segments. Since both sides can fully learn each other's routes, no NAT is required, and both sides can directly see each other's source addresses. Check out the other side of this story! From my friend's side: Linux Operations - OSPF Networking Implementation Based on Bird for New OpenWrt » NanamiのTechLaunchTower
An Experience of Manually Installing Proxmox VE, Configuring Multipath iSCSI, and NAT Forwarding
The reason was that I rented a physical server, but the IDC did not provide Proxmox VE or Debian system images, only Ubuntu, CentOS, and Windows series. Additionally, the data disk was provided via multipath iSCSI. I wanted to use PVE for isolating different usage scenarios, so I attempted to reinstall the system and migrate the aforementioned configurations. Backup Configuration First, perform a general check of the system, which reveals: The system has two Network Interfaces: enp24s0f0 is connected to a public IP address for external access; enp24s0f1 is connected to the private network address 192.168.128.153. The data disk is mapped to /dev/mapper/mpatha. Under /etc/iscsi, there are configurations for two iSCSI Nodes: 192.168.128.250:3260 and 192.168.128.252:3260, both corresponding to the same target iqn.2024-12.com.ceph:iscsi. It can be inferred that the data disk is mounted by configuring two iSCSI Nodes and then merging them into a single device using multipath. Check the system's network configuration: network: version: 2 renderer: networkd ethernets: enp24s0f0: addresses: [211.154.[REDACTED]/24] routes: - to: default via: [REDACTED] match: macaddress: ac:1f:6b:0b:e2:d4 set-name: enp24s0f0 nameservers: addresses: - 114.114.114.114 - 8.8.8.8 enp24s0f1: addresses: - 192.168.128.153/17 match: macaddress: ac:1f:6b:0b:e2:d5 set-name: enp24s0f1 It's found to be very simple static routing. The internal network interface doesn't even have a default route; just binding the IP is sufficient. Then, save the iSCSI configuration files from /etc/iscsi, which include account and password information. Reinstall Debian Used the bin456789/reinstall script for this reinstallation. Download the script: curl -O https://cnb.cool/bin456789/reinstall/-/git/raw/main/reinstall.sh || wget -O ${_##*/} $_ Reinstall as Debian 13 (Trixie): bash reinstall.sh debian 13 Then, enter the password you want to set as prompted. If all goes well, wait about 10 minutes, and it will automatically complete and reinstall into a clean Debian 13. You can connect via SSH during the process using the set password to check the installation progress. After reinstalling, perform a source change and apt upgrade as usual to get a clean Debian 13. For changing sources, directly refer to the USTC Mirror Site tutorial. Install Proxmox VE This step mainly refers to the Proxmox official tutorial. Note: The Debian installed by the above script sets the hostname to localhost. If you want to change it, please modify it before configuring the Hostname and change the hostname in hosts to your modified hostname, not localhost. Configure Hostname Proxmox VE requires the current hostname to be resolvable to a non-loopback IP address: The hostname of your machine must be resolvable to an IP address. This IP address must not be a loopback one like 127.0.0.1 but one that you and other hosts can connect to. For example, my server IP is 211.154.[CENSORED], I need to add the following record in /etc/hosts: 127.0.0.1 localhost +211.154.[CENSORED] localhost ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters After saving, use hostname --ip-address to check if it outputs the set non-loopback address: ::1 127.0.0.1 211.154.[CENSORED]. Add Proxmox VE Software Repository Debian 13 uses the Deb822 format (though you can use sources.list if you want), so just refer to the USTC Proxmox Mirror Site: cat > /etc/apt/sources.list.d/pve-no-subscription.sources <<EOF Types: deb URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve Suites: trixie Components: pve-no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF Here, a keyring needs to be migrated but I couldn't find one after searching online, so I chose to pull a copy from an existing Proxmox VE server. It's available here: proxmox-keyrings.zip Extract the public key file and place it in /usr/share/keyrings/, then run: apt update apt upgrade -y This will sync the Proxmox VE software repository. Install Proxmox VE Kernel Use the following command to install the PVE kernel and reboot to apply the new kernel: apt install proxmox-default-kernel reboot Afterwards, uname -r should show a kernel version ending with pve, like 6.17.2-2-pve, indicating the new kernel is successfully applied. Install Proxmox VE Related Packages Use apt to install the corresponding packages: apt install proxmox-ve postfix open-iscsi chrony During configuration, you will need to set up the postfix mail server. Official explanation: If you have a mail server in your network, you should configure postfix as a satellite system. Your existing mail server will then be the relay host which will route the emails sent by Proxmox VE to their final recipient. If you don't know what to enter here, choose local only and leave the system name as is. After this, you should be able to access the Web console at https://<your server address>:8006. The account is root, and the password is your root password, i.e., the password configured during the Debian reinstallation. Remove Old Debian Kernel and os-prober Use the following commands: apt remove linux-image-amd64 'linux-image-6.1*' update-grub apt remove os-prober to remove the old Debian kernel, update grub, and remove os-prober. Removing os-prober is not mandatory, but it is recommended by the official guide because it might mistakenly identify VM boot files as multi-boot files, adding incorrect entries to the boot list. At this point, the installation of Proxmox VE is complete and ready for normal use! Configuring Internal Network Interface Because the iSCSI network interface and the public network interface are different, and the reinstallation lost this configuration, the internal network interface needs to be manually configured. Open the Proxmox VE Web interface, go to Datacenter - localhost (hostname) - Network, edit the internal network interface (e.g., ens6f1 here), enter the backed-up IPv4 in CIDR format: 192.168.128.153/17, and check Autostart, then save. Then use the command to set the interface state to UP: ip link set ens6f1 up Now you should be able to ping the internal iSCSI server's IP. Configure Data Disk iSCSI In the previous step, we should have installed the open-iscsi package required for iscsiadm. We just need to reset the nodes according to the backed-up configuration. First, discover the iSCSI storage: iscsiadm -m discovery -t st -p 192.168.128.250:3260 This should yield the two original LUN Targets: 192.168.128.250:3260,1 iqn.2024-12.com.ceph:iscsi 192.168.128.252:3260,2 iqn.2024-12.com.ceph:iscsi Transfer the backed-up configuration files to the server, overwriting the existing configuration in /etc/iscsi. Also, in my backed-up config, I found the authentication configuration: # /etc/iscsi/nodes/iqn.2024-12.com.ceph:iscsi/192.168.128.250,3260,1/default # BEGIN RECORD 2.1.5 node.name = iqn.2024-12.com.ceph:iscsi ... # Some unimportant configurations omitted node.session.auth.authmethod = CHAP node.session.auth.username = [CENSORED] node.session.auth.password = [CENSORED] node.session.auth.chap_algs = MD5 ... # Some unimportant configurations omitted # /etc/iscsi/nodes/iqn.2024-12.com.ceph:iscsi/192.168.128.252,3260,2/default # BEGIN RECORD 2.1.5 node.name = iqn.2024-12.com.ceph:iscsi ... # Some unimportant configurations omitted node.session.auth.authmethod = CHAP node.session.auth.username = [CENSORED] node.session.auth.password = [CENSORED] node.session.auth.chap_algs = MD5 ... # Some unimportant configurations omitted Write these configurations to the new system using: iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.authmethod -v CHAP iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.username -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.password -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.chap_algs -v MD5 iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.session.auth.authmethod -v CHAP iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.session.auth.username -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.session.auth.password -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.chap_algs -v MD5 (I don't know why the auth info needs to be written separately, but testing shows it won't log in without rewriting it.) Then, use: iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 --login iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 --login to log into the Targets. Then use: iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.startup -v automatic iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.startup -v automatic to enable automatic mounting on boot. At this point, checking disks with tools like lsblk should reveal two additional hard drives; in my case, sdb and sdc appeared. Configure Multipath To identify if it's a multipath device, I tried: /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdb /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdc Checking the scsi_id of the two disk devices revealed they were identical, confirming they are the same disk using multi-path for load balancing and failover. Install multipath-tools using apt: apt install multipath-tools Then, create /etc/multipath.conf and add: defaults { user_friendly_names yes find_multipaths yes } Configure multipathd to start on boot: systemctl start multipathd systemctl enable multipathd Then, use the following command to scan and automatically configure the multipath device: multipath -ll It should output: mpatha(360014056229953ef442476e85501bfd7)dm-0LIO-ORG,TCMU device size=500G features='1 queue_if_no_path' hwhandler='1 alua'wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 14:0:0:152 sdb 8:16 active ready running `-+- policy='service-time 0' prio=50 status=active `- 14:0:0:152 sdc 8:16 active ready running This shows the two disks have been recognized as a single multipath device. Now, you can find the multipath disk under /dev/mapper/: root@localhost:/dev/mapper# ls control mpatha mpatha is the multipath aggregated disk. If it's not scanned, try using: rescan-scsi-bus.sh to rescan the SCSI bus and try again. If the command is not found, install it via apt install sg3-utils. If all else fails, just reboot. Configure Proxmox VE to Use the Data Disk Because we used multipath, we cannot directly add an iSCSI type storage. Use the following commands to create the PV and VG: pvcreate /dev/mapper/mpatha vgcreate <vg name> /dev/mapper/mpatha Here, I configured the entire disk as a PV. You could also create a separate partition for this. After completion, open the Proxmox VE management interface, go to Datacenter - Storage, click Add - LVM, select the name of the VG you just created for Volume group, give it an ID (name), and click Add. At this point, all configurations from the original system should have been migrated. Configure NAT and Port Forwarding NAT Because only one IPv4 address was purchased, NAT needs to be configured to allow all VMs to access the internet normally. Open /etc/network/interfaces and add the following content: auto vmbr0 iface vmbr0 inet static address 192.168.100.1 netmask 255.255.255.0 bridge_ports none bridge_stp off bridge_fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.100.0/24 -o ens6f0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-up iptables -A FORWARD -i vmbr0 -j ACCEPT post-down iptables -t nat -D POSTROUTING -s 192.168.100.0/24 -o ens6f0 -j MASQUERADE post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -D FORWARD -i vmbr0 -j ACCEPT Here, vmbr0 is the NAT bridge, with the IP segment 192.168.100.0/24. Traffic from this segment will be translated to the IP of the external network interface ens6f0 for outgoing traffic, and translated back to the original IP upon receiving replies, enabling IP sharing. Then, use: ifreload -a to reload the configuration. Now, the VMs should be able to access the internet. Just configure a static IP within the 192.168.100.0/24 range during installation, set the default gateway to 192.168.100.1, and configure the DNS address. Port Forwarding Got lazy, directly prompted an AI. Had an AI write a configuration script /usr/local/bin/natmgr: #!/bin/bash # =================Configuration Area================= # Public network interface name (Please modify according to your actual situation) PUB_IF="ens6f0" # ==================================================== ACTION=$1 ARG1=$2 ARG2=$3 ARG3=$4 ARG4=$5 # Check if running as root if [ "$EUID" -ne 0 ]; then echo "Please run this script with root privileges" exit 1 fi # Generate random ID (6 characters) generate_id() { # Introduce nanoseconds and random salt to ensure ID uniqueness even if the script runs quickly echo "$RANDOM $(date +%s%N)" | md5sum | head -c 6 } # Show help information usage() { echo "Usage: $0 {add|del|list|save} [parameters]" echo "" echo "Commands:" echo " add <Public Port> <Internal IP> <Internal Port> [Protocol] Add forwarding rule" echo " [Protocol] optional: tcp, udp, both (default: both)" echo " del <ID> Delete forwarding rule by ID" echo " list View all current forwarding rules" echo " save Save current rules to persist after reboot (Must run!)" echo "" echo "Examples:" echo " $0 add 8080 192.168.100.101 80 both" echo " $0 save" echo "" } # Internal function: add single protocol rule _add_single_rule() { local PROTO=$1 local L_PORT=$2 local T_IP=$3 local T_PORT=$4 local RULE_ID=$(generate_id) local COMMENT="NAT_ID:${RULE_ID}" # 1. Add DNAT rule (PREROUTING chain) iptables -t nat -A PREROUTING -i $PUB_IF -p $PROTO --dport $L_PORT -j DNAT --to-destination $T_IP:$T_PORT -m comment --comment "$COMMENT" # 2. Add FORWARD rule (Allow packet passage) iptables -A FORWARD -p $PROTO -d $T_IP --dport $T_PORT -m comment --comment "$COMMENT" -j ACCEPT # Output result printf "%-10s %-10s %-10s %-20s %-10s\n" "$RULE_ID" "$PROTO" "$L_PORT" "$T_IP:$T_PORT" "Success" # Remind user to save echo "Please run '$0 save' to ensure rules persist after reboot." } # Main add function add_rule() { local L_PORT=$1 local T_IP=$2 local T_PORT=$3 local PROTO_REQ=${4:-both} # Default to both if | -z "$T_IP" || -z "$T_PORT" ; then echo "Error: Missing parameters" usage exit 1 fi # Convert to lowercase PROTO_REQ=$(echo "$PROTO_REQ" | tr '[:upper:]' '[:lower:]') echo "Adding rule..." printf "%-10s %-10s %-10s %-20s %-10s\n" "ID" "Protocol" "Public Port" "Target Address" "Status" echo "------------------------------------------------------------------" if "$PROTO_REQ" == "tcp" ; then _add_single_rule "tcp" "$L_PORT" "$T_IP" "$T_PORT" elif "$PROTO_REQ" == "udp" ; then _add_single_rule "udp" "$L_PORT" "$T_IP" "$T_PORT" elif "$PROTO_REQ" == "both" ; then _add_single_rule "tcp" "$L_PORT" "$T_IP" "$T_PORT" _add_single_rule "udp" "$L_PORT" "$T_IP" "$T_PORT" else echo "Error: Unsupported protocol '$PROTO_REQ'. Please use tcp, udp, or both." exit 1 fi echo "------------------------------------------------------------------" } # Delete rule (Delete in reverse line number order) del_rule() { local RULE_ID=$1 if -z "$RULE_ID" ; then echo "Error: Please provide rule ID" usage exit 1 fi echo "Searching for rule with ID [${RULE_ID}]..." local FOUND=0 # --- Clean NAT table (PREROUTING) --- LINES=$(iptables -t nat -nL PREROUTING --line-numbers | grep "NAT_ID:${RULE_ID}" | awk '{print $1}' | sort -rn) if ! -z "$LINES" ; then for line in $LINES; do iptables -t nat -D PREROUTING $line echo "Deleted NAT table PREROUTING chain line $line" FOUND=1 done fi # --- Clean Filter table (FORWARD) --- LINES=$(iptables -t filter -nL FORWARD --line-numbers | grep "NAT_ID:${RULE_ID}" | awk '{print $1}' | sort -rn) if ! -z "$LINES" ; then for line in $LINES; do iptables -t filter -D FORWARD $line echo "Deleted Filter table FORWARD chain line $line" FOUND=1 done fi if $FOUND -eq 0 ; then echo "No rule found with ID $RULE_ID." else echo "Delete operation completed." echo "Please run '$0 save' to update the persistent configuration file." fi } # Save rules to disk (New feature) save_rules() { echo "Saving current iptables rules..." # netfilter-persistent is the service managing iptables-persistent in Debian/Proxmox if command -v netfilter-persistent &> /dev/null; then netfilter-persistent save if [ $? -eq 0 ]; then echo "✅ Rules successfully saved to /etc/iptables/rules.v4, will be automatically restored after system reboot." else echo "❌ Failed to save rules. Please check the status of the 'netfilter-persistent' service." fi else echo "Warning: 'netfilter-persistent' command not found." echo "Please ensure the 'iptables-persistent' package is installed." echo "Install command: apt update && apt install iptables-persistent" fi } # List rules list_rules() { echo "Current Port Forwarding Rules List:" printf "%-10s %-10s %-10s %-20s %-10s\n" "ID" "Protocol" "Public Port" "Target Address" "Target Port" echo "------------------------------------------------------------------" # Parse iptables output iptables -t nat -nL PREROUTING -v | grep "NAT_ID:" | while read line; do id=$(echo "$line" | grep -oP '(?<=NAT_ID:)[^ ]*') # Extract protocol if echo "$line" | grep -q "tcp"; then proto="tcp"; else proto="udp"; fi # Extract port after dpt: l_port=$(echo "$line" | grep -oP '(?<=dpt:)[0-9]+') # Extract IP:Port after to: target=$(echo "$line" | grep -oP '(?<=to:).*') t_ip=${target%:*} t_port=${target#*:} printf "%-10s %-10s %-10s %-20s %-10s\n" "$id" "$proto" "$l_port" "$t_ip" "$t_port" done } # Main logic case "$ACTION" in add) add_rule "$ARG1" "$ARG2" "$ARG3" "$ARG4" ;; del) del_rule "$ARG1" ;; list) list_rules exit 0 ;; save) save_rules ;; *) usage exit 1 ;; esac save_rules This script automatically adds/deletes iptables rules for port forwarding. Remember to chmod +x. Use iptables-persistent to save the configuration and load it automatically on boot: apt install iptables-persistent During configuration, you will be asked whether to save the current rules; Yes or No is fine. When adding a forwarding rule, use natmgr add <host listen port> <VM internal IP> <VM port> [tcp/udp/both]. The script will automatically assign a unique ID. Use natmgr del <ID> to delete. Use natmgr list to view the existing forwarding list. Reference Articles: bin456789/reinstall: 一键DD/重装脚本 (One-click reinstall OS on VPS) - GitHub Install Proxmox VE on Debian 12 Bookworm - Proxmox VE PVE连接 TrueNAS iSCSI存储实现本地无盘化_pve iscsi-CSDN博客 ProxmoxVE (PVE) NAT 网络配置方法 - Oskyla 烹茶室
DN42&OneManISP - Troubleshooting OSPF Source Address in a Coexistence Environment
Backstory As mentioned in the previous post of this series, because the VRF solution was too isolating, the DNS service I deployed on the HKG node (172.20.234.225) became inaccessible from the DN42 network. Research indicated this could be achieved by setting up veth or NAT forwarding, but due to the scarcity of available documentation, I ultimately abandoned the VRF approach. Structure Analysis This time, I planned to place both DN42 and clearnet BGP routes into the system's main routing table, then separate them for export using filters to distinguish which should be exported. For clarity, I stored the configuration for the DN42 part and the clearnet part (hereinafter referred to as inet) separately, and then included them from the main configuration file. Also, since there should ideally only be one kernel configuration per routing table, I merged the DN42 and inet kernel parts, keeping only one instance. After multiple optimizations and revisions, my final directory structure is as follows: /etc/bird/ ├─envvars ├─bird.conf: Main Bird config file, defines basic info (ASN, IP, etc.), includes sub-configs below ├─kernel.conf: Kernel config, imports routes into the system routing table ├─dn42 | ├─defs.conf: DN42 function definitions, e.g., is_self_dn42_net() | ├─ibgp.conf: DN42 iBGP template | ├─rpki.conf: DN42 RPKI route validation | ├─ospf.conf: DN42 OSPF internal network | ├─static.conf: DN42 static routes | ├─ebgp.conf: DN42 Peer template | ├─ibgp | | └<ibgp configs>: DN42 iBGP configs for each node | ├─ospf | | └backbone.conf: OSPF area | ├─peers | | └<ibgp configs>: DN42 Peer configs for each node ├─inet | ├─peer.conf: Clearnet Peer | ├─ixp.conf: Clearnet IXP connection | ├─defs.conf: Clearnet function definitions, e.g., is_self_inet_v6() | ├─upstream.conf: Clearnet upstream | └static.conf: Clearnet static routes I separated the function definitions because I needed to reference them in the filters within kernel.conf, so I isolated them for early inclusion. After filling in the respective configurations and setting up the include relationships, I ran birdc configure and it started successfully. So, case closed... right? Problems occurred After running for a while, I suddenly found that I couldn't ping the HKG node from my internal devices, nor could I ping my other internal nodes from the HKG node. Strangely, external ASes could ping my other nodes or other external ASes through my HKG node, and my internal nodes could also ping other non-directly connected nodes (e.g., 226(NKG)->225(HKG)->229(LAX)) via the HKG node. Using ip route get <other internal node address> revealed: root@iYoRoyNetworkHKG:/etc/bird# ip route get 172.20.234.226 172.20.234.226 via 172.20.234.226 dev dn42_nkg src 23.149.120.51 uid 0 cache See the problem? The src address should have been the HKG node's own DN42 address (configured on the OSPF stub interface), but here it showed the HKG node's clearnet address instead. Attempting to read the route learned by Bird using birdc s r for 172.20.234.226: root@iYoRoyNetworkHKGBGP:/etc/bird/dn42/ospf# birdc s r for 172.20.234.226 BIRD 2.17.1 ready. Table master4: 172.20.234.226/32 unicast [dn42_ospf_iyoroynet_v4 00:30:29.307] * I (150/50) [172.20.234.226] via 172.20.234.226 on dn42_nkg onlink Looks seemingly normal...? Theoretically, although the DN42 source IP is different from the usual, DN42 rewrites krt_prefsrc when exporting to the kernel to inform the kernel of the correct source address, so this issue shouldn't occur: protocol kernel kernel_v4{ ipv4 { import none; export filter { if source = RTS_STATIC then reject; + if is_valid_dn42_network() then krt_prefsrc = DN42_OWNIP; accept; }; }; } protocol kernel kernel_v6 { ipv6 { import none; export filter { if source = RTS_STATIC then reject; + if is_valid_dn42_network_v6() then krt_prefsrc = DN42_OWNIPv6; accept; }; }; } Regarding krt_prefsrc, it stands for Kernel Route Preferred Source. This attribute doesn't manipulate the route directly but instead attaches a piece of metadata to it. This metadata directly instructs the Linux kernel to prioritize the specified IP address as the source address for packets sent via this route. I was stuck on this for a long time. The Solution Finally, during an unintentional attempt, I added the krt_prefsrc rewrite to the OSPF import configuration as well: protocol ospf v3 dn42_ospf_iyoroynet_v4 { router id DN42_OWNIP; ipv4 { - import where is_self_dn42_net() && source != RTS_BGP; + import filter { + if is_self_dn42_net() && source != RTS_BGP then { + krt_prefsrc=DN42_OWNIP; + accept; + } + reject; + }; export where is_self_dn42_net() && source != RTS_BGP; }; include "ospf/*"; }; protocol ospf v3 dn42_ospf_iyoroynet_v6 { router id DN42_OWNIP; ipv6 { - import where is_self_dn42_net_v6() && source != RTS_BGP; + import filter { + if is_self_dn42_net_v6() && source != RTS_BGP then { + krt_prefsrc=DN42_OWNIPv6; + accept; + } + reject; + }; export where is_self_dn42_net_v6() && source != RTS_BGP; }; include "ospf/*"; }; After running this, the src address became correct, and mutual pinging worked. Configuration files for reference: KaguraiYoRoy/Bird2-Configuration
DN42&OneManISP - Using VRF to Run Clearnet BGP and DN42 on the Same Machine
Background Currently, clearnet BGP and DN42 each use a separate VPS in the same region, meaning two machines are required per region. After learning about VRF from a group member, I explored using VRF to enable a single machine to handle both clearnet BGP and DN42 simultaneously. Note: Due to its isolation nature, the VRF solution will prevent DN42 from accessing services on the host. If you need to run services (like DNS) on the server for DN42, you might need additional port forwarding or veth configuration, which is beyond the scope of this article. (This is also the reason why I ultimately did not adopt VRF in my production environment). Advantages of VRF Although DN42 uses private IP ranges and internal ASNs, which theoretically shouldn't interfere with clearnet BGP, sharing the same routing table can lead to issues like route pollution and management complexity. VRF (Virtual Routing and Forwarding) allows creating multiple routing tables on a single machine. This means we can isolate DN42 routes into a separate routing table, keeping them apart from the clearnet routing table. The advantages include: Absolute Security and Policy Isolation: The DN42 routing table is isolated from the clearnet routing table, fundamentally preventing route leaks. Clear Operation and Management: Use commands like birdc show route table t_dn42 and birdc show route table t_inet to view and debug two completely independent routing tables, making things clear at a glance. Fault Domain Isolation: If a DN42 peer flaps, the impact is confined to the dn42 routing table. It won't consume routing computation resources for the clearnet instance nor affect clearnet forwarding performance. Alignment with Modern Network Design Principles: Using VRF for different routing domains (production, testing, customer, partner) is standard practice in modern network engineering. It logically divides your device into multiple virtual routers. Configuration System Part Creating the VRF Interface Use the following commands to create a VRF device named dn42-vrf and associate it with the system's routing table number 1042: ip link add dn42-vrf type vrf table 1042 ip link set dev dn42-vrf up # Enable it You can change the routing table number according to your preference, but avoid the following reserved routing table IDs: Name ID Description unspec 0 Unspecified, rarely used main 254 Main routing table, where most ordinary routes reside default 253 Generally unused, reserved local 255 Local routing table, contains 127.0.0.1/8, local IPs, broadcast addresses, etc. Cannot be modified Associating Existing Network Interfaces with VRF In my current DN42 setup, several WireGuard interfaces and a dummy interface are used for DN42. Therefore, associate these interfaces with the VRF: ip link set dev <interface_name> master dn42-vrf Note: After associating an interface with a VRF, it might lose its IP addresses. Therefore, you need to readd the addresses, for example: ip addr add 172.20.234.225 dev dn42 After completion, ip a should show the corresponding interface's master as dn42-vrf: 156: dn42: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master dn42-vrf state UNKNOWN group default qlen 1000 link/ether b6:f5:28:ed:23:04 brd ff:ff:ff:ff:ff:ff inet 172.20.234.225/32 scope global dn42 valid_lft forever preferred_lft forever inet6 fd18:3e15:61d0::1/128 scope global valid_lft forever preferred_lft forever inet6 fe80::b4f5:28ff:feed:2304/64 scope link valid_lft forever preferred_lft forever Persistence I use ifupdown2 to automatically load the dummy interface and VRF device on boot. auto dn42-vrf iface dn42-vrf inet manual vrf-table 1042 auto dn42 iface dn42 inet static pre-up ip link add $IFACE type dummy || true vrf dn42-vrf address <IPv4 Address>/32 address <IPv6 Address>/128 post-down ip link del $IFACE My dummy interface is named dn42; modify accordingly if yours is different. After creation, use ifup dn42-vrf && ifup dn42 to start the dummy interface. Note: The number prefix for the VRF device file should be smaller than that of the dummy interface file, ensuring the VRF device starts first. WireGuard Tunnels Add PostUp commands to associate them with the VRF and readd their addresses. Example: [Interface] PrivateKey = [Data Redacted] ListenPort = [Data Redacted] Table = off Address = fe80::2024/64 + PostUp = ip link set dev %i master dn42-vrf + PostUp = ip addr add fe80::2024/64 dev %i PostUp = sysctl -w net.ipv6.conf.%i.autoconf=0 [Peer] PublicKey = [Data Redacted] Endpoint = [Data Redacted] AllowedIPs = 10.0.0.0/8, 172.20.0.0/14, 172.31.0.0/16, fd00::/8, fe00::/8 Then restart the tunnel. Bird2 Part First, define two routing tables for DN42's IPv4 and IPv6: ipv4 table dn42_table_v4; ipv6 table dn42_table_v6 Then, specify the VRF and system routing table number in the kernel protocol, and specify the previously created v4/v6 routing tables in the IPv4/IPv6 sections: protocol kernel dn42_kernel_v6{ + vrf "dn42-vrf"; + kernel table 1042; scan time 20; ipv6 { + table dn42_table_v6; import none; export filter { if source = RTS_STATIC then reject; krt_prefsrc = DN42_OWNIPv6; accept; }; }; }; protocol kernel dn42_kernel_v4{ + vrf "dn42-vrf"; + kernel table 1042; scan time 20; ipv4 { + table dn42_table_v4; import none; export filter { if source = RTS_STATIC then reject; krt_prefsrc = DN42_OWNIP; accept; }; }; } For protocols other than kernel, add the VRF and the independent IPv4/IPv6 tables, but do not specify the system routing table number: protocol static dn42_static_v4{ + vrf "dn42-vrf"; route DN42_OWNNET reject; ipv4 { + table dn42_table_v4; import all; export none; }; } protocol static dn42_static_v6{ + vrf "dn42-vrf"; route DN42_OWNNETv6 reject; ipv6 { + table dn42_table_v6; import all; export none; }; } In summary: Configure a VRF and the previously defined routing tables for everything related to DN42. Only the kernel protocol needs the system routing table number specified; others do not. Apply the same method to BGP, OSPF, etc. However, I chose to use separate Router IDs for the clearnet and DN42, so a separate Router ID needs to be configured: # /etc/bird/dn42/ospf.conf protocol ospf v3 dn42_ospf_iyoroynet_v4 { + vrf "dn42-vrf"; + router id DN42_OWNIP; ipv4 { + table dn42_table_v4; import where is_self_dn42_net() && source != RTS_BGP; export where is_self_dn42_net() && source != RTS_BGP; }; include "ospf/*"; }; protocol ospf v3 dn42_ospf_iyoroynet_v6 { + vrf "dn42-vrf"; + router id DN42_OWNIP; ipv6 { + table dn42_table_v6; import where is_self_dn42_net_v6() && source != RTS_BGP; export where is_self_dn42_net_v6() && source != RTS_BGP; }; include "ospf/*"; }; # /etc/bird/dn42/ebgp.conf ... template bgp dnpeers { + vrf "dn42-vrf"; + router id DN42_OWNIP; local as DN42_OWNAS; path metric 1; ipv4 { + table dn42_table_v4; ... }; ipv6 { + table dn42_table_v6; ... }; } include "peers/*"; After completion, reload the configuration with birdc c. Now, we can view the DN42 routing table separately using ip route show vrf dn42-vrf: root@iYoRoyNetworkHKGBGP:~# ip route show vrf dn42-vrf 10.26.0.0/16 via inet6 fe80::ade0 dev dn42_4242423914 proto bird src 172.20.234.225 metric 32 10.29.0.0/16 via inet6 fe80::ade0 dev dn42_4242423914 proto bird src 172.20.234.225 metric 32 10.37.0.0/16 via inet6 fe80::ade0 dev dn42_4242423914 proto bird src 172.20.234.225 metric 32 ... You can also ping through the VRF using the -I dn42-vrf parameter: root@iYoRoyNetworkHKGBGP:~# ping 172.20.0.53 -I dn42-vrf ping: Warning: source address might be selected on device other than: dn42-vrf PING 172.20.0.53 (172.20.0.53) from 172.20.234.225 dn42-vrf: 56(84) bytes of data. 64 bytes from 172.20.0.53: icmp_seq=1 ttl=64 time=3.18 ms 64 bytes from 172.20.0.53: icmp_seq=2 ttl=64 time=3.57 ms 64 bytes from 172.20.0.53: icmp_seq=3 ttl=64 time=3.74 ms 64 bytes from 172.20.0.53: icmp_seq=4 ttl=64 time=2.86 ms ^C --- 172.20.0.53 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3006ms rtt min/avg/max/mdev = 2.863/3.337/3.740/0.341 ms Important Notes If the VRF device is reloaded, all devices originally associated with the VRF need to be reloaded as well, otherwise they won't function correctly. Currently, DN42 cannot access services inside the host configured with VRF. A future article might explain how to allow traffic within the VRF to access host services (Adding to the TODO list). I learned from a friend that by setting net.ipv4.tcp_l3mdev_accept=1 and net.ipv4.udp_l3mdev_accept=1, it is possible to allow the listening sockets in the global space to accept connection requests from the VRF domain, thus achieving cross-vrf listening services. Reference Articles:： Run your MPLS network with BIRD
Configuring Multi-Exit Routing for a Dual-NIC VPS Using PBR
Warning: This is a low-value/watered-down post Background I got a Shenzhen-Hong Kong IEPL machine from a friend. It has two network interfaces, eth0 and eth1, but by default, all traffic goes through eth0. eth1 had no routing configured. I planned to use metrics for basic traffic splitting initially, and then use PBR (Policy-Based Routing) to implement rule-based routing configuration. Configuration The machine's network was initially configured by cloud-init: # This file is generated from information provided by the datasource. Changes # to it will not persist across an instance reboot. To disable cloud-init's # network configuration capabilities, write a file # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: version: 2 ethernets: eth0: addresses: - 10.10.1.31/16 gateway4: 10.10.0.1 match: macaddress: bc:24:11:f8:42:7a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth0 eth1: addresses: - 10.20.1.31/16 - [REDACTED]/64 gateway4: 10.20.0.1 gateway6: fe80::be24:11ff:fe80:66bb match: macaddress: bc:24:11:50:96:0a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth1 After backing up the config, I added metrics: # This file is generated from information provided by the datasource. Changes # to it will not persist across an instance reboot. To disable cloud-init's # network configuration capabilities, write a file # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: version: 2 ethernets: eth0: addresses: - 10.10.1.31/16 gateway4: 10.10.0.1 match: macaddress: bc:24:11:f8:42:7a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth0 + routes: + - to: "default" + via: "10.10.0.1" + # Set metric=50 as the backup exit + metric: 50 eth1: addresses: - 10.20.1.31/16 - [REDACTED]/64 gateway4: 10.20.0.1 gateway6: fe80::be24:11ff:fe80:66bb match: macaddress: bc:24:11:50:96:0a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth1 + routes: + - to: "default" + via: "10.20.0.1" + # Set metric=25 as the preferred exit + metric: 25 Create the PBR configuration: # /etc/netplan/90-pbr.yaml network: version: 2 ethernets: eth0: routes: - to: default via: 10.10.0.1 table: 10 routing-policy: - from: 10.10.0.0/16 table: 10 - to: 202.46.[REDACTED]/32 table: 10 eth1: routes: - to: default via: 10.20.0.1 table: 20 routing-policy: - from: 10.20.0.0/16 table: 20 - to: 38.47.[REDACTED]/32 table: 20 - to: 23.149.[REDACTED]/32 table: 20 For IPs that need to be accessed via a specific exit, just add a 'to' type rule and bind it to the corresponding routing table. After finishing, run netplan apply to update the configuration.
OneManISP - Ep.2 Announcing Our Own IP Prefix to the World
Preface In the previous article, we successfully registered an ASN and obtained an IPv6 address block. Now, we will announce this block to the world. Setting Up the Subnet Object in the RIPE Database It's important to note that the minimum IPv6 prefix allowed for announcement on the public internet is /48. This means if you only have a single /48 block, you cannot break it down into smaller segments. Therefore, I later leased a separate /40 block, intending to split it into multiple /48s for announcement. The IPv6 block I obtained is 2a14:7583:f200::/40, and I plan to split out 2a14:7583:f203::/48 for use with Vultr. If you don't need to split your block, please skip directly to the "Creating the Route Object" section. Splitting the Prefix First, go to Create "inet6num" object - RIPE Database and fill in the following: inet6num: The IP block you want to split out, in CIDR format. netname: Network name. country: The country to which the IP block belongs, must conform to the ISO 3166 standard (can be selected directly in the RIPE DB). admin-c: The primary key value of the Role object created earlier. tech-c: The primary key value of the Role object created earlier. status: Keep ASSIGNED This step splits a smaller /48 address block from your obtained allocation. Creating the Route Object Go to Create "route6" object - RIPE Database and fill in the following: route6: The IPv6 address block you intend to announce, in CIDR format. origin: The ASN you applied for, including the 'AS' prefix. This step declares that your ASN is permitted to use this address block for originating BGP routes. Applying for BGP Session with a VPS Provider This time I'm using a machine from Vultr. Their BGP Session setup is very beginner-friendly, with their own validation system. Furthermore, their upstream has good filters ensuring that incorrect route advertisements generally won't affect the public internet. (I forgot to take screenshots during my configuration, but you can refer to the section 申请 Vultr 的 BGP 广播功能 in Bao Shuo's article 年轻人的第一个 ASN for reference.) Go to BGP - Vultr.com, select Get Started, and fill in your ASN and IPv6 block information as required. For the LOA (Letter Of Authorization), you can refer to this template: LOA-template.docx (I rewrote one for individuals as most templates found online are for companies). After submission, the system will automatically create a ticket, and you will see your ASN and IP block in a pending verification state: Click Start, and the system will send a verification email to the abuse-mailbox email address registered with your Role object: The received email looks like this: The top link represents approving the authorization for Vultr to announce your IP block, and the bottom one is for disapproval. Click the top link, which will take you to Vultr's webpage: Then click Approve Announcement. Both the ASN and the IP block need to be verified once. Next, wait for the Vultr staff to review and complete the process. Then, in your VPS control panel, you will see the BGP tab, where you can find the upstream information: I must commend Vultr's ticket efficiency here; it took me an average of only about 10 minutes from creating the ticket requesting authorization to completion. (In contrast, the average weekday ticket response time at iFog GmbH was around 1 day, which is much slower in comparison). The process with other VPS providers is generally similar. You need to inform their staff of the ASN and IP block you want to announce. After verifying ownership, the staff will configure the corresponding BGP Session for you. Advertisement! You should have received the following information from your upstream: Upstream's ASN Upstream's IP address for the BGP Session (Optional) Password The operating system I use is Debian 12 Bookworm, using Bird2 as the routing software. I updated Bird2 to the latest version following the section "Update Bird2 to v2.16 or above" in this article. The upstream ASN Vultr gave me is 64515, the upstream BGP Session address is 2001:19f0:ffff::1, and the VPS's BGP Session address is 2001:19f0:0006:0ff5:5400:05ff:fe96:881f. My Bird2 configuration file is modified from the configuration file used in DN42: log syslog all; define OWNAS = 205369; define OWNIPv6 = 2a14:7583:f203::1; define OWNNETv6 = 2a14:7583:f203::/48; define OWNNETSETv6 = [ 2a14:7583:f203::/48+ ]; router id 45.77.x.x; protocol device { scan time 10; } function is_self_net_v6() { return net ~ OWNNETSETv6; } protocol kernel { scan time 20; ipv6 { import none; export filter { if source = RTS_STATIC then reject; krt_prefsrc = OWNIPv6; accept; }; }; }; protocol static { route OWNNETv6 reject; ipv6 { import all; export none; }; } template bgp upstream { local as OWNAS; path metric 1; multihop; ipv6 { import filter { if net ~ [::/0] then reject; accept; }; export filter { if is_self_net_v6() then accept; reject; }; import limit 1000 action block; }; graceful restart; } protocol bgp 'Vultr_v6' from upstream{ local 2001:19f0:0006:0ff5:5400:05ff:fe96:881f as OWNAS; password "123456"; neighbor 2001:19f0:ffff::1 as 64515; } A few noteworthy points: The import rule in the upstream template here rejects the default route. This prevents the routing table sent by the upstream from overwriting local default gateway routes and other routing information. If we have multiple BGP neighbors, this could cause detours or even routing loops. The upstream template specifies multihop (multihop;) because Vultr's BGP peer is not directly reachable. Without setting multihop, the BGP session would get stuck in the Idle state. If your BGP upstream is directly connected, you can omit this line or set it to direct;. After filling in the configuration file, run birdc configure to load the configuration. Run birdc show protocols to check the status. If all goes well, you should see the BGP session state as Established: At this point, you can take a break and wait for global routing convergence. After about half an hour, open bgp.tools and query your /48 block. You should see that it has been successfully received by the global internet, and you can see our upstream information: Next, we create a dummy interface on the VPS and assign a single IPv6 address from the block allocated for this machine. For example, I assigned 2a14:7583:f203::1 to my machine: ip link add dummy0 type dummy ip addr add 2a14:7583:f203::1/128 dev dummy0 Then, using your own PC, you should be able to ping this address, and traceroute will show the complete routing path: Thanks to Mi Lu for the technical support! Reference Articles: 自己在家开运营商 Part.2 - 向世界宣告 IP 段 (BGP Session & BIRD) 年轻人的第一个 ASN - 宝硕博客 BGPlayer 从零开始速成指北 - 开通 Vultr 的 BGP 广播功能 - AceSheep BGP (2) 在 Vultr 和 HE 使用自己的 IPV6 地址 - 131's Blog
Introduction This article documents my complete process of applying for an ASN through the RIPE NCC. The content is suitable for beginners. If you find any errors, please feel free to contact me via email, and I will correct them promptly. Now that we've learned the basic BGP concepts on DN42, it seems a bit of a waste not to play with the public internet, right? Basic Concepts Currently, the allocation of public ASN and IP resources is managed by five Regional Internet Registries (RIRs) worldwide: ARIN: Manages the North American region. RIPE NCC: Manages the European region. APNIC: Manages the Asia-Pacific region. LACNIC: Manages the Latin American region. AfriNIC: Manages the African region. RIRs do not provide services directly to end users. Instead, they allocate resources to Local Internet Registries (LIRs), which then assign them to end users. Of course, individual users can also register as an LIR, but this is generally not cost-effective. If you're willing to pay thousands of dollars in annual fees, then forget I said that. Among these, RIPE NCC is considered more friendly towards individual applications, followed by ARIN and APNIC. Compared to RIPE NCC, APNIC's fees are generally about 30% higher. Furthermore, RIPE NCC provides an online management system allowing users to modify information and check progress themselves, whereas with APNIC, you typically need to contact an LIR for changes. Overall, I chose to apply for an ASN through the RIPE NCC. The resources obtained (both ASN and IPs) are generally categorized into two types: PA (Provider Aggregatable) Resources: Belong to the LIR and are assigned for your use by the LIR. PI (Provider Independent) Resources: Belong to you directly. These are generally more expensive. Preparation Stage Choosing an LIR Search online for LIR Service to find many companies offering such services. Currently, RIPE NCC charges an annual administrative fee of 50 EUR for PI resources. This means the cost from an LIR for registering an ASN generally won't be lower than 50 EUR per year (approximately 60 USD at the time of writing). Here, I chose NoPKT LLC, recommended by peers. Their pricing is quite reasonable and includes a /48 block of PA IPv6 addresses with the ASN. The activation speed was also very fast – it only took half a day from submitting the required documents to getting the ASN. Preparing Documents Proof of Identity Individual: Provide an ID card or passport (I submitted photos of the front and back of my national ID card). Company: Provide a valid business license. If the applicant is a minor, usually written consent from their legal guardian is required, and the guardian must fulfill corresponding responsibilities. All submitted documents must be authentic and valid, and should be originals or notarized copies. Contact Information Postal Address: Used for registration in the RIPE Database. Technical Contact Email. Abuse Contact Email. Technical Justification Billing from a BGP-capable provider within the European region. Options include Vultr, BuyVM, iFog, V.PS, etc. Note: Vultr uses a post-payment system, generating invoices at the beginning of the month. If you need the documents ready quickly, consider other providers. ASNs of two upstream providers you plan to connect to. (In practice, the reviewers won't strictly verify the specific upstream ASNs you list. Therefore, you can fill in common, publicly known ASNs to make it look reasonable. Don't overthink it too much. You can even put mine.) Registering a RIPE DB Account and Creating Objects Go to the RIPE Database and register an account. For Chinese, it's recommended to use the 拼音(Pinyin) of your real legal name. Enabling 2FA is mandatory, so please install a TOTP app on your phone beforehand. Creating a Role Object and Maintainer Object Go to Create role and maintainer pair - RIPE Database to create a role object. Here, a 'role' is an abstract concept describing the contact information for a team, department, or functional role – it represents a role, such as NOC (Network Operations Center), Abuse Team, Hostmaster, etc. mntner: The identifier for the maintainer object. It can contain uppercase/lowercase letters, numbers, and -_. For example, I used IYOROY-MNT. role: The name for the role object. It can contain uppercase/lowercase letters, numbers, and ][)(._"*@,&:!'+/-. For example, I used IYOROY-NETWORK-NOC. address: The office address for this role. e-mail: The email address for this role. Click SUBMIT after filling out the form to create both the role object and the maintainer object. Please note the returned primary key name, which usually ends with -RIPE. You will need this for future modifications and submissions to the LIR. The maintainer object identifier here is conceptually different from the role object. The maintainer signifies who has the authority to maintain (create/modify/delete) objects in the database – it's the maintaining entity. The relationship between different concepts in the RIPE Database can be referenced in the diagram later in the article. Adding an Abuse Contact Mailbox Go to Query - RIPE Database and search for the primary key of the role you just created. You should find the entry you created. Click "Update Object" on the right. Click the plus sign (+) next to the email field to add an abuse-mailbox attribute and fill in your abuse contact email address: Click SUBMIT to save. Note: RIPE periodically checks if the abuse-mailbox is functional. Please ensure you provide a real, active email address. Creating an Organization Object The Organization object here is an abstraction of a legal entity or organization (company, university, ISP, individual user, etc.). It serves as the top-level ownership information for resource objects (like aut-num, inetnum, inet6num) in the RIPE Database. This means subsequent ASN and IP resources will be assigned to this Organization object. Go to Create Organization - RIPE Database and fill in the following information: organisation: A unique ID. Keep it as the default AUTO-1 to let RIPE NCC assign one. org-name: The name of the organization. For Chinese individuals, use your full name in Pinyin. address: Postal address. country: Country code, refer to ISO 3166. For China, use CN. e-mail: The organization's email address. admin-c / tech-c: Administrative and technical contact objects (referencing the role handle). abuse-c: Specifies the abuse contact (must be a role object linked to the abuse-mailbox in that role). mnt-ref: Specifies which maintainer(s) can create objects referencing this organisation. mnt-by: Specifies who can maintain this organisation object itself. Click SUBMIT after filling out the form and note the returned object identifier, which follows a format like ORG-XXXX-RIPE. If you need to make changes after submission, go to Query - RIPE Database and search for the previously noted Role primary key or the Organization object identifier to find the update option. Paying the LIR Fee and Submitting Documents Submit the following documents to your chosen LIR: Proof of Identity Full Name Address (recommended to match your ID document) Photos of the front and back of your ID card RIPE Database Information org: Organization object identifier as-name: AS Name admin-c: Primary key of the role object created earlier tech-c: Primary key of the role object created earlier abuse-c: Primary key of the role object created earlier nic-hdl: Primary key of the role object created earlier mnt-by: Name of the maintainer object created earlier Technical Justification VPS Bill/Invoice Upstream ASNs The LIR will likely ask you to add a mnt-ref attribute to your Organization object, pointing to the LIR's maintainer. This allows the LIR to assign the AS and IP resources to your Organization. Once the LIR reviews and approves your application, they will submit the request to RIPE. Then, it's a waiting game. Generally, it takes 3-5 working days to get your ASN. At this point, we have successfully registered our own ASN on the public internet. Supplement: Relationships Between Concepts in the RIPE Database graph LR %% ========== ORG Layer ========== subgraph Org["Organisation"] ORG["organisation\n(ORG-XXX-RIPE)"] end %% ========== Resource Layer ========== subgraph Resource["Resources"] INETNUM["inetnum\n(IPv4 Block)"] INET6NUM["inet6num\n(IPv6 Block)"] AUTNUM["aut-num\n(ASN)"] ASSET["as-set\n(ASN Set)"] end %% ========== Routing Layer ========== subgraph Routing["Routing"] ROUTE["route\n(IPv4 Route Announcement)"] ROUTE6["route6\n(IPv6 Route Announcement)"] end %% ========== Contact Layer ========== subgraph Contact["Contacts"] ROLE["role\n(Team/Function)\nnic-hdl"] PERSON["person\n(Individual)\nnic-hdl"] end %% ========== Authorization Layer ========== subgraph Maintainer["Authorization"] MNT["mntner\n(Maintainer)"] end %% ========== Contact Links ========== INETNUM --> ROLE INET6NUM --> ROLE AUTNUM --> ROLE ASSET --> ROLE ROUTE --> ROLE ROUTE6 --> ROLE ROLE --> PERSON %% ========== Organization Assignment ========== ORG --> INETNUM ORG --> INET6NUM ORG --> AUTNUM ORG --> ASSET %% ========== Authorization ========== ORG --> MNT INETNUM --> MNT INET6NUM --> MNT AUTNUM --> MNT ASSET --> MNT ROUTE --> MNT ROUTE6 --> MNT ROLE --> MNT PERSON --> MNT %% ========== Route Binding ========== ROUTE -->|origin| AUTNUM ROUTE6 -->|origin| AUTNUM %% ========== Route Scope ========== ROUTE -->|belongs to| INETNUM ROUTE6 -->|belongs to| INET6NUM Special thanks to Mi Lu for providing technical support and answering questions! Reference Articles: 自己在家开运营商 Part.1 - 注册一个 ASN - LYC8503 从0开始注册一个ASN并广播IP | Pysio's Home 青年人的第一个运营商：注册一个 ASN | liuzhen932 的小窝
Foreword I am a novice in BGP. This article may contain imprecise content/naive understandings/elementary mistakes. I kindly ask the experts to be lenient. If you find any issues, you are welcome to contact me via email, and I will correct them as soon as possible. If you find this unacceptable, it is recommended to close this article now. What are BGP Communities? TL;DR: BGP Communities "tag" routes, allowing others to use these tags for route optimization. This concept might be unfamiliar to newcomers (like me), who might not initially understand its purpose. Simply put, BGP Communities are a mechanism for tagging routes, similar to adding labels. They allow network administrators to attach one or more "tags" (i.e., community values) to routes propagated via BGP. These tags themselves do not alter the route's path attributes (like AS_PATH, LOCAL_PREF, MED, etc.), but they provide a signaling mechanism to indicate what policy or processing should be applied to that route by other routers within the same AS or in downstream peer ASes. BGP Communities can be used to: Simplify Policy Configuration: Routers inside the network or in downstream ASes only need to configure policies based on community values (like setting LOCAL_PREF, adding NO_EXPORT, applying route-maps, etc.), without needing to know the specific prefix details. This makes policies more centralized, easier to manage, and less prone to errors. Convey Policy Intent to Downstream ASes: An AS can attach community values to routes it advertises to its downstream customer or peer ASes. These communities convey requirements or suggestions on how these routes should be handled, such as route optimization based on geographic location, latency, or bandwidth. Coordinate Policies within an AS: Inside a large AS, when using IBGP full-mesh or route reflectors, edge routers (receiving EBGP routes or redistributing routes) can tag routes with community values. Core routers or route reflectors within the AS can recognize these communities and apply corresponding internal policies (like setting LOCAL_PREF, MED, deciding whether to advertise to certain IBGP peers, adding other communities, etc.), without needing complex prefix-based policies on every internal router. DN42 has its own set of Communities specifications. For details, please refer to: BGP-communities - DN42 Wiki Configuration The general idea and approach in this article are largely based on Xe_iu's method, focusing on adding BGP Communities for geographic information and performing route optimization. Generally, this is sufficient. (Another reason is that I haven't fully grasped the others yet) Note: We should ONLY add geographic information-related BGP Communities to routes originating from our own AS. We should not add such entries to routes received from neighbors. Adding our own regional Communities to a neighbor's routes constitutes forging the route origin, potentially leading to route hijacking. Downstream networks might misjudge the traffic path, routing traffic that should be direct through your network, increasing latency and consuming your network's bandwidth. (Large Communities are an exception, as they have a verification mechanism to prevent this, but that's beyond the scope of this article). The idea is clear. When exporting routes, we first need to verify if it's our own route. If it is, we add the communities tag to the route. The sample configuration provided by DN42 already includes two functions, is_self_net() and is_self_net_v6(), to check if a route is our own. Therefore, writing the configuration part is straightforward. Adding Communities to Routes First, we need to define the geographic region information for the current node at the beginning of the node configuration file. Please check BGP-communities - DN42 Wiki for details: define DN42_REGION = 52; # 52 represents East Asia define DN42_COUNTRY= 1344; # 1344 represents Hong Kong Please modify these values according to your node's actual geographic location. Then, modify the export filter in the dnpeers template: }; export filter { - if is_valid_network() && source ~ [RTS_STATIC, RTS_BGP] then accept; + if is_valid_network() && source ~ [RTS_STATIC, RTS_BGP] then{ + if (is_self_net()) then { # Check if it's our own route + bgp_community.add((64511, DN42_REGION)); # Add continent-level region info + bgp_community.add((64511, DN42_COUNTRY)); # Add country/region info + } + accept; + } reject; }; import limit 1000 action block; During export, it checks if it's our own route. If it is, it sets the bgp_community according to the defined DN42_REGION and DN42_COUNTRY. Here, 64511 is the reserved public AS number identifier specifically for geographic tags (Region/Country); you can just copy it. Apply the same method to the IPv6 export rules, but replace is_self_net() with is_self_net_v6(). Route Optimization Based on Communities Here we need to introduce another concept: local_pref (Local Preference). It is used within an AS to indicate the priority of a route. Its default value is 100, and a higher value indicates higher priority. Furthermore, in BGP route selection logic, local_pref has the highest priority, even higher than AS_PATH length. This means that by setting local_pref, we can adjust the priority of routes to achieve route optimization. Combining this with the BGP Communities mentioned above, we can set the corresponding local_pref based on Communities for optimization. Also, since BGP.local_pref is propagated within the AS, we need to modify the import logic for both eBGP and iBGP routes. My logic for handling BGP.local_pref here references (basically copies) Xe_iu's approach: For routes from the same region, priority +10 For routes from the same country, priority +5 additionally For routes received via direct peering with us, priority +20 additionally Create a function to calculate the priority: function ebgp_calculate_priority() { int priority = 100; # Base priority # Same region detection (+10) if bgp_community ~ [(64511, DN42_REGION)] then priority = priority + 10; # Same country detection (+5) if bgp_community ~ [(64511, DN42_COUNTRY)] then priority = priority + 5; # Direct eBGP neighbor detection (+20) if bgp_path.len = 1 then priority = priority + 20; return priority; } Then, in the import filter within the dnpeers template, set bgp_local_pref to the value calculated by the function: template bgp dnpeers { local as OWNAS; path metric 1; ipv4 { import filter { if is_valid_network() && !is_self_net() then { if (roa_check(dn42_roa, net, bgp_path.last) != ROA_VALID) then { print "[dn42] ROA check failed for ", net, " ASN ", bgp_path.last; reject; } + bgp_local_pref = ebgp_calculate_priority(); accept; } reject; }; Apply the same method for IPv6. After running birdc configure, we should be able to see that our routes have been tagged with Communities labels at our neighbors: (Screenshot source:https://lg.milu.moe/route_all/hk/172.20.234.224) Special thanks to Nuro Trace and Xe_iu. They helped deepen my understanding of BGP Communities and provided much assistance. Reference Articles: [DN42] bird2的配置文件 – Xe_iu's Blog | Xe_iu的杂物间 [DN42] 谈一谈如何配置 BGP community – Xe_iu's Blog | Xe_iu的杂物间 BGP-communities - DN42 Wiki