DN42 Technical Documentation

Scope

This page documents the technical configuration used to connect my lab to DN42. It is a configuration reference, not a project story. The design uses one Linux LXC as the DN42 edge, one Cisco C1101 as the internal router, and one Proxmox service LXC directly attached to a public DN42 VLAN.

Private WireGuard keys, passwords and SSH material are intentionally replaced by redacted placeholders. Public routing values, public endpoints, VLAN IDs and service addresses are kept because they are required to understand and reproduce the setup.

Address plan

DN42 registry

ASN: AS4242420621

Maintainer: XYDRIVE-MNT

IPv4 prefix: 172.22.149.64/27

IPv6 prefix: fd51:fbc4:f614::/48

Internal addressing

VLAN 10 gateway: 172.22.149.65/27, fd51:fbc4:f614:1::1/64

VLAN 99 interco: 10.255.255.0/30, fd51:fbc4:f614:ffff::/64

DN42 LXC: 10.255.255.1, fd51:fbc4:f614:ffff::1

Cisco: 10.255.255.2, fd51:fbc4:f614:ffff::2

Services

drive42: 172.22.149.66/27

drive42: fd51:fbc4:f614:1::42/64

HTTP: http://172.22.149.66/

Minecraft: 172.22.149.66:25565

Client access

Arch laptop example: 172.22.149.67/27

Arch laptop example: fd51:fbc4:f614:1::67/64

Access port: GigabitEthernet0/1/1 in VLAN 10

DNS used during tests: 172.22.149.66

Topology

Proxmox is a Layer 2 bridge host for this DN42 segment. The Linux DN42 gateway terminates WireGuard and runs FRR. The Cisco is the local router for the service/client VLAN and receives DN42 routes from the gateway over iBGP.

RoutedBits Paris AS4242420207             Kioubit FR1 AS4242423914
        |                                          |
        | WireGuard peer1 + eBGP                  | WireGuard kioubit-fr1 + eBGP
        |                                          |
        +------------------+-----------------------+
                           |
                    LXC Dn42 gateway
                    FRR + WireGuard
                    AS4242420621
                           |
                           | VLAN 99 / iBGP
                           | 10.255.255.1/30 <-> 10.255.255.2/30
                           |
                    Cisco C1101
                           |
                           | VLAN 10 / public DN42 LAN
                           |
             +-------------+------------------+
             |                                |
      Proxmox vmbr1 trunk               Cisco access ports
             |                                |
       drive42 service LXC              laptop
       172.22.149.66                    172.22.149.67

How DN42 peering works

DN42 is built from private point-to-point links between operators. To connect to someone, both sides first agree on peering details: ASN, WireGuard public key, public endpoint, tunnel port, link-local BGP address, allowed prefixes and routing policy. After that, each side creates a WireGuard peer and runs BGP over the tunnel.

1. Exchange peering information with another DN42 operator
2. Create a WireGuard tunnel between both public Internet endpoints
3. Assign tunnel addresses, often IPv6 link-local addresses like fe80::621
4. Start a BGP session over the WireGuard interface
5. Import the remote DN42 routes
6. Export only the prefixes that my AS is allowed to announce

WireGuard provides the encrypted transport. BGP exchanges the routes. eBGP is used between different autonomous systems, such as RoutedBits and Kioubit. iBGP is used inside my own AS between the LXC gateway and the Cisco.

VLAN and bridge design

The physical link between Proxmox and the Cisco is a trunk. A trunk is an Ethernet link that carries several VLANs at the same time using 802.1Q tags. Instead of using one cable for the service LAN and another for the BGP interconnect, the same Proxmox-to-Cisco cable transports VLAN 10 and VLAN 99.

VLAN 10 carries public DN42 hosts and clients. VLAN 99 carries the routed interconnect between the DN42 gateway LXC and the Cisco.

VLAN 10 - LAN-DN42
Purpose: public DN42 services and clients
Cisco SVI: Vlan10
Cisco IPv4: 172.22.149.65/27
Cisco IPv6: fd51:fbc4:f614:1::1/64
Proxmox: vmbr1 with LXC VLAN tag 10
drive42: 172.22.149.66/27, fd51:fbc4:f614:1::42/64
Laptop: access VLAN 10, for example 172.22.149.67/27

VLAN 99 - INTERCO-BGP
Purpose: point-to-point routing between Cisco and the DN42 gateway LXC
Cisco SVI: Vlan99
Cisco IPv4: 10.255.255.2/30
Cisco IPv6: fd51:fbc4:f614:ffff::2/64
LXC gateway IPv4: 10.255.255.1/30
LXC gateway IPv6: fd51:fbc4:f614:ffff::1/64

Proxmox bridge configuration

The Proxmox host keeps the Cisco-facing USB interface inside vmbr1. The host itself does not route the DN42 IPv4 service LAN. It only bridges tagged frames to LXC interfaces.

auto lo
iface lo inet loopback

iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.0.2/24
        gateway 10.10.0.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports usb-cisco
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

In the target design, vmbr1 is the Cisco-facing bridge. LXC interfaces are attached to this bridge with the required VLAN tag: tag=10 for public DN42 services and tag=99 for the BGP interconnect.

Cisco C1101 VLAN configuration

The important Cisco state is the trunk on GigabitEthernet0/1/0, access ports in VLAN 10 for clients, and two SVIs. Vlan10 is the public DN42 LAN. Vlan99 is the routed link to the gateway LXC.

vlan 10
 name LAN-DN42
!
vlan 99
 name INTERCO-BGP
!
interface GigabitEthernet0/1/0
 description PROXMOX-VMbr1-TRUNK
 switchport trunk allowed vlan 10,99
 switchport mode trunk
!
interface GigabitEthernet0/1/1
 description DN42-CLIENT-ACCESS
 switchport access vlan 10
 switchport mode access
!
interface GigabitEthernet0/1/2
 description DN42-CLIENT-ACCESS
 switchport access vlan 10
 switchport mode access
!
interface GigabitEthernet0/1/3
 description DN42-CLIENT-ACCESS
 switchport access vlan 10
 switchport mode access
!
interface Vlan10
 description DN42-PUBLIC-LAN
 ip address 172.22.149.65 255.255.255.224
 ipv6 address FD51:FBC4:F614:1::1/64
!
interface Vlan99
 description DN42-LXC-GATEWAY-INTERCO
 ip address 10.255.255.2 255.255.255.252
 ipv6 address FD51:FBC4:F614:FFFF::2/64

The final checks showed VLAN 10 active on Gi0/1/1, Gi0/1/2 and Gi0/1/3, with Gi0/1/0 trunking VLANs 10,99.

Cisco BGP and routing

The Cisco runs inside the same AS as the LXC gateway, so the session toward 10.255.255.1 is iBGP. The LXC gateway uses next-hop-self and update-source dn42-port so the Cisco receives routes with a reachable next-hop on the VLAN 99 interconnect.

ip routing
ipv6 unicast-routing
!
ipv6 route FD51:FBC4:F614::/48 Null0
!
router bgp 4242420621
 bgp router-id 10.255.255.2
 neighbor 10.255.255.1 remote-as 4242420621
 neighbor 10.255.255.1 description LXC-DN42-GATEWAY
 !
 address-family ipv4
  network 172.22.149.64 mask 255.255.255.224
  neighbor 10.255.255.1 activate
 exit-address-family
 !
 address-family ipv6
  network FD51:FBC4:F614::/48
  neighbor 10.255.255.1 activate
 exit-address-family

The static IPv6 route to Null0 exists so the Cisco can originate the full fd51:fbc4:f614::/48 summary while still routing the active fd51:fbc4:f614:1::/64 service subnet locally.

DN42 gateway LXC - system networking

The gateway LXC is the DN42 edge. It must forward traffic and must not drop valid routed packets through reverse path filtering. These values are persisted in /etc/sysctl.d/99-dn42.conf.

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0

The gateway also needs an interface toward VLAN 99 with 10.255.255.1/30. With iBGP toward the Cisco, next-hop-self is enough because iBGP would otherwise keep the original external next-hop.

WireGuard - RoutedBits Paris

peer1 is the RoutedBits Paris tunnel. BGP runs over IPv6 link-local addresses, but IPv4 DN42 routes also arrive over this session. The important fix is the point-to-point IPv4 peer statement for 172.20.19.79, because FRR requires the IPv4 next-hop to be considered connected.

[Interface]
PrivateKey = <wg-private-key-redacted>
ListenPort = 42420
Address = fe80::621/64
PostUp = ip addr replace 10.255.255.1 peer 172.20.19.79/32 dev %i
MTU = 1420
Table = off
PostUp = ip route replace 172.20.19.79/32 dev %i scope link
PostUp = iptables -P FORWARD ACCEPT
PostUp = sysctl -w net.ipv4.conf.%i.rp_filter=0 net.ipv6.conf.%i.forwarding=1

[Peer]
PublicKey = 2eANKa/mvPXwkAFyBcE9YJ6lLxmDTRsUP0C5oyIFMCQ=
Endpoint = router.par1.routedbits.com:50621
AllowedIPs = 0.0.0.0/0, ::/0

WireGuard - Kioubit FR1

kioubit-fr1 is the second external DN42 peer. It uses a separate local WireGuard port so it does not conflict with peer1. This peer improved route visibility and made several DN42 services reachable.

[Interface]
PrivateKey = <wg-private-key-redacted>
Address = fe80::621/64
Address = 192.168.217.35/32
ListenPort = 42421
MTU = 1420
Table = off
PostUp = sysctl -w net.ipv4.conf.%i.rp_filter=0 net.ipv6.conf.%i.forwarding=1

[Peer]
PublicKey = sLbzTRr2gfLFb24NPzDOpy8j09Y6zI+a7NkeVMdVSR8=
Endpoint = fr1.g-load.eu:20036
AllowedIPs = 0.0.0.0/0, ::/0
PersistentKeepalive = 25

FRRouting configuration

FRR terminates all BGP sessions. External neighbors receive only my own DN42 prefixes through prefix-lists and outbound route-maps. Toward the Cisco, iBGP uses standard next-hop-self with update-source dn42-port. External eBGP peers still use next-hop-self force because they are outside the AS.

frr version 10.3
frr defaults traditional
hostname Dn42
log syslog informational
no zebra nexthop kernel enable
service integrated-vtysh-config
!
ip prefix-list XYDRIVE-V4 seq 10 permit 172.22.149.64/27
ipv6 prefix-list XYDRIVE-V6 seq 10 permit fd51:fbc4:f614::/48
!
route-map ALLOW-ALL permit 10
exit
!
route-map FROM-KIOUBIT permit 10
 set local-preference 150
exit
!
route-map TO-KIOUBIT-V4 permit 10
 match ip address prefix-list XYDRIVE-V4
exit
route-map TO-KIOUBIT-V4 deny 100
exit
!
route-map TO-KIOUBIT-V6 permit 10
 match ipv6 address prefix-list XYDRIVE-V6
exit
route-map TO-KIOUBIT-V6 deny 100
exit
!
route-map TO-ROUTEDBITS-V4 permit 10
 match ip address prefix-list XYDRIVE-V4
exit
route-map TO-ROUTEDBITS-V4 deny 100
exit
!
route-map TO-ROUTEDBITS-V6 permit 10
 match ipv6 address prefix-list XYDRIVE-V6
exit
route-map TO-ROUTEDBITS-V6 deny 100
exit
!
ip route 172.22.149.64/27 10.255.255.2
ip route 172.20.19.79/32 peer1
ip route 172.20.53.102/32 kioubit-fr1
ipv6 route fd51:fbc4:f614::/48 fd51:fbc4:f614:ffff::2
router bgp 4242420621
 bgp router-id 10.255.255.1
 no bgp network import-check
 neighbor 10.255.255.2 remote-as 4242420621
 neighbor 10.255.255.2 description ROUTEUR-CISCO-C1101
 neighbor 10.255.255.2 update-source dn42-port
 neighbor fe80::207 remote-as 4242420207
 neighbor fe80::207 description ROUTEDBITS-PARIS
 neighbor fe80::207 interface peer1
 neighbor fe80::ade0 remote-as 4242423914
 neighbor fe80::ade0 description KIOUBIT-FR1
 neighbor fe80::ade0 interface kioubit-fr1
 neighbor fe80::ade0 capability extended-nexthop
 !
 address-family ipv4 unicast
  network 172.22.149.64/27
  neighbor 10.255.255.2 next-hop-self
  neighbor 10.255.255.2 soft-reconfiguration inbound
  neighbor fe80::207 next-hop-self force
  neighbor fe80::207 route-map ALLOW-ALL in
  neighbor fe80::207 route-map TO-ROUTEDBITS-V4 out
  neighbor fe80::ade0 next-hop-self force
  neighbor fe80::ade0 route-map FROM-KIOUBIT in
  neighbor fe80::ade0 route-map TO-KIOUBIT-V4 out
 exit-address-family
 !
 address-family ipv6 unicast
  network fd51:fbc4:f614::/48
  neighbor 10.255.255.2 activate
  neighbor fe80::207 activate
  neighbor fe80::207 next-hop-self force
  neighbor fe80::207 route-map ALLOW-ALL in
  neighbor fe80::207 route-map TO-ROUTEDBITS-V6 out
  neighbor fe80::ade0 activate
  neighbor fe80::ade0 route-map FROM-KIOUBIT in
  neighbor fe80::ade0 route-map TO-KIOUBIT-V6 out
 exit-address-family

drive42 LXC on Proxmox

drive42 is VMID 108. It is a Debian 12 unprivileged LXC with its only production interface bridged to vmbr1 and tagged into VLAN 10.

NET0="name=eth0,bridge=vmbr1,tag=10,ip=172.22.149.66/27"
NET0="${NET0},gw=172.22.149.65"
NET0="${NET0},ip6=fd51:fbc4:f614:1::42/64"
NET0="${NET0},gw6=fd51:fbc4:f614:1::1"

pct set 108 --net0 "$NET0"

Inside the container, the expected routes are simple. The Cisco is the default gateway for IPv4 and IPv6.

eth0: 172.22.149.66/27
eth0: fd51:fbc4:f614:1::42/64
default via 172.22.149.65 dev eth0 onlink
172.22.149.64/27 dev eth0 proto kernel scope link src 172.22.149.66
default via fd51:fbc4:f614:1::1 dev eth0
fd51:fbc4:f614:1::/64 dev eth0 proto kernel metric 256

drive42 services

The service LXC runs Nginx, a PaperMC Minecraft server, Unbound for DN42 DNS tests, and nftables. Minecraft was configured with online-mode=false because the DN42-only service path does not rely on Mojang public Internet authentication.

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name drive42;

    root /var/www/drive42;
    index index.html;
}
# /opt/minecraft/server.properties
server-port=25565
online-mode=false
motd=drive42 DN42 Minecraft
enable-query=false

drive42 DNS resolver

Unbound listens on the service IPs and forwards dn42. queries to DN42 recursive resolvers. In testing, the local service was correct, but some upstream DN42 DNS resolvers timed out depending on the currently selected BGP path.

server:
  interface: 172.22.149.66
  interface: fd51:fbc4:f614:1::42
  access-control: 172.22.149.64/27 allow
  access-control: fd51:fbc4:f614:1::/64 allow
  do-ip4: yes
  do-ip6: yes
  do-udp: yes
  do-tcp: yes
  verbosity: 1
  harden-dnssec-stripped: no
  domain-insecure: "dn42."

forward-zone:
  name: "dn42."
  forward-addr: 172.20.0.53
  forward-addr: 172.23.0.53
  forward-addr: fd42:d42:d42:54::1
  forward-addr: fd42:d42:d42:53::1

drive42 firewall

The firewall allows DNS for DN42 resolution tests, HTTP/HTTPS, Minecraft and ICMP. SSH is disabled on this service host and is not opened in the firewall. Forwarding is dropped because drive42 is not a router.

table inet filter {
  chain input {
    type filter hook input priority 0;
    policy drop;
    iif lo accept
    ct state established,related accept
    ip protocol icmp accept
    ip6 nexthdr ipv6-icmp accept
    tcp dport { 53, 80, 443, 25565 } accept
    udp dport { 53, 25565 } accept
  }

  chain forward {
    type filter hook forward priority 0;
    policy drop;
  }

  chain output {
    type filter hook output priority 0;
    policy accept;
  }
}

Routing behavior

The LXC gateway peers externally over eBGP and internally with the Cisco over iBGP. The external peers only receive 172.22.149.64/27 and fd51:fbc4:f614::/48. The Cisco receives DN42 routes from the gateway and forwards service traffic to VLAN 10.

Inbound traffic to 172.22.149.66:
DN42 remote host
  -> external AS path
  -> RoutedBits or Kioubit
  -> WireGuard tunnel
  -> LXC Dn42 gateway
  -> iBGP-learned internal path
  -> Cisco C1101
  -> VLAN 10
  -> drive42

Outbound traffic from drive42:
drive42
  -> default gateway 172.22.149.65 / fd51:fbc4:f614:1::1
  -> Cisco C1101
  -> route learned from gateway over iBGP
  -> LXC Dn42 gateway
  -> selected external WireGuard peer
  -> DN42

Validation commands

These commands were the useful checks for this design. They validate L2, addressing, BGP state, route installation, service listeners and reachability.

# Cisco
show vlan brief
show interfaces trunk
show interfaces GigabitEthernet0/1/0 switchport
show interfaces GigabitEthernet0/1/1 switchport
show ip interface brief
show ipv6 interface brief
show bgp ipv4 unicast summary
show bgp ipv4 unicast neighbors 10.255.255.1 routes
show bgp ipv4 unicast 172.20.0.53/32
show ip route bgp
show ipv6 route FD51:FBC4:F614::/48
show arp | include 172.22.149.66
show ipv6 neighbors | include FBC4:F614:1::42
# DN42 gateway LXC
wg show
ip -br addr
ip route show proto bgp
ip -6 route show proto bgp
vtysh -c "show bgp ipv4 unicast summary"
vtysh -c "show bgp ipv4 unicast neighbors 10.255.255.2 advertised-routes"
vtysh -c "show bgp ipv4 unicast neighbors 10.255.255.2 routes"
vtysh -c "show bgp ipv4 unicast 172.20.0.53/32"
vtysh -c "show bgp nexthop"
# Proxmox and drive42
pct config 108
pct exec 108 -- ip -br addr
pct exec 108 -- ip route
pct exec 108 -- ip -6 route
pct exec 108 -- ss -lntup
pct exec 108 -- ping -c 3 172.22.149.65
pct exec 108 -- ping -6 -c 3 fd51:fbc4:f614:1::1
pct exec 108 -- curl -4 http://172.22.149.66/
tcpdump -eni vmbr1 'arp or icmp6'

Issues and fixes

IPv4 next-hop over WireGuard

RoutedBits sent IPv4 routes over an IPv6 link-local BGP session. FRR rejected the IPv4 next-hop until peer1 was made point-to-point toward 172.20.19.79.

BGP policy required

FRR marked received IPv4 routes as policy-rejected until an inbound route-map was applied. The simple ALLOW-ALL route-map made the session explicit.

Missing VLAN tag

drive42 originally sent ARP on the wrong VLAN. Adding tag=10 to the Proxmox LXC interface placed it correctly behind Cisco Vlan10.

iBGP next-hop cleanup

A temporary static route-map rewriting the next-hop to 10.255.255.1 was used during debugging. The final setup only needs next-hop-self toward the Cisco, validated with show bgp ipv4 unicast summary and 1127 IPv4 routes received on the Cisco.

DN42-only package installs

Once drive42 was moved to DN42-only networking, normal Debian mirrors were no longer reachable. The fix was to temporarily add a LAN interface for package installation, then remove it.

Related page

The project overview and peering information are available on the DN42 Network Lab page. This page keeps the configuration reference separate from the story of the build.