Building a network

How networks are designed and physically put together, from a house to a data center, an ISP and a cloud account: the parts at each scale, address and VLAN plans, redundancy, racks and cabling, security zones and monitoring. The technologies inside are explained in Wired networks, Wi-Fi and Autonomous systems & BGP; server-side commands are in Sysadmin.

The same parts at every scale

Every network, however big, is built from the same four layers. Learn to spot them and any diagram becomes readable.

Home

• The ONT (fiber), cable modem or DSL modem turns the ISP's medium into Ethernet. ISPs often combine it with the router in one box.
• The router does NAT, DHCP, DNS forwarding, the firewall and usually Wi-Fi (see A packet's journey). If you use your own router behind the ISP's, put the ISP box in bridge mode to avoid two layers of NAT.
• One access point rarely covers a multi-story house: walls, floors and distance eat the signal. Add access points with wired backhaul (an Ethernet run to each one), or use MoCA (Ethernet over the TV coax) or powerline adapters where you can't run cable.
• Mesh systems link access points over the air instead. Easy to install, but every wireless hop shares airtime with clients and adds latency; a mesh with a dedicated backhaul radio, or better a cable, does much better. Placement and channels are in Wi-Fi.
• For new cable runs use Cat6 (10 Gbit/s up to about 55 m) or Cat6A (10 Gbit/s to 100 m); Cat5e still carries 1 Gbit/s, and 2.5 Gbit/s on most runs, to 100 m.
• Put smart-home gadgets on a separate guest or IoT network if the router offers one, and turn on IPv6: the ISP delegates a prefix (often a /56 or /60) and the router hands out /64s.

Small office

A VLAN (virtual LAN) splits one physical switch into several separate networks. An access port belongs to one VLAN and its device never knows; a trunk carries many VLANs at once, each frame marked with a 4-byte 802.1Q tag holding its VLAN ID (1–4094). The firewall has one interface per VLAN on its trunk and decides which VLAN may talk to which. VLANs themselves are explained in Wired networks.

A worked subnet plan: one /24 per VLAN, the VLAN ID as the third octet, the gateway always .1.

• 10.10.0.0/16 leaves room for 256 such VLANs; IPv6 gets a /64 per VLAN from the office's prefix, e.g. 2001:db8:abcd:10::/64 for VLAN 10.
• Phones learn their voice VLAN from the switch via LLDP-MED, so one cable can serve a phone and the PC plugged into it.
• Keep 192.168.0.0/24 and 192.168.1.0/24 out of office plans: staff on VPN from home networks using the same range will find half the office unreachable.

Campus and enterprise

• Access switches sit in wiring closets (an IDF, intermediate distribution frame, on each floor). Copper to desks is limited to 100 m per channel: 90 m of fixed cable plus 10 m of patch cords.
• Distribution switches collect a building's closets (in the MDF, main distribution frame) and are where VLANs are routed and policy applied.
• The core joins buildings and the data center over a fiber backbone (single-mode between buildings, where distances run to kilometers) and does nothing but forward fast. Smaller sites merge distribution and core into a collapsed core.
• Redundancy: every switch has two uplinks to two different switches. Links are bundled with LACP (IEEE 802.1AX) and the switch pairs act as one (MLAG, multi-chassis link aggregation), so both uplinks carry traffic. Spanning Tree (RSTP or MSTP) stays on as a safety net against loops; designs that route all the way to the access switch avoid it entirely.
• First-hop redundancy (VRRP, or Cisco's HSRP) gives hosts one gateway address shared by two routers; see Design principles.
• 802.1X authenticates every port and Wi-Fi connection before it passes traffic: the device (the supplicant) proves who it is to the switch or access point (the authenticator), which asks a RADIUS server. EAP-TLS with device certificates is the strong option; the server can also drop the device into the right VLAN. Printers and other devices that can't do 802.1X fall back to MAC authentication (MAB), which is easy to spoof, so give them a restricted VLAN.
• Branch offices connect back over the internet with VPNs or SD-WAN, or over private circuits from a carrier.

Data center

A data center network carries mostly east-west traffic (server to server: a web request fans out to dozens of services and databases) rather than north-south (in from and out to users). The modern answer is a spine-leaf fabric, also called a Clos network after Charles Clos's 1953 telephone switch design.

• Every leaf (a top-of-rack, or ToR, switch) connects to every spine. Leaves never link to each other, and neither do spines. Any server reaches any other in three switch hops: leaf, spine, leaf, with the same latency and bandwidth whichever pair you pick.
• ECMP (equal-cost multi-path) routing gives each leaf one equal path per spine. Switches hash each packet's addresses, protocol and ports, so a flow sticks to one path (no reordering) while thousands of flows spread over all of them.
• To grow, add leaves for more server ports and spines for more bandwidth; past that, a 5-stage design adds a "super-spine" layer above several spine-leaf pods.
• Oversubscription compares server bandwidth with uplink bandwidth. A leaf with 48 × 25 Gbit/s server ports (1.2 Tbit/s) and 8 × 100 Gbit/s uplinks (800 Gbit/s) is 1.5:1. AI training clusters usually get separate back-end fabrics at 1:1, because their traffic is all east-west at full speed.
• Fabrics usually run routing, not Spanning Tree, everywhere: often eBGP with a private ASN per leaf (RFC 7938), or OSPF or IS-IS. Servers are typically dual-homed to two leaves.
• Tenants and applications that need their own layer-2 or layer-3 networks get them as an overlay: VXLAN wraps their frames in UDP (24-bit segment IDs, so about 16 million segments) and EVPN, a BGP extension, tells the leaves where every address lives (RFC 7348, RFC 8365).
• North-south traffic enters through border leaves connected to firewalls, load balancers and the WAN or internet edge; load balancing is in System design.

ISP and carrier

• A POP (point of presence) is a building where the ISP has equipment: a local exchange or central office holding access and aggregation gear, or a carrier hotel where it meets other networks.
• Core links are protected twice: by physically separate fiber routes (two fibers in the same trench fail together) and by fast reroute in MPLS or segment routing, which switches to a precomputed backup path in tens of milliseconds.
• How the ISP connects to the rest of the internet (peering, transit, BGP policy) is in Autonomous systems & BGP.

Cloud networks

In a cloud you build the same things through an API: no cables, but the same ideas of address space, subnets, routes, gateways and firewalls.

• A subnet is "public" only because its route table sends 0.0.0.0/0 to an internet gateway. Put load balancers and NAT in public subnets and everything else in private ones.
• Spread every tier across at least two zones, and give each zone its own NAT gateway, so losing a zone (a separate building with its own power) leaves the application running.
• Pick address space that doesn't overlap your offices, data centers or other cloud networks: you can't peer or route between overlapping ranges, and changing a VPC's range later means rebuilding.
• Peering is usually not transitive: if A peers with B and B with C, A can't reach C through B. Many networks need a hub.

Design principles

When a link breaks, a routed network heals itself. How fast depends mostly on how quickly the failure is detected:

For hosts on a LAN, the weak point is the default gateway: a host has one gateway address and no routing protocol. First-hop redundancy protocols such as VRRP (RFC 5798) let two routers share a virtual gateway address; the backup takes it over when the active router stops sending heartbeats.

IP address planning

• Plan in a hierarchy: region → site → function → VLAN, so each level summarizes into one route: for example 10.0.0.0/12 for Europe, 10.16.0.0/12 for the Americas, a /16 per site, a /24 per VLAN.
• Leave room: reserve the next block of each size for growth; renumbering a live network is painful.
• Avoid overlaps with partners, acquisitions, cloud networks, home networks of VPN users, and 100.64.0.0/10 (carrier-grade NAT).
• IPv6: a site typically gets a /48, which holds 65,536 /64 subnets. Every VLAN gets exactly a /64 (SLAAC requires it); encode the VLAN in the fourth group (2001:db8:abcd:10::/64 for VLAN 10); use /127 on router-to-router links (RFC 6164) and /128 for loopbacks. Homes get a /56 (256 subnets), a /60 (16) or sometimes a single /64. RIPE-690 has the operational advice.
• Record every allocation in an IPAM (IP address management) tool, not a spreadsheet nobody updates.

Naming and documentation

A network nobody can describe can't be fixed at 3 a.m. Decide conventions before the first device arrives.

• Keep one source of truth: a DCIM and IPAM tool such as NetBox or Nautobot that holds sites, racks, devices, cables, addresses, VLANs and circuits. Generate configurations from it instead of typing them.
• Back up every device configuration automatically into Git (Oxidized and RANCID do this), so each change has a diff and a date.
• Draw two kinds of diagram: physical (racks, devices, cables, ports) and logical (subnets, VLANs, routing, firewall zones). One diagram trying to show both shows neither.
• Write a runbook for the failures you expect: ISP down, core switch dead, power out.

Physical build

How copper and fiber carry signals is in Wired networks.

Security zones and segmentation

Group systems into zones by how much you trust them, and make the firewall between zones deny everything except the flows you list.

• Default deny between zones, explicit allow rules, and logs for the denies.
• Segment inside zones too (microsegmentation): host firewalls and cloud security groups limit what each server can reach, so one compromised machine doesn't open the whole zone. Zero trust designs go further and grant access per user and device, wherever they connect from.
• Harden the access layer: 802.1X, DHCP snooping and dynamic ARP inspection (stop rogue DHCP servers and ARP spoofing), BPDU guard on user ports (stop loops), unused ports shut down.
• Keep firmware current and management protocols encrypted (SSH, HTTPS, SNMPv3).
• Host firewall commands (nftables, ufw) are in Sysadmin.

Monitoring

Alert on what users feel and what predicts it: links or neighbors down, error counters rising (a bad cable or optic), sustained utilization above your upgrade threshold, packet loss between sites, failing power supplies or fans, optics with falling light levels, and certificates or licenses about to expire. Prometheus with snmp_exporter, LibreNMS and Zabbix are common open-source starting points; dashboards are in Grafana.

Build checklist

• Requirements: users and devices now and in three years, applications, uptime target, budget
• Address plan: IPv4 and IPv6 blocks per site and VLAN, recorded in IPAM
• VLAN and zone plan, with the firewall policy between zones written down
• Naming conventions for devices, interfaces, cables and DNS
• Physical: rack layout, power (A and B feeds, UPS runtime), cooling and airflow, cable routes
• Cabling installed, tested and labeled at both ends; patch panels documented
• Device configurations generated from templates; default passwords changed; firmware updated
• Management network isolated; SSH and SNMPv3 only; logins via central authentication
• Redundancy tested by pulling links and powering off devices, one at a time
• Monitoring, logging and alerts live before users arrive
• Configuration backups automated
• Diagrams, runbooks and the source of truth match what was built

Recipes

Put a Linux host on a VLAN

When a server or router needs an interface in a tagged VLAN on a trunk port.

sudo ip link add link eth0 name eth0.20 type vlan id 20
sudo ip addr add 10.10.20.2/24 dev eth0.20
sudo ip link set eth0.20 up
ip -d link show eth0.20   # "vlan protocol 802.1Q id 20"

These commands don't survive a reboot. With NetworkManager:

sudo nmcli con add type vlan con-name vlan20 \
  dev eth0 id 20 ipv4.method manual \
  ipv4.addresses 10.10.20.2/24

Bond two NICs with LACP

When a server has two links to a switch pair configured as one LACP bundle (MLAG).

sudo ip link add bond0 type bond mode 802.3ad
sudo ip link set eth1 down
sudo ip link set eth1 master bond0
sudo ip link set eth2 down
sudo ip link set eth2 master bond0
sudo ip link set bond0 up
cat /proc/net/bonding/bond0   # partner MAC, active links

Generate a subnet plan

When you want the VLAN-numbered /24 plan above for any site block.

python3 - <<'EOF'
import ipaddress
site = ipaddress.ip_network("10.10.0.0/16")
nets = list(site.subnets(new_prefix=24))
for vlan in (10, 20, 30, 40, 99):
    net = nets[vlan]
    print(vlan, net, "gateway", net[1])
EOF

Check what a port is cabled to

When the documentation and reality may disagree. Needs lldpd on the host and LLDP on the switch.

sudo lldpctl                 # neighbor name, port, VLAN
networkctl lldp              # systemd-networkd hosts

References

• RFC 1918: Address Allocation for Private Internets (opens in a new tab): the private IPv4 ranges
• RIPE-690: IPv6 prefix assignment for end users (opens in a new tab): /48, /56 and /64 in practice
• RFC 7938: Use of BGP for Routing in Large-Scale Data Centers (opens in a new tab): eBGP spine-leaf fabrics
• RFC 7348: VXLAN (opens in a new tab): the overlay encapsulation
• RFC 8365: EVPN as a Network Virtualization Overlay (opens in a new tab): the control plane for VXLAN fabrics
• Meta Engineering: Introducing data center fabric (opens in a new tab): a large spine-leaf design explained
• RFC 5798: VRRP version 3 (opens in a new tab): first-hop redundancy
• RFC 5880: Bidirectional Forwarding Detection (opens in a new tab): fast failure detection
• RFC 6164: Using 127-bit IPv6 prefixes on inter-router links (opens in a new tab): point-to-point addressing
• RFC 7011: IPFIX (opens in a new tab): flow export
• Amazon VPC User Guide (opens in a new tab): VPCs, subnets, gateways
• Azure Virtual Network overview (opens in a new tab): VNets and their features
• Google Cloud VPC overview (opens in a new tab): global VPC networks and regional subnets
• NetBox documentation (opens in a new tab): a source of truth for DCIM and IPAM
• ip-link(8) (opens in a new tab): VLAN and bond interfaces on Linux
• Linux bonding driver (opens in a new tab): bonding modes, including 802.3ad