merging IP and MAC addressing
for Cheap Gigabit Ethernet networks
Andris Sidorovs, Riga Technical University, Janis Lacis, University of Latvia, Karlis Ogsts, Tieto Konts Financial Systems Ltd., Guntis Barzdins, Taide Network AS
Abstract. Address Resolution Protocol (ARP) is one of the key protocols in IP stack, used on LANs to map 32bit IP addresses into 48bit MAC (hardware) addresses. Regular ARP as defined in RFC 826 relies on MAC layer broadcasts to perform the mapping. In this paper a server-based ARP extension, named smartARP, is proposed, which allows to extend ARP functionality beyond a single MAC layer broadcast domain. Advantage of using smartARP is that it presents a cheap and fast alternative to employing a router to forward packets between MAC layer broadcast domains, called VLANs. SmartARP is transparent to hosts and LAN switches, operates independent of LAN speed, and is designed to scale for big networks.
Ethernet LAN technology today is universally adapted nearly everywhere, starting from small Departmental networks and ending with huge University campus and Company-wide networks. Invented in early 1970-ies as a simple 10Mbps shared LAN technology, due to its simplicity and popularity, Ethernet afterwards has been extended with a number of powerful techniques:
All this together has made Ethernet not only very popular, but also very cheap, reliable, and truly wire-speed LAN technology. But with speeds and WAN transmission techniques mentioned above, should Ethernet forever remain just a LAN technology? There are two major obstacles why Ethernet does not scale to real WAN:
Numerous techniques have been proposed to deal with these Ethernet scaling problems. VLANs, Layer 3 switching, NHRP, MPOA are just some of them [2,3].
Our goal in this paper is more modest: we propose a solution how to overcome Ethernet scaling problems for IP traffic only, while ignoring any other protocols, like IPX, NetBIOS, and also multicast (since multicast is widely used by routing protocols, it is briefly discussed at the end of paper). This limitation allows to take advantage of merging 32bit IP addressing scheme with 48bit MAC addressing scheme, and thus providing substantial hardware and processing savings in how IP packets are moved between Ethernet segments. These savings become particularly apparent when 1Gbps networking is considered.
The underlying idea of the paper is: in IP protocol stack MAC layer broadcasts are crucial only for one function - they are used by ARP protocol to map next-hop IP address into next hop MAC address. By implementing ARP in a server-based fashion (we call it smartARP) without the use of MAC layer broadcasts, for IP packet delivery we can use Ethernet without MAC layer broadcast functionality at all! How to implement this simple idea and what are its consequences is what this paper is all about. Described techniques are supposed to be nearly transparent to the hosts and LAN switches, and operate independent of the LAN speed.
The novelty of the proposed approach is substantiated by the fact that none of the popular Ethernet switch vendors (3Com, Cisco) have implemented in their switches a simple option to disable MAC broadcast forwarding. A reply from Cisco in this respect  was "none of Cisco switches allow you to completely eliminate broadcasts, as that is truly a function of a layer3 device (i.e. a router.)" Nevertheless, all below described scenarios are tested to work with Cisco and non-Cisco equipment.
1 VLANs and Broadcast domains
When building an Ethernet access network in the University campus (or other big institution), network managers are faced with the problem of network segmentation [??]. Although underlying Ethernet cabling and switching technology can easily provide direct LAN connectivity across the whole campus, the requirement to isolate individual workgroups from each other (since they use different IP subnets, and frequently also internal Novell or MS Networks) requires employing of routers between such workgroups. Segmentation requirement leads to the typical campus network structure depicted in Fig.1.
Fig.1. Typical router-based campus network
A more advanced network design in such cases suggests to use Ethernet switches with VLAN support and inter-VLAN routing based on IEEE 802.10 or some vendor specific encapsulation, as depicted in Fig.2. It makes network design more flexible and allows to concentrate routing functionality on one or few more powerful routers. But use of VLANs does not change the fact that a router is used to move packets between workgroup VLANs, which from logical point of view are completely independent LAN segments.
Fig.2. Campus network built using VLAN Ethernet switches and single router
The two above described approaches (and their vendor specific optimizations) are the only options available today to build segmented campus-wide LAN networks. These approaches were adequate for 10Mbps multiprotocol environments, and with powerful VLAN switches and routers can be scaled up to 100Mbps (e.g. Cisco Catalyst 5500 series). But is this going to scale to 1Gbps networks running IP-only traffic? In some cases highly manageable routing between workgroups indeed is necessary and expensive 1Gbps routers can be used for that. But what about more simple cases where the only goal of workgroup segmentation is to contain MAC broadcasts, non-IP protocols, impacts of misconfigured hosts, and to provide separate IP subnets for workgroups? Our proposal is to use simple MAC broadcast (and multicast, if used) filter to separate workgroups, as depicted in Fig.3. Without MAC broadcast functionality between workgroups, no protocol (IP or non-IP) will be able to communicate with other workgroups, because unicast MAC addresses used in other workgroups will remain unknown (since MAC broadcasts normally are used to discover them). Meanwhile direct MAC unicast communication between workgroups is still possible, if only hosts somehow learn unicast MAC addresses of hosts in other workgroups (such "learning" will be the task of smartARP described in next chapter.)
Fig.3. Campus network segmented into workgroups by MAC broadcast filters
How to implement broadcast filter on Ethernet switch, while preserving unrestricted MAC unicast frame forwarding? To our big surprise we could not find a single switch vendor [3,4] allowing to disable broadcast forwarding (per port or for the whole switch)! Probably MAC broadcasts are so fundamental to the very concept of LAN, that no switch vendor has considered adding this simple feature among the dozens of other configurable features like broadcast storm suppression, etc. Therefore for testing purposes we used a regular Cisco router with two Ethernet ports, configured as a transparent Ethernet bridge discarding all MAC broadcasts:
no ip routing
bridge 1 protocol ieee
bridge 1 address ffff.ffff.ffff discard
no ip address
no ip address
We also succeeded to modify Ethernet bridging software found in LINUX kernel to act as a transparent bridge discarding all MAC broadcasts.
2 Regular ARP and smartARP
ARP (Address Resolution Protocol) was defined in 1982 by David C. Plummer and described in RFC 826 . Since then ARP has been in use unchanged over more than 15 years (!). In Fig.4 is shown ARP frame format as defined by RFC 826 for IP over Ethernet.
Fig.4. ARP frame format (in white are shown "interesting" non-constant fields)
Regular ARP process operates as shown in Fig.5: if host Y wants to communicate with host Z, then Y broadcasts an ARP query for Z's MAC address, Z receives this query and sends ARP response back to Y as a unicast packet. During this process both Y and Z record MAC addresses of each other in their own ARP tables and afterwards can exchange IP packets directly.
Fig.5. Regualr ARP process: host Y sends query and host Z replies
Unlike regular ARP process which runs on all hosts, smartARP is a server program running on only one host in each MAC broadcast domain (or on several hosts for redundancy.) It answers all ARP requests about target IP addresses outside the given broadcast domain. Fig.5. shows the graphic interface of experimental smartARP implementation .
Fig.6. Interface of experimental smartARP implementation
How does smartARP server knows what to answer about ARP requests targeted outside current broadcast domain? There are several ways to know that, therefore smartARP is configurable. To some extent smartARP configuration file is similar to the router configuration, where the network administrator can "express his wishes" about who is allowed to communicate with whom and in what way (directly, via router, or not at all). The smartARP configuration consists of the number of lines (rules) which define MAC address mappings for ranges of target IP addresses. Below is shown a sample smartARP configuration file, illustrating different ways to obtain the target MAC address:
10.1.1.2 255.255.255.255 CONST 00:00:22:d5:e6:f7
10.1.2.0 255.255.255.0 FORWARD 00:00:00:11:11:11
10.1.1.0 255.255.255.0 LOCAL
10.1.3.0 255.255.255.0 IP 00:11
10.2.2.0 255.255.255.0 DNS mac.mydomain.com
10.1.3.1 255.255.255.255 SILENT
10.3.3.0 255.255.255.0 PROXY 10.1.1.2
Each smartARP configuration rule consists of four fields: target IP address and subnet mask (these define the range of target IP addresses to which this rule applies), operation code, and an optional parameter. Following operation codes are available:
For each ARP query to be processed, smartARP uses the first rule in configuration file, which is applicable to given target IP address. If no rule applies, no action is taken. SmartARP processes only ARP queries received through broadcast address or unicast address of server itself (this is important to avoid ARP query looping). SmartARP has no internal states - each ARP query is processed independently.
3 SmartARP application Examples
We will describe two smartARP application examples: one for a small (and friendly) private network, and another for a big (and hostile) public network. In real life a mixture of both approaches is likely to be appropriate.
3.1 Small and friendly Private network
In a private network where no intentional security breaches are expected and the only goal of workgroup segmentation is to limit broadcast domains, non-IP protocols, and effects of host mis-configuration, the smartARP application illustrated in Fig.6 can be used.
Fig.7. Network segmented by broadcast filters and re-connected by smartARP servers
Some IP renumbering in the network might be necessary to switch to smartARP based workgroup segmentation, because like ordinary ARP, smartARP will be consulted by the hosts only about target IP addresses in the same IP subnet as the host itself belongs to. For that reason all hosts in all workgroups formally have to belong to the same big IP subnet, i.e. subnet 10.1.0.0/16 in case of example in Fig.6. Actual assignment of valid IP address ranges (within this big IP subnet) to individual workgroups is achieved by smartARP configuration. It is important to distinguish between formal IP subnets (configured on hosts and routers) and actual (smaller) valid IP address ranges assigned to individual workgroups through smartARP configuration.
Below are shown configuration files of four smartARP servers A,B,C,D depicted in Fig.6 to achieve direct IP connectivity between all workgroups, and backbone:
smartARP server A:
10.1.1.0 255.255.255.0 LOCAL
0.0.0.0 0.0.0.0 FORWARD 00:00:44:d1:d2:d3
smartARP server B:
10.1.2.0 255.255.255.0 LOCAL
0.0.0.0 0.0.0.0 FORWARD 00:00:44:d1:d2:d3
smartARP server C:
10.1.3.0 255.255.255.0 LOCAL
0.0.0.0 0.0.0.0 FORWARD 00:00:44:d1:d2:d3
smartARP server D:
10.1.4.0 255.255.255.0 LOCAL
10.1.1.0 255.255.255.0 FORWARD 00:00:11:a1:a2:a3
10.1.2.0 255.255.255.0 FORWARD 00:00:22:b1:b2:b3
10.1.3.0 255.255.255.0 FORWARD 00:00:33:c1:c2:c3
Following is a sample scenario when host X in workgroup A wants to send an IP packet to a host Y in workgroup C:
Alternatively, in some cases it might be necessary to force some workgroups to communicate not directly, but via a router (for example, to implement some packet filtering). This can be achieved by defining in smartARP the router to be a proxy (configuration for smartARP server A):
10.1.1.0 255.255.255.0 LOCAL
10.1.4.0 255.255.255.0 FORWARD 00:00:44:d1:d2:d3
0.0.0.0 0.0.0.0 PROXY 10.1.4.254
Above configuration will force router (10.1.4.254) to be used as a next-hop for all frames sent from workgroup A to other workgroups. In this case icmp redirect messages have to be disabled on the Ethernet interface of the router by no ip redirects command (in Cisco router.)
For added manageability and debugging of the network, all smartARP transactions can be logged and later reviewed by the network administrator.
3.2 Big and hostile Public network (HANE)
Let us assume that a hypothetical global telecom GlobalX has introduced a new global data communications service (along with Frame Relay, ATM, and SMDS) called Global HANE (Global Hierarchically Addressed Non-broadcast Ethernet). Following is a more precise definition of the Global HANE service:
Is it complicated for a hypothetical telecom GlobalX to start offering HANE service? Not at all: HANE network can be built using standard Ethernet switches (nowadays easily running at 1Gbps) pre-configured with static prefix-based forwarding tables. Assigning addresses in hierarchical manner and maintaining static prefix-based routing tables can be handled similar to telephone numbering. Additional configuration should be applied to the switch ports acting as edge devices of HANE network: these must be configured to discard all broadcasts and multicasts, and also check the source MAC address to be in the address range assigned to the customer.
Fig.8. HANE network structure: user LANs can connect directly or via a router
How the potential customers could use HANE network? If over HANE network customer wants to interconnect just few routers, then changing the MAC address on the Ethernet interface of the router (to the MAC address assigned by the HANE operator) + defining static ARP entries for other connected routers is all what is needed (the rest of the configuration is like for regular Ethernet segment.) On regular Cisco router commands mac-address <desired-mac-address>, and arp <IPaddress> <MACaddreess> arpa would do the job.
But if the customer site consists of just few hosts interconnected by Ethernet? Or HANE connection is over fiber and really fast, let say 1Gbps? Do we really need a router to interconnect customer LAN to HANE? Fortunately no. We can connect customer Ethernet directly to HANE, only we have to change MAC addresses of connected hosts to the MAC addresses assigned by the HANE operator. And unless we want to configure huge static ARP tables on each host, we have to start a smartARP server on one of the hosts (with configuration similar to the one described in Section 3.1.)
Is it possible to change MAC address of the host? There is a popular thought that "MAC addresses of the hosts can not be changed, because they are unique numbers burned into Ethernet interface cards". This statement is not quite true. A correct statement would be that "unique MAC address is burned into each Ethernet interface card, but a device driver will use it only if no MAC address is supplied in the host configuration". Below is description how to change MAC address for hosts running popular operating systems (it is nearly as simple as to configure IP address of the host):
Finally it should be noted that other hierarchical MAC addressing schemes can be used for HANE besides the country-code based scheme mentioned above. If HANE network is going to be used for Internet hosts primarily, the HANE operator might opt to assign 48bit MAC addresses to customers based on their already assigned unique and hierarchical 32bit Internet IP addresses. In this case 16 zero bits can be prepended to the host IP address to get a HANE-wide unique MAC address for the host. If all hosts on the HANE network have such IP-derived MAC addresses, smartARP can be configured a lot simpler (and thus better scaling) by using single "IP" rule in configuration file:
0.0.0.0 0.0.0.0 IP 00:00:
Alternatively, if we do not want to change MAC addresses of the hosts connected to HANE network, smartARP command "DNS" can be used instead. In this case MAC addresses of all HANE connected hosts should be registered in DNS under common domain, for example, mac.globalX.com. Of course, such HANE network will be less scaleable due to non-hierarchical address assignment.
The described broadcast domain limitation together with smartARP is a powerful new technique to implement LAN segmentation at low cost and without performance degradation in high-speed networks. We hope that some switch vendor will eventually add the simple feature of broadcast discarding on switch ports, to make this technique easy deployable. To implement HANE on wide scale, also a prefix-based forwarding table configuration should be supported by the switch (most switches today in forwarding tables support only complete MAC addresses, not prefixes.)
In this paper we have not discussed IP multicast and other protocols, like IPX. We believe that our approach can be extended also to these cases by tunneling broadcasts and multicasts encapsulated inside unicast Ethernet frames between workgroups. Since broadcast and multicast frames usually are much shorter than maximum Ethernet frame size, this should not require fragmentation in most cases.
It is also questionable if regular Spanning-tree protocol is anymore needed in HANE type of network. A more intelligent loop-avoidance with redundant paths could be considered in the absence of broadcasts.
Finally, it should be noted that 48bit addresses used in HANE potentially allow to spare out few bits for other purposes, like QoS or accounting class identification (similar to 800 phone numbers). This could make QoS enforcement and accounting universally available on per source and per destination address.