HOWTO - Configure Load Balancing

This HOWTO assumes you want the DNS server to respond with different IP addresses (or change the order of a given set of addresses) in order to provide a simple load balancing solution. You have a choice of solutions depending on what you want to do:

Balancing Mail
Balancing Other Services
Balancing Services using SRV
Balancing Services using Split-Horizons
Controlling the Round Robin
Effectiveness of DNS Load Balancing

Balancing Mail

The normal method of handling mail fail-over is using the 'preference' value which is described in this note. However, using the MX record you can balance mail in two ways.

Define multiple MX records with the same priority, for example:
```
; zone file fragment
      IN  MX  10  mail.example.com.
      IN  MX  10  mail1.example.com.
      IN  MX  10  mail2.example.com.
....
mail  IN  A       192.168.0.4
mail1 IN  A       192.168.0.5
mail2 IN  A       192.168.0.6
```
The name server will deliver the MX records in the order defined by the rrset-order and the receiving SMTP software will select one based on its algorithm. In some cases the SMTP selection algorithm may work against the definition of the rrset-order statement. Current versions of sendmail (8.13.x), Exim (4.44) and Postfix (2.1 or 2.2) all have definitive references to indicate they randomly select equal preference servers (Postfix allows control of the behaviour with the smtp_randomize_addresses parameter) and consequentially the SMTP server may use an address which the rrset-order has carefully tried to change! qmail, courier-mta and Microsoft (Exchange and IIS SMTP) documentation does not appear to have definitive references to indicate how they handle the multiple MX RR case.
An alternate approach is to define multiple A records with the same mail server name:
```
; zone file fragment
        IN  MX  10  mail.example.com.
....
mail    IN  A       192.168.0.4
        IN  A       192.168.0.5
        IN  A       192.168.0.6
```
In this case the load-balancing effect is under the control of BIND and the rrset-order record. In order to avoid problems if the receiving mail system does reverse look-up as a spam check then the PTR records for 192.168.0.4, 192.168.0.5, 192.168.0.6 above must all define (resolve to) mail.example.com.

In all the above cases each mail server must be capable of handling and synchronising the load for all the mail boxes served by the domain, either using some appropriate back-end, access to a common file system (NAS, NFS etc.) or by defining all but one server to be a mail relay or forwarder.

Balancing Other Services

Assuming you want to load share your ftp, web or VPN services then you simply define multiple A records with the same name and different IPs as in the example below.

; zone file fragment

ftp   IN  A   192.168.0.4
ftp   IN  A   192.168.0.5
ftp   IN  A   192.168.0.6
www   IN  A   192.168.0.7
www   IN  A   192.168.0.8

; or use this format which is functionally identical
ftp   IN  A   192.168.0.4
      IN  A   192.168.0.5
      IN  A   192.168.0.6
www   IN  A   192.168.0.7
      IN  A   192.168.0.8

Note: While above example shows IPv4 addresses using A RRs, the principle applies equally to IPv6 addresses using AAAA RRs.

The DNS will deliver all the IP addresses defined, the first IP address in the returned list will be in a (default) cyclic a.k.a. round robin order (controlled by the rrset-order 'named.conf' statement). The ftp and www servers must all be exact (synchronized) replicas of each other in this scenario. In summary, multiple RRs are an extremely effective load balancing tool and can even provide powerful failover capabilities depending on the application.

There are, however, a couple of wrinkles, as always, in this perfect world. The first is the use of the sortlist 'named.conf' statement which can cause the IP address order to be changed when delivered from a resolver (sortlist is a complicated statement and not commonly implemented). The second relates to IP address selection when using the standard POSIX library function getaddrinfo implemented in LIBC (C standard function library). RFC 6724 (and its predecesor RFC 3484) nominally defines address selection for IPv6 but is also applicable to IPv4 (especially, but not exclusively, in dual stack operations) depending on the LIBC implementation being used. Certainly glibc (GNU libc used on most *nix systems) implements the RFC features for address selection (many thanks to Dennis Leeuw for the heads-up on this issue). From GLIBC version 2.5 to at least GLIBC 2.16 (GLIBC uses a bizarrely exotic numbering system) the RFC 3484 definition was implemented meaning that certain IPv4 addresses (depending on the precise IPv4 set) can be excluded from ever appearing first in the address list thus defeating any load balancing attempts by authoritative servers. This behavior can be controlled by use of a /etc/gai.conf file (see man gai.conf for details). RFC 6724 restored the supremacy of the DNS sourced list (all other things being equal) though at this time (February 2014) it is not known if the RFC 6724 behavior has been implemented in any GLIBC release (the earliest would have been 2.17). In summary, depending on the implementation of RFC 3484/6724 the client can have a definitive (and final) effect on the ordering of IP (IPv4/IPv6) addresses irrespective of any attempts by the authoritative domain owner to change this.

Note: Applications using the (nominally obsolete, but still supported) gethostbyname standard function which provides, essentially, the same functionality are not affected by the getaddrinfo issues described above.

Balancing Services

The SRV record allows an application to discover the server name (and optional port number) on which a service, such as, SIP or LDAP or whatever is provided. As such, it provides another approach to load balancing. SRV provides both priority and weight fields allowing a fine level of granular configuration as well as providing some level of fail-over. The SRV record description contains an example illustrating this kind of flexibility. However, the end application must be SRV-aware for this approach to work. Application support for SRV is patchy at best - varying from very high in SIP (VoIP) to non-existent (browsers).

Balancing Services with Split-Horizon

An alternative approach to load balancing may be provisioned using BIND's view clause to create a Split-Horizon configuration. Split-Horizon can use the users source-ip address to respond with a service IP address thus balancing for geographic or even service provider-specific traffic sources.

Controlling the order of RRs

You can control the order of RR that BIND supplies in response to queries by use of a rrset-order statement which works for any set of equal records (an RRset). The default behaviour is now defined to be cyclic. Note: Since BIND 9.6 the fixed value of rrset-order is invoked with a build option - standard Linux/BSD packages do not build with this option and thus will not support fixed rrset-order configurations.

However, just to make life truly interesting the sortlist statement can entirely destroy the effect of any rrset manipulation.

Effectiveness of DNS Load Balancing

The previous sections have addressed some of the techniques that may be used to balance load using DNS functionality. However, the real question is, how effective can the DNS be in providing this balancing?

The effects of caching will distort the effectiveness of any IP address allocation algorithm unless a 0 TTL is used - which has the effect of significantly increasing the load on the DNS and is not always implemented consistently. Even if a 0 TTL worked consistently (and it may not be supported consistently by all resolvers) the cure may be worse than the disease. Good news we have great load balancing on our web servers. Bad news we need 17 more DNS servers! Note: You may also want to read this note on TTL values. Then again, you may not want to read it.

Intuitively, and without running any experiments to verify, we would suggest that given a normal TTL (12 hours or more) and ANY IP allocation algorithm other than a single static/fixed list, loads should be reasonably balanced (measured by request arrivals at all the destination IPs in the RRset) given the following assumptions:

Traffic is balanced over a number of DNS caches, that is, traffic originates from a number of ISPs or customer DNS cache locations. Specifically, there are no pathalogical patterns where 90% (or some large'ish number) of the load originates from a particular cache/service.
Traffic volume is reasonably high (perhaps > 10,000 accesses per week) - since pathalogical patterns are more likely in small traffic volumes.

What DNS load balancing cannot do is to account for service loading, for instance, certain transactions may generate very high CPU or resource loads. For this type of control only a local load balancer - one which measures service response times - will be effective.

Finally on this topic, if you still consider that a DNS solution will do the trick if only you could perfectly control the order of IP address generation, then you could use BIND 9's SDB API to achieve the result (or one of the available libraries).

Problems, comments, suggestions, corrections (including broken links) or something to add? Please take the time from a busy life to 'mail us' (at top of screen), the webmaster (below) or info-support at zytrax. You will have a warm inner glow for the rest of the day.