Configuring DNS failover through a Fortigate
You may be in a situation where creating a failover group of DNS servers would be valuable to you.
Let’s say you have an on site DNS server performing lookups for some domains. It forwards DNS requests it can not solve to an externally configured “forwarder.” In this list of forwarders, you have the following addresses configure: Primary ISPs DNS server 1, Primary ISPs DNS server 2, Secondary ISPs DNS server 1, Secondary ISPs DNS server 2.
The server configuration causes a timeout for a DNS lookup at 5 seconds, then moves on to the next server until the lookup can be serviced.
What if your primary ISP doesn’t allow any address from the Secondary ISP to access its DNS server?
If your primary ISP fails connectivity to “Primary ISPs DNS server 1” and “Primary ISPs DNS server 2” both fail completely. In essence, the DNS server configured above to forward to your “failover” list of DNS Servers services a single DNS lookup in no less than 10 seconds.
This renders the whole “multiple” server thing useless unless your users can tolerate a Google homepage load in (likely) 260 seconds (disregarding HTTP request timeouts).
“But Matt” you say, “to solve this problem, you can use something to detect when you’re on your backup ISP and adjust the DNS server configuration to prioritize the ‘Secondary ISPs DNS server 1’ and ‘Secondary ISPs DNS server 2.'” Well, I couldn’t find anything that solved that problem (could always write something with my below average C# skills, but it would take way too long). In fact, I’m quite happy I didn’t, because I arrived on a much more elegant and efficient solution: Creating a Virtual Server Load Balancing Group on our Internet Fortigate.
I took the example from the Fortigate handbook titled: HTTP load balancing to three real web servers and flipped it so that the load balancing targets were on the outside.
1) Collect your public DNS servers from your ISPs by contacting them, capturing traffic over UDP/TCP port 53, etc.
Here, none of the clients are allowed to perform DNS lookups to servers other than the site’s DNS server. The site’s DNS server has forwarding servers configured, so I was able to grab these from there, as well as reaching out to tech support at the ISPs to make sure I had them all.
2) Verify that the primary DNS servers are unreachable via one of the following methods from the secondary ISP: TCP SYN or ICMP.
In my case, neither are accessible. While on the secondary ISP, to do this, I use a tool I’ve posted about before called tcping. You can also use nmap as follows:
nmap -PS [DNS server IP] #syn ping or nmap -PA [DNS server IP] #ack ping
Configure the Load Balancing group:
1) Verify that the Load Balance firewall feature is available to your UI. I believe that the first release that included it is v4.0 MR3.
At Global scope, in the web UI, go to System> Admin> Settings and verify that Load Balance is checked under “Display Options on GUI.”
2) At the VDOM scope, in the web UI, go to Firewall Objects> Load Balance> Health Check
3) Click “Create New.” Using the verified check, either TCP (note the Port option) for TCP SYN pinging or PING for TCMP pinging, configure the “Type” accordingly. Set the “Name” to something significant (like “Ping_10_2_3”).
Interval, Timeout, and Retry should be considered. Interval is the check interval. Timeout is the amount of time the check takes to consider a check with no return status as failed. The Retry is the total amount of failures before rolling to the next server member.
Click OK when finished.
4) At the VDOM scope, in the web UI, go to Firewall Objects> Load Balance> Virtual Server
5) Click “Create New” and configure the Virtual Server as follows:
|Name||something significant (like “Public_DNS_Servers”).|
|Interface||The interface where your DNS client(s)/on site DNS servers live.|
|Virtual Server IP||Select an IP to dedicate to be a virtual representation of the external DNS Servers.|
|Virtual Server Port||53|
|Load Balance Method||First Alive|
|Health Check||Select the health check you just create (“Ping_10_2_3”) and click so that it moves from Available to Selected. Clearly you can configure multiple checks.|
Click OK when done.
Note that you can test that the virtual server is accessible from the network by ARPing for it. It does not have an ping response daemon tied to it.
6) At the VDOM scope, in the web UI, go to Firewall Objects> Load Balance> Real Server
7) Click “Create New” and create entries in the order you want them to be prioritized. There isn’t a weight or priority parameter per Real Server object, so the Fortigate assumes that the creation order is the priority (first has the highest). Create the objects as follows:
|Virtual Server||Whatever Virtual Server you just created (“Public_DNS_Servers”)|
|IP Address||The IP address of the public DNS server (destination/forwarded-to server).|
|Max Connections||0 [for unlimited]|
Click OK when done. There is a maximum number of total Real Servers of three per Virtual Server.
8) At the VDOM scope, in the web UI, go to Policy> Policy> Policy and click “Create New”
9) The policy should read as follows:
|Source Interface/Zone||The interface/zone where your DNS client(s)/on site DNS servers live.|
|Source Address||The firewall address object that encompasses your DNS client(s)/on site DNS servers.|
|Destination Interface/Zone||The interface/zone where the public DNS servers live (destination/forwarded-to servers).|
|Destination Address||The name of the Virtual Server object you created earlier (“Public_DNS_Servers”)|
The rest of the settings are up to you. Remember you may need to NAT.
Click OK when done.
Create additional policies for all other ISPs.
I configured the firewall policies to only allow TCP/UDP 53 access for the Public_DNS_Servers from the onsite DNS servers (happens to be Windows DNS server). Doing this breaks some resiliency in Windows DNS server as reviewing DNS logs (with both my eyes and using the free software DNS Log Analyzer) revealed the following:
Time CNAME’s A records is asked for by the Windows DNS server:
20130227 12:59:02 (asking Fortigate's Virtual Server Address) 20130227 12:59:08 (asking geographically_dispersed_SOA_server_1) 20130227 12:59:12 (asking geographically_dispersed_SOA_server_2) 20130227 12:59:16 (asking geographically_dispersed_SOA_server_3) 20130227 12:59:16 (asking geographically_dispersed_SOA_server_4) 20130227 12:59:23 (asking Fortigate's Virtual Server Address)
Time responded to:
20130227 12:59:23 (answered by Fortigate's Virtual Server Address)
Using the following command, I’m able to see the SOA servers FQDNs and see the list of geographically dispersed SOA servers correlate:
nslookup -qa=A -debug [FQDN of A record] [Fortigate's Virtual Server address]
So, if you want your onsite DNS Servers to be resilient against delayed or absent responses from your group of Real Servers, you will want to allow your DNS servers to access ‘any’ address on the internet over port 53.
1) Test failover.
To debug, use the following command:
diag debug app vs -1
There is another solution to this problem as well, but requires much more “work” for the Fortigate unit. You can configure the Fortigate unit to be a DNS server, then have it’s DNS server be assigned to use the System DNS settings. The System DNS (server) settings can be over-ridden by the DNS servers dynamically assigned to a dynamically assigned interface using the “Override internal DNS” option within the interface configuration. However, you can not do this for interfaces with static IP configurations.
This latter option is how I originally was planning to configure DNS “failover;” but while writing this post in the Fortigate forum, I came upon this alternate, much more efficient configuration idea.