DNS Troubleshooting - Everything Depends on It
Gideon T. Rasmussen - CISSP, CISM, CFSO, SCSA

The Domain Name System (DNS) service is required to access e-mail, browse Web sites and use hostnames in general. DNS resolves hostnames to IP addresses and back (e.g. www.cyberguard.com translates to 64.94.50.88). This article details how DNS works under normal circumstances and provides troubleshooting tips.

DNS INTERNET CONCEPTS

DNS Topology
DNS is a hierarchical, decentralized approach to resolving names. It uses client/server architecture to maintain and distribute hostnames and IP addresses. The root servers are at the top of the inverted DNS tree. Beneath the root servers are top-level DNS servers (.com, .net, etc.) and ultimately authoritative DNS servers. The authoritative DNS servers contain DNS entries for their respective domains (e.g. www.cyberguard.com).

Establishing a Domain
When an organization decides to establish a new Internet presence, a domain must be purchased from a registrar organization such as Network Solutions (http://www.networksolutions.com). From the registrar"s management console, the domain owner configures which DNS server(s) will be authoritative for that domain. The information is then uploaded to the 13 root DNS servers.

Reverse DNS Resolution
Reverse DNS resolution (IP address -> name) throughout the Internet is essentially broken. The history behind this is that when large networks were sold and subnetted, the owners did not always create in-addr.arpa entries to support reverse DNS lookups. Reverse DNS starts at the largest network and works its way down (e.g. with 192.0.34.166, it would start with the 192 class A network). In other words, the reverse DNS request may never make it to the domain"s authoritative DNS servers.

DNS Caching
DNS servers cache resolution responses by the Time to Live (TTL) configured on the authoritative DNS server for the respective domain. RFC1912 2.2 recommends at least 3 days if the DNS environment is fairly stable. If a domain is configured in this manner, it will take up to 3 days for DNS changes to populate throughout the Internet. This timeframe can be lessened by reducing the TTL at least 3 days before the actual change.

HOW DNS INTERNET TRAFFIC FLOWS

Inbound Traffic
Inbound traffic can be defined as how Internet traffic resolves a domain and ultimately the systems that correspond to the supplied hostname.

DNS clients request name resolution from their local DNS server. If the DNS server does not have a response cached, it refers the request up to the root DNS servers and consecutive name servers until it receives a response. The local DNS server passes on the response to the DNS client and caches it for future reference until the TTL expires.

The nslookup command is useful to illustrate how DNS traffic flows and to troubleshoot issues. The first step is to determine the authoritative DNS servers for that domain from the SOA (statement of authority) record:


C:\> nslookup
> set type=SOA
> cyberguard.com

Non-authoritative answer:
cyberguard.com
primary name server = q3.cyberguard.com
responsible mail addr = postmaster.mail.cyberguard.com
serial = 2004033002
refresh = 43200 (12 hours)
retry = 7200 (2 hours)
expire = 1296000 (15 days)
default TTL = 86400 (1 day)

cyberguard.com nameserver = q3.cyberguard.com
q3.cyberguard.com internet address = 64.94.50.80

From this, we see that all inbound DNS traffic to cyberguard.com flows to q3.cyberguard.com (64.94.50.80).

Next, change to the authoritative DNS server for that domain and request resolution:

C:\> nslookup
> server 64.94.50.80
Default Server: q3.cybg.com
Address: 64.94.50.80

> www.cyberguard.com

Name: www.cyberguard.com
Address: 64.94.50.88

Querying the authoritative DNS server provides the actual DNS entry without the chance of receiving a cached response.

Still using the authoritative DNS server, we can check the mail server for this domain:

> set type=MX
> cyberguard.com

Non-authoritative answer:
cyberguard.com MX preference = 10, mail exchanger = q3-ext.cyberguard.com
cyberguard.com MX preference = 10, mail exchanger = mail.cyberguard.com

cyberguard.com nameserver = q3.cyberguard.com
mail.cyberguard.com internet address = 64.94.50.85
q3.cyberguard.com internet address = 64.94.50.80

From the nslookup output above, we can see that the CyberGuard mail servers are q3-ext.cyberguard.com and mail.cyberguard.com. The corresponding IP addresses are also provided. Mail is delivered to the system with the lowest preference value. In this case, the values are the same so mail is sent to either system.

Outbound Traffic
Outbound traffic is a bit more difficult to describe because infrastructure configurations vary. In the table below, DNS traffic flows up from the DNS client until resolution is provided (when the response is not cached).

System
Action
Purpose
6
Root servers
Resolution or forwards to top level DNS servers
Refers to top level DNS servers
5

ISP DNS server(s)
Resolution or forwards to root DNS servers
Resolves names for Internet systems
4
External DNS server(s)
Resolution or forwards to ISP DNS server(s)
Resolves names & authoritative DNS
3

CyberGuard Firewall
Resolution or forwards to external DNS server(s)
Resolution & recursive query protection
2

Internal DNS server(s)
Resolution or forwards to firewall
Resolves names for intranet systems
1
DNS client
Queries internal DNS server(s)
Provides name resolution to applications


TROUBLESHOOTING

Inbound Troubleshooting
If DNS traffic does not resolve from the Internet, use the nslookup command to trace the traffic flow from the root DNS servers down and confirm that the DNS entry is correct:

  1. Start with the SOA record (set type=SOA). Is the IP address of the authoritative DNS server what you expect?
  2. Switch nslookup to the DNS server identified in the SOA record (e.g. server 64.94.50.80). Does the DNS server respond?
  3. Query the DNS server for the hostname in question (e.g. www.cyberguard.com). Does the hostname resolve? Is the IP address what you expect?

Outbound Troubleshooting

1. From a DNS client, use the nslookup command to troubleshoot name resolution (nslookup www.cyberguard.com).

2. If name resolution fails, try to ping the DNS server (may require firewall rules).

a. If the ping command fails, there is an issue with the network or the availability of the DNS server.

b. If the ping command is successful, the issue is DNS related

3. If name resolution fails, does the nslookup command display the DNS server you expect?

4. On Windows systems, try the following:

a. Flush the DNS client resolver cache (ipconfig /flushdns)

b. Renew the DNS client registration (ipconfig /registerdns)

5. If name resolution still fails, check the DNS server:

a. Is the DNS process running (ps -ef | grep named)?

b. If the DNS process is running, and name resolution fails on the DNS server itself, there may be an issue with the DNS forwarder. A DNS forwarder is either a DNS server or the root DNS servers, just another link in the chain. Try using the troubleshooting steps detailed above to determine why it is not responding.

6. If you are using a CyberGuard firewall with Split DNS:

a. On the Servers tab, do you have the appropriate interfaces selected? If an internal interface is not selected, the network connected to it will not be able to use the firewall for DNS. The same applies for the external interface and systems on the Internet.

b. Capture a workstation DNS session with "netguard -An | grep workstation IP address". A "P" in the second column indicates the traffic was permitted. If you see a "D" in that column, the traffic was denied. If you do not see anything, the traffic never reached the firewall.

It is important to note that DNS servers essentially comprise a distributed database of hostnames to IP addresses. DNS traffic flow is quite simple. When troubleshooting inbound, be mindful of SOA records and request resolution from the authoritative DNS server. Conduct outbound troubleshooting from the DNS client to the DNS server(s).

How Network Traffic Flows - Getting Started

Network Troubleshooting - A Complex Process Made Simple

E-mail Troubleshooting - The Mail Must Get Through!









Copyright © 2005 CyberGuard Corporation All Rights Reserved.
Reprinted with Permission