What are the differences between the "internal" and "external" DNS mechanisms in squid?
This is the Frequently Given Answer to such questions.
First, let's clear up some confusion: The "internal" and "external" DNS mechanisms in squid are both DNS clients, that communicate with one or more proxy DNS servers, which they rely upon to perform the grunt work of query resolution. squid is not a DNS server, does not contain a DNS server, and does not perform DNS query resolution.
The difference between the two DNS clients is in their architectures. Both allow DNS lookups to proceed in parallel with other squid operations. (The system-supplied DNS Client library operates synchronously, blocking the calling thread until the lookup completes, which is unsuitable for a single-threaded server program like squid.) But they each do so in a different way.
squid's "external" DNS client is a very simple wrapper around the system-supplied DNS Client library that uses multiple child processes for parallelism.
When squid has been built to use its "external" DNS client, it
forks() off one or more child processes, which then
exec() the configured cache_dns_program program. It
communicates with those child processes via pipes, writing lookup commands to
the inputs of the programs and receiving the lookup results from their outputs.
Each child process is expected to operate serially, performing one lookup at a
time. squid itself manages farming out DNS lookup requests to
idle child processes.
The "cache_dns_program" program is (usually) dnsserver. It reads
lookup commands from its input, calls the system-supplied DNS Client library
via the gethostbyname() and gethostbyaddr() functions
to actually perform the lookups, and writes the results to its output.
squid's "internal" DNS client is a home-grown DNS Client library that is integrated into squid's primary event-driven processing mechanism. The network I/O to communicate with the proxy DNS servers, to perform any DNS lookups, is handled by the same mechanism as the I/O to and from HTTP servers.
squid's "internal" DNS client speaks the DNS protocol directly, constructing DNS query datagrams and decoding DNS response datagrams itself.
The default, as of squid version 2.3, is for squid to be built to use the "internal" DNS client. However, whilst it may seem from this that the "internal" DNS client is unequivocally preferable, the actual situation is not as clear cut as that. There are several differences between the two DNS clients that, depending from need, may make either preferable to the other.
DNS primarily uses UDP, but it falls back to using TCP if DNS/UDP datagrams are
too small for the task at hand. The system-supplied DNS Client library (used by the
gethostbyname() and gethostbyaddr() functions) on
most systems will correctly fall back to using DNS/TCP to communicate with the
proxy DNS servers,
in the event that the result of a DNS lookup is too big to contain in a DNS/UDP
datagram. (At the time of writing, a reverse lookup for
70.27.197.128.in-addr.arpa. required a 1144-octet response, for
example.) Some system-supplied DNS Client libraries are also capable of
negotating larger DNS/UDP datagram size limits (the default being 512 octets)
with the proxy DNS servers via EDNS0. Thus, when it has been built to use its
"external" DNS client, squid will make use of DNS/TCP when necessary
and EDNS0 if the system supports it.
In contrast, the home-grown DNS Client library in squid, that it uses when it has been built to use its "internal" DNS client, is neither capable of handling DNS/TCP nor capable of using EDNS0 to negotiate larger DNS/UDP datagram size limits. If the response from a proxy DNS server containing the result of a DNS lookup happens to require exceeding the 512 octet size limit, the home-grown DNS Client library in squid will fail to obtain the result of the lookup.
The system-supplied gethostbyname() and
gethostbyaddr() functions, used when squid has been
built to use its "external" DNS client, consult sources other than one's
configured proxy DNS servers. Depending from the operating system and how it
has been configured, they may also consult (for example): WINS, a
HOSTS file, an /etc/hosts file, or NIS.
In contrast, squid's home-grown DNS client library, that it uses when it has been built to use its "internal" DNS client, consults only the DNS.
The difference between the two is thus the same kind of difference as the difference between nslookup and ping.
The DNS protocol, and DNS servers, deal exclusively with fully-qualified domain names. DNS Client libraries, however, allow applications (and thus users) to use either fully-qualified domain names or non-fully-qualified local shorthands. They convert the shorthands to fully-qualified domain names, using locally configured conversion rules, before sending the queries to proxy DNS servers using the DNS protocol.
The system-supplied DNS Client library is usually the BIND DNS Client library on most systems. The BIND DNS Client library provides a shorthand conversion facility known as a "search path", appending a series of suffixes to the shorthand to turn it into a fully-qualified domain name, and performing and DNS lookup on each resulting domain name, until a lookup succeeds and returns a positive result (or the list of suffixes is exhausted). Other DNS client libraries have similar mechanisms.
When squid has been built to use its "external" DNS client, it can be configured to enable this shorthand conversion facility (or, rather, not to explicitly disable it) via the dns_defnames option.
In contrast, squid's home-grown DNS client library, that it uses when it has been built to use its "internal" DNS client, has no such shorthand conversion feature. (This is the reason that the dns_defnames option does not work for the "internal" DNS client.)
In addition to whatever caching is provided by the proxy DNS servers, squid caches name-to-address and address-to-name mapping information, in an internal cache of its own. To cache DNS lookup results properly, an application must take note of the TTL information returned in the responses from the proxy DNS servers, and cache any mapping information for no longer than its specified TTL period. (Not doing so causes difficulties with stale data, which the positive_dns_ttl and negative_dns_ttl configuration options do not adequately address.)
squid's home-grown DNS client library, that it uses when it has been built to use its "internal" DNS client, speaks the DNS protocol directly, takes the TTL information returned in responses from the proxy DNS servers, and records that in squid's own internal name-to-address and address-to-name mapping caches. The internal caching, of such mappings, that squid does thus respects the TTL values of the underlying DNS data.
In contrast, the system-supplied gethostbyname() and
gethostbyaddr() functions, used when squid has been
built to use its "external" DNS client, provide no direct and official means
for returning TTL information for the results that they return. (This is,
of course, because they may consult
non-DNS sources of information,
to which the concept of a TTL for the result of a lookup simply does not apply.)
squid is capable of using an unofficial back-channel to obtain the
TTL information from the DNS lookups that the system-supplied
gethostbyname() and gethostbyaddr() functions do.
However there are two requirements for having this back-channel available:
The system-supplied DNS Client library must be the BIND DNS Client library. This is true for most, but not all, systems.
The BIND DNS Client library must be patched with an appropriate source-code patch and re-built from source.