VK Cloud logo
Updated atMarch 19, 2024   02:14 PM

Local DNS cache configuration

A local CoreDNS-based caching DNS server can be configured on each cluster node. This is also known as NodeLocal DNS. This will improve the stability and performance of the DNS service in the cluster, without changing existing applications.

Overview of the DNS architecture in a cluster

Let some pod initiate a DNS query.

  • Without using caching DNS servers, the following will happen:

    1. the DNS query will be sent to the IP address of the kube-dns service of the cluster.
    2. This IP address will be translated by kube-proxy into the IP address of the kube-dns endpoint using iptables rules. In doing so, iptables will use conntrack to track connections.
    3. When a response is received from kube-dns, the reverse process will be performed.

    If DNS queries are mostly sent via UDP protocol, then high load on kube-dns (for example, if any application is actively sending DNS queries) may lead to issues:

    • Race condition for conntrack. This results in significant (up to several times) slowdown of responses to DNS queries.
    • Overflow of service tables for conntrack. Records for UDP are removed from these tables only by timeout (default — 30 seconds). If tables are full, new DNS queries sent via UDP will be dropped.
  • When using caching DNS servers, the following will happen:

    1. Pods will refer to the local caching DNS-server, which is located on the same node as the pods.

      This will avoid address translation (Dynamic NAT, DNAT), the use of iptables and conntrack. The issues described above will be eliminated.

    2. The caching DNS-server itself will address to service kube-dns using iptables and conntrack, but via TCP protocol.

      In this case load on kube-dns is reduced, because it is queried directly by a limited number of DNS-servers and not by all services of the cluster that require DNS for their work. Also, when using TCP, the latency associated with UDP packet loss and timeouts is reduced.

See official Kubernetes documentation for more details.

1. Preparatory steps

  1. Create a Kubernetes cluster of the most current version.

    Choose the cluster parameters at your own discretion.

  2. Make sure that you can connect to the cluster using kubectl.

2. Deploy a caching DNS server on each node

The DNS server will be deployed in the following configuration:

  • Deployment type: DaemonSet so that the server is available on all nodes in the cluster.

  • IP addresses that the server listens to:

    • A local 169.254.0.10 IP address on each node. This link-local address is specifically chosen so that there is no overlap with addresses from other subnets used by the cluster.
    • The IP address of the kube-dns service. In Cloud Containers clusters, this is always 10.254.0.10.

    This configuration is used because kube-proxy in Cloud Containers clusters operates in iptables mode.

  • The port to which Prometheus will connect to collect metrics: 9153.

  • Label for service selection: kube-dns: coredns.

  • Cluster domain: cluster.local.

See official Kubernetes documentation for more details about configurations and settings.

To deploy a DNS server:

  1. Create a manifest file for NodeLocal DNS:

    This file is based on the manifest from the official Kubernetes repository, taking into account the configuration above.

  2. Create the Kubernetes resources required for NodeLocal DNS based on the manifest:

    kubectl apply -f nodelocaldns.yaml

    The following information about the created resources should be displayed:

    1serviceaccount/node-local-dns created
    2service/kube-dns-upstream created
    3configmap/node-local-dns created
    4daemonset.apps/node-local-dns created
    5service/node-local-dns created

3. Check the operation of the caching DNS server

  1. Create pod that includes utilities to work with DNS:

    kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
  2. Verify that the pod is successfully created by running the command:

    kubectl get pod dnsutils

    Output should give you the similar information:

    1NAME       READY   STATUS    RESTARTS    AGE
    2dnsutils   1/1     Running   ...         ...
  3. Connect to a bash session inside this pod:

    kubectl exec -it dnsutils -- bash
  4. Check that the DNS queries are successful:

    1nslookup cloud.vk.com && \
    2nslookup kubernetes.default

    Output should give you the similar information:

    1Server:         10.254.0.10
    2Address:        10.254.0.10#53
    3
    4Non-authoritative answer:
    5Name:   cloud.vk.com
    6Address: 95.163.254.194
    7Name:   cloud.vk.com
    8Address: 95.163.254.192
    9Name:   cloud.vk.com
    10Address: 95.163.254.193
    11
    12Server:         10.254.0.10
    13Address:        10.254.0.10#53
    14
    15Name:   kubernetes.default.svc.cluster.local
    16Address: 10.254.0.1
  5. End the bash session in the dnsutils pod:

    exit

Delete unused resources

  1. If the Kubernetes resources you created are no longer needed, delete them.

    1kubectl delete -f https://k8s.io/examples/admin/dns/dnsutils.yaml
    2kubectl  -f nodelocaldns.yaml
  2. A running cluster consumes computing resources. If you no longer need it: