Automating /etc/hosts file entries via API

I ran into a really frustrating problem at work today. My employer-provided laptop is locked down extremely tight for security and compliance reasons and part of that lockdown is the utilization of Cisco's Umbrella software to route certain DNS queries only via VPN.

For whatever reason, DNS resolution via Umbrella is completely broken at the moment for my Windows 10 device. This means any of the internal DNS zones I need to get to... I can't. This includes critical tools to my day-to-day like our internal ticketing system and code repositories. And since I don't have admin privileges either, my options are limited.

Now there are easier ways to solve this than what I'm about to outline:

• remove Cisco Umbrella completely
• have an IT admin add static entries to my hosts file

But neither of these solved the problem because 1. IT hasn't been able to figure out how to get their MDM software to stop re-installing Cisco Umbrella and 2. the DNS entries I need rely on dynamic, load-balanced AWS IPs. So, I had to get more creative. They are also zero fun.

In typical fashion, Linux will bail Windows out here. I have an Ubuntu Linux VM running in VirtualBox on my laptop that I use for various tasks (git, *nix tools, etc.). So, I just had to find a way to bypass Cisco Umbrella DNS resolution, which would be far easier from inside the Ubuntu guest where I have total control.

The best way I could think of was to use a DNS API (which bypasses traditional DNS) to query the public A records for the hosts I needed and then format those and add them to /etc/hosts. Of course all of this would need to be scheduled or at least automated, ideally. Here's how I did it:

1. I chose a free (up to 500 queries a month) DNS Lookup API that I've had some exposure to in the past.
2. I then started working on a bash one-liner that could: query the records via curl, grep only the relevant IP addresses, format them into the hosts file format, and get those into the hosts file. Here's what I came up with:

curl -s "https://www.whoisxmlapi.com/whoisserver/DNSService?apiKey=[mykey]&domainName=[externalDomain]&type=A&outputFormat=JSON" | grep -E -o "[0-8]{1,2}[.][0-9]{1,3}[.][0-9]{1,3}[.][0-9]{1,3}" | sort -u | sed -e 's/^/[externalDomain] /'

The hardest part of this process for me personally was by far the regular expression. Because the JSON output from the Whois API includes both rawText and address fields it was very difficult to get to a regular expression that would pick up reliably on the IP addresses I needed without excluding some IPs altogether or adding additional characters I didn't want. I ended up making the concession that I'm excluding 9 from the first octet of IPs because I couldn't figure out how to get the RegEx to stop picking up on the raw text u0009 that comes before the IP address in that field. I am comfortable with that sacrifice for this application.

It was at this point that I realized I needed to either dump the output into /etc/hosts directly, or find some other way. After some brief searching I found another project I have used previously, update-hosts-file. After getting it all configured with the desired hosts file entries (I also had some static internal addresses I wanted to maintain), I created a simple shell script that..

1. Deletes the existing hosts modules in update-hosts-file's config

   rm -f /usr/share/update-hosts-file/modules/custom/available/sub_domain_com
   

2. Re-creates them based on the one-liner I wrote earlier with fresh hosts entries

   curl -s "https://www.whoisxmlapi.com/whoisserver/DNSService?apiKey=[mykey]&domainName=[externalDomain]&type=A&outputFormat=JSON" | grep -E -o "[0-8]{1,2}[.][0-9]{1,3}[.][0-9]{1,3}[.][0-9]{1,3}" | sort -u | sed -e 's/^/[externalDomain] /'
   

3. Executes an update of the hosts file via update-hosts-file

   update-hosts-file update skip
   

Once I confirmed that was all working as expected, I wanted to ensure this script runs itself on some frequent basis. This VM is actually shut down/rebooted once or more per day, so I added the script to systemd so that it executes on each boot. This means every time I start the VM, the DNS entries I need will be queried and added to the hosts file automatically. The systemd unit file looks like this:

[Unit]
Description=Update Corp host IPs on startup
[Service]
ExecStart=/home/darryl/.config/[script_name].sh
[Install]
WantedBy=multi-user.target

Note that I put the script in my .config directory since it's a git repo.

I'm certain that more seasoned Linux admins could have done this in a more seamless way, but it works for me. :)