Last Friday at around 14:05 we noticed that our website (www.scrt.ch) along with some other services we use internally were no longer accessible. We immediately tried to figure out why that was and quickly noticed that our DNS requests were not returning the correct IP addresses.
As these responses were not only returned by our DNS server, but all servers globally, we had a short moment of anguish where we wondered whether we had forgotten to renew our domain name or if our account at our registrar had been hacked!
Thankfully none of these had happened as we observed that the DNS configuration at the registrar and in our name servers was not altered. So at 14:17 we contacted nic.ch to ask them whether they could tell us any more about what had happened. They informed us that they had received a change request from our registrar (Gandi) and that we should check with them. Following further discussion with them, they found that similar changes had been requested for 94 .ch and .li domain names.
We then contacted Gandi directly who told us that 751 of their domain names had been hijacked and they subsequently started to rollback the changes. By 15:05, everything was back in order, though we still haven’t received any news as to what made all of this possible in the first place.
Here’s some background information on the DNS protocol to better understand what happened.DNS resolution
Each time a web site or any resource is accessed using a fully qualified domain name such as www.scrt.ch, the DNS resolution process is run to obtain the IP address of the target system. DNS lookup walks through a hierarchical DNS structure which, abstracting any cache mechanism, works in the following manner:
- The client (eg a web browser, a mail gateway, a web proxy) queries its DNS resolver. In our case, the resolver is queried for the record www.scrt.ch. The resolver will then handle the resolution (recursive query) on behalf of the client.
- First, It will query one of the root DNS servers serving the internet globally. The root DNS server will consider the Top Level Domain (TLD), which in this example is .ch, and will provide the resolver with the IP address of the name server in charge of this TLD which for .ch is nic.ch (Switch).
- Then the resolver will query the name server at nic.ch which will in turn provide the resolver with the IP address of the name servers responsible for scrt.ch.
- Finally, the resolver will query the name server authoritative for scrt.ch and will get the IP address matching the record www.scrt.ch.
DNS configuration
As a domain owner, you must first use a registrar to register your domain name, which will handle most administrative tasks for you. The main information the registrar owns which is important for this discussion is the authoritative name servers for your domain.
The registrar will typically propagate this information through the DNS global infrastructure, which basically means informing the TLD registry about the IP addresses of the authoritative name server for your domain.
Here we are
The domain scrt.ch is registered at Gandi where we configure the IP addresses of our name servers. Gandi is responsible for propagating this information to nic.ch, enabling the resolution for our domain scrt.ch globally.
The attack
Last Friday, an attacker was able to compromise a technical provider used by Gandi to communicate with various TLD registries. This compromise allowed him to request changes to registries, including nic.ch, to modify the name server information for several domains, including ours.
At this point, a rogue DNS server was introduced in the DNS resolution path. Looking at the DNS resolution process described above, the hijack happened in step 3 where nic.ch was providing the rogue DNS server instead of the valid one to any resolver, allowing the attacker to redirect any requests for impacted domains to IP addresses owned by the attacker himself.
Gandi’s incident notification can be found here.
On our end, the impacts this attack had were:
- For an hour, the www.scrt.ch website was redirected to another host serving up an exploit kit that would try to infect vulnerable browsers. However, if you had visited our site previously, you would not have been affected as HTTP Strict-Transport-Security would have forced your browser to use a valid HTTPS which the attacker could not emulate, therefore resulting in a connection error. The only case where you might have been at risk is if you had never visited our site previously and you had an outdated vulnerable browser (and/or browser plugin), though if this is the case, attackers would probably not wait for you to visit our website to compromise you.
If you want to check for the occurrence of a connection from one of your systems to the malicious web server, you can look for connections to the following IPs: 46.183.219.205, 46.183.219.206, 46.183.219.227.
- During the attack, all emails sent to scrt.ch addresses were not delivered to our server but to a foreign mail server which, according to tests performed during the attack, was not configured to accept emails for our domain. While it doesn’t seem to have been the objective here, this type of attack could potentially be used to read incoming emails while the name servers are poisoned.
In our case, we have always treated emails as inherently insecure and make sure to use other channels and encryption to transfer sensitive information.
Despite the fact that this incident was entirely out of our control, we will be taking steps to further reduce the impact such an attack could have in the future and of course recommend these to all of you reading this, as such attacks can happen to anyone:
- Preload Strict-Transport-Security into browsers for our website in order to protect even first-time visitors.
- Monitor DNS resolution by not only querying authoritative name servers directly but by crawling the full hierarchy.
- Discuss with nic.ch whether detection of such events could be improved. Having over 90 domains owned by different entities changed at the same time to point to a single name server is really suspicious.
- Implement DNSSEC. While in this case the attacker could have been able to either suppress or update the DS record in the TLD zone, the cache mechanism would have mitigated the attack until DS record TTL’s expiration, as confirmed by a discussion with SWITCH-CERT.
For reference, blog posts of Gandi and SWITCH-CERT