[buug] semi-random example of DNS, etc. troubleshooting (www.meetup.com., etc.)

Michael Paoli Michael.Paoli at cal.berkeley.edu
Fri Feb 28 07:00:26 PST 2014

So, random example of some DNS (etc.) troubleshooting, and what *not* to do
with your production DNS (at least if you can avoid it! - not always
possible in some circumstances).

So, yesterday, 2014-02-27, a bit after noon, I notice some issues
accessing www.meetup.com. and secure.meetup.com.  I figure it might be
issue with the proxy/firewall stuff at work, so I figure I'll check
again later where I don't have anything potentially mucking about with
my general access to The Internet.  So, many hours later, I check from
home ... still not working, dig a little deeper.

What am I finding in DNS - and likely many other DNS servers doing the
appropriate expected caching?

$ dig -t A www.meetup.com. +noall +answer | grep '^[    ]*[^    ;]'
www.meetup.com.         208     IN      A

and test connectivity with that:
$ nc -z 80 || echo FAILED
It doesn't work.  Also use of traceroute with options and option
arguments -n -T -p 80 didn't show anything particularly interesting
(never got much of a response beyond about 1st hop or so, and never

So, I do a trace on DNS, to see if results are rather consistent - and
alas, quite different:
$ dig -t A www.meetup.com. +trace | grep '^[    ]*[^    ;]' | fgrep  
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A

But do those IP addresses work for TCP connections on port 80?
$ (for a in; do nc -z "$a" 80 || echo FALIED; done)
Yes, they all connected, no errors on any of them.

So, why/where are we getting different DNS data from cached and not?
Our cached NS records:
$ dig -t NS meetup.com. +noall +answer | grep '^[       ]*[^    ;]'
meetup.com.             5793    IN      NS      ns4.p06.dynect.net.
meetup.com.             5793    IN      NS      ns3.p06.dynect.net.
meetup.com.             5793    IN      NS      ns1.p06.dynect.net.
meetup.com.             5793    IN      NS      ns2.p06.dynect.net.

What about uncached?
$ dig -t NS meetup.com. +trace +noall +answer | grep '^[        ]*[^    
  ;]' | fgrep meetup.com
meetup.com.             86400   IN      NS      tom.ns.cloudflare.com.
meetup.com.             86400   IN      NS      lisa.ns.cloudflare.com.
A very different set of results.

@what are the com. NS records? (these also change quite infrequently)
$ dig -t NS com. +noall +answer | grep '^[      ]*[^    ;]'
com.                    159466  IN      NS      b.gtld-servers.net.
com.                    159466  IN      NS      j.gtld-servers.net.
com.                    159466  IN      NS      i.gtld-servers.net.
com.                    159466  IN      NS      l.gtld-servers.net.
com.                    159466  IN      NS      a.gtld-servers.net.
com.                    159466  IN      NS      d.gtld-servers.net.
com.                    159466  IN      NS      h.gtld-servers.net.
com.                    159466  IN      NS      m.gtld-servers.net.
com.                    159466  IN      NS      g.gtld-servers.net.
com.                    159466  IN      NS      k.gtld-servers.net.
com.                    159466  IN      NS      c.gtld-servers.net.
com.                    159466  IN      NS      f.gtld-servers.net.
com.                    159466  IN      NS      e.gtld-servers.net.

$ let's see what one of those has to say about NS for meetup.com.
$ dig @e.gtld-servers.net. -t NS meetup.com. +noall +answer +authority  
+comment | sed -ne '/^;; A/{;p;d;};/^[^;]/p'
meetup.com.             172800  IN      NS      tom.ns.cloudflare.com.
meetup.com.             172800  IN      NS      lisa.ns.cloudflare.com.
most particularly note the TTL of 172800 seconds (== 48 hours)
TTLs of Authority records for NS of subdomains directly under com. also
very rarely change - so those have probably been TTLs of 48 hours, for
a very long time.  So, almost certainly the change has occurred within
the past 48 hours.

So, would appear meetup.com. quite recently changed their NS servers,
however the older NS servers give A records to IPs for www.meetup.com.
that no longer work - and these haven't yet expired from cache.  So that
means that for many, meetup.com has broken their service for possibly as
long as up to 48 hours for many users.

Do we see other corroborating evidence of such a recent change?
$ 2>&1 whois -H meetup.com | fgrep -i -e 'Name Server' -e 'Updated Date:'
    Updated Date: 27-feb-2014
Updated Date: 2014-02-27T14:12:40-0800
Name Server: lisa.ns.cloudflare.com
Name Server: tom.ns.cloudflare.com
Yes - looks like they updated it 2014-02-27

If I take no manual explicit action, when should it be "all better"?
$ dig -t NS meetup.com. +noall +answer | grep '^[       ]*[^    ;]'
meetup.com.             4179    IN      NS      ns3.p06.dynect.net.
meetup.com.             4179    IN      NS      ns1.p06.dynect.net.
meetup.com.             4179    IN      NS      ns4.p06.dynect.net.
meetup.com.             4179    IN      NS      ns2.p06.dynect.net.
In another 4179 seconds, when the apparently no longer usefully
functional for [www.]meetup.com. NS records expire.  At any point after
that, queries for data in/under meetup.com. will have to go back up to
com. NS server, and will pick up the updated authority records, and
follow that and find and get the updated NS records, and then all will
be fine again.  Doing a wee bit 'o searching, appears I'm not the only
one having run into meetup.com's booboo today, e.g. searching Twitter.com.:
Anna Brown
Is there a DNS outage today? http://meetup.com and  
http://statcounter.com are both down.
12:45 PM - 27 Feb 2014

Now, ... not sure exactly what issues meetup.com was dealing with, but
if they could've switched their NS servers more than 48 hours in
advance of the old IPs and old service no longer working, they would
have avoided all disruptions of service.  But in not having managed to
do that (perhaps there was unexpected failure of the old IPs/service?),
they caused at least some disruptions.  And it's not with their power
(or anyone else's) to flush out the older DNS cached data ahead of the
TTL values it's already existed under.

And, something over 4179 seconds later, we now have:
$ dig -t NS meetup.com. +noall +answer | grep '^[       ]*[^    ;]'
meetup.com.             86400   IN      NS      lisa.ns.cloudflare.com.
meetup.com.             86400   IN      NS      tom.ns.cloudflare.com.
Sweet, ... lets see if that all looks good now ...
$ dig -t A www.meetup.com. +noall +answer | grep '^[    ]*[^    ;]'
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
www.meetup.com.         300     IN      A
$ (for a in; do nc -z "$a" 80 || echo FALIED; done)
$ dig -t A secure.meetup.com. +noall +answer | grep '^[         ]*[^    ;]'
secure.meetup.com.      300     IN      A
secure.meetup.com.      300     IN      A
secure.meetup.com.      300     IN      A
secure.meetup.com.      300     IN      A
secure.meetup.com.      300     IN      A
$ (for a in; do nc -z "$a" 443 || echo FALIED; done)
All that looks good, ... and the acid test - try using site with browser
from same client host ...
Well, no longer a DNS issue anyway, ... but they still are having issues:
Website is offline No cached version of this page is available.
Error 522 Ray ID: 103d8d1171de0295
Connection timed out
San Jose
What happened?
The initial connection between CloudFlare's network and the origin web  
server timed out. As a result, the web page can not be displayed.
What can I do?
If you're a visitor of this website:
Please try again in a few minutes.
If you're the owner of this website:
Contact your hosting provider letting them know your web server is not  
completing requests. An Error 522 means that the request was able to  
connect to your web server, but that the request didn't finish. The  
most likely cause is that something on your server is hogging  
resources. Additional troubleshooting information here.

Following the link at the end:
Gives some general details about the 522 error that they're reporting,
but without any specifics about exactly why (e.g. DNS name used or
attempted, IP address(es) used or attempted, port, response or timeout
or whatever) ... so, something for meetup.com to figure out between
themselves and their service provider (apparently cloudflare.com).

doing some searches ... Google ... Google News ... Twitter ...:
Jonathan Carter #@jonathanrcarter  4h
@Register_com Also we are slowly coming back up with our DNS requests.  
  http://meetup.com  had a ddos yesterday - could be linked
Collapse  Reply  Retweet  Favorite   More
1:02 AM - 28 Feb 2014

Hmmmm, possibly related?  But seems likely unrelated - at least to the
DNS issue seen earlier - though it might explain why
http://www.meetup.com/ is still effectively down (serves up a hosted
error page).  In any case, http://www.meetup.com/ and
https://secure.meetup.com/login/ show highly similar hosted page errors.

Let's see what else we can see on Twitter:
*Lots* of buzz about "meetup.com" being down:
Also, fair amount of buzz on that about no tech news having picked it up
yet, and of meetup.com apparently being under DDoS attack (though I
can't say I've yet spotted the claims of alleged/reported DDoS attacks
being reported from a necessarily reliable and/or authoritative source
... yet).
Still spotting nothing relevant on Google News searches - tried
"meetup.com" long with relevant terms, etc.:
down OR ddos OR dns OR cloudflare
... still nothing ... yet
Internet Storm Center https://isc.sans.edu/
Nothing especially noteworthy at present
Cloudflare itself?  Presently shows all good for today and yesterday,
shows all good most recent 6 days including today, except a pair of
issues in two non-US locations on 2014-02-25.
Cloudeflare shows some CloudFlareStatus tweets in the last couple days,
but nothing that immediately appears to be relevant to meetup.com.
Checking a bit more on searches ...
So, ... does appear meetup.com claims to be and likely is under DDoS
attack.  And some bits there would also explain the recent DNS changes,
Meetup @Meetup
We are implementing a solution, but can't yet publish a time estimate  
of when Meetup services will be available globally.
about 13 hours ago
and more recently:
Meetup @Meetup
Meetup is still under a DDoS attack. Our team is fighting back, but  
unfortunately, we're still seeing intermittent outages.
about 2 hours ago
Looks like this was Meetup's first tweet on the matter:
Meetup Support #@meetup_support  22h
Meetup is down for the moment. Our team is working on fixing it right  
now. Sorry for the inconvenience!
7:44 AM - 27 Feb 2014

So, in brief summary, looks like meetup.com. had issues, presumably from
DDoS attack.  They did some (disruptive) DNS changes to attempt to fix
the issue (apparently use or change content delivery providers (CDN))
... but that appears to not yet suffice - even after the DNS issues
corrected themselves over time (max 48 hours to pick up changes due to
TTL on authority records for meetup.com. NS records - had they done or
been able to also change some other records on the older NS server,
that might have possibly reduced the time for the DNS issue, but not
necessarily all that much, and until issue with (apparently new) CDN
provider (between then and meetup.com) is worked out, the DNS being
"all better" still wasn't enough to get them effectively up and running
again).  So, why still down?  Guestimate, CDN & DDoS - CDN can
typically very well withstand DDoS.  However, CDN also has to talk to
back-end ("origin") servers for the hosted site - if those aren't
protected from attack, and/or if the CDN is being access in ways that
most or all traffic must be passed back to origin site (e.g. things
that can't be cached, such as submitting updates to a page - e.g. add
comment) - then the DDoS attack impacts may be mostly passed through
back to origin site, overwhelming those.  Anyway, maybe when it's "all
better now", Meetup.com will have a nice technical write-up of it

More information about the buug mailing list