SiteGround and Googlebot DNS problems, sites losing indexing.

SiteGround has confirmed a serious problem that has prevented Googlebot from crawling many sites hosted there since Monday. Publishers panic when e-commerce store web pages and products leave Google.

SiteGround has posted various updates to the problem, but it is not yet clear what exactly is causing the problem. This in turn is causing more concern in some publishers who tweet that they are done and move to other WordPress hosting.

The problem I see is that Googlebot is unable to retrieve the site's current robots.txt file and therefore does not crawl the site or go back to an older version of the robots.txt file.

The problem apparently started on Monday

The problem on SiteGround started on Monday when publishers noticed that Google had stopped crawling their sites.

SiteGround initially denies the problem, but eventually admits

SiteGround initially tweeted Wednesday that they couldn't identify any problems on their part.

SiteGround has replied on Twitter :

“We have received reports that a number of Google bot networks are unable to crawl some websites using our DNS service.

After a thorough check of our DNS, we can confirm that there are no blocks on our part preventing this scan, nor do we see any logs suggesting a problem with our systems. "

SiteGround “forwards” the problem to Google

SiteGround then forwarded the issue to Google, apparently assuming the issue was there and that Google could fix it.

SiteGround has published :

“We have forwarded the issue to Google and are working to resolve and identify the cause of the issue. We will keep you updated as soon as we have more information or the problem is resolved. "

SiteGround denies the problem again in the end

SiteGround then distanced itself from the problem with a subsequent tweet to essentially say they can't fix it because nothing at their end is broken.

SiteGround has tweeted : 

“Problems that originate outside our environment are quite difficult to predict, but we fully understand the inconvenience they cause. We will update our official post with more information once it becomes available: "

The problem was not widespread and did not affect other web hosts. This was happening at SiteGround, which could indicate a SiteGround-specific problem, although it may not be within the SiteGround systems themselves.

SiteGround passes the ball to AWS and Google

Eventually SiteGround seemed to identify a problem on Thursday morning, tweeting : 

“Update on the problem reported by some users with Google bots unable to crawl their sites: lo
we traced it to a network issue between the AWS Global Accelerator service and Google. We are working with engineers from both teams to fix it. We appreciate your patience! "

A customer tweeted that he felt SiteGround may not have identified the problem and posted a screenshot of an email he received:

SiteGround's customers are losing patience

Understandably, SiteGround's customers have long since lost patience with many tweeting their state of horror, shock and despair:

Real effect on a SiteGround customer who asked us for help

We received about 20 requests for help from yesterday and today SiteGround customers deeply disappointed from this problem with major damage to positioning and organic traffic.

Here is a Screen that shows how since 11 November the site is literally dead.

Solution to the problem?

A SiteGround customer seemed to confirm that the issue is related to SiteGround's DNS and that moving their website's DNS to an external DNS solved the problem:

At the time of writing, the problem is still ongoing.