Table of contents of the article:
SiteGround has confirmed a serious problem that has prevented Googlebot from crawling many sites hosted there since Monday. Publishers panic when e-commerce store web pages and products leave Google.
SiteGround has posted various updates to the problem, but it is not yet clear what exactly is causing the problem. This in turn is causing more concern in some publishers who tweet that they are done and move to other WordPress hosting.
The problem I see is that Googlebot is unable to retrieve the site's current robots.txt file and therefore does not crawl the site or go back to an older version of the robots.txt file.
The problem apparently started on Monday
The problem on SiteGround started on Monday when publishers noticed that Google had stopped crawling their sites.
SiteGround initially denies the problem, but eventually admits
SiteGround initially tweeted Wednesday that they couldn't identify any problems on their part.
SiteGround has replied on Twitter :
“We have received reports that a number of Google bot networks are unable to crawl some websites using our DNS service.
After a thorough check of our DNS, we can confirm that there are no blocks on our part preventing this scan, nor do we see any logs suggesting a problem with our systems. "
SiteGround “forwards” the problem to Google
SiteGround then forwarded the issue to Google, apparently assuming the issue was there and that Google could fix it.
“We have forwarded the issue to Google and are working to resolve and identify the cause of the issue. We will keep you updated as soon as we have more information or the problem is resolved. "
SiteGround denies the problem again in the end
SiteGround then distanced itself from the problem with a subsequent tweet to essentially say they can't fix it because nothing at their end is broken.
SiteGround has tweeted :
“Problems that originate outside our environment are quite difficult to predict, but we fully understand the inconvenience they cause. We will update our official post with more information once it becomes available: "
The problem was not widespread and did not affect other web hosts. This was happening at SiteGround, which could indicate a SiteGround-specific problem, although it may not be within the SiteGround systems themselves.
SiteGround passes the ball to AWS and Google
Eventually SiteGround seemed to identify a problem on Thursday morning, tweeting :
“Update on the problem reported by some users with Google bots unable to crawl their sites: lo
we traced it to a network issue between the AWS Global Accelerator service and Google. We are working with engineers from both teams to fix it. We appreciate your patience! "
A customer tweeted that he felt SiteGround may not have identified the problem and posted a screenshot of an email he received:
SiteGround's customers are losing patience
Understandably, SiteGround's customers have long since lost patience with many tweeting their state of horror, shock and despair:
Real effect on a SiteGround customer who asked us for help
We received about 20 requests for help from yesterday and today SiteGround customers deeply disappointed from this problem with major damage to positioning and organic traffic.
Here is a Screen that shows how since 11 November the site is literally dead.
Solution to the problem?
A SiteGround customer seemed to confirm that the issue is related to SiteGround's DNS and that moving their website's DNS to an external DNS solved the problem:
At the time of writing, the problem is still ongoing.
Update at 14:40 on November 12th
We had a lot of customer inquiries that we went through on our Hostings yesterday, mentioning that the site again points to SiteGround hosting and not ours.
By checking the DNS zone of the domain still on SiteGround, it appears that the DNS zone has been updated by resetting the changes we made.
This is probably due to the resolution operations that SiteGround is doing, but which in this case with the sites already migrated to external Hosting are only creating further problems to the already problems created by the proceeding problem that we have covered in this article.