Our sites are hosted on SiteGround servers, and when we add the XML sitemap to the Knowledge Base, it returns a blank result. I contacted SiteGround support to ask why Retell AI crawlers might be blocked, and this was their response:
==> 185.5.147.51 is the IP address making the requests. It is currently blocked by our firewall (as suggested in the PDF under "Finding 4: Anti-Bot System Interference");
To solve the problem, the service provider must stop violating our security rules i.e.:
-> Using forged and outdated user-agents for its requests - ones which contain "Chrome/120" (an extremely old version of Google Chrome); -> Requesting various files in "/wp-includes/" which is a pattern similar to security vulnerability scans.
If they confirm they have made the changes, we can unblock the IP and it will not be blocked again (assuming it no longer generates seemingly malicious traffic as mentioned above).
This is the best solution as it does not involve whitelisting the IP nor disabling security rules for your website.
==> The other possible solution is for us to whitelist the IP address on your site. You can then try again and let us know if you are still experiencing problems. If so, we will probably have to disable one or two security rules on your site.
We already have other clients that doesn’t have any issues with RetellAI Crawlers because they were hosted in Cloudways Server. This is an important issue to solved because we do have more sites hosted in SiteGround that might be getting the same errors in the future when we setup the RetellAI.
Would it be possible to implement the request from SiteGround support on your end? We want to ensure that any solution follows their recommendations without compromising security, especially if we were to proceed with the second option they suggested.
My issue is that when I add the sitemap link to the Knowledge Base, the result appears blank. I ran a full diagnostic analysis using ManusAI to identify the cause, and it indicated that SiteGround has implemented a firewall security rule that may be blocking the crawler.
I then contacted SiteGround to ask if they could allow Retell AI crawlers on their hosted sites, and the message above was their response.
Is there a way to make the request from SiteGround possible on your end?
Our website crawling is performed by an external crawler on our behalf, and we unfortunately don’t control SiteGround’s firewall or their allowlist directly. The change must be made on SiteGround’s side.
No, they are referring these changes on your end. SiteGround is just requesting if you can update your crawling mechanism or whatever you call it that is not violating their security rules. If this request change is not possible on your crawlers, then I will let SiteGround whitelist your IP address hoping not to compromise or infect our website. thank you.
