I would not let AI crawlers eat my server budget without checking logs first. A lot of “mystery” hosting bills are just bots hitting pages that were never cached well.
AI companies crawl hard. So do SEO tools, scrapers, and badly behaved monitors. The fix is rarely panic. It is measurement.
Quick Answer
Check access logs and hosting metrics, identify which bots hit which paths, improve caching for public pages, and rate-limit or block only the offenders you can name. Do not take your whole site offline because one crawler got aggressive.
The mistake is upgrading hosting before fixing cache
Teams love to solve bot spikes by doubling the server.
Sometimes that is needed. Often the site is serving uncached HTML to bots on pages that rarely change.
Static articles, tool pages, and old blog posts should not crush PHP or database on every hit.
I would separate search bots from training crawlers
Not every bot deserves the same treatment.
You may want Googlebot and Bingbot healthy. You may want to limit aggressive AI training crawlers that add no traffic and no value.
Read the user-agent, read the path, read the frequency. Then decide.
Protect uptime like a business owner
If bot traffic threatens checkout, login, or paid tool usage, protect those paths first.
Public content can often be cached harder. Sensitive endpoints may need stricter limits.
Do not block the whole site because a crawler found your sitemap.
