Tag Archive: crawlers

March 26, 2015   /   by Marco   /   bots, crawlers, google, robots, seo

Best way to stop crawler bots

I recently discovered one of my VPS servers was running constantly at 1-2 CPU load and received notifications from the VPS provider that it was using more than my fair share of CPU resources and temporarily suspended the VPS to prevent it from impacting my neighbouring customers – which was fair enough. Upon investigation there were […]