While my wordpress website was in production, I created a robots.txt file to disallow everything. When the site was read