How to check your robots.txt file
Open your web browser and enter www.yourdomain.com/robots.txt to view the contents of your robots txt file.
Do You Know
1. There are only two official commands for the robots.txt file: User-agent and Disallow.
2. Start with the user-agent line and then add the disallow commands:
User-agent: *
Disallow: /cgi-bin/
3. Use an extra Disallow line for every directory:
User-agent: *
Disallow: /support
Disallow: /cgi-bin/
Disallow: /images/
4. The casing of your directories matters. If the name of the directoy you want to disallow is “Optimize”, don’t write “optimize” in the robots.txt file.
Leave a Reply