Robots.txt file commonly used writing sample

Robots.txt file usage example:

1. Allow all robot visit

Allow the user-agent: * : and/or the user-agent: * Disallow:

2. The ban on all access to any part of the web site search engine

The user-agent: *

Disallow:/

3. Only prohibit Baiduspider visit your website

The user-agent: Baiduspider

Disallow:/

4. Allow only Baiduspider visit your website

The user-agent: Baiduspider

Disallow:

5. Prohibit spiders to access specific directory

The user-agent: *

Disallow:/cgi - bin/

Disallow:/TMP/

Disallow:/- Joe/
The above is what we needed in seo writing