Robots.txt file controls Web spiders
| decod-robots-check (1533) |  Low Risk |
Description:
The robots.txt file is commonly placed in the root directory of a system's Web server to control the actions of Web robots (often called crawlers or spiders). All robots that adhere to the Robots Exclusion Standard (see References) will check this file on your server before proceeding to index or search your site. A user who is able to modify the contents of the robots.txt file could control the actions of Web robots on your server.
Consequences:
Obtain Information
Remedy:
This is not a vulnerability. Administrators should review the contents of the robots.txt file to check if the information is consistent with the policies of their organization.
References:
- The Web Robots Pages Web site: A Standard for Robot Exclusion.
Platforms Affected:
- IETF HTTP/1.1
- Various vendors Any application
Reported:
Not available
The information within this database may change without notice. Use of this information constitutes acceptance for use in an AS IS condition. There are NO warranties, implied or otherwise, with regard to this information or its use. Any use of this information is at the user's risk. In no event shall the author/distributor (IBM Internet Security Systems X-Force) be held liable for any damages whatsoever arising out of or in connection with the use or spread of this information.
For corrections or additions please email xforce@iss.net