Robots.txt file controls Web spiders
| decod-robots-check (1533) |  Low Risk |
Description:
The robots.txt file is commonly placed in the root directory of a system's Web server to control the actions of Web robots (often called crawlers or spiders). All robots that adhere to the Robots Exclusion Standard (see References) will check this file on your server before proceeding to index or search your site. A user who is able to modify the contents of the robots.txt file could control the actions of Web robots on your server.
Platforms Affected:
- Various vendors, Any application
- Various vendors, HTTP
Remedy:
This is not a vulnerability. Administrators should review the contents of the robots.txt file to check if the information is consistent with the policies of their organization.
Consequences:
Obtain Information
References:
- The Web Robots Pages Web site, A Standard for Robot Exclusion at http://www.robotstxt.org/wc/norobots.html.
Reported:
Not available
The information within this database may change without notice. Use of this information constitutes acceptance for use in an AS IS condition. There are NO warranties, implied or otherwise, with regard to this information or its use. Any use of this information is at the user's risk. In no event shall the author/distributor (Internet Security Systems X-Force) be held liable for any damages whatsoever arising out of or in connection with the use or spread of this information.
Copyright (c) 1994-2008 Internet Security Systems, Inc. All rights reserved worldwide.
For corrections or additions please email xforce@iss.net