# robots.txt # Each record contains lines of the form: # : # User-agent: The robot the record is describing. # Disallow: Specifies a full or partial URL that is not to be visited. # Any empty value, indicates that all URLs can be retrieved. # At least one Disallow field needs to be present in a record. User-agent: * Disallow: /contact Disallow: /exchange