Specialist SEO Agency with clients in Oxford, across Oxfordshire & the Thames Valley
SEO Video Training Course  |  SEO Knowledgebase

How should my robots.txt file look?

SEO Knowledgebase > ,

A robots.txt file is a list of simple commands to Google search, so as such it looks fairly plain. It’s a txt document with lines of text with code and instructions. You’ll see commands like ‘Disallow: /pagenameexample’ or ‘Allow: /site.example’. These commands tell Googlebot what to crawl and what to ignore.
get in
touch