Specialist SEO Agency with clients in Oxford, across Oxfordshire & the Thames Valley
01865 306 889   [email protected]

How should my robots.txt file look?

SEO Knowledgebase > ,

A robots.txt file is a list of simple commands to Google search, so as such it looks fairly plain. It’s a txt document with lines of text with code and instructions. You’ll see commands like ‘Disallow: /pagenameexample’ or ‘Allow: /site.example’. These commands tell Googlebot what to crawl and what to ignore.
get in