How should my robots.txt file look?
A robots.txt file is a list of simple commands to Google search, so as such it looks fairly plain. It’s a txt document with lines of text with code and instructions. You’ll see commands like ‘Disallow: /pagenameexample’ or ‘Allow: /site.example’. These commands tell Googlebot what to crawl and what to ignore.