Advanced Robots TXT Generator

A/D User agent Directory or File

See how to use Advanced Robots TXT Generator

Robots Txt Generator (by Ann Smarty) is an easy-to-use tool to create proper Robots.txt directives for your site: Easily copy and tweak Robots.txt files from other sites or create your own

When search engine spiders crawl a website, they typically start by identifying a robots.txt file at the root domain level. Upon identification, the crawler reads the file’s directives to identify directories and files that may be blocked. Blocked filed can be created with the robots.txt generator; these files are, in some ways, the opposite of those in a website’s sitemap, which typically includes pages to be included when a search engine crawls a website.

Our Robots.txt Generator

Creating an edited/new robots.txt file for a website is easy when employing a robots.txt generator.

  • To upload an existing file and pre-populate the generator tool, type or copy/paste in the root domain’s URL in the provided text box and select ‘Upload’.
  • To customize the generated robots.txt file, use the ‘Allow’ and ‘Disallow’ functions. Please note that the tool defaults to ‘Allow’.
  • To apply a new directive to your list, click ‘Add Directive’.
  • To remove or edit an existing directive, select ‘Remove Directive’, then create a new one using the ‘Add Directive’ function.

Customize User Agent Directives

Our robots.txt generator allows for the specification of several different search engines, including Google.

  • To specify alternative directives for specific search engine crawler(s), select the ‘User Agent’ box, showing as (*) by default, to specify the bot.
  • Select ‘Add Directive’, to add the custom section to the list with the generic directive as part of the new custom directive.
  • To revise a generic disallow directive to reflect an allow directive for a custom user agent, create a new allow directive to accommodate the specific user agent for the content. The matching disallow directive is then removed for the custom user agent.
  • To add a link to an XML-based sitemap file, copy/paste or type in the full sitemap file’s full URL into the specified text box, then select ‘Update’ to add this command to the robots.txt file list.
  • When complete, select ‘Export’, in order to save the new robots.txt file, then use FTP to upload the robots.txt to the website’s domain root. Once uploaded, the robtos.txt will dictate to Google and other search engines which pages, folders, and directories or the website to display and not display when a search engine user enters a query.

The Ultimate Guide to Blocking Your Content in Search is a great resource for those interested in learning more about robots.txt files and directives.