Does your business need
to re-invent its marketing because of COVID-19?


We go through everything a small business owner needs to do BEFORE choosing an SEO Company.


Include a Link to Your Sitemap in Robots.txt for Better Deep Page Indexing
Posted by Cameron Francis on December 07, 2016 in

Include a Link to Your Sitemap in Robots.txt for Better Deep Page Indexing

Are you having trouble getting your website’s inner pages indexed? There are two main reasons for this:

  1. You don’t have a sitemap; or
  2. Your sitemap is not indexed in your robots.txt file

In case you didn’t know, the robots.txt file is the one that tells the search engines which webpages to index and which ones to exclude.

Thus, it allows the search engine bots to access to your sitemap with a few commands that will ensure your deeper web pages get indexed.

Furthermore, having a robots.txt file helps eliminate unnecessary 404 errors.

So, if your website doesn’t have a sitemap, then you need to create one right now and give the search engine bots access to it.

You can generate your sitemap by using either:

  • Standard Text Format
  • XML Format

Whichever format you choose, make sure that you have a comprehensive and updated list without any broken lines of code.

This is particularly important for the XML sitemap. In fact, if you’ll be using the XML format, it’s best to generate the file using a sitemap generator.

If you need help with this, then let the experts at eTraffic set up your sitemap for you. We guarantee that your inner pages will all get indexed. We will also carry out a free website audit to ensure that everything is working perfectly.


author Cameron Francis

designation Co-Founder

Cameron Francis is the CoFounder & Managing Director of ETRAFFIC, Melbourne's #1 Creative & Digital Marketing Agency. For over a decade Cameron has been providing growth marketing and digital strategies for over 2,000 small businesses, start-ups & entrepreneurs around the country.
  • facebook
  • Twitter
  • linkedin
  • Youtube
  • Google Plus
  • Pinterest
  • Instagram
  • Google Partner

About Cameron Francis

Cameron Francis is the co-founder of The eTraffic Group which started as a 2 man start-up from his lounge room, to a 50 person+, multi million dollar company.

For over 10 years Cameron has been providing growth hacking and digital marketing strategies for small businesses, start ups & entrepreneurs. He helps companies like Retreat Caravans, Melbourne Real Estate, and Roth Newton grow their revenue.

Testimonials Testimonials

"Cam has increased our website traffic and leads by 600%"

Jamie Brookes of
Testimonials Btn
Hire me to speak

Cam Francis is available to speak at your corporate event or conference. He brings an energetic and high impact approach to provide the audience with instantly actionable digital marketing strategies guaranteed to improve your business.