How to update robots.txt for WooCommerce

Crawlers are a massive part of digital marketing whether they are crawling for search engines looking to index and rank your page or a marketing platform providing you with data and technical information.

However, sometimes we don’t actually want pages to be crawled if we know that the page doesn’t actually bring any value to our overall website, in fact, if anything it brings the value of the site down but is still necessary for say user experience. Or maybe it’s that your site has a lot of pages that would waste your crawl budget and therefore the pages you want to rank aren’t even getting crawled by bots.

Robot.txt files allow us to block bots from unnecessarily crawling specific URLs.

Once you have installed the Yoast SEO you want to go into your WooCommerce sites backend dashboard, then navigate to the Yoast section in the left-hand menu and then the “tools” section within the Yoast Menu, as shown below. Once you’re in the “tools” section of Yoast you’ll want to click on the file editor

From here you’ll be met with your robot.txt file editor where you can edit and control what bots will and will not be able to crawl. But what is it that you’re specifically looking to blog? Is it a singular page, a file or the entire website itself? Next, we will look at how to edit the robot.txt file for a specific purpose.

So we have established how to access your robots.txt file, but now we are going to look at a small variety of inputs you may need for your WooCommerce website.

You may be looking to block a file or a folder from a bot to save you on your crawl budget, or because these aren’t in your opinion important in comparison to others on your site.

For example, if you were looking to block bots from crawling your wp-admin folder, you would use the command to the right.

This isn’t something we would advise doing as it would definitely mean that you could be missing out on organic traffic opportunities.

But let’s say for whatever reason you were looking to block a specific search engine bot from your site, like Bing’s search engine bot you would simply update your robot.txt file as shown in the image on the left-hand side.

If your WooCommerce website is already live it is highly unlikely that you would want or need to use this input, however if you’re WooCommerce site is in development and you don’t want bots to crawl it at all, then you can use this input.

Finally, we will show you how to allow all bots to crawl your site, say you’ve made a mistake or that you blocked a bot for a specific reason and now you’re ready to allow them to crawl at will. You would simply type the input to the left.

Also in this series:

Related posts:

Storytime. You are creating a game. You just made this beautiful gif of clouds and you want to add it to your background canvas in the game. How do you do this? In normal JavaScript, you would be…

When the winds blow wet and warm. When gray winter retreats from blooming-green-becoming. When new leaves dance like dyads in the rain. When the passion of the equinox stirs the blood and the sap…

As an international student, I’ve often wondered about my fellow foreigners here in Australia. Who are they? Why choose to study here? Far more questions than answers. Yet the presence of…