Tutorial: How to Use Unavailable_After Tags to Increase SEO Crawling Efficiency

Table of Contents

Introduction to Unavailable_After

Practical Uses

Implementing “Unavailable_After” Using the RankSense SEO Tool

Using the RankSense SEO tool to verify your changes

Introduction to Unavailable_After

Put simply, unavailable_after tags send a signal to search engines letting them know that the contents of a page should not be crawled after a certain date. Unlike “noindex”  tags which are directives, the “unavailable_after” tag is a hint instead that could technically be ignored. However, if it is followed, this means that a web crawler such as Googlebot will not crawl the page as frequently, allowing the more important pages to take up the majority of the site’s crawl budget. Since the crawler could still visit the page occasionally, the page could remain in the index. 

It is important to note that unavailable after tags should not be used for evergreen pages. Just because the content of a page is not changing, does not mean that it should not be crawled regularly. This tag should only be used for pages that are not important anymore and should not be ranking in the search results. 

In this tutorial, we will go over when to use unavailable_after tags, how to implement them using the RankSense Cloudflare app, and how to track the efficiency of the tags with RankSense.

Practical Uses

Suppose you have an events page on your website, linking to a different event page every week. Perhaps you post job listings that are only available for a month. Pages like these, showcasing a limited-time event, are great examples of pages that you would not want Google to crawl frequently after a certain date.  

In these cases, implementing an unavailable_after tag would be the perfect solution as the page will not be crawled as often and could remain in the index for anyone who happens to search for the exact content on the page. (Note that these pages should be moved to an “archived” part of the website so that they are not on any main pages as they are no longer important.)

Implementing “Unavailable_After” Using the RankSense SEO Tool

Step 1: Installation and rule sheets

If you do not have the RankSense app installed, you can do so by following the steps in this article. If you are not familiar with creating custom “rules” or SEO changes using the RankSense app, I recommend that you read this article. 

Step 2: Filling out your rule sheet

Unavailable_after tags are implemented on the “Meta Robots Tags” rule sheet. On this Google sheet, you will see a column for the URL, the robots tag, and comments.

In the “URL” column, simply paste the URL for the page that you would like to implement the unavailable_after tag for. In the “robots” column we will specify the “unavailable_after” tag and the exact date for the tag to take action. 

These are the ways the date can be formatted:

• year-month-date
• year/month/date
• month-date-year
• month/date/year

Specifying the exact time and timezone is optional. If the user specifies a time but does not list a timezone, the rule will default to the user’s timezone. If the user does not specify a time or a timezone, the default time will be used.

Here are some examples of ways to format the unavailable_after tag rule:

Once the rule sheet is complete, navigate to “Settings>SEO rules” on the RankSense Dashboard and click the “+ File” button to add your rule sheet. Paste the URL of your Google sheet and add the appropriate tags.

(Note: We recommended that users publish to staging first to make sure that the changes are applied properly)

Using the RankSense SEO tool to verify your changes

To verify whether the unavailable_after tags are working as intended, simply open up the RankSense Cloudflare app and scroll to the bottom of the “Dashboard” tab until you see the “Crawler statistics” section. The crawler statistics provide you with information regarding every URL that Google Bot, or your search engine of preference, has recently crawled on your page.  

Under “Latest pages crawled”, you can see the URL path and the “Last crawl” of that page. After the date that you set the unavailable_after tag for has passed, which can be checked by using our RankSense Chrome Extension, the page should not be crawled as frequently. This will let you know the tags are working as intended. 

You can even use the search tool to filter for specific URLs to check their last crawl date. You can filter these URLs by selecting the tags you applied to them when importing the rules. This way you do not need to go through all of the pages to find the ones with the unavailable_after tags.

We learned that:

• Unavailable_after tags are used to send a signal to search engines to let them know which pages are no longer important and should therefore not be crawled often.
• The web crawler may still visit the page occasionally.
• Limited time event pages are great examples of pages that should contain an unavailable_after tag once they are no longer important.
• You can apply unavailable_after tags to pages by filling out a rule sheet and importing it through the RankSense Cloudflare app.
• It is easy to verify your changes are working by using the “Crawler statistics” section of the RankSense Dashboard.