How to Think Like an SEO Expert

If you want to become an expert you need to start thinking like one. People perceive you as an authority in your field not because you claim you are, but by listening to what you say or reading what you write. From my personal experience, the key seems to be the originality, usefulness and depth of what you have to share. Recently I was very honored to contribute to a link-building project. I wanted to share with you my idea, but more than that, in this blog I like to take extra time to explain the original thought process that helped me come up with the idea in the first place.
The Challenge
Toolbar PageRank was a very important factor in measuring the quality of a link for a long while. But Google has played so much with it that it can hardly be considered reliable these days. I like to see problems like these as challenges and opportunities, so I decided to look hard for alternatives. I know there are several other methods (like using the Yahoo backlink count, number of indexed pages, etc.) but I did not feel these directly reflected how the link was important to Google, or to any other specific search engine. Each search engine has its own evaluation criteria when it comes to links, so using metrics from one to measure another is not a reliable gauge in my opinion.
I knew the answer was out there, and I knew just where to look.
Research: Putting the pieces together
Especially beyond the beginner stage, you need to make it a habit to research and read as much as possible. For advanced SEO knowledge, my favorite sources of research material are search engine–related patents and information retrieval research papers. In order to avoid getting lost while reading such documents, I recommend first reading the excellent tutorials on linear algebra and Information Retrieval from Dr. Garcia. Bill Slawsky also has an excellent blog where he unearths useful patents and provides excellent commentary. In fact Bill unearthed a very relevant patent, which provided a valuable insight for my Toolbar PageRank challenge. It is called: Anchor tag indexing in a web crawler system. Here is part of Bill’s conclusion:

The information about crawling rates, and the possible role of PageRank along with frequency of changes in content, which could influence how frequently a page is crawled is also the most detailed that I can recall seeing in a patent from Google.

From the patent I learned that search engines (at least Google) define their crawler priorities based in part on how important they think the page is and how frequently the page is updated.
So here comes a logical conclusion. If Google (and maybe other search engines) use the real PageRank or importance of the page as a criterion to determine how frequently they should visit the page, then by studying how frequently they visit a page I can indirectly determine the real PageRank or importance of the page to the search engine. Bingo!
Digging Deeper: Your own research
If you want to dig deeper, and you definitely should, you must learn to find valuable sources yourself and come up with your own conclusions. I personally love Google’s patent search. You simply need to use keywords like “pagerank,” “yahoo,” “google,” “anchor text,” and so on; you can order them by date and select “Issued Patents” or “Patent Applications” to come up with a goldmine of information and topics. Many of the ideas are never implemented, but some are—and it is good to be a step ahead of the rest of your competitors. In order to find interesting research papers you can use Google Scholar and type in queries related to search engine research. The patent search usually returns more recent documents, however.
(Another rich source of research material is the bibliography and references of these research papers and patents. Sometimes I follow so many references that I lose track and focus, so be aware.)
Now, the real trick is how you go about reading this information—back to front. There’s no need to read the entire paper from start to finish to find something valuable. Check the title and if it is interesting enough, read the abstract. If the abstract is interesting then you can go straight to the conclusion and learn the most valuable information. Why? Because most papers concentrate on proving their conclusions, but since the authors have already taken great pains to do so, you don’t need to waste your time, too! Of course if you are curious about something you always have the option to go back to the explanation and see how the author came up with it.
Making Something Useful
Now, with my challenge identified, my research bearing valuable fruit, I just needed to put it all together to create a solution. As a technical guy, my job was now to figure out how to determine the crawl rate and update rate of the page. In reality, it didn’t take much research. Google stamps the cache pages with the time of the crawl/download and I was already familiar with the concept of “conditional HTTP GET”(this is explained in the article). It was just a matter of putting all the pieces together. I also included the indexing rate because it is useful in detecting duplicate content issues and hard-to-detect penalization problems, but in reality is not a requirement for the technique to work.
I hope you’ll read the full article to appreciate this new technique to evaluate links and pages.
Final Thoughts
Here’s a schematic to generalize the concepts explained here for your own challenges.

1. Identify a problem/challenge and note the major roadblocks.
2. Research and test alternative solutions to the problem. Each problem can be solved in many different ways. Some are better than others depending on what the goals and constraints are.
3. Put all the pieces of your research and testing together by making logical connections. It is incredible how simple ideas can turn into great ones by combining the right pieces of information.
4. Share and get peer feedback.
5. Adjust your idea based on constructive criticism.
6. Go back to step 1.