How a Google Rank Tracker works behind the scenes

Any SERP tracker’s job is to google for keywords and “document” whatever results (websites/ranks) they find. And track “movements” over time.

Storing and presenting data in a useful manner is pretty straightforward. The tricky part is doing the actual search and getting accurate results. At an industrial level, this goes against Google and other search engines’ terms of service. Sooner or later you’ll face CAPTHAs, 503 Service Unavailable errors, or just inaccurate search results, which is the worst case scenario. To add to the woes – doing everything programmatically raises a few other issues (occasional HTML markup changes, IP/proxy fatigue, etc.) And the bad news isn’t over just yet. Google delivers personalized search results based on factors like the device you’re using, your location, search history, search settings, and many others that only they know.

So how does SEOlium solve all these challenges and track depersonalized search results? The short and simple answer is: we simulate a real user’s web activity. We ‘google’ keywords on your behalf using emulated browsers which are completely similar to the real ones. The only difference is there’s no man in front of the screen. As far as Google is concerned, there is a screen in a nearby location, there’s a keyboard and a mouse, there’s JavaScript, browser extensions, network turbulences, etc.

Let us get into more detail. SEOlium is probably the most accurate rank tracker there is. But Google delivers personalized search results to everyone. So by „accurate results” we mean – „depersonalized” search results that a random user would see under normal circumstances. As such, an „accurate rank tracker”, aka SEOlium, is the one that picks the most „average” search result set which Google would show to most users for a keyword or search query. To achieve that, here are a few strategies we employ.

To emulate the average web user without raising any suspicion from Google, we’re using different:

• Devices (laptop/desktop/smartphone/etc.)
• Operating systems (Windows, Android, iOS, Ubuntu, etc.)
• Screen resolutions
• Browsers (Chrome, Safari, Firefox, Opera, Edge, Android, etc.)
• Browser window sizes
• Browser extensions
• Browser User-Agent strings

This will occasionally lead to minor day to day rank fluctuations but it perfectly fits a normal use case scenario.

Logged-in users get their search results heavily personalized which is why we’re never doing a keyword search while logged into the Google account.

We want to avoid googling for keywords while having an extensive search history, which would trigger personalized SERPs. We regularly close & open browser sessions. And occasionally we clear the cache, delete all cookies, browser history, etc.

Searching from an incognito window is a normal use-case scenario which also helps to get depersonalized search results at the same time.

Appending the “&pws=0” parameter to any Google search URL further depersonalizes search results.

Here’s how a URL looks like https://www.google.com/search?q=seolium&pws=0

Otherwise, this is how search results look with personalization https://www.google.com/search?q=seolium

Glad you asked!

Tracking local keywords (think “burger chicago”, “car rental near me”, “east village hair salon”) is where it gets hotter and more complicated. In fact, we’d say this is where serious rank trackers are separated from… not so serious rank trackers.

For starters, accuracy for local search results is intended to be personalized, and this means Google relies heavily on the searcher’s location (mostly IP address, but other factors as well) for queries like “car dealer Houston”.

How we fix this is by using a set of extremely high-quality residential proxies covering any zip-code in the world. Sounds simple, right? Well, it’s not! And how we choose these residential proxies shall continue to be a secret. But this is key in enabling us to search from the most accurate location possible.

To fine-tune our results, we sometimes also employ other technologies, including using advanced Google search URL parameters for country/region and language.

Absolutely!

Like with any other business model, we face numerous challenges daily but two of them stand out:
– Google Search service prohibitions
– Search Engine Result Pages (SERP) HTML markup changes

No matter how organized we try to be, Google still outsmarts us sometimes and catches our IPs/machines. When this happens, we instantly drop those and let them rest for a while. The best bit though is we have a highly distributed infrastructure, which means any glitch only affects a small local part and never our service as a whole.

Google can be random! Occasionally, it will serve fresh markup and catch a lot of rank tracker tools off-guard. Luckily, SEOlium is never broken by this, and we’ll tell you why.

First, we have a system that detects whenever such significant changes occur. We immediately delegate to the appropriate HTML interpreter, especially when Google is only rotating the HTML markup.

But when the markup is completely new and unknown to us, our on-call engineer will shortly release a new HTML interpreter to resolve the issue.

Running a Google Rank Tracker is not easy, and there’s so much that goes on behind the scenes that it’s impossible to explain it in a single blog post. Nonetheless, we hope you now have a clearer picture of the rank tracking process – at least the SEOlium way!

From the SEOlium blog – https://www.seolium.com/how-a-rank-tracker-works/