Case Study: How the Cookie Monster Ate 22% of Our Visibility

Previous calendar year, the team at Homeday — just one of the primary house tech firms in Germany — made the choice to migrate to a new content material management technique (CMS). The ambitions of the migration were, among the other points, elevated webpage speed and producing a state-of-the-artwork, long term-evidence website with all the necessary features. Just one of the primary motivators for the migration was to permit material editors to do the job far more freely in building pages with no the enable of builders. 

Soon after analyzing many CMS options, we determined on Contentful for its fashionable technology stack, with a top-quality working experience for both editors and builders. From a specialized viewpoint, Contentful, as a headless CMS, permits us to pick out which rendering method we want to use. 

We’re currently carrying out the migration in many stages, or waves, to decrease the possibility of challenges that have a massive-scale unfavorable effect. Through the initially wave, we encountered an problem with our cookie consent, which led to a visibility reduction of nearly 22% in just five days. In this posting I’ll describe the difficulties we were facing for the duration of this to start with migration wave and how we settled them.

Setting up the very first test-wave 

For the very first test-wave we chose 10 Search engine marketing web pages with higher targeted visitors but reduced conversion costs. We founded an infrastructure for reporting and monitoring all those 10 internet pages: 

• Rank-monitoring for most suitable keywords 

• Search engine optimisation dashboard (DataStudio, Moz Professional,  SEMRush, Lookup Console, Google Analytics)

• Typical crawls 

Immediately after a extensive organizing and testing section, we migrated the 1st 10 Seo web pages to the new CMS in December 2021. While many difficulties happened for the duration of the testing phase (increased loading occasions, even bigger HTML Doc Item Design, and so on.) we determined to go stay as we did not see large blocker and we wanted to migrate the to start with testwave in advance of xmas. 

Initially general performance critique

Pretty thrilled about reaching the to start with action of the migration, we took a glance at the overall performance of the migrated internet pages on the future day. 

What we observed subsequent actually did not remember to us. 

Right away, the visibility of tracked keyword phrases for the migrated web pages decreased from 62.35% to 53.59% — we lost 8.76% of visibility in one day! 

As a end result of this steep fall in rankings, we conducted a different in depth spherical of testing. Among the other items we examined for coverage/ indexing concerns, if all meta tags were being provided, structured information, internal back links, web site pace and cell friendliness.

Next general performance assessment

All the posts experienced a cache day after the migration and the content material was completely indexed and currently being read by Google. Furthermore, we could exclude several migration hazard elements (alter of URLs, articles, meta tags, structure, and many others.) as resources of error, as there has not been any alterations.

Visibility of our tracked keyword phrases endured a further drop to 40.60% above the subsequent several times, making it a full fall of nearly 22% inside five days. This was also clearly shown in comparison to the competitors of the tracked keyword phrases (here “approximated targeted visitors”), but the visibility seemed analogous. 

As other migration threat components plus Google updates experienced been excluded as resources of problems, it undoubtedly had to be a technical concern. As well a lot JavaScript, low Main World wide web Vitals scores, or a much larger, more complicated Document Object Product (DOM) could all be potential results in. The DOM signifies a webpage as objects and nodes so that programming languages like JavaScript can interact with the web site and modify for case in point model, structure and articles.

Subsequent the cookie crumbs

We experienced to recognize problems as swiftly as doable and do brief bug-fixing and minimize extra detrimental results and traffic drops. We eventually obtained the to start with serious hint of which complex motive could be the trigger when a person of our applications showed us that the amount of webpages with high external linking, as properly as the amount of pages with optimum content measurement, went up. It is important that web pages never exceed the greatest articles size as web pages with a really huge total of entire body material may well not be thoroughly indexed. About the significant exterior linking it is significant that all external back links are trusted and applicable for end users. It was suspicious that the amount of external back links went up just like this.

Both metrics were being disproportionately large in comparison to the amount of pages we migrated. But why?

Just after checking which exterior backlinks had been additional to the migrated pages, we noticed that Google was reading and indexing the cookie consent variety for all migrated webpages. We executed a web-site research, examining for the content of the cookie consent, and observed our concept verified: 

This led to a number of complications: 

1. There was tons of duplicated content designed for every web page thanks to indexing the cookie consent type. 

2. The articles dimension of the migrated web pages dramatically elevated. This is a challenge as internet pages with a extremely large volume of overall body content might not be absolutely indexed. 

3. The range of exterior outgoing links substantially elevated. 

4. Our snippets all of a sudden showed a date on the SERPs. This would counsel a blog or information post, while most content on Homeday are evergreen written content. In addition, owing to the day showing, the meta description was slice off. 

But why was this occurring? In accordance to our company company, Cookiebot, lookup motor crawlers obtain internet websites simulating a entire consent. As a result, they gain access to all material and duplicate from the cookie consent banners are not indexed by the crawler. 

So why was not this the circumstance for the migrated pages? We crawled and rendered the webpages with diverse user brokers, but continue to could not discover a trace of the Cookiebot in the supply code. 

Investigating Google DOMs and hunting for a solution

The migrated pages are rendered with dynamic information that will come from Contentful and plugins. The plugins consist of just JavaScript code, and occasionally they occur from a companion. 1 of these plugins was the cookie manager companion, which fetches the cookie consent HTML from exterior our code foundation. That is why we failed to locate a trace of the cookie consent HTML code in the HTML supply files in the 1st location. We did see a more substantial DOM but traced that back again to Nuxt’s default, more complicated, larger DOM. Nuxt is a JavaScript framework that we work with.

To validate that Google was looking at the copy from the cookie consent banner, we utilised the URL inspection resource of Google Look for Console. We as opposed the DOM of a migrated page with the DOM of a non-migrated web page. Within just the DOM of a migrated webpage, we last but not least located the cookie consent material:

A little something else that got our awareness have been the JavaScript information loaded on our aged webpages as opposed to the files loaded on our migrated pages. Our website has two scripts for the cookie consent banner, provided by a 3rd bash: a person to demonstrate the banner and get the consent (uc) and a single that imports the banner information (cd).

• The only script loaded on our old pages was uc.js, which is responsible for the cookie consent banner. It is the a person script we need in every webpage to manage person consent. It shows the cookie consent banner without indexing the information and will save the user’s selection (if they agree or disagree to the use of cookies).

• For the migrated webpages, apart from uc.js, there was also a cd.js file loading. If we have a webpage, in which we want to exhibit extra details about our cookies to the user and index the cookie data, then we have to use the cd.js. We believed that both files are dependent on just about every other, which is not proper. The uc.js can operate by itself. The cd.js file was the rationale why the articles of the cookie banner acquired rendered and indexed.

It took a when to come across it since we believed the next file was just a pre-need for the initially 1. We established that merely getting rid of the loaded cd.js file would be the remedy.

Performance review soon after implementing the alternative

The day we deleted the file, our search term visibility was at 41.70%, which was still 21% decrease than pre-migration. 

However, the working day just after deleting the file, our visibility elevated to 50.77%, and the upcoming working day it was virtually back again to ordinary at 60.11%. The estimated traffic behaved equally. What a reduction! 

Conclusion

I can visualize that numerous SEOs have dealt with tiny problems like this. It appears trivial, but led to a major fall in visibility and website traffic throughout the migration. This is why I propose migrating in waves and blocking ample time for investigating technical errors right before and after the migration. Additionally, preserving a near seem at the site’s performance within the months right after the migration is essential. These are surely my crucial takeaways from this migration wave. We just completed the next migration wave in the commencing of May 2022 and I can point out that so far no important bugs appeared. We’ll have two far more waves and finish the migration hopefully productively by the stop of June 2022.

The general performance of the migrated web pages is practically back to typical now, and we will proceed with the next wave.