Case Study: How the Cookie Monster Ate 22% of Our Visibility

Last yr, the team at Homeday — just one of the primary residence tech corporations in Germany — designed the choice to migrate to a new written content management system (CMS). The targets of the migration have been, between other factors, greater webpage velocity and creating a point out-of-the-art, long term-proof site with all the needed options. Just one of the main motivators for the migration was to enable content material editors to perform a lot more freely in producing webpages without having the aid of builders. 

Following assessing various CMS options, we made a decision on Contentful for its contemporary technologies stack, with a exceptional experience for equally editors and builders. From a complex viewpoint, Contentful, as a headless CMS, lets us to decide on which rendering tactic we want to use. 

We’re currently carrying out the migration in quite a few stages, or waves, to reduce the chance of problems that have a significant-scale detrimental influence. During the 1st wave, we encountered an challenge with our cookie consent, which led to a visibility loss of practically 22% within just five days. In this article I will describe the difficulties we ended up dealing with all through this 1st migration wave and how we fixed them.

Placing up the first exam-wave 

For the 1st check-wave we selected 10 Seo webpages with superior website traffic but minimal conversion costs. We proven an infrastructure for reporting and checking all those 10 internet pages: 

• Rank-tracking for most related keywords 

• Website positioning dashboard (DataStudio, Moz Professional,  SEMRush, Look for Console, Google Analytics)

• Normal crawls 

After a thorough scheduling and screening section, we migrated the to start with 10 Website positioning webpages to the new CMS in December 2021. Despite the fact that various problems happened during the testing stage (enhanced loading moments, even bigger HTML Doc Item Product, and so on.) we made the decision to go dwell as we didn’t see major blocker and we preferred to migrate the initially testwave just before christmas. 

Initial performance evaluation

Extremely psyched about attaining the 1st move of the migration, we took a search at the effectiveness of the migrated pages on the subsequent day. 

What we observed next genuinely didn’t please us. 

Overnight, the visibility of tracked key phrases for the migrated webpages minimized from 62.35% to 53.59% — we dropped 8.76% of visibility in 1 day! 

As a result of this steep drop in rankings, we done a further extensive spherical of testing. Amongst other matters we analyzed for coverage/ indexing problems, if all meta tags were bundled, structured information, inside hyperlinks, site pace and cellular friendliness.

Next effectiveness assessment

All the article content had a cache day after the migration and the written content was totally indexed and staying read by Google. What’s more, we could exclude various migration risk components (adjust of URLs, written content, meta tags, structure, and so forth.) as sources of mistake, as there hasn’t been any adjustments.

Visibility of our tracked key terms endured an additional drop to 40.60% in excess of the subsequent couple of days, creating it a whole fall of pretty much 22% inside five times. This was also evidently shown in comparison to the levels of competition of the tracked keywords (in this article “estimated website traffic”), but the visibility appeared analogous. 

As other migration hazard factors in addition Google updates had been excluded as resources of glitches, it definitely had to be a complex issue. Far too considerably JavaScript, low Main Web Vitals scores, or a larger, extra advanced Document Item Model (DOM) could all be possible causes. The DOM signifies a site as objects and nodes so that programming languages like JavaScript can interact with the webpage and transform for illustration design and style, composition and content material.

Adhering to the cookie crumbs

We experienced to detect concerns as promptly as possible and do fast bug-correcting and reduce additional damaging effects and site visitors drops. We last but not least got the 1st genuine hint of which technological rationale could be the bring about when one particular of our tools confirmed us that the selection of internet pages with high exterior linking, as well as the number of internet pages with highest material dimensions, went up. It is critical that web pages you should not exceed the greatest written content dimension as pages with a pretty massive total of physique content material may well not be absolutely indexed. With regards to the large exterior linking it is critical that all exterior hyperlinks are reputable and applicable for people. It was suspicious that the range of exterior hyperlinks went up just like this.

Both equally metrics have been disproportionately substantial as opposed to the range of internet pages we migrated. But why?

Right after checking which exterior one-way links experienced been included to the migrated pages, we observed that Google was reading through and indexing the cookie consent kind for all migrated web pages. We performed a web page lookup, examining for the content material of the cookie consent, and saw our theory verified: 

This led to quite a few difficulties: 

1. There was tons of duplicated information made for each page owing to indexing the cookie consent kind. 

2. The articles dimensions of the migrated webpages dramatically improved. This is a dilemma as internet pages with a very large total of body written content may possibly not be completely indexed. 

3. The range of external outgoing inbound links considerably enhanced. 

4. Our snippets out of the blue confirmed a date on the SERPs. This would recommend a website or information report, when most content articles on Homeday are evergreen content. In addition, owing to the date appearing, the meta description was lower off. 

But why was this taking place? In accordance to our support supplier, Cookiebot, research motor crawlers entry web-sites simulating a full consent. For this reason, they gain access to all information and copy from the cookie consent banners are not indexed by the crawler. 

So why was not this the case for the migrated webpages? We crawled and rendered the webpages with diverse user agents, but continue to could not locate a trace of the Cookiebot in the source code. 

Investigating Google DOMs and hunting for a answer

The migrated internet pages are rendered with dynamic data that will come from Contentful and plugins. The plugins incorporate just JavaScript code, and sometimes they come from a lover. A person of these plugins was the cookie manager partner, which fetches the cookie consent HTML from outside our code base. That is why we failed to locate a trace of the cookie consent HTML code in the HTML supply files in the first put. We did see a larger sized DOM but traced that back to Nuxt’s default, a lot more complex, much larger DOM. Nuxt is a JavaScript framework that we do the job with.

To validate that Google was looking through the duplicate from the cookie consent banner, we made use of the URL inspection resource of Google Look for Console. We compared the DOM of a migrated website page with the DOM of a non-migrated page. In just the DOM of a migrated website page, we at last found the cookie consent articles:

A thing else that bought our interest ended up the JavaScript documents loaded on our outdated pages vs . the files loaded on our migrated pages. Our web site has two scripts for the cookie consent banner, supplied by a 3rd social gathering: one to present the banner and grab the consent (uc) and a person that imports the banner material (cd).

• The only script loaded on our outdated internet pages was uc.js, which is dependable for the cookie consent banner. It is the just one script we require in each individual webpage to tackle consumer consent. It shows the cookie consent banner devoid of indexing the material and saves the user’s conclusion (if they agree or disagree to the use of cookies).

• For the migrated webpages, apart from uc.js, there was also a cd.js file loading. If we have a site, where we want to display extra data about our cookies to the consumer and index the cookie facts, then we have to use the cd.js. We believed that each documents are dependent on every single other, which is not suitable. The uc.js can operate alone. The cd.js file was the motive why the content of the cookie banner acquired rendered and indexed.

It took a though to locate it for the reason that we assumed the next file was just a pre-prerequisite for the initial one particular. We determined that only getting rid of the loaded cd.js file would be the option.

General performance assessment right after applying the remedy

The working day we deleted the file, our search term visibility was at 41.70%, which was even now 21% decreased than pre-migration. 

Having said that, the working day right after deleting the file, our visibility improved to 50.77%, and the next working day it was pretty much again to regular at 60.11%. The believed traffic behaved equally. What a reduction! 

Conclusion

I can visualize that numerous SEOs have dealt with very small concerns like this. It appears trivial, but led to a considerable drop in visibility and site visitors for the duration of the migration. This is why I propose migrating in waves and blocking more than enough time for investigating complex glitches prior to and just after the migration. Also, keeping a close appear at the site’s general performance within just the weeks after the migration is important. These are certainly my crucial takeaways from this migration wave. We just completed the second migration wave in the beginning of May perhaps 2022 and I can state that so considerably no important bugs appeared. We’ll have two far more waves and total the migration ideally efficiently by the finish of June 2022.

The performance of the migrated web pages is just about back again to regular now, and we will carry on with the following wave.