Measurement Update 4.5.1: More Stable Metrics, Faster Tests

Last week, changes in AWS infrastructure caused temporary instability in measurement results, affecting our Watchdog and its notifications.

In response to these unexpected changes, we focused on ensuring more stable results and took the opportunity to implement updates that had been on our radar for some time.

What exactly changed, and how might it appear in your charts? The changes are most noticeable in the FCP metric and also in LCP, particularly on mobile devices:

1. From Saturday, September 21st, you might have noticed deteriorated and unstable values in the FCP and sometimes LCP metrics, stemming from AWS test instability. This change is marked as “AWS Changes (Unstable Results)” in the automatic note we send out to everyone.
2. From Friday, September 27th, you could see more stable results. Various tests at different times returned similar results. Consequently, the FCP and LCP metric data will be slightly worse than before.
3. From Tuesday, October 1st, we've accelerated the test machine settings to better align with current connection speeds in Central Europe, specifically reducing RTT from 150 to 100 ms and slightly speeding up the test machine's connection. This will result in a subsequent improvement in metric values. In the automatic note, this is marked as “Measurement Change (Faster Tests)”.

Between September 21st and October 2nd, we disabled automatic notifications for these two metrics, as the vast majority of Monitoring PLUS clients would have received falsely positive messages about speed deterioration.

The new values for the FCP and LCP metrics (among others) obtained on mobile devices will be slightly different from the state before September 21st.

We're continuing to refine measurements and new features that will enable you to have faster websites.