Getting different results of the same page in the same test run.

[chapura]

Hello!
I am using a puppeteer script for my lighthouse tests. The problem is that when I measure the same page twice in the same test run I get different results. Second measurement is always faster. Seems like the page was cached or something like that. Also, I am using a --disable-storage-reset=true flag in my browser setting, which as I understand should disable cache. What am I doing wrong?

[adamraine]

Also, I am using a --disable-storage-reset=true flag in my browser setting, which as I understand should disable cache.

This flag does the opposite actually. Lighthouse will clear the cache at the start of the run by default, and this flag will disable that behavior. In your case the second run could be pulling resources from the cache.

[chapura]

Well, I also tried to run my tests without it, but see no difference in results. Here are the flags that I use in my headless browser.
'--allow-no-sandbox-job',
'--allow-sandbox-debugging',
'--no-sandbox',
'--disable-gpu',
'--disable-gpu-sandbox',
'--display',
'--ignore-certificate-errors',
'--disable-dev-shm-usage',

[adamraine]

Can you share the Puppeteer script you are using to test? Please also read our variability docs in case this is just normal variance.

[chapura]

It will take some time to do that. But for now it would be great just to understand next situation : opening one and the same page couple of times in the same test run, each new measurement gives better results. Whether I use flags or not. So if I open same page 3 times in a row I get next total scores 52, 77,81. But the thing is, if I restart my browser before every new opening I get 52, 52, 52. Аnd as I mentioned before using flags don't make much of a difference. Seems like some static is sent from page to page and this is exactly what I am trying to get rid off. Please shed some light on this if possible.

Thanks Adam, really appreciate your help!

[AxelGros]

Yeah that seems to be expected behavior from Lighthouse. The flag that you mentioned clears cache only before the run, not during it.
So if your task is to measure the same page several times, then separate launches are the solution. However, restarting the browser in the same run will do its thing too.