Finding a better perceptual metric than FPS

In a highly framerate-heterogenous 3D environment, jitter seems more important than average FPS

At Second Life, we’re recognizing that FPS isn’t a very good match to subjective quality-of-experience, for several reasons:

• We have an unusually high diversity of GPU’s amongst the population of people using Second Life, ranging from Intel graphics to Nvidia 4090s and everything in-between.

• Because the content is all user-created, scenes can range widely in complexity, with average framerates varying wildly even on high-performance hardware.

• Rapid changes to performance can occur during an experience, for example when lots of complex avatars show up to an event.

For these reasons, we are testing new potential metrics that we hope will give us a better indication whether we improved or worsened subjective experience when we make a software update.

Jitter Explorer

‘Jitter’ refers to the difference between when a frame is displayed and when it was expected to be displayed. For example, a game running at 60FPS is supposed to display a new frame every 16 milliseconds. But in reality it sometimes is late… once in while the computer might be busy and the next frame might appear 25 milliseconds after the one before it, meaning there is a ‘jitter’ of 9 milliseconds.

Even when the average framerate is high, jitter is very noticeable. This is because 3D environments present us with lifelike scenes in which there are moving objects. Your brain is incredibly good at predicting where moving objects are going, and when a frame shows up late, it means that moving objects appear in a different place from where you predicted them. This cause a jolt of ‘surprise’ in your brain, which is unpleasant and takes you out of immersion.

To explore your sensitivity to this phenomena, try this Jitter Explorer. It shows you a dot moving in a circle, and lets you add one or more frames of delay at one or more points in the rotation. Start by adding a one-frame delay at the first frame…. you will probably easily be able to see it! If you have a really fast display (120FPS or higher), you may have to add more delay for it to be visible. You can check the “Show Delay” box to briefly color the delayed frames red, which will help you see what you are looking for.

Normalized Jitter

To measure jitter in a way that can be used across many different sessions to try to approximate subjective Quality-of-Experience, we tried summing the time differences between each frame and it’s prior one, and then dividing by the total time elapsed. As a formula, this looks like:

\(NormalizedJitter = \sum_{frames}^{n}abs(frametime_{n}-frametime_{n-1}) / \sum_{frames}^{n}frametime_{n}\)

We also tried something we named normalized frametime variance (NFTV), using a slightly different formula:

\(NFTV = stdev(frametime)/mean(frametime)\)

The first statistic has the desirable property that it is sensitive to the ordering of the delayed frames, with frames that are delayed but right next to each adding less to the metric. This seems to be a better fit to the subjective experience, because we seem to notice each glitch more than the duration of the glitch.

Trying these out in Second Life

If you download this project viewer (scroll to bottom for binaries), you can press Ctrl+Shift+S and look for “Normalized Period Jitter” and “normalized frametime variation” metrics just under jitter in the statistics window. Teleport around to different experiences and see which statistic you think is a better match for your experience, and feel free to share what number(s) you think are the edge between “Good” and “Bad” for you.