\chapter{Starting point for design}
Traditionally, measuring of video playback performance has focused on the image quality. This was reasonable with analog systems, where the primary problems are noise and distortion. Digital video systems have altogether different failure modes: it is far more common to encounter gaps and jitter than for the actual image quality to be degraded.

\section{Background on video playback performance}
The term \emph{video playback performance} is used here for the aspects that affect the video reproduction quality and happen during the playback, as opposed to those that happen during the video production. Figure \ref{fig_videoprocess} shows the typical path from video production to video playback, and the part of the process that this thesis focuses on.


\begin{figure}[h]
\centering
\begin{tikzpicture}
\tikzstyle{block} = [draw, align=center];
\tikzstyle{result} = [align=center];

\node [block] (scene)
{ Original\\scene };

\node [block] (camera) [right = 0.5cm] at (scene.east)
{ Video\\camera };

\node [block] (compression) [right = 0.5cm] at (camera.east)
{ Compression\\algorithm };

\node [result] (file) [right = 0.5cm] at (compression.east)
{ Video file };

\node [block, fill=lightgray] (decompression) [right = 0.5cm] at (file.east)
{ Decompression\\algorithm };

\node [block, fill=lightgray] (display) [right = 0.5cm] at (decompression.east)
{ Video\\display };

\draw [->,thick] (scene.east)       -- (camera.west);
\draw [->,thick] (camera.east)      -- (compression.west);
\draw [->,thick] (compression.east) -- (file.west);
\draw [->,thick] (file.east)        -- (decompression.west);
\draw [->,thick] (decompression.east) -- (display.west);

\tikzstyle{brace}=[decorate, decoration={brace,amplitude=2mm,raise=2mm}]
\draw [brace] (scene.north west) -- (compression.north east)
    node [midway, above=5mm] {Video production};
\draw [brace] (decompression.north west) -- (display.north east)
    node [midway, above=5mm] {Video playback};
\end{tikzpicture}

\caption{A typical process of producing and displaying digital video. This thesis is concerned with measuring the performance of the video playback portion of the process.}
\label{fig_videoprocess}
\end{figure}

The overall user experience depends on a wide variety of factors, as illustrated in Figure \ref{fig_mindmap}, of which the playback performance is one important part. For detailed discussion and examples, see \cite{mm_quality}-\cite{movie}. Leaving aside the audio portion and using a strict definition of "video" as only the moving image, the video playback performance consists of two aspects:

\begin{description}
\item[Image quality] is the visual aspects of each individual video frame as shown on the screen. It depends on things such as the display resolution, color reproduction and decompression accuracy.
\item[Motion quality] is the dynamic behavior of video frame changes. In the most basic case, it means how accurate the frame change time is. Further considerations are how fast the change occurs, whether the whole display changes in unison and whether all the video frames are actually displayed.
\end{description}

Of these, image quality can be roughly defined as the apparent clarity of the image, while motion quality can be defined as the apparent smoothness of motion. In the end, the perceived playback quality is always a complex combination of video quality, audio quality, content material and personal characteristics of the viewer \cite{mm_quality}.

\begin{figure}[tbp]
\centering
\begin{tikzpicture}[mindmap,
    level 1 concept/.append style={sibling angle=60, level distance=4.4cm, minimum size=2.5cm, font=\normalsize},
    level 2 concept/.append style={sibling angle=60, level distance=3.2cm, font=\small, minimum size=2cm},
    level 3 concept/.append style={sibling angle=60, level distance=2.6cm, font=\footnotesize, minimum size=2cm},
    decoration={start radius=2.5cm,end radius=2cm,amplitude=3mm,angle=16}
    ]
\node [concept, minimum size=4cm] {User experience\\when watching\\video}
    [clockwise from=60] child
    {
        node[concept, fill=lightgray] {Playback performance}
        [clockwise from=90]
        child {
            node[concept, fill=lightgray] {Motion quality}
            [clockwise from=110]
            child{ node[concept, fill=lightgray] {Frame timing}}
            child{ node[concept, fill=lightgray] {Missing frames}}
        }
        child {
            node [concept] {Image quality}
            [clockwise from=70]
            child{ node[concept] {Decom\-pression}}
            child{ node[concept] {Re\-scaling}}
        }
    }
    child
    {
        node[concept] {Physical aspects of device}
        [clockwise from=30]
        child {
            node[concept] {Form factor}
            [clockwise from=50]
            child{ node[concept] {Weight}}
            child{ node[concept] {Heat}}
        }
        child {
            node[concept] {Screen}
            [clockwise from=10]
            child{ node[concept] {Size}}
            child{ node[concept] {Colors}}
        }
    }
    child
    {
        node[concept] {Video material}
        [clockwise from=-30]
        child {
            node[concept] {Format}
            [clockwise from=-10]
            child {node[concept] {Frame rate}}
            child {node[concept] {Res\-o\-lu\-tion}}
        }
        child {
            node[concept] {Subject matter}
            [clockwise from=-50]
            child {node[concept] {Move\-ment}}
            child {node[concept] {Details}}
        }
    }
    ;
\end{tikzpicture}
\caption{Illustration of the wide variety of aspects that affect the user experience when watching a video played back on an electronic device. The parts that are of central interest for this thesis are shaded in gray.}
\label{fig_mindmap}
\end{figure}

Because video playback is a primary use for so many consumer electronic devices, it is clear that the quality of the playback affects the user's perceptions of the device. The smoothness of the image and the smoothness of the motion can have a great deal of effect on the market success of the tablet, computer or phone. A testimony to this is the decision of Apple to adopt the Retina displays \cite{retina}, which provide high image quality, and the constant drive towards higher performance processors in tablets \cite{androidbenchmark} to provide higher motion quality. Even the user interface and games are closely related to video playback, although some of the content is generated in real-time.

Therefore, it is beneficial for the manufacturer to optimize the video playback performance of their device. One way to attempt this is just to select the highest performance hardware on the market, and hope that it overcomes any deficiencies in the software. However, it would be more cost effective to improve the software, as the improvements there can be distributed to all devices for no extra cost.

Our goal is to find and eliminate performance bugs from the software. Some of them can be found out by simple testing by a human, but others are not immediately obvious when watching the video playback. Humans have a high tolerance for poor quality video when it is viewed alone, and the deficiencies only become obvious when compared against a better version. Even if such a golden reference is available, avoiding the subjective bias will be difficult \cite{winkler}.

To achieve objective results, we want to leave the content material and the personal characteristics of the viewer out of the consideration. This can be achieved by carefully designed double-blind studies, such as the \emph{Double Stimulus Continuous Quality Scale} methodology \cite[p.159]{itu_opavc}, but the cost of such studies is prohibitive for the product development phase. It would be optimal to have an electronic device, which could accurately estimate the perceptions of the average viewer. Such a device could then be used during the whole product development project to improve the quality of the product, instead of just rating the product after it is already almost finished.

Electronic measurements that try to estimate the perceived video quality are usually termed \emph{metrics}. The typical way to establish a metric is to define one or more measurable characteristics of the video, and then construct a model that maps the measurements into a single value. The accuracy of the metric can be studied using the double-blind studies. The goal is for the metric to correspond as well as possible to the average perceptions of the users.

There are multiple widely used metrics for video quality, ranging from the PSNR (Peak Signal-to-Noise Ratio) to more advanced metrics that more closely approximate the human perception \cite{winkler}. However, most of them focus on the image quality, ignoring the motion quality. There are several reasons for this. Historically, the signal path has consisted mostly of analog systems that could degrade the image quality, but could not change the timing of the material as there was no buffering. Similarly, with the analog systems, changing the dynamic characteristics such as the frame rate of the material was not feasible and it was not important to measure something which you cannot improve.

However, the introduction of digital video has radically changed the practical aspects of video quality. Because most digital video systems buffer one or many frames before playback, the timing of the frames can be easily and also accidentally altered. Similarly, higher quality equipment can use higher frame rates than traditional analog systems have. Consequently, it is now equally important to measure motion quality as it is to measure the image quality.

There exists ongoing research into the development of metrics for video motion quality. Notable examples include the \emph{Motion-based Video Integrity Evaluation} index \cite{movie} and \emph{V-Factor} \cite{winkler}. However, there is no general consensus on the reliability of these metrics, and neither has achieved widespread use.

The use of metrics to estimate the perceptions of the user can also be misleading. The traditional \emph{Peak signal-to-noise ratio} (PSNR) metric for the image quality has been widely used to optimize the compression in video codecs, mostly because of its mathematical simplicity. However, further research has shown that this actually leads to excessively blurry images, and codecs that rate better in this metric can actually look worse in reality \cite{x264}. Therefore if a metric is to be used, it must be carefully selected to avoid any undesired bias.

\begin{figure}[h]
\centering
\begin{tikzpicture}
\tikzstyle{block} = [draw, align=center];
\tikzstyle{result} = [align=center];

\node [result] (video) {Video file};
\node [block, right=1cm, yshift=-1.5cm] at (video.east) (encode) {Compression};
\node [block, right=of encode] (decode) {Decompression};
\node [block, right=7cm] at (video.east) (display) {Playback\\device};
\node [result, right=of display] (observer) {Observer};

\draw [->,thick] (video.east) -- +(0.5cm,0cm) |- (encode);
\draw [->,thick] (video.east) -- (display);
\draw [->,thick] (encode) -- (decode);
\draw [->,thick] (decode) -| (display);
\draw [->,thick] (display) -- (observer);
\end{tikzpicture}
\caption{A single playback device can be used to perform subjective comparisons of video quality by alternating between two sources, for example between original and compressed material. However, measuring the quality of the playback device itself is not possible with this setup.}
\label{fig_subjective}
\end{figure}

Both the double-blind studies and most of the metrics rely on the availability of a golden reference. The test subjects and the metrics compare \emph{degraded material} against the source material. The principle is shown in Figure \ref{fig_subjective}. This type of study is well suited for video compression, transmission and storage testing, where the degraded material and source material can be played on the same screen. However, when we are interested in the degradation that occurs in the playback device itself, how do we play back the reference? Or in the case of a metric, how do we capture the degraded material?

There does not exist a general solution to this problem. It might be feasible to use a very high-quality video display as the reference, and set the test up so that the two devices are indistinguishable to retain the double-blindness. For comparative studies, comparing the playback from several screens is sufficient, but this also requires careful arrangements to avoid subjective bias if the devices can be distinguished. For the use of metrics, a high-quality video camera can be used to record the display, and through careful calibration and positioning the recording can be compared to the source material.

However, there is also a different way to approach the problem. We can define how the ideal video playback device should function, even though it cannot be implemented in reality. Objective measurements can then be used to determine how close the device is to the ideal behavior, and what aspects should be improved. Because the reference device doesn't actually exist this precludes the use of human subjects for the testing, and we need accurate measurement devices to replace them instead.

\subsection{Ideal video playback}
Video playback can be considered to be ideal when it fulfills the following criteria:

\begin{enumerate}
\item The color values for every pixel in every frame are exactly the same as in the source material.
\item Each frame is displayed for the same length of time, which is equal to 1/frame rate.
\item Frame changes are immediate and all pixels on the screen change the color instantly and at the same moment.
\end{enumerate}

Note that the quality of the source material is ignored in the consideration of video playback performance. Even a poor quality video can be played back ideally, and the quality of the material depends solely on the quality of the recording device and any compression that has been applied. It is also assumed that there is an agreed upon way to decode the source material, i.e. that the color formats or compression is not ambiguous or implementation-dependent in any way.

Even with these restrictions, the ideal video playback cannot be achieved in practice due to several inherent physical constraints. There are also many possible implementation problems that can cause the performance to degrade. If we cannot achieve the ideal performance, how close to it should we strive to be?

\subsection{Limits of human perception}
In some market segments, it is enough to optimize the video playback performance so that it satisfies the average user. In the high-fidelity market, the manufacturer's goal is to push the quality beyond what their most demanding users can perceive. However, in all situations it is unnecessary to optimize beyond the limits of the human perception.

The human eye functions quite differently from a camera \cite{watson}. Instead of evenly spaced frames, each light receptor transmits its own sequence of neuroimpulses to the brain. When a new object becomes visible, brains are aware of its position and velocity much sooner than the actual color or details of the object. A camera in similar situation would have all the details of the object available at the same time, when the first frame is captured.

It is relatively straight-forward to determine the maximum visual acuity and color accuracy for the human eye \cite{kalloniatis}. Based on these, it can be calculated how many pixels and how many color values can be seen when a display is viewed from a certain distance. Modern high-DPI (Dots Per Inch) displays, such as Apple's Retina, approach or exceed these limits, leading to a viewing experience where the individual pixels can no longer be distinguished. When combined with good color quality, it is theoretically possible to reproduce still scenes exactly as they would appear in real life, except for depth perception.

We can also define a limit for the perception of the motion. Even though the eye operates in such a way that it is impossible to define a strict frame rate for it, it does have a limit after which an increase of the frame rate of a display cannot be detected. Such studies have been conducted, and the  typical result is in the range of 30--120~FPS \cite{kuroki}. The large variance is to be expected both due to individual differences and due to differences in the video content.

Even if a user cannot distinguish the difference in the frame rate, any \emph{variation} in the frame times may still be visible. Some studies suggest that the impact of dropped frames on perceived video quality decreases quickly at frame rates over 25 FPS \cite{quan}, which would mean that a jitter (difference of shortest and longest frame time) of up to 40~ms (1~s~/~25) would be allowable. However, there is a lack of studies that would have actually tested higher frame rates, so the impact is difficult to estimate. Personal experience shows that variations of around 10~ms are visible, especially in slowly panning aerial imagery. Even though determining an exact limit of perceptibility would require further studies, it can be assumed to be somewhere in the tens of milliseconds.

These values give an idea of how accurately we should aim to measure the display behavior, and how close to the perfect result the displays should aim to be.

\subsection{Challenges in video playback}
Any video playback device has physical restrictions that prevent it from reaching the ideal video playback. Many of these are cost issues, so a compromise can be reached which provides adequate performance compared to the price of the device.

The most common physical limitations are:

\begin{enumerate}
\item Color gamut of the display, limiting the reproduction of color values.
\item Refresh rate of the display, limiting the frame rate.
\item Latency of the pixel color changes, limiting the frame change speed.
\end{enumerate}

However, there are also several problems that are primarily caused by the software implementation of the video playback device. Often these can be overcome with careful design, without increasing the price of the hardware.

Most common software limitations are:

\begin{enumerate}
\item Inaccuracies in decoding of the video contents or color space conversions.
\item Unpredictable timing in multiprocessing operating systems.
\item Gaps in playback caused by unexpected delays in fetching the source material from disk or network.
\item Mismatch between display frame rate and content frame rate.
\end{enumerate}

Of these, limitation 1. causes problems in image quality, while limitations 2-4. cause problems in motion quality. Particularly 2. and 3. are difficult to test, as they can vary depending on background programs and network load, requiring long test runs in different kinds of environments.

\begin{figure}[h]
\centering
\begin{tikzpicture}
\begin{axis}[name=dev1, x=2cm, y=1mm, ymin=0, ymax=60, xmax=3.0,
    every axis y label/.style={at={(ticklabel cs:0.5)},rotate=90,anchor=near ticklabel},
    ytick={0, 16.66667, 33.33333, 50.00000, 66.66667}, yticklabels={0, 17, 33, 50, 67},
    ylabel={Frame time (ms)}, xlabel={Video time (s)}]

\addplot [very thick] plot table [x index=0, y index=1] {plots/good_jitter.txt};
\end{axis}
\begin{axis}[name=dev2, at={(dev1.east)}, anchor=west, xshift=1cm,
    x=2cm, y=1mm, ymin=0, ymax=60, xmax=3.0,
    ytick={0, 16.66667, 33.33333, 50.00000, 66.66667}, yticklabels={,,,,},
    xlabel={Video time (s)}]

\addplot [very thick] plot table [x index=0, y index=1] {plots/bad_jitter.txt};
\end{axis}
\node [above = 0.5cm] at (dev1.north) {Device 1};
\node [above = 0.5cm] at (dev2.north) {Device 2};

\end{tikzpicture}
\caption{Frame timings measured from two video playback devices.}
\label{fig_jitter}
\end{figure}

Least obvious of the problems is number 4, which is caused by the refresh rate of the display hardware not being evenly divisible by the frame rate of the content. If the refresh rate can not be changed, the best compromise is to duplicate single frames at even intervals. However, if not explicitly designed and verified, it is much more common for the frame duplication to vary randomly based on minute timing variations of the system. This can lead to some frames being displayed for e.g. 3 refresh cycles, while other frames would be shown only for 1 cycle.

Figure \ref{fig_jitter} shows an example of this behavior by comparing the behaviour of two playback devices. On both devices, the display refresh rate is approximately 60~Hz and the video frame rate is 30~FPS. Even though the nominal frequencies are divisible by each other, small variations can cause them to drift relative to each other. Device 1 handles the situation well, only requiring an occasional shorter frame to compensate, while the device 2 unnecessary also lengthens the frame times and causes excessive variation.

The jitter in frame times is seen as unevenness in motion, but the visibility of the problem depends heavily on the contents that is being played. In the example case, device 1 has a jitter of 17 milliseconds and device 2 has a jitter of 33 milliseconds. The jitter also occurs much less often on device 1, which will further reduce the visibility of it.

Overall, the visibility of the software limitations listed above depends on the user. Some users will tolerate even high inaccuracies in the playback, and in some kinds of content the problems may not even be visible. However, it is reasonable to expect that the differences will be noticed by reviews and when customers are comparing devices side-by-side in a store.

\addglos{Frame time}{The length of time for how long a specific video frame is visible on the display}


\subsection{Need for timing measurement}
Comprehensive testing of video playback performance will require the combination of several methods. Some of the aspects can be verified using traditional software testing methods, such as running a known file through the decoder and verifying the output. Other aspects depend more heavily on the interaction between software and hardware, and require therefore measurements from the real device.

One useful characteristic to measure is the timing of various events, most importantly the frame changes. This timing information can provide a good estimate of the motion quality of the playback. It is also relatively straightforward to measure, if the played back material is modified to include suitable markers.

The central point of this thesis is the development of a method and device for such timing measurements. It is also important that the solution can be integrated in a larger measurement system, which can then measure the complete scope of video playback performance.

\subsection{Existing solutions}
There exist several ways in which the timing information of video playback can be captured:

\begin{enumerate}
\item By modifying the playback software to capture timestamps when frames are shown.
\item Using generic electronic instruments, such as an oscilloscope connected to the display hardware.
\item Using a specialized instrument, which monitors the display using a sensor or a camera.
\end{enumerate}

Method 1 is the most inexpensive, and it has found widespread use. However, it doesn't usually capture the delays introduced in the graphics card or display hardware. Furthermore, as it requires modification of the software, it cannot be used for competitor analysis nor for testing the final production units.

Method 2 can be very flexible in the kinds of measurements that can be performed. An example of this method is an application note by Rohde \& Schwarz on lip-sync measurement using an oscilloscope \cite{rohde}. However, the signal analysis is rarely automated. This means that long test runs will require laborious manual verification of the waveforms, or the development of custom signal analysis software for the purpose. In the latter case, the complexity of the system would approach that of the method 3.

Method 3 is ideally the least laborious and the most widely applicable option. If the algorithms included in the instrument perform well, the method can be applied with little modification to any kind of video playback device. However, the downside is that this method requires the purchase of a specialized instrument, the expense of which may not always be justified for a single kind of measurement.

\section{Towards a measurement instrument}
The measurement instrument developed in this thesis uses a light sensor, which is attached to the screen of the video playback device. A microprocessor will then analyze the waveforms captured using the light sensor and collect the requested data derived from the frame timings. Figure \ref{fig_meassetup} shows an overview of the measurement setup.

In addition to measuring the frame changes, it is useful to be able to output a synchronization pulse to an external camera. This capability allows building a larger measurement system, where this instrument is used to detect the timing of the video frames, and another instrument is used to verify the content of the images. An example of such a system is the OptoFidelity AV100 \cite{av100}.

\begin{figure}[h]
\centering
\begin{tikzpicture}
\tikzstyle{block} = [draw, text centered, text width=3cm];
\tikzstyle{result} = [text width=3cm];

\node [block] (display)
{
	Playback device's display
};

\node [block] (sensor) [right=0.5cm] at (display.east)
{
	Light sensor
};

\node [block] (device) [right=0.5cm] at (sensor.east)
{
	Detection of frame changes
};

\node [result] (camera) [above right=1cm] at (device.east)
{
	Synchronization pulse to camera
};

\node [result] (fps) [below right=1cm] at (device.east)
{
	Timestamps of frame changes
};

\draw [->,thick] (display.east) -- (sensor.west);
\draw [->,thick] (sensor.east) -- (device.west);
\draw [->,thick] (device.east) -- +(0.5cm,0) |- (camera.west);
\draw [->,thick] (device.east) -- +(0.5cm,0) |- (fps.west);
\end{tikzpicture}

\caption{The basic measurement setup for camera synchronization and frame rate measurements.}
\label{fig_meassetup}
\end{figure}

The need to measure the frame timings and to synchronize a camera is central to many video playback performance measurement systems developed by OptoFidelity. Therefore, the kind of device described in this thesis was first developed very early in the company history. First version was under the name of \emph{Synchronization Module} and then refined as \emph{Frame Rate Meter}, shown in Figure \ref{fig_frm}. The device described in this thesis is a continuation of this product line, named \emph{Video Multimeter} to reflect the wider measurement possibilities.

\begin{figure}[h]
\centering
\includegraphics[width=6cm]{pictures/frm120.jpg}
\caption{The OptoFidelity Frame Rate Meter is a predecessor to the device described in this thesis.}
\label{fig_frm}
\end{figure}

Throughout the various versions, the basic measurement setup has remained essentially the same. However, the use of more refined hardware and algorithms allow more accurate data to be extracted from the measurements. Equally important goal of the new version is to address some limitations that have restricted the applicability of the previous versions.

\subsection{Issues of the previous version}
The Frame Rate Meter is a device that can detect a black-and-white marker on the display using a fiber-optic light sensor. The actual detection of the light levels is implemented as an analog filter circuit, the output of which is then monitored by an 8-bit microcontroller. The device can perform the two basic functions: output a synchronization pulse, and capture timestamps when a change is detected. The analog signal path has proven to have predictable operation and low latency of just a few microseconds, both of which are important for this kind of a device. 

However, it does also have drawbacks: adapting the device to different purposes is difficult, because the only way to change the signal path is through hardware modifications.
It also translates to a large number of discrete parts in the hardware, which increases production costs.
Some of the parts used are no longer manufactured, leading to reduced availability.

\addglos{Dropped frame}{A frame that is missing in video playback, usually due to insufficient performance of the playback device}
The sensor used is a single photodiode, connected to the fiber-optic cable. This kind of a sensor can sense the brightness of the light at one point of the screen. It is designed to be used with a test video, which is modified to contain a black-and-white blinking square in one corner. Detecting the difference of black and white light levels is reliable, but it has one severe limitation: if a frame is missing in playback, the square will be the same color for two frame times. For example, if a frame with black marker is dropped, the marker will remain white for two frames in a row. Consequently, one dropped frame in playback leads to two dropped frames in capture. Similarly, two dropped frames can go undetected.

This problem has previously been circumvented using a hardware modification, which links two units together. In this way it has been possible to monitor two separate synchronization markers on the screen.
By having these markers use different frequencies, dropped frames can be handled more accurately.
Essentially this forms a 4-state marker, by using 2 bits.
However, the system consisting of two separate devices and fibers is difficult to set up and is rarely used.
Also, even with the modification, the device does not collect any information about the dropped frames.

Another problem that has been noticed in practical use is the limited capture rate of cameras.
When emitting the synchronization pulse to the camera, it is usually desirable to minimize the latency between the frame change and the pulse. However, if two frames occur very close to each other, the external camera cannot capture them fast enough and will ignore the second trigger pulse. This shows a need for flexibility when integrating into other parts of a larger measurement system: to adapt to the speed of the camera, a delay has to be inserted between two synchronization pulses if they occur too closely. Previously this has also required a hardware modification, which is rarely practical.

Finally, the old version does not attempt to handle the backlight flicker of the display.
In practice this means that the video playback device has to be set to full brightness, an option that may not be available on all devices.

\subsection{Strengths of the previous version}
One advantage of the analog signal processing is the guaranteed low latency and jitter, which are just a few microseconds.
The camera synchronization requires predictable timing of the synchronization pulses, and is then adjusted to give the best possible captured image quality.
To have reliable measurement results, the timing has to be equal for all captured frames.

\addglos{PC}{Personal Computer, a general purpose desktop or laptop computer, usually based on Intel x86 architecture}
\addglos{GUI}{Graphical User Interface, a computer program's interface composed of graphical elements, such as buttons, images and text areas}

\begin{figure}[h]
\centering
\includegraphics[width=14cm]{pictures/frm120gui.png}
\caption{The graphical user interface previously developed for the Frame Rate Meter.}
\label{fig_frm120gui}
\end{figure}

A PC program, shown in Figure \ref{fig_frm120gui}, has been developed to act as a GUI for the device.
It can show the frame rate in real time, and measure the variations in it.
The previous version has already been integrated to various other measurement systems, which would benefit from retaining the same USB interface and functionality.

The previous version has also been extensively used in the field, and found to perform reliably.
When acting as a synchronization device, it is only a small part of the whole measurement system.
Therefore it is especially important that synchronization is accurate and consistent, as errors in results are difficult to pinpoint to a single part of the system. The simplicity and predictability of the analog signal processing path has helped in achieving this goal.

\subsection{New market needs}
Despite fulfilling its immediate goal, the Frame Rate Meter has not been a large success on the market. Feedback from the sales indicates that the narrow application area and limitations in measurement are the prime cause for this. Many customers consider the expense of a separate instrument too large, if it can only perform a basic frame rate measurement.

Early in the planning of this thesis project, several new possible market areas were identified. Examples of these are: more thorough frame rate measurement, with dropped frames detection; camera latency measurement; audio-video synchronization measurement. All of these have something in common with the basic task, namely the measurement of timing information from a display device. However, they all require a different method of analyzing the signal, and also extra hardware for some of the tasks.

Overall, it was identified that the new instrument would have to be more flexible and extensible than the previous version. This would allow the sale of a single hardware unit, which could later be expanded using different software options.

\subsection{Similar products and methods}
There are not many products on the market that perform frame rate measurement from the playback device's display. All of these rely on markers embedded in the video feed.

VDelay \cite{vdelay}, shown in Figure \ref{fig_vdelay}, is a program for latency and frame rate measurements in video calls, developed by Columbia University. It uses a small display in front of the transmitting camera to show a bar code, which is detected at the receiving end using software. By synchronizing the clocks of the devices, the latency of the transfer can be computed. By detecting when the barcode changes, the frame rate of the video is known. The main challenges in the system are related to the refresh rate of the LCD display and the camera shutter speed, which affect the blurriness and thus detection of the barcode.

\begin{figure}[h]
\centering
\includegraphics[width=14cm]{pictures/vdelay.jpg}
\caption{Screen shot of the vDelay application. Image from \cite{vdelay}.}
\label{fig_vdelay}
\end{figure}

An off-the-shelf high-speed video camera can be used to study the frame-rate in single-use scenarios. This involves recording the display at a high frame rate and careful studying of the resulting video to identify frame change points. The method is limited by the length of video that the high-speed camera can record, which is often just a few seconds, and by the manual work required to process the resulting video. These limitations make studying longer periods of playback infeasible.

\begin{figure}[h]
\centering
\includegraphics[width=14cm]{pictures/chromatic.jpg}
\caption{Running measurement with Spirent Chromatic test system. Image from \cite{chromatic}.}
\label{fig_chromatic}
\end{figure}

Spirent Communications (previously known as Metrico Wireless) has a test system called Chromatic \cite{chromatic} for measuring frame rate and other video performance aspects from a device display. It uses a high-speed video camera to capture spinning circular markers that have been embedded in the video. An example measurement setup is shown in Figure \ref{fig_chromatic}. The results are processed on a normal PC laptop. The same camera is also used to perform further analysis of the video content, such as detecting transmission errors. There is no public information available on the speed of the camera used nor the measurement accuracy that can be achieved using this setup, but it is reasonable to assume that it is limited by the camera speed and available processing power.

Pixel Instruments has a specialized instrument for lip-sync measurements. The LipTracker \cite{liptracker} automatically detects the speaker from the image and uses computer vision algorithms to match the lip movements to the audio signal. In this way the measurement can be performed without a reference source and without any added markers.

Compared to the similar products, the Frame Rate Meter stands apart by its use of a simpler marker scheme. Instead of using a 1-dimensional marker, such as a barcode, or a 2-dimensional marker like a spinning disc, it uses a single blinking rectangle. This allows the instrument to capture the marker using a simple photodiode sensor, which can operate at a significantly faster sample rate than a camera. This yields higher accuracy measurements, but on the other hand limits the amount of information that can be conveyed by the marker.

\section{The idea: a Video Multimeter}
Early in the project, the idea began to form about a highly extensible hardware device, which could interface various sensors and perform real-time signal processing on them. Although the name \emph{Video Multimeter} was coined only later, it encompasses well the purpose of this device: to be a convenient instrument to measure many aspects of video playback, providing both quick feedback during development and accurate results for quality assurance.

To achieve this goal, the hardware has to be designed to be both powerful and extensible. Furthermore, the user interface needs a good usability, in order to make a versatile tool. All this had to be done in the scope of a relatively small research and development project, in order to keep the development costs reasonable. Fortunately, with modern digital parts and ready-made software components, the task is much more feasible than even just a decade ago.

\subsection{Basic principle for new version}
Advances in microprocessor technology have increased the performance steadily over the years. Modern microcontrollers and small microprocessor systems are already fast enough to allow the complete implementation of the signal processing path in software. By capturing all the sensor readings into digital form as early as possible, the signal path can be defined in software to suit the needs of each measurement task. It also reduces the part count, as many microcontrollers now include a fast analog-to-digital converter and a large selection of other peripherals on the same chip.

In addition to performing well in the measurement tasks that have been decided ahead of time, the device needs to be adaptable for new purposes. Therefore sufficient extension capabilities must be designed into the hardware, and also the software needs to allow efficient reuse and modification.

\subsection{New features}
The black-and-white marker of the Frame Rate Meter is the simplest possible way to detect frame changes, but over the years it has turned out to be too limiting. Instead, a multi-state marker was required to be able to gather more information about the video playback. Multiple options were considered, such as using a row of binary markers, but eventually it was decided to focus primarily on color information to provide the additional states.

By using the three color channels (red, green and blue) in a binary fashion, 8 combinations can be created. This is enough to detect multiple dropped frames in the video playback, while the marker can still be read through a single optical fiber. This is useful as it takes up minimal space on the display, which may need to remain visible for other measurement devices.

The multi-state marker allows for the most important new feature: ability to detect dropped frames. This allows accurate frame rate and dropped frame measurements, which were difficult with the previous version if any frame dropping occurred.

However, just doing frame rate measurement is a too narrow market segment to cover the development costs. By leveraging the flexibility of software, the device can be used for a variety of tasks: latency measurements, lip sync measurements, camera synchronization and backlight analysis to name a few. Some of these were implemented for the first release, in the scope of this project, while others were left for future development.


