\chapter{Software architecture}
The high-level purpose of the device is straightforward: capture data from sensors, apply signal processing 
algorithms, show results or take an action.
The design of the necessary filtering algorithms is the most difficult part of the core tasks.
It is more of a signal processing problem than a software architecture problem.

However, to support these core tasks, a significant amount of low-level software is necessary.
Because the hardware is custom, there is no premade operating system that would work without configuration.
Also some device drivers will have to be written, because no operating system contains drivers for all of the wide 
variety of chips on the market.



\section{Real-time operating system}
Most important part of the support software is the operating system that will run on the device.
The purpose of a RTOS (Real-Time Operating System) is to support multiple parallel tasks, without compromising 
the strict timing requirements of the tasks.
Most RTOS also provide a HAL (Hardware Abstraction Layer), which contains device drivers and a programming interface for the various parts of the hardware.

\subsection{Requirements}
In the scope of this project, there are no resources to port or significantly extend the chosen RTOS.
Therefore it is necessary that the RTOS already contains support for the STM32F4 processor.
Most important subsystems should also be included or easy to integrate: USB driver, graphics toolkit and file system.

Furthermore, to be commercially useful, the operating system must have reasonable support and licensing model and reasonably secure future availability.
This can mean either paid-for support package, or an open source community around the project.
In the latter case, any features not implemented by the community can be implemented by the designer, only to the limit of available time.

\subsection{Options}
Table \ref{tbl_rtos} shows 4 candidate operating systems that were considered for the project. All of them could fullfill the requirements, either by themselves or when combined by other components. In some cases, however, this would require adapting separate libraries to work with the chosen RTOS, while in other cases an already integrated solution exists.

{\bf ChibiOS/RT}\cite{chibios} is an open-source RTOS, which contains basic thread facilities and a 
HAL with drivers for most STM32F4 peripherals \cite{chibios_hal}.
It does not include a file system nor C standard library, but separate libraries can be used.

{\bf {\textmu}C/OS-II}\cite{ucos} is a commercial RTOS developed by Micrium.
There is no official STM32F4 support yet, but third-party ports exist \cite{ucos_stm32}.
Advanced file system and graphics modules are available.

{\bf NuttX}\cite{nuttx} is an open-source RTOS that aims "to be a tiny Linux-compatible OS for MCUs."
Consequently it supports many POSIX interfaces, such as pthreads.
It has drivers for most STM32F4 peripherals, but some are not fully working or tested.

{\bf FreeRTOS}\cite{freertos} is an open-source RTOS with long history and extensively reviewed code base.
The basic system includes only the kernel and thread functionality, but device drivers are available from third 
parties \cite{ucos_stm32}.

Some well-known operating systems were left out of comparison because of lack of hardware support, such as NutOS, eCos or VXWorks. These systems did not support the ARM Cortex-M4F processor at the time when the device was 
being designed.

Overall, NuttX contained the most features that were required by this project. Especially its integrated graphics subsystem, which supports multiple application threads and has a built-in window manager, contributed towards the decision. Furthermore, being open source and BSD licensed made it an inexpensive option that can be modified as necessary.

\begin{table}
\caption{Feature matrix of the software components included in each of the considered RTOS.
Features marked as \textbf{separate} or \textbf{third-party} are sold separately by either the same company or by another one. Features marked as \textbf{no} do not have an existing implementation for the operating system.}
\label{tbl_rtos}
\begin{tabular}{llllll}
\toprule
RTOS & Graphics & C library & File system & HAL & STM32F4 USB Driver \\
\midrule
ChibiOS/RT & No & No & No & Yes & Yes \\
{\textmu}C/OS-II & Separate & Separate & Separate & Yes & Third-party \\
NuttX & Yes & Yes & Yes & Yes & Incomplete \\
FreeRTOS & No & No & Third-party & No & Third-party \\
\bottomrule
\end{tabular}
\end{table}

\section{Programming language}
The traditional de facto programming language for embedded systems has been C.
However, modern microcontrollers are powerful enough to support higher level languages, which may offer faster software development and easier to understand code.

There are two main components in the system: the signal processing and the user interface.
For the signal processing part, real-time requirements such as high performance and predictable latency are critical.
For the user interface, performance is not as critical as the efficiency of software development.

\subsection{Options}
The Cortex-M4 processor is widely supported by many programming languages. Some of the most commonly used programming languages are:

{\bf C} is supported by compilers from many vendors and is compiled to native machine code for the target.
It is a relatively low-level language, allowing efficient use of the hardware but often requires more complex code for implementation than other languages.

{\bf C++} is a high-level object-oriented language, also supported by compilers from many vendors.
Compared to C, it has higher level features but still allows everything that is possible through C and has equal performance.

{\bf C\#} is a high-level object-oriented language, which uses garbage collection to simplify development. It can be run on microcontrollers by the means of .NET Micro Framework \cite{netmf}.
Unlike the full .NET Framework, the Micro Framework only interprets the byte code and does not include just-in-time compilation. This leads to 2--10 times slower execution than the languages that are compiled to native code.

{\bf Lua} \cite{lua} is a script language, which is widely used to implement the higher-level portions of applications and games. It is rarely used to implement complete programs, but is used as a method to allow easy customization of applications for various purposes. In the scope of this project, it could allow fast development of measurement tasks once the lower level components have been finished.

The NuttX operating system itself is implemented in C and C++. Therefore the use of any other language would require the development of a separate interface layer to provide access to the NuttX functions from the rest of the software. Choosing C++ as the development language was a reasonable way to maximize the performance while also providing high-level features for development.

\section{Implementation}
The software components implemented as part of this project can be divided into three categories: device drivers, signal processing filters, and user interface. The overview of the software architecture and the components implemented in this project are shown in Figure \ref{fig_softarch}.

\begin{figure}[h]
\centering
\begin{tikzpicture}[node distance=1cm]
\tikzstyle{block} = [draw, align=center, minimum height=1.5cm];
\tikzstyle{hblock} = [align=center];
\tikzstyle{mblock} = [draw, align=center, minimum height=1.5cm, fill=lightgray];

\node[mblock] (app) {Measurement\\application};
\node[block, below=of app, xshift=-4.7cm] (libc) {Support\\libraries};
\node[block, right=of libc] (gui) {Graphics\\subsystem};
\node[block, right=of gui] (kernel) {Operating system\\kernel};
\node[mblock, right=of kernel] (sigproc) {Signal processing\\algorithms};
\node[mblock, below=of gui] (ddriv) {Display\\driver};
\node [block, below=of kernel, xshift=-1cm] (bdriv) {Built-in\\drivers};
\node [mblock, below=of kernel, xshift=1cm] (cdriv) {Custom\\drivers};
\node [hblock, below=of ddriv] (disp) {Display\\hardware};
\node [hblock, below=of bdriv] (microsd) {MicroSD\\card};
\node [hblock, below=of cdriv] (sensors) {Sensors};

\draw [thick] (app) -| (libc);
\draw [thick] (app) -| (gui);
\draw [thick] (app) -| (kernel);
\draw [thick] (app) -| (sigproc);
\draw [thick] (gui) -- (ddriv) -- (disp);
\draw [thick] (kernel) -- (bdriv) -- (microsd);
\draw [thick] (kernel) -- (cdriv) -- (sensors);
\end{tikzpicture}
\caption{Overview of the software architecture. The user interacts with the measurement application, which communicates with the hardware through the kernel and driver layers. The shaded parts were implemented in this thesis work, while the other parts are reused open source components.}
\label{fig_softarch}
\end{figure}

The complete system also contains the scheduler, device drivers and graphics libraries from the NuttX operating system, some of which were extended for this project but most of which are used as-is. Most changes to the NuttX core have been contributed back to the central project, which has the benefit of not needing to maintain them separately over the changes in future operating system versions.

\subsection{Device drivers}
The lowest level of the software implementation is the device drivers, which interface to hardware resources such as the color sensor or the synchronization port. These can be considered to form a part of the NuttX kernel, as applications communicate with them through standard kernel interfaces such as the device nodes in \emph{/dev}.

\begin{figure}[tb]
\centering
\begin{tikzpicture}
[node distance=5 mm]
\tikzstyle{block} = [draw, align=center];
\tikzstyle{result} = [align=center];
\tikzstyle{outerblock} = [draw, rounded corners, inner sep=2mm];
\tikzstyle{outerblock2} = [draw, rounded corners, very thick, inner sep=5mm, minimum width=13cm];

\node [block] (sensor) {Sensor chip};
\node [block, above=of sensor] (adc) {Analog-to-digital\\converter};
\node [block, above=of adc] (dma) {DMA controller};
\node [outerblock2, fit=(sensor) (adc) (dma)] (hardware) {};

\node [block, above=of dma, yshift=3mm] (memory) {Double-buffered\\memory area};

\node [block, above=of memory, yshift=5mm] (hfilters) {High samplerate\\filters};
\node [block, above=of hfilters] (decimate) {Samplerate\\reduction};
\node [outerblock, fit=(hfilters) (decimate)] (irq) {};
\node [block, left=of decimate, xshift=-1cm] (queue1) {Message queue};
\node [block, right=of decimate, xshift=1cm] (queue2) {Message queue};
\node [block, above=of queue1] (filter1) {Low samplerate\\filters};
\node [block, above=of filter1] (gui) {User interface};
\node [block, above=of queue2, yshift=1cm] (save) {Filesystem\\writing};
\node [outerblock, fit=(filter1) (gui)] (app1) {};
\node [outerblock, fit=(save)] (app2) {};

\node [align=left, above left=1mm] at (hardware.south east) {\bf Hardware};
\node [align=left, above=0mm] at (irq.north) {Interrupt handler};
\node [align=left, above=2mm] at (app1.north) (a1) {Application 1};
\node [align=left, above=2mm] at (app2.north) (a2) {Application 2};

\node [outerblock2, xshift=2mm, fit=(app1) (app2) (a1) (a2) (irq)] (software) {};
\node [align=left, above left=1mm] at (software.south east) {\bf Software};

\draw [thick, ->] (sensor) -- (adc);
\draw [thick, ->] (adc) -- (dma);
\draw [thick, ->] (dma) -- (memory);
\draw [thick, ->] (memory) -- (hfilters);
\draw [thick, ->] (hfilters) -- (decimate);
\draw [thick, ->] (decimate) -- (queue1);
\draw [thick, ->] (decimate) -- (queue2);
\draw [thick, ->] (queue1) -- (filter1);
\draw [thick, ->] (filter1) -- (gui);
\draw [thick, ->] (queue2) -- (save);

\end{tikzpicture}
\caption{Data flow in the system when two applications are reading the data from the color sensor device driver.}
\label{fig_dataflow}
\end{figure}

To meet the performance requirements, most of the device driver code runs in interrupts when new samples are available from the hardware. Also DMA (Direct Memory Access) techniques are used to offload the processing of the data to the hardware. When the samples are ready in a memory buffer, the interrupt passes them on to applications by means of message queues, which are implemented by the NuttX kernel.

Figure \ref{fig_dataflow} shows the flow of the data throughout the hardware and software of the device. Hardware handles the most time critical parts of collecting the samples. When one of the memory buffers is full, the interrupt handler triggers, performs some of the more demanding filtering tasks, reduces sample rate of the signal and passes on the samples to memory queues. Each reading application has its own queue of incoming samples, which it can then read and process in whatever way is appropriate for the application's purpose.

The message queue mechanism provides thread safety, but it also has a negative performance impact when large amounts of data are being processed. Consequently, it is important that the device drivers do not have to to pass excessive amount of data to the applications. This is accomplished by running some of the signal processing algorithms in the interrupt handlers, such as the backlight filtering. In order to function effectively, the backlight filter needs to process every input sample at the full 100 kHz sample rate, but for further analysis a smaller sample rate is sufficient.

A traditional approach to interrupt handlers is to keep them as short as possible, because a running interrupt handler blocks all other tasks on the system. However, modern CPU cores such as the Cortex-M4 have multiple pre-emptible interrupt priorities, so that a higher priority interrupt can interrupt another, longer running handler. This is beneficial in exactly this kind of applications, where there is a need to run long processing tasks in the interrupts. By setting a lower priority for the longer interrupt handlers, the shorter ones can pre-empt it. In this specific case, it allows the reduction of the sample rate as early as possible, without compromising the overall system interrupt latency.

\subsection{Signal processing}
Digital signal processing forms a core part of the software. It is necessary to both implement the currently required measurements, and to allow easy extensibility for future needs. Furthermore, efficiency, reliability and testability are important.

\addglos{FIR}{Finite Impulse Response, a type of digital filter where a finite number of previous samples are combined using multiplication and addition}
The signal processing tasks in this system consist of fairly basic algorithms: FIR (Finite Impulse Response) filters, sliding window variances and averages, derivatives and some custom non-linear filtering. It is reasonable that these basic algorithms should be implemented as reusable blocks, which can be combined in various ways to realize the measurement tasks.

The reusability was achieved by implementing each basic algorithm or filter block as a separate C++ class. The classes share a basic interface, which allows chaining of several instances together to form a filter chain. Input samples are given through a function call on the chain's first filter, and the samples then progress automatically through the chain. Output data is collected through a callback function on the last filter. An example of such a filter chain is shown in Figure \ref{fig_filterblocks}.

\begin{figure}[h]
\centering
\begin{tikzpicture}
\tikzstyle{block} = [draw, align=center];
\tikzstyle{result} = [align=center];

\node [result] (input) {Samples from\\the color sensor};
\node [block] (cspace) [above = 0.5cm] at (input.north) {Color space\\conversion};
\node [block] (cdetect) [right = 0.5cm] at (cspace.east) {Color change\\detection};
\node [block] (crecog) [right = 0.5cm] at (cdetect.east) {Color\\recognition};
\node [block] (cseq) [right = 0.5cm] at (crecog.east) {Color sequence\\interpretation};
\node [result] (count) [below = 1cm] at (cseq.south) {List of\\dropped frames};
\node [result] (times) [below = 1cm] at (crecog.south) {Frame times};

\draw [->,thick] (input.north) -- (cspace.south);
\draw [->,thick] (cspace.east) -- (cdetect.west);
\draw [->,thick] (cdetect.east) -- (crecog.west);
\draw [->,thick] (crecog.east) -- (cseq.west);
\draw [->,thick] (cseq.south) -- (count.north);
\draw [->,thick] (crecog.south) -- (times.north);
\end{tikzpicture}
\caption{Simplified diagram of a filter chain for detecting the frame times and dropped frames from a color marker. Each block in the diagram is implemented as a separate filter, which can be chained in different layouts for measurement tasks.}
\label{fig_filterblocks}
\end{figure}

For this thesis, the filter blocks implemented include such functions as backlight filtering, color space conversion, color change detection and color recognition. The algorithms used cannot be described here in detail due to their proprietary nature. An important goal was that the device would not require complex setup, but would instead automatically adapt to many kinds of display devices under measurement. This was accomplished through careful design of the algorithms, so that they will capture the fundamental characteristics of the signal and not be affected by variations between display types.

Because of the very large variations in the kinds of display devices in use, the algorithms used are necessarily somewhat heuristic in nature. Verifying the performance of the algorithms is a major task, as even small changes made to accommodate one kind of display could cause problems for another kind. In order to verify the performance of the filters, a regression test suite was needed.

Figure \ref{fig_waveforms} shows two examples of the various kinds of display behaviors that the signal processing algorithms have to adapt to and tested for. The OLED display is characterized by much faster color changes, which are in fact so abrupt that the internal refresh cycles of the screen can be seen as glitches. The data shown in the graph has already been preprocessed by a color calibration algorithms, which separates the R, G and B channels and normalizes the range of color values to 0--255.

\addglos{LCD}{Liquid Crystal Display, a display technology based on the polarization change of liquid crystals}
\addglos{OLED}{Organic Light Emitting Diode, a display technology based on an array of small light emitting diodes that act as the pixels of the screen}

\begin{figure}[h]
\centering
\begin{tikzpicture}
\begin{axis}[name=graph1, x=0.5mm, y=0.1mm, ymin=-50, ymax=350, xmax=100,
    every axis y label/.style={at={(ticklabel cs:0.5)},rotate=90,anchor=near ticklabel},
    ytick={0, 255}, yticklabels={0, 255}, ylabel={Brightness (RGB)},
    xtick={0, 28, 61, 93}, xticklabels={1, 2, 3}, xlabel={Frame number},
    x tick label as interval]

\addplot [very thick] plot table [x index=0, y index=1] {plots/waveform1.txt};
\addplot [very thick, dashed] plot table [x index=0, y index=2] {plots/waveform1.txt};
\addplot [very thick, dotted] plot table [x index=0, y index=3] {plots/waveform1.txt};
\end{axis}
\begin{axis}[name=graph2, at={(graph1.east)}, anchor=west, xshift=2cm,
    x=0.5mm, y=0.1mm, ymin=-50, ymax=350, xmax=100,
    every axis y label/.style={at={(ticklabel cs:0.5)},rotate=90,anchor=near ticklabel},
    ytick={0, 255}, yticklabels={}, ylabel={},
    xtick={0, 28, 61, 93}, xticklabels={1, 2, 3}, xlabel={Frame number},
    x tick label as interval]

\addplot [very thick] plot table [x index=0, y index=1] {plots/waveform2.txt};
\addplot [very thick, dashed] plot table [x index=0, y index=2] {plots/waveform2.txt};
\addplot [very thick, dotted] plot table [x index=0, y index=3] {plots/waveform2.txt};
\end{axis}
\node [above = 0.5cm] at (graph1.north) {LCD display};
\node [above = 0.5cm] at (graph2.north) {OLED display};
\end{tikzpicture}
\caption{Comparison of OLED and LCD display behaviors. Each graph shows the measured brightness of three color channels: red, green and blue.}
\label{fig_waveforms}
\end{figure}

The very basic behavior of the filters can be verified using synthetic unit tests. For example, a color change detector can be tested by feeding a generated sequence of 100 samples of one color and then 100 samples of another color, and checking that the output shows a single transition at the correct time. 

However, testing the adaptability to various display types cannot be done using synthetic data. Instead, throughout the project test data was collected from various kinds of devices and added to the test suite. Once the data is processed through the filters, the output is checked visually using graphing tools. For example, graphing the detected color changes against the raw color data gives a graph that is feasible, if tedious, to check manually.

After the output has been manually checked once, the results are stored as a reference file. Any further runs will compare the output of this (filter, data file) pair against the reference file, and only report differences that exceed a predefined threshold. This allows automatic checking of all changes against the full test data set, which greatly reduces the possibility of regressions in filter performance.

Further development could be done in the area of comparing the reference files and actual filter output. Currently the checks report considerably often false positives, as there is only a single threshold defined for the whole test. For displays with slow color transition, like LCD in Figure \ref{fig_waveforms}, the uncertainty in the results is necessarily larger than for displays with very fast transition. Similarly, the glitches in OLED waveforms can result in false transitions which must be filtered out separately, a step which complicates the individual testing of the filter blocks.

As the signal processing subsystem is implemented as standard C++, it is very portable to various platforms. For example, the tests can be readily run on the development PC, which speeds up the process of running through all the test data files. Similarly, even if a future version of this device is designed based on a different hardware platform and operating system, the signal processing portion can be reused.

\subsection{User interface}
NuttX includes its own GUI and window manager libraries, called \emph{NxWidgets}\cite{nxwidgets} and \emph{NxWM}\cite{nxwm}. These are designed to fit the resources available on common microcontrollers, and are therefore a good fit for the hardware in this project. The programming language for the graphical side is C++, and the libraries use the familiar object-oriented approach to GUI programming.

\begin{figure}[h]
\centering
\includegraphics[width=8cm]{pictures/startscreen.png}
\caption{The basic NxWM user interface, showing the taskbar on the left and the list of applications in the start window on the right.}
\label{fig_nxwm}
\end{figure}

The NXWidgets library contains user interface elements (widgets) for the most common functions, such as buttons, text boxes and labels. Each widget is a separate C++ class, and new widgets can be implemented by the user of the library. Unlike the GUI libraries typically used on PC's, NXWidgets does not include any automatic layout support: each widget has to be manually placed based on pixel coordinates. As long as the screen size remains constant, however, this is not a large problem.

The NxWM window manager provides facilities for starting applications, switching between them and closing them. The NxWM's basic user interface, shown in Figure \ref{fig_nxwm}, is the taskbar at the side of the screen, and the start window which acts as a list of applications. Each application is implemented as a C++ class, all of which implement a common interface \emph{IApplication}. The application is responsible for creating its own windows, processing input events and painting the graphics when requested by the window manager.

Overall, the NXWidgets and NxWM combination provides a flexible graphics subsystem, which can run multiple applications simultaneously and each application can also have multiple threads itself.
This is useful for example in the frame rate measurement application, shown in Figure \ref{fig_measgui}, where the graph can be updated by one thread while the rest of the user interface is handled in another. For this to work, one aspect that must be taken care of is the threading model between the applications and user input events.

\begin{figure}[h]
\centering
\includegraphics[width=8cm]{pictures/fps_stats.png}
\caption{User interface during frame rate measurement.}
\label{fig_measgui}
\end{figure}

Depending on which thread delivers the asynchronous input events from the user, there may be need to use interlocking between several threads to prevent corruption of the program state. As part of the development work relating to this thesis, the core of NxWM was modified to pass all asynchronous events through a single worker thread, as shown in Figure \ref{fig_multithread} This change was also contributed to the upstream NuttX project, and it simplifies the development of multithreaded applications for this platform.

\begin{figure}[h]
\centering
\begin{tikzpicture}[node distance=2cm]
\tikzstyle{block} = [draw, align=center, minimum height=1.5cm];

\node[block] (ts) {Touchscreen\\driver};
\node[block, right=of ts] (gui) {GUI\\elements};
\node[block, below=of gui] (wt) {System-wide\\worker thread};
\node[block, right=of gui] (sig) {Signal processing\\thread};

\draw[->,thick] (ts) |- (wt) node [midway, below right] {Touch events};
\draw[->,thick] (sig) |- (wt) node [midway, below left] {Data to display};
\draw[->,thick] (wt) -- (gui) node [midway, right, align=left] {Method\\calls};
\end{tikzpicture}
\caption{Multithreading model used in the user interface. All events are synchronized by the use of a single worker thread, which makes it simpler to write reliable multithreaded code.}
\label{fig_multithread}
\end{figure}

By passing each event through the work queue, there is no need to perform interlocking as all GUI actions are delivered by a single thread. Applications can still also use additional threads when necessary, as long as they implement the interlocking themselves.

The applications developed for the device in this thesis are straightforward in architecture. As the signal processing subsystem handles most of the actual measurement task, the application only has to provide means to display and save the results. This is accomplished through a few custom controls, such as the scrolling frame time graph, and a single class per each application. The class instantiates the necessary controls, places them on the screen and runs its own thread to for the signal processing filters.

