\documentclass{sacj}
\bibliographystyle{sacj}
\usepackage{graphicx}
\usepackage{color}

\newcommand{\DGK}[1]{ \textbf { \textcolor{cyan}{ DGK: [ {#1} ] } } }


%SACJ commands
\SACJtitle{Quality in Software Engineering: a pragmatic approach using metrics}
%\SACJauthor[dan]{Daniel Acton}{mja@danielacton.com}
\SACJauthor[up]{Daniel Acton}{mja@danielacton.com}
\SACJauthor[up]{Derrick G Kourie}{dkourie@fastar.org}
\SACJauthor[us]{Bruce W Watson}{watson@cs.up.ac.za}
\SACJaddress[up]{Fastar Research Group, Department of Computer Science, %
                         University of Pretoria, South Africa}
\SACJaddress[us]{Fastar Research Group, Department of Information Science, %
                         University of Stellenbosch, South Africa}
%\SACJaddress{Computer Science Department, University of Pretoria, Hatfield, 0022, South Africa}
\SACJabstract{As long as software has been produced, there have been efforts to strive for quality in software products. In order to understand quality in software products, researchers have built models of software quality that rely on metrics in an attempt to provide a quantitative view of software quality. The aim of these models is to provide software producers with the capability to define and evaluate metrics related to quality and use these metrics to improve the quality of the software they produce over time.  The main disadvantage of these models is that they require effort and resources to define and evaluate metrics from software projects. \\ \\
This article briefly describes some prominent models of software quality in the literature and continues to describe a new approach to gaining insight into quality in software engineering projects. A case study based on this new approach is described and results from the case study are discussed.}
\SACJkeywords{Software Engineering, quality, metrics, measurement}

%Start text

\begin{document}

\maketitle

\section{Introduction}\label{section:Intro}
\emph{Quality} has many definitions in the literature. The Institute for Electrical and Electronic Engineers (IEEE) Standard Glossary of Software Engineering Terminology\cite{IEEE:Glossary} defines quality as ``the degree to which a system, component, or process meets specified requirements or customer or user needs or expectations''. The ISO9000\cite{SANS9000} standard defines quality as ``the degree to which a set of inherent distinguishing features fulfils a need or expectation that is stated, generally implied or obligatory''. In the context of using metrics to measure software quality, the IEEE standard for a Software Quality Metrics Methodology\cite{IEEE:SQMetricsMethod} defines quality as ``the degree to which software possesses a desired combination of quality attributes''.

As discussed by Garvin \cite{Garvin:ProductQuality}, determining the quality of a product can be approached in a number of ways. The \emph{transcendent approach} states that quality cannot be defined precisely, but is rather experiential, and one can learn to recognise quality through experience. The \emph{product-based approach} holds that quality is measurable and related to the degree to which the product possesses some attribute. The \emph{user-based approach} holds that quality is subjective, and is determined by the user of the product based on their perspective. The \emph{manufacturing-based approach} equates quality to conformance to requirements: if a product is built as specified, it exhibits quality. The \emph{value-based approach} holds that a quality product is a product that provides ``performance at an acceptable price or conformance at an acceptable cost''.

There are many approaches in the literature that attempt to gain insight into the quality of software. Industry standards, such as the ISO9000\cite{SANS9000} family of standards, CMMI \cite{CMMI:overview} and The Personal Software Process (PSP) \cite{SEI:PSP}, can be seen as a distillation of industry experiences into ``best practice''. These industry standards stress the importance of process in the pursuit of product quality. The ISO25000 \cite{SANS25000} family of standards, along with work by Boehm et al. \cite{boehm:charsq}, Hyatt and Rosenberg \cite{hyatt:sqmodel} and McCall et al. \cite{Mccall:SQFactors} propose measuring software quality using metrics. Other approaches, such as Cleanroom \cite{Mills:Cleanroom} address quality by ensuring that formal methods are followed in the construction of software. Approaches such as those proposed by Barkmann et al. \cite{similar:swedish} and Gousios and Spinellis \cite{similar:greek} suggest the use of Software Configuration Management (SCM) systems to gain insight into software quality.

This article describes an approach to software quality measurement that relies on the processes often present in the course of a software project (such as requirements gathering, testing and development) and the software often used (such as an SCM system). It also describes a case study that observes the use of the proposed approach in a real-world software project.

 Section~\ref{section2} gives an overview of some of the approaches in the literature, along with analysis of these approaches. Section~\ref{section3} describes a new approach to software quality measurement. Section~\ref{section4} describes an implementation of the approach discussed in Section~\ref{section3} and Section~\ref{section5} discusses a case study that uses this new approach. Section~\ref{section5} provides a conclusion and suggests areas of future work.

\section{Software Quality Models in the literature}
\label{section2}
This section briefly describes eight models in the literature that use metrics as a method of gaining insight into quality in software engineering. These models were chosen as they represent a diverse range of models, across time (ranging from 1977 until 2009), focus (metrics and process) and source (academic and industry). The section closes with a critique of these methods, indicating that there is a need for a more pragmatic approach to quality measurement which does not impose a heavy administrative burden on software developers and managers.

\subsection{McCall's Quality Factors}
In their 1977 technical reports to the United States Air Force~\cite{Mccall:SQFactors}, McCall, et al. describe an approach to assessing software quality using metrics. The approach relies on a set of factors and criteria which affect the quality of the software. Metrics are used to measure these criteria and give a quantitative indication of the quality of the software product across the development life cycle. An example metric measures the cross referencing that relates modules to requirements. This metric therefore attempts to measure the so-called \emph{traceability} criterion, which in turn is believed to contribute to what the authors call the \emph{correctness} factor. 

\subsection{Hyatt and Rosenburg's Software Quality Model}
Hyatt and Rosenburg's model~\cite{hyatt:sqmodel} relies on two core requirements: that the software works well enough and that it is available when needed. This model is based on the perspective of a project manager. It identifies areas of risk to the fulfilment of these two core requirements. Metrics are used to provide an indication of the risk in each of these risk areas. An example metric is the \emph{number of weak phrases}, which attempts to measure the ambiguity of the requirements. This contributes to the so-called \emph{ambiguity} attribute, which in turn affects the \emph{requirements quality} goal. This goal is related to the \emph{correctness risk} area: the higher the requirements quality, the lower the correctness risk.

\subsection{Boehm's Characteristics of Software Quality}
In their book ``Characteristics of Software Quality''~\cite{boehm:charsq}, Boehm et al. discuss an approach to analysing software quality based on a set of characteristics that are generally desirable for software to exhibit. These characteristics are grouped into three main areas: as-is utility (how well the software is usable as it is), maintainability (how easy it is to maintain the software), and portability (how easy it us to use the software in different environments). These three characteristics are used in order to gain some insight into the general utility of the software. In other words: software that is easy to use and maintain in different environments would be classified (according to Boehm, et al.) as having high utility to the user or purchaser of the software. These high-level characteristics are further classified into lower-level characteristics. Metrics are used to measure these characteristics. The metrics used in this model were chosen based on their correlation with software quality, the importance of having a high measure for that metric, the cost of automatically evaluating the metric, and the feasibility and completeness of automated evaluation. An example of a metric described by Boehm et al. is ``Metric Number 1.12'' which measures whether the code is readable or not. This metric contributes to the low-level characteristic of \emph{legibility} which contributes to the intermediate characteristic of \emph{understandability}, which contributes to the high level characteristic of \emph{maintainability}. 

\subsection{Cleanroom Software Engineering}
The origin of the \emph{cleanroom} method is a paper titled ``Certifying the Reliability of Software'' in which Currit et al. \cite{CURRIT:Reliability} discuss a method of certifying the reliability of software before the software is released to users. The reliability of the software is measured by its Mean Time To Failure (MTTF). Currit et al. suggest that users of the software  will not be as concerned with the number of defects in the software as they will be with the reliability of the software.  Currit et al. assume that the output of the development phase of production is of high quality and focus more attention on testing the reliability of the software. This they do by performing a subset of operations that real users would perform in an environment that closely mimics that of the user.

Building on this initial work on software reliability, Mills et al. \cite{Mills:Cleanroom} turned their attention to the assumption that Currit et al. had made, namely that the development output exhibits high quality.  Mills et al. proposed a more formal approach to the development effort in order to ensure that the output does indeed exhibit high quality. The result of this work was the \emph{cleanroom} software engineering approach. 

Cleanroom uses mathematical foundations in the design, specification and correctness verification of the product. The focus on a formal proof that the software works as required mollifies the need to debug developed outputs. The use of formal methods of object-based box-structure specification and design ensures that the specification and design of the product are well-defined.  Function-theoretical methods are used to verify the correctness of software. Statistical usage testing is used to certify the quality of the product.

\subsection{SCM-based approaches}
Barkmann et al. \cite{similar:swedish} and Gousios and Spinellis \cite{similar:greek} propose the automated evaluation of metrics based on a project's source code (stored in an SCM system) in order to gain insight into the quality of the software. Barkmann et al. propose an approach that uses external open source software code repositories to determine which metrics apply to a given project. Gousios and Spinellis propose an approach that allows software producers to use metrics based on more than just the source code (i.e. they could also include metrics based on information from bug-reporting software and email messages).

\subsection{ISO25000}
The ISO25000 family of standards (also known as Software product Quality Requirements and Evaluation or SQuaRE) \cite{SANS25000} is an evolution of an earlier standard, ISO9126 \cite{SANS9126}. Both of these standards propose an approach to measuring software quality using metrics. The ISO25000 family describes a framework for quality models that categorises product quality into a hierarchical view: characteristics, subcharacteristics, sub-subcharacteristics, quality properties and quality measures. The family of standards also provides two instances of the model, namely a quality-in-use model (used to measure the quality of a product as it might be perceived by users of the product), and a product quality model (used to measure the quality of the finished product or interim products during development).

\subsection{ISO9000}
The ISO9000 \cite{SANS9000} family of standards suggests that an organisation wishing to provide quality products or services should focus on the processes used to produce their product or provide their service. If the organisation implements, maintains and continually improves these processes, it is suggested that the outputs of these processes should be of consistent or improving quality. ISO9000 also promotes a focus on the requirements and satisfaction of the customer, and if the organisation aligns their processes with this focus, they should be producing software that customers perceive as satisfying their requirements and hence exhibiting consistent or increasing quality.

\subsection{CMMI}
CMMI focusses on the implementation and improvement of the processes involved in software development, and bases it's focus on the \emph{process management principle}: ``the quality of a system is highly influenced by the quality of the process used to acquire, develop, and maintain it.'' \cite{CMMI:overview}. An organisation that chooses to implement CMMI is effectively implementing a model that aims to make their processes more effective. The organisation progresses through CMMI levels from \emph{initial} (where there are no --- or very few --- processes in place) through \emph{managed}, \emph{defined}, \emph{quantitatively managed} and finally \emph{optimizing}, where there are well-defined processes, and data from the processes are used to optimize the processes further. The suggestion CMMI makes is that if an organisation continually ensures that the processes meet business requirements and are improved over time, the quality of the output should be of a consistent and increasing level.

\subsection{The Personal Software Process (PSP)}
The PSP focuses on having the engineeers or developers producing the product be an integral part of driving the quality of the product. A project is made up of tasks and phases. In each phase, the engineers or developers must estimate the size and effort of the tasks in the phase. As they perform the work, they record information about their work, including any defects they may find or fix, and the actual time taken to complete the task. Once the work is complete, the estimations are compared with the actual data and as a result, future estimations should be more accurate. The metrics that are collected and calculated over the course of the project include quality measures. The indicators of quality in PSP include those based on defects, for example defects per thousand lines of code, defects detected per phase and defects fixed per phase, and those based on productivity. The PSP maintains that in order to strive for quality, engineers or developers must spend time and effort on detecting and fixing defects, particularly aiming to remove defects early and prevent defects through keeping records of defects, ensuring a good design, and reducing development time.

\subsection{Analysis}
The models discussed in this section strive in various ways to provide software producers with the ability to gain some insight into and/or control over  the quality of their software product. Having a model that uses metrics and provides feedback about quality at different points in the development lifecycle is important to software producers so that they may gain some insight into the quality of their product and how they can maintain and increase the level of quality over time. The main disadvantages that the models described in this section have in common is that, for the most part, effort is required to define the metrics to be measured, measure them, and analyse their results for an insight into quality. 

The approach proposed by McCall et al. has been criticised for being too subjective \cite{article:swqelusive}. In addition, the assessment of quality is performed using a set of worksheets that require values for each metric. The effort required to implement this approach is a limiting factor to its implementation, and may yield results too late to be acted upon. 

The approach proposed by Boehm et al. suggest that metrics be defined for a project based on factors like the intended users, characteristics and environments. This also creates a barrier to adoption since the effort to define them is required for every project. While this approach does suggest automated evaluation of metrics, there could still be metrics for which manual evaluation is required. 

The approach proposed by Hyatt and Rosenburg is biased to the view of the project manager, and this may mean that the approach does not cater for other perspectives.

It is important to bear in mind that industry standards are not the definitive instruction on a matter --- they are merely guidelines based on experience. Thus, it is natural to expect that there are supporters and detractors of these standards. The efficacy of ISO9000 in increasing quality has been questioned by Stelzer et al. \cite{Stelzer1997}. When organisations investigate implementing CMMI, there is evidence\cite{Staples:2007} and \cite{Khurshid:2009} indicating that they may decide against implementing CMMI because the cost of implementing is seen as unjustified against the benefits. The effort requireds to implement the PSP approach could be a barrier to its adoption: engineers or developers need to estimate and calculate a number of metrics during their work, and while the benefit is supposed higher quality, the cost is extra effort that would not otherwise have had to be expended.

The approach described in the next section attempts to provide some insight into the quality of a software product without requiring the implementor to expend effort over and above the effort that would normally be expended to write requirements, create tests and monitor defects.

\section{A pragmatic approach}
\label{section3}
This section provides an overview of a pragmatic approach to gaining insight into quality in software engineering. More details about the approach are covered in subsequent sections.

A software engineering project can be defined as a project undertaken to provide some software product that fulfills some requirements of a user or set of users \cite{SANS9000}. As discussed in the literature covered in Section~\ref{section2}, \emph{one} of the indicators of quality in a software engineering project is the degree to which the produced software fulfills the stated requirements of the user. Such a view of quality corresponds to the \emph{manufacturing-based} approach to quality mentioned in the introductory remarks of Section~\ref{section:Intro}. Code is developed in order to meet these requirements, and tests are run against the code to confirm whether requirements have been met. In cases where tests fail (indicating a requirement has not been met) a defect is raised. The aim of a software project, then, is to fulfil all of the requirements and confirm these requirements are met, handling defects as they arise. In general, the approaches discussed in the Section~\ref{section2} have some view of requirements, tests and defects and the relationship between them.

The main difference between the approaches discussion in Section~\ref{section2} and the approach discussed in this Section is the effort required to gather and calculate metrics associated with quality. In this approach, metrics pertaining quality are calculated automatically based on processes and information that would normally be part of the development process anyway (tests, requirements and defects) so relatively little effort is required to implement this approach.

In the text below, a metric or product quality indicator will be proposed to measure the extent to which stated requirements have been met. To this extent, the approach is also classifiable as a \emph{product-based} approach to quality---i.e. it holds that quality is measurable and related to the degree to which the product possesses attributes that are specified as requirements. However, it should be emphasised that the measure is not being proposed as a panacea to the multi-dimensional problem of assessing quality,  but rather as a single and pragmatic mechanism to assist in addressing that problem.  

\subsection{Assumptions}
\label{section:qassumptions}

A software project consists of a set of deliverables. Among these deliverables are code and documentation. The delivered code and documentation are created from some source code (and documentation) that are developed over the course of the project by software developers. The source code is represented by a set of files. Over the course of the project, these source files will be created, modified and tested until they fulfill the stated requirements.

Most software engineering methodologies require a testing phase, so at certain points of the software development life cycle, tests will be run against an interim software product, or \emph{release}. The execution of these tests may uncover differences between the stated requirements and what the interim product actually delivers. These differences can be handled as defects. Successive phases or iterations of the development life cycle, then, will attempt to address these defects, with the goal being no deviation between the product and the requirements (i.e. there are no defects).

The assumptions used in this approach are listed below.

\begin{itemize}
\item There is a set of requirements indicating the requirements that the developers of the software need to fulfill. The requirements are easily identifiable and can be enumerated and can change over the course of the project. The requirements can be functional or non-functional, but this approach assumes that (1) the requirements (functional and non-functional) have been stated in some set of requirements and (2) all of the stated requirements can be verified using some test.
\item There is a set of source files that are developed to fulfill the requirements of the software product. Each source file contributes to the fulfillment of zero or more requirements. Some source files may provide helper functions such as string formatting and are therefore not directly involved in fulfilling requirements. A requirement may be fulfilled by a single source file or a number of source files.
\item There is a set of tests that are run against a set of source files to verify that there is no deviation between what is delivered and what is required (i.e. there are no defects). A test exposes only one defect and a defect is raised by only one test. This does imply some constraint on how tests are designed.
\item Each requirement will be verified by one or more test(s). Because source files are associated with requirements, this implies that a set of tests is associated with a set of source files --- the cardinality of the relationship being unrestricted.
\item There is a set of defects, which represent the deviation between delivered product and stated requirements. Each defect is found as a result of a test execution. Each source file may be associated with zero or more defect(s) and each defect may be associated with one or more file(s). As noted by Dijkstra in \cite{dijkstra:bugpresence}, ``Program testing can be used to show the presence of bugs, but never to show their absence'', and the approach proposed here only deals with those defects that are observable as a result of running tests on the software. If there are latent defects that are not uncovered by testing (for example, because the testing was not rigorous enough), then these defects would not be considered in this approach.
\end{itemize}

\subsection{The Product Quality Indicator}
It is generally considered good practice for software producers to use some form of SCM system in order to manage changes to their software as the software development lifecycle progresses. Subversion\cite{Pragmatic:SVN} is a popular SCM system and supports the notion of executing arbitrary processes when certain events occur, such as a change to some file under the control of the repository. The approach proposed here uses this mechanism to automatically calculate values for metrics as the code for the project is modified and to store the values of the metrics with the source code in the SCM repository. The values are used to provide an insight into the quality of the product under development. The benefit of this approach is that it uses processes and concepts that are generally part of the software development process.

The metrics proposed here are defined based on metrics intrinsic to most software engineering projects: the proportion of the number of requirements met to the total number of requirements and the proportion of the number of tests that pass to the total number of tests. The definitions of these metrics are as follows.

\begin{equation}
R =  \frac{ R_{met} }{ R_{total} }
\label{eq:eq1}
\end{equation}
where $R_{met}$ and $R_{total}$ are the number of requirements claimed to have been met and the total number of requirements, respectively. The concept of using the proportion of total requirements to the requirements met or defects is also discussed in \cite{paper:req-metrics}, \cite{propreq1} and \cite{propreq2}.

\begin{equation}
T = \frac{ T_{run} - D }{ T_{total} }
\label{eq:eq2}
\end{equation}
where $T_{run}$ is the number of tests run (pass or fail), $T_{total}$ is the total number of tests defined, and $D$ is the total number of defects raised by the tests that have run. Each test can give rise to zero or one defects.

$R$ provides an indication of the completeness of the project. $T$ provides an indication of how much the value of $R$ is validated by tests that pass; a confidence factor of sorts. The value of $R$ could be high because all of the requirements are stated as met, but without the tests to corroborate that claim, the quality cannot be said to be high. These metrics are combined to give a view of quality, $Q$, as below. The concept of using the proportion of tests passing to failing is discussed in \cite{relatedwork:testspass} and \cite{relatedwork:testspass2}.

\begin{equation}
Q =  \alpha \times R + (1 - \alpha) \times T
\label{eq:eq3}
\end{equation}
where $\alpha$ is the weight of the value of $R$ and $0 < \alpha < 0.5$. The value of $\alpha$ is less than 0.5 because the relative importance of $T$ is higher as it provides a confidence factor for the value of $R$. 

As indicated in Equation~\ref{eq:eq3}, $Q$ is calculated as a linear combination of $R$ and $T$ where the sum of the weights (or co-efficients) is 1. The reason for this is so that different importance might be placed on each of $R$ and $T$. For example, a relatively low value of $\alpha$ would ensure that $Q$ remains realistic in a context where a high proportion of requirements have been met but a low proportion of tests passing have been passed. Theoretically, the value of $\alpha$ is between 0 and 1. Here however $\alpha$ is constrained between 0 and 0.5, so that the value of $T$ is given a higher weight than the value of $R$ in the calculation of $Q$ because the weighted value of $R$ will always be less than 0.5. The values of $R$ and $T$ range between 0 and 1 and hence $Q$ ranges between 0 and 1.

The next section describes \emph{Metaversion}, an implementation of the approach described here using Subversion as an SCM repository.

\section{Metaversion}
\label{section4}
This section describes an implementation of the approach described in Section~\ref{section3} called \emph{Metaversion}.

\subsection{Collecting source file meta-data}
\label{section:collectingarts}

As mentioned previously in this section, for each of the source files in a software project, the following information must be stored (as per the assumptions noted in Section~\ref{section:qassumptions}):

\begin{itemize}
\item details of the tests that include or act upon the source file;
\item details of the defects that the source file participate in --- or contribute to; and
\item details of the requirements that are claimed (and are proven using tests) to be met by the source file.
\end{itemize}

It should be borne in mind that there may need to be significant effort expended in order to determine which file gives rise to a defect, but this effort would have to be expended in the course of fixing the defect anyway.

Using Subversion, which has meta-data capabilities using \emph{properties}, the required meta-data can be stored. The approach discussed here does not rely on Subversion being used as the SCM system. The approach would work with any SCM system that supports setting arbitrary metadata on individual files in the repository. Subversion was chosen for this implementation, although there are other SCM systems such as Git\cite{Pragmatic:GIT} that could be used.

Subversion properties can have any name and are version-controlled along with the file itself. This means that over time, as the contents of the file change and requirements, tests and defects are associated with it, the values of the properties also change. The property names and values that are to be set against each version of a source file (as applicable) are listed below. Note that the identifiers referred to below are the test numbers, requirements numbers or defect numbers that identify each of the tests, requirements or defects in the project.

\begin{description}
\item[mv:tests-pass] A list of identifiers for the tests that have run and are known to have passed for this version of the file.
\item[mv:tests-total] A list of identifiers for all of the tests that are associated with this version of the file.
\item[mv:tests-fail] A list of identifiers for the tests that have run and are known to have failed for this version of the file.
\item[mv:defects-open] A list of identifiers for the defects that have been reported against this version of the file and are as yet unresolved.
\item[mv:defects-closed] A list of identifiers for the defects that have been reported against this version of the file and are resolved.
\item[mv:req-met] A list of identifiers for requirements that have been proven to be fulfilled by this version of the file.
\item[mv:req-unmet] A list of identifiers for requirements that have been proven not to be fulfilled by this version of the file (e.g. after a defect has been raised against a previously met requirement).
\item[mv:req-total] A list of identifiers for requirements that this version of the file is supposed to fulfill.
\end{description}

The \emph{mv:req-total} property would be set based on the requirements specification---hence the assumption in Section~\ref{section:qassumptions} that, for ease of reference, there should be an enumerated list of requirements. Based on these requirements, a set of tests would be created, and the \emph{mv:tests-total} property would be set from these.

If testing were an automated process, the setting of the \emph{mv:tests-pass} and \emph{mv:tests-fail} properties could be done by the software tool performing the testing. If a defect management system was in use; when a defect is raised, it could set the value of the \emph{mv:defects-open} property. Similarly, when a defect was resolved, the \emph{mv:defects-closed} property could be set. The \emph{mv:req-met} and \emph{mv:req-total} properties could have their values automatically set by some requirements management tool, if one were in use. If there are details about which tests are related to which requirements, then details about met and unmet requirements can be inferred from passed and failed tests respectively. The properties \emph{mv:req-unmet} and \emph{mv:tests-fail} are used to store this information.

If no automated software is in use for these areas of the software project, the values could be set manually as part of the planning (for requirements and tests) and development and testing (for tests and defects).

\begin{figure}[h!]
  \centering
  \includegraphics[width=3in]{gathering-props.png}
  \caption[Gathering source file meta-data]%
  {Gathering source file meta-data}
  \label{fig:gathering-props}
\end{figure}

Figure~\ref{fig:gathering-props} depicts the collection of the properties from the various sources in the software project.

%\newpage

\subsection{Using source file meta-data}

The value of storing the property meta-data is in the analysis thereof. Using the values of these properties and the quality equation described in Equation~\ref{eq:eq3} (and any other metrics that can be derived from these properties), it is possible to get some idea of the quality of a particular source file, but perhaps of more value is the collective quality of the set of source files: the quality of the software product as a whole. This approach (as mentioned before) is not prescriptive regarding the mechanism for extracting the meta-data nor the tools to use to do it. This implementation is a suggestion of how this could be done.

One approach to using the source file meta-data would be to step through every file in the repository, read the values of the properties and build a report from that data. As the size of the repository grows, however, this approach would take longer to produce results because there are more source files for which to store meta-data and hence more meta-data.

A more scalable approach would be to have the Subversion repository react when a property is set on a file (and that property is one of the properties mentioned in Section~\ref{section:collectingarts}) by storing the data in some intermediate location. The Subversion application provides this event-based mechanism through the use of \emph{hooks}. These hooks allow custom applications to run when some activity occurs in the repository. The hook of interest to Metaversion is the \emph{post-commit} hook. This hook is triggered when any file or change to a file is committed to the repository (including property changes). These hooks allow for an event-driven approach to the analysis of the meta-data as opposed to a polling approach. The data needed to perform analysis and reporting will be built up over time and the reporting application need only consult the intermediate store of data to build reports and not the entire Subversion repository.

\begin{figure}[h!]
  \centering
  \includegraphics[width=3in]{post-commit-hook.png}
  \caption[Reacting to post-commit hook]%
  {Reacting to post-commit hook}
  \label{fig:post-commit-hook}
\end{figure}

Figure~\ref{fig:post-commit-hook} shows how the properties set on source files in the repository will be stored in an intermediate database. The post-commit hook will be configured to call an application. This application is called \emph{MetaversionCommitHandler}, and was produced as part of this study.

The meta-data stored with source files will be extracted using a reporting application. This reporting application will gather the meta-data from the repository and produce reports that will aid the development team, managers and sponsors of the software product to gain insight into the quality performance of the software. This reporting application was called \emph{MetaversionReporter} and was produced as part of this study.

\subsection{Metaversion Data Model}
Metaversion reflects the relationship between tests, defects and requirements as indicated in Figure~\ref{fig:mvdatamodel}. Each file can have zero or more tests, zero or more defects and zero or more requirements associated with it.

\begin{figure}[h!]
  \centering
  \includegraphics[width=3in]{metaversion-ri-class.png}
  \caption[Metaversion data model]%
  {Metaversion data model}
  \label{fig:mvdatamodel}
\end{figure}

%\newpage

\subsection{MetaversionCommitHandler}

When a user (developer, manager, sponsor, etc.) sets a property on a file in the repository and commits the changes to the Subversion repository, the repository invokes the post-commit hook. The hook, in turn, invokes the \emph{MetaversionCommitHandler} application. The \emph{MetaversionCommitHandler} application calculates which properties have changed (in particular, the properties from the list in Section~\ref{section:collectingarts} for each file for which properties have been set). A MySQL \cite{mysqlmanual} database is updated with the details stored in the properties: requirements, tests and defects are associated with files, as appropriate. Figure~\ref{fig:mvcommitprocess} shows this process. \emph{MetaversionCommitHandler} is used in the case study in Section~\ref{section5}.

\begin{figure}[h!]
  \centering
  \includegraphics[width=3in]{metaversion-ri-commit.png}
  \caption[Metaversion commit process]%
  {Metaversion commit process}
  \label{fig:mvcommitprocess}
\end{figure}

%\newpage

\subsection{MetaversionReporter}
\label{section:mversionreporter}
As mentioned, the value of the meta-data stored in the intermediate database is in the analysis thereof. The \emph{MetaversionReporter} application is run by a member of the team developing the software in order to gain insight into the quality of the product.  (Here no specific position is taken about whether this role should be assigned to the developer, the manager, the sponsor, etc.) The \emph{MetaversionReporter} application reads the information from the MySQL database that is populated by the \emph{MetaversionCommitHandler} application. This information is then used to calculate the value of $Q$ (as defined in Equation~\ref{eq:eq3}). The information is then displayed to the user. This process is depicted in Figure~\ref{fig:mvreportprocess}. \emph{MetaversionReporter} is used in the case study in Section~\ref{section5}.

\begin{figure}[h!]
  \centering
  \includegraphics[width=3in]{metaversion-ri-report.png}
  \caption[Metaversion reporting process]%
  {Metaversion reporting process}
  \label{fig:mvreportprocess}
\end{figure}

The next section presents a case study where this approach was implemented using the implementation described in this section.

\section{Case Study}
\label{section5}
This section describes a small case study that was performed using an implementation, called Metaversion, of the approach described in Section~\ref{section4}. The value of $Q$ as defined in Section~\ref{section4} is provided across the 6 interim products that the project yielded. For all calculations of $Q$, a value of 0.3 was used for $\alpha$---i.e.~the proportion of tests run and consequent defects exposed was weighted more heavily than the proportion of requirements met.  A survey was used as a comparison against the values of $Q$ that were automatically calculated after each interim product. 

The case study was performed on a software project called Timezonr\footnote{The TimeZonr application can be found at http://lnkr.co.za/timezonr}, which is an Android\cite{Android} application. The application is a small application of around 1500 lines of code. The application was written by a single developer---the first author of this paper. It was tested by two independent people---i.e. non-authors of the paper. The testers answered the survey questionnaire after each interim build, and from this questionnaire, an indication of ``perceived quality'' was found. This ``perceived quality'' was deduced, along with an indication of the completeness of the product and the confidence that the respondents of the survey had in their perception of completeness. These responses were compared with the calculated value for $Q$, $R$ and $T$, respectively.

\begin{figure}[h!]
  \centering
  \includegraphics[width=3in,height=3in]{case-study-graph-quality.eps}
  \caption[Quality: perceived vs. calculated]%
  {Quality: perceived vs. calculated}
  \label{fig:csg-quality}
\end{figure}
\begin{figure}[h!]
  \centering
  \includegraphics[width=3in,height=3in]{case-study-graph-completeness.eps}
  \caption[Completeness: perceived vs. calculated]%
  {Completeness: perceived vs. calculated}
  \label{fig:csg-completeness}
\end{figure}
\begin{figure}[h!]
  \centering
  \includegraphics[width=3in,height=3in]{case-study-graph-confidence.eps}
  \caption[Completeness confidence: perceived vs. calculated]%
  {Completeness confidence: perceived vs. calculated}
  \label{fig:csg-confidence}
\end{figure}
Figures~\ref{fig:csg-quality}, \ref{fig:csg-completeness} and \ref{fig:csg-confidence}  show the values for $Q$, $R$ and $T$, respectively, as calculated by the approach, against the average responses of the survey.

Figure~\ref{fig:csg-quality} indicates that the perceived quality increased as the product was developed. This implies that the perception of the testers was that as the project was being developed, the product produced got closer to what they wanted. The calculated quality $Q$ also increased as the product was being developed. This is because, as the product was developed, more of the requirements were being met, and more of the tests indicating adherence to requirements were passing.

Figure~\ref{fig:csg-completeness} indicates how the completeness of the product was perceived by the testers and the calculated values of completeness. In general, both perceived and calculated completeness increased over the course of the project, and this indicates that testers believed that the project was progressing towards a state of completion, and that the number of requirements (which were known to the testers) being met was increasing over the course of the project. At build 4 and 5, the perceived completeness was less than the calculated completeness. This could well be because of the combination of the testers' understanding of the term \emph{completeness} and the discrete interval from which they had to choose values for completeness.  The calculated and perceived values were only equal once other than the start and end of the project. At build 4, the calculated completeness was 1.0, which implies that all requirements were claimed to have been met. The perception of the respondents, however, was that the project was not complete. Since the respondents knew the requirements, this difference could be attributed to the presence of defects. The respondents knew that the software had defects, and they may have surmised therefore that the software was incomplete.

Figure~\ref{fig:csg-confidence}  indicates that the confidence in completeness increased as the project progressed. The testers perceived that they were more sure of their assessment of completion as the project progressed. This could be due to familiarity with the product, and an increase in understanding of how the application worked. In addition to this, this measure indicates that the testers felt they had enough of a view of the completeness of the product to be sure of it. For the calculated confidence in completion, this indicates that a higher proportion of the tests that were being run were passing, which lends some degree of confidence to the completeness of the product. The values of the completeness confidence from the respondents were all high. This could be attributed to two things. Firstly, the responses only had discrete values, and so the respondent would not have been able to give an answer inbetween the values. Thus, they may have, for example, chosen 4 out of 5, when they actually wished to give a response of 3.5. Secondly, the respondents knew the requirements of the build when they were testing, so they could actually prove their answer to the completeness question. Since all of the responses were not 5 out of 5, this could imply they did not do this, or it could imply that there was some other factor influencing their understanding of the question.

These figures show that, in this exercise, the ``perceived quality'' of the product had a fairly close similarity to the metric calculated by the new approach. This supports the view that the metric $Q$ could be used to gain some insight into the quality of the software product. 

\section{Conclusion}
\label{section6}

This paper provided an overview of approaches to gaining insight into the quality of software products and pointed to the fact that there appears to be a deficit in quality metrics that can be easily and pragmatically gathered. A pragmatic approach was proposed in Section~\ref{section3} to provide software producers with insight into product quality based on processes, systems and artifacts usually present in a software project. The case study described in Section~\ref{section5} is seen as a limited pilot study to test the feasibility of implementing the proposed approach \emph{in situ}.

The case study is clearly limited in terms of project size, number of testers and the self-involvement of one of the authors. It therefore does not claim to show universal applicability of the approach proposed. Nevertheless, it does provided evidence that quality as perceived by non-developers (one of the testers was actually also the client of the product) coheres with the proposed metrics. Of course, more evidence as to the level of insight the approach provides into software quality should be attained by implementing the approach across a wider variety of projects of varying sizes.

Executing the implementation of this approach also brought to light further possibilities for future work to improve the approach. The most immediate are as follows:
\begin{itemize}
\item As stated, the approach requires that the implementor sets requirements, tests and defects against each file in the repository. This approach still requires effort on the part of the implementor. Future work could be done to automate this process, by integrating requirements gathering software and defect tracking software. Such integrations do exist, for example Scmbug \cite{scmbug}. 

\item Currently, the approach proposes only three metrics, and further work could be done to provide further metrics based on the SCM repository. An example could be productivity (perhaps based on the number of lines of code submitted per day). 

\item Currently, the approach proposes Subversion as an SCM, and future work could investigate making the approach more generic so that any SCM could be used. 
%In addition to these, more evidence as to the level of insight the approach provides into software quality could be attained by implementing the approach across a wider variety of projects of varying sizes.
\end{itemize}
From a broader perspective, the study highlights the possibilities of using version control systems more effectively for gathering statistical evidence to monitoring quality as software projects progress through their life-cycles.

\bibliography{Journal-Article-DanielActon-2012}

\end{document}