The most important aspect in any measurement program is to very explicitly understand and define the purpose of the measurements. Therefore, it should be clear what is the property that is measured, how the measurement data will be used and what is the purpose and goal of the measurements. Common goals of quality measurements include:

• give feedback for developers and track progress
• set standard thresholds and identify deviations
• compare with other persons, teams or organizations

The target of quality measurement can be either the development process or the developed software or system product. The process metrics are typically indirect indicators of software quality and more typical in traditional, document-driven processes. The process metrics are used to ensure that the defined processes are followed. Product metrics, instead, aim at measuring the achieved product quality by more direct means.

Measurement data can be used on many organizational levels. The simplest is when individual developers, for example, use the data for immediate feedback for their own tasks. The next level is the team level where the data is used inside a team to track, control, or improve the quality or practices by the team itself. When the data is used outside the team, the situation is always more sensitive and also the visibility of how the data is used and by whom requires more careful attention. The frequency of monitoring the measurements results also needs to be considered. It is dependent on the purpose and needs of the collected measurement data. For example, for individual developers, the feedback loop on the low level code quality must be as short as possible, but on the project level quality aspects it might be enough to get the data within some weeks after the project has completed.

The degree of automation in data collection affects the required effort in data collection as well as the availability and reliability of the measurement data. The more automated the data collection and presentation can be made the more reliably and frequently the data will be available. In becomes also important to recognize that the measurability of quality goals can differ significantly, see attached flyer the different types of measurability

When measurements target individual people and their activities the data is always sensitive. It is important to decide on what level of the organization the data is used and who gets access to it. People must know who is using the measurements and for what purpose. Especially if people feel that they are evaluated, compared, and judged based on some metrics they change their behaviour, intentionally or not, so that the measured attribute gets better - or what they think is better.

Different stakeholders have different concerns. Inviting them to express their concerns and define relevant quality objectives is likely to add to the range of concerns that are relevant for a given project. A Quality Objective Setting Method is a collaborative way of identifying a relevant set of quality objectives and designing a quality model for an organization or project (see description of a QOS method).

A quality viewpoint represents the key concerns of a particular stakeholder or stakeholder group while the quality objective characterizes the software product or process from some quality viewpoint. Viewpoints are a highly abstract idea that allows us to capture a wide variety of issues, aspects and concerns. Defining the quality viewpoints helps to determine:

1. The point in time at which predictions should be carried out and, therefore, what information will be available to define abstractions and measures;
2. The definition of descriptive models for the quality focus, that is the relevant dependent attributes: For example, from the user’s point of view, reliability may be defined as the mean time to failure, whereas, from the tester’s point of view, it may be defined as the number of faults detected over a period of time;
3. What information is costly or difficult to acquire and, consequently, what information should possibly be left out of the model.

Viewpoints are a highly abstract idea that allows us to capture a wide variety of issues, aspects and concerns. What we want to achieve with this variety, is to support the many ways in which viewpoints can be useful in a concrete software project. Each stakeholder has a potentially unique set of concerns. Thus, before embarking in creating a quality model framework, the organization – in cooperation with the project – needs to agree on how this variety is exploited. Two relevant questions for this discussion would be:

• Does the project want to create a unified set of concerns?
• Does the project need to eliminate overlaps in viewpoints, or are overlaps actually a good approach to draw out the relevant concerns towards QA work in the project?

It may be difficult for software organizations to establish good goals from a business perspective and to clearly identify the business relevance of the metrics. Hence, it may be a good idea not to conflate quality viewpoints, but rather elaborate upon the various roles and concerns of the different stakeholders, preferably with rationales etc.

In short the Goal/Question/Metric (GQM) technique is about defining specific improvement goals, ask concrete questions, define metrics, collect measurement data, answer questions and draw conclusions on whether the goals were reached or not. Goal definition in GQM asks for specific viewpoints from which the quality will be analysed. Example viewpoints in use are ‘user’, ‘customer’, ‘manager’, ‘developer’, ‘testers’, ‘development team’ and ‘corporation.’ A GQM model “(The structure of a GQM model)” contains information necessary to implement goal oriented measurement and analysis. The elements in the model are goals, questions and metrics.

• A goal (see image below) specifies an analytical task that should be achieved using measurement:

“Analyze the

• A question expresses a need for information. The answers should contribute to reaching the goal. The form of the questions depends on what we want to achieve. We need information both on quality focus – what we want to know something about – and on the environment – for later interpretation and reuse.

• A metric is a quantitative specification of data or information. Metrics are derived from the questions and contribute the data required for answering them. One or more metrics are needed for each question.

The validators’ job relies on his/her personal experience and knowledge of the actual project. The check requires a good understanding of the content and intent of the metric and of the environment in which the data were collected.

Validating data should be done as quickly as possible after the collection. When defining indicators and metrics for a quality objective, an important consideration is the reliability and validity of the defined metrics. (See Evaluating quality metrics.). Before data can be used in measuring, it needs to be validated. This is because a measurement needs accurate and consistent data. In software projects, validation of data is especially important in the startup period, before all project members have reached a common understanding of the metrics.

The reliability of measures is threatened by all aspects that can cause random or systematic deviations in the measurement data. Such issues affecting the reliability of measurements are, for example, procedures and instruments used for collecting measurement data and the source of data. Human collected or reported data can suffer personal bias or perspective of the collector/reporter. Also the skills, experience, and motivations of the individuals affect the data.

#