Continuously assessing and improving software quality with software analytics tools: a case study
Silverio Martínez-Fernández, Anna Maria Vollmer, Andreas Jedlitschka, Xavier Franch, Lidia López, Prabhat Ram, Pilar Rodríguez, Sanja Aaramaa, Alessandra Bagnato, Michał Choraś and Jari Partanen
Abstract. In the last decade, modern data analytics technologies have enabled the creation of software
analytics tools offering real-time visualization of various aspects related to software development and
usage. These tools seem to be particularly attractive for companies doing agile software development.
However, the information provided by the available tools is neither aggregated nor connected to higher
quality goals. At the same time, assessing and improving software quality has also been a key target for the
software engineering community, yielding several proposals for standards and software quality models.
Integrating such quality models into software analytics tools could close the gap by providing the
connection to higher quality goals. This study aims at understanding whether the integration of quality
models into software analytics tools provides understandable, reliable, useful, and relevant information at
the right level of detail about the quality of a process or product, and whether practitioners intend to use it.
Over the course of more than one year, the four companies involved in this case study deployed such a tool
to assess and improve software quality in several projects. We used standardized measurement instruments
to elicit the perception of 22 practitioners regarding their use of the tool. We complemented the findings
with debriefing sessions held at the companies. In addition, we discussed challenges and lessons learned
with four practitioners leading the use of the tool. Quantitative and qualitative analyses provided positive
results; i.e., the practitioners’ perception with regard to the tool’s understandability, reliability, usefulness,
and relevance was positive. Individual statements support the statistical findings and constructive feedback
can be used for future improvements. We conclude that potential for future adoption of quality models
within software analytics tools definitely exists and encourage other practitioners to use the presented seven
challenges and seven lessons learned and adopt them in their companies.
