Penalty1vs0.tex


Data Quality Assessment of Maintenance Reporting Procedures

Manik Madhikermia, Sylvain Kublerb,∗, Jérémy Robertb, Andrea Budaa, Kary Främlinga

aAalto University, School of Science

P.O. Box 15400, FI-00076 Aalto, Finland
bUniversity of Luxembourg, Interdisciplinary Centre for Security, Reliability & Trust

4 rue Alphonse Weicker L-2721 Luxembourg

Abstract

Today’s largest and fastest growing companies’ assets are no longer physical, but rather digital (software, al-

gorithms. . . ). This is all the more true in the manufacturing, and particularly in the maintenance sector where

quality of enterprise maintenance services are closely linked to the quality of maintenance data reporting pro-

cedures. If quality of the reported data is too low, it can results in wrong decision-making and loss of money.

Furthermore, various maintenance experts are involved and directly concerned about the quality of enterprises’

daily maintenance data reporting (e.g., maintenance planners, plant managers. . . ), each one having specific needs

and responsibilities. To address this Multi-Criteria Decision Making (MCDM) problem, and since data quality is

hardly considered in existing expert maintenance systems, this paper develops a Maintenance Reporting Quality

Assessment (MRQA) dashboard that enables any company stakeholder to easily – and in real-time – assess/rank

company branch offices in terms of maintenance reporting quality. From a theoretical standpoint, AHP is used

to integrate various data quality dimensions as well as expert preferences. A use case describes how the pro-

posed MRQA dashboard is being used by a Finnish multinational equipment manufacturer to assess and enhance

reporting practices in a specific or a group of branch offices.

Keywords: Data Quality, Information Quality, Multi-Criteria Decision Making, Analytic Hierarchy Process,

Decision Support Systems, Maintenance.

1. Introduction

Data and Information quality is one of the most

competitive advantages for an organization in today’s

digital age, for example, with the rapid evolution of

Internet of Things, Industry 4.0, Big Data and Cloud

Computing (Xu et al., 2010; Chen et al., 2014). Com-

panies are trying hard to find out relevant strategies to

make their products (physical or virtual) standout with

respect to their competitors. Quality improvement of

products, processes and services requires the collec-

tion and analysis of data to solve quality-related prob-

lems (Li et al., 2015; Köksal et al., 2011). Companies

need to provide after-sales services such as mainte-

nance and warranty services to ensure that the deliv-

ered product is reliable and in full accordance with the

customer requirements. Nonetheless, providing such

services inevitably generate costs for businesses (Fang

& Huang, 2008). As indicated by Mobley (2002),

one third of all maintenance costs is wasted as the

∗Corresponding author

Email addresses: manik.madhikermi@aalto.fi (Manik

Madhikermi), sylvain.kubler@uni.lu (Sylvain Kubler),

jeremy.robert@uni.lu (Jérémy Robert),

andrea.buda@aalto.fi (Andrea Buda),

kary.framling@aalto.fi (Kary Främling)

result of unnecessary or improper maintenance prac-

tices. More recent studies have confirmed that mainte-

nance is a major cost issue, with a ratio between main-

tenance costs and added-value higher than 25% in

some sectors (Sophie et al., 2014). In fact, data qual-

ity practices – including maintenance reports – have

a considerable impact on maintenance tasks, risks and

business performance since poor data quality results

in losses across a number of fronts (Arputhamary &

Arockiam., 2015), and reciprocally, high data qual-

ity fosters enhanced business activities and decision-

making.

A successful maintenance program often relies on

a detailed planning and intelligent decision-making

support systems. This is all the more true given

that planning maintenance involves managing a set of

complex tasks and resources to guarantee the max-

imum possible operational availability of equipment

(Palma, 2010). Various stakeholders with different re-

sponsibilities are involved in this management, such

as (i) Maintenance planners who are responsible for

scheduling planned maintenance activities; (ii) Plant

managers who are responsible for cost reporting and

savings; (iii) Maintenance managers who are respon-

sible for the execution of planned/unplanned mainte-

nance activities, and so on. All these experts have

Preprint submitted to Elsevier June 28, 2016



a common goal: reducing maintenance downtime to

increase productivity. In this respect, they usually

make use of maintenance reports as decision support

tools, which contain useful record information such

as technical maintenance logs, asset location, descrip-

tion of defect location codes, scheduled maintenance

date, etc. It is thus of importance to develop and im-

plement strategies for enhanced reporting practices,

data quality control and management (Jones-Farmer

et al., 2014). Nonetheless, requirements related to the

data and associated quality attributes are tightly cou-

pled with the stakeholder’s needs and responsibilities.

For example, maintenance managers pay more atten-

tion to technical log records for their daily decision-

making, whereas plant managers rather use defect and

asset location-related information to manage their in-

ventory. All this provides irrefutable evidence of the

complexity of developing a flexible, intelligent and

integrated decision-making support system for data

quality assessment and maintenance management; it

implies to take into consideration various stakeholder

roles, needs, quality dimensions, and other techni-

cal and organizational aspects (Vujanovićet al., 2012;

Shafiee , 2015). Given the Multi-Criteria Decision

Making (MCDM) nature of the problem, this paper

investigates and develops a Maintenance Reporting

Quality Assessment (MRQA) tool, whose underly-

ing framework relies on AHP. The primary goal of

this tool is to help companies to dynamically assess

quality of daily maintenance data reporting activities,

while taking into account specific needs or role of the

end-user (i.e., a company stakeholder).

The summary of the paper is as follows: Sec-

tion 2 conducts a thorough literature review of both

(i) existing expert maintenance systems making use

of MCDM techniques, and (ii) existing data qual-

ity frameworks, against which our research is moti-

vated. Section 3 provides insight into the research

methodology underlying the MRQA framework/tool

development. Section 4 thoroughly details the MRQA

framework and underlying mathematical theory. Sec-

tion 5 describes a use case that shows how the pro-

posed MRQA decision-making support dashboard is

being used by a Finnish multinational Original Equip-

ment Manufacturer (OEM) company to assess and

rank company branch offices in terms of maintenance

reporting quality. Conclusions, implications, limita-

tions and future research are discussed in Section 6.

2. Data quality in expert maintenance systems

To understand how crucial and complex it is to

properly address data quality in maintenance settings,

section 2.1 discusses the key maintenance business

levels, along with previous research works that have

used MCDM techniques to address challenges at each

of these levels. Section 2.2 discusses existing frame-

works for data quality analysis and management in

maintenance processes.

2.1. Expert maintenance systems

Maintenance is a complex process that is usually

triggered by an equipment failure or planned repair.

This process requires planning, scheduling, control-

ling as well as deploying maintenance resources to

perform the necessary maintenance actions (Duffuaa

et al., 2001). Adopting an efficient approach to or-

ganize maintenance management (MM) activities is a

prerequisite to its success. Several MM frameworks

have been developed and applied for this purpose,

one of the earliest being put forward by Pintelon &

Gelders (1992) who pointed out three important busi-

ness levels in the decision-making process, including

the (i) Operational level: decision regarding market-

ing and finance; ii) Planning & control level: deci-

sions regarding resource and scheduling management,

and performance reporting; iii) Managerial level: de-

cisions regarding how to optimize actions and poli-

cies to be performed on-site. Later on, Levrat et al.

(2008) proposed a similar three business level-based

MM framework, namely:

• Strategic level: strategic axis are expressed in

quantitative and qualitative terms, and organi-

zational maintenance strategies are defined such

as corrective and preventive maintenance, risk-

based or condition-based maintenance, etc.;

• Tactical level: maintenance actions such as

scheduling and resource planning are planned;

• Operational Level: actual work is carried out in

addition to access performance and future equip-

ment conditions.

Making decisions at each of these three levels im-

plies dealing with multiple, conflicting, and incom-

mensurate criteria and/or objectives, as well as hu-

man judgments. Research on human judgements and

decision making shows that the human brain is able

to consider only a limited amount of information at

any one time (Simpson, 1996), which makes it unreli-

able to take decisions when facing complex problems.

MCDM techniques, such as AHP, TOPSIS, ELEC-

TRE, PROMETHEE, Fuzzy MCDM, etc., have been

proven to be of great value in supporting decision-

makers at each MM level, as summarized in Table 1.

At the “Strategic level”, MCDM techniques are

considered for various purposes, including (i) main-

tenance policy selection, (ii) tool/contractor selection,

and (iii) cost estimation. Table 1 provides an “at a

glance” overview of scientific papers that have made

use of MCDM techniques for each of these purposes.

Bevilacqua and Braglia (2000); Wang et al. (2007);

2



Table 1: MCDM Techniques Applied in Maintenance Industry
AHP FAHP ANP DEA VIKOR ELECTRE PROMOTHEE MAUT

S
tr

a
te

g
ic

L
e
v
e
l

Maintenance Policy Se-

lection

(Bertolini and

Bevilacqua, 2006;

Bevilacqua and

Braglia, 2000;

Goossens et al.,

2015; Labib et al.,

1998; Pramod et

al., 2007; Shyjith et

al., 2008; Tan et al.,

2011; Zaim et al.,

2012)

(Azizi et al., 2014;

Ilangkumaran & Ku-

manan, 2009; Hos-

seini et al., 2015;

Ferdousmakan et al.,

2014; Wang et al.,

2007; Fouladgar et

al., 2012)

(Kumar & Maiti,

2012; Shahin et al.,

2012; Pourjavad et

al., 2013; Zaim et al.,

2012)

Azadeh et al. (2014);

Sheikhalishahi

(2014)

(Ilangkumaran &

Kumanan, 2012;

Ahmadi et al., 2010)

(Zhangqiong &

Guozheng , 1999; Li

et al., 2007)

(Emovon et al. ,

2015; de et al.,

2015c; Monte et al.,

2015)

Tools and companies Se-

lection

(Bertolini et al.,

2004; Garcı́a-

Cascales et al., 2009;

Ha et al., 2008;

Triantaphyllou et al.,

1997)

(Durán, 2011) (Ha et al., 2008) (dGonçalves et al.,

2014; Gomez et al.,

2011)

(Kuo et al., 2012; de

et al., 2015a)

Cost Estimation (Chou , 2009) (Chen et al., 2005)

T
a
c
ti

c
a
l

L
e
v
e
l

Maintenance prioritiza-

tion

(Farhan & Fwa,

2009; Moazami et

al., 2011; Taghipour

et al., 2011)

(Ouma et al., 2015) (Wakchaure & Jha,

2011)

(Liu et al., 2012;

Cafiso et al., 2002;

Hankach et al., 2011;

Trojan & Morais,

2012b)

(Monte & de

Almeida-Filho,

2016; de et al.,

2015b)

Resource Planning (Azadeh et al., 2013) (Cavalcante et al.,

2010; Almeida et al.,

2013)

(Garmabaki et al.,

2016; de Almeida.,

2001; Liu & Fran-

gopol, 2006)

Maintenance Scheduling (Coulter et al., 2006;

Eslami et al., 2014)

(Van et al., 2013) (Certa et al., 2013) (Almeida, A. T.,

2012)

O
p
e
ra

ti
o
n
a
l

L
e
v
e
l

Critical Component Iden-

tification

(Dehghania et al.,

2012)

(Cavalcante et al. ,

2007, 2010)

Measuring/Assessment

Efficiency

(Wang et al., 2010) (Muchiri et al.,

2011; Vujanovićet

al., 2012; Van &

Pintelon, 2014)

Sun (2004); Peck

et al. (1998); Ozbek

et al. (2010a,b);

Hjalmarsson et al.

(1996); Liu & Yu

(2004); Rouse et al.

(2002); Fallah-Fini

et al. (2015); Jeon et

al. (2011); Roll et al.

(1989); Charnes et

al. (1984)

(de un Caso, 2008) (e Costa et al., 2012)

Maintenance Action Se-

lection

(Kumar & Maiti,

2012)

(Alarcón et al., 2007;

Thor et al., 2013)

3



Tan et al. (2011); Fouladgar et al. (2012) developed

MCDM-based maintenance policy selection frame-

works taking into account maintenance cost, added-

value and safety dimensions. Shahin et al. (2012)

rather focused on the selection of appropriate (opti-

mum) maintenance strategies, paying special attention

to reliability, availability and maintainability criteria

and potential interdependencies (via ANP). Gomez

et al. (2011); Durán (2011) developed a similar ap-

proach, considering the same criteria, but rather ap-

plying ELECTRE II and FAHP respectively. Select-

ing appropriate tools and/or contractors for outsourc-

ing activities plays also an important role at the strate-

gic level, as it affects the whole maintenance manage-

ment process. In this respect, Bertolini et al. (2004)

developed an AHP-based outsourcing service selec-

tion model considering maintenance-related criteria.

Maintenance budgeting and cost estimation are other

important strategic decisions that need to be properly

managed. To this end, Chou (2009) and Chen et al.

(2005) develop two distinct utility-based assessment

approaches, respectively relying on AHP and ELEC-

TRE II, which enable decision-makers to estimate –

based on historical data of similar projects – pave-

ment and pipeline maintenance costs.

Looking at the “Tactical level” now, MCDM tech-

niques are mainly applied for maintenance work plan-

ning purposes, which includes (i) task prioritization,

(ii) task scheduling, and (iii) resource planning. Ta-

ble 1 reports some scientific papers that have made

use of MCDM techniques for each of these purposes.

Cafiso et al. (2002); Farhan & Fwa (2009); Moazami

et al. (2011); Ouma et al. (2015) and Babashamsi et al.

(2016) have all studied prioritization of road mainte-

nance with the objective to reduce the overall cost (cri-

teria considered in this studies being traffic volume,

road safety, pavement width. . . ). Other studies such

as (Trojan & Morais, 2012a,b; Monte & de Almeida-

Filho, 2016) developed MCDM-based frameworks for

maintenance prioritization in the context of water sup-

ply networks, looking at strategies for reducing costs

and water losses. Taghipour et al. (2011) devel-

oped a framework in the context of healthcare main-

tenance management for medical equipment prioriti-

zation, considering mission criticality, age, risk, re-

call and hazard alerts as main prioritization criteria.

Resource planning is also a very critical aspect to be

tackled at the tactical level, as resources can be either

human or non-human in nature. For example, Van et

al. (2013) develop a three-stage approach of personnel

rostering (i.e., human scheduling) for aircraft mainte-

nance, whereas de Almeida. (2001) are seeking to op-

timize spare-provisioning (i.e., non-human resource

allocation).

Finally, at the “Operational level”, MCDM tech-

niques are often applied for (i) task efficiency assess-

ment, (ii) Critical component identification, and (iii)

Maintenance action selection. Table 1 shows that

most of the papers implement a data envelopment

analysis (DEA) for task efficiency assessment, which

is a well-known tool for benchmarking in operations

management. The identification of critical compo-

nents is also very important to addressed to offer en-

hanced predictive maintenance services (Cavalcante

et al. , 2010; Dehghania et al., 2012). Along with crit-

ical component identification comes the challenge of

making the right decisions and actions on the field to

avoid causing any disruption, delay or monetary loss.

A few studies have been using MCDM techniques

to select the best maintenance action(s) on-site, such

as Nyström & Söderholm (2010) who are seeking to

improve railway track maintenance practices, or still

Alarcón et al. (2007) who apply ELECTRE for mini-

mizing telecommunication network disruption during

maintenance activities.

Given the significant number of papers discussed

above and classified in Table 1 (40 papers at the

Strategic level, 19 at the Tactical level, and 20 at the

Operational level), we feel it is appropriate to analyze

and identify criteria that are the most commonly used

at each MM level, which will therefore help us to state

whether or not existing studies takes into account data

quality-related criteria. The outcome of our analysis

shows that the three most commonly used criteria at

each MM level are (see Appendix B for a complete

overview of the analysis outcome and criteria lists):

• Strategic level: (i) Cost (22.7%), (ii) Resource

Availability & Utilization (10.1%), Added value

(7.6%);

• Tactical level: Cost (21.9%), Environmental/Op-

erational Conditions (8.7%) and Safety (9.4%);

• Operational Level: Cost (25.8%), Resource

Availability & Utilization (19.4%) and Added

Value (4.8%).

These results clearly show that data quality is hardly

considered in the reviewed papers, whereas it can have

a major impact on expert decisions, as previously dis-

cussed. The only paper that integrates a data qual-

ity criterion is (Van & Pintelon, 2014), where the au-

thors measure the “accuracy” of maintenance report

records. Given the fact that existing expert mainte-

nance systems fail, or have no specific interest, to take

into account various data quality dimensions, as well

as in providing experts with the possibility to spec-

ify – in real-time – their own preferences regarding

each of these dimensions, our research aims to fulfill

this gap by adapting existing data quality frameworks

to the maintenance sector (the next section discussing

such frameworks). From the MM’s viewpoint, our re-

search primarily addresses the “Tactical” and “Oper-

ational” levels with the objective to assess quality of

enterprises’ daily maintenance reporting activities.

4



2.2. Data quality frameworks

Although first data or information quality frame-

works were introduced back in the 90′ (Krogstie et

al., 1995; Wang & Strong, 1996; Jarke & Vassiliou,

1997), research efforts has recently gained momentum

in an increasingly number of sectors due to the digi-

talization of almost every industry, e.g. in the context

of i) smart cities for open data portal quality assess-

ment, e.g. in (Umbrich et al., 2015) whose metadata

items are assessed in terms of quality; ii) product life-

cycle management, e.g. in (Wellsandt et al., 2015)

where authors separate ‘fitting’ from ‘unfitting’ in-

formation from a manufacturing/design decision pro-

cess perspective; iii) query processing, e.g. in (Sam-

paio et al., 2015) for incorporating data quality pro-

filing dimensions in the processing of queries involv-

ing quality-aware query language extensions; or more

recently (iv) Google’s Analytics Advocate shared his

own framework “TITE” (time, interactions, trends,

and events) to help marketers gain context and get ac-

tionable insights from their data (Waisberg, 2015).

Although some data quality frameworks are generic

enough to be applied in different contexts and sectors

(Krogstie et al., 1995; Kahn et al., 2002; Maurino &

Batini, 2009), they often need to be tuned/adapted to

each application case. It is, nonetheless, difficult to

state in what respects one framework is better than

another since data quality is commonly thought of as

a multi-dimensional concept with varying attributed

characteristics, which depend on the author’s philo-

sophical viewpoint, past experience, application do-

mains, and so forth (Ofner et al., 2013). In our re-

search, we decided to consider the framework intro-

duced by Krogstie et al. (1995) which, even if it dates

back to 1995, provides a very detailed and complete

overview of data quality concepts and relationships.

Section 3 provides greater detail on the Krogstie’s

framework, and to what extent it is instanciated to

cope with maintenance reporting procedures.

3. Research Methodology: Data quality frame-

work instantiation to MRQA purposes

The research methodology used in this study for de-

termining what data quality dimensions must be in-

tegrated to our model (in light of the MRQA prob-

lem) is presented in this section. To this end, sec-

tion 3.1 presents the high-level concepts and relation-

ships covered by the Krogstie’s framework, while sec-

tion 3.2 describes both what concepts/relationships

from that framework are relevant to our problem and

how they are integrated based on AHP.

3.1. Krogstie framework concepts and definitions

Concepts and relationships underlying the

Krogstie’s data quality framework are depicted

in Figure 1 and described hereinafter:

Modeling

Domain

Knowledge
Quality

Model
Externalization

Semantic
Quality

Syntactic
Quality

Physical
Quality

Language

Extension

Language
Quality

Participant
Knowledge

Audience
Interpretation

Social Technical

Language Quality

Social

Quality

Perceived
Semantic
Quality

Pragmatic
Quality

Pragmatic
Quality

Figure 1: Krogstie’s data quality framework

• Physical Quality: about externalizability (i.e.,

the knowledge of some social actors has been

externalized by the use of a conceptual model-

ing language) and internalizability (i.e., the ex-

ternalized model is persistent and available, thus

enabling participants to make sense of it);

• Syntactic Quality: correspondence between the

model and the language extension of the lan-

guage in which the model is written;

• Semantic Quality: correspondence between the

model and domain, where domain is considered

as the ideal knowledge about the situation to be

modeled. Krogstie’s framework contains two se-

mantic goals: Validity and Completeness;

• Perceived Semantic Quality: correspondence be-

tween the actor interpretation of a model and

his/her current knowledge of the domain. In line

with the semantic quality, two goals are defined

by the authors: Perceived Validity and Perceived

Completeness;

• Pragmatic Quality: correspondence between the

model and the “Audience Interpretation” of it;

• Social Quality: about people “agreement”;

• Knowledge Quality: from a pure standpoint of

social construction, it is difficult to talk about the

quality of explicit knowledge. On the other hand,

within certain areas such as mathematics, what

is regarded as ‘true’ is comparatively stable, and

it is inter-subjectively agreed that certain peo-

ple have more valid knowledge of an area than

others. The ‘quality’ of the participant knowl-

edge can thus be expressed by the relationship

between the audience knowledge and domain;

• Language Quality: appears as means for model

quality in the framework. The authors have re-

grouped factors from earlier discussions on lan-

5



Table 2: Criteria and its sub-criteria description related to the data quality dimensions

Criteria Sub-Criteria Description Type

Believability (CB)

Length of Work Description (CB1) Length of the work description related to a work order. I
CB1
avg

Work Log Conflict (CB2) Work description conflict among different form fields related to a

same report.

I
CB2
var

Technician Log Variation (CB3) Technical log variation among a set of reports related to a same

maintenance work order.

I
CB3
var

Completeness (CC )

Asset Location reported (CC1 ) Asset location (in the product) where maintenance was performed. I
CC1
fill

Description reported (CC2 ) Description of work to be done in particular maintenance work. I
CC2
fill

Start & Finish Date reported (CC3 ) Actual Start and Finish dates and times of work completed. I
CC3
fill

Target Start Date reported (CC4 ) Targeted start date of the maintenance work. I
CC4
fill

Target Finish Date reported (CC5 ) Targeted finish date of the maintenance work. I
CC5
fill

DLC Code reported (CC6 ) Actual location of the defect within product (DLC standing for

“Defect Location Code”).

I
CC6
fill

Schedule Start Date reported (CC7 ) Scheduled start date of the maintenance work. I
CC7
fill

Schedule Finish Date reported (CC8 ) Scheduled Finish date of the maintenance work. I
CC8
fill

Timeliness (CT ) This is average delay of reporting on individual site I
CT
avg

guage quality as follows: i) Domain appropriate-

ness; ii) Participant Knowledge Appropriateness;

iii) Technical actor interpretation enhancement.

3.2. MCDM-based Krogstie framework instanciation

Given the above definitions, and based on the

Finnish OEM company’s requirements, three key con-

cepts/relationships and a working assumption form

the foundation of our framework. First, we assume

that the Physical Quality (cf. Figure 1), and particu-

larly the externalized model, is 100% persistent and

available, thus enabling participants to make sense of

it. A potential study assessing how persistent their

implementations are compared with the initial ex-

pert statements/knowledge will be achieved in future

work. The OEM company then expressed require-

ments regarding three of the Krogstie’s framework

concepts/relationships, as highlighted in red/bold in

Figure 1, namely:

1. Semantic Quality: the OEM company wants to

know to which extent the service data reported

by each operator (on each site) can be trusted, or

more exactly can be considered as “true”, “real”

and “credible”, in order to carry out the plan-

ning activities. This is referred to as the “Be-

lievability” criterion (CB), whose various facets

of CB are formalized in the form of sub-criteria,

i.e. as Believability quality indicators denoted by

{CB1..CB3} (see Table 2 for more information);

2. Language Quality: the OEM company wants to

know to which extent the service data reported

by each operator is complete, or is of sufficient

depth and breadth for the task at hand. To put it

another way, this criterion, referred to as Com-

pleteness (CC ), reflects the level of details re-

ported by each operator with regard to each re-

port/form field that needs to be entered (in ac-

cordance with the company’s business logic).

Similarly to CB, several Completeness quality

indicators are defined, respectively denoted by

{CC1 . . .CC8} (see Table 2);

3. Knowledge Quality: the OEM company wants to

know to which extent the service data reported by

each operator is sufficiently “up to date”, which

is depending on the time difference between the

maintenance work achievement and the task re-

porting. This criterion, referred to as Timeliness

CT , is based on the assumption that the longer

the time spent to submit the report, the lesser the

quality of the reporting (operator are likely to for-

get details over time). As emphasized in Table 2,

no sub-criterion is defined for this dimension.

To ease the understanding of those data quality di-

mensions and sub-dimensions, an illustration of the

overall maintenance reporting quality assessment is

presented in Figure 2. First, maintenance opera-

tors carry out maintenance work orders/tasks on each

OEM site (denoted by Site 1. . . Site z), thus generating

multiple reports. Figure 2 presents a simplified view

of (i) the report’s content, and (ii) the comparison pro-

cess based on the criteria introduced in Table 2. We

emphasize, using “smileys”, how and why a report’s

content (or form field content) can positively or nega-

tively impact on the maintenance reporting quality.

In light of the MCDM problem – integration of

various quality dimensions, expert preferences, report

contents. . . – AHP has been chosen to help organiz-

ing critical aspects of the problem in a manner sim-

ilar to that used by the human brain in structuring

the knowledge (Saaty, 1980). As highlighted in our

literature review (cf. section 2), there are a number

of MCDM techniques such as AHP/ANP, TOPSIS,

ELECTRE, PROMETHEE, MAUT, and much more

(e.g., hybrid MCDM combining these techniques to-

gether or even with other theories such as fuzzy logic)

(Behzadian et al., 2012; Mardani et al., 2015). There

6



SITE 1 SITE 2 . . . SITE z

x x
x

x

y xx
x
x y

x
x

xy y

OEM
Database

Maintenance Operator (Site 1)

Report ID : 1D

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

ID : 1389706

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

ID : 1389706

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Power Controller 4v

Done

28/08/2014

23/08/2014

24/08/2014

27/08/2014

Front axle 34.8YH

Done

02/05/2014

07/06/2014

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 2)

Report ID : 2A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zD

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

ID : 1389706

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

ID : 1389706

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site z)

Report ID : zA

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Fuel System 01X.2

System changed by...

28/08/2014

23/08/2014

24/08/2014

27/08/2014

Chassis has been re...

28/08/2014

23/08/2014

24/08/2014

27/08/2014

Maintenance

Operators per

OEM site

Example when comparing operator
reports between Sites 1 and z

Maintenance Reporting Quality
Assessment of the OME company

CB1 : Length of Work Description
One world (”Done”) is too short to properly

describe the maintenance opration
The description seems to be long enough
in reports zA & zD

CB2 : Work Log Conflict
High number of conflict/variation in the

set of reports available on site 1
Conflicts/Variations in content of the
reports rarely occur

CC1 : Asset Location Reported
Field “Asset Location” filled out in report 1A

as well as in report 1D
Field “Asset Location” filled out in
report zA but not in report zD. . .

. . .. . . . . .

CT : Average Delay of Reporting
Reports 1A was made 1h after the task, while
report 1D was made with a delay of 3 weeks

Both Reports zA and zD have been
made with a delay inferior to 2h

MCDM technique
Site ranking considering all reports available on the
different OEM sites : {Site 1, Site 2, Site 3. . . Site z}

2
3

SITE 1
1

SITE 2

SITE z

Figure 2: Stages composing the maintenance reporting quality assessment framework

are no better or worse techniques but some techniques

are better suited to particular decision-making prob-

lems than others. For example, AHP only deals with

linear preferences and not with contextual preferences

where values of one or several criteria may affect the

importance or utility of other criteria. In this study,

AHP is used (and combined with TOPSIS) for two

reasons: i) we only deal with linear preferences and

ii) AHP provides a powerful impartial, logical, and

easy-to-use grading system, thus reducing personal

biases and allowing for comparing dissimilar alterna-

tives. Those characteristics are probably the main rea-

sons for its success. According to a recent survey on

MCDM techniques (Mardani et al., 2015), AHP is the

second most used technique (applied in 16% of the

reviewed literature1) after Hybrid MCDM (19.89%).

1 In total, 150 scientific journal papers were reviewed.

4. AHP-based MRQA framework

The hierarchical structure defined using AHP con-

sists consists of four levels, as depicted in Figure 3,

namely:

• Level 1: the overall goal of the study is to rank

the different OEM company sites in terms of

maintenance reporting quality;

• Levels 2 and 3: the set of data quality dimensions

(criteria) and sub-criteria defined in Table 2;

• Level 4 the OEM company sites representing the

alternatives.

Given this hierarchy, AHP does perform the follow-

ing computation steps for identifying the final ranking

of the alternatives with respect to the overall goal:

1. Compare each element in the corresponding level

and calibrate them on the numerical scale. This

7



Site 1 Site 2 Site 3 Site 4 . . . . . . . . . . . . . . . Site 54

CB1 CB2 CB3 CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8 CT

Believability Completeness Timeliness

Reporting Quality Assessment and Ranking of OEM Sites
Level 1

Level 2

Level 3

Level 4

Figure 3: AHP structure of the maintenance reporting quality assessment process

requires
n(n−1)

2
pairwise comparisons, where n is

the number of elements (diagonal elements be-

ing equal to “1” and the other elements being the

reciprocal of the earlier comparisons);

2. Perform calculation to find the maximum eigen-

value, consistency index (CI), consistency ratio

(CR), and normalized values;

3. If the computed eigenvalue, CI and CR are sat-

isfactory, then decision/ranking is done based on

the normalized values.

These three stages 1, 2 and 3 are detailed in the

following sections. In an effort to facilitate the un-

derstanding, a scenario is considered in the following,

whose parts are preceded by the symbol “➫”.

4.1. Pairwise comparison based preference measure-

ment

According to Blumenthal (1977), two types of

judgment exist, namely:

i) “Comparative judgment, which is the identifica-

tion of some relations between two stimuli both

present to the observer”;

ii) “Absolute judgment, which involves the relations

between a single stimuli and some information

held in short term memory about some former

comparison stimuli, or about some previously ex-

perienced measurement scale using which the ob-

server rates the single stimulus.”

In a comparative/relative measurement, each alterna-

tive is compared with many other alternatives, that

is why this is also referred in the AHP literature

to as “pairwise comparisons as ratio measurement”

(Mumpower et al., 2012). In an absolute measure-

ment, each alternative is compared with an ideal alter-

native the expert knows of or can imagine, that is why

this is referred to as “pairwise comparison based pref-

erence measurement”. This section details the “pair-

wise comparison based preference measurement” ap-

proach, which is used at level 2 and 3 of the AHP

structure (cf. Figure 3), while section 4.2 details the

“pairwise comparisons as ratios” approach, which is

used at level 4.

Regarding pairwise comparison based preference

measurement, decision makers have to evaluate the

importance of one criterion (or sub-criterion) with re-

spect to the others. To this end, OEM stakehold-

ers perform pairwise comparisons among the iden-

tified criteria as formalized in Eq. 1, with m the

number of criteria to be compared (e.g., at level 2:

m = |{CB,CC,CT}| = 3). The expert evaluation is

carried out based on the 1- to 9-point Saaty’s scale:

{1,3,5,7,9}; wi j = 1 meaning that Ci and C j are

of equal importance and wi j = 9 meaning that Ci is

strongly favored over C j. Note that all variables used

in this paper are summarized in Table 3.

P =

























C1 . . . Cm

C1 w11 . . . w1m
..
.

..

.
...

..

.

Cm wm1 . . . wmm

























(1)

The computation of the normalized eigenvector of

P then enables to turn qualitative data into crisp ra-

tios. Although several approaches exist in the lit-

erature for normalized eigenvector computation, the

Simple Additive Weighting (SAW) method (Hwang &

Yoon, 1981) is used in this study, namely:

WCi =

∑m
j=1 wi j

∑m
k=1

∑m
j=1 wk j

, w ji =















1 i = j
1

wi j
i , j

(2)

WC = [WC1, ..,WCi, ..,WCm ]

P is characterized as consistent if, and only if Eq. 3

is respected. However, it is not that simple to fulfill

this prerequisite when dealing with real expert prefer-

ences, or when the number of criteria increases. Saaty

(1980) proved that for consistent reciprocal matrix,

the largest eigenvalue is equal to the size of the com-

parison matrix (or λmax = m) and, accordingly, intro-

duced CI as the deviation or degree of consistency2

2RI corresponds to the consistency index of a pairwise matrix

generated randomly.

8



Table 3: Variable definitions

Variables Description

Cx abbreviation for criterion x with x = {1,2, ..,m}. Three criteria are defined at level 2 of the hierarchy structure, namely:

CB, CC , CT (cf. Table 2).

Cxh abbreviation for a sub-criterion of criterion x with h = {1,2, ..,y}. In this study, h = {1..3} for x = B (i.e., three sub-criteria

of CB), h = {1..8} for x = C (i.e., three sub-criteria of CC ) and h = ∅ for x = T (i.e., CT does not have sub-criteria).

P abbreviation for “Pairwise Comparison matrix”, whether at level 2, 3 or 4 of the AHP structure.

wi j crisp value of a pairwise comparison matrix located at row i, column j of P.

Al represents an alternative l in the AHP structure, with l = {1,2, ..,z}. In this study, l is the set of OEM sites to be

assessed/ranked in terms of maintenance reporting quality.

WCx , WCxh represents the eigenvalue of criterion Cx or sub-criterion Cxh (the eigenvector results from the P’s weight derivation

process). In practice, it indicates the importance of one (sub)criterion over the others.

I
Cxh
φ

(Al) represents a digital indicator used for computing pairwise comparisons as ratios (i.e., measurable elements). Two indica-

tors are defined, namely φ= {fill,avg,var} (see Eqs. 10, 14 and 16).

W
Al
Cxh

represents the eigenvalue of alternative Al with respect to sub-criterion Cxh. In practice, it indicates how good (or bad)

the data quality is, regarding alternative/site l, with respect to Cxh .

Rep Total(Al) function returning the total number of reports available on Site l.

Size Delay(r,Al) function returning either (i) the reporting delay value or (ii) description length value of a given report (denoted by r)

available on Site l.

Rep filled(Al) function returning the number of reports (on Site l) for which a given “form field” has been filled out.

Rep var(Al) function returning the number of reports/work orders (on Site l) containing content variation/conflicts.

(see Eq. 4). If CR is smaller or equal to 10%, the in-

consistency is regarded as acceptable.

wi j = wik × wk j
∀i,k∈N|i,k; j∈N−{i,k}

(3)

CI =
λmax − m

m − 1
CR =

CI

RI
(4)

➫ In this scenario, pairwise comparisons are filled

out by the OEM’s executive officer. Eq. 5 shows the

officer preference specifications regarding criteria at

Level 2 of the AHP structure. The computed normal-

ized eigenvector shows that the officer judges all cri-

teria (at this level) of equal importance. Eq. 6 shows

the pairwise comparisons carried out at Level 3 of the

AHP structure, with regard to sub-criteria CBx | x =

{1,2,3} (see Eq. 7 for the details of WCB1 computa-

tion). The eigenvector (cf. Eq. 6) emphasizes that

the officer judges “Length of Work Description” (i.e.,

CB1) slightly more important than “Work Log Con-

flict” (i.e., CB2) for his/her own task, and highly more

important than “Technician Log Variation” (i.e., CB3).



























CB CC CT

CB 1 1 1

CC 1 1 1

CT 1 1 1



























➠



























WCB 0.33

WCC 0.33

WCT 0.33



























(5)

CR=0





























CB1 CB2 CB3

CB1 1 3 5

CB2
1
3

1 3

CB3
1
5

1
3

1





























➠



























WCB1 0.61

WCB2 0.29

WCB3 0.10



























(6)

CR=0.040

WCB1 =
1 + 3 + 5

1 + 3 + 5 + 1
3
+ 1 + 3 + 1

5
+

1
3
+ 1

(7)

= 0.61

Similarly, the OEM officer carries out pair-

wise comparisons between sub-criteria CCx | x =

{1,2, ..,8}, as detailed in Eq. 8. According to the re-

sulting eigenvector, CC1 is deemed as the most impor-

tant sub-criterion (WCC1 ), followed by CC3 and CC2 re-

spectively. Since CT does not have any sub-criterion,

no pairwise comparison is required.

4.2. Pairwise comparisons as ratio measurement

As previously mentioned, pairwise comparisons as

ratio measurement is used at level 4 of the AHP struc-

ture in order to compare alternatives with respect to

each criterion, based upon measurable/supervised sys-

tem parameters, e.g. how many times the field “DLC

Code” (CC6) has been reported on Site l compared

with the other sites (i.e., how many times such as form

field has been left empty by maintenance operators).

Eq. 9 gives insight into the pairwise comparisons as

ratio matrix of the set of alternatives/sites Al with re-

spect to the monitored system parameter Cxh. The ra-

tio value is denoted by I
Cxh
φ

(Al), as described in Ta-

ble 3. The normalized eigenvector values of the pair-

wise comparisons as ratio matrix with respect to cri-

terion Cxh are denoted by W
Al
Cxh

in Eq. 9.

9



CR=0.096





























































































CC1 CC2 CC3 CC4 CC5 CC6 CC7 CC8

CC1 1 3 1 3 7 3 9 3

CC2 1/3 1 1/3 3 5 3 5 3

CC3 1 3 1 3 5 3 5 3

CC4 1/3 1/3 1/3 1 3 3 5 1

CC5 1/7 1/5 1/5 1/3 1 1 3 3

CC6 1/3 1/3 1/3 1/3 1 1 5 1/3

CC7 1/9 1/5 1/5 1/5 1/3 1/5 1 1/5

CC8 1/3 1/3 1/3 1 1/3 3 5 1





























































































➠





























































































WCC1 0.266

WCC2 0.167

WCC3 0.242

WCC4 0.099

WCC5 0.065

WCC6 0.058

WCC7 0.023

WCC8 0.080





























































































(8)

































































A1 A2 . . . Az

A1 1
I
Cxh
φ

(A1 )

I
Cxh
φ

(A2 )
. . .

I
Cxh
φ

(A1 )

I
Cxh
φ

(Az )

A2
I
Cxh
φ

(A2 )

I
Cxh
φ

(A1 )
1 . . .

I
Cxh
φ

(A1 )

I
Cxh
φ

(Az )

..

.
..
.

..

.
...

..

.

Az
I
Cxh
φ

(Az )

I
Cxh
φ

(A1 )

I
Cxh
φ

(Az )

I
Cxh
φ

(A2 )
. . . 1

































































➠









































W
A1
Cxh

W
A2
Cxh

..

.

W
Az
Cxh









































(9)

Three digital indicators I
Cxh
φ

(Al) are defined (i.e.,

φ = {fill,avg,var}), which are described below. Ta-

ble 2 highlights what indicators is used with regard to

each criterion (see column named “Type”):

• I
Cxh
fill

(Al) (Filled Indicator – Eq. 10): used to cal-

culate the proportion of reports where a given

“field” was filled out on Site l ; Rep filled(Al)

returning the number of reports that have been

filled out, and Rep Total(Al) returning the total

number of reports available on Site l:

I
Cxh
fill

(Al) =
Rep filled(Al)

Rep Total(Al)
(10)

➫ Let us consider pairwise comparisons as ra-

tio measurements between Sites 1 and 2, with re-

spect to CC6. On Site 1, 76 maintenance reports

have been carried out and 45 of them contain the

DLC code (meaning that 59% of the available

reports contain the requested information, see

Eq. 11), while on Site 2 only 44% of the avail-

able reports contain this information (see Eq. 12).

The resulting pairwise comparisons as ratio ma-

trix with respect to CC6 is given in Eq. 13, in

which the above computed I
CC6
fill

(A1) and I
CC6
fill

(A2)

are considered for the pairwise comparison as

ratio measurement between Sites 1 and 2 (see

row 1/column 2 of the matrix, and vice-versa).

The resulting eingevector indicates how good (or

bad) the reporting quality with respect to CC6 in

this case – is regarding each site.

I
CC6
fill

(A1) =
45

76
= 59% (11)

I
CC6
fill

(A2) =
49

88
= 44% (12)







































A1 A2 . . . A54

A1 1
59
44

. . . 0.15

A2
44
59

1 . . . 0.67

.

.

.
.
.
.

.

.

.
...

.

.

.

A54 6.64 1.50 . . . 1







































➠















































W
A1
CC6

0.187

W
A2
CC6

0.002

.

.

.
.
.
.

W
A54
CC6

3E-06















































(13)

Pairwise comparison as ratios can lead to com-

putational issues when dividing
I
Cxh
φ

(Ai )

I
Cxh
φ

(A j )
since the

denominator may be null. For example, consid-

ering the above scenario, if I
CC6
fill

(A2) = 0 (mean-

ing that the DLC Code was never reported by any

operator on Site 2), then
I
CC6
fill

(A1 )

I
CC6
fill

(A2 )
=

59
0

, which pre-

vents from performing the division. To bypass

this problem, a penalty score θ is assigned to the

corresponding site (i.e., A j) with respect to cri-

terion Cxh, i.e. W
A j
Cxh
= θ. However, a more in-

depth study must be conducted to identify what

penalty score should be assigned, how it affects

the overall results, and so on. This study is pre-

sented in Appendix B to avoid overloading the

paper.

• I
Cxh
avg(i) (Average Indicator – Eq. 14): used to cal-

culate the average delays for maintenance re-

porting per site (i.e., regarding CT ), or the av-

erage length of work description (i.e., CB1) per

site. Mathematically, I
Cxh
avg (Al) is computed based

on Eq. 14, where Size Delay(k,Al) is either (i)

the reporting delay value or (ii) the description

length value, of a given report (denoted by r)

available on Site l.

I
Cxh
avg(Al) =

Rep Total(Al )
∑

r=1

SizeDelay(r,Al)

Rep Total(Al)
(14)

10



WCB1

0.610

WCB2

0.290

WCB3

0.100

WCC1

0.266

WCC2

0.167

WCC3

0.242

WCC4

0.099

WCC5

0.065

WCC6

0.058

WCC7

0.023

WCC8

0.080

WCB
= 0.33 WCC

= 0.33 WCT
= 0.33

Reporting Quality Assessment and Ranking of OEM Sites
1.00

Site 1 : W
A1

C
B1

Site 2 : W
A2

C
B1

Site 54 : W
A54

C
B1

Site 1 : W
A1

C
B3

Site 2 : W
A2

C
B3

Site 54 : W
A54

C
B3

Site 1 : W
A1

C
C6

= 0.187

Site 2 : W
A2

C
C6

= 0.002

Site 54 : W
A54

C
C6

= 3E-06

Site 1 : W
A1

C
T

Site 2 : W
A2

C
T

Site 54 : W
A54

C
T

Level 1

Level 2

Level 3

Level 4

S
e
e

E
q
.
5

S
e
e

E
q
.
6

&
8

S
e
e

E
q
.
1
3

Figure 4: AHP structure and associated weights

➫ Let us assume that 4 reports are available on

Site 1 (i.e. Rep Total(A1) = 4) and that the work

description length is respectively equal to 44, 5,

13 and 101. The average indicator with regard

to CB1 (on Site 1) is therefore equal to 40.75, as

detailed in Eq. 15. The resulting pairwise com-

parisons as ratios matrix is not presented due to

similarities with the matrix detailed in Eq. 13.

I
CB1
avg (Al) =

44 + 5 + 13 + 101

4
= 40.75 (15)

• I
Cxh
var (i) (Variation Indicator – Eq. 14): used to cal-

culate the number of reports and/or work orders

that contain variations or conflicts. One possi-

ble conflict could be that the operator indicates a

DLC code related to the car’s wheel (see CC6),

while indicating in the Work Description (see

CC2 ) that the car’s pump has been fixed. I
Cxh
var (Al)

is computed based on Eq. 16, with Rep var(Al)

the number of reports that contain content varia-

tion (or conflicts) on Site l.

I
Cxh
var (Al) =

Rep var(Al)

Rep Total(Al)
(16)

In an effort to summarize all the variables and

weights computed in this section, we provide an “at

a glance” representation of the AHP hierarchy in Fig-

ure 4, which refers to the different equations consid-

ered to compute the variable weights. In the next sec-

tion, we present how those different weights are ag-

gregated to obtain the final site ranking.

4.3. Alternative ranking using TOPSIS

The different weights must now be aggregated in or-

der to obtain a global weight of each alternative with

respect to all criteria, which is computed based on

Eq. 17. All these global weights are summarized in

the form of a matrix in Eq. 18.

GW
Al
Cxh
= W

Al
Cxh
× WCxh × WCx (17)



































































CB1 . . . CB3 CC1 . . . CC8 CT

A1 GW
A1
CB1

. . . GW
A1
CB3

GW
A1
CC1

. . . GW
A1
CC8

GW
A1
CT

A2 GW
A2
CB1

. . . GW
A2
CB3

GW
A2
CC1

. . . GW
A2
CC8

GW
A2
CT

.

.

.

.

.

.
.
.
.

.

.

.

.

.

.
.
.
.

.

.

.

.

.

.

Az GW
Az
CB1

. . . GW
Az
CB3

GW
Az
CC1

. . . GW
Az
CC8

GW
Az
CT



































































(18)

➫ For illustration purposes, Eq. 19 details the

global weight computation for A1 (i.e., Site 1) with

respect to criterion CC6, which implies to takes into

account WCC -related weight.

GW
A1
CC6
= W

A1
CC6
× WCC6 × WCC (19)

= 0.187 × 0.058 × 0.333 = 0.0036

The global weights (cf. Eq. 17) must now be aggre-

gated for each alternative in order to obtain the final

quality score, based on which the final site ranking is

generated. To this end, the TOPSIS method is em-

ployed or, to be more accurate, combined with AHP.

TOPSIS introduces for each alternative the closeness

coefficient denoted by R(Al), which implies comput-

ing for each criterion xh the positive ideal solution

(PIS) denoted by d+
xh

and negative ideal solution (NIS)

denoted by d−
xh

, as formalized in Eq. 20 and 21 respec-

tively. The distances measuring the separation from

PIS and NIS are then computed in Eq. 22 and 23, re-

spectively denoted D+
Al

and D−
Al

).

d
+
xh
= max

l=1..z

(

GW
Al
Cxh

)

(20)

d
−
xh
= min

l=1..z

(

GW
Al
Cxh

)

(21)

D
+
(Al) =

√

∑

xh

(

GW
Al
Cxh
− d+

xh

)2
l = 1, ..,z (22)

D
−
(Al) =

√

∑

xh

(

GW
Al
Cxh
− d−

xh

)2
l = 1, ..,z (23)

A prior alternative has a longer distance to NIS and

a shorter distance to PIS. Consequently, the closeness

11



Table 4: Alternative ranking illustration
One ranking per quality dimension Overall Ranking

Believability Completeness Timeliness

Site 1 30th 3rd 2nd 12th

Site 2 4th 15th 27th 15th

. . . . . . . . . . . . . . .

Site 11 34rd 7th 1st 14th

. . . . . . . . . . . . . . .

Site 32 1nd 2th 18th 7th

. . . . . . . . . . . . . . .

Site 47 19th 35th 31th 28th

. . . . . . . . . . . . . . .

coefficient to the ideal solution for each alternative can

be formulated as in Eq. 24, where R(Al) denotes the

final performance score of Site l. The larger the R(Al)

score, the better the maintenance reporting quality on

the corresponding site.

R(Al) =
D−(Al)

D+(Al) + D
−(Al)

l = 1, ..,z (24)

The overall site ranking can therefore be generated

based on the R(Al) performance scores. Nonetheless,

let us note that in Eq. 22 and 23, if:

• xh = {CB1, ..,CB3,CC1, ..,CC8,CT}: a single and

overall ranking of the sites is generated (i.e., all

criteria/sub-criteria are aggregated), as shown in

Table 4 (see column “Overall Ranking”);

• xh = {CB1, ..,CB3} or xh = {CC1, ..,CC8} or

xh = {CT}: one ranking per quality dimension

(i.e., CC , CB or CT ) is generated, as shown in Ta-

ble 4 (see column named “One ranking per qual-

ity dimension”). Having indicators per dimen-

sion enable e.g. a site manager to further inves-

tigate i) what dimension(s) must be enhanced in

the short, medium or long term, ii) to track the

evolution over time of the reporting quality of

one or a group of sites with respect to specific

dimensions, etc.

➫ Figure 5 gives insight into how a company’s

stakeholder can use the “ranking per quality dimen-

sion” to better understand how a site behaves (i.e.,

how good/bad it is) with respect to one or more di-

mensions3: the larger the surface areas in Figure 5,

the better the site ranking and, as a consequence, the

better the reporting quality on this site.

5. OEM use case

Two distinct use cases, defined at the tactical and

operational levels, are presented in this section and

show how the Finnish OEM company takes advan-

tage of the MRQA dashboard. Figure 6 provides an

3 The four sites highlighted in bold in Table 4 are considered and

displayed in this example.

Timeliness

1
st

12th
23

th

34
th

45
th

1
st

12th
23

th

34
th

45
th

54
th

45
th

34
th

23
th

12th

1
st

Site 11

Site 32

Site 47

Completeness

Believability

Figure 5: Comparison of sites 11, 32 and 47 (cf. Table 4)

overview of the architecture and associated tools that

have been developed/set up in the company: main-

tainers on the different sites report maintenance work

order-related information using the company’s online

form (see “Reporting Service” in Figure 6). A screen-

shot of the company’s online form is provided in Fig-

ure 7(a), which has been annotated to help the reader

to understand what form fields correspond to what

AHP (sub-)criteria. In total, by summing all reports

from all sites, 275 585 reports have been processed

and analyzed.

The MRQA dashboard thereby enables any site

stakeholder (e.g., plant manager, head officer. . . ) to

assess – at a given point in time and based on the

his/her own preferences – the quality of reporting of

the 54 branch offices. As highlighted in Figure 6,

when a stakeholder requests for the site ranking ser-

vice, the overall ranking is computed at the head of-

fice (i.e., in Finland). In practice, a set of SQL queries

is performed against the different database systems –

spread over the 54 sites – that contain the maintenance

reports. The retrieved information is then used as in-

puts of the pairwise comparisons as ratio measure-

ment process. A screenshot of the MRQA dashboard

is given in Figure 7(b), which provides the stakeholder

with the possibility to:

• access it through a web browser;

• vizualize in a user-friendly way both the loca-

tions of the OEM sites (see the dashboard ele-

ment named “Map of Company Sites”) and their

corresponding quality/ranking (see “Final As-

sessment & Ranking”);

• deeper investigate the maintenance reporting

quality of one or more sites with respect to one or

more quality dimensions (see the “Disintegrated

Quality View” dashboard element);

12



Internet

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

MRQA algorithm(s)

{ {

{ {

{ {

Head Office

(OEM)

b

➠

b

➠

b

➠

✓ T
ask

1

Tas
k 2

✓ T
ask

3

✓ T
ask

1

Tas
k 2

✓ T
ask

3

✓ T
ask

1

Tas
k 2

✓ T
ask

3

Reporting Service

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

MRQA dashboard

Site
Manager
or other

b

➠

b

➠

b

➠

Reporting Service

✓ T
ask

1

Tas
k 2

✓ T
ask

3

✓ T
ask

1

Tas
k 2

✓ T
ask

3

✓ T
ask

1

Tas
k 2

✓ T
ask

3

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Maintenance Operator (Site 1)

Report ID : 1A

Asset Location

Description

Actual End-Date

Target Start Date

. . .
Scheduled Start Date

Scheduled End Date

Site
Manager
or other

MRQA dashboard

Other sites

Figure 6: Overall infrastructure underlying the MRQA dashboard

• adjust the assessment period, e.g., to assess/com-

pare sites over the last few weeks, months or

years (see the “Assessment period” dashboard el-

ement);

• modify his/her preferences related to the crite-

ria importance, e.g. if he/she wants to give – at

a specific point in time – further importance to

one or more dimensions (e.g., Completeness over

Believability) or sub-dimensions (e.g., CC4 over

CC1 ). This is discussed further in this section, but

the reader can already refers to Figure 10 to have

an overview of the dashboard functionality.

Two distinct scenarios are presented in sections 5.1

and 5.2 respectively. The first one provides in-

sight into the reporting quality assessment results

when considering the importance between criteria as

roughly equivalent, while the second scenario high-

lights how the MRQA dashboard can be used for a

specific purpose, namely to set up a cost-reduction ac-

tion plan in the proposed scenario.

5.1. Scenario 1: Equivalence between criteria

At the operational level, the head officer wants to

have an overview of the maintenance reporting qual-

ity regarding all sites, without prioritizing any qual-

ity dimension. To this end, the officer does perform

pairwise comparisons by specifying that all criteria

are equal in importance (as carried out in Eq. 5), and

similarly for the sub-criteria. Figure 8(a) gives insight

– in the form of a histogram – into the quality assess-

ment results, where the x-axis refers to the 54 sites and

the y-axis to the quality score obtained after applying

AHP. It can be observed that three sites stand out (hav-

ing the highest quality scores), namely sites 46, 12 and

18 respectively.

The officer wants to further investigate the reasons

behind the low quality score of Sites 6 and 54 (i.e.,

sites having the poorest quality). To this end, the of-

ficer selects – in the “Disintegrated Quality View”

dashboard element in Figure 7 – these two sites, thus

having a deeper insight into the site level quality with

respect to each level 2 quality dimension. It can be

observed that Site 6 has a particularly poor ranking

regarding both the “Timeliness” and “Completeness”

dimensions, while Site 54 has a poor ranking regard-

ing “Timeliness” and “Believability”. The officer can

even go a step further in the analysis in order to under-

stand the reasons behind the low quality score of a site

regarding one of these dimensions. For example, in

the dashboard’s screenshot (cf. “Disintegrated quality

view (level 3)” in Figure 7), the officer has selected the

‘Completeness’ dimension and can visualize the per-

centage of form fields (which have been turned into

sub-criteria CC1 to CC8) that have been or not reported

with respect to the total number of reports on the se-

lected sites (i.e., on Sites 6 and 54). It can be observed

that maintainer operators on Site 6 report less often

information than on Site 54, which is the main reason

why Site 6 has a lower quality rank than Site 54, as

pointed out above. Nonetheless, an interesting point

in this graph is that CC6 reports more CC6 -related in-

formation (i.e., the ‘DLC Code’), namely 55% against

18% for Site 54.

In summary, this first scenario offered an overview

of the different MRQA dashboard functionalities, and

how the associated views can be used as decision

13



CB1 ➙

➙

CC2

CC3

➙

CC4➙

CC5➙

CC7➙

CC8

Legend

Form fields assessed in terms of “Believability”: CB = {CB1,CB2,CB3} Form fields assessed in terms of “Timeliness”: CT

Form fields assessed in terms of “Completeness”: CC = {CC1,CC2, . . . ,CC8}

(a) Online form used/filled out by maintainer operators on each OEM site

Disintegrated Quality View (level 3)

(b) MRQA dashboard interface

Figure 7: Screenshots of the “Maintenance Work Order System” & the “MRQA dashboard”

14



R
e
la

ti
v
e

Q
u
a
li

ty
S

c
o
re

0

0.2

0.4

0.6

0.8

1

1
2

3
4

5
6

7
8

9
10

11
12

13
14

15
16

17
18

19
20

21
22

23
24

25
26

27
28

29
30

31
32

33
34

35
36

37
38

39
40

41
42

43
44

45
46

47
48

49
50

51
52

53
54

Site number (i.e., set of alternatives)

(a) Scenario 1: equivalence between criteria

0

0.024

0.048

0.072

0.096

0.12

1
2

3
4

5
6

7
8

9
10

11
12

13
14

15
16

17
18

19
20

21
22

23
24

25
26

27
28

29
30

31
32

33
34

35
36

37
38

39
40

41
42

43
44

45
46

47
48

49
50

51
52

53
54

Site number (i.e., set of alternatives)

(b) Scenario 2: Cost-reduction action plan

Figure 8: Overall site ranking: Scenarios 1 & 2

Failure

✹ Create
Work Order

Spare Parts

Arrived

Start repairing

at scheduled date

b

Start

date

End

date

Repair work
completion

Report after
customer checking

& Validation

y✔

T I M E L I N E S S
B

Maintenance work
order tracking system

(see Figure 7(a))

T I M E L I N E S S
B

Figure 9: Problem to be addressed in the maintenance reporting process to avoid financial losses

support tools by various maintenance stakeholders.

The second scenario, presented in the next section,

puts further emphasis on how a site stakeholder can

take advantage of the dashboard for improving cost-

reduction action plan.

5.2. Scenario 2: Cost-reduction action plan

Over the past few years, the OEM company has

been facing a major problem in the maintenance re-

porting process at the tactical level, leading to sub-

stantial financial losses. The traditional maintenance

process4 is depicted in Figure 9, where a work order

is created by the maintenance planner as soon as a

problem/failure is reported by the customer. Once the

spare parts are supplied on site, the site manager re-

ports it into the system, and the maintainer can start to

repair the defective equipment at the scheduled time.

As emphasized in Figure 9, the maintainer has to spec-

ify – into the maintenance work order tracking sys-

tem – the “Start date” and “End date” (both dates be-

ing taken into account in our AHP under the “Com-

pleteness” dimension, and particularly with CC3 ). Fol-

lowing the “Repair work completion” (i.e., End date),

it is necessary to wait until the customer checks and

validates the maintenance service as depicted in Fig-

ure 9. Such a time interval actually corresponds to

4 Only the Scheduled Maintenance Process is described (not the

Unscheduled process) to ease the understanding.

the “Timeliness” criterion. This interval is very crit-

ical for the OEM company because the company has

to pay a penalty fee that depends on the time inter-

val (obviously under the condition that the customer

does not validate, or complain against the service, and

obtains a favorable ruling).

Given the above-mentioned problem, the head of-

ficer wants to draw up an assessment of the overall

situation (i.e., with regard to each site), and to shape a

proper action plan for reducing financial losses. This

plan consists first in identifying and managing sites

that have the poorest timeliness quality, since they

have the highest financial risk. To this end, the offi-

cer does specify that Timeliness (CT ) is strongly more

important than Believability (CB) and Completeness

(CC ) in order to bring to light the sites with the poor-

est Timeliness quality. Figure 10 shows how this can

be specified through the MRQA dashboard (see red/-

dashed frame). The final ranking is generated and

given in Figure 8(b), showing that Sites 18, 42, 12,

6 and 5 are respectively the branch offices that have

the highest risk for monetary losses.

Although the action plan to be set up by the OEM

company to reduce such risks on the identified sites

is out of scope of this paper, it is worth noting that

IoT messaging protocols will likely be implemented

in the future to address part of the problem. At a more

concrete level, such protocols will be used to gener-

ate ‘event-based’ notifications to the customer as soon

15



Pairwise comparison at level 1 of the AHP structure
by the OEM head officer

Figure 10: Dashboard/User Interface to adjust the pairwise comparison based preference measurement

as the repair work is completed (i.e., when the “End

date” is entered in the system), e.g. by explicitly stat-

ing that the customer has a n-day deadline to check

and validate the maintenance work. From a practical

viewpoint, the recent IoT standards published by The

Open Group (Främling et al., 2014) will first be im-

plemented on the riskiest sites.

6. Conclusions, implications, limitations and fu-

ture research

6.1. Conclusions

Data, information and knowledge are the “new oil”

of the digital era, and are at the heart of all busi-

ness operations. It is therefore crucial for companies

to implement the right infrastructure to monitor and

improve the quality of data generated throughout the

lifecycle of the company’s assets (either physical or

virtual assets). It is a fact that the data quality has

a significant impact on the overall incomes and ex-

penditures of companies: poor data quality impacting

the downstream processes, and reciprocally, high data

quality fostering enhanced business activities and de-

cision making. However, it remains challenging to as-

sess information quality, as information is not as tan-

gible as physical assets.

The literature review carried out in this paper brings

to light the fact that current expert maintenance sys-

tems fail, or have no specific interest, to take into ac-

count data quality dimensions in the maintenance re-

porting quality assessment process. To fulfill this gap,

this paper develops a Maintenance Reporting Qual-

ity Assessment (MRQA) dashboard that enables any

company stakeholder to easily – and in real-time –

assess/rank company branch offices in terms of main-

tenance reporting quality. In this respect, AHP is used

to integrate various data quality dimensions as well

as expert preferences. The paper presents two scenar-

ios showing how the MRQA dashboard is being used

by a Finnish multinational equipment manufacturer to

assess and enhance maintenance reporting practices in

one or more branch offices. This should contribute to

enhance other organization activities such as:

• after-sales services: the quality of maintenance

reports makes it possible to assess the mainte-

nance work, thus helping to reach a higher qual-

ity after-sales services;

• on the design of future generations of products:

processing and analyzing relevant maintenance

reports help to better understand how company

assets behave throughout their lifecycle which,

in turn, help to enhance the design of future gen-

erations of products (Främling et al., 2013);

• predictive maintenance strategies: providing

real-time and remote predictive maintenance is

becoming a very promising area in the so-called

IoT, whose objective is to provide systems with

the capability to discover and process real-time

16



data and contexts so as to make pro-active deci-

sions (e.g., to self-adapt the system before a pos-

sible failure) (Främling et al., 2014). Although

real-time data is of the utmost importance in the

predictive maintenance process, combining such

data with historical maintenance reporting data

(regarding a specific product item) has the poten-

tial to generate new knowledge and lead to more

effective and product-centric decisions;

• government regulation compliance: in some do-

mains, it is mandatory to comply with govern-

ment regulations (e.g., in automotive, avionics,

or healthcare domains). In this respect, assess-

ing the quality of maintenance reporting can pre-

vent the company from having regulation non-

compliance issues, e.g. by carefully following

the data quality on each branch office and identi-

fying as soon as possible quality issues regarding

one or more dimensions;

6.2. Implications

This research presents three main theoretical im-

plications. First, it contributes to the literature on

maintenance management (MM) by proposing a thor-

ough state-of-the-art on the use of MCDM techniques

at each MM level (Strategic, Tactical, Operational),

which helps identifying criteria at each of these levels

(cf. Table 5). This list of criteria can be of potential

value to future researchers working in MM. Second,

this research contains an approach to identify relevant

data quality dimensions (based on existing data qual-

ity frameworks), and to turn them into a hierarchical

AHP structure. A theoretical framework is then pro-

posed, enabling the assessment and ranking of differ-

ent company branch offices in terms of maintenance

reporting quality. To the best of our knowledge, and

as evidenced through our state-of-the-art, this is the

first research work that addresses this specific goal.

Finally, the research also contributes three main

managerial implications. First, it enables organization

stakeholders to realize how important it is to moni-

tor and assess maintenance reporting practices, as it

can impact downstream but also upstream activities

of the organization. Second, the proposed dashboard

helps practitioners to quickly identify, based on their

needs and preferences, how one or a group of sites

behave (i.e., how good/bad they are) with respect to

one or more dimensions. This is helpful to estab-

lish their strategic plans to improve current practices,

which may result in savings of both money and time.

6.3. Limitations

The theoretical implications discussed above rely

both on the Krogstie’s data quality framework to iden-

tify key data quality dimensions, and AHP as MCDM

technique to structure these quality dimensions in the

form of a hierarchy that makes easier for maintenance

stakeholders to specify their needs. However, this re-

search has several limitations. First, only a few con-

cepts and relationships from the Krogstie’s framework

were considered (see red/bold elements in Figure 1),

which is due to the data that has been made available

by the Finnish OEM company, as well as to their own

expectations/needs. In future research work, the pro-

posed AHP framework and underlying criteria should

be extended to take into consideration the other con-

cepts/relationships such as Language Quality (e.g., for

domain appropriateness, participant knowledge ap-

propriateness. . . ), Syntactic Quality, etc.. Such an

extension might potentially require to combine AHP

with other tools and techniques for semantic pro-

cessing and matching purposes for example, or still

for handling uncertainty and vagueness in the expert

judgments/preferences (e.g., using fuzzy logic).

Furthermore, although it is already a great achieve-

ment for the Finnish company to be able to iden-

tify how good/bad their branch offices are in report-

ing maintenance data, we would have liked to carry

out a post-analysis to evaluate benefits of the post-

action plans carried out in the different branch of-

fices. For example, we are aware that the company

have developed on-site training programs, which have

been customized according to the quality results re-

lated to each site (e.g., if a site fails in addressing one

or more data quality dimensions, the training program

is customized accordingly). However, such a post-

analysis and insightful implications cannot be pro-

vided because of contractual obligations and project

constraints.

6.4. Future research

From a research perspective, we developed a frame-

work that makes possible the ranking of maintenance

sites based on their respective reporting quality. Cur-

rently, the proposed MRQA dashboard is limited to

the visualization of the site’s data quality score and as-

sociated rank. In future research work, we would like

to extend this tool, and particularly to re-use the final

site’s (AHP) data quality score as an input parameter

of a more advanced framework (e.g., that would in-

tegrate live sensor data from manufacture equipment)

that would make it possible to decide – in real-time

– what predictive failure model for machine is better

suited. This is based on two working assumptions:

• a weak assumption: the higher the maintenance

reporting quality score on-site (denoted by R(Al)

in this study), the higher the confidence of the

failure prediction;

• a strong assumption: the confidence of one or

more predictive models (e.g., binary logic model,

cox regression model, regression trees model. . . )

17



Table 5: Percentage of Criteria used in the Maintenance Management (MM) literature

Strategic Level Tactical Level Operational Level

Cost 22.7 Cost 21.9 Cost 25.8

Resource Availability & Utilization 10.1 Environment./Operation. Condition 14.1 Resource Availability & Utilization 19.4

Added Value 7.6 Safety 9.4 Added Value 4.8

Safety 7.6 Resource Availability & Utilization 9.4 DownTime & Time to repair 4.8

Reliability 7.6 Risk/Severity 9.4 Quality 4.8

Environment./Operation. Condition 5.0 DownTime & Time to repair 7.8 Risk/Severity 4.8

Quality 5.0 Reliability 6.3 Organizational Process 4.8

Risk/Severity 5.0 Added Value 3.1 Failure Frequency 3.2

Failure Frequency 4.2 Knowledge 3.1 Knowledge 3.2

Feasibility (implementation) 4.2 Resources Age 3.1 Environment./Operation. Condition 3.2

Repairability 3.4 Failure Frequency 1.6 Reliability 3.2

DownTime & Time to repair 3.4 Detectability 1.6 Safety 1.6

Flexibility 3.4 Feasibility (implementation) 1.6 Repairability 1.6

Knowledge 1.7 Maintenance Frequency 1.6 Abilities & development 1.6

Geographical Location 2.5 Comfort 1.6 Collaboration with Stakeholders 1.6

Component Failed 1.7 Automation 1.6 Operational Time 1.6

Detectability 0.8 Laws and Regulation 1.6 Maintenance Impact 1.6

Difficulty and Challenges 0.8 Number of affected people 1.6 Performance 1.6

Support and Services 0.8 Customer Category 1.6

Management & Organization 0.8 Number of sorties flown 1.6

Machine Uses 1.6

might evolve according to the reporting quality

score, whose evolution might even (potentially)

differ from one model to another. To put it an-

other way, we might assume that according to the

on-site maintenance reporting quality score (e.g.,

if R(Al) < 60%), a binary logic model might pro-

vide more confident predictions than a regression

trees model, or vice-versa.

The objective of future research will be to validate (or

invalidate) these two working assumptions and, if val-

idated, to propose a more advanced framework that is

able to switch between two or more predictive models

and react accordingly.

7. Acknowlegement

The research leading to this publication is sup-

ported by the Finnish Metals and Engineering Compe-

tence Cluster (FIMECC) S4Fleet program and the Na-

tional Research Fund Luxembourg (grant 9095399).

Appendix A. Percentage of criteria considered in

the Maintenance literature

Based on the summary matrix given in Table 1, re-

porting what MCDM techniques is commonly used at

each MM level, we carried out an in-depth analysis

to identify the most commonly used criteria at each of

these level in order to see whether data quality is prop-

erly addressed in the maintenance literature, and par-

ticularly regarding maintenance reporting activities.

The analysis outcome, regarding each MM level, is

given in the form of tabular in Table 5, which high-

light that data quality is hardly considered in the re-

viewed papers knowing that “Quality” refers, in most

of the reviewed papers, to other quality aspects than

Data Quality, except in (Van & Pintelon, 2014).

Appendix B. Penalty score selection

The methodology defined to tune the penalty score

consists in studying whether the introduced penalty

has a significant impact on the overall ranking. Let us

consider, in Eq. B.1, the pairwise comparisons as ratio

matrix introduced as example in section 4.2, where it

is assumed now that the form field(s) related to crite-

rion CC6 has/have been left empty in all reports car-

ried out on Site 1 (i.e., in 100% of the reports). Con-

sequently, I
CC6
fill

(A1) = 0, as highlighted in the first

column and row of the matrix in Eq. B.1. Our strat-

egy is to give out a penalty score (denoted by θ) as

eigenvalue to the corresponding site (i.e., to Site 1) as

shown in Eq. B.1.





































































A1 A2 . . . A54

A1 1 ✓✓
0

44
. . .

✚
✚
✚0

I
CB1
fill

(Al )

A2 ✓✓
44
0

1 . . . 0.67

.

.

.
.
.
.

.

.

.
...

.

.

.

A54 ✚
✚
✚ICB1

fill
(Al )

0
1.50 . . . 1





































































➠























































θ

W
A2
CC6

.

.

.

W
A54
CC6







































































































A1 A2 . . . A54

A1 — — — —

A2 — 1 . . . 0.67

.

.

.
.
.
.

.

.

.
...

.

.

.

A54 — 1.50 . . . 1

















































➠























































θ

W
A2
CC6

.

.

.
.
.
.

W
A54
CC6























































(B.1)

In order to select the penalty score θ, we propose to

carry out an analysis to determine whether the intro-

duced score impacts substantially or not on the overall

18



%corr

Siteranking

1

0

0

54

{CB = 0.6, CC = 0.2, CT = 0.2}

Jacc.

{CB = 0.4, CC = 0.4, CT = 0.2}

Jacc.

{CB = 0.4, CC = 0.2, CT = 0.4}

Jacc.

{CB = 0.2, CC = 0.6, CT = 0.2}

Jacc.

{CB = 0.2, CC = 0.4, CT = 0.4}

Jacc.

MRQA tool

{CB = 0.2, CC = 0.2, CT = 0.6}

θ = 0

Jacc.

{CB = 0.6, CC = 0.2, CT = 0.2}

{CB = 0.4, CC = 0.4, CT = 0.2}

{CB = 0.4, CC = 0.2, CT = 0.4}

{CB = 0.2, CC = 0.6, CT = 0.2}

{CB = 0.2, CC = 0.4, CT = 0.4}

{CB = 0.2, CC = 0.2, CT = 0.6}

θ = −1

Figure A.11: Comparison process – based on the Jaccard similarity coefficients – set up for similarity measurements between distinct site

rankings (i.e., considering various criteria preferences and penalty scores)

ranking. To this end, two distinct penalty scores are

considered:

• θ = 0: the site is penalized compared with the

other sites since any site that does not get a

penalty has automatically an eignevalue greater

than zero, or to be more precise 0 < W
Al
Cxh
< 1;

• θ = −p | p ∈ R−: the site is penalized compared

to the other sites, whose effect (unlikeθ= 0) is to

bring down the overall ranking when aggregating

all AHP dimensions/criteria.

To identify whether the penalty scores impact (in a

substantial manner) the overall ranking, we propose

– as depicted in Figure A.11 – to generate/compute

the alternative ranking for each penalty score (con-

sidering a given set of criteria weights/preferences)

and to compare whether the two rankings vary from

each other. This process has been tested for six com-

binations of criteria weights, as emphasized in Fig-

ure A.11. The similarity measure between distinct

rankings is based on the Jaccard similarity coeffi-

cients, whose principle is described in section Ap-

pendix B.1. Results and concluding remarks about the

penalty score selection are presented in section Ap-

pendix B.2.

Appendix B.1. Jaccard-based similarity measure

The Jaccard similarity coefficients (Tan et al., 2006)

can be used to measure a similarity between two dis-

tinct lists A and B, as formalized in Eq. B.2 (i.e., the

size of the list intersection divided by the size of the

list union). In our study, the union size is equal to the

number of alternatives/sites z. A Jaccard similarity

coefficient goes from 0 (no common list) to 1 (identi-

cal lists).

J(A, B) =
|A ∩ B|

|A ∪ B|
=
|A ∩ B|

z
(B.2)

Let A, B and C be three distinct lists consisting of

five sites {S1, ..,S5}, where each site receives a final

rank as presented in Figure B.12. In this example, two

Jaccard similarity coefficients J(A, B) and J(A,C) are

calculated. Both coefficients are equal because the in-

tersections |A∩B|and |A∩C| have the same cardinality.

A B C

S1 1 1 7

S2 2 2 6

S3 3 3 1

S4 4 6 2

S5 5 7 3

J(A, B) =
|A∩B|

z
=

|1,2,3|

|1,2,3,4,5|
=

3
5

J(A,C) =
|A∩C|

z
=

|1,2,3|

|1,2,3,4,5|
=

3
5

Figure B.12: Computation of Jaccard similarity coefficients

In our study, sites are ordered according to their

data quality score. It could be worthwhile to define

a similarity coefficient that would take into account

the rank. To this end, let us define Lq to be a sub-

list of L, where Lq consists of sites from rank 1 to q

(q ≤ z). A progressive similarity coefficient Jq(A, B)

can therefore be computed as in Eq. B.3.

Jq(A, B) = J(Aq, Bq) (B.3)

Figure B.13 details the evolution of the Jaccard

progressive coefficients Jq(A, B) and Jq(A,C) with q =

1,2, ...,5 (see lists A, B, and C given in Figure B.12).

Appendix B.2. Penalty score impact and selection

As previously stated and summarized in Fig-

ure A.11, the alternative ranking for each penalty

19



J1(A, B) =
|A1∩B1|

1
= 1.00 J1(A,C) =

|A1∩C1|

1
= 0.00

J2(A, B) =
|A2∩B2|

2
= 1.00 J2(A,C) =

|A2∩C2|

2
= 0.00

J3(A, B) =
|A3∩B3|

3
= 1.00 J3(A,C) =

|A3∩C3|

3
= 0.33

J4(A, B) =
|A4∩B4|

4
= 0.75 J4(A,C) =

|A4∩C4|

4
= 0.50

J5(A, B) =
|A5∩B5|

5
= 0.60 J5(A,C) =

|A5∩C5|

5
= 0.60

Figure B.13: Computation of Jaccard progressive coefficients

0

0.2

0.4

0.6

0.8

1

0 9 18 27 36 45 54

S
im

il
a
ri

ty
c
o
e
ffi

c
ie

n
ts

Site ranking

Criteria Preferences: {CB = 0.6,CC = 0.2,CT = 0.2}

Criteria Preferences: {CB = 0.4,CC = 0.4,CT = 0.2}

Criteria Preferences: {CB = 0.4,CC = 0.2,CT = 0.4}

Criteria Preferences: {CB = 0.2,CC = 0.6,CT = 0.2}

Criteria Preferences: {CB = 0.2,CC = 0.4,CT = 0.4}

Criteria Preferences: {CB = 0.2,CC = 0.2,CT = 0.6}

Figure B.14: Penalty score impact on site ranking: θ= 0 vs. θ=−1

score (i.e., for θ = 0 and θ = −1) is generated,

where the two resulting rankings are compared based

on the Jaccard similarity measure. In total, six sim-

ilarity comparisons are performed (cf. Figure A.11),

whose results are displayed in Figure B.14. These re-

sults show that the choice of the penalty score does

not lead to significant changes in the final ranking (al-

though a few sites move up or down between the 27th

and 36th positions), and is not dependent on the crite-

ria weights. Given this observation, the penalty score

θ = 0 has been chosen in this study. An additional

reason for choosing this score is that the sum of the

eigenvector values are equal to 1 (thus respecting the

eigenvector property/axiom), which is not true when

choosing θ=−1.

References

Ahmadi, A., Gupta, S., Karim, R. and Kumar, U. (2010). Selection

of maintenance strategy for aircraft systems using multi-criteria

decision making methodologies. International Journal of Relia-

bility, Quality and Safety Engineering, 17, 223–243.

Alarcón, M. J., Grau, J. B. and Torres, J. (2007). Application of

ELECTRE I method to restoration actions in telecommunication

network maintenance. IEEE.

Almeida, A.T. (2012). Multicriteria model for selection of preven-

tive maintenance intervals. Quality and Reliability Engineering

International, 28, 585–593.

Almeida-Filho, A., Ferreira, R.J. and Almeida, A. (2013). A DSS

based on multiple criteria decision making for maintenance

planning in an electrical power distributor. Springer.

Arputhamary, B. and Arockiam, L. (2015). Data Integration in Big

Data Environment. Bonfring International Journal of Data Min-

ing, 5, 1.

Azadeh, A., Sheikhalishahi, M., Firoozi, M. and Khalili,

S.M. (2013). An integrated multi-criteria Taguchi computer

simulation-DEA approach for optimum maintenance policy and

planning by incorporating learning effects. International Journal

of Production Research, 51, 5374–5385.

Azadeh, A., Sheikhalishahi, M., Khalili, S. M. and Firoozi, M.

(2014). An integrated fuzzy simulation–fuzzy data envelopment

analysis approach for optimum maintenance planning. Interna-

tional Journal of Computer Integrated Manufacturing, 27,181–

199.

Azizi, A. and Fathi, K. (2014). Selection of optimum maintenance

strategies based on a fuzzy analytic hierarchy process. Manage-

ment Science Letters, 4, 893–898.

Babashamsi, P., Golzadfar, A., Yusoff, N. I. M., Ceylan, H. and Nor,

N. G. M. (2016). Integrated fuzzy analytic hierarchy process and

VIKOR method in the prioritization of pavement maintenance

activities. nternational Journal of Pavement Research and Tech-

nology, 9, 112–120.

Behzadian, M., Khanmohammadi Otaghsara, S., Yazdani, M. and

Ignatius, J. (2012). A state-of the-art survey of TOPSIS applica-

tions. Expert Systems with Applications, 39, 13051–13069.

Bertolini, M., Bevilacqua, M., Braglia, M. and Frosolini, M. (2004).

An analytical method for maintenance outsourcing service se-

lection. International Journal of Quality & Reliability Manage-

ment, 21, 772–788.

Bertolini, M. and Bevilacqua, M. (2006). A combined goal pro-

gramming – AHP approach to maintenance selection problem.

Reliability Engineering & System Safety, 91, 839–848.

Bevilacqua, M. and Braglia, M. (2000). The analytic hierarchy pro-

cess applied to maintenance strategy selection. Reliability Engi-

neering & System Safety, 70, 71–83.

Blumenthal, A. L. (1977). The process of cognition. Prentice Hall/-

Pearson Education.

Cafiso, S., Di, G. A., Kerali, H. and Odoki, J. (2002). Multicri-

teria analysis method for pavement maintenance management.

Transportation Research Record: Journal of the Transportation

Research Board, 1816, 73–84.

Cavalcante, C. A. V. and Costa, A. P. C. S. (2010).Multicriteria

Model of Preventive Maintenance. Brazilian Journal of Oper-

ations & Production Management, 3,71–86.

Cavalcante, C. A. V., and Ferreira, R. J. P. and de Almeida, A.

T. (2010). A preventive maintenance decision model based on

multicriteria method PROMETHEE II integrated with Bayesian

approach. IMA Journal of Management Mathematics, 21, 333–

348.

Cavalcante, C.A.V. and De Almeida, A.T. (2007). A multi-criteria

decision-aiding model using PROMETHEE III for preventive

maintenance planning under uncertain conditions. Journal of

Quality in Maintenance Engineering, 13, 385–397.

Certa, A., Enea, M. and Lupo, T. (2013). ELECTRE III to dynami-

cally support the decision maker about the periodic replacements

configurations for a multi-component system. Decision support

systems, 55,126–134.

Charnes, A., Clark, C. T., and Cooper, W. W. and Golany, B. (1984).

A developmental study of data envelopment analysis in measur-

ing the efficiency of maintenance units in the US air forces. An-

nals of Operations Research, 2,95–112.

Chen, M., Mao, S., & Liu, Y. (2014). Big data: A survey. Mobile

Networks and Applications, 19, 171–209.

Chen, L., Weng, M. and Zhang, G. (2005). Utility Optimality

Method for Pipeline Integrity Maintenance Costs. Petroleum En-

gineering Construction, 2,004.

Chou, J. (2009). Web-based CBR system applied to early cost bud-

geting for pavement maintenance project. Expert Systems with

Applications, 36, 2947–2960.

Coulter, E. D., Sessions, J. and Wing, M. G. (2006). Scheduling

forest road maintenance using the analytic hierarchy process and

heuristics. Silva Fennica, 40, 143.

20



de Almeida, A. T. (2001). Multicriteria decision making on mainte-

nance: spares and contracts planning. European Journal of Op-

erational Research, 129,235–241.

de Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira, R.

J. P., de Almeida-Filho, A. T and Garcez, T. V. (2015). Decision

on Maintenance Outsourcing. Springer.

de Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira,

R. J. P., de Almeida-Filho, A. T. and Garcez, T. V. (2015).

Decisions on Priority Assignment for Maintenance Planning.

Springer.

de Almeida, A. T., Cavalcante, C. A. V., Alencar, M. H., Ferreira,

R. J. P., and de Almeida-Filho, A. T. and Garcez, T. V. (2015).

Preventive Maintenance Decisions. Springer.

de un Caso, E. (2008). The Efficiency of Preventive Maintenance

Planning and the Multicriteria Methods: A Case Study. Com-

putación y Sistemas, 12,208–215.

Dehghanian, P., Fotuhi-Firuzabad, M., Bagheri-Shouraki, S. and

Kazemi, A. A. R. (2012). Critical component identification in

reliability centered asset management of power distribution sys-

tems via fuzzy AHP. Systems Journal, 4, 593–602.

dGonçalves, C. D. F. and Dias, J. A. M. and Cruz-Machado, V. A.

(2014). Decision Methodology for Maintenance KPI Selection:

Based on ELECTRE I. Springer.

Duffuaa, S. O., Ben-Daya, M., Al-Sultan, K. S. and Andijani, A.

A. (2001). A generic conceptual simulation model for mainte-

nance systems. Journal of Quality in Maintenance Engineering,

7, 207–219.

Durán, O. (2011). Computer-aided maintenance management sys-

tems selection based on a fuzzy AHP approach. Advances in En-

gineering Software, 42, 821–829.

e Costa, C. A .B., Carnero, M. C. and Oliveira, M. D. (2012). A

multi-criteria model for auditing a Predictive Maintenance Pro-

gramme. European Journal of Operational Research, 217,381–

393.

Emovon, I., Norman, R. A. and Murphy, A. J.(2010). Hybrid

MCDM based methodology for selecting the optimum mainte-

nance strategy for ship machinery systems. Journal of Intelligent

Manufacturing, ,1–13.

Eslami, S., Sajadi, S. M. and Kashan, A. H. (2014). Selecting a

preventive maintenance scheduling method by using simulation

and multi criteria decision making. International Journal of Lo-

gistics Systems and Management, 18, 250–269.

Fallah-Fini, S., Triantis, K., Rahmandad, H. and Jesus, M. (2015).

Measuring dynamic efficiency of highway maintenance opera-

tions. Omega, 50,18–28.

Fang, C.-C., & Huang, Y.-S. (2008). A Bayesian decision analysis

in determining the optimal policy for pricing, production, and

warranty of repairable products. Expert Systems with Applica-

tions, 35, 1858–1872.

Farhan, J. and Fwa, T. (2009). Pavement maintenance prioritiza-

tion using analytic hierarchy process. Transportation Research

Record: Journal of the Transportation Research Board, 12–24.

Ferdousmakan, M., Vasili, M., Vasili, M., Tang, S.H. and Lim, N.T.

(2014). Selection of Appropriate Risk-based Maintenance Strat-

egy by Using Fuzzy Analytical Hierarchy Process. In 4rd Euro-

pean Seminar on Computing, Pilsen, Czech Republic (pp. 77).

Främling, K., Holmström, J., Loukkola, J., Nyman, J., and Kaustell,

A. (2013). Sustainable PLM through Intelligent Products. Engi-

neering Applications of Artificial Intelligence, 26, 789–799.

Fouladgar, M.M., Yazdani-Chamzini, A., Lashgari, A., Zavadskas,

E. K. and Turskis, Z. (2012). Maintenance strategy selection us-

ing AHP and COPRAS under fuzzy environment. International

journal of strategic property management, 16, 85–104.

Främling, K., Kubler, S., and Buda, A. (2014). Universal Messag-

ing Standards for the IoT from a Lifecycle Management Per-

spective. IEEE Internet of Things Journal, 1, 319–327.

Garcı́a-Cascales, M. S., and Lamata, M. T. (2009). Selection of

a cleaning system for engine maintenance based on the ana-

lytic hierarchy process. Computers & Industrial Engineering,

56, 1442–1451.

Garmabaki, A. H. S., Ahmadi, A. and Ahmadi, M. (2016).

Maintenance Optimization Using Multi-attribute Utility Theory.

Springer.

Gomez, A. and Carnero, M. C. (2011). Selection of a Computerised

Maintenance Management System: a case study in a regional

health service. Production Planning and Control, 22,426–436.

Goossens, A. J. M. and Basten, R. J. I. (2015). Exploring main-

tenance policy selection using the Analytic Hierarchy Process;

an application for naval ships. Reliability Engineering & System

Safety, 142, 31–41.

Ha, S. H. and Krishnan, R. (2008). A hybrid approach to supplier

selection for the maintenance of a competitive supply chain. Ex-

pert Systems with Applications, 34, 1303–1311.

Hankach, P. and Lepert, P. (2011). Multicriteria Decision Analysis

for Prioritizing Road Network Maintenance Interventions. Inter-

national Journal of Pavements, 10,.

Hjalmarsson, L. and Odeck, J. (1996). Efficiency of trucks in road

construction and maintenance: an evaluation with data envelop-

ment analysis. Computers & operations research, 23,393–404.

Hosseini Firouz, M. and Ghadimi, N. (2015). Optimal preven-

tive maintenance policy for electric power distribution systems

based on the fuzzy AHP methods. Complexity,, DOI 10.1002/c-

plx.21668.

Hwang, C. L., and Yoon, K. (1981). Multiple Attribute Decision-

Making Methods and Applications. Springer Verlag, Berlin, Hei-

delberg, New York.

Ilangkumaran, M. and Kumanan, S. (2009). Selection of mainte-

nance policy for textile industry using hybrid multi-criteria de-

cision making approach. Journal of Manufacturing Technology

Management, 20, 1009–1022.

Ilangkumaran, M. and Kumanan, S. (2012). Application of hybrid

VIKOR model in selection of maintenance strategy. Interna-

tional Journal of Information Systems and Supply Chain Man-

agement (IJISSCM), 5,59–81.

Jarke, M. and Vassiliou, Y. (1997). Data Warehouse Quality: A Re-

view of the DWQ Project. In Proceedings of the Conference on

Information Quality (IQ1997), Cambridge, MA (pp. 299–313).

Jeon, J., Kim, C. and Lee, H. (2011). Measuring efficiency of total

productive maintenance (TPM): a three-stage data envelopment

analysis (DEA) approach. Total Quality Management & Busi-

ness Excellence, 22, 911–924.

Jones-Farmer, L. A., Ezell, J. D. and Hazen, B. T. (2014). Applying

control chart methods to enhance data quality. Technometrics,

56, 29–41.

Kahn, B. K., Strong, D. M., and Wang, R. Y. (2002). Information

quality benchmarks: product and service performance. Commu-

nications of the ACM, 45, 184–192.

Köksal, G., Batmaz, l., and Testik, M. C. (2011). A review of data

mining applications for quality improvement in manufacturing

industry. Expert systems with Applications, 38, 13448–13467.

Krogstie, J., Lindland, O. I., and Sindre, G. (1995). Defining quality

aspects for conceptual models. In Proceedings of the IFIP8.1

Working Conference on Information Systems Concepts: Towards

a Consolidation of Views (ISCO), Marburg, Germany (pp. 216–

231).

Kumar, G. and Maiti, J. (2012). Modeling risk based maintenance

using fuzzy analytic network process. Expert Systems with Ap-

plications, 39, 9946–9954.

Kuo, T. C. and Wang, M. L. (2012). The optimisation of mainte-

nance service levels to support the product service system. In-

ternational Journal of Production Research, 50, 6691–6708.

Labib, A. W., O’Connor, R. F. and Williams, G. B. (1998). An ef-

fective maintenance system using the analytic hierarchy process.

Integrated Manufacturing Systems, 9, 87–98.

Levrat, E., and Iung, B. and Crespo Marquez, A. (2008). E-

maintenance: review and conceptual framework. Production

Planning & Control, 19, 408–429.

Li, C., and Xu, M. and Guo, S. (2007). ELECTRE III based on

ranking fuzzy numbers for deterministic and fuzzy maintenance

strategy decision problems. IEEE.

Li, J., Tao, F., Cheng, Y. and Zhao, L.. (2015). Big data in product

lifecycle management. The International Journal of Advanced

21



Manufacturing Technology, 81, 667–684.

Liu, J. and Yu, D. (2004). Evaluation of plant maintenance based on

data envelopment analysis. Journal of Quality in Maintenance

Engineering, 10,203–209.

Liu, M. and Frangopol, D. M. (2006). Decision support system for

bridge network maintenance planning. Springer.

Liu, J., LU, X. and QU, C. (2012). A Priority Sorting Approach

of Maintenance Task During Mission Based on ELECTRE TRI.

Fire Control & Command Control, ,S1.

Mardani, A., Jusoh, A. and Zavadskas, E. K. (2015). Fuzzy multi-

ple criteria decision-making techniques and applications – Two

decades review from 1994 to 2014. Expert systems with Appli-

cations, 42, 4126–4148.

Maurino, A. and Batini, C. (2009). Methodologies for data quality

assessment and improvement. ACM Computing Surveys, 41, 1–

52.

Moazami, D., Behbahani, H. and Muniandy, R. (2011). Pavement

rehabilitation and maintenance prioritization of urban roads us-

ing fuzzy logic. Expert Systems with Applications, 38, 12869–

12879.

Mobley, R. K. (2002). An introduction to predictive maintenance.

Butterworth-Heinemann.

Monte, M. B. S. and others. (2015). A MCDM Model for Preventive

Maintenance on Wells for Water Distribution. In IEEE Interna-

tional Conference on QSystems, Man, and Cybernetics (SMC),

2015 (pp.268–272).

Monte, M. B.S and de Almeida-Filho, A. T. (2016). A Multicrite-

ria Approach Using MAUT to Assist the Maintenance of a Wa-

ter Supply System Located in a Low-Income Community. Water

Resources Management, , 1–14.

Muchiri, P., Pintelon, L., Gelders, L. and Martin, H. (2011). De-

velopment of maintenance function performance measurement

framework and indicators. International Journal of Production

Economics, 131, 295–302.

Mumpower, J. L., Phillips, L. D., Renn, O. and Uppuluri, V. R. R.

(2012). Expert Judgment and Expert Systems. Springer Science

& Business Media.

Nyström, B. and Söderholm, P. (2010). Selection of maintenance

actions using the analytic hierarchy process (AHP): decision-

making in railway infrastructure. Structure and Infrastructure

Engineering, 6, 467–479.

Ofner, M., Otto, B. and Österle, H. (2013). A Maturity Model for

Enterprise Data Quality Management. Enterprise Modelling and

Information Systems Architectures, 8, 4–24.

Ouma, Y. O., Opudo, J. and Nyambenya, S.(2015). Comparison of

Fuzzy AHP and Fuzzy TOPSIS for Road Pavement Maintenance

Prioritization: Methodological Exposition and Case Study. Ad-

vances in Civil Engineering, ,DOI 10.1155/2015/140189.

Ozbek, M. E., de la Garza, J. M. and Triantis, K. (2010). Data and

modeling issues faced during the efficiency measurement of road

maintenance using data envelopment analysis. Journal of Infras-

tructure Systems, 16, 21–30.

Ozbek, M. E., de la Garza, J. M. and Triantis, K. (2010). Effi-

ciency measurement of bridge maintenance using data envelop-

ment analysis. Journal of Infrastructure Systems, 16, 31–39.

Palma, J., de León Hijes, F. C. G., Martı́nez, M. C. and Cárceles, L.

G. (2010). Scheduling of maintenance work: A constraint-based

approach. Expert Systems with Applications, 37, 2963–2973.

Peck, M. W., Scheraga, C. A. and Boisjoly, R. P. (1998). Assess-

ing the relative efficiency of aircraft maintenance technologies:

an application of data envelopment analysis. Transportation Re-

search Part A: Policy and Practice, 32, 261–269.

Pintelon, L. M. and Gelders, L.F. (1992). Maintenance management

decision making. European journal of operational research, 58,

301–317.

Pourjavad, E., Shirouyehzad, H. and Shahin, A. (2013). Selecting

maintenance strategy in mining industry by analytic network

process and TOPSIS. International Journal of Industrial and

Systems Engineering, 15, 171–192.

Pramod, V. R., Sampath, K., Devadasan, S.R., Jagathy Raj, V.P. and

Moorthy, G. D. (2007). Multicriteria decision making in main-

tenance quality function deployment through the analytical hier-

archy process. International Journal of Industrial and Systems

Engineering, 2, 454–478.

Roll, Y., Golany, B. and Seroussy, D.(1989). Measuring the effi-

ciency of maintenance units in the Israeli Air Force. European

Journal of Operational Research, 43, 136–142.

Rouse, P., Putterill, M. and Ryan, D. (2002). Integrated perfor-

mance measurement design: insights from an application in air-

craft maintenance. Management Accounting Research, 13, 229–

248.

Saaty, T. L. (1980). The Analytic Hierarchy Process. New York:

McGraw-Hill.

Sampaio, S. F. M., Dong, C. and Sampaio, P. (2015). DQ 2 S –

A framework for data quality-aware information management.

Expert Systems with Applications, 42, 8304–8326.

Shafiee, M. (2015). Maintenance strategy selection problem: an

MCDM overview. Journal of Quality in Maintenance Engineer-

ing, 21, 378–402.

Shahin, A., Pourjavad, E. and Shirouyehzad, H. (2012). Selecting

optimum maintenance strategy by analytic network process with

a case study in the mining industry. International Journal of Pro-

ductivity and Quality Management, 10, 464–483.

Sheikhalishahi, M. (2014). An integrated simulation-data envelop-

ment analysis approach for maintenance activities planning. In-

ternational Journal of Computer Integrated Manufacturing, 27,

858–868.

Shyjith, K., Ilangkumaran, M. and Kumanan, S. (2008). Multi-

criteria decision-making approach to evaluate optimum mainte-

nance strategy in textile industry. Journal of Quality in Mainte-

nance Engineering, 14, 375–386.

Simpson, L. (1996). Do decision makers know what they prefer?:

MAVT and ELECTRE II. Journal of the Operational Research

Society, 47, 919–929.

Sophie, S.-Z., Thomas, A., Dominik, L., Ralf, H., Pierrick, B.

and Javier, G.-S. (2014). Supreme sustainable predictive main-

tenance for manufacturing equipment. In European Congress

& Expo on Maintenance and Asset Management (EuroMainte-

nance), Helsinki, Finland, (pp. 1–6).

Sun, S. (2004). Assessing joint maintenance shops in the Taiwanese

Army using data envelopment analysis. Journal of Operations

Management, 22, 233–245.

Taghipour, S., Banjevic, D. and Jardine, A. K. S. (2011). Prioriti-

zation of medical equipment for maintenance decisions. Journal

of the Operational Research Society, 62, 1666–1687.

Tan, P.-N., Steinbach, M. and Kumar, V. (2006). Introduction to

data mining. Addison-Wesley Longman Publishing Co., Inc.

Tan, Z., Li, J., Wu, Z., Zheng, J. and He, W. (2011). An evalua-

tion of maintenance strategy using risk based inspection. Safety

science, 49, 852–860.

Thor, J., Ding, S. and Kamaruddin, S. (2013). Comparison of multi

criteria decision making methods from the maintenance alter-

native selection perspective. The International Journal of Engi-

neering and Science, 2, 27–34.

Triantaphyllou, E., Kovalerchuk, B., Mann, L. and Knapp, G. M.

(1997). Determining the most important criteria in maintenance

decision making. Journal of Quality in Maintenance Engineer-

ing, 3, 16–28.

Trojan, F. and Morais, D. C. (2012). Using ELECTRE TRI to sup-

port maintenance of water distribution networks. Pesquisa Op-

eracional, 32, 423–442.

Trojan, F. and Morais, D. C. (2012). Prioritising alternatives for

maintenance of water distribution networks: a group decision

approach. Water Sa, 38, 555–564.

Umbrich, J., Neumaier, S. and Polleres, A. (2015). Quality assess-

ment & evolution of Open Data portals. In 3rd International

Conference on Future Internet of Things and Cloud (FiCloud),

Roma, Italy (pp. 404–411).

Van den Bergh, J., De Bruecker, P., Beliën, J., De Boeck, L. and De-

meulemeester, E. (2013). A three-stage approach for aircraft line

maintenance personnel rostering using MIP, discrete event sim-

ulation and DEA. Expert Systems with Applications, 40, 2659–

22



2668.

Van Horenbeek, A. and Pintelon, L. (2014). Development of a

maintenance performance measurement frameworkusing the an-

alytic network process (ANP) for maintenance performance in-

dicator selection. Omega, 42, 33–46.

Vujanović, D., Momčilović, V., Bojović, N. and Papić, V. (2012).

Evaluation of vehicle fleet maintenance management indicators

by application of DEMATEL and ANP. Expert Systems with Ap-

plications, 39, 10552–10563.

Waisberg, D. (September 2015). Data Analytics: A Matrix for

Better Decision Making. https://www.thinkwithgoogle.

com/articles/data-analysis-a-matrix-for-better-

decision-making.html#utm_source=LinkedIn&utm_

medium=social&utm_campaign=Think

Wakchaure, S. S. and Jha, K. N. (2011). Prioritization of bridges

for maintenance planning using data envelopment analysis. Con-

struction Management and Economics, 29, 957–968.

Wang, R. Y., and Strong, D. M. (1996). Beyond accuracy: What

data quality means to data consumers. Journal of management

information systems, 12, 5–33.

Wang, J., Fan, K. and Wang, W. (2010). Integration of fuzzy AHP

and FPP with TOPSIS methodology for aeroengine health as-

sessment. Expert Systems with Applications, 37, 8516–8526.

Wang, L., Chu, J. and Wu, J. (2007). Selection of optimum main-

tenance strategies based on a fuzzy analytic hierarchy process.

International Journal of Production Economics, 107, 151–163.

Wellsandt, S., Wuest, T., Hribernik, K., and Thoben, K.-D. (2015).

Information Quality in PLM: A Product Design Perspective.

In Advances in Production Management Systems: Innovative

Production Management Towards Sustainable Growth (AMPS),

Tokyo, Japan (pp. 515–523).

Xu, L. and He, W. and Li, S. (2010). Internet of Things in indus-

tries: A survey. IEEE Transactions on Industrial Informatics,

10, 2233–2243.

Zaim, S., Turkyilmaz, A., Acar, M. F., Al-Turki, U. and Demirel, O.

F. (2012). Maintenance strategy selection using AHP and ANP

algorithms: a case study. Journal of Quality in Maintenance En-

gineering, 18, 16–29.

Zhangqiong, W. and Guozheng, S. (1999). A Study of Appraisal

and Decision-making Support System for Maintenance Scheme

of Metal Structure of Crane. Journal of Wuhan University, 5,

007.

23

https://www.thinkwithgoogle.com/articles/data-analysis-a-matrix-for-better-decision-making.html#utm_ source=LinkedIn&utm_medium=social&utm_campaign=Think
https://www.thinkwithgoogle.com/articles/data-analysis-a-matrix-for-better-decision-making.html#utm_ source=LinkedIn&utm_medium=social&utm_campaign=Think
https://www.thinkwithgoogle.com/articles/data-analysis-a-matrix-for-better-decision-making.html#utm_ source=LinkedIn&utm_medium=social&utm_campaign=Think
https://www.thinkwithgoogle.com/articles/data-analysis-a-matrix-for-better-decision-making.html#utm_ source=LinkedIn&utm_medium=social&utm_campaign=Think

	Introduction
	Data quality in expert maintenance systems
	Expert maintenance systems
	Data quality frameworks

	Research Methodology: Data quality framework instantiation to MRQA purposes
	Krogstie framework concepts and definitions
	MCDM-based Krogstie framework instanciation

	AHP-based MRQA framework
	Pairwise comparison based preference measurement
	Pairwise comparisons as ratio measurement
	Alternative ranking using TOPSIS

	OEM use case
	Scenario 1: Equivalence between criteria
	Scenario 2: Cost-reduction action plan

	Conclusions, implications, limitations and future research
	Conclusions
	Implications
	Limitations
	Future research

	Acknowlegement
	Percentage of criteria considered in the Maintenance literature
	Penalty score selection
	Jaccard-based similarity measure
	Penalty score impact and selection