Methods of expert estimations concordance for integral quality estimation M. P. Kuznetsova,∗, V. V. Strijovb aMoscow Institute of Physics and Technology, Institutskiy lane 9, Dolgoprudny city, Moscow region, 141700, Russia bNational Research University Higher School of Economics, 20 Myasnitskaya Ulitsa, Moscow 101000, Russia Abstract The paper presents new methods of alternatives ranking using expert estima- tions and measured data. The methods use expert estimations of objects quality and criteria weights. This expert estimations are changed during the computa- tion. The expert estimation are supposed to be measured in linear and ordinal scales. Each object is described by the set of linear, ordinal or nominal crite- ria. The constructed object estimations must not contradict both the measured criteria and the expert estimations. The paper presents methods of expert esti- mations concordance. The expert can correct result of this concordance. Keywords: expert estimations, integral quality, object ranking, preference learning 1. Introduction To make decisions about managed objects, for example, about nature pro- tected areas or state regions, one must rank this objects according to an integral quality estimations, or, equivalently, construct a binary preference function over the set of objects. To construct the integral quality estimation several steps must be performed [1]. 1. A quality criterion must be chosen to compare the objects. 2. A set of objects must be selected according to the quality criterion. 3. The expert must select a set of features describing the objects. 4. A design matrix ”objects-features” must be fulfilled. 5. The expert estimations of the objects quality and the criteria weights must be collected. Further we suppose that multicollinearity of the criteria is not significant. ∗Corresponding author Email addresses: mikhail.kuznecov@phystech.edu (M. P. Kuznetsov), strijov@ccas.ru (V. V. Strijov) Preprint submitted to Expert Systems with Applications July 29, 2013 Inna Typewriter Kuznetsov M.P., Strijov V.V. Methods of expert estimations concordance for integral quality estimation // Expert Systems with Applications, 2014, 41(4-2) : 1988-1996. We consider various scales for the expert estimations [2]: linear, ordinal and nominal. Each scale defines a method of transformation to be applied: for instance, any monotonic transformation can be applied to the ordinal scale. Decision making and preference learning propose several methods to esti- mate the integral quality of objects [3, 4]. Unsupervised methods construct the estimation using the objects description and the quality criterion. This paper introduces the principal component analysis as an example of the unsupervised methods [5, 6]. According to this method, an integral quality estimation is a projection of the objects to a first component of the design matrix. Also Pareto slicing and metric method can be regarded as the unsupervised methods. The supervised methods use expert estimations of the objects quality or the criteria weights [7] besides the design matrix. This paper presents the linear regression method [1], where the target variable is a vector of the expert-given object estimations. We consider linear and ordinal scaled expert estimations. The paper considers a linear model for objects quality estimation [8]. The expert estimations of the criteria weights and objects quality are measured in the linear or ordinal scale. In general, model-computed object estimations doesn’t equal expert-given estimations. This means that expert estimations and model- computed estimations contradict each other [9, 10]. We propose the method of the expert estimations concordance. We consider three various cases corre- sponding to the different types of measurement scales. The first case considers linear scaled both expert estimations and measured data. We propose the estimations concordance method as follows. A set of the admissible expert estimations is a segment restricted by the maximum and minimum value of the estimation. The model uses a structure parameter to find the solution as a point of this segment. The second case considers expert estimations of objects and criteria weights to be ordinal scaled. Ordinal-scaled expert estimations define a convex polyhe- dral cone. The design matrix defines a linear mapping of this cone to object space. The mapped cone can intersect a cone defined by the expert estimations of the objects. In this case the expert estimations of criteria weights and ob- jects supposed to be concordant. In the converse case, we present a method of ordinal concordance. The method minimizes a distance between the vectors in the cones. The third case considers ordinal scaled criteria [11]. The proposed method of objects quality estimation is as follows. Each criterion corresponds to the convex polyhedral con in objects space. According to the linear model, an admissible set of the object estimations is a Minkowski sum of the corresponding cones [12, 13]. The computed estimation is a projection of the expert estimation to the admissible set of values. The data: Nature Protected Areas’ Annual Report. We use the report to illus- trate the proposed methods. Table 1 shows a part of the data. The problem is to estimate efficiency of each NPA using the measured data and the expert estimations. 2 Table 1: Nature Protected Areas report with the expert object qualities and criteria weights estimations O b je ct n u m b er N a ti o n a l p a rk n a m e E x p e r t e s ti m a ti o n s o f o b je c t q u a li ti e s N u m b er o f en v ir o n m en ta l ex p er ti se s N u m b er o f in d iv id u a l g ra n ts N u m b er o f P h D s N u m b er o f em p lo y ee s N u m b er o f o rg a n iz a ti o n s a t th e N P A te rr it o ry N u m b er o f jo u rn a l p a p er s N u m b er o f jo u rn a l a u th o rs N u m b er o f st u d en ts 1.00 0.85 0.78 0.70 0.69 0.58 0.48 0.29 x1 NPA 1 1.00 3 0 1 4.5 3 3 2.5 0 x2 NPA 2 0.83 1 7 1 8 2 8.5 5 40 x3 NPA 3 0.67 2 1 1.5 9 1.5 9.5 6.5 66 x4 NPA 4 0.63 1 3 3 5 4.5 18 11 7 x5 NPA 5 0.58 0 12 8 19 11 7.5 11 62 x6 NPA 6 0.50 0 0 2 5 2.5 2.5 3.5 11 x7 NPA 7 0.44 1 5 4 11 20 16.5 15.5 4 x8 NPA 8 0.38 0 0 3 7 1 4.5 2.5 0 x9 NPA 9 0.33 0 6 0 7 1.5 7 4.5 1 x10 NPA 10 0.17 0 0 1 4 2 2.5 3.5 14.5 3 2. The integral quality estimation problem Denote by X = {xij} m,n i,j=1 a design matrix ”objects-features”. The object description is a vector xi, the i th string of the matrix X. The object ranking problem is to find a binary relation ≺ defined on the set of object pairs, xi ≺ xk. To solve this problem we find a mapping f : X → R, where X is a set of object values. A set of values R of the mapping f has a natural binary relation, that is finding a mapping f is sufficient to solve object ranking problem. Denote an integral quality estimation of the object xi by yi. We consider a linear model, where the value of integral quality yi is a linear combination of the criteria, elements of the vector xi: yi = f(w, xi) = n∑ j=1 wjxij. (1) Denote by f a vector of the values of the function f over the set of objects, f = [f(w, x1), ..., f(w, xm)] T = Xw, (2) where w is a vector of criteria weights. Let each criteria be mapped to the scale [0, 1]: xij 7→ xij − min i xij max i xij − min i xij , i ∈ {1, ..., m}, j ∈ {1, ..., n}. This paper considers the case of the full rank of X, rank(X) = n, with m > n. 3. Unsupervised integral quality estimation We will consider the principal component analysis as an unsupervised method for integral quality estimation. This method finds the objects’ projections to the principal component coordinates such that the sum of squared distances between the objects and their projections to the first component is minimum. Consider the orthogonal matrix W from the linear combination ZT = XTW where columns z1, ..., zn of the matrix Z have the maximum sum of variances, n∑ j=1 σ2(Zj) → max, where σ2(Zj) = 1 m m∑ i=1 (zi − z)2, z = 1 m m∑ i=1 zi. Columns of the matrix W are the eigenvectors of the covariance matrix Σ = XTX. The matrix Σ = XTX can be found using singular values decomposition 4 of the matrix XTX. Since Σ = XTX = WΛWT, it follows that ΣW = WΛ is an eigenvectors system of the matrix ΣW. Therefore the vector of object estimations ŷPCA is the projection of the row- vectors of the matrix X to its first principal component, and w is the first row- vector of the matrix W. This vector corresponds to the maximum eigenvalue of the matrix Σ. 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 x1 y1 x2 y2 x3 y3 x4 y4 x5 y5 x6 y6 x7 y7 x8 y8 x9 y9 x10 y10 Number of PhDs N u m b e r o f e m p lo y e e s Figure 1: The principle component method illustration Figure 1 illustrates principal component analysis for objects quality estima- tion. The black points are the NPAs from the table 1 describing by the criteria ¡¡Number of PhDs¿¿ and ¡¡Number of Employees¿¿. The PCA estimations yi of the objects xi are the points projections to the first principal component indicated by the blue line. 4. Supervised integral quality estimation The supervised methods use the model (1), the design matrix X, the expert estimations of the objects y0 or of the criteria weights w0. 4.1. The linear-scaled expert estimations Weighted sum. Consider the linear-scaled expert estimations of the criteria weights w0. The vector of the object estimations is the linear combination, ŷ = Xw0. This is the simplest method of objects estimation. The main drawback is the lack of robustness of the result estimations. The small changes of the expert estimations w0 may cause enormous changes in the result estimation. 5 0 0.2 0.4 0.6 0.8 1 0.5 1 1.5 2 PCA estimation, ŷP C A E s t im a t io n , y 1 = X w 0 x1x2x3 x4 x5 x6 x7 x8 x9 x10 Figure 2: Comparison of the weighted sum method with the principal component method Fig. 2 shows a comparison of the weighted sum estimation y1 = Xw0 with the principal component estimation ŷPCA. Each point is an object, NPA. The x- axis shows the principal component estimation ŷPCA, the y-axis — the weighted sum estimation y1. Expert-statistical method. Consider the linear-scaled expert estimations y0. The method obtains the criteria weights ŵ as the argument of minimum of a distance between the expert estimations y0 and the computed estimations y ′ 0 = Xŵ, ŵ = arg min w∈Rn ∥Xw − y0∥2. The solution of this problem is given by the ordinal least squares method, ŵ = (XTX)−1XTy0. (3) The vector of the object estimations y′0 = Xŵ. This vector is contained in space of the columns of the matrix X and is the nearest vector to the y0. 5. The expert estimations concordance In this section we will consider both expert estimations of the objects y0 and of the criteria weights w0. 6 5.1. Linear concordance of the expert estimations Consider the computed object estimations y1 = Xw0 using the vector w0 and the computed criteria weights w1 = X +y0 using the vector y0. Here the pseudo-inverse linear operator X+ = (XTX)−1XT. In other words, the linear operator X maps the expert estimations w0 to the vector y1, and pseudo- inverse linear operator X+ maps the expert estimations y0 to the vector w1. In general case the computed and the expert-given estimations are different, y1 ̸= y0, w1 ̸= w0. Call the expert estimations y and w concordant if the following conditions hold: y = Xw, w = X+y. (4) Hereafter we find the expert estimations under the conditions of concordance (4). Denote by y′0 = XX +y0 the projection of the vector y0 to the space of the columns of the matrix X. α-concordance method of the expert estimations. To resolve the contradiction in expert estimations let us consider the estimations wα ∈ [w0, w1] and yα ∈ [y1, y′0]. (5) y0 y′0 X+ X y1 w1 w0 X subspace, dim n Objects space, dim mFeatures space, dim n wα yα Figure 3: The α-concordance method illustration The pair of vectors wα, yα for the given α is defined by the following condi- tions, wα = αw0 + (1 − α)X+y′0, yα = (1 − α)y ′ 0 + αXw0. Theorem 1. The vectors wα, yα are concordant (4). Proof. It is easily proved that Xwα = yα and X +yα = wα: Xwα = αXw0 + (1 − α)XX+y′0 = αXw0 + (1 − α)y ′ 0 = yα, X+yα = (1 − α)X+y′0 + αX +Xw0 = (1 − α)X+y′0 + αw0 = wα. 7 Fig. 3 illustrates the α− concordance method. The vectors y0 w0 are the expert estimations of the objects and of the criteria weights. y′0 is the nearest point to the y0 in the n-dimensional subspace of the columns of the matrix X. The pair of vectors yα, wα is the concordant pair of the expert estimations. The parameter α defines expert preferences to the expert estimations of the objects versus the expert estimations of the criteria weights. If α tends to zero the expert prefers the estimations of the objects; if α tends to zero the expert prefers the estimations of the criteria weights. One could allow expert to assign the parameter α according to his own preferences. Another way to define the parameter α is to compute it as the argument of minimum of the residuals sum Q, Q = ∥w0 − wα∥ n + ∥y′0 − yα∥ m → min α . (6) γ-concordance method of the expert estimations. The γ-concordance method refuses from the restrictions (5). It finds concordant estimations in the neigh- borhoods of the vectors w0, y ′ 0 as a solution of the optimization problem (6) with a regularization parameter γ2 ∈ [0, +∞): wγ = arg min w∈Rn (ε2 + γ2δ2), where ε2 = ∥w0 − wγ∥2 and δ2 = ∥y′0 − yγ∥2. The solution of this problem is the vector of criteria weights, wγ = (X TX + γ2In) −1(XTy′0 + γ 2w0), (7) and the concordant estimations of objects, yγ = Xwγ. The parameter γ 2 defines expert preferences to the expert estimations of the objects versus the expert estimations of the criteria weights, so as α. y′0 w0 Objects space, dim mFeatures space, dim n X+ X ε2 = ∥w0 − wα∥2 δ2 = ∥y′0 − yα∥2 wα yα Figure 4: The γ-concordance method illustration Fig. 4 illustrates the method of γ−concordance. The radiuses of the circum- circles of the points w0 and y ′ 0 equal ε and δ, respectively. 8 0 0.2 0.4 0.6 0.8 1 0.5 1 1.5 2 α, γ(α) O b je ct es ti m a ti o n s, ŷ y1 y1 y2 y2 y3 y3 y4 y4 y5 y5 y6 y6 y7 y7 y8 y8 y9 y9 y10 y10 α−concordance γ−concordance (a) The change of the object estimations 0 0.2 0.4 0.6 0.8 1 0.74 0.76 0.78 0.8 0.82 0.84 0.86 0.88 α, γ(α) E rr o r v a lu e, Q α−concordance γ−concordance (b) The change of the error function Figure 5: α- and γ-concordance illustration Fig. 5 compares α- and γ-concordance methods. The x-axis shows the values of the parameter α changing from 0 to 1, whereas parameter γ is the function of α, γ = α 1 − α , so γ changes from 0 to ∞. The left figure shows object estimations chang- ing. The blue lines indicate object estimations for α-concordance, the red lines indicate γ-concordance. The extreme cases correspond to the expert estima- tions y′0 and y1 = Xw0, respectively. The right figure shows changing of the error function (6). In the case of γ-concordance this function has a global min- imum corresponding to the optimal value of the parameter γ. Fig. 6 compares the expert-statistical method and the γ-concordance method for the optimal value of γ defined by minimal value of the error function (6). The x-axis shows the expert-statistical estimations ŷOLS. The y-axis shows the γ-concordance estimations ŷγ. 5.2. Ordinal concordance of the expert estimations Let the expert estimations y0, w0 be measured in ordinal scales. It means that the set of the estimations is linearly ordered. Consider a cone in Euclidean space corresponding to this set. Without loss of generality suppose that the cone is described by the set of vectors y ∈ Rm with the following restrictions on the components of y: y1 > y2 > ... > ym > 0; w1 > w2 > ... > wn > 0. The set of such vectors y is described by the system of linear inequalities, Jmy 6 0, 9 0.2 0.4 0.6 0.8 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 OLS estimation, ŷOLS γ -E s t im a t io n , ŷ γ x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 Figure 6: Comparison of the expert-statistical method with the γ-concordance method where Jm is the m × m matrix, Jm =   −1 1 0 · · · 0 0 0 −1 1 · · · 0 0 ... ... ... ... ... ... 0 0 0 · · · −1 1 0 0 0 · · · 0 −1   . In the case of a random order yi1 > yi2 > yim > 0, the matrix of the system will be constructed from the Jm by permutations of the corresponding columns. In the same way, the cone of vectors w corresponding to the expert estima- tions of the criteria weights is described by the system of linear inequalities with the n × n-matrix Jn. Therefore the expert estimations y0 and w0 correspond to the m×m and n× n matrices Jm and Jn, respectively. Correspondence of convex polyhedral cones to the expert estimations. Denote by Y0 and W0 the cones, corresponding to the expert estimations in the space of objects and in the space of features, respectively, Y0 = {y|Jmy 6 0}, W0 = {y|Jnw 6 0}. The linear operator X maps the cone W0 of the expert estimations of the criteria weights w0 to the computed cone XW0. The linear operator XX+ maps the 10 cone Y0 of the expert estimations of the objects y0 to the cone Y′0 = XX +Y0. The cone Y′0 consists of the vectors from the subspace of columns of the ma- trix X. Fig. 7 illustrated the defined cones. W0 X+Y′0 Y0 Y′0 XW0X+ X Objects space, dim mFeatures space, dim n Figure 7: Cones corresponding to the expert estimations The ordinal-scaled expert estimations w0 and y0 are called concordant if the cones XW0 and Y′0 have a non-empty intersection besides the vector 0. In this case we can found vectors ŷ ∈ Y′0 and ŵ ∈ W0 satisfying the concordance conditions (4). Properties of polyhedral cones. Now we introduce the following properties of the polyhedral cones corresponding to the expert estimations. Lemma 2. If two cones have vertices in the origin, their intersection is a polyhedral cone. Proof. A polyhedral cone with the vertex in the origin is described by the sys- tem of linear inequalities. Let the first cone be described by the system X1w > 0 and the second cone — by the system X2w > 0. The intersection of this cones is described by the system with the matrix ( X1 X2 ) . In other words, their inter- section is the polyhedral cone with a vertex in the origin. Lemma 3. The locus Xw is a cone if X is a linear mapping. Proof. For any vector w ∈ W a vector λw ∈ W. Therefore, if y ∈ Y, we get λy = λXw = X(λw) ∈ Y = XW. This completes the proof. It follows that if W is a polyhedral cone, the linear operator X maps it to the polyhedral cone XW. A corresponding pseudo-inverse operator X+ maps the cone Y to the cone X+Y. 11 Lemma 4. W0 ∩ X+Y′0 = {0} ⇐⇒ XW0 ∩ Y′0 = {0}. Proof. Let us note that the cones W0 and Y′0 = XX+Y0 have the same dimen- sion n since rank(X) = n. This means that operators X, X+ imply one-to-one correspondence from the features space to the space of the columns of the ma- trix X. Let the vector w ∈ W0 ∩ X+Y′0 w ̸= 0. Then the corresponding vector Xw ∈ XW0 as well as Xw ∈ XX+Y′0 = Y ′ 0. That is the cones XW0 Y′0 intersect at non-zero vector Xw. Let the vector y ∈ Y′0 ∩ XW0 y ̸= 0. Then the corresponding vector X+y ∈ X+Y′0 as well as X+y ∈ X+XW0 = W0. That is the cones W0 and X+Y′0 intersect at non-zero vector X+y. From Lemma 4 it follows that for any vector wp of the cone Wp = W0 ∩ X+Y′0 there exists a unique vector y′p ∈ Y′p = XW0 ∩ Y′0 such that the vectors wp, y′p satisfy the conditions of concordance 4. Moreover, from the definition of the cone Y′0 it follows that for any vector y′p ∈ Y′p there exists a unique vector yp ∈ Y0 such that y′p = XX+yp. This implies that a concordant pair ŵ, ŷ must satisfy following conditions,   Jnŵ 6 0, ŷ = XX+y, Jmy 6 0. Optimization problem for ordinal-scaled expert estimations concordance. Now we formulate an optimization problem for ordinal-scaled expert estimations con- cordance. We will find the nearest vectors ŵ and y1 in the cones W0 and Y0 as follows: (ŵ, y1) = arg min w∈W0,y∈Y0 ∥X+y − w∥, subject to y ∈ Y0, w ∈ W0, ∥X+y∥ = 1, ∥w∥ = 1, (8) where ∥ · ∥ is the Euclidean metrics in the space Rm. 12 W0 X+Y′0 ŵ X+y1 Features space Figure 8: The nearest vectors of the cones This means that the computed vector of criteria weights ŵ is a monotonic transformation of the vector w0. At the same time the objects estimation ŷ = XX+y1 is the nearest point to the expert estimation y0 from the subspace of the columns of the matrix X. This method illustrated with fig. 8. The problem of the nearest vectors can be solved by maximizing the rank correlation. That is, we will find the vectors ŵ ∈ W0 and y1 ∈ Y0 such that Kendall correlation between ŵ and y1 is maximum: (ŵ, y1) = arg max w∈W0,y∈Y0 ρ(X+y, w) : ∥X+y∥ = 1, ∥w∥ = 1. The algorithm of minimizing distance between vectors in cones. Rewrite the problem (8) as follows: minimize ∥X+y − w∥ subject to (X+y)T X+y = 1 and wT w = 1, Jnw 6 0 Jmy 6 0. To solve this problem we propose an iterative algorithm consequently finding approximations of vectors y(2k), w(2k+1) at every even and odd iteration. Define the vector w(0) = w0 at the iteration k = 0. Denote by a = y (2k) and b = w(2k+1) the solutions of two consequent optimization problems: 2k : 2k + 1 : minimize ∥X+a − w(2k)∥ minimize ∥X+y(2k+1) − b∥ subject to (X+a)T X+a = 1, subject to bT b = 1 = 1, Jma 6 0. Jnb 6 0. While solving the optimization problem, define the constants w(2k) = a(2k−1) and y(2k+1) = b(2k). Since the target function and inequality constraints are convex, the solution will be found for finite number of iterations. Methods of convex optimization 13 to solve this problem are provided in [14]. To solve the problem of the rank correlation maximization we use a genetic algorithm. In the case of a non-trivial intersection of the cones W0 and X+Y0 the solution of (8) is a vector ŵ from an intersection of this cones and a vector ŷ satisfying ŷ = XX+y1 = Xŵ. That is, vectors ŷ, ŵ satisfy the concordance conditions (4). If an intersection of the cones is trivial, the proposed algorithm find the nearest non-concordant vectors. As in the case of linear scales, we present a method of the expert estimations concordance using a structure parameter α, yα = (1 − α)ŷ + αXŵ. Here the vector yα and the corresponding vector wα = X +yα define the cones Yα and Wα, respectively. Furthermore, the intersection Yα ∩ XWα ̸= ∅. As in the case of linear-scaled expert estimation concordance, the parameter α defines expert preferences to the expert estimations of the objects versus the expert estimations of the criteria weights. Below we present a method of con- structing the linear-scaled estimations from the computed ordinal-scaled esti- mations. Stable estimations with respect to the design matrix disturbance. Consider the computed cone Yp = Yα ∩ Wα and the design matrix X. Disturb the elements of this matrix, X∆ = X + ∆, with a normal-distributed noise, ∆ = δI, δ ∼ N(0, σ2). An image of the linear mapping y = X∆w is also normally distributed. According to the hypothesis call a stable solution yp the central vector of the cone Yp under the condi- tion ∥yp∥ = 1. The so-called Chebyshev point yp is a center of an incircle of the cone Yp. Find the maximum distance from the target vector yp to the cone faces as follows: ŷp = arg max yp∈Yp {∥yp − b∥2 : b ∈ Rm \ Yp, ∥yp∥2 6 1}. (9) Fig. 9 illustrates the Chebyshev point yp of the cone Yp. Expert estimation concordance using isotonic regression. Consider the special case of the problem (8) such that the expert estimations of the objects y0 are linearly scaled and the expert estimations of the criteria weights are ordinal- scaled. In this case, the problem can be formulated in the terms of the well- known isotonic regression problem [15, 16]. 14 Yp ŷp Objects space Figure 9: An stable solution: Chebyshev point yp Let w̃ = X+y0. Find the monotonic sequence w1 6 ... 6 wn as the nearest to the vector w̃, ŵ = arg minw∈Rn n∑ i=1 (w̃i − wi)2, wn > ... > w1. To make the expert estimation concordant, rewrite this problem with the struc- ture parameter λ. If λ tends to zero the expert prefers the estimations of the objects; if λ tends to zero the expert prefers the estimations of the criteria weights. Find the vector ŵ such that: ŵ = arg min w∈Rn ( 1 2 n∑ i=1 (w̃i − wi)2 + λ n−1∑ i=1 (wi − wi+1)+ ) . To solve this problem we use an algorithm proposed at [15]. 6. Expert estimations concordance for the ordinal-scaled criteria This section considers the case of the ordinal-scaled criteria. Write the ma- trix X as the concatenation of its columns, X = [χ1, ..., χn]. In the case of the ordinal criteria, the geometric shapes corresponding to the columns of X are cones X1, ..., Xn. As described above, each cone is defined by the system of linear inequalities, Xj = {xj|Jmj xj 6 0}, j = 1, ..., m, with the m × m matrices Jmj . 15 Consider a linear model of the objects quality estimation. In the terms of ordinal scales it means that the admissible set X of the object estimations ŷ is a set consisting of the all possible sums of vectors, X = {x| x = x1 + ... + xn, x1 ∈ X1, ..., xn ∈ Xn}. For the following consideration let us recall definition of the Minkowski sum. The Minkowski sum of two subsets L1 and L2 of the linear space is a set L ′ consisting of all possible sums of vectors from L1 and L2. Call an admissible set for the linear model a Minkowski sum, X = X1 + ... + Xn. To estimate vector of object qualities we construct an admissible set as the Minkowski sum of the convex polyhedra X1, ..., Xn. To do this we use a method from [13]. The proposed method computes a matrix of a system of linear in- equalities describing the sum of the polyhedra. The description of this method is given below. Let two convex polyhedra X1 and X2 be described by the following system of inequalities: X1 = {x1|J1x1 6 b1}, X2 = {x2|J2x2 6 b2}. The Minkowski sum of the polyhedra is the vector x satisfying the following conditions:   x − x1 − x2 = 0, J1x1 6 b1, J2x2 6 b2. Transform the system replacing the variable x1 = x − x2:{ J1x − J2x2 6 b1, J2x2 6 b2. (10) The following lemma describes the Minkowski sum of two polyhedra. Lemma 5. x ∈ X if and only if it exists x2 satisfying (10). This means that to find a vector x one must solve the system of linear inequalities, Cx2 6 d, C = ( −J1 J2 ) , d = ( b1 − J1x b2 ) . To solve this system we use the following version of the Minkowski-Farkas lemma. Lemma 6. Let J and b be a matrix and a vector. The system of linear in- equalities Jx 6 b is solvable iff yb > 0 for any vector y satisfying the following conditions: y > 0, yJ = 0. 16 In our case write the Minkowski-Farkas lemma as following, ∃x2 : Cx2 6 d ⇔ ∀z : CT z = 0, z > 0 → (d, z) > 0. Let V be a fundamental system of solutions (FSS) for this case. Therefore V = ( V1 V2 ) , where Vi is a FSS, corresponding to the matrix Ji. It follows that the condi- tion (d, z) > 0 must be rewritten as VT1 (b1 − J1x) + V T 2 b2 > 0. Denote J = VT1 J1, b = V T 1 b1 + V T 2 b2, and obtain the parameters J, b of the system of inequalities describing the Minkowski sum X1 + X2. To find the non-negative FSS of the system with the matrix V we use a method proposed in [13]. The solution of the concordance problem is the point ŷ nearest to the expert estimation y0 such that ŷ ∈ X . Having constructed the set X , define the com- puted object estimations as the projection PX (y) ∈ X satisfying the following conditions: ŷ = PX (y) = arg min z∈X ∥y − z∥. (11) The projection is unique due to the convexity of the set X . Fig. 10 illustrates a projection of the vector y0 to the admissible set X . X y0 ŷ y1 y2 y3 Figure 10: Projection of the point y0 to the cone X Fig. 11 compares the method of ordinal expert data (9) with the method of ordinal criteria (11). The x-axis shows the Chebyshev point estimation ŷCheb. The y-axis shows the projections ŷCones to the admissible set X . 17 0.2 0.4 0.6 0.8 0.45 0.5 0.55 0.6 0.65 0.7 0.75 Estimation ŷCheb for ordinal experts E st im a ti o n ŷ C o n e s fo r o rd in a l fe a tu re s x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 Figure 11: Comparison of the method of the ordinal expert data with the method of ordinal criteria 7. Computational experiment Results for the real data. Table 2 shows estimations of the Nature Protected Areas obtained by the four proposed methods. The results are comparable. The specific method should be chosen according to some knowledge or assumptions about the data structure. Analysis of the algorithms accuracy. The results of four proposed algorithms are demonstrated in the table 3. As the quality criterion we propose a cor- relation coefficient between expert estimation of objects y0 and the computed estimations ŷ. We use Pearson correlation coefficient, r(y0, ŷ), to measure the Table 2: Integral quality estimations for the Nature Protected Areas Object number ŷOLS ŷγ ŷCheb ŷCones x1 1 2 1 1 x2 5 5 4 3 x3 6 3 2 2 x4 2 6 5 5 x5 3 1 6 6 x6 8 9 9 9 x7 4 4 3 4 x8 9 8 8 8 x9 7 7 7 7 x10 10 10 10 10 18 Table 3: Accuracy of the proposed algorithms ŷOLS ŷγ ŷCheb ŷCones Pearson, r 0.69 0.55 0.6 0.66 Kendall, τ 0.47 0.47 0.38 0.51 quality in the linear scales and we use Kendall correlation coefficient, τ(y0, ŷ), to measure quality in the ordinal scales. Note that the estimation ŷ was com- puted using Leave-One-Out method. Thus we can estimate a generalization ability of the proposed algorithms. The results show the object estimations ŷ computed by the four proposed methods. 1. ŷOLS — estimations computed by the expert-statistical method (3), 2. ŷγ — estimations computed by the γ-concordance method (7), 3. ŷCheb — estimations computed by the Chebyshev point finding (9), 4. ŷCones — estimations computed by ordinal criteria method (11). The results show that Pearson correlation has the maximum value for the OLS- estimation, whereas Kendall correlation is maximum for the ordinal criteria method. Analysis of the algorithms stability. To analyse stability of the proposed algo- rithms we disturb the elements of the matrix X. Consider the matrix X∆ = X+∆, where ∆(i, j) ∼ N(0, σ). We change the standard deviation σ of the dis- turbance from its minimum value σ = 0 to its maximum value σmax. Fig. (12) shows how changes quality criteria for the estimations ŷ = f(ŵ, X∆). The left figure shows the changing of Pearson correlation. For the non-disturbed matrix X the expert-statistical method gives the best result but this result is less stable. The most stable results are indicated with the green line (ordinal criteria) and the black line (ordinal expert data). Software implementation. The proposed methods were realized using MATLAB language. The open access code of algorithms and the computational experiment are located at [17]. The code consists of the two main modules. The module comparison.m performs pairwise algorithms comparison. The results of this module are illustrated at fig. (2), (11), (6). The module test_noise.m tests algo- rithms (9), (7), (11), (3) precision and stability. The results of this module are shown above in this section. The input data structures X, y0, w0 are the design matrix X and the expert estimations of the object qualities and of the criteria weights, respectively. This data correspond to the table 1. 19 0 0.1 0.2 0.3 0.4 0.5 0.2 0.3 0.4 0.5 0.6 Noise std, σ P e a r s o n r OLS Gamma Ordinal features Ordinal experts (a) Pearson correlation, r 0 0.1 0.2 0.3 0.4 0.5 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Noise std, σ K e n d a ll τ OLS Gamma Ordinal features Ordinal experts (b) Kendall correlation, τ Figure 12: Analysis of the algorithms stability for the disturbed matrix X 8. Conclusion The paper presents the methods of objects integral quality estimation based on expert estimations and measured data. Unsupervised and supervised meth- ods are considered. The paper proposes the methods of the linear and ordinal expert estimations concordance. The methods use a structure parameter defin- ing expert preferences to the expert estimations of the objects versus the expert estimations of the criteria weights. The paper presents the method of the stable object estimations construction and the method of the ordinal-scaled objects in- tegral quality estimation. The error analysis is performed. The integral quality estimations are constructed for the Nature Protected Areas’ annual reports. References [1] V. Strijov, G. Granic, et al, Integral indicator of ecological impact of the croatian thermal power plants, Energy 36 (2011) 4144–4149. [2] A. Khurshid, H. Sahai, Scales of measurements: An introduction and a selected bibliography, Quality and Quantity 27 (1993) 303–324. [3] J. Fuernkranz, E. Huellermeier, Preference learning, Springer, 2011. [4] V. F. Lopez, F. de la Prieta, M. Ogihara, D. D. Wong, A model for multi- label classification and ranking of learning objects, Expert Systems with Applications 39 (2012) 8878–8884. [5] I. T. Jolliffe, Principal Component Analysis, Springer, 2002. [6] D. Kim, B.-J. Yum, Collaborative filtering based on iterative principal component analysis, Expert Systems with Applications 28 (2005) 823–830. 20 [7] S. Jullien-Ramasso, G. Mauris, P. B. L. Valet, A decision support system for animated film selection based on a multi-criteria aggregation of referees ordinal preferences, Expert Systems with Applications 39 (2012) 4250– 4257. [8] S. R. Searle, Linear models, John Wiley and Sons, 1997. [9] V. I. Danilov, A. I. Sotskov, Social Choice Mechanisms, Springer-Verlang, 2002. [10] S. D. G., Mathematics and voting, Notices of the American Mathematical Society (4/2008). [11] H. M. Moshkovich, A. I. Mechitov, D. L. Olson, Rule induction in data mining: effect of ordinal scales, Expert Systems with Applications 22 (2002) 303–311. [12] E. Fogel, D. Halperin, Exact and efficient construction of minkowski sums of convex polyhedra with applications, Computer-Aided Design 39 (2007) 929–940. [13] M. Uhanov, Polygons sum construction algorithm, Bulletin of South Ural State University, Series ”Mathematics, Physics, Chemistry” 1 (2001) 39–44. [14] S. Boyd, L. Vandenberghe, Convex Optimization, Cambridge University Press, 2006. [15] R. Tibshirani, H. Hoefling, R. Tibshirani, Nearly-isotonic regression, Tech- nometrics 53 (2010) 54–61. [16] H. Hoefling, A path algorithm for the fused lasso signal approximator, Journal of Computational and Graphical Statistics 19 (2010) 984–1006. [17] M. P. Kuznetsov, V. V. Strijov, Algorithms of the integral quality estimation, 2013. URL: http://svn.code.sf.net/p/mlalgorithms/code/PreferenceLearning/. 21