UNIVERSITY OF 
 
 ILLINOIS LIBRARY 
 
 AT URBANA-CHAMPAIGN 
 
 BOOKSTACKS 
 
Digitized by the Internet Archive 
 
 in 2011 with funding from 
 
 University of Illinois Urbana-Champaign 
 
 http://www.archive.org/details/generalizedconst63ster 
 
Faculty Working Papers 
 
 GENERALIZED CONSTRAINED DERIVATIVES 
 Ronald J. Stern 
 
 #63 
 
 College of Commerce and Business Administration 
 
 University of Illinois at U r ba n a - C h a m p a i g n 
 
FACULTY WORKING PAPERS 
 College of Comtnerce and Business Administration 
 University of Illinois at Urbana-Champaign 
 September 12, 1972 
 
 GENERALIZED CONSTRAINED DERIVATIVES 
 Ronald J. Stern 
 
 #63 
 
 To be submitted to the Journal of Operations Research 
 
GENERALIZED CONSTRAINED DERIVATIVES 
 by 
 Ronald J. Stem* 
 
 k 
 
 ABSTRACT 
 
 In this paper we deal with a general equality constrained minimi- 
 zation problem. The concept of the constrained derivative as due to 
 Wilde et al. , [5] - [8] is extended by considering more general parti- 
 tions of the vector of unknowns into state and decision components and 
 utilizing the properties of the generalized matrix inverse. The main 
 result is a generalization of a sufficient condition of Wilde, guaran- 
 teeing the existence of a local minimum. 
 
 ^Department of Business Administration 
 University of Illinois 
 Urbana, Illinois 61801 
 
GENERALIZED CONSTRAINED DERIVATIVES 
 
 In equality constrained rainimization problems , a test for a local 
 minimum is to verify the stationarity of a Lagrangian and the definite- 
 ness of a certain quadratic form. Using the concepts of state and 
 decision variables and constrained derivatives Wilde et al. [5] - [8] 
 showed that stationarity of the Lagrangian and definiteness of a quadratic 
 form of reduced dimension is sufficient. In this paper we introduce 
 the concept of^a generalized constrained derivative and derive a 
 generalization of Wilde's sufficient condition. The result obtained here 
 does not depend on the local existence of a non-singular sub-Jacobian 
 of the constraint Jacobian, but rather a sxib-Jacobian of maximal full 
 celuBtn rank. 
 
 1. NOTATION AND PRELIMINARIES 
 
 Let A be a real mxn matrix. We denote the range of A by R(A) and 
 the nulls pace of A by N(A). The transpose of A is denoted by A*. The 
 generalized inverse of A is denoted by A"*". If A is square and non-singular 
 then A"*" = A~^, the inverse of A. 
 
 For references on the generalized inverse the reader is referred to 
 e.g. [1] and [2]. A"*" is uniquely defined for any (raxn) ve^l matrix A, 
 and computational methods for determining it may be;-found for example in 
 [23. Some particular properties of A"*" relevant to this paper are now given. 
 
 (1.1) AA"*"' = P 
 
 D ( A 1 ' the projection onto R(A) , 
 
 (1.2) I - A'^'A = Pjj/.x» the projection onto N(A) , 
 Projection maps are discussed in [4], 
 
In view of (1.1) and (1.2) we have the following result on linear 
 equations: If yeR(A) then the general solution to Ax = y is given by 
 
 (1.3) X = A'^'y + (I-A''"A)s for any zeR". 
 
 We will make lise of the following analytic notation. Let h:R" ■*■ R^. 
 Let I [ I I be the Eucledian norm. Then 
 
 h(t) = o|I t 11^ means lim J^ = 0. 
 
 *^ I! t !|k 
 
 The Jacob ian of h at t^ , if it exists, will be denoted (12.) . Here 
 
 o 
 
 and the Jacobian is the (mxn) matrix |~j r jJHi — ^ I inhere (i = 1,2,..., in). 
 
 9h,-(to) 
 
 f ah \ _ / 9hi(t 
 
 Uj^^ \ 3t, 
 
 (j = 1,2,... ,n) and to eR"^. 
 
 If m = 1 we write [— J = V^h(tQ) , the visual gradient notation. 
 
 to 
 
 Also, when m = 1, the Hessian of h at t^, if it exists, i;: -.h'-. (nxn) 
 
 /A 
 
 matrix r^^'^^oM where (i = 1,2,. ..n) and (j = 1,2, ...,n). 
 I3ti3tjl 
 
 2. CONSTRAINED DERIVATIVES 
 
 In this section a brief review of the concepts to be generalized vril' 
 be presented. The problem which we will consider, denoted P, is given by 
 
 F- minimize y(x) 
 
 f^(x) =0 (i = 1,. .. ,m) 
 
 subject to 
 
 xeR 
 
 -2- 
 
The fHonctions y, f-,^,. . . jf^^^ are assumed to be twice differentiable on 
 r'^. The constraints are also written f(x) = 0. 
 
 Partition the vector x = (-g-j , where ser'" is referred to as the 
 state vector and deR ~ is called the decision vector. Assume a feasible 
 
 " ~ Lf\ "~ — 
 
 point Xq is such that the Jacobian [.^j is non-singular. By Taylor's 
 
 l9slx^ 
 
 theorem ' ' 
 
 (2.1) 3y = 7^y(xQ)3d + Vgy(xo)9s + oI|3x|I 
 
 If Xq is a feasible point then by Taylor's theorem the feasibility of 
 x + 9x is equivalent to 
 
 (2.2) (lii 9s +[lf| 3d + oM9x| 
 9s /x^ »9d/x^ 
 
 Since f—\ is invertible (2.2) yields 
 
 ''-'' ^^ = -(C te)x/^-°"^'^"" 
 
 The constrained derivative of y at x with respect to the partition 
 i---| is defined by 
 
 (2 
 
 ,.) (U) ^^ = ,,.(.„) - v.<s, (If) - (II) ^ 
 
 From (2.1) - (2.4) we obtain 
 (2.5) 9y = [^ 8d + o|!9xj| 
 
 \^^} Xq 
 
 Since 9d is free to range over R^""" we have the following result: 
 
 A necessary condition for x,, to be a local minimum of ¥ is that j^iLj = 0. 
 
 \oaJv 
 
Again consider problem P but now assume y, f,, ..., f^o are three . 
 times differentiable. Let H denote the Hessian of the objective function 
 and let H^ denote the Hessian of the ith constraint at x . For any set 
 of numbers (Xj^, X2, ...» ^■^) we have 
 
 (2.6) 9y = 1- 
 
 8x 
 
 y- E X.f. 8x + i ax* IH - E X.H. j 3x + o| | 9x| | 
 
 i=l J 2 ^ i=i -- 
 
 If (X , ... X ) are Lagrange multipliers , that is, satisfy 
 
 m 
 
 m 
 
 (2.7) Vy(xQ),= E X.Vf^(x ) 
 
 i=l 
 
 then 
 
 / m _ \ 
 
 (2.8) 9y = i 3x* H - E X^H^ | Sx + o| 1 3x| j". 
 
 Notice that a unique vector of Lagrange multipliers is guaranteed to 
 
 exist at x , by our assumption on j— ) 
 
 m _ \9s/Xq 
 
 Write P = H - E X. H.. Appropriate partitioning P, we have 
 i=l ■'■ """ 
 
 (2.9) 3y = 1 Od'^ 9s*) L!^^J~as_j hd\ ^ o||9xl|^ 
 
 \ CIS I SS / 
 
 Upon defining 
 
 and making use of (2.3) and (2.9) we obtain 
 
 (2.10) 3y.= J 3d* S^^ + o||3x||^ 
 
 which yields the following result : A sufficient condition for Xp to be a 
 
 m 
 local minimum of problem F is that the Lagrangian y - E T.f . is stationary 
 
 i=l ^ ^ 
 at x^ and S^^ is positive definite . 
 
All the material in section 2 can be found in [7], 
 
 3. GENERALIZED CONSTRAINED DERIVATIVES 
 
 Again consider problem P, with the functions y, f-|_, ..., f^^ being 
 twice differentiable where x is a feasible point. Now take a partition 
 y = (-J-) where s is not necessarily an Tn-vector . 
 
 An auxiliary problem, denoted P , is noftt introduced: 
 
 ^O 
 
 P^ ■* minimize y(x) 
 
 s^iectto ^M.) my f(x) = 
 ^ 'x^ ^ 'x„ 
 
 n 
 xeR 
 
 Assume thatf — -j is a maximal full column rank submatrix of {—] . 
 Feasibility of x^ + 5^^ for problem P^^ is equivalent to 
 
 Since l^\ |li.\ - P / , . \ and by the above assumption on |— | , 
 
 l3sL l3sL R 3i\ ^ Ui 
 
 "o 
 (3.1) may be rewritten as follows: 
 
 o i\a<=/ I o 
 
 "•^> (fi '^*(i) ''i^] (sr°ii-ii=° 
 
 '^O '^o 
 
 -5- 
 
Since all three terms in this equation are in R i — ^1 we can solve for 3s 
 
 and obtain 
 
 ^o '^o 
 
 The generalized constrained derivative of y at x^ with respecf to 
 the partition |-|-| is defined by 
 
 (3.., (|.) - v,y(x„, - V(.<o' (f); (||}_^ 
 
 '^o -. ' '^o ^o 
 
 Similarly to (2.5) we have 
 (3.5) ay = f^ 9d + oi |3x| | 
 
 ^o 
 
 and the following fact: 
 
 A necessary condition for x^^ to be a local minimum of P is that 
 
 ^o 
 
 ^\ = 0. 
 
 It is not necessarily true that the generalized constrained derivative = 
 
 V^d.Xo 
 
 at a local minimum of problem P, however, as is evidenced by the following 
 
 counterexample. Consider the problem 
 
 ... , . 2 2 
 minimize y(x) = x + x + x 
 
 9 1 
 
 subject to f-j^(x) = 3<j - ^2 ~ ^3 ~ ^ 
 
 f^Cx) = Xg^ ^ 
 
 The solution obviously is (0,0,0) and (— -j = ( j. Upon setting 
 
 /x \ IT '•* o ^ 
 
 s = X, , d ~ f 2|we obtain f=Xj = (0,1). 
 ^- 1^3/ Ud/ 
 
 3' 
 This occurs for the following reason: In expression (2.2), when 
 
 
 s not square , the terms o j | 3x | | need not be in R { — j , and one 
 
 o 
 
 -6- 
 
cannot proceed to solve (2.2) for 3s by using a generalized inverse. 
 This, it can be verified, is precisely the case in the aboye counter- 
 example. This difficulty cannot occur, of course, if the constraints 
 in problem P are linear ; that is, f(x) = Ax-b. The generalized constrained 
 derivative for a problem with linear constraints is 
 
 • ^l^j = V^y(xo) - V^yCx^) A^ A^ 
 
 where A = (A^ ' A,) = f™V 
 •^ i a \3x/ 
 
 For the case where P has linear constraints it is necessary •'' hat the 
 generalized constrained derivative be at a local minimum . If y(x) is 
 linear and Ag is non-singular, then the last statement is equivalent to 
 saying that the simplex criterion usually denoted by z-j-c is for variables 
 X. in the basis. Different partitions I ---j correspond to different vertices 
 of the feasible region. The state variables are the basic variables and 
 the decision variables are the variables not in the basis. (See e.g. [3]). 
 Observe that 
 
 (3.6) X is a local minimum solution of P ==£> x is a local minimum 
 
 O Xq o 
 
 solution of P 
 Thus a sufficient condition for x,^ to be a local miniraini of P is 
 
 O V 
 
 ^O 
 
 sufficient condition for P as well. Such a condition will now be derived. 
 
 Let X be a feasible point for problem P. It is clearly then feasible 
 
 for P„ . Again assume that ( — | is a maximal full column rank submatrix of 
 ^o V^sL 
 
 /8f\ ° 
 
 l~~-] . AI90, assume now that the functions y, f, , ..., fj„ are three times 
 
 ^'O 
 
 differentiable. Assume 
 
(3,7) Vy(x ) G R fM^ 
 
 o 
 
 This implies the existence of a (not necessarily unique) vector of Lagrange 
 multipliers ( A-, , . . . , A ) . ' 
 
 Following the development from (2.6) - (2.10), using (3.3) in place 
 of (2.3), and (3,6), we see that a sufficient condition for x„ to he a local 
 
 ■ ■ I > - 1 ■ . o III 
 
 rainimuiTi of P is that (3.7) holds and S* is positive definite. Here Si, 
 is the square matrix 
 
 ^dd ~ ^dd " ^ds 
 
 (i): (M) 
 
 \ft 
 
 p fMV /9i\ 
 
 ss \3sl \^^] 
 
 Xq/ x^ x^ 
 
 m 
 
 where P = H - Z A.H. is appropriately partitioned . 
 1=1 
 
 CONCLUDING REMARKS 
 
 /9f\ 
 Upon locating a non-singular sub-Jacobian of (-— 1 and verifying 
 
 _ ■^o 
 
 that the Lagrangian of problem P is stationary at x for a Lagrange 
 
 multiplier A, Wilde has defined a matrix of "constrained second derivatives," 
 
 B^,... vrhose definition involves a. S,, is of dimension (n-m x n-m) , 
 
 where rn is the nuirher of constraints. The definiteness of Sj, guarantees 
 
 that X is a local sxtremum of problem P. 
 
 In this paper we have treated the case where a non-singular sub- 
 
 Jacobijan may not exist , but we considered maximal full column-rank sub- 
 
 Jacobians. • By 5-ntroducing an "artificial" problem, P^ , and using the 
 
 Xq 
 
 properties of the genereilized matrix inverse, vre defined SI,, which is an 
 
 -8- 
 
(n-p X n-p) analog of S^^. Here p <_ m. Under stationarity of the Lagrangian 
 of F the definiteness of S^^ guarantees a local extremura. It can be seen, 
 by following the arguments of section 3, that if p = m, then our result is 
 precisely that of Wilde. 
 
 ACKJIOWLEDGEMENT 
 
 The author wishes to thank Professor Adi Ben-Israel for discussions 
 on this subject. Comments and corrections made by the preliminary 
 referee were very helpful. The example in section 3 is due to his report. 
 
 ~9- 
 
REFERENCES 
 
 [1] A, Ben-Israel and A. Chsjnnes , "Contributions to the Theory of 
 Generalized Inverses," J. SIAM , 11, (1963), 667-699. 
 
 [2] A. Ben-Israel and D. Cohen, "Oi. Iterative Computation of Generalized 
 
 Inverses and Associated Projections," J. SIAH-Num. An. , 3_ (1966, 410-1+19. 
 
 [3] S. Gass, Linear Programming , McGraw-Hill, Nevj York, 1954- . 
 
 [4] P. Halmos , Finite-Dimensional Vector Spaces , Van Nostrand, Princeton, 
 1958. 
 
 [5] D. Wilde, "Differential Calculus in Nonlinear Programming," 
 Operation^ Research , 10 (1952), 764-773. 
 
 [6] D. Wilde, "Jacobians in Constrained Nonlinear Optimiaat'.on," 
 
 Operations Research , 13 (1965), 848-856. • ' 
 
 [7] D. Wilde 5 C. Beightler, Foundations of Optimization , Prentice Hall, 
 Englewood Cliffs, 1967. 
 
 [8] D. Wilde £ G. Reklaitis , "A Computationally Compact Sufficient 
 
 Condition for Constrained Minima," Operations Research , 17 (1969), 
 425-235.