137816 529..550 Expert systems and mass appraisal John Kilpatrick Greenfield Advisors LLC, Seattle, Washington, USA Abstract Purpose – The purpose of this paper is to examine the usefulness of a heuristic expert system, to show its applicability to real-world valuation problems, and to suggest several avenues for statistical testing. Design/methodology/approach – The expert systems follow a traditional sales adjustment grid format, with sufficient data for non-parametric testing. Findings – The paper finds that, while non-parametric statistics provide weaker results than traditional (e.g. hedonic regression) modeling, the technique provides a statistically testable model useful in situations with limited data and/or poorly characterized probability functions. Practical implications – This paper addresses the conundrum faced by real estate valuers on the lack of statistical underpinnings of traditional heuristic models. Originality/value – This is one of the first empirical studies in the valuation literature exploring statistical characterization of heuristic valuation methods. Keywords Expert system, Property values, Sales adjustment grid, Non-parametric statistics, Property, Fair value, Sales management, United States of America Paper type Research paper 1. Introduction The purpose of this paper is to explore what is known about expert systems, a set of methodologies within the category of real estate appraisal sales comparison approaches which combines the heuristic characteristics of the sales adjustment grid with some of the statistical power of regression modeling. Expert systems are useful when the appraiser is confronted with small data sets or the likelihood of non-normality in values, but nonetheless has a sufficiently large array of data to at least take advantage of some non-parametric statistical characterizations. Regression relies on the appraiser’s judgment in the modeling process, but lets the data essentially speak for themselves in the adjustment phase. Expert systems rely on appraiser judgment at both levels; but by applying a larger set of data than can be comfortably managed with a sales grid, this method allows for a heightened degree of accuracy, reliability, and replicability in the process. Expert systems draw from Bayesian estimation, and constitute a maximum likelihood estimator of value, which results in the same coefficients as the least squares estimator derived from a hedonic model, but approaching the problem from a different perspective Real estate occupies a unique place on the asset spectrum. The real estate market is notoriously inefficient, and unlike securities markets, which provide severe penalties for taking advantage of certain kinds of inefficiencies (e.g. insider trading), real estate markets actually foster and encourage participants to in this regard. Transactions require substantial intermediation, high degrees of leverage, and lengthy clearing periods. The assets themselves suffer from locational monopoly and high degrees of The current issue and full text archive of this journal is available at www.emeraldinsight.com/1463-578X.htm Expert systems and mass appraisal 529 Received December 2010 Accepted March 2011 Journal of Property Investment & Finance Vol. 29 No. 4/5, 2011 pp. 529-550 q Emerald Group Publishing Limited 1463-578X DOI 10.1108/14635781111150385 both temporal and spatial autocorrelation. (See, for example, Pace et al., 1998, and Des Rosiers et al., 2000) As such, raw asset prices themselves reveal very little about the true value of real estate; yet an understanding of the actual underlying value is critical for a number of reasons, including business decisions (particularly financing), forced acquisition litigation (either through eminent domain or through trespass, such as encroachment or contamination), and taxation (e.g. – property, estate). This has given rise to a rather stylized appraisal process. In the USA, and in most other countries, appraisal methods have developed heuristically over the years[1]. We can trace appraisal methods and standards in the USA back to the Virginia House of Burgesses in the 1600s, at which time they gave instructions on the assessment of property for tax purposes. Professional appraisal organizations arose in the USA in the 1930s, amid a clamor for better organization of financial markets in general. Various professional appraisal organizations came together in the 1980s to codify the Uniform Standards of Professional Appraisal Practice (USPAP), which were then transferred to the newly-formed Appraisal Foundation, which was empowered via the 1989 Financial Institutions Reform, Recovery, and Enforcement Act to promulgate both appraisal standards and qualifications for state-based licensure. Appraisal methods – as distinct from appraisal standards – continued to be developed heuristically. The “body of knowledge” evolved on basically a two-track system, with academic researchers exploring values via statistically robust methods, such as regression, contingent valuation, or time-series modeling, and practitioners relying more heavily on professional organizations for methodological guidance. Of course, in practice, this sort of bi-modality was not so clearly defined, as many practitioners also held academic appointments, and many academics contributed to practitioner texts and coursework. Nonetheless, for good or bad, the appraisal profession did not follow the same path carved out by accountants, who have a more well-developed integration of academia and accounting practitioners. In recent years, two separate evolutions have given the profession some pause. First, arguably, the three largest uses of appraisals in the USA are for property tax purposes, mortgage financing, and eminent domain “takings”. The first of these is more-or-less governed by supplemental standards and methods promulgated by the International Association of Assessing Officers (IAAO). While the IAAO is nominally a part of the USPAP universe, tax assessors usually adhere to a mass appraisal paradigm (provided for by USPAP Standard 6), which, in its best applications, closely resembles hedonic regression. They generally are required to adhere to a certain degree of statistical rigor, and the IAAO actually promulgates minimum confidence levels, such as maximum acceptable coefficients of dispersion. Second, mortgage finance appraisal, however, has no such published thresholds, and that segment of the profession has been roiled with accusations of inaccuracies, inarguably contributing to the overall problems with residential mortgage finance today. A thorough examination of these problems is beyond the scope of this study, but it is sufficient to say that as of this writing, that segment of the appraisal profession is currently casting about for a better way to do things. To follow-up on Shiller and Weiss (1999), mortgage lending has apparently recently erred on the side of minimizing Type I errors (failure to make a deserved loan) but at the expense of increased Type II errors (making loans that should JPIF 29,4/5 530 not have been made). At either extreme of the bell curve (assuming it IS a bell curve), there is presently a lack of statistical power to assess the likelihood of either type of error. Third, eminent domain is also one of the largest purposes for which appraisal is currently conducted. In most circumstances, eminent domain appraisal follows essentially the same methodological paradigm as financing appraisal. In 1993, the USA Supreme Court issued the now-famous Daubert ruling, in which it set down certain criteria for the admission of expert scientific testimony[2]. It was not actually until 1999 that these criteria were extended to technical experts, including appraisers, via the Kumho Tire ruling[3], but nonetheless the criteria as applied to real estate valuation continues to be referred to as the Daubert test. Many states amended their rules of evidence to conform to Daubert, and as of this writing, most states use Daubert for civil litigation[4]. The emphasis of Daubert is hypothesis testing. In its opinion, the Court stated: Ordinarily, a key question to be answered in determining whether a theory or technique is scientific knowledge that will assist the trier of fact will be whether it can be (and has been) tested. Scientific methodology today is based on generating hypotheses and testing them to see if they can be falsified; indeed, this methodology is what distinguishes science from other fields of human inquiry(emphasis added). Academic researchers will immediately recognize that this places a requirement for statistical characterization of the data, analysis, and results, but the Court went further to establish a four-pronged test to provide guidance for trial courts[5]: (1) Do(es) the method(s) center upon a testable hypothesis? (2) Is there a known or potentially knowable error rate associated with the method(s)? (3) Has the method been subject to peer review? (4) Is the method generally accepted in the relevant scientific community? Clearly, valuation methodology used in the courtroom must carry with it some measure of statistical validity in order to meet these criteria. Thus, all three of the leading uses of appraisals in America now have significant reason to explore statistical characterization. One solution to this problem would be the use of hedonic regression modeling for all real estate valuation. OLS regression has been an important component of the econometric tool box since Gauss and Markov developed their eponymous theorem[6]. The appraisal body of knowledge has discussed the applicability of regression analysis for many years, and Smith (1971) provides a good summary of the early thinking within the profession. While a more widespread use of regression may be appealing, and in fact may be feasible in many areas and for many property types, at the very least it is overkill and at worst the requirements for linear regression make it infeasible in many real estate valuation situations. In the former, hedonic modeling requires a substantial level of data gathering. One might envision a situation such as exists in Germany, where local boards keep track of property values on a mass-basis, but even those situations require an appraiser to handle individual property characteristics. Worse, in a mass appraisal Expert systems and mass appraisal 531 scenario, practical application of ordinary least squares (OLS) regression requires that several assumptions should be satisfied: (1) Data are independently and identically distributed (iid) draws from their joint distribution. (2) Strict exogeneity, that is, the conditional means of the error terms is zero, and the errors are uncorrelated with the regressors: E 1jX � � ¼ 0 ð1Þ and E X 01 � � ¼ 0: ð2Þ (3) No multicollinearity, which requires that all of the regressors are linearly independent, and the matrix Q is positive and semi-definite, with moments up to at least the second: Pr rank Xð Þ ¼ p � � ¼ 1 ð3Þ and Qxx ¼ E X 0X=n � � : ð4Þ (4) Spherical errors, where I is an n £ n identity matrix: Var 1jX � � ¼ s 2In: ð5Þ (5) The most common violations of spherical errors are heteroskedasticity and autocorrelation: E 12i jX � � – s 2 ð6Þ and E 1i1jjX � � – 0; for i – j: ð7Þ (6) Normality: 1j ~XN 0; s 2In � � : ð8Þ JPIF 29,4/5 532 Also, in a practical sense, cross-sectional hedonic modeling requires a much larger data set than is typically available or easily procured. Given the typical practical admonition that appraisers use data which are spatially and temporally as close as possible to the subject (implicitly admitting that assumptions (3) and (4) are violated), then raising an adequate data set is often not practical. While a large enough data set (and, implicitly, a large enough budget) allows a researcher to attack these issues through a variety of well-accepted econometric tools (e.g. weighted least squares, logarithmic transformations, etc.), these tools and techniques are simply beyond the scope of the vast majority of appraisal situations. Further, use of these tools and techniques usually requires a very specialized skill-set in econometrics. Thus, the profession is left with the need for methodologies which have solid statistical characterization but which do not require the strict assumptions of OLS. Prior research has explored neural networks as an alternative, but with a focus on either reviewing “canned” software packages or examining the ways neural networks can improve hedonic pricing models. Worzala et al. (1995), McGreal et al. (1998), Liu et al. (2006), and Peterson and Flanagan (2009) provide a good review of these lines of research. Neither line of research suggests a solution to the adjustment-grid-versus- hedonics conundrum. The exploration of neural networks also introduced the term “fuzzy logic” to the appraisal lexicon, although, in fact, traditional appraisal methodology is simply a heuristic application of fuzzy logic. The term is generally credited to Zadeh (1964)[7], who was extending prior work on fuzzy sets: [. . .] a “fuzzy set” [. . .] extends the concept of membership in a set to situations in which there are many, possibly a continuum, of grades of membership. The concept of fuzzy sets and fuzzy logic is necessary in neural networks, because it is the best way – if not only way – to instruct computers to select not just comparables which are exact matches (an impossibility in real estate) but instead to learn from comparables which are close matches. Experienced real estate appraisers will immediately recognize that this is what they have always heuristically endeavored to accomplish in a sales adjustment grid. Thus, the application of fuzzy logic simply provides a means for computers to mine data sets for the closest comparables. Expert systems, on the other hand, provide the potential to provide the appraiser with statistical properties without the strict assumptions of OLS. Expert systems are adaptive to non-parametric data, and are useful for smaller data sets than would normally be required in hedonic models. The principal shortcoming of expert systems is that they do not come in a “one-size-fits-all” package, and require some modeling skills on the part of the appraiser. The remainder of this paper explores the theoretical basis for a mass appraisal, with an eye to describing the equilibrium condition of residential real estate markets so as to understand the nature of the mass appraisal model. The paper then presents two case studies of expert systems applications to demonstrate the modeling and statistical characterization techniques. For simplicity, this paper focuses on residential valuation. However, we have observed non-residential expert systems in place and used successfully in tax assessment situations. Expert systems and mass appraisal 533 2. The residential real estate market – a brief primer on equilibrium and modeling The nature of the residential real estate equilibrium is one of the least understood theoretical underpinnings of modern appraisal practice. While a thorough exploration deserved a separate stream of research, it is useful to mention the highlights to set the stage for the inputs to an expert systems model. Students of economics begin their studies with a purposefully simplistic example of supply and demand, as shown in Figure 1, to illustrate that an increase in demand (from D1 to D2) results in an increase in the quantity supplied (from Q1 to Q2) along with an increase in price (P1 to P2). Students who pay close attention in class will note that this is a partial equilibrium model, in which the demand, supply, quantity, and price of all complimentary and substitute goods are held constant. Demand and supply curves are assumed to be downward and upward sloping, respectively, universally differentiable (that is, no points of inflection) and concave[8]. Of course, in practice, nothing could be further from the truth. Housing supply is sticky in the short- and intermediate term, substitute goods abound, locational monopolies predominate, and consumer expectations (and resultant demand functions) are frequently incompatible with each other and with existing supply. Thus, at the very least, the actual model in practice more closely resembles Figure 2. In practice, and particularly in the short-run, the residential real estate market is more accurately described with a Nash equilibrium, in which a finite number of “demanders” are competing against each other to optimize their utility in a market of dissimilar but finite supply. Demanders take into account each others’ equilibrium strategy, and no one player can gain an advantage by unilaterally changing his or her strategy. The Nash equilibrium takes into account the notion that different demanders enter the market place with different strategies, but are faced with the same vector of heterogeneous supply. The idea of a mixed-strategy game was not unique to John Nash’s, 1951 paper, and in fact his work built on the earlier work of Von Neumann and Morganstern (1947). However, their earlier work assumed the special case of a zero-sum game. Nash generalized their work to show that a collective optimization could result in a Pareto-optimality that was not zero-sum[9]. Figure 1. Simple model of supply/demand JPIF 29,4/5 534 What are the implications for real estate valuation? Simply put, direct comparison of property transaction prices in order to develop a value must take into account the fact that each transaction price was arrived at via a slightly different “demander” strategy. Heuristic appraisal methods are able to take this into account – see the foregoing discussion of fuzzy logic – albeit in a fashion not well characterized from a statistical perspective. Indeed, explorations into the Daubert-esque characteristics of individual appraisals have revealed significant problems (see Kilpatrick, 2010). This is not to cast dispersions on OLS-type models, which do not incorporate such heuristic accommodation of strategy variation. Indeed, OLS-type models accommodate such idiosyncrasies as long as those idiosyncrasies are reasonably well behaved. Statistical theory tells us that with large sample sizes, the OLS assumptions at least asymptotically are true. Nonetheless, individualized appraisals, such as those conducted for mortgage lending and most eminent domain work, do not have the luxury of large data sets. To substitute for this, appraisers draw on so-called “appraisal judgment” in arriving at a reconciled “opinion of value.” The former of these terms actually draws from a field of statistical inference called Bayesian estimation, and the latter constitutes a maximum likelihood estimator. As will be shown in the next section, an expert system draws from these fields as well, albeit in a statistically characterizable fashion. Hence, a brief exploration of the two topics is in order. The Rev. Thomas Bayes (1702-1761) was a Presbyterian theologian and mathematician, who published on both topics. He is also well known for his defense of Isaac Newton’s development of calculus. In the early part of the eighteenth century, statisticians were fascinated with what were then called inverse probabilities, which we now refer to as conditional probabilities. In essence, the question asked is “What is the probability of something happening if we already know some predecessor information about that event?” The classic example is an urn filled with an equal number of black and white balls (say, ten of each). An introductory statistics student should be able to guess that the probability of drawing either a black or a white ball is 50-50, the same as a coin flip – a very simple binomial distribution. Now, what if five white balls have been drawn out of the urn in a row, but each time they are replaced in the urn and the urn shaken so that subsequent draws are totally random? On each Figure 2. Supply/demand with perfectly inelastic supply Expert systems and mass appraisal 535 subsequent draw, the same binomial distribution continues to apply. Counter-intuitively, the probability of a white versus black ball remains 50-50, despite the seemingly endless but totally random draw of white balls[10]. Conversely, what if the balls are not replaced, so after five draws, the urn is known to only possess five white balls, but ten black ones. What now is the probability of a white draw or a black one? Mathematically, this can be expressed as equation (9): Pr HjE � � ¼ PrðEjHÞ PrðHÞ PrðEÞ ; ð9Þ where: H ¼ The hypothesis being tested. E ¼ Prior knowledge observed by the researcher. Pr(H) ¼ The prior probability of H, before the researcher gained the prior knowledge. Pr(EjH) ¼ The conditional probability of seeing the evidence E if the hypothesis H is actually true. Pr(E) ¼ The marginal probability of observing E with or without H being true (or across all possible outcomes of Hi) – P PrðEjHiÞ PrðHiÞ: Pr(HjE) ¼ The posterior probability of H given the observation of E. Indeed, the student of Bayesian statistics will quickly see that the sales adjustment grid process is simply a very practical manifestation of Rev. Bayes’ theorem. The hypothesis being tested is the value of the property, and the prior knowledge is not only the comparable data, but also the appraiser’s judgment in making adjustments to such data. As a launch-point for the next discussion, Pr(EjH) is would be recognized by a statistician as a maximum likelihood estimator (MLE) and is the mathematical expression for the judgmental process by which the appraiser draws comparables for the sales comparison approach. The MLE asks the simple question, “Given what we know about these data, what probability process best fits”? Ordinary estimation procedures start with some predecessor assumption about the distributional characteristics of the data (i.e. normality). In OLS, the asymptotic properties allow us to make this assumption[11]. MLE, conversely, begins with the data instead, and then looks for a probability distribution that fits. In most cases, the MLE has desirable properties, and these properties fit well with appraisal assumptions: . Consistency. The ML estimator converges asymptotically to the value being estimated. From an appraisal perspective, this means that there is a benefit to experience and professionally-developed judgment. . Asymptotic normality. As sample size increases, the MLE distribution tends toward a normal distribution. To an appraiser, this suggests that the reliability of the value opinion improves with more comparable sales. . Efficiency. There is no asymptotically unbiased estimator that has lower asymptotic mean square error. To an appraiser, this suggests that the MLE minimizes the risk of error. JPIF 29,4/5 536 From a more technical perspective, suppose the appraiser draws a sample of data, x1, x2, x3, . . . xn, where the n observations are all independent and identically distributed. The distribution, f0( *) is unknown, but is believed to be a part of a family of distributions, so that f 0 ¼ fð* jFÞ where F is the true, but unknown value. Thus, it would be desirable to find some estimator, Fx, which would be as close in value to F as possible. The actual likelihood function is as shown in equation (11), and in practice, the log-likelihood (equation (12)) is used for simplicity (and also has certain properties which make it desirable in real estate analyses[12]): L Fjx1; x2:::xn � � ¼ f x1; x2:::xnjF � � ¼ Yn i¼1 f xijF � � ð10Þ ln L Fjx1; x2:::xn � � ¼ f x1; x2:::xnjF � � ¼ Xn i¼1 ln f xijF � � : ð11Þ Note that the iid assumption can be relaxed for the underlying data so long as it is possible to write a joint density function and that the parameter, F, has a finite dimension that does not depend on the sample size. This is a handy simplification in real estate, since the data are most likely not independent, but temporally and spatially correlated. In effect, the ML estimator is also usually the most probable Bayesian estimator. The literature tying MLE to appraisal practice is scant but growing. Assane (2007) uses MLE methods to reconcile spatial and hedonic models, and Ross et al. (forthcoming) extend this work using Monte Carlo simulation to determine the welfare implications one can draw from distance variables used in hedonic regression models. However, both of those papers, and other work in the area, endeavor to make improvements on the hedonic model, and demonstrate the usefulness of more data. The gap in the knowledge base is to apply these statistical tools in a fashion that provides statistical characterization in a limited dataset/non-parametric world. Thus, we are left with three important observations which tie heuristic appraisal methods to underlying economic and econometric theory: (1) The real estate transactional market constitutes a Nash equilibrium, in which all participants take into account the strategies of other participants in the goal of optimizing their utility. Thus, spatial and temporal autocorrelation are part of the process, rather than aberrations to the model. (2) Real estate valuation takes into account what is already known about the market, and indeed temporal and spatial autocorrelation makes it highly likely that the value of the nth property is tied inexorably to the value of the (n þ 1)th and (n 2 1)th properties. Considerable literature, not discussed here, discusses the appraisal smoothing problem (source?). Nonetheless, without the Bayesian priors that give rise to appraisal smoothing, heuristic methods would not be possible. (3) The appraiser, faced with a set of data and a set of prior observations about the underlying market, uses fuzzy logic to formulates a maximum likelihood estimator to determine the true value of the property. S/he is able to do this with a limited data set based on the Bayesian priors already known about the probable behavior of the market. The challenge now is to formulate a way to explain all of this in a statistically valid fashion. The next discusses the theory behind expert systems. Expert systems and mass appraisal 537 3. Expert systems – theoretical and methodological underpinnings Some of what we think of today as “expert systems” grew out of the data mining studies in the 1990s. Increasing amounts of data were being stored in relatively easily accessible databases. Researchers were interested in a variety of data analyses, including classification, discovery of associations, pattern identification, temporal modeling, deviation detection, dependency modeling, clustering, and characteristic rule discovery. Individual data analytical techniques, such as OLS, proved problematic working across disparate data sets. Out of that, fuzzy logic and hybrid systems developed which drew on the various strengths of different techniques. Goonatilake and Khebbal (1995) provide a contemporaneous outline of this emerging data analysis trend. McCluskey and Anand (1999) were among the first to thoroughly outline the application of such expert systems as they apply to mass appraisal. However, as early as 1989, Scott and Gronow outlined models that would simulate appraisal expertise (Scott and Gronow, 1989). Follow-up research in this vein was done by Nawawi and Gronow (1991) and Nawawi et al. (1997). Early research pointed to the problem of transparency in the expert system model. McCluskey and Anand (1999) basically outline two analytical strategies, both of which rely on separating the data (in this case, comparable transactions) into two sets: a comparable set (which will be directly used for valuation) and a separate data set which will be used for the system to “learn”. In their loosely coupled hybrid system (see Figure 3), the learning process is an artificial neural network which estimates attribute values. The weights on various values is computed using an equation they developed. Alternatively, they present a tightly coupled hybrid system (see Figure 4) in which a genetic algorithm starts out with a set of likely solutions. The various solutions are iteratively applied in a survival-of-the-fittest mode. The solution that best fits the data becomes the MLE which is then applied to the comparable data. McCluskey and Anand (1999) also present, and dismiss, what they call the “domain expert” model. In this model, relative weights and factors are determined by an expert, who has prior knowledge of the valuation equation. They state that the principal shortcoming of this model is that it requires an “expert” and thus is not self-learning. Given the foregoing discussions about Bayesian analysis, however, this domain expert, coupled with the two-stage paradigm of their tightly coupled hybrid system (which produces an MLE), provide a powerful underlying basis for a useable expert system. Figure 3. JPIF 29,4/5 538 4. Practical expert system applications – two case studies The challenge at this juncture is to develop expert systems that are actually adaptable in real life for actual appraisal problems. Much like heuristic or OLS models, expert systems will require appraiser input both in the modeling and the data selection. Unlike neural networks, the expert system will not “teach itself” but rather the appraiser will utilize the data, in a fuzzy-logic fashion, to develop adjustment factors. In practice, the expert system closely resembles heuristic sales adjustment grids, but with the added advantage of the ability for statistical characterizations. Case study 1 – Plaquemines Parish, Louisiana Plaquemines Parish has been in the news lately – it is “ground zero” for the April 2010 Deepwater Horizon explosion and subsequent oil spill. Many of the major news reports came out of the towns of Belle Chase and Venice. However, Plaquemines Parish was also “ground zero” or nearly so for Hurricane Katrina in 2005. Kilpatrick and Dermisi (2007) discuss much of the real estate research following that event, and one area of research was the impacts of flooding on property values. Considerable attention was paid to two major proposed class actions versus the Army Corps of Engineers, regarding breeches of the Lake Ponchartrain levee and the Mississippi River Gulf Outlet (MRGO) levee. In the former case, the Federal District Court ruled that the Army enjoyed protection from lawsuit by the Flood Control Act of 1936. In the latter, however, the Court ruled that the Act did not apply, since the MRGO was not dredged for flood control but rather for navigation. In Plaquemines Parish, however, the levees were not built and maintained by the Army, but rather by the Parish itself, which did not enjoy sovereign immunity. Residences were damaged by flooding from allegedly poorly designed, built, and/or maintained levees, and a class action suit was filed (Bermaster v. Plaquemines Parish). The complexity of the case was compounded by four somewhat related factors: (1) Properties in the parish are very thinly traded, and appraisals are “noisy” in statistical terms. (2) Tax assessment data in Louisiana are problematic compared to other jurisdictions, and in the case of Plaquemines Parish, records were destroyed when the Parish Courthouse burned down (ironically, not Katrina related). Figure 4. Expert systems and mass appraisal 539 (3) The case was significantly delayed when a judge died and a new judge had to be appointed. (4) Many properties in the Parish were affected by the flooding and were totally destroyed, and the occupants moved away. Adjudication of a class action required some methodology for establishing base-line values of the homes. Hedonic modeling would require data quality that was simply not available, and individualized appraisals of all of the affected properties would have been prohibitively expensive. For that reason, some sort of expert system was needed. The first step in the expert system was to identify a study area (Figure 5), which encompassed all of the proposed class area. Within this study area, comparables were collected (Figure 6) from transactions over a period of several years prior to the flooding. It was immediately obvious that the comparables were extraordinarily heterogeneous, and data verification was an important component of the study. The next step in the process was to apply appraisal expertise to the data in order to find common themes to the valuation of properties in this market place (see equation (12) below). This is analogous to the model specification process in a hedonic regression. Next, a hold-out sample was selected (Figure 7) which consisted of a cohesive neighborhood of properties which spanned most property types in the region, and for which good baseline values were either known or knowable through good tax assessment records, recent arms-length transactions, recent appraisals, or interviews with local appraisers or brokers: Vi;j ¼ BasePricej þ Qi;j 2 Qbase;j � � bj;1 þ Bri;j 2 Brbase;j � � bj;2 þ Ai;j 2 Abase;j � � bj;3 þ SFi;j 2 SFbase;j � � bj;4 þ Disti;j 2 Distbase;j � � bj;5 þ 1 ð12Þ where: Vi,j ¼ Estimated Value of property i in property type j. BasePricej ¼ Base price of property type j. Qi,j ¼ Quality Level of property i in property type j. Qbase,j ¼ Base Quality level of property type j. bj,1 ¼ Quality Level coefficient for property type j (or price adjustment due to difference between Quality level of property i and base Quality level in property type j). Bri,j ¼ Presence of Brick in property i of property type j. Brbase,j ¼ Base level Presence of Brick in property type j. bj,2 ¼ Presence of Brick coefficient for property type j (or adjustment due to difference between Presence of Brick in property i and base level of Presence of Brick in property type j). Ai,j ¼ Acreage of property i in property type j. Abase,j ¼ Base Acreage of property type j. JPIF 29,4/5 540 bj,3 ¼ Acreage coefficient for property type j (or adjustment due to difference between Acreage of property i and Base Acreage in property type j). SFi,j ¼ Square Footage of property i of property type j. SFbase,j ¼ Base Square Footage of property type j. Figure 5. Plaquemines Parish study area Expert systems and mass appraisal 541 bj,4 ¼ Square Footage coefficient for property type j (or adjustment due to difference between Square Footage of property i and base Square Footage in property type j). Disti,j ¼ Distance from Ferry of property i of property type j. Distbase,j ¼ Base Distance from Ferry of property type j. Figure 6. Plaquemines Parish comparables for expert system JPIF 29,4/5 542 bj,5 ¼ Distance of Ferry coefficient for property type j (or adjustment due to difference between Distance from Ferry of property i and base Distance from Ferry in property type j). 1 ¼ Error term. An example of this methodology is shown in Table I. Accuracy of the model is best described by the coefficient of dispersion (COD), a non-parametric tool recommended Figure 7. Plaquemines Parish sample area for test 1 Expert systems and mass appraisal 543 by the IAAO and widely used by tax assessors as an accuracy benchmark. COD for n properties is computed using equation (13), and for the Plaquemines test sample, the COD is 9.06 percent. This is well within the standards for this property type recommended by the IAAO: COD ¼ P abs Price 2 Valueð Þ=n median value : ð13Þ Table II also shows the results on a property-by-property basis. Case study 2 – Lomax, Illinois Located in western Illinois, just east of the banks of the Mississippi River near the Iowa/Missouri border, this small township had a population of 477 as of the 2000 census. Homes are typically several decades old, and rarely change hands. The few transactions that are recorded are rarely arms-length, since homes pass among family members or friends frequently. People who live there were often born there or married into families there. People leave typically by marrying and moving away. It was recently discovered that an oil pipeline, owned by BP Pipelines North America, has been leaking into the drinking water and soil of the homes in Lomax. The township and many of the property owners filed suit, and one of the analytical challenges was to craft a supportable baseline value for these homes. Again, a hedonic model would have been problematic. Individual appraisals of the homes might have been feasible, but would still have lacked statistical robustness and characterizability. Table III details the 20 homes that were valued, with home types including mobile homes, ranch homes, and typical Midwestern farmhouses. Comparable sales were drawn from the same county (Henderson) as well as adjacent Mercer and Hancock counties. Over 250 comparables were analyzed, broken down into five categories: (1) Double-wide manufactured homes. (2) Single-wide mobile homes, sold with the lot as a unit. (3) Old-style homes category 1. (4) Old-style homes category 2. (5) Ranch homes. Adjustments were made based on factors specific to each group. For example, under the ranch category, adjustments had to be made for lot size, improvements size, full baths (beyond 1), half baths (beyond -0-), garage size as it differed from 575, basement Property type Base value ($) Qual/cond Brick Acres SF Distance Coefficients ($) ! 20,000 5,000 3,000 25 2500 Property type “J” base values 48,000 2 0 0.5 1,200 17 Property type “I” characteristics 2 1 0.3 1,350 21 Notes: Vali ¼ $48,000+((2 2 2)*$20,000)+((1 2 0)*$5,000)+((0.3 2 0.5)*$3,000)+((1,350 2 1,200)*25)+ ((21 2 7)* 2 500) Vali ¼ $48,000+$0+$5,000 2 $600+$3,750 2 $2,000 Vali ¼ $54,150 Table I. Case study no. 1 valuation example JPIF 29,4/5 544 ID S a le d a te S a le p ri ce ($ ) T im e a d ju st ed p ri ce ($ ) Im p r T y p e Q /C B ri ck A cr es S F D is t V a lu e ($ ) E rr o r (% ) 9 3 /4 /2 0 0 0 4 8 ,5 5 1 6 3 ,1 6 4 1 2 0 .5 0 .2 7 1 ,1 5 6 0 .1 3 7 7 ,1 4 7 2 2 2 1 5 5 /1 0 /2 0 0 1 6 0 ,0 0 0 7 6 ,5 2 8 1 2 0 2 .8 5 1 ,5 0 0 6 .4 5 6 7 ,8 4 2 1 1 1 8 .1 1 0 /1 5 /2 0 0 2 2 1 ,0 0 0 2 6 ,0 0 5 1 1 0 3 .8 8 8 3 4 1 8 .1 3 2 8 ,4 1 5 2 9 1 8 .2 8 /1 2 /2 0 0 3 2 1 ,0 0 0 2 5 ,2 4 7 1 1 0 3 .8 8 8 3 4 1 8 .1 3 2 8 ,4 1 5 2 1 3 1 9 8 /2 0 /2 0 0 1 4 0 ,0 0 0 5 1 ,0 1 9 1 2 0 1 .6 2 1 ,0 9 2 1 9 .5 8 4 7 ,3 8 4 7 2 0 8 /3 0 /2 0 0 1 5 0 ,0 0 0 6 3 ,7 7 3 1 2 0 0 .5 8 1 ,2 7 9 2 1 .0 0 4 8 ,2 2 0 2 4 2 3 1 2 /2 0 /2 0 0 2 6 7 ,0 0 0 8 2 ,9 6 7 1 3 0 0 .7 8 1 ,3 8 3 2 2 .4 3 7 0 ,6 8 9 1 5 2 6 5 /1 9 /2 0 0 4 4 5 ,0 0 0 5 1 ,0 3 9 1 2 0 0 .3 4 1 ,4 7 5 2 2 .5 7 5 1 ,6 0 0 2 1 3 2 .1 1 1 /3 /2 0 0 0 3 5 ,0 0 0 4 5 ,5 3 4 1 2 0 0 .2 8 1 ,3 6 8 2 1 .5 1 4 9 ,2 7 0 2 8 2 1 9 /1 1 /2 0 0 2 6 0 ,0 0 0 7 4 ,2 9 9 1 .5 2 0 0 .6 7 2 ,2 9 1 2 1 .8 0 8 0 ,8 3 2 2 9 2 2 1 /1 0 /2 0 0 1 4 0 ,0 0 0 5 1 ,0 1 9 1 .5 2 0 0 .5 0 1 ,0 7 0 2 1 .8 4 4 5 ,9 0 3 1 0 2 4 1 0 /5 /2 0 0 0 5 6 ,6 6 0 7 3 ,7 1 3 1 .5 3 0 0 .2 7 1 ,4 8 5 2 2 .5 5 7 3 ,5 4 1 0 2 5 8 /1 9 /2 0 0 5 1 2 0 ,0 0 0 1 2 8 ,4 0 0 1 .5 3 0 1 0 .5 8 1 ,5 1 2 2 2 .3 7 1 1 5 ,9 5 2 1 0 4 3 1 1 /2 5 /2 0 0 2 1 4 4 ,0 0 0 1 7 8 ,3 1 8 1 .5 4 0 .5 0 .2 6 3 ,1 1 7 3 .9 6 1 8 2 ,1 4 8 2 2 4 4 /1 3 /2 0 0 5 1 3 5 ,0 0 0 1 4 4 ,4 5 0 2 3 1 1 .0 5 1 ,7 0 4 0 .3 9 1 5 3 ,1 6 9 2 6 1 0 6 /1 2 /2 0 0 3 2 0 5 ,0 0 0 2 4 6 ,4 6 2 2 4 1 3 .2 0 1 ,8 4 4 2 .2 8 2 1 2 ,1 2 4 1 4 1 1 .2 4 /5 /2 0 0 4 2 0 0 ,0 0 0 2 2 6 ,8 4 0 2 4 1 8 .8 7 1 ,2 5 4 3 .4 5 2 1 3 ,6 2 3 6 1 2 1 1 /1 /2 0 0 2 1 1 0 ,0 0 0 1 3 6 ,2 1 5 2 3 1 0 .6 8 1 ,9 0 4 3 .3 7 1 5 5 ,3 6 7 2 1 4 2 7 8 /1 4 /2 0 0 0 1 2 6 ,0 0 0 1 6 3 ,9 2 3 2 4 1 0 .5 9 2 ,1 9 9 7 .1 7 2 0 2 ,0 4 3 2 2 3 2 9 1 /3 1 /2 0 0 3 1 3 8 ,0 0 0 1 6 5 ,9 1 1 2 3 1 0 .2 1 2 ,6 3 7 1 .7 5 1 9 2 ,8 8 7 2 1 6 3 0 2 /1 1 /2 0 0 5 9 9 ,5 0 0 1 0 6 ,4 6 5 2 2 1 0 .2 0 1 ,5 8 5 1 .7 8 1 0 5 ,2 0 7 1 3 1 .1 5 /2 9 /2 0 0 2 8 0 ,0 0 0 9 9 ,0 6 6 2 2 1 0 .2 9 1 ,4 8 5 4 .0 1 9 1 ,2 0 5 8 3 9 .1 2 /2 2 /2 0 0 2 1 6 0 ,0 0 0 1 9 8 ,1 3 1 2 4 1 0 .3 4 2 ,1 6 8 3 .7 5 2 1 6 ,1 2 0 2 9 3 9 .2 7 /2 3 /2 0 0 4 2 0 0 ,0 0 0 2 2 6 ,8 4 0 2 4 1 0 .3 4 2 ,1 6 8 3 .7 5 2 1 6 ,1 2 0 5 4 0 4 /3 0 /2 0 0 4 1 8 6 ,0 0 0 2 1 0 ,9 6 1 2 3 1 0 .2 9 2 ,5 6 3 3 .8 8 1 9 0 ,3 3 4 1 0 4 1 1 0 /3 1 /2 0 0 2 2 0 5 ,0 0 0 2 5 3 ,8 5 6 2 4 1 0 .3 7 2 ,1 4 9 3 .8 1 2 1 7 ,6 7 2 1 4 4 2 2 /5 /2 0 0 1 1 5 0 ,0 0 0 1 9 1 ,3 2 0 2 3 1 0 .2 4 1 ,7 5 0 3 .8 5 1 4 9 ,4 9 4 2 2 4 4 1 0 /1 7 /2 0 0 2 1 5 3 ,0 0 0 1 8 9 ,4 6 3 2 3 1 0 .7 0 2 ,3 0 0 3 .9 9 1 7 4 ,0 1 9 8 4 6 3 /3 1 /2 0 0 3 2 3 2 ,0 0 0 2 7 8 ,9 2 2 2 4 1 0 .2 8 2 ,8 0 3 3 .9 2 2 3 7 ,2 3 0 1 5 1 1 1 /7 /2 0 0 3 1 8 2 ,0 0 0 2 1 8 ,8 1 0 3 3 0 .5 0 .3 4 2 ,7 3 5 3 .5 0 2 0 7 ,7 8 3 5 3 .3 1 0 /1 8 /2 0 0 1 3 1 5 ,0 0 0 4 0 1 ,7 7 3 3 5 1 3 .9 7 3 ,4 8 4 0 .4 7 4 0 4 ,4 3 4 2 1 1 3 4 /1 6 /2 0 0 1 3 4 5 ,0 0 0 4 4 0 ,0 3 7 3 4 0 .5 1 9 .9 7 3 ,3 3 6 4 .3 7 4 1 5 ,2 4 2 6 2 8 5 /1 5 /2 0 0 1 1 3 0 ,0 0 0 1 6 5 ,8 1 1 3 3 1 0 .3 4 2 ,8 1 8 7 .1 1 2 0 3 ,0 7 5 2 2 2 3 3 5 /1 /2 0 0 3 2 3 0 ,0 0 0 2 7 6 ,5 1 8 3 4 1 0 .3 4 2 ,7 0 0 3 .8 1 2 6 2 ,6 0 6 5 3 5 8 /3 0 /2 0 0 1 1 9 0 ,0 0 0 2 4 2 ,3 3 9 3 3 0 0 .2 4 3 ,8 2 3 3 .7 9 2 7 4 ,5 2 3 2 1 3 3 6 .1 8 /2 1 /2 0 0 0 2 2 8 ,0 0 0 2 9 6 ,6 2 3 3 3 1 0 .2 4 3 ,6 6 4 3 .7 9 2 6 2 ,4 8 4 1 2 3 6 .2 3 /2 7 /2 0 0 1 2 3 5 ,0 0 0 2 9 9 ,7 3 5 3 3 1 0 .2 4 3 ,6 6 4 3 .7 9 2 6 2 ,4 8 4 1 2 3 6 .3 5 /7 /2 0 0 4 2 4 2 ,5 0 0 2 7 5 ,0 4 4 3 3 1 0 .2 4 3 ,6 6 4 3 .7 9 2 6 2 ,4 8 4 5 3 7 1 0 /2 7 /2 0 0 0 1 2 9 ,0 0 0 1 6 7 ,8 2 6 3 3 1 0 .2 9 2 ,5 0 0 3 .7 5 1 9 0 ,4 5 8 2 1 3 3 8 9 /3 /2 0 0 4 1 9 5 ,0 0 0 2 2 1 ,1 6 9 3 3 0 0 .2 4 2 ,0 7 2 3 .7 8 1 6 9 ,4 6 6 2 3 4 5 1 2 /1 /2 0 0 4 1 7 9 ,0 0 0 2 0 3 ,0 2 2 3 3 1 0 .3 8 2 ,3 4 1 3 .9 9 1 8 3 ,3 6 6 1 0 Table II. Plaquemines test results Expert systems and mass appraisal 545 (full, partial, or finished), fireplace (beyond a base of -0-), deck, porch, enclosed porch, patio, shed, proximity to a railroad, and condition. Note that different adjustments were made for different categories of homes. Table IV shows the CODs for Lomax property types (see also Figure 8). As an example, the median value for ranch homes determined by the comparables was $70,364. The average dispersion was $5,399, resulting in a coefficient of dispersion of 7.67 percent. Across the five categories, the range was 7.67 percent to 35.21 percent. For three of the categories, the properties were within recommendations of IAAO, the fourth category was nearly so, and only one category (mobile homes) was well outside of the range of standards. However, at least these measures of dispersion were determinable, and so in a Daubert setting, the evidence would be acceptable. 5. Summary, conclusions, and recommendations for future research One of the current challenges in real estate valuation is to measure the statistical properties of the appraisal estimates. Colwell et al. (2009) demonstrate that when the data are available, appraisers should always opt for a statistically characterizable GA-ID Lot SF Yr blt Bldg SF Rms BR Ba .5 Ba Gar SF Descr 1 1,936 1820 1,936 7 5 1 0 0 Cape Cd 2 19,602 1998 1,836 7 3 1 0 0 Doubwide 3 43,734 1970 1,064 5 3 1 0 576 Ranch 4 25,570 1973 1,266 5 3 1 0 576 Ranch 5 7,656 1900 929 4 2 1 0 368 1-stry 6 15,312 1920 1,163 6 2 1 0 490 1-stry 7 8,712 1920 1,542 6 2 1 0 637 1.5 stry 8 10,890 1920 1,428 5 3 1 0 576 1-stry 9 55,321 1998 1,232 5 3 2 0 0 m-home 10 7,950 1935 1,186 6 3 1 0 1,080 1-stry 11 8,557 1900 1,554 7 4 1 0 0 1.5 stry 12 33,055 1900 1,114 4 2 1 0 0 1-stry 13 22,704 1920 1,055 5 2 1 0 450 1-stry 14 18,480 1986 1,248 6 3 1 1 576 Ranch 15 16,368 1981 1,068 5 3 1 0 732 Ranch 16 8,712 1900 792 4 1 1 0 720 1-stry 17 12,000 1950 1,184 4 2 1 0 624 1-stry 18 12,000 1954 1,144 5 3 1 1 704 Ranch 19 6,000 1996 1,224 5 3 2 0 0 M-home 20 9,000 1910 742 4 2 1 0 0 1-stry Table III. Homes in Lomax, Illinois COD Property type (%) Double-wide 8.05 Mobile home 35.21 Old style 1 21.81 Old style 2 13.40 Ranch 7.67 Table IV. Lomax coefficients of dispersion JPIF 29,4/5 546 methodology, and in fact, in the courtroom, it is generally required that the appraiser report on the measurable error rate of the estimator. While many advanced methods such as hedonic modeling, meta analysis, or survey research provide for such statistical measurements, these methods require significantly large data sets and computational undertakings. The goal of this study is to present some of the theoretical underpinnings of expert systems, to help define its use, to give some examples of such use, and to briefly discuss ways to statistically characterize the output. The concept of statistical characterization is key – both for acceptability in litigation, but also to aid Figure 8. Lomax, Illinois Expert systems and mass appraisal 547 mortgage-financing appraisers in understanding the error rates of sales adjustment-grid appraisals. Expert systems provide potential for a solution that lies between the purely heuristic sales adjustment grid and the more computationally intensive hedonic model. The benefits of statistical characterization are important, and with risk of pontification, this paper hopes to drive that forward so as to minimize, as Shiller and Weiss put it, the Type II errors in residential mortgage lending. Additional work can quickly be done in two areas: (1) building on Kilpatrick (2010) to find ways to better characterize individual appraisals; and (2) extend this research to add more non-parametric measurement to the expert systems calculations. Notes 1. In some countries, real estate appraisal takes on different characteristics, and is often more highly regulated. For example, Germany has a strictly codified process dependent on local councils which periodically publish land value, to which the depreciated value of the improvements can be added. In most parts of Mexico, a sales comparison is typically not useful and again a variant of a cost approach is preferred. 2. Daubert v. Merrell Dow Pharmaceuticals, 509 USA 579 (1993), now formally captured in Rule 702, Federal Rules of Evidence. 3. Kumho Tire Co. v. Charmichael, 526 USA 137 (1999). 4. Notably, some states which have not adopted Daubert nonetheless have rules of evidence which emulate Daubert. See Kaufman (2006). 5. Of the remaining 20 states, many adhere to what are known as the Frye standards, from Frye v. United States, 293 F. 1013 (DC Cir. 1923). It establishes a bar of general acceptance, but is silent as to the need for a known or knowable error rate. 6. A complete restatement of the theorem is beyond the scope of this paper, and while most econometric students should be familiar with it, a simple summary may be useful: In a linear regression model, if the expected value of the errors is zero, the errors are uncorrelated, and have homoskedastic variances, the best linear unbiased estimator is the ordinary least squares estimator. Note that the errors do not have to be normal or even identically distributed. (As a bit of an aside, Carl Friedrich Gauss (1777-1855) and Andrey Markov (1856-1922) never actually worked together. In the 1800s, the theorem was credited to Gauss alone. Markov’s contributions – as a Russian – were allegedly overlooked in the west, according to a 1934 article in the Journal of the Royal Statistical Society. Most writers attribute the first use of the term “Gauss Markov Theorem” to the book by Scheffe (1959), Analysis of Variance.) 7. Zadeh also published about fuzzy sets in 1965 and 1968 (Zadeh, 1965, 1968). 8. This more-or-less standard graphical representation is generally attributed to Alfred Marshall. Simple presentations often use straight lines to represent demand and supply, but concavity is necessary for there to be a marginal rate of substitution. Fisher suggests that this second-order economic behavior may be biologically imprinted. 9. Note that a Pareto optimality does not imply a singular, unique point of maximum utility, but encompasses the possibility of multiple solutions. In a Nash equilibrium, no one player can change the mix with unilateral decisions, but collectively, multiple solutions could emerge. JPIF 29,4/5 548 10. Casino owners watch with glee as roulette players see a long string of “reds” show up, and thus bet heavily on black, thus further enhancing the profitability to the casino. The opening scene of Tom Stoppard’s classic play, Rosencranz and Gildenstern Are Dead uses this phenomenon to set the stage for the play’s commentary on the fickleness of fate. 11. This is the second time asymptotic normality has been referred to in this paper, and perhaps it useful to define it a bit better. As the number of observations goes to infinity, the distributional characteristics emulate a normal one. This can be mathematically expressed per equation A-1: fðXÞ , N ðas n ) 1Þ: ðA-1Þ 12. Since the log is a monotonic transformation, a log-likelihood solution is equivalent to a likelihood solution. In addition, real estate prices and values are famously nonlinear and have a truncated lower bound at zero. References Assane, D.D. (2007), “Estimating the effect of air quality: spatial versus traditional hedonic price models”, Southern Economic Journal, Vol. 73, April, pp. 1088-111. Colwell, P., Trefzger, J.W. and Heller, J.A. (2009), “Expert testimony: regression analysis and other systematic methodologies”, The Appraisal Journal, Vol. 77, Summer, pp. 253-62. Des Rosiers, F., Thériault, M. and Villeneuve, P. (2000), “Sorting out access and neighbourhood factors in hedonic price modeling”, Journal of Property Valuation & Investment, Vol. 18 No. 3, pp. 291-315. Goonatilake, S. and Khebbal, S. (1995), “Intelligent hybrid systems: issues, classifications, and future directions”, in Goonatilake, S. and Khebbal, S. (Eds), Intelligent Hybrid Systems, John Wiley & Sons, Chichester. Kaufman, M.S. (2006), “The status of Daubert in state courts”, Atlantic Legal Foundation, March 31. Kilpatrick, J.A. (2010), “What is the error rate of an appraisal?”, paper presented at the American Real Estate Society Annual Meeting. Kilpatrick, J.A. and Dermisi, S. (2007), “The aftermath of Katrina: recommendations for real estate research”, Journal of Real Estate Literature, Vol. 15 No. 2, pp. 213-27. Liu, J.-G., Zhang, X.-L. and Wu, W.-P. (2006), “Application of fuzzy neural network for real estate prediction”, Advances in Neural Networks, Springer, Berlin and Heidelberg, pp. 1187-91. McCluskey, W. and Anand, S. (1999), “The application of intelligent hybrid techniques for the mass appraisal of residential properties”, Journal of Property Investment & Finance, Vol. 17 No. 3, pp. 218-38. McGreal, S., Adair, A., McBurney, D. and Patterson, D. (1998), “Neural networks: the prediction of real estate values”, Journal of Property Valuation & Investment, Vol. 16 No. 1, pp. 57-70. Nash, J. (1951), “Non-cooperative games”, The Annals of Mathematics, Vol. 54 No. 2, pp. 286-95. Nawawi, A.H. and Gronow, S. (1991), “Expert systems in rating valuation”, Journal of the Society of Surveying Technicians, Vol. 18 No. 8, pp. 66-72. Nawawi, A.H., Jenkins, D. and Gronow, S. (1997), “Expert system development for the mass appraisal of commercial property”, in McCluskey, W.J. and Adair, A.S. (Eds), Computer Assisted Mass Appraisal: An International Review, Ashgate, London. Pace, R.K., Barry, R., Clapp, J.M. and Rodriguez, M. (1998), “Spatiotemporal autoregressive models of neighborhood effects”, Journal of Real Estate Finance and Economics, Vol. 17 No. 1, pp. 15-33. Expert systems and mass appraisal 549 Peterson, S. and Flanagan, A.B. (2009), “Neural network hedonic pricing models in mass real estate appraisal”, Journal of Real Estate Research, Vol. 31 No. 2, pp. 147-64. Scheffe, H. (1959), Analysis of Variance, John Wiley, New York, NY. Scott, I. and Gronow, S. (1989), “Valuation expertise: its nature and application”, Journal of Valuation, Vol. 8 No. 4, pp. 362-75. Shiller, R.J. and Weiss, A.N. (1999), “Evaluating real estate valuation systems”, Journal of Real Estate Finance and Economics, Vol. 18 No. 2, pp. 147-61. Smith, T.R. (1971), “Multiple regression and the appraisal of single family residential properties”, The Appraisal Journal, Vol. 39 No. 2, April, pp. 277-84. Von Neumann, J. and Morganstern, O. (1947), The Theory of Games and Economic Behavior, Princeton University Press, Princeton, NJ. Worzala, E., Lenk, M. and Silva, A. (1995), “An exploration of neural networks and its application to real estate valuation”, Journal of Real Estate Research, Vol. 10 No. 2, pp. 185-201. Zadeh, L.A. (1964), Memorandum 4307-PR 1, Rand Corporation, Santa Monica, CA. Zadeh, L.A. (1965), “Fuzzy sets”, Information and Control, Vol. 8, pp. 338-53. Zadeh, L.A. (1968), “Probability measures and fuzzy events”, Journal of Mathematical Analysis and Applications, Vol. 23, pp. 421-7. Further reading Barndorff-Nielson, O.E. and Cox, D.R. (1989), Asymptotic Techniques for Use in Statistics, Chapman & Hall, London. McLean, D., Mundy, B. and Kilpatrick, J. (1999), “Summation of evidentiary rules for real estate experts mandated by Daubert v. Merrell Dow Pharmaceuticals, Inc.”, Real Estate Issues, Fall. Mahle, S. (1999), “The impact of Daubert v. Merrell Dow Pharmaceuticals, Inc., on expert testimony: with applications to securities litigation”, The Florida Bar Reporter, April. About the author Dr John Kilpatrick is the CEO of Greenfield Advisors, a 34-year-old firm specializing in complex real estate problem-solving with offices in Seattle and Atlanta. He has a PhD in Real Estate Finance, and prior to joining Greenfield he taught real estate and corporate finance at the University of South Carolina. He was also one of the founders, and the first Administrator, of the South Carolina Supercomputer Network. Dr Kilpatrick is a Visiting Scholar in Real Estate at the Zicklin School of Business, Baruch College, City University of New York, and serves on the Finance Department Advisory Board for Washington State University. He is the author of four books on real estate and contributing author to three others, as well over 100 monographs, scholarly journal articles, and working papers. Dr Kilpatrick is a Fellow of the American Real Estate Society, the Membership Chair of the Real Estate Counseling Group of America, and serves on the Editorial Board of the Journal of Sustainable Real Estate. He is featured in the current editions of Who’s Who in America and Who’s Who in Business and Finance. He has been featured in the Wall Street Journal, the New York Times, the Boston Herald, and numerous other major periodicals. John Kilpatrick can be contacted at: john@greenfieldadvisors.com JPIF 29,4/5 550 To purchase reprints of this article please e-mail: reprints@emeraldinsight.com Or visit our web site for further details: www.emeraldinsight.com/reprints