Visualization and 3D reconstructed models have become more and more an established feature of research in the field of cultural heritage (CH). Since the 1980s, 3D handmade models have developed a strong tradition as a medium of communication in knowledge transfer. The most popular
Author's address: Stefanie Wefers, Ashish Karmacharya and Frank Boochs, i3mainz, Institute for Spatial Information and Surveying Technology, Hochschule Mainz, University of Applied Sciences, Lucy-Hillebrand-Straße 2, D-55128 Mainz, Germany; email: stefanie.wefers@hs-mainz.de; ashish.karmacharya@hs-mainz.de; frank.boochs@hs-mainz.de; Mieke Pfarr-Harfst, Technische Universität Darmstadt, El-Lissitzky-Straße 1, 64287 Darmstadt, Germany; email: pfarr@dg.tu-darmstadt.de
Permission to make digital or hardcopies of part or all of this work is granted without fee according to the open access policy of SDH.© 2017 SDH Open Access Journal
application is displaying these digital 3D reconstructed models in a video, especially in the context of an exhibition. One of the reasons for this application is that the 3D models transfer a message through images that are understood as a universal language requiring no further encoding [PfarrHarfst 2016]. Three-dimensionality within these models makes it possible for a broad public to get an idea of complex spatial interrelation and to contextualize CH objects within a 3D setting. Therefore, digital 3D reconstructed models are understood (in addition to written text and spoken words) as a further medium for storing and transferring knowledge [Pfarr-Harfst 2016].
Creating scholarly digital 3D reconstructed models of CH objects is a multidisciplinary task [PfarrHarfst and Wefers 2016]. The knowledge about the details of the object that is to be reconstructed comes from CH experts such as archaeologists, art historians, or historians. In a first step, the basis for the modeling has to be set through (1) a compilation of relevant available publications, 3D data, spectral data, images, descriptions, drawings, paintings etc. and, if necessary, (2) a digitization of non-digital information. Relevant information is compiled by CH experts; however, the data acquisition is done by technical experts capable of properly applying technologies such as 3D recording or spectral recording. Afterwards, modelers begin to create the digital 3D reconstruction, working with the compiled information and data. They regularly discuss interim reconstruction results with the CH experts, allowing for adjustments and clarification of (so far) neglected details, until the digital 3D representation reflects the vision of the CH expert. How detailed this information needs to be is very often underestimated by CH experts who, both as information recipients and providers, are used to two-dimensional visualizations within their domain. In contrast to 3D reconstructed models, such 2D visualizations easily allow missing information to be hidden. For digital 3D reconstructed models, however, detailed three-dimensional descriptions of the CH object and its constructive parts are needed; if it has been visualized in its context, its spatial position and context are also required. Many decisions have to be made by the CH expert: for some of these, no scholarly evidence might exist; for other decisions further reconstruction options, which are still under scholarly discussion and might not be concluded at all, have to be neglected. Supporting such visualization applications could be achieved by implementation of application requirements in a machine-readable ontology-based knowledge representation. Through semantic web technologies, knowledge can be inferred from ontology-based knowledge representations. The overall idea is to create a recommendation platform for CH experts that highlights emerging challenges during 3D reconstruction projects while taking individual input interactively into account.
With the overall intention of prescribing the optimal selection of spatial and spectral recording technologies and the technical strategy or strategies for fulfilling the data-specific demands of CH applications, an ontology-based knowledge representation is currently under development. It is called COSCHKR, and acronym for Color and Space in Cultural Heritage Knowledge Representation [Karmacharya et al. 2016; Wefers et al. 2016]. COSCH is the acronym of the European COST Action TD1201: Color and Space in Cultural Heritage (COSCH), during which the foundation of the knowledge representation was developed (http://www.cosch.info). Cultural application defines the requirements and demands that are necessary for successful completion of the application. These requirements and demands are related to the nature and quality of data, while the technologies and the underlying components are able to generate required quality data. Therefore, COSCHKR is driven through the primary axis of CH Applications → Data ← Technologies. This axis is represented through the top-level classes, CH Applications, Data and Technologies, and is defined through their interrelationships. The classes are further defined through rules, which describe their semantic constructs. In addition to these three classes defining the primary axis of the ontology, the ontology includes classes representing the physical (CH) objects through the top-level class Physical Thing, as well as other external impacts through the top-level class External Influences. Both classes represent restraining/supporting constraints to the technologies. These constraints are semantically constructed through rules inside these two classes. Physical objects represented in Physical Thing provide restraining/supporting constraints on technologies, e.g., through their size and/or shape. Likewise, specifications of the project, such as available budget, which are encoded inside External Influences, may have an impact on the recommendation of an optimal recording device, e.g., due to higher costs than are available in the project budget. The structure, relationships, and rules are logic-based constructs encoded into the ontology through machine interpretable language (OWL). This enables machines to participate and assist in interpreting and concluding, through reasoning, the logic-based facts. The top-level classes and relations of COSCHKR are illustrated in Fig. 1 [for details see Karmacharya; Wefers et al. 2016].
Figure 1. Top-level classes and relationships of the ontology COSCHKR.
The rules binding CH Applications and Technologies through Data initiate navigation through the ontology that requires knowledge assertions from the user. In many cases, these assertions reflect the actual discussion between a humanities expert, who would like to record a physical CH asset, and a technical expert, who is actually recording the asset. This assertive mechanism feeds the facts into the ontology, and these facts provide the basis for the rules to be inferred inside the ontology.
The word "semantic" implies "meaning" or "understanding". The technologies implementing semantics aim at explicitly describing the meaning of content; they do not aim at the content itself. The evolution of the Semantic Web framework in the late 20th century has given a boost to technologies such as artificial intelligence, which exploit the semantics of content to reason onto the conclusions [Berners-Lee 2006]. In this context, ontologies play a major role: They are traditionally used to structure knowledge by defining terms and relationships describing a specific knowledge domain [Heflin 2004] and were first used in philosophy dealing with the theory of existence [Hofweber 2011]. They define the descriptive semantics of the various entities related to a specific knowledge domain. The Semantic Web considers ontologies as the main medium to express and represent structured knowledge. It uses a standardized Web Ontology Language (OWL) (https://www.w3.org/OWL/) bringing expressive and reasoning power to the semantic web.
So far, ontologies have been rarely used for implementing and representing 3D visualization because:
- few studies have been carried out to evaluate and classify 3D visualization techniques; they mainly focus on the interaction techniques [V-MusT 2013] and/or software/hardware configurations [Potter and Wright 2006] used in specific applications; and
- different classifications, terminologies, and taxonomies have been defined, each for specific aims and applications [Shu et al. 2008]; due to the huge variety they cannot be considered for a heuristic view, which is needed to develop an ontology with the above described purpose.
Even when such kinds of ontologies are developed, they are used to define problems in designing visualization techniques and/or their preferred application areas, without giving details on the specifications of these techniques and where and how they are intended to be used. For example, the Top Level Visualization Ontology (TLVO) provides a common vocabulary to describe visualization data, processes, and products [Brodie et al. 2004]. More recent research analyzes visualization taxonomies and proposes modifications of TLVO [Pérez et al. 2010]. Other examples are:
- a visualization ontology that adds semantics for the discovery of visualization services [Shuet al. 2008];
- "unifying ontology" for visualization systems that allows reasoning on the optimal use/reuse/synthesis of graphical representation for a special situation [Voigt and Polowinski 2011]. For our purpose, the ontology lacks many basic concepts, e.g., there is no representation of 3D visualizations or the targeted applications, and last but not least, the ontology is no longer accessible; and
- a query based selection of relevant visualization techniques [Metral et al. 2012].
The work on visualization ontologies primarily focuses on either classifying hardware/software and their configurations or selecting specific representations. They primarily work on querying asserted knowledge for the answers. However, the selection of visualization techniques and workflows, based on the demand(s) of the required data for a specific purpose, requires the following essential components to be listed in an ontology: (1) visualization techniques/workflows, (2) technologies dictating the data generation, (3) the content and quality required for the visualization, and (4) the accessibilities of information/data that will generate the required content. Moreover, such a kind of an ontology would need to bind these components together through proper relationships and rules. If these preconditions were met, the ontology could be used to develop optimal visualization techniques and workflows adapted to the needs of the targeted application.
Currently, COSCHKR relies on facts, which are either given by the existential physical facts of the CH asset or through proven technical capabilities of technologies, which are needed for the entire process of an optimal data acquisition and processing adapted to the requirements of the targeted application. In both cases, the ontology infers from the facts, which are logically encoded as rules, in various related classes. The ontology requires asserted knowledge of these facts before inferring them.
An illustration of such an assertive process is presented through the example of an already implemented typical CH application, which focuses on the "Analysis of Geometric Alteration". An example for such an application is a project focusing on archaeological waterlogged wooden samples, which were 3D documented before and after conservation treatment in order to be able to evaluate geometric alterations caused by the conservation treatment through comparison of the data sets [e.g., Mazzola 2009] (http://www.rgzm.de/kur/index.cfm?Layout=holz&Content=start). As an example, we present the first basic semantic rules for this CH application Geometric Alteration (which is a subclass of the top-level class CH ApplicationsM; for more details, see Karmacharya, Wefers, Boochs 2016; Wefers, Karmacharya, Boochs 2016). It requires high quality 3D data representing the object for at least two instances. The class is semantically constructed through the rule that states this requirement. The rules are defined through Description Logic Statements (DL) [Baader 2003], but for simple explanations, we use equivalent lexical statements. Equation 1 states that the class Geometric Alteration requires at least two 3D data of the object, and equation 2 states that these data need to be of high quality.
Geometric Alteration has Requirement on Data minimum 3D Data
has Representation of one Physical Thing(Eq. 1)
Geometric Alteration has Requirement on Data minimum 3D Data has
Quality High Quality(Eq. 2)
The equations 3 to 5 illustrate the rules inside the ontology, which semantically define different components of the technology, while equation 6 displays the inference result of those rules.
Structured Light 3D Scanning hashas Main Operating Instrument
Structured Light 3D Scanners(Eq. 3)
Structured Light 3D Scanners has Measurement Principles
Triangulations(Eq. 4)
Triangulations has Generation Of Data 3D Data has Quality High
Quality OR Medium Quality OR Low Quality(Eq. 5)
This permits the inference:
Structured Light 3D Scanning has Generation Of Data 3D Data
has Quality High Quality OR Medium Quality OR Low Quality(Eq. 6)
The above-described first selection of technologies is afterwards iteratively inferred for their optimal suitability against the restraining/supporting constraints of the characteristics of physical objects (inside Physical Thing) or other external impacting factors (inside External Influences). For example, the size of the object, in this case the archaeological waterlogged wooden samples, plays a major role in filtering out technologies. The size of these wooden samples is on average 10 x 6 x 6 cubic cm and is asserted as "small". Through this classification, instruments and their subsequent technical processes suitable for recording only large physical objects such as Laser Scanners and Laser Scanning are filtered out.
These assertions are the facts that the ontology requires and they are put forward to the user (in most cases CH experts). They are formulated through predefined knowledge inside the ontology describing its individual components. The system asks for further assertions about the knowledge of the object and other environmental or project-oriented issues in order to give a recommendation as to which technology should be considered due to its optimal suitability. Fig. 2 displays a User Interface simulation that recommends structured light 3D scanning for recording waterlogged wooden samples with the purpose of evaluating possible geometric alterations.
As mentioned above, however, CH applications encompass not only analysis of spectral or spatial data but also very often focus on visualization of data. A wide variety of visualization types (interactive, animation, graphical etc.) exist for 3D and/or spectral data representing existing physical CH objects, reconstructions of only partly preserved CH objects, and 3D reconstructed models. Whether COSCHKR should also address these manifold visualization types, especially the implementation of visualization applications for digital 3D reconstructed models [Pfarr-Harfst and Wefers 2016], is a challenge (see below).
Figure 2. Simulated User Interface for knowledge assertion of waterlogged wooden samples.
As regards CH applications which focus on visualization, the big difference from other CH applications so far implemented in COSCHKR (see geometric alteration case study described above) is that non-existing or only partly existing physical things are digitally visualized in their complete and/or former condition. An example is a project focused on the scholarly digital 3D reconstruction of the Byzantine city of Ephesos [Grellert et al. 2010]. This visualization is characterized by the fact that the entire city, including its location in the landscape, its city-walls, streets, squares, buildings and monuments, is reconstructed based on the outcomes of archaeological excavations (foundations of buildings and monuments with associated but isolated building blocks and other construction parts, small finds found within the buildings, etc.) and their scholarly interpretation [e.g.,Mangartz 2010; Pülz 2010; Wefers 2015]. All physically existing finds and features of the excavations are real evidence; their spatial and spectral information can be used for modeling without contradiction. However, the interpretation of these finds and features, which means putting them into a context, such as a room or object and ascribing them to a specific spatial position within a room or object, is a hypothesis. The validity of such hypotheses differs depending on the preservation condition and/or completeness of the physical object and the level of information provided by other sources. Based on this, hypotheses can be classified by a number of levels and implemented into COSCHKR. However, the granularity of these levels still needs to be determined and cross-checked with other approaches. For in stance, Kuroczyński et al. [2015] set-up nine Levels of Hypotheses to be able to classify the scholarly content of a digital 3D reconstructed model. For this purpose, they define a hypothesis as a combination of the Level of Information provided by the source and the Level of Detail displayed in the digital model. Due to the purpose of COSCHKR and as described above, the level of hypothesis has to be defined differently. However, to allow a linkage of both concepts it might be of benefit to either have the same granularity or at least reflect the granularity of Kuroczyński et al. [2015] in a possiblyhigher or lower granularity of COSCHKR.
Implementing such kinds of Levels of Hypotheses into COSCHKR would allow us to give recommendations for the required involvement of a CH expert during the 3D modeling process. The level of hypothesis has impact on the 3D modeling workflow: that is, a higher level of hypothesis requires more and frequent involvement of the CH expert during the 3D modeling process. For example, in the project focusing on the scholarly digital 3D reconstruction of the Byzantine city of Ephesos, a water-powered stone sawing machine was included. Evidence for the stone-sawing machine was found within one room of terrace house 2, but actually not a single construction part of the machine itself is preserved. Only supports, postholes, chutes, the waterwheel raceway, and stones with cut-marks are preserved. This evidence, together with the expert's knowledge about stone-sawing machines and further contextual information, such as time of application, gave reason to set up the hypothesis that a water-powered stone-sawing machine was constructed in this room of terrace house 2 [Mangartz 2010]. Together with ground plots and sections of the preserved tangible CH 2D reconstruction, drawings were prepared as a basis for the digital 3D reconstruction. Especially during the digital 3D reconstruction of the machine, more detailed descriptions were requested by the 3D modelers (e.g., concerning the construction of the push rod suspension, extender wheel, frame saws, suspension of the frame saws, water wheel, mounting of the water wheel, etc.). It would have been advantageous for the whole reconstruction project if this verification process could have been planned at the very beginning.
Besides the above-described implementation of the level of hypotheses, further information about the object to be modeled and the targeted application would be required from the user as input, in order to be able to give more detailed support.
However, the biggest challenge related to CH applications of visualization is that they are very different from each other and rarely share similarities. Each case needs to be planned, implemented and applied independently. This is because CH visualizations depend on hypotheses provided by CH experts' interpretations. These hypotheses affect data processing steps in varying degrees. Every hypothetical interpretation needs to be independently and carefully handled during the data processing, which is in this case the digital 3D reconstruction or modeling. This is not the same as in the case described above (geometric alteration), which deals with facts and evidence.
Data processing activities form a workflow and are used to create a digital 3D reconstruction, which is used for a visualization application. These activities are listed as relevant subclasses of the top-level class Data Processing. They are related through rules that link them together and determine their workflow. These rules are created on facts and thus can fail to comply when asserted through hypothetical interpretations, especially when there are individual hypotheses for individual visualization cases. Such lapses can/will alter/break the sequence in the workflow. One of the greatest challenges within COSCHKR is including hypotheses and linking them to data processing tasks without interfering with the inference mechanism. We have suggested the classification of hypotheses based on their correspondence with the proven evidence. This, however, requires humanities experts' involvement in data processing steps. The balance between such involvement and their influence on adjusting parameters of the defined semantic rules within data processing activities (defined through subclasses of the top-level class Technologies) is a challenge that requires further clarification through future discussion and activities.
The authors would like to thank MEng. Guido Heinz (RGZM) for his essential input regarding the development of the Technologies classes. Part of this work was supported by the COST Action TD1201 "Color and Space in Cultural Heritage" (www.cosch.info).
2003. The Description Logic Handbook: Theory, Implementation and Applications. Cambridge: Cambridge University Press.
2006. Artificial Intelligence and the Semantic Web, 14. http://www.w3.org/2006/Talks/0718-aaai-tbl/Overview.html
2004. Visualization Ontologies: Report of a Workshop Held at the Nationale Science Centre. Report e-Science Institute
2004. OWL Web Ontology Language - Use Cases and Requirements. https://www.w3.org/TR/2004/REC-webont-req-20040210
2011. Logic and Ontology. Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/logic-ontology
2016. Knowledge Based Recommendation on Optimal Spectral and Spatial Recording Strategy of Physical Cultural Heritage Objects. In Proceedings of the Tenth International Conference on Advances in Semantic Processing, Semapro Venice: IARIA, 49–59.
2010. Ephesos - Byzantinisches Erbe des Abendlandes. Digitale Simulation und Rekonstruktion der Stadt Ephesos im 6. Jahrhundert. In Falko Daim and Jörg Drauschke (eds.), Byzanz - das Römerreich im Mittelalter. Schauplätze. Monographien des RGZM 84, 2, 2. Mainz: Verlag des RGZM, 241–254.
2015. Virtual Museum of Destroyed Cultural Heritage - 3D Documentation, Reconstruction and Visualisation in the Semantic Web. In Virtual Archaeology (Methods and benefits). Proceedings of the Second International Conference held at the State Hermitage Museum 1-3 June 2015. Saint Petersburg, 54–61.
2010. Die byzantinische Steinsäge von Ephesos. Baubefund, Rekonstruktion, Architekturteile. Monographien des RGZM 86. Mainz: Verlag des RGZM.
2009. What to do with "Large Quantity Finds in Archaeological Collections" – A KURproject. News in Conservation 6 (December 2009), 6.
2012. An Ontology of 3D Visualization Techniques for Enriched 3D City Models. In Usage, Usability, and Utility of 3D City Models–European COST Action TU0801 (p. 02005). EDP Sciences.
2010. An Enhanced Visualization Ontology for a Better Representation of the Visualization Process. In International Conference on ICT Innovations. Berlin, Heidelberg: Springer-Verlag GmbH, 342–347.
2016. Digital 3D Reconstructed Models - Structuring Visualisation Project Workflows. In M. Ioannides et al. eds. EuroMed 2016, Part I, LNCS 10058 544–555. DOI: 10.1007/978-3-319-48496-9_43
2016. Typical Workflows, Documentation Approaches and Principles of 3D Reconstructions. In M. Pfarr-Harfst et al., eds. How to Manage Data and Knowledge Related to Interpretative Digital 3D Reconstructions of Cultural Heritage? Berlin, Heidelberg: Springer-Verlag GmbH, 32-46.
2006. An Ontological Approach to Visualization Resource Management. In International Workshop on Design, Specification, and Verification of Interactive Systems. Berlin, Heidelberg: Springer-Verlag GmbH, 151-156.
2010. Das sog. Lukasgrab von Ephesos: eine Fallstudie zur Adaption antiker Monumente in byzantinischer Zeit. Forschungen in Ephesos 4, 4. Wien: Verl. der Österr. Akad. der Wiss.
2008. Bringing Semantics to Visualization Services. Advances in Engineering Software, 39, 6, 514–520.
2013. Virtual Museums by Interaction Technology. http://www.v-must.net/virtual-museums/categories/interaction-technology
2011. Towards a Unifying Visualisation Ontology. Technische Berichte Technische Universität Dresden, Fakultät Informatik 2011, 01. http://nbn-resolving.de/urn:nbn:de:bsz:14-qucosa-67559.
2015. Die Mühlenkaskade von Ephesos. Technikgeschichtliche Studien zur Versorgung einer spätantiken bis frühbyzantinischen Stadt. Monographien des RGZM 118. Mainz: Verlag des RGZM.
2016. Development of a Platform Recommending 3D and Spectral Digitisation Strategies. Virtual Archaeology Review 7, 15, 18–27.