Improved spectral imaging microscopy for cultural heritage through oblique illumination Oakley et al. Herit Sci (2020) 8:27 https://doi.org/10.1186/s40494-020-00369-0 R E S E A R C H A R T I C L E Improved spectral imaging microscopy for cultural heritage through oblique illumination Lindsay Oakley1, Stephanie Zaleski1, Billie Males1, Oliver Cossairt2 and Marc Walton1* Abstract This work presents the development of a flexible microscopic chemical imaging platform for cultural heritage that utilizes wavelength-tunable oblique illumination from a point source to obtain per-pixel reflectance spectra in the VIS–NIR range. The microscope light source can be adjusted on two axes allowing for a hemisphere of possible illu- mination directions. The synthesis of multiple illumination angles allows for the calculation of surface normal vectors, similar to phase gradients, and axial optical sectioning. The extraction of spectral reflectance images with high spatial resolutions from these data is demonstrated through the analysis of a replica cross-section, created from known painting reference materials, as well as a sample extracted from a painting by Pablo Picasso entitled La Miséreuse accroupie (1902). These case studies show the rich microscale molecular information that may be obtained using this microscope and how the instrument overcomes challenges for spectral analysis commonly encountered on works of art with complex matrices composed of both inorganic minerals and organic lakes. Keywords: Spectral imaging microscopy, Oblique illumination, Optical sectioning, Surface shape, Dictionary learning, Micro analysis, Picasso © The Author(s) 2020. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creat iveco mmons .org/publi cdoma in/ zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Introduction The technical study of objects from art and archaeology often involves the removal of microsamples for chemical analysis and imaging, which provide fundamental micro- structural information to explore how these objects were created and to determine how their constituent materi- als may deteriorate over time [1, 2]. Cross-sections of artworks often present a complicated heterogeneity of organic and inorganic components, and it is therefore beneficial to identify each component with spatial sen- sitivity. Analytical techniques such as scanning electron microscopy/energy dispersive x-ray spectroscopy (SEM/ EDS) can provide element analysis at high resolution, but molecular information and the identity of organic constituents must be obtained from other analytical methods such as Raman or Fourier Transform Infrared (FTIR) microspectroscopy [3–5]. While Raman or FTIR instruments have been configured for high-resolution imaging [6–9], these measurements can be time con- suming and consequently within most cultural herit- age laboratories these techniques are primarily used for point analysis rather than molecular mapping. Elemen- tal images created via EDS mapping routines requires exposure of a sample to an electron beam while achiev- ing sufficient counting statistics. These conditions often cause radiative damage that precludes further imaging and analysis of the same sample location [1, 10]. As an alternative to these traditional workhorse techniques of analysis, here it is shown that microscale chemical maps can be produced using less damaging optical wavelengths through a spectral imaging microscope. We build on pre- vious work in cultural heritage [11] towards highly sen- sitive spectral reflectance imaging at the microscale by Open Access *Correspondence: marc.walton@northwestern.edu 1 Center for Scientific Studies in the Arts, Northwestern University, Evanston, IL, USA Full list of author information is available at the end of the article http://orcid.org/0000-0002-4436-3546 http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/publicdomain/zero/1.0/ http://creativecommons.org/publicdomain/zero/1.0/ http://crossmark.crossref.org/dialog/?doi=10.1186/s40494-020-00369-0&domain=pdf Page 2 of 11Oakley et al. Herit Sci (2020) 8:27 producing an instrument that provides both traditional optical images of microscopic features, as well as material maps, with as little as possible radiative damage to the sample. We also demonstrate the use of computational techniques to obtain diffraction-limited chemical images of both organic and inorganic materials present in cross- sections in a single experiment. The majority of optical microscopy experiments in cultural heritage are performed in reflection where the cross-section is prepared as a thick polished slab [1]. Because polished cross sections are excellent specular reflectors, it is necessary to configure the sample illumi- nation to suppress such reflections which carry none of the desired molecular information. While it is possible to avoid specularity using a transmission illumination geometry, this method requires thin sections prepared by microtomy. Although possible, the material properties of some samples can make them challenging to section and often the necessary cutting instrumentation is not readily available in art conservation laboratories [12–14]. Care- ful consideration of the limitations and advantages of specular suppression methods informed the design of the spectral microscope described here in order maximize signal and spatial resolution. One method to eliminate specular reflections is through the use of cross-polarization filters. Specular reflections that retain the polarization state of the source light are efficiently rejected by a second filter oriented orthogonally to a source filter, while randomly polarized light rays associated with diffuse scattering are allowed [15, 16]. However, precise crossed alignment is required to achieve peak efficiency and maximize the number of photons collected for imaging [17, 18]. More importantly, neither the extinction nor transmission efficiency of most polarization filters is uniform across the visible range which is not ideal for spectral microscopy over wide wavelength ranges (e.g., 400–1000  nm). Standard wide- field polarized light microscopes also have a large depth of field which contributes to a large volume of forward scattered light (on the order of 10–100 s of microns) and significantly increases spectral blur based on the absorp- tion properties of the sample matrix. To our knowledge, these large interaction volumes inherent to axial reflec- tion-geometries of a standard widefield microscope are the main reason spectral microscopy has not been adopted as a micron-resolution analytical technique in cultural heritage science. As an alternative to bright field microscopy, dark field (DF) spectral microscopes have been demonstrated in the context of biological applications with excellent spa- tial resolution, contrast, and shallow light penetration depths, providing a promising avenue for the exami- nation of cross-sections [19–25]. In DF microscopy, a hollow cone of light with a numerical aperture (NA) larger than the objective illuminates the sample. Usu- ally a beam stop is placed in the illumination path to match the size of the back aperture to create these con- ditions. Progressively dimmer images are produced with increasing magnification as the diameter of the beam stop must increase with the NA of the objective. Such a light path can be problematic when trying to efficiently recover light reflected from a highly absorb- ing sample, such as pigments routinely encountered in objects from cultural heritage. Additionally, while the DF setup should prevent diffracted light (less than first order) from entering the objective [26], certain orien- tations of microfacets due to surface roughness can still guide light into the objective and contaminate the image with specular reflections. Recent innovations have achieved enhanced contrast and spatial resolution through tunable-angle oblique- illumination (OI) as an alternative to DF illumination [26, 27]. OI from a single point source can increase con- trast and enhance details in the direction of incident illumination [27]. It has also been demonstrated that optimizing the angle of OI increases sensitivity based on the particle size and shape of the scattering ele- ment [28, 29]. Ma et  al. has drawn on recent advances in oblique ptychographic illumination from an annular array of lights to improve the confocal gate of a stand- ard brightfield microscope [30]. They demonstrated that individual point sources (LEDs) of monochro- matic light retain partial coherence and thus improve the lateral resolution of the microscope. When multi- ple sub-images from each illumination direction were synthetically combined, the contrast of the in-focus ele- ments was strongly enhanced while out-of-focus back- ground was smeared, thus improving the microscope’s overall axial sectioning capabilities. Using these princi- ples, an OI spectral imaging microscope will allow for enhanced absorption contrast at the sample surface, thus overcoming out-of-plane contributions to spec- tral blurring that are encountered in standard widefield microscopes. The spectral microscope we present here implements OI using mobile axes of illumination, as demonstrated by the analysis of paint cross sections of both known refer- ence materials and a sample from an early twentieth cen- tury painting by Pablo Picasso. These samples provide challenging case studies with complex heterogeneity at the microscale and imperfect surface features, which aids in highlighting the usefulness of OI in separating sub- surface light/matter interactions from the image plane of interest. In addition, OI successfully captures spectral signatures of organic pigments that would have been oth- erwise missed by elemental analysis alone. Page 3 of 11Oakley et al. Herit Sci (2020) 8:27 Results and discussion OI microscope design approach A schematic of the microscope is shown in Fig. 1, where the motorized movement of the illumination optics is indicated as “rotation angle α”. This illumination arm is decoupled from the detection axis of the microscope in order to send incident light to the sample obliquely from adjustable polar angles. Illumination around the speci- men was achieved by rotating the sample to multiple azimuthal angles (“rotation angle β”, Fig. 1) and taking an image at each position. Altogether, this multi-angle dome of illumination positions produces a large solid angle of light delivered to the sample in reflection and enables extraction of diffuse reflection measurements from mul- tiple views of the specimen. We implement a tunable light source not only for the advantageous possibility to conduct excitation and emission studies to image the spectral luminescence of the specimen. Also, on account of this configuration, the reflected light is then collected directly by the camera without  the need for additional dispersive optical elements or specialized objectives. The key idea behind the design of the microscope is that light reflected and scattered from sample’s surface is dependent on the oblique angle of illumination. The effect of illumination angle on acquired spectral data is illustrated  with the cross-section in Fig.  2 which was removed from the tacking edge of Pablo Picasso’s 1902 painting entitled La Miséreuse accroupie, painted dur- ing his Blue Period. From this cross-section, spectral image stacks were collected, and two illumination angles were selected and used to calculate  RGB images  using CIE 1931  color  matching functions: Fig.  2a, b show the sample illuminated at β = 0° and 180°, respectively. Some significant differences between the 0° and 180° views are indicated by arrows in the enlarged insets: the orange arrow shows dissimilar interreflection highlights caused by surface concavities around a void, and the blue arrow indicates a feature below the embedding resin missing in the 0° image yet appearing at 180°. The green arrow points toward a yellow-orange particle from which the reflectance spectra, shown Fig.  2c, were extracted. The spectrum at β = 0° has a shoulder at 535  nm which is greatly diminished at 180°. These observed changes in reflectance can be attributed to interreflections, subsur- face scattering, and volumetric scattering that fluctuate with the illumination angle (Additional file 1: Movie S1). Fig. 1 Schematic of the oblique illumination spectral microscope. The tunable illumination angle can be adjusted to change the angles of light in the polar direction as indicated by “rotation angle α” and the sample can be rotated in the azimuthal direction (“rotation angle β”) to produce multiple views of the specimen with different lighting angles Page 4 of 11Oakley et al. Herit Sci (2020) 8:27 However, diffusely reflected light from the in-focus plane should remain constant at all angles in our observation geometry, provided that the sample is topographically flat and the surface normal vector is aligned with the optical axis [31]. Here, following a simple calibration procedure, we suggest that a minimum projection through all images can provide an upper bound on the estimated value of directly reflected diffuse light based on the premise that these other scattering mechanisms listed above only add to the intensity of a given pixel [32]. It corresponds that taking the difference of a maximum and minimum image provides an estimation of these effects [33, 34]. The capability to separate light scattered from different focal planes is of significance to the design of our microscope because this light is responsible for undesired spectral blurring and prevents the collection of reflectance spec- tra at the highest possible spatial resolutions [35–38]. Surface shape estimation and min–max projections Although careful sample preparation can produce sam- ples with mirror-like finishes to the unaided eye, based on the resolution of our microscope, even techniques like microtomy or mechanical polishing with fine dia- mond slurries will leave cross sections with surface microfacets that can introduce angle dependent high- lights and shadow artifacts. Light reflection from these facets can contribute to an over- or under-estimation of diffusely reflected light depending on whether it is a topographical peak or valley, respectively, in the mini- mum projection of the image stack. Likewise, depend- ent on the orientation of the illumination arm to the detection axis, there will be a cosine falloff of illumina- tion, following Lambert’s law, that requires calibration to compensate. Such topography and sample orienta- tion, however, can be computationally flattened by nor- malizing the images to the surface shape gradient at each angle of illumination. Here we estimate the surface shape by using uncali- brated photometric stereo [39] given by Eq. 1: In this parameterization, the albedo (diffuse color) k is absorbed into the surface normal so |N | = k . I is the image intensity given by I (x, y) under the illumina- tion vector L · N is recovered by solving Eq.  1 via least squares given the known lighting directions. Figure  3a shows a series of RGB images of a reference cross-section comprised of four layers with pigment compositions indicated in Additional file  2: Table  S2. From these images, a normal vector map was calculated which demonstrates that there is measurable surface tex- ture in all four layers, even after polishing with slurries down to 0.5 μm. The most significant texture seems to be isolated to the second layer which contains a high con- centration of the organic pigment, madder lake. Based on this observed morphology, the polishing procedure (1)I = N · L , N = kN Fig. 2 RGB images of a cross section from Picasso’s La Miséreuse accroupie acquired at a β = 0° and b β = 180°. The arrows highlight distinct differences between images due to illumination direction. c Reflectance spectra extracted from the yellow-orange particle at β = 0° (red trace) and β = 180° (black trace). See Additional file 1: Movie S1 for multiple illumination angles of this sample Page 5 of 11Oakley et al. Herit Sci (2020) 8:27 employed may have selectively dissolved the water-solu- ble organic components. In addition to qualitative shape information provided by the normal map, surface gradients, which are the dot product of the surface normal vector and lighting direc- tion ( N · L ), were calculated for each illumination direc- tion. These gradients were then used to normalize the input images prior to calculating the output average, minimum, and difference as presented in Fig.  3b. More formally, the diffuse color is calculated for every illumi- nation angle (l) and wavelength (j) following, where Nxyz is the normal vector calculated from the RGB images using Eq.  1. Then the minimum intensity image is computed over all illumination angles at a given wave- length to filter out the interreflections and subsurface contributions: The per wavelength difference image is, (2)klj = Ilj ( Nxyz · Ll )−1 (3)min ∑ l k l j = I min j This difference image provides information about interreflections around pigment particles as well as multibounce light at layer interfaces: note, for instance, in Fig.  3b the blue scattered light between bottom lay- ers 1 and 2 as well as the orange scattered light between the top layers 3 and 4. In Fig.  3c and the corresponding inset detail, the average image is blurry at high magnifi- cations, yet the minimum projection (Fig. 3d) has resolv- able features below 1 µm, as demonstrated by the cluster of three pigment particles marked with a red arrow. This sub-micron resolution is at the theoretical Rayleigh dif- fraction limit of our imaging system using a ×20, 0.42 NA objective (~ 0.78 µm at λ = 535 nm). The explanation for this increase in the observed resolution between the average and minimum projections is provided schemati- cally in Fig. 3e. Two light rays incoming at oblique angles illuminate the sample surface. The forward scattering of the material matrix will produce lobes of volumetri- cally scattered light which is delineated by dashed lines. In the average projection image, which is the equivalent to standard darkfield illumination (See  Additional file  2: (4)differencej = Imaxj − I min j Fig. 3 Spectral image processing workflow of a reference cross-section. a Images were acquired in 10 angle increments from β = 0 to 360°, aligned, and used to calculate a surface normal vector map. Using this normal map, gradients were calculated for every illumination angle and then used to normalize the input images. b From the surface shape corrected image set, an average projection (darkfield equivalent), minimum projection (focal plane diffuse scattering only), and max–min (extraneous additional scatter) images were created. c Detail of the average image shows blur in the 1–10 µm range, but the minimum image d displays pigment particles with resolvable features below 1 µm. e The schematic diagram represents the overlap of two scattering paths which is explained in more detail in the text below. The minimum projection provides a measure of the overlap between the scattering regions (gray region) Page 6 of 11Oakley et al. Herit Sci (2020) 8:27 Figure S2 for more detail), patches adjacent to the overlap between the two lobes contain scattered contributions from both directions of illumination which contributes to blur. However, with the minimum projection method, all the light except that confined to the overlap region (gray shaded region, Fig. 3e) is computationally removed. Thus, in the minimum image, light scattered in the axial and lateral directions is effectively confined to a smaller interaction volume which serves to increase spatial reso- lution. This observed improvement in resolution over traditional Köhler illumination closely matches the reso- lution improvements detailed by Ma et al., who describe annular illumination from an array of point sources in transmission [30]. Also, in comparison to other advanced imaging techniques that can achieve high spatial resolu- tions, such as 2-photon or confocal methods, our method offers comparable optical sectioning and spatial resolu- tions for cultural heritage materials [40]. Mapping pigment distributions with sparse modeling While the multiple sample views can be used to extract structural and shape information, the spectral stack contains important information about the artist’s pro- cess based on the identity and distribution of colorants in the sample which we can calculate. A set of refer- ence reflectance spectra for the pure pigments included in the layered mock-up were formed into a diction- ary to perform a sparse linear unmixing and create pigment distribution maps. Our approach has been previously described [41] where we find a sparse rep- resentation w of the spectral stack x with respect to a dictionary D composed of known spectral signatures. This is achieved by solving a constrained optimization problem: Here, the l0 pseudo-norm counts the number of non- zero elements in w . Equation  5 was solved by using a greedy algorithm, orthogonal matching pursuit (OMP), where the number of non-zero elements for each pixel was set to two for this example and optimized sequentially. The resultant pigment distribution maps for the mockup sample are shown in Fig.  4. Based on the known content of the layers, the algorithm successfully isolated pigments to the correct strata. The only excep- tion is where the algorithm incorrectly identified Prus- sian blue in layer 2 of the specimen. This false positive result is not surprising since the dense mixture of mad- der and ultramarine in this example is highly absorb- ing and exhibits a spectral shape with an appearance similar to Prussian blue: an almost flat, near zero reflec- tance profile across the visible range. However, the loading of these high tint pigments in this mockup is extreme, and such a scenario is unlikely to be encoun- tered in a real sample. (5)Minimize �w0� such that x = Dw Fig. 4 Cross-section of the multi-layered mockup sample including the reconstructed RGB image and the corresponding pigment maps created by sparse modeling of the diffuse spectral dataset. The maps were created using an orthogonal matching pursuit (OMP) algorithm with a dictionary comprised of pure pigment spectra (ultramarine, cadmium yellow, vermillion, madder lake, zinc white, and Prussian blue). Distribution weights are normalized between 0 and 1 on a per pixel basis Page 7 of 11Oakley et al. Herit Sci (2020) 8:27 The limitations of the method The false positive result discussed above is a general limitation of visible spectral imaging with either highly absorbing or reflecting pigments and such spectra showing minimal spectral features should be examined carefully. The IR range (~ 900–1800 nm) of the source could also be utilized for a more detailed identifica- tion of pigments by a modular switching of the camera sensor and lenses appropriate for those wavelengths. However, perhaps the most significant step for creat- ing accurate maps of pigment distributions is the crea- tion of the database of potential pigments that could be found in the sample which requires some intui- tive or historical knowledge of likely artist materials. Finally, the current microscope design requires a long working distance objective in order to properly illumi- nate the sample and access multiple angles of illumina- tion. A higher NA objective would provide increased resolution but would also reduce the working distance and consequently the number of available angles for oblique illumination. Case study using a cross‑section from picasso’s La Miséreuse accroupie In 1902, the artist Pablo Picasso (1881–1973) painted a crouching woman on top of a landscape scene of a park. This hidden scene was discovered during routine exami- nation by X-radiography (see Fig.  5a, where the X-radi- ograph is rotated by 90°) conducted by the Art Gallery of Ontario, where the painting currently resides. To bet- ter understand the pigments used to paint the sky of the hidden painting, a small sample was removed from the tacking edge of the painting as indicated by the arrow in Fig. 5a. The sample was embedded in resin and the surface in Fig.  5b was revealed by cutting away excess resin using ultramicrotomy. Using the same procedure as described for the mockup cross-section, the spectral microscope was implemented to interrogate the identity and spatial distribution of the pigments distributed in four layers as indicated by the minimum projection image through 10 illumination angles shown in Fig.  5b. Figure  5c provides the surface normal vector map which reveals grooves cut into the sample by the microtomy knife. The image in Fig.  5d compliments this information and demonstrates surface interreflections from these grooves. In addition, Fig. 5 a Photograph of Pablo Picasso’s La Miséreuse accroupie (1902) and its corresponding X-radiograph rotated by 90° to reveal an underlying landscape painting (image courtesy of the Art Gallery of Ontario). The cross section was removed from the upper right tacking edge, indicated by the white arrow in a, imaged and decomposed into the b minimum projection, c surface shape components, and d the maximum-minimum. e–h Representative spectra are compared to references from each of the non-white paint layers Page 8 of 11Oakley et al. Herit Sci (2020) 8:27 features below the top surface of the transparent resin may be observed as a blurry orange or white glows (cor- responding to the top and bottom of the section) that are removed from the minimum image using our filtering procedure. Average area spectra were extracted from each layer as shown in Fig.  5e–h and pigment identifications were proposed based on comparison with reference paint mix- tures. A summary of the complimentary analytical tests that were performed and the proposed pigment identi- fications can be found in Additional file  2: Table  S1. In every layer, the microscope allowed for the complete complex heterogeneity of the sample to be examined in detail. In the top layer 4 it was proposed, based on the ele- mental information from SEM–EDS, that there might be more than one blue pigment present. Although many of Picasso’s blue period paintings are known to have uti- lized Prussian blue, the simultaneous use of ultramarine has only been documented in rare instances [42]. A com- parison of a reference paint created with a 180:1:1 w/w ratio of lead white: ultramarine: Prussian blue produces an excellent spectral match which was confirmed using Raman spectroscopy (Fig. 5e). In the thickest subsequent red layer 3, the spectral signature is dominated by the presence of vermilion (Fig.  5f ). However, upon closer inspection there is an anomalous yellow particle as pre- viously highlighted in Fig.  2. The spectrum of the region (Fig.  2c) displays a shoulder at 535  nm and matches a cadmium yellow reference (not shown); this was cor- roborated by the presence of cadmium from SEM–EDS, which demonstrates the effectiveness of identifying sin- gle particles in a complex matrix with our instrument. In the purple layer 2, the extracted spectrum clearly repre- sents a mixture with multiple absorption bands observed at 540, 567, and approximately 630  nm (Fig.  5g). SEM– EDS indicated that this layer was aluminum-rich, previ- ously inspiring the hypothesis that this layer contained an organic lake pigment. Consequently, a reference sample containing a 180:1:1 w/w mixture of lead white: ultra- marine: carmine naccarat was created. This reference matched the spectral features observed in the historical sample. Indeed, when spectra were extracted from iso- lated red particles (Fig.  5b, inset) in both the historical and reference samples, as viewed in Fig. 5h, characteristic absorption bands can be observed; these are separated by approximately 35–40 nm due to the nature of the ground and excited electronic states of the anthraquinone col- orant present [43]. Moreover, recent work on improved protocols for identifying organic reds from reflectance data pushes this identification further to support the specific presence of a cochineal based rather than plant based (madder) pigment and suggest that the substrate used for precipitation was aluminum rather than tin [44]. The unambiguous presence of single cochineal lake pig- ment particles is clearly indicated from the micro-spec- tral imaging; the only other technique currently capable of such sensitivity to organic-based colorants is surface- enhanced Raman spectroscopy [45]. Lastly, the presence of ultramarine, indicated by the band at 630  nm, was also confirmed by Raman spectroscopy. The sparse mod- eling of pigment distributions in all the layers is shown in Fig. 6. In this instance the number of non-zero elements in w, from Eq.  5, was constrained to three. This dataset demonstrates the power of OI spectral microscopy as a sensitive, low-dosage method to characterize pigments in cross-section microsamples that may be challenging to detect with alternative analytical techniques. Conclusions The OI spectral imaging microscope demonstrated in this work effectively captures the sample heterogene- ity observed in complex cultural heritage objects. The spatial distribution of chemical information from reflec- tance spectra can be routinely captured in addition to surface shape information made possible by the multi- axis movement of the illumination. Also, the hyperspec- tral datasets are generated using low intensity radiation, which translates to decreased potential for sample dam- age. Ultimately, as demonstrated from the case study of a cross section sample from La Miséreuse accroupie, the OI spectral imaging microscope provides a wealth of molec- ular data on microscale features and material properties that can be mined computationally. Materials and methods OI spectral microscope The OI spectral microscope was built on a Cerna modu- lar microscope frame (Thorlabs). A tunable light source consisting of a 300-watt xenon light source filtered through a monochromator (Newport TLS260-300X), enabling down to 5  nm wavelength resolution, was used to illuminate the sample. The broad nature of absorption bands for most pigments in this wavelength range means that the 5  nm resolution is sufficient to capture any unique and identifying spectral features. A 4f arrange- ment of achromatic lenses (Thorlabs AC254-06/30-A- ML) were used to refocus the beam to a point at a far distance from the focal plane (~ 20  mm) as is shown in Fig. 1. Two DT-80 stages (Physik Instrumente) were used to rotate the sample platform and the illumination arm via a microprocessor board (Tiny-G CNC controller) pro- grammed with custom Python scripts. We implemented a scientific complementary metal oxide sensor (sCMOS, Photometrics PrimeBSI) with a 4.2-megapixel array. To optimize data management, most experiments utilized Page 9 of 11Oakley et al. Herit Sci (2020) 8:27 an annular array of azimuthal lighting positions col- lected at a fixed polar angle of 50° relative to the sample surface normal. To accommodate for the movement of the illumination arm and stage, a long working distance objective (MY20X-804-20X Mitutoyo Plan Apochromat Objective, 0.42 NA, 20 mm WD) was used to collect the sample scattering. Spectral image stacks were collected across the visible and near-infrared (430–1000  nm) with an image captured at 2 or 5 nm increments. Image post‑processing In a typical experiment, 115 sub images are collected from 430 to 1000  nm before the sample stage is rotated to 9 additional azimuthal angles (every 36°) for a total of 1150 images. The images at each wavelength form a wavelength or lambda stack. For each azimuthal angle, the lambda stacks are registered by the known rotation angle as well as small translations due to micron scale wobble and center- ing misalignments of the sample platform. The translation transform is determined by a scale-invariant feature trans- form (SIFT), template matching, or by hand labelling fea- tures in each image. The translation is determined on RGB images calculated from each lambda stack: the wavelength images are first collapsed into a XYZ tristimulus image by multiplying each wavelength by a color matching function and summing across all wavelengths to produce X, Y, and Z bands prior to a transformation into RGB color space [46]. After the translation transform is known for each sample rotation angle, it is then applied to all 1150 images to produce a well-aligned four dimensional dataset com- posed of 2 spatial dimensions plus wavelength and lighting angles. After registration, these multiple angled views of the sample are collapsed into a single image by calculating an average or minimum composite image. This procedure is repeated for each wavelength and the minimum projec- tions are concatenated to create a spectral stack of diffuse- only scattered light. All registration, color transformation, photometric stereo, and matching pursuit steps are imple- mented as custom written Python scripts in the Fiji image processing suite which are available in the NU-ACCESS Github repositories (https ://githu b.com/NU-ACCES S/) [47]. Extracting spectral information To extract spectral information, a white reference target of fused silica (Heraeus, HOD®–High Purity Fused Silica Diffusor, 25  mm diameter × 1  mm thick), was used to normalize the obtained reflectance data between 430 and 1000 nm [48–50]. Noise in the measurement increases as the sensitivity of the camera decreases in the NIR range. Fifty dark images obtained with the source shuttered were averaged and then subtracted from both the reflec- tion and white reference images to mitigate the effect of background noise from the sensor. The reflectance for each wavelength at each pixel was calculated using Eq. 6, where I is the intensity of the pixel values for the sample Fig. 6 Images of a cross section from La Miséreuse accroupie (1902): reconstructed RGB image (top left) and the pigment maps for Prussian blue, vermilion, carmine naccarat (an aluminum lake of carminic acid), ultramarine, and lead white. Distribution weights are normalized between 0 and 1 https://github.com/NU-ACCESS/ Page 10 of 11Oakley et al. Herit Sci (2020) 8:27 images and white reference and D is the average pixel intensity of the set of dark images. Complementary analytical techniques SEM images were acquired on a Hitachi S-3400  N-II using a 20  kV accelerating voltage and 50  μA probe cur- rent. EDS maps were constructed using a higher probe current (70  μA) and a minimum of 30,000 counts per spectrum were required for analysis. Samples were coated in carbon tape to mitigate charge accumulation. Raman spectra were acquired on a Horiba LabRAM HR Evolution Confocal Raman instrument. The excitation wavelength was λex = 633  nm, using ×50 or ×100 LWD objectives and a 600  gr/mm grating. Acquisition times ranged from 5 to 50 s with 3.2–5% of the maximum laser power delivered at the sample. Painting cross sections Reference paintouts were prepared by mixing pigments obtained from Kremer with Galkyd Medium (Gamb- lin Artist’s Colors). Pigment mixtures were based on weight percentages shown in Additional file  2: Table  S2 and the alkyd medium was added dropwise until a work- able, homogeneous paint mixture was achieved by grind- ing pigment and medium together on a glass plate. Paint layers were applied evenly to glass slides and dried in ambient conditions or at 40 ℃. A multi-layered mockup was prepared by painting additional layers with a brush so that each underlying layer was not visible. When dry, samples were excised with a scalpel, mounted as a cross- section in an epoxy resin (Epo-Thin 2, Buehler) and pol- ished using a series of diamond suspensions in water to finish. The sample extracted from La Miséreuse accroupie was also mounted in an epoxy resin and an exposed sam- ple surface was achieved by microtomy using a glass knife. Supplementary information Supplementary information accompanies this paper at https ://doi. org/10.1186/s4049 4-020-00369 -0. Additional file 1. Movie showing registered RGB images of the Picasso cross sectional sample from 10 different lighting angles. Additional file 2. Supplemental information including an image of the OI microscope, images of a 1951 Air Force Resoultion target and compari- sons to traditional brightfield and dark field microscopy as well as tables summarizing the results of complimentary analyses of the Picasso cross section and compositional information about the prepared mock-up. (6)Refl (�,x,y) = ( I image (�,x,y) − Davg (x, y) ) ( [I white (�, x, y) − Davg (x, y)]� ) Abbreviations DF: Dark field; FTIR: Fourier transform infrared; LWD: Long working distance; NA: Numerical aperture; OI: Oblique illumination; SEM/EDS: Scanning electron microscopy/energy dispersive x-ray spectroscopy; SIFT: Scale-invariant feature transform; VIS-NIR: Visible through near infrared. Acknowledgements This collaborative initiative is part of the Center for Scientific Studies in the Arts broad portfolio of activities, made possible by the Andrew Mellon foundation. In addition this work made use of the EPIC and Keck II facilities of Northwestern University’s NUANCE Center, which has received support from the Soft and Hybrid Nanotechnology Experimental (SHyNE) Resource (NSF ECCS-1542205); the MRSEC program (NSF DMR-1720139) at the Materials Research Center; the International Institute for Nanotechnology (IIN); the Keck Foundation; and the State of Illinois, through the IIN. This work also made use of the MatCI Facility supported by the MRSEC program of the National Science Foundation (DMR- 1720139) at the Materials Research Center of Northwestern University. We thank Sandra Webster-Cook and Kenneth Brummel at the Art Gallery of Ontario for the motivation to better understand Picasso’s painting materials and methods using a cross-section removed from their painting. The authors also wish to thank Eme- line Pouyet for help in preparing the Picasso cross-section through microtomy. Authors’ contributions LO, SZ, OC, and MW designed the research; LO, BM and MW performed the research; LO, MW, OC, and SZ analyzed data. LO, SZ, and MW wrote the paper. All authors read and approved the final manuscript. Funding This research was made possible by the Andrew Mellon foundation. Availability of data and materials Image processing scripts are available through the NU-ACCESS GitHub linked in the Materials and Methods section. Image stacks are available upon request. Competing interests The authors declare that they have no competing interests. Author details 1 Center for Scientific Studies in the Arts, Northwestern University, Evanston, IL, USA. 2 Department of Computer Science, Northwestern University, Evanston, IL, USA. Received: 11 January 2020 Accepted: 28 February 2020 References 1. Khandekar N. Preparation of cross-sections from easel paintings. Stud Conserv. 2003;48:52–64. 2. Sandu ICA, Schäfer S, Magrini D, Bracci S, Roque CA. Cross-section and staining-based techniques for investigating organic materials in painted and polychrome works of art: a review. Microsc Microanal. 2012;18:860–75. 3. Schreiner M, Melcher M, Uhlir K. Scanning electron microscopy and energy dispersive analysis: applications in the field of cultural heritage. Anal Bioanal Chem. 2007;387:737–47. 4. Franquelo ML, Duran A, Herrera LK, Jimenez de Haro MC, Perez-Rodriguez JL. Comparison between micro-Raman and micro-FTIR spectroscopy tech- niques for the characterization of pigments from Southern Spain cultural heritage. J Mol Struct. 2009;924–926:404–12. 5. Casadio F, Daher C, Bellot-Gurlet L. Raman spectroscopy of cultural heritage materials: overview of applications and new frontiers in instrumentation, sampling modalities, and data processing. In: Mazzeo R, editor. Analytical chemistry for cultural heritage, topics in current chemistry collections. Berlin: Springer; 2017. p. 161–211. 6. Henderson EJ, Helwig K, Read S, Rosendahl SM. Infrared chemical mapping of degradation products in cross-sections from paintings and painted objects. Herit Sci. 2019;7:71. 7. Prati S, Joseph E, Sciutto G, Mazzeo R. New advances in the application of FTIR microscopy and spectroscopy for the characterization of artistic materi- als. Acc Chem Res. 2010;43:792–801. https://doi.org/10.1186/s40494-020-00369-0 https://doi.org/10.1186/s40494-020-00369-0 Page 11 of 11Oakley et al. Herit Sci (2020) 8:27 8. Ropret P, Miliani C, Centeno SA, Tavzes Č, Rosi F. Advances in Raman map- ping of works of art. J Raman Spectrosc. 2010;41:1462–7. 9. Salzer R, Siesler HW. Infrared and Raman spectroscopic imaging. Hoboken: Wiley; 2014. 10. Janssens K, Grieken RV. Non-destructive micro analysis of cultural heritage materials. Amsterdam: Elsevier; 2004. 11. van der Weerd J, van Veen MK, Heeren RMA, Boon JJ. Identification of pig- ments in paint cross sections by reflection visible light imaging microspec- troscopy. Anal Chem. 2003;75:716–22. 12. Derrick M, Souza L, Kieslich T, Florsheim H, Stulik D. Embedding paint cross-section samples in polyester resins: problems and solutions. J Am Inst Conserv. 1994;33:227–45. 13. Pouyet E, et al. Preparation of thin-sections of painting fragments: classical and innovative strategies. Anal Chim Acta. 2014;822:51–9. 14. Pouyet E, et al. Thin-sections of painting fragments: opportunities for combined synchrotron-based micro-spectroscopic techniques. Herit Sci. 2015;3:3. 15. Wolff LB, Boult TE. Constraining object features using a polarization reflec- tance model. IEEE Trans Pattern Anal Mach Intell. 1991;13:635–57. 16. Debevec P, et al. Acquiring the reflectance field of a human face In: Proceed- ings of the 27th annual conference on computer graphics and interactive techniques—SIGGRAPH’00. New York: ACM Press. 2000. p. 145–56. 17. Yang ZY, Lu YF. Broadband nanowire-grid polarizers in ultraviolet-visible- near-infrared regions. Opt Express. 2007;15:9510. 18. Wang JJ, et al. High-performance nanowire-grid polarizers. Opt Lett. 2005;30:195. 19. Théoret T, Wilkinson KJ. Evaluation of enhanced darkfield microscopy and hyperspectral analysis to analyse the fate of silver nanoparticles in wastewa- ters. Anal Methods. 2017;9:3920–8. 20. El-Khoury PZ, Joly AG, Hess WP. Hyperspectral dark field optical microscopy of single silver nanospheres. J Phys Chem C. 2016;120:7295–8. 21. Verebes GS, et al. Hyperspectral enhanced dark field microscopy for imag- ing blood cells. J Biophotonics. 2013;6:960–7. 22. Roth GA, Tahiliani S, Neu-Baker NM, Brenner SA. Hyperspectral microscopy as an analytical tool for nanomaterials. Wiley Interdiscip Rev Nanomed Nanobiotechnol. 2015;7:565–79. 23. Zamora-Perez P, Tsoutsi D, Xu R, Rivera_Gil P. Hyperspectral-enhanced dark field microscopy for single and collective nanoparticle characterization in biological environments. Materials. 2018;11(2):243. 24. Roth GA, del Sosa Peña MP, Neu-Baker NM, Tahiliani S, Brenner SA. Identification of metal oxide nanoparticles in histological samples by enhanced darkfield microscopy and hyperspectral mapping. JoVE J Vis Exp. 2015;106:53317. 25. Novikova IV, et al. Multimodal hyperspectral optical microscopy. Chem Phys. 2017;498–499:25–32. 26. Sanchez C, et al. Oblique illumination in microscopy: a quantitative evalua- tion. Micron. 2018;105:47–54. 27. Piekos WB. Diffracted-light contrast enhancement: a re-examination of oblique illumination. Microsc Res Tech. 1999;46:334–7. 28. Mappes T, et al. The invention of immersion ultramicroscopy in 1912—the birth of nanotechnology? Angew Chem Int Ed. 2012;51:11208–12. 29. Taylor MA, Bowen WP. Enhanced sensitivity in dark-field microscopy by optimizing the illumination angle. Appl Opt. 2013;52:5718–23. 30. Ma X, Zhang Z, Yao M, Peng J, Zhong J. Spatially-incoherent annular illumination microscopy for bright-field optical sectioning. Ultramicroscopy. 2018;195:74–84. 31. Hanrahan P, Krueger W. Reflection from layered surfaces due to subsurface scattering In: Proceedings of the 20th annual conference on computer graphics and interactive techniques—SIGGRAPH’93. New York: ACM Press; 1993. p. 165–74. 32. Szeliski R, Avidan S, Anandan P. Layer extraction from multiple images containing reflections and transparency. In: Proceedings IEEE Conference on computer vision and pattern recognition. CVPR 2000 (Cat. No.PR00662), vol.1. 2000. p. 246–53. 33. Park J, Sabharwal A, Veeraraghavan A. Direct-global separation for improved imaging photoplethysmography. In: 2018 IEEE/CVF conference on com- puter vision and pattern recognition workshops (CVPRW). New York: IEEE; 2018. p. 1456–609. 34. Nayar SK, Krishnan G, Grossberg MD, Raskar R. Fast separation of direct and global components of a scene using high frequency illumination. In: ACM SIGGRAPH 2006 Papers, SIGGRAPH’06. New York: ACM Press; 2006. p. 935–44. 35. Selci S. Phase and index of refraction imaging by hyperspectral reflectance confocal microscopy. Molecules. 2016;21:1727. 36. Zheng G, Kolner C, Yang C. Microscopy refocusing and dark-field imaging by using a simple LED array. Opt Lett. 2011;36:3987. 37. Shaw PJ, Agard DA, Hiraoka Y, Sedat JW. Tilted view reconstruction in optical microscopy Three-dimensional reconstruction of Drosophila melanogaster embryo nuclei. Biophys J. 1989;55:101–10. 38. Tian L, Wang J, Waller L. 3D differential phase-contrast microscopy with computational illumination using an LED array. Opt Lett. 2014;39:1326–9. 39. Li Z, Li YF. Microscopic photometric stereo: a dense microstructure 3D measurement method In: 2011 IEEE International conference on robotics and automation. New York: IEEE; 2011. p. 6009–14. 40. Latour G, et al. In situ 3D characterization of historical coatings and wood using multimodal nonlinear optical microscopy. Opt Express. 2012;20(22):24623–35. 41. Rohani N, et al. Automatic pigment identification on roman Egyptian paintings by using sparse modeling of hyperspectral images. In: 2016 24th European signal processing conference EUSIPCO. 2016. p. 2111–5. 42. Favero PA, et al. Reflectance imaging spectroscopy and synchrotron radia- tion X-ray fluorescence mapping used in a technical study of the blue room by Pablo Picasso. Herit Sci. 2017;5:13. 43. Morales KM. a note on characterization of the cochineal dyestuff on wool using microspectrophotometry. E Preserv Sci. 2015;12:8–14. 44. Fonseca B, Patterson CS, Ganio M, MacLennan D, Trentelman K. Seeing red: towards an improved protocol for the identification of madder- and cochineal-based pigments by fiber optics reflectance spectroscopy (FORS). Herit Sci. 2019;7(1):92. 45. Pozzi F, et al. Surface-enhanced raman spectroscopy: using nanoparticles to detect trace amounts of colorants in works of art. In: Dillmann P, Bellot- Gurlet L, Nenner I, editors. Nanoscience and cultural heritage. Paris: Atlantis Press; 2016. p. 161–204. 46. Berns RS. Digital color reconstructions of cultural heritage using color- managed imaging and small-aperture spectrophotometry. Color Res Appl. 2019;44:531–46. 47. Rueden CT, et al. Image J2: imageJ for the next generation of scientific image data. BMC Bioinform. 2017;18:529. 48. Mason JD, et al. Robust commercial diffuse reflector for UV-VIS-NIR applica- tions. Appl Opt. 2015;54:7542–5. 49. ElMasry G, Sun DW. CHAPTER 1—Principles of hyperspectral imaging tech- nology. In: Sun DW, editor. Hyperspectral imaging for food quality analysis and control. Cambridge: Academic Press; 2010. p. 3–43. 50. Chang CI, editor. Hyperspectral data exploitation: theory and applications. Hoboken: Wiley; 2007. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in pub- lished maps and institutional affiliations. Improved spectral imaging microscopy for cultural heritage through oblique illumination Abstract Introduction Results and discussion OI microscope design approach Surface shape estimation and min–max projections Mapping pigment distributions with sparse modeling The limitations of the method Case study using a cross-section from picasso’s La Miséreuse accroupie Conclusions Materials and methods OI spectral microscope Image post-processing Extracting spectral information Complementary analytical techniques Painting cross sections Acknowledgements References