key: cord-0325176-o5cn4vvc authors: Borish, Michael; Post, Brian K.; Roschli, Alex; Chesser, Phillip C.; Love, Lonnie J. title: Real-Time Defect Correction in Large-Scale Polymer Additive Manufacturing via Thermal Imaging and Laser Profilometer date: 2020-12-31 journal: Procedia Manufacturing DOI: 10.1016/j.promfg.2020.05.091 sha: 8ac5ec9584a32482417d06f81eb6095a8099e15c doc_id: 325176 cord_uid: o5cn4vvc Abstract Defects can result in a failed part and are costly in terms of time and material. This cost is even greater in the context of large-scale additive manufacturing where the objects can be very large. As a result, a great deal of research has focused on defect identification and mitigation. To address defects during object construction, researchers at Oak Ridge National Laboratory’s Manufacturing Demonstration Facility investigated an in-situ control system comprised of two sensors: a thermal camera and laser profilometer. This control system adjusted material flow and build speed to mitigate three types of defects: low layer times, underfill, and overfill. Several test objects were constructed. The control system was found to adjust build parameters to handle low layer times of approximately 15 seconds and height deviations from -100% underfill (the absence of a layer) to 50% overfill. Within two layers, height deviations could be returned to within 10% of the expected layer height. Further, preliminary results suggest the system can compensate for uneven build surfaces. In large-scale additive manufacturing (AM), printing a large object can routinely take more than a day and involve hundreds of pounds of material. With such an investment of time and material, defects that occur during the build can be costly if the build ends in failure. Even without complete failure, the presence of defects can easily impact the object's physical properties to make it worthless. Defects increase the cost of production and create waste. That waste is at odds with lean manufacturing, a key concept of Industry 4.0 [1] . As a result, a great deal of effort has been devoted to techniques to identify and mitigate defects with varied types of sensors [2] . This work focuses on the identification and mitigation of three types of defects in a polymer-based process: underfill, overfill, and low layer times. Underfill occurs when less than the ideal amount of material is deposited. Overfill is the opposite whereby too much material is deposited. Low layer times are just that, layers with build times that are too low to allow the polymer sufficient time to cool. Each defect can easily lead object construction to fail. To address these defects, researchers at Oak Ridge National Laboratory (ORNL) utilized a laser profilometer and thermal camera. The laser profilometer was utilized to identify underfill and overfill while the thermal camera was used to identify low layer build times. This identification was accomplished through the creation of height maps and thermal thresholds, respectively. The laser profilometer and thermal camera were attached to BAAM (Big Area Additive Manufacturing) [3] . BAAM is a large-scale polymer additive manufacturing In large-scale additive manufacturing (AM), printing a large object can routinely take more than a day and involve hundreds of pounds of material. With such an investment of time and material, defects that occur during the build can be costly if the build ends in failure. Even without complete failure, the presence of defects can easily impact the object's physical properties to make it worthless. Defects increase the cost of production and create waste. That waste is at odds with lean manufacturing, a key concept of Industry 4.0 [1] . As a result, a great deal of effort has been devoted to techniques to identify and mitigate defects with varied types of sensors [2] . This work focuses on the identification and mitigation of three types of defects in a polymer-based process: underfill, overfill, and low layer times. Underfill occurs when less than the ideal amount of material is deposited. Overfill is the opposite whereby too much material is deposited. Low layer times are just that, layers with build times that are too low to allow the polymer sufficient time to cool. Each defect can easily lead object construction to fail. To address these defects, researchers at Oak Ridge National Laboratory (ORNL) utilized a laser profilometer and thermal camera. The laser profilometer was utilized to identify underfill and overfill while the thermal camera was used to identify low layer build times. This identification was accomplished through the creation of height maps and thermal thresholds, respectively. The laser profilometer and thermal camera were attached to BAAM (Big Area Additive Manufacturing) [3] . BAAM is a large-scale polymer additive manufacturing 48th SME North American Manufacturing Research Conference, NAMRC 48 (Cancelled due to machine. That is, it is a gantry-style system that uses a FDMtype process that is pellet rather than filament fed. BAAM has a build volume of approximately 7'x20'x6' and an example of the build volume is shown in Figure 1 . In addition to the sensors, multiple software changes were made to support the system. These changes were made as part of the slicing software, Human-Machine interface (HMI), and a separate data collection and processing PC. These changes allowed BAAM to adjust material flow and layer times to maintain process stability and produce parts that would have otherwise failed to complete. These changes were built upon previous work done with BAAM and other metal-based systems. Defect identification and mitigation is an important avenue of research as evidenced by the large body of work behind it. There are numerous overviews of which Everton et al. is one example [2] . This survey, focused on metal-based AM, describes numerous methodologies for process prediction and defect analysis. Further, numerous types of sensors have been evaluated for the purpose of process stability [2, 4, 5, 6] . These works cover a wide range of sensors including piezoceramic, high speed laser scanning, laser profilometers, RGB cameras, acoustic sensors, thermal IR cameras, and spectroscopy. The high-level goal of all this work is the same: maintain a stable build process free of defects in order to produce the expected part. When utilizing these sensors and methodologies, there are generally two types of processing: in-situ and post. In-situ processing refers to inspection and mitigation processes that occur during object construction. An example of this is the use of in-situ process controls by Tan et al. to achieve consistent quality metal AM parts [7] . Post processing occurs as part of a process after construction of the object is complete. This post processing could take the form of additional physical processes such as heat treatments or milling to guarantee specific object qualities [8, 9] . Post processing can also encapsulate an inspection process such as nondestructive testing [10, 11] . We build upon this work by utilizing similar sensors, however, there are unique challenges associated with polymer vs metal-based processes. There has also been significant work done with defect identification and mitigation in polymer-based systems. Similar to metal-based research, a variety of sensors have been used. In work by Faes et al., a laser scanner was used to monitor the part under construction [12] . In Narayanan et al., an image-based approach was utilized in combination with a machine learning model to identify when a defect was likely to occur [13] . Beyond monitoring the part in-situ, other work such as Anderegg et al. focused on monitoring the polymer extrusion process directly [14] . In this work, the authors monitored the material flow and extrusion pressure through the extruder assembly to improve process performance. We build upon this work by utilizing similar approaches. However, unlike this work, we focus on much larger scale systems. Relevant work with an in-situ laser profilometer has previously been done at ORNL [15] . In this work, Borish et al. used a laser profilometer to create height maps for a large-scale metal-based process. This height map was then used to adjust material flow to maintain appropriate layer height. Though the current work uses the same hardware, the change from metal to polymer introduces unique challenges that differentiate them from one another. Previous relevant work has also been completed with in-situ inspection via a thermal camera [16] . In this work, Borish et al. utilized a thermal camera on a polymer-based system. The thermal camera computed thermal thresholds of the object under construction. This thermal feedback was used to pause printing once a layer completed in order to give the layer enough time to cool. This component was tested in isolation. We now build upon this system component by utilizing it in conjunction with a laser profilometer. This allows more complex defect mitigation behavior by the system as a whole. Defect identification and mitigation is made possible through both hardware and software modifications to BAAM. In terms of hardware, BAAM was outfitted with a laser profilometer and a thermal camera. Software modifications were made to support these hardware additions. With these additions in place, the system functions as follows: • The slicing software generates additional commands at the end of each layer to signal the machine to collect data. • After a layer is constructed, the laser profilometer is passed over the object to collect data. • This data is used to generate height maps for mitigating underfill and overfill in subsequent layers. • The gantry is then moved into position for the thermal camera. • The camera arm is extended, and the temperature monitored. Construction is paused until the temperature falls below an appropriate threshold defined as part of the slicing process. • Once the layer temperature is below the appropriate threshold, the camera is retracted, and the build continues. • Subsequent layers' material flow is adjusted based on the height map information. If the height map indicates overfill, less material is deposited, and if the height map indicates underfill, additional material is deposited. We will now discuss relevant changes in more detail. First, we will discuss hardware, then software. In previous work, both the laser profilometer and thermal camera were discussed in isolation and the results for the laser profilometer focused on a metal-based system. We now expand these systems to work together and focus on results related to a polymer-based system. As such, the hardware is relatively unchanged from previous work. A general view of the hardware is shown in Figure 2 . We will summarize each component briefly. The laser profilometer was used for the detection of underfill and overfill, while the thermal camera was used to adjust layer build times based on layer temperatures. In previous work, the profilometer was mounted to both a metal and polymer-based system. A full explanation of the hardware and mounting can be found there [15] . To summarize, a Keyence laser profilometer was mounted to the gantry near the extruder nozzle. This allows full control to position the profilometer for collecting data. The profilometer was mounted between two copper blocks filled with air that act as heat sinks to keep the sensor cool. A close-up view of this hardware is shown in Figure 3 . Similar to the aforementioned laser profilometer, the thermal camera was also described in previous work [16] . Again, to summarize, the thermal camera was mounted on a gravitycontrolled gimbal attached to a telescoping arm. This allows the gantry to be moved out of the way and the arm extended, so the camera can collect data. Once data collection is complete, the arm is retracted, and the build can continue. A close-up view of this hardware is shown in Figure 4 . Just like the hardware, the foundations of the software were begun in previous work [15, 16] . Alterations were previously made to the slicing software, Human-Machine Interaction (HMI) software, and additional software dedicated to data collection, analysis, and decision making. Compared to the previous work, the alterations to the slicer software remain unchanged. ORNL has developed our own slicer, ORNL Slicer, that was altered to provide relevant information as part of the gcode output. As part of the typical slicing process, a 3D object is transformed into a sequential set of 2D cross sections called layers. At the end of each layer, additional information is inserted to create commands for the data collection of the laser profilometer and thermal camera. Additional information in the form of "ideal" cross sections are also output with the gcode. These ideal cross sections are used by the laser profilometer to create height maps from the scanned data by identifying Unlike the ORNL Slicer, alterations to the HMI and sensor software were new. Once a layer was constructed, BAAM would utilize both the laser profilometer and thermal camera to investigate the object. In our previous work, the sensor collection software had been written to generate height maps with a metal-based system. We expected this code to remain unchanged, but multiple steps were impacted due to differences between polymer and metal-based systems. We will describe the differences and analysis in more detail in the result section. However, in general, the creation of a height map is accomplished through several discrete steps. After a layer is constructed the laser profilometer is passed over the bounding box of the object in a raster pattern. Then, each individual pass of the laser profilometer is stitched together to create one single image. This stitching algorithm attempts to align the edges of various sections of a picture into a larger, composite image similar to the panoramic function on many devices. This image is then denoised using a simple box filter in this case. Following this, some minor smoothing is done. Finally, the laser profilometer image is masked using the "ideal" representation of the layer as generated by the slicing software and a height map of appropriate values is calculated. These calculations are accomplished through the help of the OpenCV library [17] . Further details can be found in previous work [15] . Similarly, the sensor collection software also handled thermal image analysis. This analysis was previously performed using a polymer-based system and so is relatively unchanged from previous work. However, the thermal camera component must now operate in conjunction with the laser profilometer. To summarize, the system waits until a layer is constructed. The new combined system first allows the laser profilometer algorithm to run. Then, the gantry is moved out of the way of the object. The telescoping arm where the camera is mounted is then extended. The camera monitors the temperature and calculates an average surface temperature at a regular time interval. Just as in the previous case with the height maps, OpenCV is used to create an image matrix from which the average is calculated. Once the surface temperature has reached the appropriate thermal threshold as defined as part of the slicing process, the data collection stops, the arm retracts, and the machine is signaled to continue object construction. The final component was the HMI and associated changes. Similar to our previous work with the metal-based system, the HMI for the polymer-based system would receive height map and thermal information for each layer. Once the appropriate threshold was reached, building would continue. As building continued for the subsequent layer, the HMI would interpolate into a matrix that represented the height map. Based on this height deviation, material flow would be adjusted as appropriate. Material flow was adjusted by altering two parameters: build speed and extruder RPMs. BAAM can already alter extruder RPMs to maintain material flow and does this as part of its normal operation. The reason for this is curves. As you build around a curve, you have a nonzero angular velocity as compared to building a straight line. As a result, material flow differs between the two states if no adjustment were made. For example, 100 RPM of material flow results in slightly different amounts of material exiting the extruder per unit distance around a curve as compared to a straight line. BAAM therefore adjusts extruder RPMs in relation to build speed in order to maintain consistent flow. To alter this flow, our changes to the HMI adjusted both the extruder RPM and build speed. If overfill was detected, the speed would be increased, and the extruder RPM would be lower in lockstep. On subsequent layers, this resulted in the machine building on overfilled section quicker with less material. Underfill functioned similarly in reverse. To test the efficacy of the system, a sequence of test objects was constructed. All objects were constructed on the same machine using 20% carbon fiber reinforced acrylonitrile butadiene styrene (ABS) manufactured by Techmer PM, LLC. These tests were as follows: 1. Construction of an object with low layer time (requires thermal camera processing) 2. Construction of an object with underfill (requires laser profilometer processing) 3. Construction of an object with overfill (requires laser profilometer processing) 4. Construction of an object with under and overfill (requires laser profilometer processing) 5. Construction of an object with under and overfill as well as low layer time (requires thermal camera and laser profilometer processing) The test objects for test 1-4 were hexagons that measured 60 inches in diameter. The test object for test #5 was a square with a rectangular protrusion attached. This object measured 12 inches x 24 inches. During construction of these objects, the laser scanner collected data at a resolution of 1 mm between scans. The beam itself had a resolution of .3 mm between each measurement point. Additionally, when calculating the resultant height maps, nominal build height was measurements within 10% of the command layer height while anything greater was considered either underfill or overfill. Due to the large size of these objects and slow cooling rate, the thermal camera recorded data at 1 fps and calculated thermal threshold once per 10 frames. This leads to a thermal calculation once every 10 seconds. For these tests, the thermal threshold was set to 110C so that a layer's construction would not continue unless the part was beneath that temperature. In addition, each of these tests lasted a minimum of five hours in order to guarantee the existence of multiple errors. During each build, defects were regularly and intentionally introduced. Defects were introduced every ten layers starting from layer two and were created by manipulating the gcode commands. For example, turning off the material pumps during construction would guarantee underfill. The system was then allowed to tune build parameters for subsequent layers in order to maintain a stable build process. Additionally, the under and overfill defects introduced were randomized in both location within the layer and size. One primary issue with generating height maps with a metal-based system is the reflectivity of the material. This high reflectivity creates issues for accurately measuring the height with the laser profilometer. As a result, individual pixels would routinely have poor measurements. To compensate for this, we introduced a noise reduction step that filled in image holes with neighborhood averages. Indeed, this was necessary as we found a significant amount of noise represented by error codes returned by the laser profilometer. For a more in-depth discussion of the noise, please see our previous work. The error code rate on an example metal component was approximately 12% which amounted to a few thousand pixels in absolute terms. In comparison, for the polymer-based system, the error rate was only 8%, however, the objects are much larger and resulted in tens of thousands of erroneous pixels. The issue then is that processing this many pixels takes too much time between layers. Unlike metal-based construction that could potentially continue after a significant pause, polymer-based construction is limited by material cooldown. If the material cools too much, construction cannot continue as the part will suffer from delamination. Further, we noticed a curious pattern in the data. Previously, the erroneous pixels would be distributed throughout the laser profilometer data when dealing with a metal-based part. However, the erroneous pixels of the polymer-based scan would show a regular pattern. This pattern would consist of several individual profiles of good data, followed by several profiles filled with erroneous data. We found this pattern corresponds to the individual beads of material. For example, if you look at one of the completed objects in the following sections, even though the object is solid, at the smallest scale there is a gap between beads. As a result, we found a repeating pattern representative of a bead, followed by a small gap, followed by a bead, etc. In recognizing this pattern and given the size of the image, instead of denoising these erroneous pixels as in the metalbased case, we simply filtered them out of the image. In conjunction, we implemented a small history buffer on the HMI such that as it interpolates into the height map matrix, the most recent adjustment is recorded. If a filtered pixel is found, the system continues with its most recent height adjustment or uses its nominal build parameters if no history is available. In this way, we were able to speed up image processing dramatically. However, this also represents an opportunity for future work as a GPU implementation or external computing sources could be utilized to fully process the image under appropriate time constraints. Before presenting the successful builds, examples will be provided to showcase the construction failures caused by the three types of defects. First, Figure 5 showcases an object with low layer times that did not have the benefit of a thermal camera. As this figure shows, without sufficient cooling, the material cannot support the weight of itself and eventually collapses. Next, an object with underfill. As the defect implies, there is too little material compared to the commanded amount. As a result, the object has areas where material can simply be missing causing internal voids. This defect then propagates up through subsequent layers as shown in Figures 6a-c. Finally, an object with overfill. This defect is the opposite of underfill. Overfill is the result of too much material compared to the commanded output. As a result, the object has areas that are too high, and the object is uneven. One further consequence of this is subsequent layers may impact the machine. This defect and its propagation is shown in Figures 7a-c. As can be seen, the system is able to compensate appropriately for low layer times, underfill, and overfill. For test #1, the hexagon built had a low layer time of approximately 20 seconds. Without the additional wait time, this object would have collapsed under its own weight after several layers had been constructed. This result echoes results found in our previous work which investigated the use of a thermal camera in isolation. This object is shown in Figure 8 . Tests #2 -#4 showcase BAAM's ability to adjust to various types of underfill, overfill, or a combination of both respectively. For these tests, defects were found to resolve after at most three layers. For example, the defects introduced on layer two would no longer be noticeable once layer six had begun deposition. This additional resolution time makes sense as we were adjusting material flow on subsequent layers without altering the original path planning. Test #2 is shown in Figure 9a -c. In Figure 9a , various sections of underfill can be observed throughout the layer. In Figure 9b , the previous underfill has been resolved a few layers later, and Figure 9c shows the resultant hexagon. For this test, the underfill represents a height deviation of -100% (a complete lack of a layer). The layer height targeted was .15 inches per layer so underfill defects resulted in sections of the layer being .15 inches too low. For the immediate layer after which the underfill was detected, the system would maintain material output while reducing its speed by half. This creates an output of double the amount of material for the same unit distance to compensate for the defect. Similar to test #2, test #3 introduced overfill into the object. Test #3 is shown in Figure 10a -c. In Figure 10a , the top section that was overfilled is highlighted. In Figure 10b , several layers later, the defects have resolved. Figure 10c shows the completed hexagon. In order to reduce the chance that the machine would impact the material and cause damage, overfill defects were limited to +50% height deviation. Again, a layer height of .15 inches per layer was targeted resulting in defects that were up to .225 inches high. Similar to underfill, the Test #4 combined both underfill and overfill defects. The results are similar to test #2 and test #3 and are shown in Figure 11a -c. As seen from the resulting objects, the laser profilometer was able to adjust for significant amounts of underfill and overfill. However, this is only a first step and a natural extension would be true path re-planning. One of the major limitations of this approach is the need for subsequent pathing to overlap defect areas. For example, should a defect occur on the final layer or on a layer that is not overlapped by additional geometry, the generated height map does not help. The natural extension to our system would identify those areas of disjoint geometry and create additional path planning. True path re-planning could address underfill on the final layer for example. Finally, test #5 was also successful and adjusted for underfill, overfill, and a low layer time of approximately 15 seconds. The successfully completed object is shown in Figure 12 . Unlike tests #2 -#4, the underfill and overfill in test #5 took place as part of the wall of the object rather than in the interior. As can be seen, while the object did complete successfully, the additional material from the overfill had a negative impact on the overall final quality of the objects. This does leave open the possibility for improvement, however, in many cases this small decrease in quality may be perfectly acceptable rather than complete failure of the part. Typical for polymer systems, BAAM too uses build sheets on top of the build table to increase adhesion of material. These build sheets are made of similar ABS material as the objects. The large, black build sheets on BAAM can be seen in numerous previous figures. These build sheets are held in place by taping down the edges and engaging a vacuum that pulls the sheet flat onto the build table. Though not planned, test #2 involved an additional variable by accident: an uneven build surface. This was due to a failure of the vacuum on the build table. As a result, the sheet that covers the table bowed upward creating a convex surface. The results of object construction were as previously presented, Conceptually, an uneven build surface is the same as overfill or underfill. It is an area that is above or below the standard build plane and can be treated similarly. There is also an analog between this issue and our previous work on the metal-based system. In that work, it was common to use the laser profilometer to scan the build volume before construction begins. That is because, in the metal-base system, a titanium plate is placed on top of the build table upon which the object is constructed. These build plates are never perfectly flat, and so, scanning the plate after mounting allows for material flow adjustments for the initial layer of the object. Here, the issue is largely the same, though the size of deviation was much larger. Overall though, the system was able to deal with this issue and continued to successfully construct the object. Figure 13 highlights the deviation of the build sheet and a closeup of the curve in the first few layers of the resulting object from Figure 9c. As demonstrated, BAAM can construct objects while identifying and mitigating defects in the form of underfill, overfill, and low layer times. BAAM currently accomplishes this using two sensors: a laser profilometer and thermal camera. These sensors allow the creation of a height map and monitoring of appropriate thermal thresholds respectively. The height map is used for subsequent layers to adjust material flow while the thermal analysis forces the machine to wait until a specific thermal threshold is reached before continuing construction. As mentioned, there are numerous avenues for improvement of the system. The largest natural extension would be to improve the system with true path re-planning. Such a development effort would allow the presented defect mitigation for a larger class of objects. This will be an avenue we pursue in the future concurrently with slicing software that can support such a capability. A cyber-physical systems architecture for industry 4.0-based manufacturing systems Cincinnati Big Area Additive Manufacturing (BAAM). info. ornl. gov Acoustic emission for in situ quality monitoring in additive manufacturing using spectral convolutional neural networks A multi-bead overlapping model for robotic wire and arc additive manufacturing (WAAM). Robotics and Computer-Integrated Manufacturing In situ multielemental analysis by laser induced breakdown spectroscopy in additive manufacturing Laser metal deposition in-situ process control for different build strategies Additive manufacturing of 17-4 PH stainless steel: post-processing heat treatment to achieve uniform reproducible microstructure Developments in construction-scale additive manufacturing processes Nondestructive evaluation of additive manufacturing. National Aeronatuics and Space Administration (NASA) Nondestructive evaluation of additively manufactured metallic parts: In situ and post deposition Process monitoring of extrusion based 3D printing via laser scanning Support vector machine and convolutional neural network based approaches for defect detection in fused filament fabrication In-situ monitoring of polymer flow temperature and pressure in extrusion based additive manufacturing Defect Identification and Mitigation Via Visual Inspection in Large-Scale Additive Manufacturing Situ Thermal Imaging for Single Layer Build Time Alteration in Large-Scale Polymer Additive Manufacturing. Procedia Manufacturing The OpenCV Library. Dr. Dobb's Journal of Software Tools This material is based upon work supported by the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, Office of Advanced Manufacturing, under contract number DE-AC05-00OR22725.