Real-time processing of digitally acquired data dates back to at least the 1980s1, but decades later, most data processing still takes place long after acquisition, on a separate PC. Implementing real-time processing is especially important for in-situ experiments, in which the next experimental step is often determined by the changes observed in the microscope.
While computer processing power has increased exponentially, the amount of data collected in a single microscope session has also increased dramatically. Using Gatan’s latest CMOS based in-situ cameras, more than a TB of video can easily be acquired in a single day, even when the user is careful to limit the amount of data captured. This increases the difficulty to process the data in real-time. Although live FFTs are commonly available for analyzing data in real-time, as cameras get faster, only a subset of the acquired frames are actually analyzed at the microscope. For true real-time analysis, all the data generated by the detector would be processed live.
Gatan is working toward flexible solutions for live processing on the full stream of data from high-speed cameras. In this work, a scalable parallel file system with hundreds of TB of storage from DDN was used to store all data as it was acquired. While data is still being written to the filesystem, a 2nd PC running Gatan’s GMS (Digital Micrograph) software is used to process this data via scripting. In this case, the script converted a selected area electron diffraction pattern into a 1D profile, in which subtle changes could be more easily discerned by the TEM operator. This processed data was again saved in DM4 format on the DDN file system and the analyzed data was accessed by GMS on the computer at the TEM displaying the analyzed data at the TEM next to the live view window. All of this was done in a few seconds, allowing the user to modify the experimental conditions based on this feedback.
This proof-of-concept experiment can be extended so that any amount of processing is applied and the analysis performed on an HPC platform. The key is this experiment demonstrates that the computational demands of the data analysis do not conflict with the demands of the data capture. The near real-time feedback to the TEM operator enabled phase changes to be easily recognized. Future advances will speed the data processing, increasing the frame rate of the processed data window.
1W. Krakow, Applications of real-time image processing for electron microscopy, Ultramicroscopy. 18 (1985) 197–210.