Data and Software Strategy for Transmission Electron Microscopy: Open Science and Collaboration

03/05/2023 | General

Data and Software Strategy for Transmission Electron Microscopy: Open Science and Collaboration


This article summarizes the Deliverable 1.8 which discusses the development of a data and software strategy for transmission electron microscopy (TEM) within the ESTEEM3 project, focusing on open science and collaboration among research institutes.

The importance of open source software, standardization of file formats, and access to resources for researchers is emphasized, as well as the benefits to academia and industry.



The ESTEEM3 project aims to develop a comprehensive data and software strategy for TEM, addressing open data, open science, and collaboration among research institutes, software developers, and vendors. This article provides an overview of the current state of TEM software and data management, highlighting the importance of open source solutions, standardization of file formats and metadata, and access to resources for researchers. It also discusses the potential benefits of these developments to both academia and industry.

Strategy for Open Data and Open Science

The ESTEEM3 project supports the development of open source software and data management solutions for TEM, encouraging collaboration among research institutes and vendors. One notable example is the case study on event-based detectors for 4D STEM, where ESTEEM3 partner ANT demonstrated the potential of such detectors for high-speed scanning. This led to a collaboration with ASI and JUL to develop practical application support for the Timepix3-based detector, using sparse arrays to link event-based data with numerical arrays for data analysis. By adopting open source solutions like HyperSpy, pyXem, and LiberTEM, detector vendors can focus on their core products while leveraging existing software for scientific applications.

File Formats for TEM 

Open source file reader support for raw data continues to improve, enabling the conversion and opening of most file formats with different applications. The RosettaSciIO project, for example, has made HyperSpy file readers reusable and shareable between projects. However, standardization and formalization of technical metadata remain a challenge, with several ongoing initiatives attempting to address this issue.

Open Software and Data

The publication of software and data alongside scientific papers is becoming more widespread in electron microscopy, prompting discussions around formal requirements for publication in relevant journals. Adopting good software development practices is crucial, as is the growing collection of TEM and STEM data in open repositories for data science studies and comparisons.

Data Strategy for TA Users

A modular and extensible data management system based on open source components and standardized protocols has been implemented at JUL, with plans to make it easier to deploy at other sites. This system provides TA users with access to various data analysis facilities, including remote access, file sync and share, and electronic lab notebooks integration.

Evaluation of Open Software for TEM

The list of open source software for TEM remains largely up-to-date, with notable changes including the merger of Pixstem and pyXem, the creation of RosettaSciIO for file reading code reuse, and the addition of Py4DSTEM to the list of available software.

Conclusion and Outlook

The positive developments for software and data in electron microscopy in an open science and open data context continue, though metadata for TEM remains a challenging topic. Continued funding and appreciation of research software engineering is critical for progress in this area, as it bridges the gap between "works in principle" and "works in practice." Open source solutions created with public funding incentivize openness and interoperability in the software ecosystem, benefiting both academia and industry.