Smart Microscopy as a Service#
Rafael Camacho, Centre for Cellular Imaging, University of Gothenburg
Approach to smart microscopy#
At the Centre for Cellular Imaging (CCI), Smart Microscopy is developed within a fee-for-service model, where users are trained to operate commercially available microscopy equipment from vendors such as ZEISS and Thermo Fisher. This approach prioritizes accessibility and reproducibility, ensuring that automation strategies remain compatible with vendor-provided microscope control software. Smart Microscopy at CCI is implemented with a holistic perspective, addressing not only adaptive feedback microscopy—where acquisition parameters are adjusted dynamically based on real-time sample analysis—but also downstream processes such as image processing, quality control monitoring, and data management. By considering the entire imaging pipeline from acquisition to analysis and integration, we aim to enhance efficiency, reproducibility, and scalability within a research infrastructure setting.
Methodology, Implementation details#
At CCI, we implement Smart Microscopy through automated feedback loops that connect image analysis with hardware control. Our workflows leverage machine learning-based object detection and segmentation to enable real-time adaptive imaging, adjusting acquisition parameters based on sample characteristics.
Light Microscopy, vendor ZEISS We utilize Open Application Development (OAD), a framework within ZEISS ZEN Blue that integrates an Iron Python script editor to automate microscopy workflows. OAD enables responsive imaging strategies, processing real-time image analysis cues from both internal algorithms (running within ZEN) and external tools such as Python scripts and ImageJ/FIJI plugins. Our primary efforts have focused on:
Targeted Imaging: Adaptive imaging routines that prioritize specific sample features.
Object Tracking: Automated tracking of dynamic structures in live-cell imaging applications.
Electron Microscopy (ZEISS SEM & Thermo Fisher TEM) ZEISS SEM systems lack an efficient API for direct hardware communication (an issue currently being addressed). As a result, we have concentrated on:
On-the-Fly Image Analysis and Quality Control: Implementing real-time image evaluation workflows to detect acquisition anomalies and improve data reliability.
Data Standardization: Developing pipelines for converting complex vendor-specific image data structures (e.g., multi-image projects with proprietary metadata) into OME-Zarr, an open, cloud-compatible image format. For TEM workflows, we emphasize data format standardization to ensure interoperability with downstream analysis tools, particularly through the adoption of OME-Zarr.
Data Management Infrastructure As of early 2025, CCI has deployed an OMERO-based image data management system that streamlines data flow across all facility microscopes. This infrastructure facilitates seamless data transfer from acquisition systems to computational analysis servers, and supports integration with users’ own storage solutions, working towards ensuring FAIR (Findable, Accessible, Interoperable, Reusable) data principles.
Key features and innovations#
The Smart Microscopy implementation at CCI integrates advanced automation strategies across diverse imaging modalities, ensuring seamless operation within a fee-for-service core facility model. Our key innovations focus on adaptive imaging, interoperability, and data standardization, enhancing both image acquisition and post-processing workflows to improve efficiency and reproducibility. Adaptive imaging and automated feedback loops enhance real-time decision-making, enabling more precise and efficient imaging workflows. Targeted imaging on light microscopes leverages machine learning-based object detection within ZEISS ZEN Blue’s Open Application Development (OAD) framework, dynamically selecting regions of interest (ROI) based on sample characteristics. This also facilitates object tracking for live-cell studies, though current research is focused on improving feedback speed for faster adaptive responses. In electron microscopy, on-the-fly image analysis for SEM workflows enables real-time quality assessment and anomaly detection, optimizing acquisition efficiency in high-throughput environments. These innovations create a more responsive, automated, and reproducible microscopy workflow, reducing manual intervention while enhancing data quality and throughput. CCI’s Smart Microscopy implementation also enhances interoperability and scalability, ensuring seamless integration across imaging modalities and research infrastructures. Standardized data conversion to OME-Zarr facilitates compatibility across light, SEM, and TEM workflows, ensuring structured data handling and long-term accessibility. Multi-software integration maintains flexibility, enabling workflows that connect ZEISS ZEN, FIJI/ImageJ, and Python-based analysis tools while preserving vendor compatibility. The deployment of an OMERO-based data management system further supports automated image transfer, metadata structuring, and remote access, streamlining data handling from acquisition to analysis. Designed for open-access imaging facilities and research environments, CCI’s Smart Microscopy solutions lower the technical barrier for advanced imaging automation, making cutting-edge workflows more accessible to a broader user base. The infrastructure is optimized for scalability and interoperability, allowing workflows to be shared, adapted, and deployed across multiple research institutions.
Contributions that could contribute to interoperability#
All Smart Microscopy developments at CCI are designed with open software and interoperability as core principles, ensuring seamless integration across imaging platforms and analysis tools. By adopting standardized data formats such as OME-Zarr and implementing OMERO-based data management, we facilitate structured, vendor-independent data handling. Our multi-software integration strategy ensures compatibility with ZEISS ZEN, FIJI/ImageJ, and Python-based analysis workflows, enabling flexible automation while maintaining vendor compatibility. These efforts lay the foundation for scalable, shareable, and adaptable imaging workflows, with interoperability remaining a key focus for future developments (see roadmap for details).
Current bottlenecks, Roadmap#
While CCI’s Smart Microscopy implementation has made significant progress in automation, interoperability, and data standardization, several challenges remain in fully realizing the potential of adaptive microscopy workflows. One of the main bottlenecks is limited access to vendor APIs, particularly for electron microscopy systems. While ZEISS light microscopes support automation through the Open Application Development (OAD) framework, EM platforms lack efficient software interfaces for direct hardware communication. This limitation prevents real-time adaptive feedback loops, requiring workarounds such as post-acquisition quality control rather than live adjustments. We are actively collaborating with industry partners to explore solutions that enable deeper automation and integration within these systems. Another challenge is the speed and efficiency of real-time feedback microscopy, particularly in live-cell imaging. While targeted imaging and object tracking are already implemented, response times remain constrained by latency in processing real-time image analysis outputs and sending commands back to the microscope. Further optimization of computational workflows and hardware-software communication is needed to improve the responsiveness of adaptive imaging, ensuring that real-time decision-making can keep pace with dynamic biological processes. In terms of data management and interoperability, we have successfully deployed an OMERO-based infrastructure, but ensuring seamless integration across different imaging modalities and user storage solutions remains an ongoing effort. The conversion of complex, vendor-specific data structures into standardized formats such as OME-Zarr is progressing, but further development is needed to refine metadata extraction, enhance compatibility with emerging image analysis pipelines, streamline large-scale data transfers, and integrate computational resources such as national CPU and GPU clusters for scalable analysis. Looking ahead, our roadmap prioritizes the development of a Python-based Smart Microscopy Workflow Manager, designed to act as a user-level interface that allows researchers to define imaging workflows independent of the underlying microscope hardware. This system will abstract communication with different microscopes—even from different vendors—ensuring a unified approach to controlling diverse imaging platforms. Additionally, it will provide flexibility in image analysis execution, seamlessly integrating processing across Java, Python, or proprietary software environments, enabling users to focus on scientific insights rather than software compatibility. By addressing these challenges, we aim to make Smart Microscopy at CCI more robust, accessible, and scalable, ensuring that our workflows continue to support cutting-edge research in imaging sciences while paving the way for broader adoption across research infrastructures.