Creating a Segmentation App Including Visualization with MONAI Deploy App SDK

This tutorial shows how to create an organ segmentation application for a PyTorch model that has been trained with MONAI, and visualize the segmentation and input images with Clara Viz integration.

When integrated with imaging networks and receiving DICOM instances from modalities and Picture Archiving and Communications System (PACS), an AI deploy application has to deal with a whole DICOM study with multiple series, whose images’ spacing may not be the same as expected by the trained model. To address these cases consistently and efficiently, MONAI Deploy Application SDK provides classes, called operators, to parse DICOM studies, select specific series with application-defined rules, and convert the selected DICOM series into domain-specific image format along with meta-data representing the pertinent DICOM attributes. The image is then further processed in the pre-processing stage to normalize spacing, orientation, intensity,etc, before pixel data as Tensors are used for inference.

During the development of an application, there is often times the need to visualize the transformed images between processing steps. Clara Viz is the perfect tool for this scenario, and a built-in operator in the App SDK makes the integration seamless and easy to implement, as we demonstrate below.

Creating Operators and connecting them in Application class

We will implement an application that consists of five Operators:

• DICOMDataLoaderOperator:

  • Input(dicom_files): a folder path (DataPath)

  • Output(dicom_study_list): a list of DICOM studies in memory (List[DICOMStudy])

• DICOMSeriesSelectorOperator:

  • Input(dicom_study_list): a list of DICOM studies in memory (List[DICOMStudy])

  • Input(selection_rules): a selection rule (Dict)

  • Output(study_selected_series_list): a DICOM series object in memory (StudySelectedSeries)

• DICOMSeriesToVolumeOperator:

  • Input(study_selected_series_list): a DICOM series object in memory (StudySelectedSeries)

  • Output(image): an image object in memory (Image)

• MonaiBundleInferenceOperator:

  • Input(image): an image object in memory (Image)

  • Output(pred): an image object in memory (Image)

• DICOMSegmentationWriterOperator:

  • Input(seg_image): a segmentation image object in memory (Image)

  • Input(study_selected_series_list): a DICOM series object in memory (StudySelectedSeries)

  • Output(dicom_seg_instance): a file path (DataPath)

Note

The DICOMSegmentationWriterOperator needs both the segmentation image as well as the original DICOM series meta-data in order to use the patient demographics and the DICOM Study level attributes.

The workflow of the application is illustrated below.

%%{init: {"theme": "base", "themeVariables": { "fontSize": "16px"}} }%% classDiagram direction TB DICOMDataLoaderOperator --|> DICOMSeriesSelectorOperator : dicom_study_list...dicom_study_list DICOMSeriesSelectorOperator --|> DICOMSeriesToVolumeOperator : study_selected_series_list...study_selected_series_list DICOMSeriesToVolumeOperator --|> MonaiBundleInferenceOperator : image...image DICOMSeriesSelectorOperator --|> DICOMSegmentationWriterOperator : study_selected_series_list...study_selected_series_list MonaiBundleInferenceOperator --|> DICOMSegmentationWriterOperator : pred...seg_image class DICOMDataLoaderOperator { <in>dicom_files : DISK dicom_study_list(out) IN_MEMORY } class DICOMSeriesSelectorOperator { <in>dicom_study_list : IN_MEMORY <in>selection_rules : IN_MEMORY study_selected_series_list(out) IN_MEMORY } class DICOMSeriesToVolumeOperator { <in>study_selected_series_list : IN_MEMORY image(out) IN_MEMORY } class MonaiBundleInferenceOperator { <in>image : IN_MEMORY pred(out) IN_MEMORY } class DICOMSegmentationWriterOperator { <in>seg_image : IN_MEMORY <in>study_selected_series_list : IN_MEMORY dicom_seg_instance(out) DISK }

Setup environment

# Install MONAI and other necessary image processing packages for the application
!python -c "import monai" || pip install --upgrade -q "monai"
!python -c "import torch" || pip install -q "torch>=1.12.0"
!python -c "import numpy" || pip install -q "numpy>=1.21.6"
!python -c "import nibabel" || pip install -q "nibabel>=3.2.1"
!python -c "import pydicom" || pip install -q "pydicom>=2.3.0"
!python -c "import highdicom" || pip install -q "highdicom>=0.18.2"
!python -c "import SimpleITK" || pip install -q "SimpleITK>=2.0.0"

# Install MONAI Deploy App SDK package
!python -c "import monai.deploy" || pip install -q "monai-deploy-app-sdk"

# Install Clara Viz package
!python -c "import clara.viz" || pip install --upgrade -q "clara-viz"

Note: you may need to restart the Jupyter kernel to use the updated packages.

Download/Extract input and model/bundle files from Google Drive

# Download the test data and MONAI bundle zip file
!pip install gdown
!gdown "https://drive.google.com/uc?id=1Uds8mEvdGNYUuvFpTtCQ8gNU97bAPCaQ"

# Clean up the destinaton folder for the input DICOM files
!rm -rf dcm

# After downloading ai_spleen_bundle_data zip file from the web browser or using gdown,
!unzip -o "ai_spleen_seg_bundle_data.zip"

# Need to copy the model.ts file to its own clean subfolder for pacakging, to workaround an issue in the Packager
models_folder = "models"
!rm -rf {models_folder} && mkdir -p {models_folder}/model && cp model.ts {models_folder}/model && ls {models_folder}/model

Requirement already satisfied: gdown in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (5.1.0)
Requirement already satisfied: beautifulsoup4 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from gdown) (4.12.3)
Requirement already satisfied: filelock in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from gdown) (3.13.4)
Requirement already satisfied: requests[socks] in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from gdown) (2.31.0)
Requirement already satisfied: tqdm in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from gdown) (4.66.2)
Requirement already satisfied: soupsieve>1.2 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from beautifulsoup4->gdown) (2.5)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from requests[socks]->gdown) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from requests[socks]->gdown) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from requests[socks]->gdown) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from requests[socks]->gdown) (2024.2.2)
Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages (from requests[socks]->gdown) (1.7.1)
Downloading...
From (original): https://drive.google.com/uc?id=1Uds8mEvdGNYUuvFpTtCQ8gNU97bAPCaQ
From (redirected): https://drive.google.com/uc?id=1Uds8mEvdGNYUuvFpTtCQ8gNU97bAPCaQ&confirm=t&uuid=c8244ec0-ca91-472f-8701-075a197bb44e
To: /home/mqin/src/monai-deploy-app-sdk/notebooks/tutorials/ai_spleen_seg_bundle_data.zip
100%|██████████████████████████████████████| 79.4M/79.4M [00:00<00:00, 99.7MB/s]
Archive:  ai_spleen_seg_bundle_data.zip
  inflating: dcm/1-001.dcm           
  inflating: dcm/1-002.dcm           
  inflating: dcm/1-003.dcm           
  inflating: dcm/1-004.dcm           
  inflating: dcm/1-005.dcm           
  inflating: dcm/1-006.dcm           
  inflating: dcm/1-007.dcm           
  inflating: dcm/1-008.dcm           
  inflating: dcm/1-009.dcm           
  inflating: dcm/1-010.dcm           
  inflating: dcm/1-011.dcm           
  inflating: dcm/1-012.dcm           
  inflating: dcm/1-013.dcm           
  inflating: dcm/1-014.dcm           
  inflating: dcm/1-015.dcm           
  inflating: dcm/1-016.dcm           
  inflating: dcm/1-017.dcm           
  inflating: dcm/1-018.dcm           
  inflating: dcm/1-019.dcm           
  inflating: dcm/1-020.dcm           
  inflating: dcm/1-021.dcm           
  inflating: dcm/1-022.dcm           
  inflating: dcm/1-023.dcm           
  inflating: dcm/1-024.dcm           
  inflating: dcm/1-025.dcm           
  inflating: dcm/1-026.dcm           
  inflating: dcm/1-027.dcm           
  inflating: dcm/1-028.dcm           
  inflating: dcm/1-029.dcm           
  inflating: dcm/1-030.dcm           
  inflating: dcm/1-031.dcm           
  inflating: dcm/1-032.dcm           
  inflating: dcm/1-033.dcm           
  inflating: dcm/1-034.dcm           
  inflating: dcm/1-035.dcm           
  inflating: dcm/1-036.dcm           
  inflating: dcm/1-037.dcm           
  inflating: dcm/1-038.dcm           
  inflating: dcm/1-039.dcm           
  inflating: dcm/1-040.dcm           
  inflating: dcm/1-041.dcm           
  inflating: dcm/1-042.dcm           
  inflating: dcm/1-043.dcm           
  inflating: dcm/1-044.dcm           
  inflating: dcm/1-045.dcm           
  inflating: dcm/1-046.dcm           
  inflating: dcm/1-047.dcm           
  inflating: dcm/1-048.dcm           
  inflating: dcm/1-049.dcm           
  inflating: dcm/1-050.dcm           
  inflating: dcm/1-051.dcm           
  inflating: dcm/1-052.dcm           
  inflating: dcm/1-053.dcm           
  inflating: dcm/1-054.dcm           
  inflating: dcm/1-055.dcm           
  inflating: dcm/1-056.dcm           
  inflating: dcm/1-057.dcm           
  inflating: dcm/1-058.dcm           
  inflating: dcm/1-059.dcm           
  inflating: dcm/1-060.dcm           
  inflating: dcm/1-061.dcm           
  inflating: dcm/1-062.dcm           
  inflating: dcm/1-063.dcm           
  inflating: dcm/1-064.dcm           
  inflating: dcm/1-065.dcm           
  inflating: dcm/1-066.dcm           
  inflating: dcm/1-067.dcm           
  inflating: dcm/1-068.dcm           
  inflating: dcm/1-069.dcm           
  inflating: dcm/1-070.dcm           
  inflating: dcm/1-071.dcm           
  inflating: dcm/1-072.dcm           
  inflating: dcm/1-073.dcm           
  inflating: dcm/1-074.dcm           
  inflating: dcm/1-075.dcm           
  inflating: dcm/1-076.dcm           
  inflating: dcm/1-077.dcm           
  inflating: dcm/1-078.dcm           
  inflating: dcm/1-079.dcm           
  inflating: dcm/1-080.dcm           
  inflating: dcm/1-081.dcm           
  inflating: dcm/1-082.dcm           
  inflating: dcm/1-083.dcm           
  inflating: dcm/1-084.dcm           
  inflating: dcm/1-085.dcm           
  inflating: dcm/1-086.dcm           
  inflating: dcm/1-087.dcm           
  inflating: dcm/1-088.dcm           
  inflating: dcm/1-089.dcm           
  inflating: dcm/1-090.dcm           
  inflating: dcm/1-091.dcm           
  inflating: dcm/1-092.dcm           
  inflating: dcm/1-093.dcm           
  inflating: dcm/1-094.dcm           
  inflating: dcm/1-095.dcm           
  inflating: dcm/1-096.dcm           
  inflating: dcm/1-097.dcm           
  inflating: dcm/1-098.dcm           
  inflating: dcm/1-099.dcm           
  inflating: dcm/1-100.dcm           
  inflating: dcm/1-101.dcm           
  inflating: dcm/1-102.dcm           
  inflating: dcm/1-103.dcm           
  inflating: dcm/1-104.dcm           
  inflating: dcm/1-105.dcm           
  inflating: dcm/1-106.dcm           
  inflating: dcm/1-107.dcm           
  inflating: dcm/1-108.dcm           
  inflating: dcm/1-109.dcm           
  inflating: dcm/1-110.dcm           
  inflating: dcm/1-111.dcm           
  inflating: dcm/1-112.dcm           
  inflating: dcm/1-113.dcm           
  inflating: dcm/1-114.dcm           
  inflating: dcm/1-115.dcm           
  inflating: dcm/1-116.dcm           
  inflating: dcm/1-117.dcm           
  inflating: dcm/1-118.dcm           
  inflating: dcm/1-119.dcm           
  inflating: dcm/1-120.dcm           
  inflating: dcm/1-121.dcm           
  inflating: dcm/1-122.dcm           
  inflating: dcm/1-123.dcm           
  inflating: dcm/1-124.dcm           
  inflating: dcm/1-125.dcm           
  inflating: dcm/1-126.dcm           
  inflating: dcm/1-127.dcm           
  inflating: dcm/1-128.dcm           
  inflating: dcm/1-129.dcm           
  inflating: dcm/1-130.dcm           
  inflating: dcm/1-131.dcm           
  inflating: dcm/1-132.dcm           
  inflating: dcm/1-133.dcm           
  inflating: dcm/1-134.dcm           
  inflating: dcm/1-135.dcm           
  inflating: dcm/1-136.dcm           
  inflating: dcm/1-137.dcm           
  inflating: dcm/1-138.dcm           
  inflating: dcm/1-139.dcm           
  inflating: dcm/1-140.dcm           
  inflating: dcm/1-141.dcm           
  inflating: dcm/1-142.dcm           
  inflating: dcm/1-143.dcm           
  inflating: dcm/1-144.dcm           
  inflating: dcm/1-145.dcm           
  inflating: dcm/1-146.dcm           
  inflating: dcm/1-147.dcm           
  inflating: dcm/1-148.dcm           
  inflating: dcm/1-149.dcm           
  inflating: dcm/1-150.dcm           
  inflating: dcm/1-151.dcm           
  inflating: dcm/1-152.dcm           
  inflating: dcm/1-153.dcm           
  inflating: dcm/1-154.dcm           
  inflating: dcm/1-155.dcm           
  inflating: dcm/1-156.dcm           
  inflating: dcm/1-157.dcm           
  inflating: dcm/1-158.dcm           
  inflating: dcm/1-159.dcm           
  inflating: dcm/1-160.dcm           
  inflating: dcm/1-161.dcm           
  inflating: dcm/1-162.dcm           
  inflating: dcm/1-163.dcm           
  inflating: dcm/1-164.dcm           
  inflating: dcm/1-165.dcm           
  inflating: dcm/1-166.dcm           
  inflating: dcm/1-167.dcm           
  inflating: dcm/1-168.dcm           
  inflating: dcm/1-169.dcm           
  inflating: dcm/1-170.dcm           
  inflating: dcm/1-171.dcm           
  inflating: dcm/1-172.dcm           
  inflating: dcm/1-173.dcm           
  inflating: dcm/1-174.dcm           
  inflating: dcm/1-175.dcm           
  inflating: dcm/1-176.dcm           
  inflating: dcm/1-177.dcm           
  inflating: dcm/1-178.dcm           
  inflating: dcm/1-179.dcm           
  inflating: dcm/1-180.dcm           
  inflating: dcm/1-181.dcm           
  inflating: dcm/1-182.dcm           
  inflating: dcm/1-183.dcm           
  inflating: dcm/1-184.dcm           
  inflating: dcm/1-185.dcm           
  inflating: dcm/1-186.dcm           
  inflating: dcm/1-187.dcm           
  inflating: dcm/1-188.dcm           
  inflating: dcm/1-189.dcm           
  inflating: dcm/1-190.dcm           
  inflating: dcm/1-191.dcm           
  inflating: dcm/1-192.dcm           
  inflating: dcm/1-193.dcm           
  inflating: dcm/1-194.dcm           
  inflating: dcm/1-195.dcm           
  inflating: dcm/1-196.dcm           
  inflating: dcm/1-197.dcm           
  inflating: dcm/1-198.dcm           
  inflating: dcm/1-199.dcm           
  inflating: dcm/1-200.dcm           
  inflating: dcm/1-201.dcm           
  inflating: dcm/1-202.dcm           
  inflating: dcm/1-203.dcm           
  inflating: dcm/1-204.dcm           
  inflating: model.ts                
model.ts

Set up environment variables

%env HOLOSCAN_INPUT_PATH dcm
%env HOLOSCAN_MODEL_PATH {models_folder}
%env HOLOSCAN_OUTPUT_PATH output
%ls $HOLOSCAN_MODEL_PATH

env: HOLOSCAN_INPUT_PATH=dcm
env: HOLOSCAN_MODEL_PATH=models
env: HOLOSCAN_OUTPUT_PATH=output
model/

Set up imports

Let’s import necessary classes/decorators to define Application and Operator.

import logging
from pathlib import Path

# Required for setting SegmentDescription attributes. Direct import as this is not part of App SDK package.
from pydicom.sr.codedict import codes

from monai.deploy.conditions import CountCondition
from monai.deploy.core import AppContext, Application
from monai.deploy.core.domain import Image
from monai.deploy.core.io_type import IOType
from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator
from monai.deploy.operators.dicom_seg_writer_operator import DICOMSegmentationWriterOperator, SegmentDescription
from monai.deploy.operators.dicom_series_selector_operator import DICOMSeriesSelectorOperator
from monai.deploy.operators.dicom_series_to_volume_operator import DICOMSeriesToVolumeOperator
from monai.deploy.operators.monai_bundle_inference_operator import (
    BundleConfigNames,
    IOMapping,
    MonaiBundleInferenceOperator,
)
from monai.deploy.operators.stl_conversion_operator import STLConversionOperator
from monai.deploy.operators.clara_viz_operator import ClaraVizOperator

Determining the Input and Output for the Model Bundle Inference Operator

The App SDK provides a MonaiBundleInferenceOperator class to perform inference with a MONAI Bundle, which is essentially a PyTorch model in TorchScript with additional metadata describing the model network and processing specification. This operator uses the MONAI utilities to parse a MONAI Bundle to automatically instantiate the objects required for input and output processing as well as inference, as such it depends on MONAI transforms, inferers, and in turn their dependencies.

Each Operator class inherits from the base Operator class, and its input/output properties are specified by using @input/@output decorators. For the MonaiBundleInferenceOperator class, the input/output need to be defined to match those of the model network, both in name and data type. For the current release, an IOMapping object is used to connect the operator input/output to those of the model network by using the same names. This is likely to change, to be automated, in the future releases once certain limitation in the App SDK is removed.

The Spleen CT Segmentation model network has a named input, called “image”, and the named output called “pred”, and both are of image type, which can all be mapped to the App SDK Image. This piece of information is typically acquired by examining the model metadata network_data_format attribute in the bundle, as seen in this [example] (https://github.com/Project-MONAI/model-zoo/blob/dev/models/spleen_ct_segmentation/configs/metadata.json).

Creating Application class

Our application class would look like below.

It defines App class, inheriting Application class.

The requirements (resource and package dependency) for the App can be specified by using @resource and @env decorators.

Objects required for DICOM parsing, series selection, pixel data conversion to volume image, model specific inference, and the AI result specific DICOM Segmentation object writers are created. The execution pipeline, as a Directed Acyclic Graph, is then constructed by connecting these objects through self.add_flow().

class AISpleenSegApp(Application):
    """Demonstrates inference with built-in MONAI Bundle inference operator with DICOM files as input/output

    This application loads a set of DICOM instances, select the appropriate series, converts the series to
    3D volume image, performs inference with the built-in MONAI Bundle inference operator, including pre-processing
    and post-processing, save the segmentation image in a DICOM Seg OID in an instance file, and optionally the
    surface mesh in STL format.

    Pertinent MONAI Bundle:
      https://github.com/Project-MONAI/model-zoo/tree/dev/models/spleen_ct_segmentation

    Execution Time Estimate:
      With a Nvidia GV100 32GB GPU, for an input DICOM Series of 515 instances, the execution time is around
      25 seconds with saving both DICOM Seg and surface mesh STL file, and 15 seconds with DICOM Seg only.
    """

    def __init__(self, *args, **kwargs):
        """Creates an application instance."""
        self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
        super().__init__(*args, **kwargs)

    def run(self, *args, **kwargs):
        # This method calls the base class to run. Can be omitted if simply calling through.
        self._logger.info(f"Begin {self.run.__name__}")
        super().run(*args, **kwargs)
        self._logger.info(f"End {self.run.__name__}")

    def compose(self):
        """Creates the app specific operators and chain them up in the processing DAG."""

        logging.info(f"Begin {self.compose.__name__}")

        app_context = Application.init_app_context({})  # Do not pass argv in Jupyter Notebook
        app_input_path = Path(app_context.input_path)
        app_output_path = Path(app_context.output_path)
        model_path = Path(app_context.model_path)

        # Create the custom operator(s) as well as SDK built-in operator(s).
        study_loader_op = DICOMDataLoaderOperator(
            self, CountCondition(self, 1), input_folder=app_input_path, name="study_loader_op"
        )
        series_selector_op = DICOMSeriesSelectorOperator(self, rules=Sample_Rules_Text, name="series_selector_op")
        series_to_vol_op = DICOMSeriesToVolumeOperator(self, name="series_to_vol_op")

        # Create the inference operator that supports MONAI Bundle and automates the inference.
        # The IOMapping labels match the input and prediction keys in the pre and post processing.
        # The model_name is optional when the app has only one model.
        # The bundle_path argument optionally can be set to an accessible bundle file path in the dev
        # environment, so when the app is packaged into a MAP, the operator can complete the bundle parsing
        # during init.

        config_names = BundleConfigNames(config_names=["inference"])  # Same as the default

        bundle_spleen_seg_op = MonaiBundleInferenceOperator(
            self,
            input_mapping=[IOMapping("image", Image, IOType.IN_MEMORY)],
            output_mapping=[IOMapping("pred", Image, IOType.IN_MEMORY)],
            app_context=app_context,
            bundle_config_names=config_names,
            bundle_path=model_path,
            name="bundle_spleen_seg_op",
        )

        # Create DICOM Seg writer providing the required segment description for each segment with
        # the actual algorithm and the pertinent organ/tissue. The segment_label, algorithm_name,
        # and algorithm_version are of DICOM VR LO type, limited to 64 chars.
        # https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html
        segment_descriptions = [
            SegmentDescription(
                segment_label="Spleen",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Spleen,
                algorithm_name="volumetric (3D) segmentation of the spleen from CT image",
                algorithm_family=codes.DCM.ArtificialIntelligence,
                algorithm_version="0.3.2",
            )
        ]

        custom_tags = {"SeriesDescription": "AI generated Seg, not for clinical use."}

        dicom_seg_writer = DICOMSegmentationWriterOperator(
            self,
            segment_descriptions=segment_descriptions,
            custom_tags=custom_tags,
            output_folder=app_output_path,
            name="dicom_seg_writer",
        )

        # Create the processing pipeline, by specifying the source and destination operators, and
        # ensuring the output from the former matches the input of the latter, in both name and type.
        self.add_flow(study_loader_op, series_selector_op, {("dicom_study_list", "dicom_study_list")})
        self.add_flow(
            series_selector_op, series_to_vol_op, {("study_selected_series_list", "study_selected_series_list")}
        )
        self.add_flow(series_to_vol_op, bundle_spleen_seg_op, {("image", "image")})
        # Note below the dicom_seg_writer requires two inputs, each coming from a source operator.
        self.add_flow(
            series_selector_op, dicom_seg_writer, {("study_selected_series_list", "study_selected_series_list")}
        )
        self.add_flow(bundle_spleen_seg_op, dicom_seg_writer, {("pred", "seg_image")})
        # Create the surface mesh STL conversion operator and add it to the app execution flow, if needed, by
        # uncommenting the following couple lines.
        stl_conversion_op = STLConversionOperator(
            self, output_file=app_output_path.joinpath("stl/spleen.stl"), name="stl_conversion_op"
        )
        self.add_flow(bundle_spleen_seg_op, stl_conversion_op, {("pred", "image")})

        # Create the ClaraViz operator and feed both the seg and input Image object to it
        viz_op = ClaraVizOperator(self, name="clara_viz_op")
        self.add_flow(series_to_vol_op, viz_op, {("image", "image")})
        self.add_flow(bundle_spleen_seg_op, viz_op, {("pred", "seg_image")})

        logging.info(f"End {self.compose.__name__}")


# This is a sample series selection rule in JSON, simply selecting CT series.
# If the study has more than 1 CT series, then all of them will be selected.
# Please see more detail in DICOMSeriesSelectorOperator.
Sample_Rules_Text = """
{
    "selections": [
        {
            "name": "CT Series",
            "conditions": {
                "StudyDescription": "(.*?)",
                "Modality": "(?i)CT",
                "SeriesDescription": "(.*?)"
            }
        }
    ]
}
"""

Executing app locally

We can execute the app in the Jupyter notebook. Note that the DICOM files of the CT Abdomen series must be present in the dcm and the Torch Script model at model.ts. Please use the actual path in your environment.

!rm -rf $HOLOSCAN_OUTPUT_PATH
AISpleenSegApp().run()

[2024-04-23 17:21:11,103] [INFO] (root) - Parsed args: Namespace(log_level=None, input=None, output=None, model=None, workdir=None, argv=[])
[2024-04-23 17:21:11,110] [INFO] (root) - AppContext object: AppContext(input_path=dcm, output_path=output, model_path=models, workdir=)
[2024-04-23 17:21:11,119] [INFO] (root) - End compose
[info] [gxf_executor.cpp:247] Creating context
[info] [gxf_executor.cpp:1672] Loading extensions from configs...
[info] [gxf_executor.cpp:1842] Activating Graph...
[info] [gxf_executor.cpp:1874] Running Graph...
[info] [gxf_executor.cpp:1876] Waiting for completion...
[2024-04-23 17:21:11,172] [INFO] (monai.deploy.operators.dicom_data_loader_operator.DICOMDataLoaderOperator) - No or invalid input path from the optional input port: None

2024-04-23 17:21:11.171 INFO  gxf/std/greedy_scheduler.cpp@191: Scheduling 10 entities

[2024-04-23 17:21:11,733] [INFO] (root) - Finding series for Selection named: CT Series
[2024-04-23 17:21:11,734] [INFO] (root) - Searching study, : 1.3.6.1.4.1.14519.5.2.1.7085.2626.822645453932810382886582736291
  # of series: 1
[2024-04-23 17:21:11,735] [INFO] (root) - Working on series, instance UID: 1.3.6.1.4.1.14519.5.2.1.7085.2626.119403521930927333027265674239
[2024-04-23 17:21:11,736] [INFO] (root) - On attribute: 'StudyDescription' to match value: '(.*?)'
[2024-04-23 17:21:11,736] [INFO] (root) -     Series attribute StudyDescription value: CT ABDOMEN W IV CONTRAST
[2024-04-23 17:21:11,737] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2024-04-23 17:21:11,737] [INFO] (root) - On attribute: 'Modality' to match value: '(?i)CT'
[2024-04-23 17:21:11,738] [INFO] (root) -     Series attribute Modality value: CT
[2024-04-23 17:21:11,738] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2024-04-23 17:21:11,739] [INFO] (root) - On attribute: 'SeriesDescription' to match value: '(.*?)'
[2024-04-23 17:21:11,739] [INFO] (root) -     Series attribute SeriesDescription value: ABD/PANC 3.0 B31f
[2024-04-23 17:21:11,740] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2024-04-23 17:21:11,740] [INFO] (root) - Selected Series, UID: 1.3.6.1.4.1.14519.5.2.1.7085.2626.119403521930927333027265674239
[2024-04-23 17:21:11,964] [INFO] (root) - Parsing from bundle_path: /home/mqin/src/monai-deploy-app-sdk/notebooks/tutorials/models/model/model.ts

[2024-04-23 17:21:24,861] [INFO] (monai.deploy.operators.stl_conversion_operator.STLConversionOperator) - Output will be saved in file output/stl/spleen.stl.
[2024-04-23 17:21:26,556] [INFO] (monai.deploy.operators.stl_conversion_operator.SpatialImage) - 3D image
[2024-04-23 17:21:26,557] [INFO] (monai.deploy.operators.stl_conversion_operator.STLConverter) - Image ndarray shape:(204, 512, 512)
/home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages/highdicom/valuerep.py:54: UserWarning: The string "C3N-00198" is unlikely to represent the intended person name since it contains only a single component. Construct a person name according to the format in described in https://dicom.nema.org/dicom/2013/output/chtml/part05/sect_6.2.html#sect_6.2.1.2, or, in pydicom 2.2.0 or later, use the pydicom.valuerep.PersonName.from_named_components() method to construct the person name correctly. If a single-component name is really intended, add a trailing caret character to disambiguate the name.
  warnings.warn(
[2024-04-23 17:21:38,046] [INFO] (highdicom.base) - copy Image-related attributes from dataset "1.3.6.1.4.1.14519.5.2.1.7085.2626.936983343951485811186213470191"
[2024-04-23 17:21:38,047] [INFO] (highdicom.base) - copy attributes of module "Specimen"
[2024-04-23 17:21:38,048] [INFO] (highdicom.base) - copy Patient-related attributes from dataset "1.3.6.1.4.1.14519.5.2.1.7085.2626.936983343951485811186213470191"
[2024-04-23 17:21:38,049] [INFO] (highdicom.base) - copy attributes of module "Patient"
[2024-04-23 17:21:38,051] [INFO] (highdicom.base) - copy attributes of module "Clinical Trial Subject"
[2024-04-23 17:21:38,052] [INFO] (highdicom.base) - copy Study-related attributes from dataset "1.3.6.1.4.1.14519.5.2.1.7085.2626.936983343951485811186213470191"
[2024-04-23 17:21:38,053] [INFO] (highdicom.base) - copy attributes of module "General Study"
[2024-04-23 17:21:38,054] [INFO] (highdicom.base) - copy attributes of module "Patient Study"
[2024-04-23 17:21:38,055] [INFO] (highdicom.base) - copy attributes of module "Clinical Trial Study"
[info] [gxf_executor.cpp:1879] Deactivating Graph...
[info] [gxf_executor.cpp:1887] Graph execution finished.
[2024-04-23 17:21:38,152] [INFO] (__main__.AISpleenSegApp) - End run

2024-04-23 17:21:38.150 INFO  gxf/std/greedy_scheduler.cpp@372: Scheduler stopped: Some entities are waiting for execution, but there are no periodic or async entities to get out of the deadlock.
2024-04-23 17:21:38.150 INFO  gxf/std/greedy_scheduler.cpp@401: Scheduler finished.

Once the application is verified inside Jupyter notebook, we can write the above Python code into Python files in an application folder.

The application folder structure would look like below:

my_app
├── __main__.py
└── app.py

# Create an application folder
!mkdir -p my_app && rm -rf my_app/*

app.py

%%writefile my_app/app.py

# Copyright 2021-2023 MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


import logging
from pathlib import Path

# Required for setting SegmentDescription attributes. Direct import as this is not part of App SDK package.
from pydicom.sr.codedict import codes

from monai.deploy.conditions import CountCondition
from monai.deploy.core import AppContext, Application
from monai.deploy.core.domain import Image
from monai.deploy.core.io_type import IOType
from monai.deploy.operators.dicom_data_loader_operator import DICOMDataLoaderOperator
from monai.deploy.operators.dicom_seg_writer_operator import DICOMSegmentationWriterOperator, SegmentDescription
from monai.deploy.operators.dicom_series_selector_operator import DICOMSeriesSelectorOperator
from monai.deploy.operators.dicom_series_to_volume_operator import DICOMSeriesToVolumeOperator
from monai.deploy.operators.monai_bundle_inference_operator import (
    BundleConfigNames,
    IOMapping,
    MonaiBundleInferenceOperator,
)
from monai.deploy.operators.stl_conversion_operator import STLConversionOperator
from monai.deploy.operators.clara_viz_operator import ClaraVizOperator


class AISpleenSegApp(Application):
    """Demonstrates inference with built-in MONAI Bundle inference operator with DICOM files as input/output

    This application loads a set of DICOM instances, select the appropriate series, converts the series to
    3D volume image, performs inference with the built-in MONAI Bundle inference operator, including pre-processing
    and post-processing, save the segmentation image in a DICOM Seg OID in an instance file, and optionally the
    surface mesh in STL format.

    Pertinent MONAI Bundle:
      https://github.com/Project-MONAI/model-zoo/tree/dev/models/spleen_ct_segmentation

    Execution Time Estimate:
      With a Nvidia GV100 32GB GPU, for an input DICOM Series of 515 instances, the execution time is around
      25 seconds with saving both DICOM Seg and surface mesh STL file, and 15 seconds with DICOM Seg only.
    """

    def __init__(self, *args, **kwargs):
        """Creates an application instance."""
        self._logger = logging.getLogger("{}.{}".format(__name__, type(self).__name__))
        super().__init__(*args, **kwargs)

    def run(self, *args, **kwargs):
        # This method calls the base class to run. Can be omitted if simply calling through.
        self._logger.info(f"Begin {self.run.__name__}")
        super().run(*args, **kwargs)
        self._logger.info(f"End {self.run.__name__}")

    def compose(self):
        """Creates the app specific operators and chain them up in the processing DAG."""

        logging.info(f"Begin {self.compose.__name__}")

        # Use command line options over environment variables to init context.
        app_context = Application.init_app_context(self.argv)
        app_input_path = Path(app_context.input_path)
        app_output_path = Path(app_context.output_path)
        model_path = Path(app_context.model_path)

        # Create the custom operator(s) as well as SDK built-in operator(s).
        study_loader_op = DICOMDataLoaderOperator(
            self, CountCondition(self, 1), input_folder=app_input_path, name="study_loader_op"
        )
        series_selector_op = DICOMSeriesSelectorOperator(self, rules=Sample_Rules_Text, name="series_selector_op")
        series_to_vol_op = DICOMSeriesToVolumeOperator(self, name="series_to_vol_op")

        # Create the inference operator that supports MONAI Bundle and automates the inference.
        # The IOMapping labels match the input and prediction keys in the pre and post processing.
        # The model_name is optional when the app has only one model.
        # The bundle_path argument optionally can be set to an accessible bundle file path in the dev
        # environment, so when the app is packaged into a MAP, the operator can complete the bundle parsing
        # during init.

        config_names = BundleConfigNames(config_names=["inference"])  # Same as the default

        bundle_spleen_seg_op = MonaiBundleInferenceOperator(
            self,
            input_mapping=[IOMapping("image", Image, IOType.IN_MEMORY)],
            output_mapping=[IOMapping("pred", Image, IOType.IN_MEMORY)],
            app_context=app_context,
            bundle_config_names=config_names,
            bundle_path=model_path,
            name="bundle_spleen_seg_op",
        )

        # Create DICOM Seg writer providing the required segment description for each segment with
        # the actual algorithm and the pertinent organ/tissue. The segment_label, algorithm_name,
        # and algorithm_version are of DICOM VR LO type, limited to 64 chars.
        # https://dicom.nema.org/medical/dicom/current/output/chtml/part05/sect_6.2.html
        segment_descriptions = [
            SegmentDescription(
                segment_label="Spleen",
                segmented_property_category=codes.SCT.Organ,
                segmented_property_type=codes.SCT.Spleen,
                algorithm_name="volumetric (3D) segmentation of the spleen from CT image",
                algorithm_family=codes.DCM.ArtificialIntelligence,
                algorithm_version="0.3.2",
            )
        ]

        custom_tags = {"SeriesDescription": "AI generated Seg, not for clinical use."}

        dicom_seg_writer = DICOMSegmentationWriterOperator(
            self,
            segment_descriptions=segment_descriptions,
            custom_tags=custom_tags,
            output_folder=app_output_path,
            name="dicom_seg_writer",
        )

        # Create the processing pipeline, by specifying the source and destination operators, and
        # ensuring the output from the former matches the input of the latter, in both name and type.
        self.add_flow(study_loader_op, series_selector_op, {("dicom_study_list", "dicom_study_list")})
        self.add_flow(
            series_selector_op, series_to_vol_op, {("study_selected_series_list", "study_selected_series_list")}
        )
        self.add_flow(series_to_vol_op, bundle_spleen_seg_op, {("image", "image")})
        # Note below the dicom_seg_writer requires two inputs, each coming from a source operator.
        self.add_flow(
            series_selector_op, dicom_seg_writer, {("study_selected_series_list", "study_selected_series_list")}
        )
        self.add_flow(bundle_spleen_seg_op, dicom_seg_writer, {("pred", "seg_image")})
        # Create the surface mesh STL conversion operator and add it to the app execution flow, if needed, by
        # uncommenting the following couple lines.
        stl_conversion_op = STLConversionOperator(
            self, output_file=app_output_path.joinpath("stl/spleen.stl"), name="stl_conversion_op"
        )
        self.add_flow(bundle_spleen_seg_op, stl_conversion_op, {("pred", "image")})

        # Create the ClaraViz operator and feed both the seg and input Image object to it
        viz_op = ClaraVizOperator(self, name="clara_viz_op")
        self.add_flow(series_to_vol_op, viz_op, {("image", "image")})
        self.add_flow(bundle_spleen_seg_op, viz_op, {("pred", "seg_image")})

        logging.info(f"End {self.compose.__name__}")


# This is a sample series selection rule in JSON, simply selecting CT series.
# If the study has more than 1 CT series, then all of them will be selected.
# Please see more detail in DICOMSeriesSelectorOperator.
Sample_Rules_Text = """
{
    "selections": [
        {
            "name": "CT Series",
            "conditions": {
                "StudyDescription": "(.*?)",
                "Modality": "(?i)CT",
                "SeriesDescription": "(.*?)"
            }
        }
    ]
}
"""

Writing my_app/app.py

if __name__ == "__main__":
    AISpleenSegApp().run()

The above lines are needed to execute the application code by using python interpreter.

__main__.py

__main__.py is needed for MONAI Application Packager to detect the main application code (app.py) when the application is executed with the application folder path (e.g., python simple_imaging_app).

%%writefile my_app/__main__.py
from app import AISpleenSegApp

if __name__ == "__main__":
    AISpleenSegApp().run()

Writing my_app/__main__.py

!ls my_app

app.py	__main__.py

This time, let’s execute the app in the command line.

Note

Since the environment variables have been set and contain the correct paths, it is not necessary to provide the command line options on running the application.

!rm -rf $HOLOSCAN_OUTPUT_PATH
!python my_app

[2024-04-23 17:21:43,227] [INFO] (root) - Parsed args: Namespace(log_level=None, input=None, output=None, model=None, workdir=None, argv=['my_app'])
[2024-04-23 17:21:43,229] [INFO] (root) - AppContext object: AppContext(input_path=dcm, output_path=output, model_path=models, workdir=)
[2024-04-23 17:21:43,230] [INFO] (root) - End compose
[info] [gxf_executor.cpp:247] Creating context
[info] [gxf_executor.cpp:1672] Loading extensions from configs...
[info] [gxf_executor.cpp:1842] Activating Graph...
[info] [gxf_executor.cpp:1874] Running Graph...
[info] [gxf_executor.cpp:1876] Waiting for completion...
2024-04-23 17:21:43.261 INFO  gxf/std/greedy_scheduler.cpp@191: Scheduling 10 entities
[2024-04-23 17:21:43,263] [INFO] (monai.deploy.operators.dicom_data_loader_operator.DICOMDataLoaderOperator) - No or invalid input path from the optional input port: None
[2024-04-23 17:21:43,601] [INFO] (root) - Finding series for Selection named: CT Series
[2024-04-23 17:21:43,601] [INFO] (root) - Searching study, : 1.3.6.1.4.1.14519.5.2.1.7085.2626.822645453932810382886582736291
  # of series: 1
[2024-04-23 17:21:43,601] [INFO] (root) - Working on series, instance UID: 1.3.6.1.4.1.14519.5.2.1.7085.2626.119403521930927333027265674239
[2024-04-23 17:21:43,601] [INFO] (root) - On attribute: 'StudyDescription' to match value: '(.*?)'
[2024-04-23 17:21:43,601] [INFO] (root) -     Series attribute StudyDescription value: CT ABDOMEN W IV CONTRAST
[2024-04-23 17:21:43,601] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2024-04-23 17:21:43,601] [INFO] (root) - On attribute: 'Modality' to match value: '(?i)CT'
[2024-04-23 17:21:43,601] [INFO] (root) -     Series attribute Modality value: CT
[2024-04-23 17:21:43,601] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2024-04-23 17:21:43,601] [INFO] (root) - On attribute: 'SeriesDescription' to match value: '(.*?)'
[2024-04-23 17:21:43,601] [INFO] (root) -     Series attribute SeriesDescription value: ABD/PANC 3.0 B31f
[2024-04-23 17:21:43,601] [INFO] (root) - Series attribute string value did not match. Try regEx.
[2024-04-23 17:21:43,601] [INFO] (root) - Selected Series, UID: 1.3.6.1.4.1.14519.5.2.1.7085.2626.119403521930927333027265674239
[2024-04-23 17:21:43,818] [INFO] (root) - Parsing from bundle_path: /home/mqin/src/monai-deploy-app-sdk/notebooks/tutorials/models/model/model.ts
Box(children=(Widget(), VBox(children=(interactive(children=(Dropdown(description='View mode', index=2, options=(('Cinematic', 'CINEMATIC'), ('Slice', 'SLICE'), ('Slice Segmentation', 'SLICE_SEGMENTATION')), value='SLICE_SEGMENTATION'), Output()), _dom_classes=('widget-interact',)), interactive(children=(Dropdown(description='Camera', options=('Top', 'Right', 'Front'), value='Top'), Output()), _dom_classes=('widget-interact',))))))
[2024-04-23 17:21:56,304] [INFO] (monai.deploy.operators.stl_conversion_operator.STLConversionOperator) - Output will be saved in file output/stl/spleen.stl.
[2024-04-23 17:21:57,924] [INFO] (monai.deploy.operators.stl_conversion_operator.SpatialImage) - 3D image
[2024-04-23 17:21:57,924] [INFO] (monai.deploy.operators.stl_conversion_operator.STLConverter) - Image ndarray shape:(204, 512, 512)
/home/mqin/src/monai-deploy-app-sdk/.venv/lib/python3.10/site-packages/highdicom/valuerep.py:54: UserWarning: The string "C3N-00198" is unlikely to represent the intended person name since it contains only a single component. Construct a person name according to the format in described in https://dicom.nema.org/dicom/2013/output/chtml/part05/sect_6.2.html#sect_6.2.1.2, or, in pydicom 2.2.0 or later, use the pydicom.valuerep.PersonName.from_named_components() method to construct the person name correctly. If a single-component name is really intended, add a trailing caret character to disambiguate the name.
  warnings.warn(
[2024-04-23 17:22:10,502] [INFO] (highdicom.base) - copy Image-related attributes from dataset "1.3.6.1.4.1.14519.5.2.1.7085.2626.936983343951485811186213470191"
[2024-04-23 17:22:10,502] [INFO] (highdicom.base) - copy attributes of module "Specimen"
[2024-04-23 17:22:10,502] [INFO] (highdicom.base) - copy Patient-related attributes from dataset "1.3.6.1.4.1.14519.5.2.1.7085.2626.936983343951485811186213470191"
[2024-04-23 17:22:10,502] [INFO] (highdicom.base) - copy attributes of module "Patient"
[2024-04-23 17:22:10,503] [INFO] (highdicom.base) - copy attributes of module "Clinical Trial Subject"
[2024-04-23 17:22:10,503] [INFO] (highdicom.base) - copy Study-related attributes from dataset "1.3.6.1.4.1.14519.5.2.1.7085.2626.936983343951485811186213470191"
[2024-04-23 17:22:10,503] [INFO] (highdicom.base) - copy attributes of module "General Study"
[2024-04-23 17:22:10,503] [INFO] (highdicom.base) - copy attributes of module "Patient Study"
[2024-04-23 17:22:10,503] [INFO] (highdicom.base) - copy attributes of module "Clinical Trial Study"
2024-04-23 17:22:10.588 INFO  gxf/std/greedy_scheduler.cpp@372: Scheduler stopped: Some entities are waiting for execution, but there are no periodic or async entities to get out of the deadlock.
2024-04-23 17:22:10.588 INFO  gxf/std/greedy_scheduler.cpp@401: Scheduler finished.
[info] [gxf_executor.cpp:1879] Deactivating Graph...
[info] [gxf_executor.cpp:1887] Graph execution finished.
[2024-04-23 17:22:10,590] [INFO] (app.AISpleenSegApp) - End run

!ls output

1.2.826.0.1.3680043.10.511.3.28732173401945631043953175363609741.dcm  stl

Packaging app

Clara-Viz operators present in an application are used for interactive visualization, so the application shall not be packaged.

Creating a Segmentation App Including Including Visualization with Clara-Viz Creating a Segmentation App Supporting Multiple Models