Event handlers

Model checkpoint loader

class monai.handlers.CheckpointLoader(load_path, load_dict, name=None)

CheckpointLoader acts as an Ignite handler to load checkpoint data from file. It can load variables for network, optimizer, lr_scheduler, etc. If saving checkpoint after torch.nn.DataParallel, need to save model.module instead as PyTorch recommended and then use this loader to load the model.

Parameters

• load_path (str) – the file path of checkpoint, it should be a PyTorch pth file.

• load_dict (dict) –

  target objects that load checkpoint to. examples:

  {'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler}
  

• name (Optional[str]) – identifier of logging.logger to use, if None, defaulting to engine.logger.

Model checkpoint saver

class monai.handlers.CheckpointSaver(save_dir, save_dict, name=None, file_prefix='', save_final=False, save_key_metric=False, key_metric_name=None, key_metric_n_saved=1, epoch_level=True, save_interval=0, n_saved=None)

CheckpointSaver acts as an Ignite handler to save checkpoint data into files. It supports to save according to metrics result, epoch number, iteration number and last model or exception.

Parameters

• save_dir (str) – the target directory to save the checkpoints.

• save_dict (dict) –

  source objects that save to the checkpoint. examples:

  {'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler}
  

• name (Optional[str]) – identifier of logging.logger to use, if None, defaulting to engine.logger.

• file_prefix (str) – prefix for the filenames to which objects will be saved.

• save_final (bool) – whether to save checkpoint or session at final iteration or exception.

• save_key_metric (bool) – whether to save checkpoint or session when the value of key_metric is higher than all the previous values during training.keep 4 decimal places of metric, checkpoint name is: {file_prefix}_key_metric=0.XXXX.pth.

• key_metric_name (Optional[str]) – the name of key_metric in ignite metrics dictionary. if None, use engine.state.key_metric instead.

• key_metric_n_saved (int) – save top N checkpoints or sessions, sorted by the value of key metric in descending order.

• epoch_level (bool) – save checkpoint during training for every N epochs or every N iterations. True is epoch level, False is iteration level.

• save_interval (int) – save checkpoint every N epochs, default is 0 to save no checkpoint.

• n_saved (Optional[int]) – save latest N checkpoints of epoch level or iteration level, ‘None’ is to save all.

Note

CheckpointHandler can be used during training, validation or evaluation. example of saved files:

• checkpoint_iteration=400.pth

• checkpoint_iteration=800.pth

• checkpoint_epoch=1.pth

• checkpoint_final_iteration=1000.pth

• checkpoint_key_metric=0.9387.pth

completed(engine)

Callback for train or validation/evaluation completed Event. Save final checkpoint if configure save_final is True.

Return type

None

exception_raised(engine, e)

Callback for train or validation/evaluation exception raised Event. Save current data as final checkpoint if configure save_final is True.

Return type

None

interval_completed(engine)

Callback for train epoch/iteration completed Event. Save checkpoint if configure save_interval = N

Return type

None

metrics_completed(engine)

Callback to compare metrics and save models in train or validation when epoch completed.

Return type

None

CSV saver

class monai.handlers.ClassificationSaver(output_dir='./', filename='predictions.csv', overwrite=True, batch_transform=<function ClassificationSaver.<lambda>>, output_transform=<function ClassificationSaver.<lambda>>, name=None)

Event handler triggered on completing every iteration to save the classification predictions as CSV file.

Parameters

• output_dir (str) – output CSV file directory.

• filename (str) – name of the saved CSV file name.

• overwrite (bool) – whether to overwriting existing CSV file content. If we are not overwriting, then we check if the results have been previously saved, and load them to the prediction_dict.

• batch_transform (Callable) – a callable that is used to transform the ignite.engine.batch into expected format to extract the meta_data dictionary.

• output_transform (Callable) – a callable that is used to transform the ignite.engine.output into the form expected model prediction data. The first dimension of this transform’s output will be treated as the batch dimension. Each item in the batch will be saved individually.

• name (Optional[str]) – identifier of logging.logger to use, defaulting to engine.logger.

Mean Dice metrics handler

class monai.handlers.MeanDice(include_background=True, to_onehot_y=False, mutually_exclusive=False, sigmoid=False, logit_thresh=0.5, output_transform=<function MeanDice.<lambda>>, device=None)

Computes Dice score metric from full size Tensor and collects average over batch, class-channels, iterations.

Parameters

• include_background (bool) – whether to include dice computation on the first channel of the predicted output. Defaults to True.

• to_onehot_y (bool) – whether to convert the output prediction into the one-hot format. Defaults to False.

• mutually_exclusive (bool) – if True, the output prediction will be converted into a binary matrix using a combination of argmax and to_onehot. Defaults to False.

• sigmoid (bool) – whether to add sigmoid function to the output prediction before computing Dice. Defaults to False.

• logit_thresh (float) – the threshold value to round value to 0.0 and 1.0. Defaults to None (no thresholding).

• output_transform (Callable) – transform the ignite.engine.state.output into [y_pred, y] pair.

• device (torch.device) – device specification in case of distributed computation usage.

See also

monai.metrics.meandice.compute_meandice()

compute()

Computes the metric based on it’s accumulated state.

This is called at the end of each epoch.

Returns

the actual quantity of interest.

Return type

Any

Raises

NotComputableError – raised when the metric cannot be computed.

reset()

Resets the metric to it’s initial state.

This is called at the start of each epoch.

update(output)

Updates the metric’s state using the passed batch output.

This is called once for each batch.

Parameters

output (Sequence[Union[Tensor, dict]]) – the is the output from the engine’s process function.

ROC AUC metrics handler

class monai.handlers.ROCAUC(to_onehot_y=False, softmax=False, average=<Average.MACRO: 'macro'>, output_transform=<function ROCAUC.<lambda>>, device=None)

Computes Area Under the Receiver Operating Characteristic Curve (ROC AUC). accumulating predictions and the ground-truth during an epoch and applying compute_roc_auc.

Parameters

• to_onehot_y (bool) – whether to convert y into the one-hot format. Defaults to False.

• softmax (bool) – whether to add softmax function to y_pred before computation. Defaults to False.

• average (Union[Average, str]) –

  {"macro", "weighted", "micro", "none"} Type of averaging performed if not binary classification. Defaults to "macro".

  • "macro": calculate metrics for each label, and find their unweighted mean.

    This does not take label imbalance into account.

  • "weighted": calculate metrics for each label, and find their average,

    weighted by support (the number of true instances for each label).

  • "micro": calculate metrics globally by considering each element of the label

    indicator matrix as a label.

  • "none": the scores for each class are returned.

• output_transform (Callable) – a callable that is used to transform the Engine process_function output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs.

• device (Union[str, device, None]) – device specification in case of distributed computation usage.

Note

ROCAUC expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values.

compute()

Computes the metric based on it’s accumulated state.

This is called at the end of each epoch.

Returns

the actual quantity of interest.

Return type

Any

Raises

NotComputableError – raised when the metric cannot be computed.

reset()

Resets the metric to it’s initial state.

This is called at the start of each epoch.

Return type

None

update(output)

Updates the metric’s state using the passed batch output.

This is called once for each batch.

Parameters

output (Sequence[Tensor]) – the is the output from the engine’s process function.

Metric logger

class monai.handlers.MetricLogger(loss_transform=<function MetricLogger.<lambda>>, metric_transform=<function MetricLogger.<lambda>>)

Segmentation saver

class monai.handlers.SegmentationSaver(output_dir='./', output_postfix='seg', output_ext='.nii.gz', resample=True, mode='nearest', padding_mode=<GridSamplePadMode.BORDER: 'border'>, scale=None, dtype=None, batch_transform=<function SegmentationSaver.<lambda>>, output_transform=<function SegmentationSaver.<lambda>>, name=None)

Event handler triggered on completing every iteration to save the segmentation predictions into files.

Parameters

• output_dir (str) – output image directory.

• output_postfix (str) – a string appended to all output file names.

• output_ext (str) – output file extension name.

• resample (bool) – whether to resample before saving the data array.

• mode (Union[GridSampleMode, InterpolateMode, str]) –

  This option is used when resample = True. Defaults to "nearest".

  • NIfTI files {"bilinear", "nearest"}

    Interpolation mode to calculate output values. See also: https://pytorch.org/docs/stable/nn.functional.html#grid-sample

  • PNG files {"nearest", "linear", "bilinear", "bicubic", "trilinear", "area"}

    The interpolation mode. See also: https://pytorch.org/docs/stable/nn.functional.html#interpolate

• padding_mode (Union[GridSamplePadMode, str]) –

  This option is used when resample = True. Defaults to "border".

  • NIfTI files {"zeros", "border", "reflection"}

    Padding mode for outside grid values. See also: https://pytorch.org/docs/stable/nn.functional.html#grid-sample

  • PNG files

    This option is ignored.

• scale (255, 65535) – postprocess data by clipping to [0, 1] and scaling [0, 255] (uint8) or [0, 65535] (uint16). Default is None to disable scaling. It’s used for PNG format only.

• dtype (np.dtype, optional) – convert the image data to save to this data type. If None, keep the original type of data. It’s used for Nifti format only.

• batch_transform (Callable) – a callable that is used to transform the ignite.engine.batch into expected format to extract the meta_data dictionary.

• output_transform (Callable) – a callable that is used to transform the ignite.engine.output into the form expected image data. The first dimension of this transform’s output will be treated as the batch dimension. Each item in the batch will be saved individually.

• name (Optional[str]) – identifier of logging.logger to use, defaulting to engine.logger.

Training stats handler

class monai.handlers.StatsHandler(epoch_print_logger=None, iteration_print_logger=None, output_transform=<function StatsHandler.<lambda>>, global_epoch_transform=<function StatsHandler.<lambda>>, name=None, tag_name='Loss', key_var_format='{}: {:.4f} ', logger_handler=None)

StatsHandler defines a set of Ignite Event-handlers for all the log printing logics. It’s can be used for any Ignite Engine(trainer, validator and evaluator). And it can support logging for epoch level and iteration level with pre-defined loggers.

Default behaviors:

• When EPOCH_COMPLETED, logs engine.state.metrics using self.logger.

• When ITERATION_COMPLETED, logs self.output_transform(engine.state.output) using self.logger.

Parameters

• epoch_print_logger (Optional[Callable]) – customized callable printer for epoch level logging. Must accept parameter “engine”, use default printer if None.

• iteration_print_logger (Optional[Callable]) – customized callable printer for iteration level logging. Must accept parameter “engine”, use default printer if None.

• output_transform (Callable) – a callable that is used to transform the ignite.engine.output into a scalar to print, or a dictionary of {key: scalar}. In the latter case, the output string will be formatted as key: value. By default this value logging happens when every iteration completed.

• global_epoch_transform (Callable) – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to print synced epoch number with the trainer engine.

• name – identifier of logging.logger to use, defaulting to engine.logger.

• tag_name (str) – when iteration output is a scalar, tag_name is used to print tag_name: scalar_value to logger. Defaults to 'Loss'.

• key_var_format (str) – a formatting string to control the output string format of key: value.

• logger_handler (logging.handler) – add additional handler to handle the stats data: save to file, etc. add existing python logging handlers: https://docs.python.org/3/library/logging.handlers.html

attach(engine)

Register a set of Ignite Event-Handlers to a specified Ignite engine.

Parameters

engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

epoch_completed(engine)

Handler for train or validation/evaluation epoch completed Event. Print epoch level log, default values are from Ignite state.metrics dict.

Parameters

engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

exception_raised(engine, e)

Handler for train or validation/evaluation exception raised Event. Print the exception information and traceback.

Parameters

• engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

• e (Exception) – the exception caught in Ignite during engine.run().

iteration_completed(engine)

Handler for train or validation/evaluation iteration completed Event. Print iteration level log, default values are from Ignite state.logs dict.

Parameters

engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

Tensorboard handler

class monai.handlers.TensorBoardStatsHandler(summary_writer=None, log_dir='./runs', epoch_event_writer=None, iteration_event_writer=None, output_transform=<function TensorBoardStatsHandler.<lambda>>, global_epoch_transform=<function TensorBoardStatsHandler.<lambda>>, tag_name='Loss')

TensorBoardStatsHandler defines a set of Ignite Event-handlers for all the TensorBoard logics. It’s can be used for any Ignite Engine(trainer, validator and evaluator). And it can support both epoch level and iteration level with pre-defined TensorBoard event writer. The expected data source is Ignite engine.state.output and engine.state.metrics.

Default behaviors:

• When EPOCH_COMPLETED, write each dictionary item in engine.state.metrics to TensorBoard.

• When ITERATION_COMPLETED, write each dictionary item in self.output_transform(engine.state.output) to TensorBoard.

Parameters

• summary_writer (SummaryWriter) – user can specify TensorBoard SummaryWriter, default to create a new writer.

• log_dir (str) – if using default SummaryWriter, write logs to this directory, default is ./runs.

• epoch_event_writer (Optional[Callable]) – customized callable TensorBoard writer for epoch level. Must accept parameter “engine” and “summary_writer”, use default event writer if None.

• iteration_event_writer (Optional[Callable]) – customized callable TensorBoard writer for iteration level. Must accept parameter “engine” and “summary_writer”, use default event writer if None.

• output_transform (Callable) – a callable that is used to transform the ignite.engine.output into a scalar to plot, or a dictionary of {key: scalar}. In the latter case, the output string will be formatted as key: value. By default this value plotting happens when every iteration completed.

• global_epoch_transform (Callable) – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to use trainer engines epoch number when plotting epoch vs metric curves.

• tag_name (str) – when iteration output is a scalar, tag_name is used to plot, defaults to 'Loss'.

attach(engine)

Register a set of Ignite Event-Handlers to a specified Ignite engine.

Parameters

engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

epoch_completed(engine)

Handler for train or validation/evaluation epoch completed Event. Write epoch level events, default values are from Ignite state.metrics dict.

Parameters

engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

iteration_completed(engine)

Handler for train or validation/evaluation iteration completed Event. Write iteration level events, default values are from Ignite state.logs dict.

Parameters

engine (Engine) – Ignite Engine, it can be a trainer, validator or evaluator.

LR Schedule handler

class monai.handlers.LrScheduleHandler(lr_scheduler, print_lr=True, name=None, epoch_level=True, step_transform=<function LrScheduleHandler.<lambda>>)

Ignite handler to update the Learning Rate based on PyTorch LR scheduler.

Parameters

• lr_scheduler (torch.optim.lr_scheduler) – typically, lr_scheduler should be PyTorch lr_scheduler object. If customized version, must have step and get_last_lr methods.

• print_lr (bool) – whether to print out the latest learning rate with logging.

• name (Optional[str]) – identifier of logging.logger to use, if None, defaulting to engine.logger.

• epoch_level (bool) – execute lr_scheduler.step() after every epoch or every iteration. True is epoch level, False is iteration level.

• step_transform (Callable) – a callable that is used to transform the information from engine to expected input data of lr_scheduler.step() function if necessary.

Raises

ValueError – argument step_transform must be a callable.

Validation handler

class monai.handlers.ValidationHandler(validator, interval, epoch_level=True)

Attach validator to the trainer engine in Ignite. It can support to execute validation every N epochs or every N iterations.

Parameters

• validator (Evaluator) – run the validator when trigger validation, suppose to be Evaluator.

• interval (int) – do validation every N epochs or every N iterations during training.

• epoch_level (bool) – execute validation every N epochs or N iterations. True is epoch level, False is iteration level.

Raises

ValueError – validator must be Evaluator ignite engine.