Event handlers#
Model checkpoint loader#
- class monai.handlers.CheckpointLoader(load_path, load_dict, name=None, map_location=None, strict=True, strict_shape=True)[source]#
CheckpointLoader acts as an Ignite handler to load checkpoint data from file. It can load variables for network, optimizer, lr_scheduler, etc. If saving checkpoint after torch.nn.DataParallel, need to save model.module instead as PyTorch recommended and then use this loader to load the model.
Usage example:
trainer = SupervisedTrainer(...) save_dict = { "trainer": trainer, "net": network, "opt": optimizer, "lr": lr_scheduler, } map_location = "cuda:0" # checkpoint needs to have same save_dict for this to work handler = CheckpointLoader(load_path="/test/checkpoint.pt", load_dict=save_dict, map_location=map_location, strict=True) handler(trainer) # Trainer now has the same state as stored, including the number of epochs and iterations completed # so you can resume an interrupted training at the place where it left
- Parameters:
load_path – the file path of checkpoint, it should be a PyTorch pth file.
load_dict –
target objects that load checkpoint to. examples:
{'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler}
name – identifier of logging.logger to use, if None, defaulting to
engine.logger
.map_location – when loading the module for distributed training/evaluation, need to provide an appropriate map_location argument to prevent a process to step into others’ devices. If map_location is missing, torch.load will first load the module to CPU and then copy each parameter to where it was saved, which would result in all processes on the same machine using the same set of devices.
strict – whether to strictly enforce that the keys and data shape in the state_dict of every item of load_dict match the state_dict of the corresponding items of checkpoint, default to True.
strict_shape – whether to enforce the data shape of the matched layers in the checkpoint, if `False, it will skip the layers that have different data shape with checkpoint content, and ignore the strict arg. this can be useful advanced feature for transfer learning. users should totally understand which layers will have different shape. default to True.
- Note: if strict_shape=False, will only load checkpoint for torch.nn.Module and skip other
items in the load_dict. For example, if the shape of some layers in current model can’t match the checkpoint, the parameter_group of current optimizer may also can’t match the checkpoint, so skip loading checkpoint for optimizer.
For more details about loading checkpoint, please refer to: https://pytorch.org/ignite/v0.4.5/generated/ignite.handlers.checkpoint.Checkpoint.html #ignite.handlers.checkpoint.Checkpoint.load_objects. https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict.
Model checkpoint saver#
- class monai.handlers.CheckpointSaver(save_dir, save_dict, name=None, file_prefix='', save_final=False, final_filename=None, save_key_metric=False, key_metric_name=None, key_metric_n_saved=1, key_metric_filename=None, key_metric_save_state=False, key_metric_greater_or_equal=False, key_metric_negative_sign=False, epoch_level=True, save_interval=0, n_saved=None)[source]#
CheckpointSaver acts as an Ignite handler to save checkpoint data into files. It supports to save according to metrics result, epoch number, iteration number and last model or exception.
- Parameters:
save_dir – the target directory to save the checkpoints.
save_dict –
source objects that save to the checkpoint. examples:
{'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler}
name – identifier of logging.logger to use, if None, defaulting to
engine.logger
.file_prefix – prefix for the filenames to which objects will be saved.
save_final – whether to save checkpoint or session at final iteration or exception. If checkpoints are to be saved when an exception is raised, put this handler before StatsHandler in the handler list, because the logic with Ignite can only trigger the first attached handler for EXCEPTION_RAISED event.
final_filename – set a fixed filename to save the final model if save_final=True. If None, default to checkpoint_final_iteration=N.pt.
save_key_metric – whether to save checkpoint or session when the value of key_metric is higher than all the previous values during training.keep 4 decimal places of metric, checkpoint name is: {file_prefix}_key_metric=0.XXXX.pth.
key_metric_name – the name of key_metric in ignite metrics dictionary. If None, use engine.state.key_metric instead.
key_metric_n_saved – save top N checkpoints or sessions, sorted by the value of key metric in descending order.
key_metric_filename – set a fixed filename to set the best metric model, if not None, key_metric_n_saved should be 1 and only keep the best metric model.
key_metric_save_state – whether to save the tracking list of key metric in the checkpoint file. if True, then will save an object in the checkpoint file with key checkpointer to be consistent with the include_self arg of Checkpoint in ignite: https://pytorch.org/ignite/v0.4.5/generated/ignite.handlers.checkpoint.Checkpoint.html. typically, it’s used to resume training and compare current metric with previous N values.
key_metric_greater_or_equal – if True, the latest equally scored model is stored. Otherwise, save the first equally scored model. default to False.
key_metric_negative_sign – whether adding a negative sign to the metric score to compare metrics, because for error-like metrics, smaller is better(objects with larger score are retained). default to False.
epoch_level – save checkpoint during training for every N epochs or every N iterations. True is epoch level, False is iteration level.
save_interval – save checkpoint every N epochs, default is 0 to save no checkpoint.
n_saved – save latest N checkpoints of epoch level or iteration level, ‘None’ is to save all.
Note
CheckpointHandler can be used during training, validation or evaluation. example of saved files:
checkpoint_iteration=400.pt
checkpoint_iteration=800.pt
checkpoint_epoch=1.pt
checkpoint_final_iteration=1000.pt
checkpoint_key_metric=0.9387.pt
- attach(engine)[source]#
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- completed(engine)[source]#
Callback for train or validation/evaluation completed Event. Save final checkpoint if configure save_final is True.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- exception_raised(engine, e)[source]#
Callback for train or validation/evaluation exception raised Event. Save current data as final checkpoint if configure save_final is True. This callback may be skipped because the logic with Ignite can only trigger the first attached handler for EXCEPTION_RAISED event.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.e (
Exception
) – the exception caught in Ignite during engine.run().
- Return type:
None
- interval_completed(engine)[source]#
Callback for train epoch/iteration completed Event. Save checkpoint if configure save_interval = N
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- load_state_dict(state_dict)[source]#
Utility to resume the internal state of key metric tracking list if configured to save checkpoints based on the key metric value. Note to set key_metric_save_state=True when saving the previous checkpoint.
Example:
CheckpointSaver( ... save_key_metric=True, key_metric_save_state=True, # config to also save the state of this saver ).attach(engine) engine.run(...) # resumed training with a new CheckpointSaver saver = CheckpointSaver(save_key_metric=True, ...) # load the previous key metric tracking list into saver CheckpointLoader("/test/model.pt"), {"checkpointer": saver}).attach(engine)
- Return type:
None
Metrics saver#
- class monai.handlers.MetricsSaver(save_dir, metrics='*', metric_details=None, batch_transform=<function MetricsSaver.<lambda>>, summary_ops=None, save_rank=0, delimiter=', ', output_type='csv')[source]#
ignite handler to save metrics values and details into expected files.
- Parameters:
save_dir – directory to save the metrics and metric details.
metrics – expected final metrics to save into files, can be: None, “*” or list of strings. None - don’t save any metrics into files. “*” - save all the existing metrics in engine.state.metrics dict into separate files. list of strings - specify the expected metrics to save. default to “*” to save all the metrics into metrics.csv.
metric_details – expected metric details to save into files, the data comes from engine.state.metric_details, which should be provided by different Metrics, typically, it’s some intermediate values in metric computation. for example: mean dice of every channel of every image in the validation dataset. it must contain at least 2 dims: (batch, classes, …), if not, will unsqueeze to 2 dims. this arg can be: None, “*” or list of strings. None - don’t save any metric_details into files. “*” - save all the existing metric_details in engine.state.metric_details dict into separate files. list of strings - specify the metric_details of expected metrics to save. if not None, every metric_details array will save a separate {metric name}_raw.csv file.
batch_transform – a callable that is used to extract the meta_data dictionary of the input images from ignite.engine.state.batch if saving metric details. the purpose is to get the input filenames from the meta_data and store with metric details together. engine.state and batch_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
summary_ops –
expected computation operations to generate the summary report. it can be: None, “*” or list of strings, default to None. None - don’t generate summary report for every expected metric_details. “*” - generate summary report for every metric_details with all the supported operations. list of strings - generate summary report for every metric_details with specified operations, they should be within list: [“mean”, “median”, “max”, “min”, “<int>percentile”, “std”, “notnans”]. the number in “<int>percentile” should be [0, 100], like: “15percentile”. default: “90percentile”. for more details, please check: https://numpy.org/doc/stable/reference/generated/numpy.nanpercentile.html. note that: for the overall summary, it computes nanmean of all classes for each image first, then compute summary. example of the generated summary report:
class mean median max 5percentile 95percentile notnans class0 6.0000 6.0000 7.0000 5.1000 6.9000 2.0000 class1 6.0000 6.0000 6.0000 6.0000 6.0000 1.0000 mean 6.2500 6.2500 7.0000 5.5750 6.9250 2.0000
save_rank – only the handler on specified rank will save to files in multi-gpus validation, default to 0.
delimiter – the delimiter character in the saved file, default to “,” as the default output type is csv. to be consistent with: https://docs.python.org/3/library/csv.html#csv.Dialect.delimiter.
output_type – expected output file type, supported types: [“csv”], default to “csv”.
CSV saver#
- class monai.handlers.ClassificationSaver(output_dir='./', filename='predictions.csv', delimiter=', ', overwrite=True, batch_transform=<function ClassificationSaver.<lambda>>, output_transform=<function ClassificationSaver.<lambda>>, name=None, save_rank=0, saver=None)[source]#
Event handler triggered on completing every iteration to save the classification predictions as CSV file. If running in distributed data parallel, only saves CSV file in the specified rank.
- __init__(output_dir='./', filename='predictions.csv', delimiter=', ', overwrite=True, batch_transform=<function ClassificationSaver.<lambda>>, output_transform=<function ClassificationSaver.<lambda>>, name=None, save_rank=0, saver=None)[source]#
- Parameters:
output_dir – if saver=None, output CSV file directory.
filename – if saver=None, name of the saved CSV file name.
delimiter – the delimiter character in the saved file, default to “,” as the default output type is csv. to be consistent with: https://docs.python.org/3/library/csv.html#csv.Dialect.delimiter.
overwrite – if saver=None, whether to overwriting existing file content, if True, will clear the file before saving. otherwise, will append new content to the file.
batch_transform – a callable that is used to extract the meta_data dictionary of the input images from ignite.engine.state.batch. the purpose is to get the input filenames from the meta_data and store with classification results together. engine.state and batch_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
output_transform – a callable that is used to extract the model prediction data from ignite.engine.state.output. the first dimension of its output will be treated as the batch dimension. each item in the batch will be saved individually. engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
name – identifier of logging.logger to use, defaulting to engine.logger.
save_rank – only the handler on specified rank will save to CSV file in multi-gpus validation, default to 0.
saver – the saver instance to save classification results, if None, create a CSVSaver internally. the saver must provide save_batch(batch_data, meta_data) and finalize() APIs.
Ignite Metric Handler#
- class monai.handlers.IgniteMetricHandler(metric_fn=None, loss_fn=None, output_transform=<function IgniteMetricHandler.<lambda>>, save_details=True, reduction=mean, get_not_nans=False)[source]#
Base Metric class based on ignite event handler mechanism. The input prediction or label data can be a PyTorch Tensor or numpy array with batch dim and channel dim, or a list of PyTorch Tensor or numpy array without batch dim.
- Parameters:
metric_fn – callable function or class to compute raw metric results after every iteration. expect to return a Tensor with shape (batch, channel, …) or tuple (Tensor, not_nans).
loss_fn – A torch _Loss function which is used to generate the LossMetric
output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean_dice of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
reduction – Argument for the LossMetric, look there for details
get_not_nans – Argument for the LossMetric, look there for details
- attach(engine, name)[source]#
Attaches current metric to provided engine. On the end of engine’s run, engine.state.metrics dictionary will contain computed metric’s value under provided name.
- Parameters:
engine (
Engine
) – the engine to which the metric must be attached.name (
str
) – the name of the metric to attach.
- Return type:
None
- compute()[source]#
- Raises:
NotComputableError – When
compute
is called before anupdate
occurs.- Return type:
Any
- reset()#
Resets the metric to it’s initial state.
By default, this is called at the start of each epoch.
- Return type:
None
- update(output)#
- Parameters:
output (
Sequence
[Tensor
]) – sequence with contents [y_pred, y].- Raises:
ValueError – When
output
length is not 2. metric_fn can only support y_pred and y.- Return type:
None
Mean Dice metrics handler#
- class monai.handlers.MeanDice(include_background=True, reduction=mean, num_classes=None, output_transform=<function MeanDice.<lambda>>, save_details=True, return_with_label=False)[source]#
Computes Dice score metric from full size Tensor and collects average over batch, class-channels, iterations.
- __init__(include_background=True, reduction=mean, num_classes=None, output_transform=<function MeanDice.<lambda>>, save_details=True, return_with_label=False)[source]#
- Parameters:
include_background – whether to include dice computation on the first channel of the predicted output. Defaults to True.
reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.num_classes – number of input channels (always including the background). When this is None,
y_pred.shape[1]
will be used. This option is useful when bothy_pred
andy
are single-channel class indices and the number of classes is not automatically inferred from data.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean dice of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
return_with_label – whether to return the metrics with label, only works when reduction is “mean_batch”. If True, use “label_{index}” as the key corresponding to C channels; if ‘include_background’ is True, the index begins at “0”, otherwise at “1”. It can also take a list of label names. The outcome will then be returned as a dictionary.
See also
monai.metrics.meandice.compute_dice()
Mean IoU metric handler#
- class monai.handlers.MeanIoUHandler(include_background=True, reduction=mean, output_transform=<function MeanIoUHandler.<lambda>>, save_details=True)[source]#
Computes IoU score metric from full size Tensor and collects average over batch, class-channels, iterations.
- __init__(include_background=True, reduction=mean, output_transform=<function MeanIoUHandler.<lambda>>, save_details=True)[source]#
- Parameters:
include_background – whether to include iou computation on the first channel of the predicted output. Defaults to True.
reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean iou of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.meaniou.compute_iou()
ROC AUC metrics handler#
- class monai.handlers.ROCAUC(average=macro, output_transform=<function ROCAUC.<lambda>>)[source]#
Computes Area Under the Receiver Operating Characteristic Curve (ROC AUC). accumulating predictions and the ground-truth during an epoch and applying compute_roc_auc.
- Parameters:
average –
{
"macro"
,"weighted"
,"micro"
,"none"
} Type of averaging performed if not binary classification. Defaults to"macro"
."macro"
: calculate metrics for each label, and find their unweighted mean.This does not take label imbalance into account.
"weighted"
: calculate metrics for each label, and find their average,weighted by support (the number of true instances for each label).
"micro"
: calculate metrics globally by considering each element of the labelindicator matrix as a label.
"none"
: the scores for each class are returned.
output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
Note
ROCAUC expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values.
Confusion matrix metrics handler#
- class monai.handlers.ConfusionMatrix(include_background=True, metric_name='hit_rate', compute_sample=False, reduction=mean, output_transform=<function ConfusionMatrix.<lambda>>, save_details=True)[source]#
Compute confusion matrix related metrics from full size Tensor and collects average over batch, class-channels, iterations.
- __init__(include_background=True, metric_name='hit_rate', compute_sample=False, reduction=mean, output_transform=<function ConfusionMatrix.<lambda>>, save_details=True)[source]#
- Parameters:
include_background – whether to include metric computation on the first channel of the predicted output. Defaults to True.
metric_name – [
"sensitivity"
,"specificity"
,"precision"
,"negative predictive value"
,"miss rate"
,"fall out"
,"false discovery rate"
,"false omission rate"
,"prevalence threshold"
,"threat score"
,"accuracy"
,"balanced accuracy"
,"f1 score"
,"matthews correlation coefficient"
,"fowlkes mallows index"
,"informedness"
,"markedness"
] Some of the metrics have multiple aliases (as shown in the wikipedia page aforementioned), and you can also input those names instead.compute_sample – when reducing, if
True
, each sample’s metric will be computed based on each confusion matrix first. ifFalse
, compute reduction on the confusion matrices first, defaults toFalse
.reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: TP/TN/FP/FN of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.confusion_matrix()
Hausdorff distance metrics handler#
- class monai.handlers.HausdorffDistance(include_background=False, distance_metric='euclidean', percentile=None, directed=False, reduction=mean, output_transform=<function HausdorffDistance.<lambda>>, save_details=True)[source]#
Computes Hausdorff distance from full size Tensor and collects average over batch, class-channels, iterations.
- __init__(include_background=False, distance_metric='euclidean', percentile=None, directed=False, reduction=mean, output_transform=<function HausdorffDistance.<lambda>>, save_details=True)[source]#
- Parameters:
include_background – whether to include distance computation on the first channel of the predicted output. Defaults to
False
.distance_metric – : [
"euclidean"
,"chessboard"
,"taxicab"
] the metric used to compute surface distance. Defaults to"euclidean"
.percentile – an optional float number between 0 and 100. If specified, the corresponding percentile of the Hausdorff Distance rather than the maximum result will be achieved. Defaults to
None
.directed – whether to calculate directed Hausdorff distance. Defaults to
False
.reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: hausdorff distance of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
Surface distance metrics handler#
- class monai.handlers.SurfaceDistance(include_background=False, symmetric=False, distance_metric='euclidean', reduction=mean, output_transform=<function SurfaceDistance.<lambda>>, save_details=True)[source]#
Computes surface distance from full size Tensor and collects average over batch, class-channels, iterations.
- __init__(include_background=False, symmetric=False, distance_metric='euclidean', reduction=mean, output_transform=<function SurfaceDistance.<lambda>>, save_details=True)[source]#
- Parameters:
include_background – whether to include distance computation on the first channel of the predicted output. Defaults to
False
.symmetric – whether to calculate the symmetric average surface distance between seg_pred and seg_gt. Defaults to
False
.distance_metric – : [
"euclidean"
,"chessboard"
,"taxicab"
] the metric used to compute surface distance. Defaults to"euclidean"
.reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: surface dice of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
Panoptic Quality metrics handler#
- class monai.handlers.PanopticQuality(num_classes, metric_name='pq', reduction=mean_batch, match_iou_threshold=0.5, smooth_numerator=1e-06, output_transform=<function PanopticQuality.<lambda>>, save_details=True)[source]#
Computes Panoptic quality from full size Tensor and collects average over batch, class-channels, iterations.
- __init__(num_classes, metric_name='pq', reduction=mean_batch, match_iou_threshold=0.5, smooth_numerator=1e-06, output_transform=<function PanopticQuality.<lambda>>, save_details=True)[source]#
- Parameters:
num_classes – number of classes. The number should not count the background.
metric_name – output metric. The value can be “pq”, “sq” or “rq”.
reduction – define mode of reduction to the metrics, will only apply reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to self.reduction. if “none”, will not do reduction.match_iou_threshold – IOU threshold to determine the pairing between y_pred and y. Usually, it should >= 0.5, the pairing between instances of y_pred and y are identical. If set match_iou_threshold < 0.5, this function uses Munkres assignment to find the maximal amount of unique pairing.
smooth_numerator – a small constant added to the numerator to avoid zero.
output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: panoptic quality of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.panoptic_quality.compute_panoptic_quality()
Mean squared error metrics handler#
- class monai.handlers.MeanSquaredError(reduction=mean, output_transform=<function MeanSquaredError.<lambda>>, save_details=True)[source]#
Computes Mean Squared Error from full size Tensor and collects average over batch, iterations.
- __init__(reduction=mean, output_transform=<function MeanSquaredError.<lambda>>, save_details=True)[source]#
- Parameters:
reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean squared error of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
Mean absolute error metrics handler#
- class monai.handlers.MeanAbsoluteError(reduction=mean, output_transform=<function MeanAbsoluteError.<lambda>>, save_details=True)[source]#
Computes Mean Absolute Error from full size Tensor and collects average over batch, iterations.
- __init__(reduction=mean, output_transform=<function MeanAbsoluteError.<lambda>>, save_details=True)[source]#
- Parameters:
reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean squared error of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
Root mean squared error metrics handler#
- class monai.handlers.RootMeanSquaredError(reduction=mean, output_transform=<function RootMeanSquaredError.<lambda>>, save_details=True)[source]#
Computes Root Mean Squared Error from full size Tensor and collects average over batch, iterations.
- __init__(reduction=mean, output_transform=<function RootMeanSquaredError.<lambda>>, save_details=True)[source]#
- Parameters:
reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean squared error of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
Peak signal to noise ratio metrics handler#
- class monai.handlers.PeakSignalToNoiseRatio(max_val, reduction=mean, output_transform=<function PeakSignalToNoiseRatio.<lambda>>, save_details=True)[source]#
Computes Peak Signal to Noise Ratio from full size Tensor and collects average over batch, iterations.
- __init__(max_val, reduction=mean, output_transform=<function PeakSignalToNoiseRatio.<lambda>>, save_details=True)[source]#
- Parameters:
max_val – The dynamic range of the images/volumes (i.e., the difference between the maximum and the minimum allowed values e.g. 255 for a uint8 image).
reduction – define the mode to reduce metrics, will only execute reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: mean squared error of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
reduction – {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,
See also
Metrics reloaded binary handler#
- class monai.handlers.MetricsReloadedBinaryHandler(metric_name, include_background=True, reduction=mean, get_not_nans=False, output_transform=<function MetricsReloadedBinaryHandler.<lambda>>, save_details=True)[source]#
Handler of MetricsReloadedBinary, which wraps the binary pairwise metrics of MetricsReloaded.
- __init__(metric_name, include_background=True, reduction=mean, get_not_nans=False, output_transform=<function MetricsReloadedBinaryHandler.<lambda>>, save_details=True)[source]#
- Parameters:
metric_name – Name of a binary metric from the MetricsReloaded package.
include_background – whether to include computation on the first channel of the predicted output. Defaults to
True
.reduction – define mode of reduction to the metrics, will only apply reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.get_not_nans – whether to return the not_nans count, if True, aggregate() returns (metric, not_nans). Here not_nans count the number of not nans for the metric, thus its shape equals to the shape of the metric.
output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: TP/TN/FP/FN of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.wrapper()
Metrics reloaded categorical handler#
- class monai.handlers.MetricsReloadedCategoricalHandler(metric_name, include_background=True, reduction=mean, get_not_nans=False, smooth_dr=1e-05, output_transform=<function MetricsReloadedCategoricalHandler.<lambda>>, save_details=True)[source]#
Handler of MetricsReloadedCategorical, which wraps the categorical pairwise metrics of MetricsReloaded.
- __init__(metric_name, include_background=True, reduction=mean, get_not_nans=False, smooth_dr=1e-05, output_transform=<function MetricsReloadedCategoricalHandler.<lambda>>, save_details=True)[source]#
- Parameters:
metric_name – Name of a categorical metric from the MetricsReloaded package.
include_background – whether to include computation on the first channel of the predicted output. Defaults to
True
.reduction – define mode of reduction to the metrics, will only apply reduction on not-nan values, available reduction modes: {
"none"
,"mean"
,"sum"
,"mean_batch"
,"sum_batch"
,"mean_channel"
,"sum_channel"
}, default to"mean"
. if “none”, will not do reduction.get_not_nans – whether to return the not_nans count, if True, aggregate() returns (metric, not_nans). Here not_nans count the number of not nans for the metric, thus its shape equals to the shape of the metric.
smooth_dr – a small constant added to the denominator to avoid nan. OBS: should be greater than zero.
output_transform – callable to extract y_pred and y from ignite.engine.state.output then construct (y_pred, y) pair, where y_pred and y can be batch-first Tensors or lists of channel-first Tensors. the form of (y_pred, y) is required by the update(). engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
save_details – whether to save metric computation details per image, for example: TP/TN/FP/FN of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.wrapper()
Metric logger#
- class monai.handlers.MetricLogger(loss_transform=<function _get_loss_from_output>, metric_transform=<function MetricLogger.<lambda>>, evaluator=None)[source]#
Collect per-iteration metrics and loss value from the attached trainer. This will also collect metric values from a given evaluator object which is expected to perform evaluation at the end of training epochs. This class is useful for collecting loss and metric values in one place for storage with checkpoint savers (state_dict and load_state_dict methods provided as expected by Pytorch and Ignite) and for graphing during training.
- Example::
# construct an evaluator saving mean dice metric values in the key “val_mean_dice” evaluator = SupervisedEvaluator(…, key_val_metric={“val_mean_dice”: MeanDice(…)})
# construct the logger and associate with evaluator to extract metric values from logger = MetricLogger(evaluator=evaluator)
# construct the trainer with the logger passed in as a handler so that it logs loss values trainer = SupervisedTrainer(…, train_handlers=[logger, ValidationHandler(1, evaluator)])
# run training, logger.loss will be a list of (iteration, loss) values, logger.metrics a dict with key # “val_mean_dice” storing a list of (iteration, metric) values trainer.run()
- Parameters:
loss_transform – Converts the output value from the trainer’s state into a loss value engine.state and loss_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
metric_transform – Converts the metric value coming from the trainer/evaluator’s state into a storable value
evaluator – Optional evaluator to consume metric results from at the end of its evaluation run
- attach(engine)[source]#
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
Logfile handler#
- class monai.handlers.LogfileHandler(output_dir, filename='log.txt', loglevel=20, formatter='%(asctime)s %(name)s %(levelname)s: %(message)s', create_dir=True)[source]#
Adds a logging.FileHandler to the attached engine’s logger when the start event occurs and removes it again when then completed event occurs.
A handler is needed to remove FileHandler object when the complete event occurs so that further runs of different engines write only to the log files they should, rather than previous files. Multiple handlers can write to the same file which allows output from train and evaluation engine objects to be condensed in one file. If the given output directory doesn’t exist it will by default be created when the start event occurs. This can be used in conjunction with CheckpointSaver to save a log file to the same destination as the saved checkpoints. Since the handler is added possibly after other logging events during initialisation, not all logging data will be retained.
- Parameters:
output_dir (
str
) – directory to save the log file tofilename (
str
) – name of the file to save log tologlevel (
int
) – log level for the handlerformatter (
str
) – format string for the logging.Formatter set for the handlercreate_dir (
bool
) – if True, create output_dir if it doesn’t exist
Training stats handler#
- class monai.handlers.StatsHandler(iteration_log=True, epoch_log=True, epoch_print_logger=None, iteration_print_logger=None, output_transform=<function StatsHandler.<lambda>>, global_epoch_transform=<function StatsHandler.<lambda>>, state_attributes=None, name='StatsHandler', tag_name='Loss', key_var_format='{}: {:.4f} ')[source]#
StatsHandler defines a set of Ignite Event-handlers for all the log printing logics. It can be used for any Ignite Engine(trainer, validator and evaluator). And it can support logging for epoch level and iteration level with pre-defined loggers.
Note that if
name
is None, this class will leverage engine.logger as the logger, otherwise,logging.getLogger(name)
is used. In both cases, it’s important to make sure that the logging level is at leastINFO
. To change the level of logging, please callimport ignite; ignite.utils.setup_logger(name)
(whenname
is not None) orengine.logger = ignite.utils.setup_logger(engine.logger.name, reset=True)
(whenname
is None) before running the engine with this handler attached.- Default behaviors:
When EPOCH_COMPLETED, logs
engine.state.metrics
usingself.logger
.When ITERATION_COMPLETED, logs
self.output_transform(engine.state.output)
usingself.logger
.
Usage example:
import ignite import monai trainer = ignite.engine.Engine(lambda x, y: [0.0]) # an example trainer monai.handlers.StatsHandler(name="train_stats").attach(trainer) trainer.run(range(3), max_epochs=4)
More details of example is available in the tutorial: Project-MONAI/tutorials.
- __init__(iteration_log=True, epoch_log=True, epoch_print_logger=None, iteration_print_logger=None, output_transform=<function StatsHandler.<lambda>>, global_epoch_transform=<function StatsHandler.<lambda>>, state_attributes=None, name='StatsHandler', tag_name='Loss', key_var_format='{}: {:.4f} ')[source]#
- Parameters:
iteration_log – whether to log data when iteration completed, default to True.
iteration_log
can be also a function and it will be interpreted as an event filter (see https://pytorch.org/ignite/generated/ignite.engine.events.Events.html for details). Event filter function accepts as input engine and event value (iteration) and should return True/False. Event filtering can be helpful to customize iteration logging frequency.epoch_log – whether to log data when epoch completed, default to True.
epoch_log
can be also a function and it will be interpreted as an event filter. Seeiteration_log
argument for more details.epoch_print_logger – customized callable printer for epoch level logging. Must accept parameter “engine”, use default printer if None.
iteration_print_logger – customized callable printer for iteration level logging. Must accept parameter “engine”, use default printer if None.
output_transform – a callable that is used to transform the
ignite.engine.state.output
into a scalar to print, or a dictionary of {key: scalar}. In the latter case, the output string will be formatted as key: value. By default this value logging happens when every iteration completed. The default behavior is to print loss from output[0] as output is a decollated list and we replicated loss value for every item of the decollated list. engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.global_epoch_transform – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to print synced epoch number with the trainer engine.
state_attributes – expected attributes from engine.state, if provided, will extract them when epoch completed.
name – identifier of logging.logger to use, if None, defaulting to
engine.logger
.tag_name – when iteration output is a scalar, tag_name is used to print tag_name: scalar_value to logger. Defaults to
'Loss'
.key_var_format – a formatting string to control the output string format of key: value.
- attach(engine)[source]#
Register a set of Ignite Event-Handlers to a specified Ignite engine.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- epoch_completed(engine)[source]#
Handler for train or validation/evaluation epoch completed Event. Print epoch level log, default values are from Ignite engine.state.metrics dict.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- exception_raised(_engine, e)[source]#
Handler for train or validation/evaluation exception raised Event. Print the exception information and traceback. This callback may be skipped because the logic with Ignite can only trigger the first attached handler for EXCEPTION_RAISED event.
- Parameters:
_engine (
Engine
) – Ignite Engine, unused argument.e (
Exception
) – the exception caught in Ignite during engine.run().
- Return type:
None
Tensorboard handlers#
- class monai.handlers.TensorBoardHandler(summary_writer=None, log_dir='./runs')[source]#
Base class for the handlers to write data into TensorBoard.
- Parameters:
summary_writer – user can specify TensorBoard or TensorBoardX SummaryWriter, default to create a new TensorBoard writer.
log_dir – if using default SummaryWriter, write logs to this directory, default is ./runs.
- class monai.handlers.TensorBoardStatsHandler(summary_writer=None, log_dir='./runs', iteration_log=True, epoch_log=True, epoch_event_writer=None, iteration_event_writer=None, output_transform=<function TensorBoardStatsHandler.<lambda>>, global_epoch_transform=<function TensorBoardStatsHandler.<lambda>>, state_attributes=None, tag_name='Loss')[source]#
TensorBoardStatsHandler defines a set of Ignite Event-handlers for all the TensorBoard logics. It can be used for any Ignite Engine(trainer, validator and evaluator). And it can support both epoch level and iteration level with pre-defined TensorBoard event writer. The expected data source is Ignite
engine.state.output
andengine.state.metrics
.- Default behaviors:
When EPOCH_COMPLETED, write each dictionary item in
engine.state.metrics
to TensorBoard.When ITERATION_COMPLETED, write each dictionary item in
self.output_transform(engine.state.output)
to TensorBoard.
Usage example is available in the tutorial: Project-MONAI/tutorials.
- __init__(summary_writer=None, log_dir='./runs', iteration_log=True, epoch_log=True, epoch_event_writer=None, iteration_event_writer=None, output_transform=<function TensorBoardStatsHandler.<lambda>>, global_epoch_transform=<function TensorBoardStatsHandler.<lambda>>, state_attributes=None, tag_name='Loss')[source]#
- Parameters:
summary_writer – user can specify TensorBoard or TensorBoardX SummaryWriter, default to create a new TensorBoard writer.
log_dir – if using default SummaryWriter, write logs to this directory, default is ./runs.
iteration_log – whether to write data to TensorBoard when iteration completed, default to True.
iteration_log
can be also a function or int. If it is an int, it will be interpreted as the iteration interval at which the iteration_event_writer is called. If it is a function, it will be interpreted as an event filter (see https://pytorch.org/ignite/generated/ignite.engine.events.Events.html for details). Event filter function accepts as input engine and event value (iteration) and should return True/False.epoch_log – whether to write data to TensorBoard when epoch completed, default to True.
epoch_log
can be also a function or int. If it is an int, it will be interpreted as the epoch interval at which the epoch_event_writer is called. If it is a function, it will be interpreted as an event filter. Seeiteration_log
argument for more details.epoch_event_writer – customized callable TensorBoard writer for epoch level. Must accept parameter “engine” and “summary_writer”, use default event writer if None.
iteration_event_writer – customized callable TensorBoard writer for iteration level. Must accept parameter “engine” and “summary_writer”, use default event writer if None.
output_transform – a callable that is used to transform the
ignite.engine.state.output
into a scalar to plot, or a dictionary of {key: scalar}. In the latter case, the output string will be formatted as key: value. By default this value plotting happens when every iteration completed. The default behavior is to print loss from output[0] as output is a decollated list and we replicated loss value for every item of the decollated list. engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.global_epoch_transform – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to use trainer engines epoch number when plotting epoch vs metric curves.
state_attributes – expected attributes from engine.state, if provided, will extract them when epoch completed.
tag_name – when iteration output is a scalar, tag_name is used to plot, defaults to
'Loss'
.
- attach(engine)[source]#
Register a set of Ignite Event-Handlers to a specified Ignite engine.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- class monai.handlers.TensorBoardImageHandler(summary_writer=None, log_dir='./runs', interval=1, epoch_level=True, batch_transform=<function TensorBoardImageHandler.<lambda>>, output_transform=<function TensorBoardImageHandler.<lambda>>, global_iter_transform=<function TensorBoardImageHandler.<lambda>>, index=0, max_channels=1, frame_dim=-3, max_frames=64)[source]#
TensorBoardImageHandler is an Ignite Event handler that can visualize images, labels and outputs as 2D/3D images. 2D output (shape in Batch, channel, H, W) will be shown as simple image using the first element in the batch, for 3D to ND output (shape in Batch, channel, H, W, D) input, each of
self.max_channels
number of images’ last three dimensions will be shown as animated GIF along the last axis (typically Depth). And if writer is from TensorBoardX, data has 3 channels and max_channels=3, will plot as RGB video.It can be used for any Ignite Engine (trainer, validator and evaluator). User can easily add it to engine for any expected Event, for example:
EPOCH_COMPLETED
,ITERATION_COMPLETED
. The expected data source is ignite’sengine.state.batch
andengine.state.output
.- Default behavior:
Show y_pred as images (GIF for 3D) on TensorBoard when Event triggered,
Need to use
batch_transform
andoutput_transform
to specify how many images to show and show which channel.Expects
batch_transform(engine.state.batch)
to return data format: (image[N, channel, …], label[N, channel, …]).Expects
output_transform(engine.state.output)
to return a torch tensor in format (y_pred[N, channel, …], loss).
Usage example is available in the tutorial: Project-MONAI/tutorials.
- __init__(summary_writer=None, log_dir='./runs', interval=1, epoch_level=True, batch_transform=<function TensorBoardImageHandler.<lambda>>, output_transform=<function TensorBoardImageHandler.<lambda>>, global_iter_transform=<function TensorBoardImageHandler.<lambda>>, index=0, max_channels=1, frame_dim=-3, max_frames=64)[source]#
- Parameters:
summary_writer – user can specify TensorBoard or TensorBoardX SummaryWriter, default to create a new TensorBoard writer.
log_dir – if using default SummaryWriter, write logs to this directory, default is ./runs.
interval – plot content from engine.state every N epochs or every N iterations, default is 1.
epoch_level – plot content from engine.state every N epochs or N iterations. True is epoch level, False is iteration level.
batch_transform – a callable that is used to extract image and label from ignite.engine.state.batch, then construct (image, label) pair. for example: if ignite.engine.state.batch is {“image”: xxx, “label”: xxx, “other”: xxx}, batch_transform can be lambda x: (x[“image”], x[“label”]). will use the result to plot image from result[0][index] and plot label from result[1][index]. engine.state and batch_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
output_transform – a callable that is used to extract the predictions data from ignite.engine.state.output, will use the result to plot output from result[index]. engine.state and output_transform inherit from the ignite concept: https://pytorch.org/ignite/concepts.html#state, explanation and usage example are in the tutorial: Project-MONAI/tutorials.
global_iter_transform – a callable that is used to customize global step number for TensorBoard. For example, in evaluation, the evaluator engine needs to know current epoch from trainer.
index – plot which element in a data batch, default is the first element.
max_channels – number of channels to plot.
frame_dim – if plotting 3D image as GIF, specify the dimension used as frames, expect input data shape as NCHWD, default to -3 (the first spatial dim)
max_frames – if plot 3D RGB image as video in TensorBoardX, set the FPS to max_frames.
LR Schedule handler#
- class monai.handlers.LrScheduleHandler(lr_scheduler, print_lr=True, name=None, epoch_level=True, step_transform=<function LrScheduleHandler.<lambda>>)[source]#
Ignite handler to update the Learning Rate based on PyTorch LR scheduler.
- __init__(lr_scheduler, print_lr=True, name=None, epoch_level=True, step_transform=<function LrScheduleHandler.<lambda>>)[source]#
- Parameters:
lr_scheduler – typically, lr_scheduler should be PyTorch lr_scheduler object. If customized version, must have step and get_last_lr methods.
print_lr – whether to print out the latest learning rate with logging.
name – identifier of logging.logger to use, if None, defaulting to
engine.logger
.epoch_level – execute lr_scheduler.step() after every epoch or every iteration. True is epoch level, False is iteration level.
step_transform – a callable that is used to transform the information from engine to expected input data of lr_scheduler.step() function if necessary.
- Raises:
TypeError – When
step_transform
is notcallable
.
Validation handler#
- class monai.handlers.ValidationHandler(interval, validator=None, epoch_level=True, exec_at_start=False)[source]#
Attach validator to the trainer engine in Ignite. It can support to execute validation every N epochs or every N iterations.
- __init__(interval, validator=None, epoch_level=True, exec_at_start=False)[source]#
- Parameters:
interval – do validation every N epochs or every N iterations during training.
validator – run the validator when trigger validation, suppose to be Evaluator. if None, should call set_validator() before training.
epoch_level – execute validation every N epochs or N iterations. True is epoch level, False is iteration level.
exec_at_start – whether to execute a validation first when starting the training. default to False. It can be useful especially for some transfer-learning cases to validate the initial model before training.
- Raises:
TypeError – When
validator
is not amonai.engines.evaluator.Evaluator
.
SmartCache handler#
- class monai.handlers.SmartCacheHandler(smartcacher)[source]#
Attach SmartCache logic to the engine in Ignite. Mainly include the start, update_cache, and shutdown functions of SmartCacheDataset.
- __init__(smartcacher)[source]#
- Parameters:
smartcacher (
SmartCacheDataset
) – predefined SmartCacheDataset, will attach it to the engine.- Raises:
TypeError – When
smartcacher
is not amonai.data.SmartCacheDataset
.
- attach(engine)[source]#
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- completed(engine)[source]#
Callback for train or validation/evaluation completed Event. Stop the replacement thread of SmartCacheDataset.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
Parameter Scheduler handler#
- class monai.handlers.ParamSchedulerHandler(parameter_setter, value_calculator, vc_kwargs, epoch_level=False, name=None, event=None)[source]#
General purpose scheduler for parameters values. By default it can schedule in a linear, exponential, step or multistep function. One can also pass Callables to have customized scheduling logic.
- Parameters:
parameter_setter (Callable) – Function that sets the required parameter
value_calculator (Union[str,Callable]) – Either a string (‘linear’, ‘exponential’, ‘step’ or ‘multistep’) or Callable for custom logic.
vc_kwargs (Dict) – Dictionary that stores the required parameters for the value_calculator.
epoch_level (bool) – Whether the step is based on epoch or iteration. Defaults to False.
name (Optional[str]) – Identifier of logging.logger to use, if None, defaulting to
engine.logger
.event (Optional[str]) – Event to which the handler attaches. Defaults to Events.ITERATION_COMPLETED.
EarlyStop handler#
- class monai.handlers.EarlyStopHandler(patience, score_function, trainer=None, min_delta=0.0, cumulative_delta=False, epoch_level=True)[source]#
EarlyStopHandler acts as an Ignite handler to stop training if no improvement after a given number of events. It‘s based on the EarlyStopping handler in ignite.
- Parameters:
patience – number of events to wait if no improvement and then stop the training.
score_function – It should be a function taking a single argument, an
Engine
object that the handler attached, can be a trainer or validator, and return a score float. an improvement is considered if the score is higher.trainer – trainer engine to stop the run if no improvement, if None, must call set_trainer() before training.
min_delta – a minimum increase in the score to qualify as an improvement, i.e. an increase of less than or equal to min_delta, will count as no improvement.
cumulative_delta – if True, min_delta defines an increase since the last patience reset, otherwise, it defines an increase after the last event, default to False.
epoch_level – check early stopping for every epoch or every iteration of the attached engine, True is epoch level, False is iteration level, default to epoch level.
Note
If in distributed training and uses loss value of every iteration to detect early stopping, the values may be different in different ranks. When using this handler with distributed training, please also note that to prevent “dist.destroy_process_group()” hangs, you can use an “all_reduce” operation to synchronize the stop signal across all ranks. The mechanism can be implemented in the score_function. The following is an example:
import os import torch import torch.distributed as dist def score_function(engine): val_metric = engine.state.metrics["val_mean_dice"] if dist.is_initialized(): device = torch.device("cuda:" + os.environ["LOCAL_RANK"]) val_metric = torch.tensor([val_metric]).to(device) dist.all_reduce(val_metric, op=dist.ReduceOp.SUM) val_metric /= dist.get_world_size() return val_metric.item() return val_metric
User may attach this handler to validator engine to detect validation metrics and stop the training, in this case, the score_function is executed on validator engine and trainer is the trainer engine.
GarbageCollector handler#
- class monai.handlers.GarbageCollector(trigger_event='epoch', log_level=10)[source]#
Run garbage collector after each epoch
- Parameters:
trigger_event – the event that trigger a call to this handler. - “epoch”, after completion of each epoch (equivalent of ignite.engine.Events.EPOCH_COMPLETED) - “iteration”, after completion of each iteration (equivalent of ignite.engine.Events.ITERATION_COMPLETED) - any ignite built-in event from ignite.engine.Events. Defaults to “epoch”.
log_level – log level (integer) for some garbage collection information as below. Defaults to 10 (DEBUG). - 50 (CRITICAL) - 40 (ERROR) - 30 (WARNING) - 20 (INFO) - 10 (DEBUG) - 0 (NOTSET)
Post processing#
- class monai.handlers.PostProcessing(transform, event='MODEL_COMPLETED')[source]#
Ignite handler to execute additional post processing after the post processing in engines. So users can insert other handlers between engine postprocessing and this post processing handler. If using components from monai.transforms as the transform, recommend to decollate engine.state.batch and engine.state.batch in the engine(set decollate=True) or in the DecollateBatch handler first.
- __init__(transform, event='MODEL_COMPLETED')[source]#
- Parameters:
transform (
Callable
) – callable function to execute on the engine.state.batch and engine.state.output. can also be composed transforms.event (
str
) – expected EVENT to attach the handler, should be “MODEL_COMPLETED” or “ITERATION_COMPLETED”. default to “MODEL_COMPLETED”.
Decollate batch#
- class monai.handlers.DecollateBatch(event='MODEL_COMPLETED', detach=True, decollate_batch=True, batch_keys=None, decollate_output=True, output_keys=None, allow_missing_keys=False)[source]#
Ignite handler to execute the decollate batch logic for engine.state.batch and engine.state.output. Typical usage is to set decollate=False in the engine and execute some postprocessing logic first then decollate the batch, otherwise, engine will decollate batch before the postprocessing.
- Parameters:
event – expected EVENT to attach the handler, should be “MODEL_COMPLETED” or “ITERATION_COMPLETED”. default to “MODEL_COMPLETED”.
detach – whether to detach the tensors. scalars tensors will be detached into number types instead of torch tensors.
decollate_batch – whether to decollate engine.state.batch of ignite engine.
batch_keys – if decollate_batch=True, specify the keys of the corresponding items to decollate in engine.state.batch, note that it will delete other keys not specified. if None, will decollate all the keys. it replicates the scalar values to every item of the decollated list.
decollate_output – whether to decollate engine.state.output of ignite engine.
output_keys – if decollate_output=True, specify the keys of the corresponding items to decollate in engine.state.output, note that it will delete other keys not specified. if None, will decollate all the keys. it replicates the scalar values to every item of the decollated list.
allow_missing_keys – don’t raise exception if key is missing.
MLFlow handler#
- class monai.handlers.MLFlowHandler(tracking_uri=None, iteration_log=True, epoch_log=True, epoch_logger=None, iteration_logger=None, dataset_logger=None, dataset_dict=None, dataset_keys=image, output_transform=<function MLFlowHandler.<lambda>>, global_epoch_transform=<function MLFlowHandler.<lambda>>, state_attributes=None, tag_name='Loss', experiment_name='monai_experiment', run_name=None, experiment_param=None, artifacts=None, optimizer_param_names='lr', close_on_complete=False)[source]#
MLFlowHandler defines a set of Ignite Event-handlers for the MLFlow tracking logics. It can be used for any Ignite Engine(trainer, validator and evaluator). And it can track both epoch level and iteration level logging, then MLFlow can store the data and visualize. The expected data source is Ignite
engine.state.output
andengine.state.metrics
.- Default behaviors:
When EPOCH_COMPLETED, track each dictionary item in
engine.state.metrics
in MLFlow.When ITERATION_COMPLETED, track expected item in
self.output_transform(engine.state.output)
in MLFlow, default to Loss.
Usage example is available in the tutorial: Project-MONAI/tutorials.
- Parameters:
tracking_uri – connects to a tracking URI. can also set the MLFLOW_TRACKING_URI environment variable to have MLflow find a URI from there. in both cases, the URI can either be an HTTP/HTTPS URI for a remote server, a database connection string, or a local path to log data to a directory. The URI defaults to path mlruns. for more details: https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri.
iteration_log – whether to log data to MLFlow when iteration completed, default to True.
iteration_log
can be also a function and it will be interpreted as an event filter (see https://pytorch.org/ignite/generated/ignite.engine.events.Events.html for details). Event filter function accepts as input engine and event value (iteration) and should return True/False.epoch_log – whether to log data to MLFlow when epoch completed, default to True.
epoch_log
can be also a function and it will be interpreted as an event filter. Seeiteration_log
argument for more details.epoch_logger – customized callable logger for epoch level logging with MLFlow. Must accept parameter “engine”, use default logger if None.
iteration_logger – customized callable logger for iteration level logging with MLFlow. Must accept parameter “engine”, use default logger if None.
dataset_logger – customized callable logger to log the dataset information with MLFlow. Must accept parameter “dataset_dict”, use default logger if None.
dataset_dict – a dictionary in which the key is the name of the dataset and the value is a PyTorch dataset, that needs to be recorded. This arg is only useful when MLFlow version >= 2.4.0. For more details about how to log data with MLFlow, please go to the website: https://mlflow.org/docs/latest/python_api/mlflow.data.html.
dataset_keys – a key or a collection of keys to indicate contents in the dataset that need to be stored by MLFlow.
output_transform – a callable that is used to transform the
ignite.engine.state.output
into a scalar to track, or a dictionary of {key: scalar}. By default this value logging happens when every iteration completed. The default behavior is to track loss from output[0] as output is a decollated list and we replicated loss value for every item of the decollated list. engine.state and output_transform inherit from the ignite concept: https://pytorch-ignite.ai/concepts/03-state/, explanation and usage example are in the tutorial: Project-MONAI/tutorials.global_epoch_transform – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to track synced epoch number with the trainer engine.
state_attributes – expected attributes from engine.state, if provided, will extract them when epoch completed.
tag_name – when iteration output is a scalar, tag_name is used to track, defaults to ‘Loss’.
experiment_name – the experiment name of MLflow, default to ‘monai_experiment’. An experiment can be used to record several runs.
run_name – the run name in an experiment. A run can be used to record information about a workflow, like the loss, metrics and so on.
experiment_param – a dict recording parameters which will not change through the whole workflow, like torch version, cuda version and so on.
artifacts – paths to images that need to be recorded after running the workflow.
optimizer_param_names – parameter names in the optimizer that need to be recorded during running the workflow, default to ‘lr’.
close_on_complete – whether to close the mlflow run in complete phase in workflow, default to False.
For more details of MLFlow usage, please refer to: https://mlflow.org/docs/latest/index.html.
- attach(engine)[source]#
Register a set of Ignite Event-Handlers to a specified Ignite engine.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- epoch_completed(engine)[source]#
Handler for train or validation/evaluation epoch completed Event. Track epoch level log, default values are from Ignite engine.state.metrics dict.
- Parameters:
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
ClearML handlers#
- class monai.handlers.ClearMLHandler(project_name, task_name, output_uri, tags, reuse_last_task_id, continue_last_task, auto_connect_frameworks, auto_connect_arg_parser)[source]#
Base class for the handlers to log everything to ClearML. For more details of ClearML usage, please refer to: https://clear.ml/docs/latest/docs/references/sdk/task
Usage example is available in the tutorial: Project-MONAI/tutorials.
- __init__(project_name, task_name, output_uri, tags, reuse_last_task_id, continue_last_task, auto_connect_frameworks, auto_connect_arg_parser)[source]#
- Parameters:
project_name – ClearML project name, default to ‘MONAI’.
task_name – ClearML task name, default to ‘monai_experiment’.
output_uri – The default location for output models and other artifacts, default to ‘True’.
tags – Add a list of tags (str) to the created Task, default to ‘None’.
reuse_last_task_id – Force a new Task (experiment) with a previously used Task ID, default to ‘True’.
continue_last_task – Continue the execution of a previously executed Task (experiment), default to ‘False’.
auto_connect_frameworks – Automatically connect frameworks, default to ‘True’.
auto_connect_arg_parser – Automatically connect an argparse object to the Task, default to ‘True’.
- class monai.handlers.ClearMLStatsHandler(project_name='MONAI', task_name='monai_experiment', output_uri=True, tags=None, reuse_last_task_id=True, continue_last_task=False, auto_connect_frameworks=True, auto_connect_arg_parser=True, *args, **kwargs)[source]#
Class to write tensorboard stats by inheriting TensorBoardStatsHandler class. Everything from Tensorboard is logged automatically to ClearML.
Usage example is available in the tutorial: Project-MONAI/tutorials.
- __init__(project_name='MONAI', task_name='monai_experiment', output_uri=True, tags=None, reuse_last_task_id=True, continue_last_task=False, auto_connect_frameworks=True, auto_connect_arg_parser=True, *args, **kwargs)[source]#
- Parameters:
project_name – ClearML project name, default to ‘MONAI’.
task_name – ClearML task name, default to ‘monai_experiment’.
output_uri – The default location for output models and other artifacts, default to ‘True’.
tags – Add a list of tags (str) to the created Task, default to ‘None’.
reuse_last_task_id – Force a new Task (experiment) with a previously used Task ID, default to ‘True’.
continue_last_task – Continue the execution of a previously executed Task (experiment), default to ‘False’.
auto_connect_frameworks – Automatically connect frameworks, default to ‘True’.
auto_connect_arg_parser – Automatically connect an argparse object to the Task, default to ‘True’.
- class monai.handlers.ClearMLImageHandler(project_name='MONAI', task_name='monai_experiment', output_uri=True, tags=None, reuse_last_task_id=True, continue_last_task=False, auto_connect_frameworks=True, auto_connect_arg_parser=True, *args, **kwargs)[source]#
This class inherits all functionality from TensorBoardImageHandler class. Everything from Tensorboard is logged automatically to ClearML.
Usage example is available in the tutorial: Project-MONAI/tutorials.
- __init__(project_name='MONAI', task_name='monai_experiment', output_uri=True, tags=None, reuse_last_task_id=True, continue_last_task=False, auto_connect_frameworks=True, auto_connect_arg_parser=True, *args, **kwargs)[source]#
- Parameters:
project_name – ClearML project name, default to ‘MONAI’.
task_name – ClearML task name, default to ‘monai_experiment’.
output_uri – The default location for output models and other artifacts, default to ‘True’.
tags – Add a list of tags (str) to the created Task, default to ‘None’.
reuse_last_task_id – Force a new Task (experiment) with a previously used Task ID, default to ‘True’.
continue_last_task – Continue the execution of a previously executed Task (experiment), default to ‘False’.
auto_connect_frameworks – Automatically connect frameworks, default to ‘True’.
auto_connect_arg_parser – Automatically connect an argparse object to the Task, default to ‘True’.
NVTX Handlers#
Wrapper around NVIDIA Tools Extension for profiling MONAI ignite workflow
- class monai.handlers.nvtx_handlers.MarkHandler(event, msg=None)[source]#
Mark an instantaneous event that occurred at some point.
- Parameters:
msg – ASCII message to associate with range
- class monai.handlers.nvtx_handlers.RangeHandler(events, msg=None)[source]#
Attach a NVTX range to a pair of Ignite events. It pushes an NVTX range at the first event and pops it at the second event. Stores zero-based depth of the range that is started.
- Parameters:
events – a string, pair of Ignite events, pair of Ignite event literals, or pair of Ignite events and literals. If a single string is provided, it should describe the base name of a pair of default Ignite events with _STARTED and _COMPLETED postfix (like “EPOCH” for Events.EPOCH_STARTED and Events.EPOCH_COMPLETED). The accepted events are: BATCH, ITERATION, EPOCH, and ENGINE. If pair of literals, each should be the literal equivalent of an Ignite event, fo instance: (“EPOCH_STARTED” and “EPOCH_COMPLETED”). One can combine events and literals, like (Events.EPOCH_STARTED and “EPOCH_COMPLETED”). For the complete list of Events, check https://pytorch.org/ignite/generated/ignite.engine.events.Events.html.
msg – ASCII message to associate with range. If not provided, the name of first event will be assigned to the NVTX range.
- attach(engine)[source]#
Attach an NVTX Range to specific Ignite events :type engine:
Engine
:param engine: Ignite Engine, it can be a trainer, validator or evaluator.- Return type:
None
- class monai.handlers.nvtx_handlers.RangePopHandler(event)[source]#
At a specific event, pop a previously pushed range. Stores zero-based depth of the range that is started.
- Parameters:
msg – ASCII message to associate with range
Utilities#
- monai.handlers.utils.from_engine(keys, first=False)[source]#
Utility function to simplify the batch_transform or output_transform args of ignite components when handling dictionary or list of dictionaries(for example: engine.state.batch or engine.state.output). Users only need to set the expected keys, then it will return a callable function to extract data from dictionary and construct a tuple respectively.
If data is a list of dictionaries after decollating, extract expected keys and construct lists respectively, for example, if data is [{“A”: 1, “B”: 2}, {“A”: 3, “B”: 4}], from_engine([“A”, “B”]): ([1, 3], [2, 4]).
It can help avoid a complicated lambda function and make the arg of metrics more straight-forward. For example, set the first key as the prediction and the second key as label to get the expected data from engine.state.output for a metric:
from monai.handlers import MeanDice, from_engine metric = MeanDice( include_background=False, output_transform=from_engine(["pred", "label"]) )
- Parameters:
keys (
Union
[Collection
[Hashable
],Hashable
]) – specified keys to extract data from dictionary or decollated list of dictionaries.first (
bool
) – whether only extract specified keys from the first item if input data is a list of dictionaries, it’s used to extract the scalar data which doesn’t have batch dim and was replicated into every dictionary when decollating, like loss, etc.
- Return type:
Callable
- monai.handlers.utils.stopping_fn_from_loss()[source]#
Returns a stopping function for ignite.handlers.EarlyStopping using the loss value.
- Return type:
Callable
[Engine
,Any
]
- monai.handlers.utils.stopping_fn_from_metric(metric_name)[source]#
Returns a stopping function for ignite.handlers.EarlyStopping using the given metric name.
- Return type:
Callable
[Engine
,Any
]
- monai.handlers.utils.write_metrics_reports(save_dir, images, metrics, metric_details, summary_ops, deli=',', output_type='csv', class_labels=None)[source]#
Utility function to write the metrics into files, contains 3 parts: 1. if metrics dict is not None, write overall metrics into file, every line is a metric name and value pair. 2. if metric_details dict is not None, write raw metric data of every image into file, every line for 1 image. 3. if summary_ops is not None, compute summary based on operations on metric_details and write to file.
- Parameters:
save_dir – directory to save all the metrics reports.
images – name or path of every input image corresponding to the metric_details data. if None, will use index number as the filename of every input image.
metrics – a dictionary of (metric name, metric value) pairs.
metric_details – a dictionary of (metric name, metric raw values) pairs, usually, it comes from metrics computation, for example, the raw value can be the mean_dice of every channel of every input image.
summary_ops –
expected computation operations to generate the summary report. it can be: None, “*” or list of strings, default to None. None - don’t generate summary report for every expected metric_details. “*” - generate summary report for every metric_details with all the supported operations. list of strings - generate summary report for every metric_details with specified operations, they should be within list: [“mean”, “median”, “max”, “min”, “<int>percentile”, “std”, “notnans”]. the number in “<int>percentile” should be [0, 100], like: “15percentile”. default: “90percentile”. for more details, please check: https://numpy.org/doc/stable/reference/generated/numpy.nanpercentile.html. note that: for the overall summary, it computes nanmean of all classes for each image first, then compute summary. example of the generated summary report:
class mean median max 5percentile 95percentile notnans class0 6.0000 6.0000 7.0000 5.1000 6.9000 2.0000 class1 6.0000 6.0000 6.0000 6.0000 6.0000 1.0000 mean 6.2500 6.2500 7.0000 5.5750 6.9250 2.0000
deli – the delimiter character in the saved file, default to “,” as the default output type is csv. to be consistent with: https://docs.python.org/3/library/csv.html#csv.Dialect.delimiter.
output_type – expected output file type, supported types: [“csv”], default to “csv”.
class_labels – list of class names used to name the classes in the output report, if None, “class0”, …, “classn” are used, default to None.
Probability Map Handlers#
- class monai.handlers.probability_maps.ProbMapProducer(output_dir='./', output_postfix='', prob_key='pred', dtype=<class 'numpy.float64'>, name=None)[source]#
Event handler triggered on completing every iteration to calculate and save the probability map. This handler use metadata from MetaTensor to create the probability map. This can be simply achieved by using monai.data.SlidingPatchWSIDataset or monai.data.MaskedPatchWSIDataset as the dataset.
- __init__(output_dir='./', output_postfix='', prob_key='pred', dtype=<class 'numpy.float64'>, name=None)[source]#
- Parameters:
output_dir – output directory to save probability maps.
output_postfix – a string appended to all output file names.
prob_key – the key associated to the probability output of the model
dtype – the data type in which the probability map is stored. Default np.float64.
name – identifier of logging.logger to use, defaulting to engine.logger.