Event handlers¶
Model checkpoint loader¶
-
class
monai.handlers.
CheckpointLoader
(load_path, load_dict, name=None, map_location=None, strict=True, strict_shape=True)[source]¶ CheckpointLoader acts as an Ignite handler to load checkpoint data from file. It can load variables for network, optimizer, lr_scheduler, etc. If saving checkpoint after torch.nn.DataParallel, need to save model.module instead as PyTorch recommended and then use this loader to load the model.
- Parameters
load_path (
str
) – the file path of checkpoint, it should be a PyTorch pth file.load_dict (
Dict
) –target objects that load checkpoint to. examples:
{'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler}
name (
Optional
[str
]) – identifier of logging.logger to use, if None, defaulting toengine.logger
.map_location (
Optional
[Dict
]) – when loading the module for distributed training/evaluation, need to provide an appropriate map_location argument to prevent a process to step into others’ devices. If map_location is missing, torch.load will first load the module to CPU and then copy each parameter to where it was saved, which would result in all processes on the same machine using the same set of devices.strict (
bool
) – whether to strictly enforce that the keys in state_dict match the keys returned by torch.nn.Module.state_dict function. default to True.strict_shape (
bool
) – whether to enforce the data shape of the matched layers in the checkpoint, if `False, it will skip the layers that have different data shape with checkpoint content. This can be useful advanced feature for transfer learning. users should totally understand which layers will have different shape. default to True.
Model checkpoint saver¶
-
class
monai.handlers.
CheckpointSaver
(save_dir, save_dict, name=None, file_prefix='', save_final=False, final_filename=None, save_key_metric=False, key_metric_name=None, key_metric_n_saved=1, key_metric_filename=None, key_metric_save_state=False, key_metric_greater_or_equal=False, epoch_level=True, save_interval=0, n_saved=None)[source]¶ CheckpointSaver acts as an Ignite handler to save checkpoint data into files. It supports to save according to metrics result, epoch number, iteration number and last model or exception.
- Parameters
save_dir (
str
) – the target directory to save the checkpoints.save_dict (
Dict
) –source objects that save to the checkpoint. examples:
{'network': net, 'optimizer': optimizer, 'lr_scheduler': lr_scheduler}
name (
Optional
[str
]) – identifier of logging.logger to use, if None, defaulting toengine.logger
.file_prefix (
str
) – prefix for the filenames to which objects will be saved.save_final (
bool
) – whether to save checkpoint or session at final iteration or exception. If checkpoints are to be saved when an exception is raised, put this handler before StatsHandler in the handler list, because the logic with Ignite can only trigger the first attached handler for EXCEPTION_RAISED event.final_filename (
Optional
[str
]) – set a fixed filename to save the final model if save_final=True. If None, default to checkpoint_final_iteration=N.pt.save_key_metric (
bool
) – whether to save checkpoint or session when the value of key_metric is higher than all the previous values during training.keep 4 decimal places of metric, checkpoint name is: {file_prefix}_key_metric=0.XXXX.pth.key_metric_name (
Optional
[str
]) – the name of key_metric in ignite metrics dictionary. If None, use engine.state.key_metric instead.key_metric_n_saved (
int
) – save top N checkpoints or sessions, sorted by the value of key metric in descending order.key_metric_filename (
Optional
[str
]) – set a fixed filename to set the best metric model, if not None, key_metric_n_saved should be 1 and only keep the best metric model.key_metric_save_state (
bool
) – whether to save the tracking list of key metric in the checkpoint file. if True, then will save an object in the checkpoint file with key checkpointer to be consistent with ignite: https://github.com/pytorch/ignite/blob/master/ignite/handlers/checkpoint.py#L99. typically, it’s used to resume training and compare current metric with previous N values.key_metric_greater_or_equal (
bool
) – if True, the latest equally scored model is stored. Otherwise, save the the first equally scored model. default to False.epoch_level (
bool
) – save checkpoint during training for every N epochs or every N iterations. True is epoch level, False is iteration level.save_interval (
int
) – save checkpoint every N epochs, default is 0 to save no checkpoint.n_saved (
Optional
[int
]) – save latest N checkpoints of epoch level or iteration level, ‘None’ is to save all.
Note
CheckpointHandler can be used during training, validation or evaluation. example of saved files:
checkpoint_iteration=400.pt
checkpoint_iteration=800.pt
checkpoint_epoch=1.pt
checkpoint_final_iteration=1000.pt
checkpoint_key_metric=0.9387.pt
-
attach
(engine)[source]¶ - Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
completed
(engine)[source]¶ Callback for train or validation/evaluation completed Event. Save final checkpoint if configure save_final is True.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
exception_raised
(engine, e)[source]¶ Callback for train or validation/evaluation exception raised Event. Save current data as final checkpoint if configure save_final is True. This callback may be skipped because the logic with Ignite can only trigger the first attached handler for EXCEPTION_RAISED event.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.e (
Exception
) – the exception caught in Ignite during engine.run().
- Return type
None
-
interval_completed
(engine)[source]¶ Callback for train epoch/iteration completed Event. Save checkpoint if configure save_interval = N
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
load_state_dict
(state_dict)[source]¶ Utility to resume the internal state of key metric tracking list if configured to save checkpoints based on the key metric value. Note to set key_metric_save_state=True when saving the previous checkpoint.
Example:
CheckpointSaver( ... save_key_metric=True, key_metric_save_state=True, # config to also save the state of this saver ).attach(engine) engine.run(...) # resumed training with a new CheckpointSaver saver = CheckpointSaver(save_key_metric=True, ...) # load the previous key metric tracking list into saver CheckpointLoader("/test/model.pt"), {"checkpointer": saver}).attach(engine)
- Return type
None
Metrics saver¶
-
class
monai.handlers.
MetricsSaver
(save_dir, metrics='*', metric_details=None, batch_transform=<function MetricsSaver.<lambda>>, summary_ops=None, save_rank=0, delimiter='\\t', output_type='csv')[source]¶ ignite handler to save metrics values and details into expected files.
- Parameters
save_dir (
str
) – directory to save the metrics and metric details.metrics (
Union
[Sequence
[str
],str
,None
]) – expected final metrics to save into files, can be: None, “*” or list of strings. None - don’t save any metrics into files. “*” - save all the existing metrics in engine.state.metrics dict into separate files. list of strings - specify the expected metrics to save. default to “*” to save all the metrics into metrics.csv.metric_details (
Union
[Sequence
[str
],str
,None
]) – expected metric details to save into files, the data comes from engine.state.metric_details, which should be provided by different Metrics, typically, it’s some intermediate values in metric computation. for example: mean dice of every channel of every image in the validation dataset. it must contain at least 2 dims: (batch, classes, …), if not, will unsequeeze to 2 dims. this arg can be: None, “*” or list of strings. None - don’t save any metric_details into files. “*” - save all the existing metric_details in engine.state.metric_details dict into separate files. list of strings - specify the metric_details of expected metrics to save. if not None, every metric_details array will save a separate {metric name}_raw.csv file.batch_transform (
Callable
) – callable function to extract the meta_dict from input batch data if saving metric details. used to extract filenames from input dict data.summary_ops (
Union
[Sequence
[str
],str
,None
]) – expected computation operations to generate the summary report based on specified metric_details. it can be: None, “*” or list of strings. None - don’t generate summary report for every specified metric_details “*” - generate summary report for every metric_details with all the supported operations. list of strings - generate summary report for every metric_details with specified operations, they should be within this list: [mean, median, max, min, 90percent, std]. default to None.save_rank (
int
) – only the handler on specified rank will save to files in multi-gpus validation, default to 0.delimiter (
str
) – the delimiter character in CSV file, default to ” “.output_type (
str
) – expected output file type, supported types: [“csv”], default to “csv”.
CSV saver¶
-
class
monai.handlers.
ClassificationSaver
(output_dir='./', filename='predictions.csv', overwrite=True, batch_transform=<function ClassificationSaver.<lambda>>, output_transform=<function ClassificationSaver.<lambda>>, name=None, save_rank=0)[source]¶ Event handler triggered on completing every iteration to save the classification predictions as CSV file. If running in distributed data parallel, only saves CSV file in the specified rank.
- Parameters
output_dir (
str
) – output CSV file directory.filename (
str
) – name of the saved CSV file name.overwrite (
bool
) – whether to overwriting existing CSV file content. If we are not overwriting, then we check if the results have been previously saved, and load them to the prediction_dict.batch_transform (
Callable
) – a callable that is used to transform the ignite.engine.batch into expected format to extract the meta_data dictionary.output_transform (
Callable
) – a callable that is used to transform the ignite.engine.output into the form expected model prediction data. The first dimension of this transform’s output will be treated as the batch dimension. Each item in the batch will be saved individually.name (
Optional
[str
]) – identifier of logging.logger to use, defaulting to engine.logger.save_rank (
int
) – only the handler on specified rank will save to CSV file in multi-gpus validation, default to 0.
Iteration Metric¶
-
class
monai.handlers.
IterationMetric
(metric_fn, output_transform=<function IterationMetric.<lambda>>, device='cpu', save_details=True)[source]¶ Class for metrics that should be computed on every iteration and compute final results when epoch completed. Similar to the EpochMetric in ignite: https://github.com/pytorch/ignite/blob/v0.4.2/ignite/metrics/epoch_metric.py#L13.
- Parameters
metric_fn (
Callable
) – callable function or class to compute raw metric results after every iteration. expect to return a Tensor with shape (batch, channel, …) or tuple (Tensor, not_nans).output_transform (
Callable
) – transform the ignite.engine.state.output into [y_pred, y] pair.device (
Union
[str
,device
]) – device specification in case of distributed computation usage.save_details (
bool
) – whether to save metric computation details per image, for example: mean_dice of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
-
attach
(engine, name)[source]¶ Attaches current metric to provided engine. On the end of engine’s run, engine.state.metrics dictionary will contain computed metric’s value under provided name.
- Parameters
engine (
Engine
) – the engine to which the metric must be attached.name (
str
) – the name of the metric to attach.
- Return type
None
-
compute
()[source]¶ - Raises
NotComputableError – When
compute
is called before anupdate
occurs.- Return type
Any
Mean Dice metrics handler¶
-
class
monai.handlers.
MeanDice
(include_background=True, output_transform=<function MeanDice.<lambda>>, device='cpu', save_details=True)[source]¶ Computes Dice score metric from full size Tensor and collects average over batch, class-channels, iterations.
- Parameters
include_background (
bool
) – whether to include dice computation on the first channel of the predicted output. Defaults to True.output_transform (
Callable
) – transform the ignite.engine.state.output into [y_pred, y] pair.device (
Union
[str
,device
]) – device specification in case of distributed computation usage.save_details (
bool
) – whether to save metric computation details per image, for example: mean dice of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.meandice.compute_meandice()
ROC AUC metrics handler¶
-
class
monai.handlers.
ROCAUC
(average=<Average.MACRO: 'macro'>, output_transform=<function ROCAUC.<lambda>>, device='cpu')[source]¶ Computes Area Under the Receiver Operating Characteristic Curve (ROC AUC). accumulating predictions and the ground-truth during an epoch and applying compute_roc_auc.
- Parameters
average (
Union
[Average
,str
]) –{
"macro"
,"weighted"
,"micro"
,"none"
} Type of averaging performed if not binary classification. Defaults to"macro"
."macro"
: calculate metrics for each label, and find their unweighted mean.This does not take label imbalance into account.
"weighted"
: calculate metrics for each label, and find their average,weighted by support (the number of true instances for each label).
"micro"
: calculate metrics globally by considering each element of the labelindicator matrix as a label.
"none"
: the scores for each class are returned.
output_transform (
Callable
) – a callable that is used to transform theEngine
process_function output into the form expected by the metric. This can be useful if, for example, you have a multi-output model and you want to compute the metric with respect to one of the outputs.device (
Union
[str
,device
]) – device specification in case of distributed computation usage.
Note
ROCAUC expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values.
-
compute
()[source]¶ Computes the metric based on it’s accumulated state.
By default, this is called at the end of each epoch.
- Returns
- the actual quantity of interest. However, if a
Mapping
is returned, it will be (shallow) flattened into engine.state.metrics whencompleted()
is called. - Return type
Any
- Raises
NotComputableError – raised when the metric cannot be computed.
Confusion matrix metrics handler¶
-
class
monai.handlers.
ConfusionMatrix
(include_background=True, metric_name='hit_rate', output_transform=<function ConfusionMatrix.<lambda>>, device='cpu', save_details=True)[source]¶ Compute confusion matrix related metrics from full size Tensor and collects average over batch, class-channels, iterations.
- Parameters
include_background (
bool
) – whether to skip metric computation on the first channel of the predicted output. Defaults to True.metric_name (
str
) – ["sensitivity"
,"specificity"
,"precision"
,"negative predictive value"
,"miss rate"
,"fall out"
,"false discovery rate"
,"false omission rate"
,"prevalence threshold"
,"threat score"
,"accuracy"
,"balanced accuracy"
,"f1 score"
,"matthews correlation coefficient"
,"fowlkes mallows index"
,"informedness"
,"markedness"
] Some of the metrics have multiple aliases (as shown in the wikipedia page aforementioned), and you can also input those names instead.output_transform (
Callable
) – transform the ignite.engine.state.output into [y_pred, y] pair.device (
Union
[str
,device
]) – device specification in case of distributed computation usage.save_details (
bool
) – whether to save metric computation details per image, for example: TP/TN/FP/FN of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
See also
monai.metrics.confusion_matrix()
Hausdorff distance metrics handler¶
-
class
monai.handlers.
HausdorffDistance
(include_background=False, distance_metric='euclidean', percentile=None, directed=False, output_transform=<function HausdorffDistance.<lambda>>, device='cpu', save_details=True)[source]¶ Computes Hausdorff distance from full size Tensor and collects average over batch, class-channels, iterations.
- Parameters
include_background (
bool
) – whether to include distance computation on the first channel of the predicted output. Defaults toFalse
.distance_metric (
str
) – : ["euclidean"
,"chessboard"
,"taxicab"
] the metric used to compute surface distance. Defaults to"euclidean"
.percentile (
Optional
[float
]) – an optional float number between 0 and 100. If specified, the corresponding percentile of the Hausdorff Distance rather than the maximum result will be achieved. Defaults toNone
.directed (
bool
) – whether to calculate directed Hausdorff distance. Defaults toFalse
.output_transform (
Callable
) – transform the ignite.engine.state.output into [y_pred, y] pair.device (
Union
[str
,device
]) – device specification in case of distributed computation usage.save_details (
bool
) – whether to save metric computation details per image, for example: hausdorff distance of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
Surface distance metrics handler¶
-
class
monai.handlers.
SurfaceDistance
(include_background=False, symmetric=False, distance_metric='euclidean', output_transform=<function SurfaceDistance.<lambda>>, device='cpu', save_details=True)[source]¶ Computes surface distance from full size Tensor and collects average over batch, class-channels, iterations.
- Parameters
include_background (
bool
) – whether to include distance computation on the first channel of the predicted output. Defaults toFalse
.symmetric (
bool
) – whether to calculate the symmetric average surface distance between seg_pred and seg_gt. Defaults toFalse
.distance_metric (
str
) – : ["euclidean"
,"chessboard"
,"taxicab"
] the metric used to compute surface distance. Defaults to"euclidean"
.output_transform (
Callable
) – transform the ignite.engine.state.output into [y_pred, y] pair.device (
Union
[str
,device
]) – device specification in case of distributed computation usage.save_details (
bool
) – whether to save metric computation details per image, for example: surface dice of every image. default to True, will save to engine.state.metric_details dict with the metric name as key.
Metric logger¶
-
class
monai.handlers.
MetricLogger
(loss_transform=<function _get_loss_from_output>, metric_transform=<function MetricLogger.<lambda>>, evaluator=None)[source]¶ Collect per-iteration metrics and loss value from the attached trainer. This will also collect metric values from a given evaluator object which is expected to perform evaluation at the end of training epochs. This class is useful for collecting loss and metric values in one place for storage with checkpoint savers (state_dict and load_state_dict methods provided as expected by Pytorch and Ignite) and for graphing during training.
- Example::
# construct an evaluator saving mean dice metric values in the key “val_mean_dice” evaluator = SupervisedEvaluator(…, key_val_metric={“val_mean_dice”: MeanDice(…)})
# construct the logger and associate with evaluator to extract metric values from logger = MetricLogger(evaluator=evaluator)
# construct the trainer with the logger passed in as a handler so that it logs loss values trainer = SupervisedTrainer(…, train_handlers=[logger, ValidationHandler(1, evaluator)])
# run training, logger.loss will be a list of (iteration, loss) values, logger.metrics a dict with key # “val_mean_dice” storing a list of (iteration, metric) values trainer.run()
- Parameters
loss_transform (
Callable
) – Converts the output value from the trainer’s state into a loss valuemetric_transform (
Callable
) – Converts the metric value coming from the trainer/evaluator’s state into a storable valueevaluator (
Optional
[Engine
]) – Optional evaluator to consume metric results from at the end of its evaluation run
-
attach
(engine)[source]¶ - Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
Segmentation saver¶
-
class
monai.handlers.
SegmentationSaver
(output_dir='./', output_postfix='seg', output_ext='.nii.gz', resample=True, mode='nearest', padding_mode=<GridSamplePadMode.BORDER: 'border'>, scale=None, dtype=<class 'numpy.float64'>, output_dtype=<class 'numpy.float32'>, squeeze_end_dims=True, data_root_dir='', batch_transform=<function SegmentationSaver.<lambda>>, output_transform=<function SegmentationSaver.<lambda>>, name=None)[source]¶ Event handler triggered on completing every iteration to save the segmentation predictions into files. It can extract the input image meta data(filename, affine, original_shape, etc.) and resample the predictions based on the meta data.
- Parameters
output_dir (
str
) – output image directory.output_postfix (
str
) – a string appended to all output file names, default to seg.output_ext (
str
) – output file extension name, available extensions: .nii.gz, .nii, .png.resample (
bool
) – whether to resample before saving the data array. if saving PNG format image, based on the spatial_shape from metadata. if saving NIfTI format image, based on the original_affine from metadata.mode (
Union
[GridSampleMode
,InterpolateMode
,str
]) –This option is used when
resample = True
. Defaults to"nearest"
.- NIfTI files {
"bilinear"
,"nearest"
} Interpolation mode to calculate output values. See also: https://pytorch.org/docs/stable/nn.functional.html#grid-sample
- NIfTI files {
- PNG files {
"nearest"
,"linear"
,"bilinear"
,"bicubic"
,"trilinear"
,"area"
} The interpolation mode. See also: https://pytorch.org/docs/stable/nn.functional.html#interpolate
- PNG files {
padding_mode (
Union
[GridSamplePadMode
,str
]) –This option is used when
resample = True
. Defaults to"border"
.- NIfTI files {
"zeros"
,"border"
,"reflection"
} Padding mode for outside grid values. See also: https://pytorch.org/docs/stable/nn.functional.html#grid-sample
- NIfTI files {
- PNG files
This option is ignored.
scale (
Optional
[int
]) – {255
,65535
} postprocess data by clipping to [0, 1] and scaling [0, 255] (uint8) or [0, 65535] (uint16). Default is None to disable scaling. It’s used for PNG format only.dtype (
Union
[dtype
,type
,None
]) – data type for resampling computation. Defaults tonp.float64
for best precision. If None, use the data type of input data. It’s used for Nifti format only.output_dtype (
Union
[dtype
,type
,None
]) – data type for saving data. Defaults tonp.float32
, it’s used for Nifti format only.squeeze_end_dims (
bool
) – if True, any trailing singleton dimensions will be removed (after the channel has been moved to the end). So if input is (C,H,W,D), this will be altered to (H,W,D,C), and then if C==1, it will be saved as (H,W,D). If D also ==1, it will be saved as (H,W). If false, image will always be saved as (H,W,D,C). it’s used for NIfTI format only.data_root_dir (
str
) – if not empty, it specifies the beginning parts of the input file’s absolute path. it’s used to compute input_file_rel_path, the relative path to the file from data_root_dir to preserve folder structure when saving in case there are files in different folders with the same file names. for example: input_file_name: /foo/bar/test1/image.nii, output_postfix: seg output_ext: nii.gz output_dir: /output, data_root_dir: /foo/bar, output will be: /output/test1/image/image_seg.nii.gzbatch_transform (
Callable
) – a callable that is used to transform the ignite.engine.batch into expected format to extract the meta_data dictionary. it can be used to extract the input image meta data: filename, affine, original_shape, etc.output_transform (
Callable
) – a callable that is used to transform the ignite.engine.output into the form expected image data. The first dimension of this transform’s output will be treated as the batch dimension. Each item in the batch will be saved individually.name (
Optional
[str
]) – identifier of logging.logger to use, defaulting to engine.logger.
Training stats handler¶
-
class
monai.handlers.
StatsHandler
(epoch_print_logger=None, iteration_print_logger=None, output_transform=<function StatsHandler.<lambda>>, global_epoch_transform=<function StatsHandler.<lambda>>, name=None, tag_name='Loss', key_var_format='{}: {:.4f} ', logger_handler=None)[source]¶ StatsHandler defines a set of Ignite Event-handlers for all the log printing logics. It’s can be used for any Ignite Engine(trainer, validator and evaluator). And it can support logging for epoch level and iteration level with pre-defined loggers.
- Default behaviors:
When EPOCH_COMPLETED, logs
engine.state.metrics
usingself.logger
.When ITERATION_COMPLETED, logs
self.output_transform(engine.state.output)
usingself.logger
.
- Parameters
epoch_print_logger (
Optional
[Callable
[[Engine
],Any
]]) – customized callable printer for epoch level logging. Must accept parameter “engine”, use default printer if None.iteration_print_logger (
Optional
[Callable
[[Engine
],Any
]]) – customized callable printer for iteration level logging. Must accept parameter “engine”, use default printer if None.output_transform (
Callable
) – a callable that is used to transform theignite.engine.output
into a scalar to print, or a dictionary of {key: scalar}. In the latter case, the output string will be formatted as key: value. By default this value logging happens when every iteration completed.global_epoch_transform (
Callable
) – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to print synced epoch number with the trainer engine.name (
Optional
[str
]) – identifier of logging.logger to use, defaulting toengine.logger
.tag_name (
str
) – when iteration output is a scalar, tag_name is used to print tag_name: scalar_value to logger. Defaults to'Loss'
.key_var_format (
str
) – a formatting string to control the output string format of key: value.logger_handler (
Optional
[Handler
]) – add additional handler to handle the stats data: save to file, etc. Add existing python logging handlers: https://docs.python.org/3/library/logging.handlers.html
-
attach
(engine)[source]¶ Register a set of Ignite Event-Handlers to a specified Ignite engine.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
epoch_completed
(engine)[source]¶ Handler for train or validation/evaluation epoch completed Event. Print epoch level log, default values are from Ignite state.metrics dict.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
exception_raised
(engine, e)[source]¶ Handler for train or validation/evaluation exception raised Event. Print the exception information and traceback. This callback may be skipped because the logic with Ignite can only trigger the first attached handler for EXCEPTION_RAISED event.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.e (
Exception
) – the exception caught in Ignite during engine.run().
- Return type
None
Tensorboard handlers¶
-
class
monai.handlers.
TensorBoardHandler
(summary_writer=None, log_dir='./runs')[source]¶ Base class for the handlers to write data into TensorBoard.
- Parameters
summary_writer (
Optional
[SummaryWriter
]) – user can specify TensorBoard SummaryWriter, default to create a new writer.log_dir (
str
) – if using default SummaryWriter, write logs to this directory, default is ./runs.
-
class
monai.handlers.
TensorBoardStatsHandler
(summary_writer=None, log_dir='./runs', epoch_event_writer=None, epoch_interval=1, iteration_event_writer=None, iteration_interval=1, output_transform=<function TensorBoardStatsHandler.<lambda>>, global_epoch_transform=<function TensorBoardStatsHandler.<lambda>>, tag_name='Loss')[source]¶ TensorBoardStatsHandler defines a set of Ignite Event-handlers for all the TensorBoard logics. It’s can be used for any Ignite Engine(trainer, validator and evaluator). And it can support both epoch level and iteration level with pre-defined TensorBoard event writer. The expected data source is Ignite
engine.state.output
andengine.state.metrics
.- Default behaviors:
When EPOCH_COMPLETED, write each dictionary item in
engine.state.metrics
to TensorBoard.When ITERATION_COMPLETED, write each dictionary item in
self.output_transform(engine.state.output)
to TensorBoard.
- Parameters
summary_writer (
Optional
[SummaryWriter
]) – user can specify TensorBoard SummaryWriter, default to create a new writer.log_dir (
str
) – if using default SummaryWriter, write logs to this directory, default is ./runs.epoch_event_writer (
Optional
[Callable
[[Engine
,SummaryWriter
],Any
]]) – customized callable TensorBoard writer for epoch level. Must accept parameter “engine” and “summary_writer”, use default event writer if None.epoch_interval (
int
) – the epoch interval at which the epoch_event_writer is called. Defaults to 1.iteration_event_writer (
Optional
[Callable
[[Engine
,SummaryWriter
],Any
]]) – customized callable TensorBoard writer for iteration level. Must accept parameter “engine” and “summary_writer”, use default event writer if None.iteration_interval (
int
) – the iteration interval at which the iteration_event_writer is called. Defaults to 1.output_transform (
Callable
) – a callable that is used to transform theignite.engine.output
into a scalar to plot, or a dictionary of {key: scalar}. In the latter case, the output string will be formatted as key: value. By default this value plotting happens when every iteration completed.global_epoch_transform (
Callable
) – a callable that is used to customize global epoch number. For example, in evaluation, the evaluator engine might want to use trainer engines epoch number when plotting epoch vs metric curves.tag_name (
str
) – when iteration output is a scalar, tag_name is used to plot, defaults to'Loss'
.
-
attach
(engine)[source]¶ Register a set of Ignite Event-Handlers to a specified Ignite engine.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
class
monai.handlers.
TensorBoardImageHandler
(summary_writer=None, log_dir='./runs', interval=1, epoch_level=True, batch_transform=<function TensorBoardImageHandler.<lambda>>, output_transform=<function TensorBoardImageHandler.<lambda>>, global_iter_transform=<function TensorBoardImageHandler.<lambda>>, index=0, max_channels=1, max_frames=64)[source]¶ TensorBoardImageHandler is an Ignite Event handler that can visualize images, labels and outputs as 2D/3D images. 2D output (shape in Batch, channel, H, W) will be shown as simple image using the first element in the batch, for 3D to ND output (shape in Batch, channel, H, W, D) input, each of
self.max_channels
number of images’ last three dimensions will be shown as animated GIF along the last axis (typically Depth).It can be used for any Ignite Engine (trainer, validator and evaluator). User can easily add it to engine for any expected Event, for example:
EPOCH_COMPLETED
,ITERATION_COMPLETED
. The expected data source is ignite’sengine.state.batch
andengine.state.output
.- Default behavior:
Show y_pred as images (GIF for 3D) on TensorBoard when Event triggered,
Need to use
batch_transform
andoutput_transform
to specify how many images to show and show which channel.Expects
batch_transform(engine.state.batch)
to return data format: (image[N, channel, …], label[N, channel, …]).Expects
output_transform(engine.state.output)
to return a torch tensor in format (y_pred[N, channel, …], loss).
- Parameters
summary_writer (
Optional
[SummaryWriter
]) – user can specify TensorBoard SummaryWriter, default to create a new writer.log_dir (
str
) – if using default SummaryWriter, write logs to this directory, default is ./runs.interval (
int
) – plot content from engine.state every N epochs or every N iterations, default is 1.epoch_level (
bool
) – plot content from engine.state every N epochs or N iterations. True is epoch level, False is iteration level.batch_transform (
Callable
) – a callable that is used to transform theignite.engine.batch
into expected format to extract several label data.output_transform (
Callable
) – a callable that is used to transform theignite.engine.output
into expected format to extract several output data.global_iter_transform (
Callable
) – a callable that is used to customize global step number for TensorBoard. For example, in evaluation, the evaluator engine needs to know current epoch from trainer.index (
int
) – plot which element in a data batch, default is the first element.max_channels (
int
) – number of channels to plot.max_frames (
int
) – number of frames for 2D-t plot.
LR Schedule handler¶
-
class
monai.handlers.
LrScheduleHandler
(lr_scheduler, print_lr=True, name=None, epoch_level=True, step_transform=<function LrScheduleHandler.<lambda>>)[source]¶ Ignite handler to update the Learning Rate based on PyTorch LR scheduler.
- Parameters
lr_scheduler (
Union
[_LRScheduler
,ReduceLROnPlateau
]) – typically, lr_scheduler should be PyTorch lr_scheduler object. If customized version, must have step and get_last_lr methods.print_lr (
bool
) – whether to print out the latest learning rate with logging.name (
Optional
[str
]) – identifier of logging.logger to use, if None, defaulting toengine.logger
.epoch_level (
bool
) – execute lr_scheduler.step() after every epoch or every iteration. True is epoch level, False is iteration level.step_transform (
Callable
[[Engine
],Any
]) – a callable that is used to transform the information from engine to expected input data of lr_scheduler.step() function if necessary.
- Raises
TypeError – When
step_transform
is notcallable
.
Validation handler¶
-
class
monai.handlers.
ValidationHandler
(interval, validator=None, epoch_level=True)[source]¶ Attach validator to the trainer engine in Ignite. It can support to execute validation every N epochs or every N iterations.
- Parameters
interval (
int
) – do validation every N epochs or every N iterations during training.validator (
Optional
[Evaluator
]) – run the validator when trigger validation, suppose to be Evaluator. if None, should call set_validator() before training.epoch_level (
bool
) – execute validation every N epochs or N iterations. True is epoch level, False is iteration level.
- Raises
TypeError – When
validator
is not amonai.engines.evaluator.Evaluator
.
SmartCache handler¶
-
class
monai.handlers.
SmartCacheHandler
(smartcacher)[source]¶ Attach SmartCache logic to the engine in Ignite. Mainly include the start, update_cache, and shutdown functions of SmartCacheDataset.
- Parameters
smartcacher (
SmartCacheDataset
) – predefined SmartCacheDataset, will attach it to the engine.- Raises
TypeError – When
smartcacher
is not amonai.data.SmartCacheDataset
.
-
attach
(engine)[source]¶ - Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
-
completed
(engine)[source]¶ Callback for train or validation/evaluation completed Event. Stop the replacement thread of SmartCacheDataset.
- Parameters
engine (
Engine
) – Ignite Engine, it can be a trainer, validator or evaluator.- Return type
None
Parameter Scheduler handler¶
-
class
monai.handlers.
ParamSchedulerHandler
(parameter_setter, value_calculator, vc_kwargs, epoch_level=False, name=None, event=<Events.ITERATION_COMPLETED: 'iteration_completed'>)[source]¶ General purpose scheduler for parameters values. By default it can schedule in a linear, exponential, step or multistep function. One can also pass Callables to have customized scheduling logic.
- Parameters
parameter_setter (Callable) – Function that sets the required parameter
value_calculator (Union[str,Callable]) – Either a string (‘linear’, ‘exponential’, ‘step’ or ‘multistep’) or Callable for custom logic.
vc_kwargs (Dict) – Dictionary that stores the required parameters for the value_calculator.
epoch_level (bool) – Whether the the step is based on epoch or iteration. Defaults to False.
name (Optional[str]) – Identifier of logging.logger to use, if None, defaulting to
engine.logger
.event (Optional[str]) – Event to which the handler attaches. Defaults to Events.ITERATION_COMPLETED.
EarlyStop handler¶
-
class
monai.handlers.
EarlyStopHandler
(patience, score_function, trainer=None, min_delta=0.0, cumulative_delta=False, epoch_level=True)[source]¶ EarlyStopHandler acts as an Ignite handler to stop training if no improvement after a given number of events. It‘s based on the EarlyStopping handler in ignite.
- Parameters
patience (
int
) – number of events to wait if no improvement and then stop the training.score_function (
Callable
) – It should be a function taking a single argument, anEngine
object that the handler attached, can be a trainer or validator, and return a score float. an improvement is considered if the score is higher.trainer (
Optional
[Engine
]) – trainer engine to stop the run if no improvement, if None, must call set_trainer() before training.min_delta (
float
) – a minimum increase in the score to qualify as an improvement, i.e. an increase of less than or equal to min_delta, will count as no improvement.cumulative_delta (
bool
) – if True, min_delta defines an increase since the last patience reset, otherwise, it defines an increase after the last event, default to False.epoch_level (
bool
) – check early stopping for every epoch or every iteration of the attached engine, True is epoch level, False is iteration level, default to epoch level.
Note
If in distributed training and uses loss value of every iteration to detect early stopping, the values may be different in different ranks. User may attach this handler to validator engine to detect validation metrics and stop the training, in this case, the score_function is executed on validator engine and trainer is the trainer engine.
GarbageCollector handler¶
-
class
monai.handlers.
GarbageCollector
(trigger_event='epoch', log_level=10)[source]¶ Run garbage collector after each epoch
- Parameters
trigger_event (
str
) – the event that trigger a call to this handler. - “epoch”, after completion of each epoch (equivalent of ignite.engine.Events.EPOCH_COMPLETED) - “iteration”, after completion of each iteration (equivalent of ignite.engine.Events.ITERATION_COMPLETED) - any ignite built-in event from ignite.engine.Events. Defaults to “epoch”.log_level (
int
) – log level (integer) for some garbage collection information as below. Defaults to 10 (DEBUG). - 50 (CRITICAL) - 40 (ERROR) - 30 (WARNING) - 20 (INFO) - 10 (DEBUG) - 0 (NOTSET)
Transform inverter¶
-
class
monai.handlers.
TransformInverter
(transform, loader, output_keys='pred', batch_keys='image', meta_key_postfix='meta_dict', collate_fn=<function no_collation>, postfix='inverted', nearest_interp=True, to_tensor=True, device='cpu', post_func=<function TransformInverter.<lambda>>, num_workers=0)[source]¶ Ignite handler to automatically invert transforms. It takes engine.state.output as the input data and uses the transforms information from engine.state.batch. The inverted data are stored in engine.state.output with key: “{output_key}_{postfix}”. And the inverted meta dict will be stored in engine.state.batch with key: “{output_key}_{postfix}_{meta_key_postfix}”.
- Parameters
transform (
InvertibleTransform
) – a callable data transform on input data.loader (
DataLoader
) – data loader used to run transforms and generate the batch of data.output_keys (
Union
[str
,Sequence
[str
]]) – the key of expected data in ignite.engine.output, invert transforms on it. it also can be a list of keys, will invert transform for each of them. Default to “pred”.batch_keys (
Union
[str
,Sequence
[str
]]) – the key of input data in ignite.engine.batch. will get the applied transforms for this input data, then invert them for the expected data with output_keys. It can also be a list of keys, each matches to the output_keys data. default to “image”.meta_key_postfix (
str
) – use {batch_key}_{postfix} to to fetch the meta data according to the key data, default is meta_dict, the meta data is a dictionary object. For example, to handle key image, read/write affine matrices from the metadata image_meta_dict dictionary’s affine field.collate_fn (
Optional
[Callable
]) – how to collate data after inverse transformations. default won’t do any collation, so the output will be a list of size batch size.postfix (
str
) – will save the inverted result into ignite.engine.output with key {output_key}_{postfix}.nearest_interp (
Union
[bool
,Sequence
[bool
]]) – whether to use nearest interpolation mode when inverting the spatial transforms, default to True. If False, use the same interpolation mode as the original transform. it also can be a list of bool, each matches to the output_keys data.to_tensor (
Union
[bool
,Sequence
[bool
]]) – whether to convert the inverted data into PyTorch Tensor first, default to True. it also can be a list of bool, each matches to the output_keys data.device (
Union
[str
,device
,Sequence
[Union
[str
,device
]]]) – if converted to Tensor, move the inverted results to target device before post_func, default to “cpu”, it also can be a list of string or torch.device, each matches to the output_keys data.post_func (
Union
[Callable
,Sequence
[Callable
]]) – post processing for the inverted data, should be a callable function. it also can be a list of callable, each matches to the output_keys data.num_workers (
Optional
[int
]) – number of workers when run data loader for inverse transforms, default to 0 as only run one iteration and multi-processing may be even slower. Set to None, to use the num_workers of the input transform data loader.