Instance Space Analysis for Algorithm Evaluation

Instance Space Analysis (ISA) is a methodology used to evaluate a portfolio of algorithms and elucidate their strengths and weaknesses. See [1] for more details. The following set of papers use Instance Space Analysis to evaluate algorithms.

ISA3: a 3-dimensional expansion of Instance Space Analysis

Authors: Connor Simpson, Mario Andrés Muñoz, Sevvandi Kandanaarachchi, Ricardo J. G. B. Campello
Venue: Machine Learning, 2025 TLDR: Moving Instance Space Analysis to 3D preserves more information and reveals clearer, previously hidden strengths of algorithms when evaluating diverse algorithm portfolios.
ISA3: a 3-dimensional expansion of instance space analysis
ISA3
A 3D instance space for anomaly detection algorithms.
Evaluating algorithms depends heavily on how diverse and representative the test set is. This work extends Instance Space Analysis into 3D, preserving more information while maintaining interpretability through optimised visualisation. Alongside a new method for identifying algorithm portfolio footprints, the approach reveals previously hidden regions of algorithm strength, demonstrated through a case study on unsupervised anomaly detection.

Benchmarking algorithm portfolio construction methods

Authors: Mario Andrés Muñoz, Hamed Soleimani, Sevvandi Kandanaarachchi
Venue: GECCO, 2022
TLDR: Smartly constructed algorithm portfolios reduce risk and outperform arbitrary ones, with some construction methods producing especially low‑risk yet differently robust portfolios.
Benchmarking algorithm portfolio construction methods
Algorithm portfolios combine multiple algorithms to reduce the risk of poor performance from any single choice. This work benchmarks five portfolio construction methods on standard ASLib scenarios, analysing how well they balance risk (unsolved instances) and robustness (reliance on a dominant algorithm). The results show that carefully constructed portfolios consistently outperform arbitrary ones, with two methods standing out for producing low‑risk portfolios with differing levels of robustness.

On normalization and algorithm selection for unsupervised outlier detection

Authors: Sevvandi Kandanaarachchi, Mario A Muñoz, Rob J Hyndman, Kate Smith-Miles
Venue: Data Mining and Knowledge Discovery, 2020
TLDR: Outlier detection performance depends heavily on both the dataset and how it’s normalised, and analysing these combinations helps identify the best method for each case.
On normalization and algorithm selection for unsupervised outlier detection
ISAAnomalyNorm
The instance space for anomaly detection and normalization variants
The effectiveness of outlier detection methods depends not only on the dataset itself, but also on how the data is normalised. This work shows that normalisation changes key properties such as nearest‑neighbour relationships and data density, directly influencing which points are identified as outliers. By analysing combinations of normalisation schemes and detection methods using instance space analysis, the study reveals their relative strengths and weaknesses and provides guidance on selecting the most suitable method for a given dataset.

Instance Space Analysis for unsupervised outlier detection

Authors: Sevvandi Kandanaarachchi, Mario A Munoz, Kate Smith-Miles
Venue: Evaluation and Experimental Design in Data Mining and Machine Learning, 2019
TLDR: By measuring key dataset properties, this work helps predict which unsupervised outlier detection method will perform best for a given problem.
ISAAnomaly
The instance space for anomaly detection algorithms
Unsupervised outlier detection methods perform best when their definition of an outlier aligns with the structure of the data. This work introduces a set of meta‑features that capture key dataset properties and uses them to conduct the first instance space analysis of unsupervised outlier detection methods. The analysis reveals method‑specific strengths and weaknesses and enables accurate recommendations of suitable outlier detection techniques for different datasets.

References

  1. Smith-Miles, Kate, and Mario Andrés Muñoz. Instance space analysis for algorithm testing: Methodology and software tools, ACM Computing Surveys 55.12 (2023): 1-31.,