From 014d004a2b77be51ebd3ca36f1d6312451f2ccaf Mon Sep 17 00:00:00 2001 From: Olaf Mersmann Date: Wed, 22 Apr 2026 15:18:21 +0200 Subject: [PATCH 1/6] doc: Add documentation for new schema Rudimentary design document and reference for the new schema that attempts to summarize the discussion at the OPL hackathon in April. The core structure was written by the author, then fleshed out using Claude and finally reviewed by again the author. Co-Authored-By: Claude Opus 4.7 --- SCHEMA.md | 301 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 301 insertions(+) create mode 100644 SCHEMA.md diff --git a/SCHEMA.md b/SCHEMA.md new file mode 100644 index 0000000..7d73c8d --- /dev/null +++ b/SCHEMA.md @@ -0,0 +1,301 @@ +# OPL Schema + +The OPL schema catalogs optimization **problems**, **suites**, **generators**, and their **implementations** in a single, machine-readable format. + +Three design choices shape everything below: + +1. **One flat library, keyed by ID.** + Every entity lives in a `Library` dict. + Suites reference problems, problems reference implementations using their respective ID. + There is no embedding of problems or implementations within suites to facilitate reuse. + F.e. an implementation might be referenced by multiple problems or suites. +2. **Numeric fields accept a scalar, a set, or a range.** + A problem may have exactly `2` objectives, one of `{2, 3, 5}`, or any value in `{min: 2, max: 50}`. + The same union type is used for variable dimensions and constraint counts. +3. **Three-valued logic for yes/no fields.** + Many boolean fields (f.e. `hard`, `allows_partial_evaluation`, ...) [`YesNoSome`](#yesnosome) as their value. + We lose some expressive power but simplify the data entry. + If we force authors to decide on yes or no, then we would need more complex structures for variables, constraints etc. and that would make the usual case unnecessarily complex. + +## Contents + +- [Library](#library) +- [Thing types](#thing-types) + - [Implementation](#implementation) + - [ProblemLike](#problemlike) (shared fields) + - [Problem](#problem) + - [Suite](#suite) + - [Generator](#generator) +- [Shared building blocks](#shared-building-blocks) + - [Variable](#variable) / [VariableType](#variabletype) + - [Constraint](#constraint) / [ConstraintType](#constrainttype) + - [Reference](#reference) / [Link](#link) + - [ValueRange](#valuerange) + - [YesNoSome](#yesnosome) + +--- + +## Library + +A `Library` is a dict from ID to a [Thing](#thing-types). +IDs are free-form but must be unique and the convention is to add a prefix marking the type to avoid collisions: + +| Prefix | Type | +|---------|------------------| +| `impl_` | Implementation | +| `fn_` | Problem | +| `suite_`| Suite | +| `gen_` | Generator | + +On load the library validates that every ID referenced by a suite (`problems`) or problem (`implementations`) exists and has the correct type. Suites also have their `fidelity_levels` auto-populated from their problems. + +```yaml +impl_coco: + type: implementation + name: COCO + description: Comparing Continuous Optimisers +fn_sphere: + type: problem + name: Sphere + objectives: [1] + implementations: [impl_coco] +suite_bbob: + type: suite + name: BBOB + problems: [fn_sphere] +``` + +--- + +## Thing types + +All entities inherit from `Thing`, which only carries a discriminator: + +```yaml +type: problem # or: suite | generator | implementation +``` + +We want to have as flat a structure as possible to make exploring and searching OPL as easy as possible. +That's one of the reasonst the top level object is a dictionary of dissimilar things. +But we need to be able to tell them apart so we have a `type` field to discriminate between them. + +### Implementation + +A pointer to code that implements one or more problems. +Intentionally minimal so that the schema describes *what* a problem is, not how to run it. +There are separate files which contain curated usage examples for problems or suites keyed by their respective IDs. + +| Field | Type | Notes | +|-------------------|-----------------------------------|----------------------------------------------| +| `name` | str | required | +| `description` | str | required | +| `language` | str? (e.g. `python`, `c`) | | +| `links` | list of [Link](#link)? | repo, release, docs… | +| `evaluation_time` | str? | free-form ("8 minutes", "fast") | +| `requirements` | str or list of str? | URL to requirements file or list of packages | + +```yaml +impl_coco: + type: implementation + name: COCO + description: Comparing Continuous Optimisers benchmarking platform + language: c + links: + - {type: repository, url: https://github.com/numbbo/coco-experiment} +impl_py_cocoex: + type: implementation + name: Python bindings for COCO + description: The Python bindings for the experimental part of the COCO framework + language: Python + links: + - {type: source, url: https://github.com/numbbo/coco-experiment/tree/main/build/python} + - {type: package, url: https://pypi.org/project/coco-experiment/} +``` + +### ProblemLike + +Fields shared by [Problem](#problem), [Suite](#suite), and [Generator](#generator). +The schema deliberately puts most descriptive fields here so suites can be characterised without explicitly having to add all problems in the suite. + +| Field | Type | Notes | +|------------------------------------------|------------------------------------------------|----------------------------------------------------| +| `name` | str | required | +| `long_name` | str? | | +| `description` | str? (markdown) | longer prose | +| `tags` | set of str? | free-form keywords | +| `references` | set of [Reference](#reference)? | | +| `implementations` | set of IDs? | must resolve to [Implementation](#implementation)s | +| `objectives` | set of int? | e.g. `{1}`, `{2, 3}` — **not** a ValueRange | +| `variables` | set of [Variable](#variable)? | | +| `constraints` | set of [Constraint](#constraint)? | omit entirely for unconstrained | +| `dynamic_type` | set of str? | `{"no"}`, `{"time-varying"}`… | +| `noise_type` | set of str? | `{"none"}`, `{"gaussian"}`… | +| `allows_partial_evaluation` | [YesNoSome](#yesnosome)? | | +| `can_evaluate_objectives_independently` | [YesNoSome](#yesnosome)? | | +| `modality` | set of str? | `{"unimodal"}`, `{"multimodal"}` | +| `fidelity_levels` | set of int? | `{1}` = single-fidelity, `{1,2}` = multi-fidelity | +| `code_examples` | set of str? | paths to example scripts | +| `source` | set of str? | `{"artificial"}`, `{"real-world"}` | + +> `objectives` is a set of integers because we don't assume extreme scalability in this property so explicit enumeration is fine. +> Dimensions of variables on the other hand are ranges because here problems often are scalable over wide ranges. + +### Problem + +One optimization problem (possibly parameterised by instances). + +Adds: + +| Field | Type | Notes | +|-------------|--------------------------------------------|--------------------------------------------| +| `instances` | [ValueRange](#valuerange) or list of str? | e.g. `{min: 1, max: 15}` or named variants | + +```yaml +fn_sphere: + type: problem + name: Sphere + objectives: [1] + variables: [{type: continuous, dim: {min: 2, max: 40}}] + modality: [unimodal] + source: [artificial] + instances: {min: 1, max: 15} + implementations: [impl_coco] +``` + +### Suite + +A curated, fixed collection of problems. + +Adds: + +| Field | Type | Notes | +|------------|--------------|-----------------------------------------------| +| `problems` | set of IDs? | must resolve to [Problem](#problem)s | + +`fidelity_levels` is auto-unioned from member problems at validation time. + +```yaml +suite_bbob: + type: suite + name: BBOB + problems: [fn_sphere, fn_rosenbrock, fn_rastrigin] + objectives: [1] + source: [artificial] + implementations: [impl_coco] +``` + +### Generator + +A parametric family of problems — unlike a [Suite](#suite), the member problems are not enumerated. Uses the same fields as [ProblemLike](#problemlike) with no additions; the distinction from [Problem](#problem) is that a generator produces instances on demand. + +```yaml +gen_mpm2: + type: generator + name: MPM2 + description: Multiple peaks model, second instantiation + objectives: [1] + variables: [{type: continuous, dim: {min: 1}}] + modality: [multimodal] +``` + +--- + +## Shared building blocks + +### Variable + +A group of decision variables of the same type. +Multi-type problems list multiple entries. +While you can have multiple entries of the same type, this should be justified in some way like when you can evaluate the problem on only one subset of variables. + +| Field | Type | Default | +|--------|-----------------------------------------------|----------------------| +| `type` | [VariableType](#variabletype) | `unknown` | +| `dim` | int, set of int, [ValueRange](#valuerange), or null | `0` | + +```yaml +variables: + - {type: continuous, dim: 10} + - {type: integer, dim: {min: 1, max: 5}} +``` + +### VariableType + +`continuous | integer | binary | categorical | unknown`. +Use `unknown` for permutation/combinatorial problems the schema doesn't yet distinguish **and** add an appropriate tag. +We are actively watching for unknown variable types and are open to extending the above list if there is a critical mass of problems to justify it. + +### Constraint + +A group of constraints. +To indicate that the problem is unconstrained, you need an _empty_ `constraints` field. +A missing `constraints` field or if it is set to `null` means it is not known if unconstrained. + +| Field | Type | Notes | +|------------|-----------------------------------------------|------------------------------------| +| `type` | [ConstraintType](#constrainttype) | default `unknown` | +| `hard` | [YesNoSome](#yesnosome)? | hard vs. soft | +| `equality` | [YesNoSome](#yesnosome)? | equality vs. inequality | +| `number` | int, set of int, [ValueRange](#valuerange), null | | + +```yaml +constraints: + - {type: box, hard: yes, number: 10} + - {type: linear, hard: some, equality: no, number: {min: 1}} +``` + +### ConstraintType + +`box | linear | function | unknown`. `function` covers non-linear/black-box constraints. + +### Reference + +Bibliographic pointer. +Requires a `title` and `authors` and optionally includes a `link` to the material. + +```yaml +references: + - title: "Honey Badger Algorithm: New metaheuristic algorithm for solving optimization problems." + authors: + - Fatma A. Hashim + - Essam H. Houssein + - Kashif Hussain + - Mai S. Mabrouk + - Walid Al-Atabany + link: {type: doi, url: "https://doi.org/10.1016/j.matcom.2021.08.013"] +``` + +### Link + +`{type?: str, url: str}`. +`type` is free-form (`repository`, `arxiv`, `paper`, `doi`, ...). +`url` is a URL to some resource. + +If `type` is `doi`, please use the full URL (starting with `https://doi.org/...`) instead of the raw DOI. + +### ValueRange + +An inclusive numeric range type. +At least one of `min`/`max` must be given. +If `min` is given and `max` is missing, it does not imply that there is no upper bound. +There might be one, it is just not known. +The same applies for the case where `max` is given and `min` is missing. + +```yaml +dim: {min: 2} # 2 or more +dim: {min: 2, max: 40} # between 2 and 40 +dim: {max: 100} # up to 100 +``` + +Used by `Variable.dim`, `Constraint.number`, `Problem.instances`. + +### YesNoSome + +Three-valued flag: `yes | no | some | ?` (the last serialises as the literal `'?'` string, meaning unknown). +`some` captures the common case where *part* of something has some property. +For example only some constraints might hard but we don't know the exact number of hard and soft constraints, only the total number. + +```yaml +constraints: [{type: box, hard: some}] +allows_partial_evaluation: "?" +``` From 1f4ad8ac78e8f202f62cc1d577c4bab3dcacd3f9 Mon Sep 17 00:00:00 2001 From: Olaf Mersmann Date: Wed, 22 Apr 2026 15:25:14 +0200 Subject: [PATCH 2/6] fix: Make all reference fields optional Fixes #176 --- SCHEMA.md | 2 +- src/opltools/schema.py | 16 +++++++++------- tests/test_reference.py | 16 ++++++++++------ 3 files changed, 20 insertions(+), 14 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index 7d73c8d..492e8a2 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -251,7 +251,7 @@ constraints: ### Reference Bibliographic pointer. -Requires a `title` and `authors` and optionally includes a `link` to the material. +Requires either a `title` or a `link` and optionally a list of `authors`. ```yaml references: diff --git a/src/opltools/schema.py b/src/opltools/schema.py index 19a1d36..ae90dc3 100644 --- a/src/opltools/schema.py +++ b/src/opltools/schema.py @@ -73,16 +73,18 @@ def __hash__(self): class Reference(BaseModel): - title: str - authors: list[str] + title: str | None = None + authors: list[str] | None = None link: Link | None = None + @model_validator(mode="after") + def _validate(self) -> Self: + if self.title is None and self.link is None: + raise ValueError("References must have either a title or a link.") + return self + def __hash__(self): - return ( - hash(self.title) - + sum([hash(author) for author in self.authors]) - + hash(self.link) - ) + return hash(self.title) + hash(self.link) class Usage(BaseModel): diff --git a/tests/test_reference.py b/tests/test_reference.py index ef6cbd5..0d388a6 100644 --- a/tests/test_reference.py +++ b/tests/test_reference.py @@ -5,13 +5,17 @@ class TestReference: - def test_minimal(self): - ref = Reference(title="A paper", authors=["Alice", "Bob"]) + def test_only_author(self): + ref = Reference(title="A paper") assert ref.title == "A paper" - assert ref.authors == ["Alice", "Bob"] + assert ref.authors is None assert ref.link is None - def test_with_link(self): + def test_only_link(self): + ref = Reference(link=Link(url="https://example.org")) + assert ref.link.url == "https://example.org" + + def test_full(self): ref = Reference( title="A paper", authors=["Alice"], @@ -19,6 +23,6 @@ def test_with_link(self): ) assert ref.link.url == "https://example.org" - def test_requires_authors(self): + def test_requires_title_or_link(self): with pytest.raises(ValidationError): - Reference(title="A paper") + Reference(authors=["A paper"]) From 3343cd8c903968f83b954c1ba0ae5126dffc8261 Mon Sep 17 00:00:00 2001 From: Olaf Mersmann Date: Wed, 22 Apr 2026 15:31:29 +0200 Subject: [PATCH 3/6] feat: Extend evaluation_time to problems Fixes #174 --- SCHEMA.md | 6 +++- examples/emdo.py | 2 +- src/opltools/schema.py | 56 +++++++++++++++++++----------------- tests/test_implementation.py | 2 +- tests/test_library.py | 16 +++++++++++ 5 files changed, 53 insertions(+), 29 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index 492e8a2..ae9c18b 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -91,7 +91,7 @@ There are separate files which contain curated usage examples for problems or su | `description` | str | required | | `language` | str? (e.g. `python`, `c`) | | | `links` | list of [Link](#link)? | repo, release, docs… | -| `evaluation_time` | str? | free-form ("8 minutes", "fast") | +| `evaluation_time` | set of str? | free-form list ("8 minutes", "fast") | | `requirements` | str or list of str? | URL to requirements file or list of packages | ```yaml @@ -135,11 +135,15 @@ The schema deliberately puts most descriptive fields here so suites can be chara | `modality` | set of str? | `{"unimodal"}`, `{"multimodal"}` | | `fidelity_levels` | set of int? | `{1}` = single-fidelity, `{1,2}` = multi-fidelity | | `code_examples` | set of str? | paths to example scripts | +| `evaluation_time` | set of str? | free-form list ("8 minutes", "fast") | | `source` | set of str? | `{"artificial"}`, `{"real-world"}` | > `objectives` is a set of integers because we don't assume extreme scalability in this property so explicit enumeration is fine. > Dimensions of variables on the other hand are ranges because here problems often are scalable over wide ranges. +When no `evaluation_time` is set, it percolates up from any referenced implementations. +The same is true for the `variables` and `constraints` properties of a suite that has references to problems. + ### Problem One optimization problem (possibly parameterised by instances). diff --git a/examples/emdo.py b/examples/emdo.py index 49bbd56..7a4504d 100644 --- a/examples/emdo.py +++ b/examples/emdo.py @@ -37,7 +37,7 @@ name="Electric Motor Design Optimization", description="Not publicly available", language="python", - evaluation_time="8 minutes" + evaluation_time=["8 minutes"] ), "fn_emdo": Problem( name="Electric Motor Design Optimization", diff --git a/src/opltools/schema.py b/src/opltools/schema.py index ae90dc3..a2c5b5b 100644 --- a/src/opltools/schema.py +++ b/src/opltools/schema.py @@ -1,4 +1,5 @@ from enum import Enum +from typing import Any from typing_extensions import Self from pydantic import BaseModel, RootModel, ConfigDict, model_validator @@ -98,7 +99,7 @@ class Implementation(Thing): description: str links: list[Link] | None = None language: str | None = None - evaluation_time: str | None = None + evaluation_time: set[str] | None = None requirements: str | list[str] | None = None @@ -118,6 +119,7 @@ class ProblemLike(Thing): can_evaluate_objectives_independently: YesNoSome | None = None modality: set[str] | None = None fidelity_levels: set[int] | None = None + evaluation_time: set[str] | None = None code_examples: set[str] | None = None source: set[str] | None = None @@ -152,35 +154,32 @@ def _check_id_references(self, ids, type: OPLType) -> None: else: raise ValueError(f"Missing {type.name} with id '{id}'") - # For a given suite, make sure the fidelty_levels property contains - # the fidelity_levels of all problems in the suite. - def _fixup_suite_fidelity(self, suite: Suite): - if suite.problems: - if not suite.fidelity_levels: - suite.fidelity_levels = set() - for pid in suite.problems: - problem = self.root[pid] - assert isinstance(problem, Problem) - if problem.fidelity_levels: - suite.fidelity_levels.update(problem.fidelity_levels) - - return suite - - def _fixup_suite_variables(self, suite: Suite): - if not suite.problems: + def _percolate_set(self, thing: Any, children: set | None, property: str): + """Propagate some `property` from child objects to the parent by calculating the union of all the child property sets. + + This is useful to propagate properties like `variables`, `constraints`, etc. from problems up to the suite. + """ + if children is None: return - if suite.variables is None: - suite.variables = set() - for pid in suite.problems: - problem = self.root[pid] - assert isinstance(problem, Problem) - if problem.variables is not None: - suite.variables.update(problem.variables) + if getattr(thing, property, None) is None: + setattr(thing, property, set()) + thing_set = getattr(thing, property) + + for child_id in children: + child = self.root[child_id] + child_set = getattr(child, property, None) + if child_set is not None: + thing_set.update(child_set) @model_validator(mode="after") def _validate(self) -> Self: - # Make sure all problems referenced in suites exists + # First check and fixup all problems + for id, thing in self.root.items(): + if isinstance(thing, Problem) and thing.implementations: + self._percolate_set(thing, thing.implementations, "evaluation_time") + + # Then check and fixup all suites because changes from the problems need to propagate to the suites for id, thing in self.root.items(): if isinstance(thing, Suite) and thing.problems: for problem_id in thing.problems: @@ -192,7 +191,12 @@ def _validate(self) -> Self: raise ValueError( f"Suite {id} references problem with id '{problem_id}' but id is a {self.root[problem_id].type.name}." ) - self._fixup_suite_fidelity(thing) + + self._percolate_set(thing, thing.problems, "fidelity_levels") + self._percolate_set(thing, thing.problems, "variables") + self._percolate_set(thing, thing.problems, "constraints") + self._percolate_set(thing, thing.problems, "evaluation_time") + return self diff --git a/tests/test_implementation.py b/tests/test_implementation.py index b579fba..63a03e2 100644 --- a/tests/test_implementation.py +++ b/tests/test_implementation.py @@ -17,7 +17,7 @@ def test_full(self): description="desc", links=[Link(url="https://example.org")], language="python", - evaluation_time="fast", + evaluation_time=["fast"], requirements=["numpy", "scipy"], ) assert impl.language == "python" diff --git a/tests/test_library.py b/tests/test_library.py index b7dc328..542bc7b 100644 --- a/tests/test_library.py +++ b/tests/test_library.py @@ -80,3 +80,19 @@ def test_fixup_fidelity_all_problems_without_levels(self): "s1": Suite(name="S1", problems={"p1"}), }) assert lib.root["s1"].fidelity_levels == set() + + def test_fixup_evaluation_time_percolates_from_implementation_to_suite(self): + lib = Library(root={ + "impl1": Implementation( + name="impl1", description="d", evaluation_time={"fast"} + ), + "impl2": Implementation( + name="impl2", description="d", evaluation_time={"8 minutes"} + ), + "p1": Problem(name="P1", implementations={"impl1"}), + "p2": Problem(name="P2", implementations={"impl2"}), + "s1": Suite(name="S1", problems={"p1", "p2"}), + }) + assert lib.root["p1"].evaluation_time == {"fast"} + assert lib.root["p2"].evaluation_time == {"8 minutes"} + assert lib.root["s1"].evaluation_time == {"fast", "8 minutes"} From a3059ed99d765f8932f6e76d5231a9cfcb5cd9d8 Mon Sep 17 00:00:00 2001 From: Olaf Mersmann Date: Wed, 22 Apr 2026 13:50:16 +0200 Subject: [PATCH 4/6] chore: Convert problem.yaml to new schema Adds an LLM generated generator for the problems defined in the old problem.yaml to generate a new style problem.yaml. For future fixes, it is probably best to update examples/problem.py, regenerate the yaml and then merge it with the old yaml by replacing entries based on id. That way additional entries added later are not clobbbered. --- examples/problems.py | 2996 ++++++++++++++++++++++++++ problems.yaml | 4855 +++++++++++++++++++++++++++++++----------- 2 files changed, 6590 insertions(+), 1261 deletions(-) create mode 100644 examples/problems.py diff --git a/examples/problems.py b/examples/problems.py new file mode 100644 index 0000000..0a2d4ea --- /dev/null +++ b/examples/problems.py @@ -0,0 +1,2996 @@ +"""Conversion of problems.yaml into the opltools schema. + +Every entry from the original YAML is preserved as a `#!` comment block +directly above the Python object(s) it was converted into. Where the +original YAML carried information that does not fit into the new schema, +a `FIXME` comment is used to flag the loss. + +IDs use the prefixes: + fn_ - single Problem + suite_ - Suite + gen_ - Generator + impl_ - Implementation +""" + +from opltools import ( + Library, + Problem, + Suite, + Generator, + Implementation, + Reference, + Link, + Variable, + Constraint, + ValueRange, +) +from pydantic_yaml import to_yaml_str + +things = {} + + +# ===================================================================== +# Shared implementations (reused by multiple YAML entries). +# ===================================================================== + +things["impl_coco"] = Implementation( + name="COCO framework", + description="Comparing Continuous Optimizers: black-box optimization benchmarking platform", + language="C/Python", + links=[Link(type="repository", url="https://github.com/numbbo/coco")], +) + +things["impl_coco_legacy"] = Implementation( + name="COCO legacy (bbob-noisy)", + description="Archived COCO download page that hosted the bbob-noisy suite", + language="C/Python", + links=[ + Link( + type="archive", + url="https://web.archive.org/web/20210416065610/https://coco.gforge.inria.fr/doku.php?id=downloads", + ) + ], +) + +things["impl_iohexperimenter"] = Implementation( + name="IOHexperimenter", + description="IOHprofiler experimenter framework", + language="C++/Python", + links=[Link(type="repository", url="https://github.com/IOHprofiler/IOHexperimenter")], +) + +things["impl_pymoo"] = Implementation( + name="pymoo", + description="Multi-objective optimization in Python", + language="Python", + links=[Link(type="repository", url="https://github.com/anyoptimization/pymoo")], +) + +things["impl_mocobench"] = Implementation( + name="mocobench", + description="Multi-objective combinatorial optimization benchmark", + language="C++", + links=[Link(type="repository", url="https://gitlab.com/aliefooghe/mocobench/")], +) + +things["impl_reproblems"] = Implementation( + name="reproblems", + description="Real-world inspired multi-objective optimization problem suite", + language="Python", + links=[Link(type="repository", url="https://github.com/ryojitanabe/reproblems")], +) + + +# ===================================================================== +# Entries +# ===================================================================== + +#! - name: BBOB +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1080/10556788.2020.1808977 +#! implementation: https://github.com/numbbo/coco +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob"] = Suite( + name="BBOB", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + references=[ + Reference( + title="COCO: a platform for comparing continuous optimizers in a black-box setting", + authors=[], + link=Link(url="https://doi.org/10.1080/10556788.2020.1808977"), + ) + ], + implementations={"impl_coco"}, +) + +#! - name: BBOB-biobj +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: 2-40 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.48550/arXiv.1604.00359 +#! implementation: https://github.com/numbbo/coco +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob_biobj"] = Suite( + name="BBOB-biobj", + objectives={2}, + variables=[Variable(type="continuous", dim=ValueRange(min=2, max=40))], + modality={"multimodal"}, + references=[ + Reference( + title="BBOB bi-objective test suite", + authors=[], + link=Link(url="https://doi.org/10.48550/arXiv.1604.00359"), + ) + ], + implementations={"impl_coco"}, +) + +#! - name: BBOB-noisy +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'yes' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://hal.inria.fr/inria-00369466 +#! implementation: https://web.archive.org/web/20210416065610/https://coco.gforge.inria.fr/doku.php?id=downloads +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob_noisy"] = Suite( + name="BBOB-noisy", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + noise_type={"noisy"}, + references=[ + Reference( + title="Real-parameter black-box optimization benchmarking: noisy functions definitions", + authors=[], + link=Link(url="https://hal.inria.fr/inria-00369466"), + ) + ], + implementations={"impl_coco_legacy"}, +) + +#! - name: BBOB-largescale +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 20-640 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.48550/arXiv.1903.06396 +#! implementation: https://github.com/numbbo/coco +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob_largescale"] = Suite( + name="BBOB-largescale", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=20, max=640))], + modality={"multimodal"}, + references=[ + Reference( + title="BBOB large-scale test suite", + authors=[], + link=Link(url="https://doi.org/10.48550/arXiv.1903.06396"), + ) + ], + implementations={"impl_coco"}, +) + +#! - name: BBOB-mixint +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 5-160 +#! variable type: integer;continuous;mixed +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3321707.3321868 +#! implementation: https://github.com/numbbo/coco +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob_mixint"] = Suite( + name="BBOB-mixint", + objectives={1}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=5, max=160)), + Variable(type="integer", dim=ValueRange(min=5, max=160)), + ], + modality={"multimodal"}, + references=[ + Reference( + title="BBOB mixed-integer test suite", + authors=[], + link=Link(url="https://doi.org/10.1145/3321707.3321868"), + ) + ], + implementations={"impl_coco"}, +) + +#! - name: BBOB-biobj-mixint +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: 5-160 +#! variable type: integer;continuous;mixed +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3321707.3321868 +#! implementation: https://github.com/numbbo/coco +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob_biobj_mixint"] = Suite( + name="BBOB-biobj-mixint", + objectives={2}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=5, max=160)), + Variable(type="integer", dim=ValueRange(min=5, max=160)), + ], + modality={"multimodal"}, + references=[ + Reference( + title="BBOB bi-objective mixed-integer test suite", + authors=[], + link=Link(url="https://doi.org/10.1145/3321707.3321868"), + ) + ], + implementations={"impl_coco"}, +) + +#! - name: BBOB-constrained +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 2-40 +#! variable type: continuous +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: http://numbbo.github.io/coco-doc/bbob-constrained/ +#! implementation: https://github.com/numbbo/coco +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_bbob_constrained"] = Suite( + name="BBOB-constrained", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=2, max=40))], + constraints=[Constraint(hard="yes")], + modality={"multimodal"}, + references=[ + Reference( + title="bbob-constrained documentation", + authors=[], + link=Link(url="http://numbbo.github.io/coco-doc/bbob-constrained/"), + ) + ], + implementations={"impl_coco"}, +) + +#! - name: MOrepo +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: '?' +#! variable type: combinatorial +#! constraints: '?' +#! dynamic: '?' +#! noise: '?' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: '' +#! implementation: https://github.com/MCDMSociety/MOrepo +#! source (real-world/artificial): '' +#! textual description: '' +things["impl_morepo"] = Implementation( + name="MOrepo", + description="Multi-objective optimisation problem repository", + links=[Link(type="repository", url="https://github.com/MCDMSociety/MOrepo")], +) +# FIXME: "combinatorial" has no direct VariableType; dimensionality "?" unknown. +things["suite_morepo"] = Suite( + name="MOrepo", + objectives={2}, + variables=[Variable(type="unknown")], + constraints=[Constraint(hard="?")], + dynamic_type={"unknown"}, + noise_type={"unknown"}, + implementations={"impl_morepo"}, +) + +#! - name: ZDT +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: scalable +#! variable type: continuous;binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1162/106365600568202 +#! implementation: https://github.com/anyoptimization/pymoo +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_zdt"] = Suite( + name="ZDT", + objectives={2}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + references=[ + Reference( + title="Comparison of multiobjective evolutionary algorithms: empirical results", + authors=["Eckart Zitzler", "Kalyanmoy Deb", "Lothar Thiele"], + link=Link(url="https://doi.org/10.1162/106365600568202"), + ) + ], + implementations={"impl_pymoo"}, +) + +#! - name: DTLZ +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/CEC.2002.1007032 +#! implementation: https://pymoo.org/problems/many/dtlz.html +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_dtlz"] = Suite( + name="DTLZ", + # FIXME: original "2+" - schema requires set[int]; truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Scalable multi-objective optimization test problems", + authors=["Kalyanmoy Deb", "Lothar Thiele", "Marco Laumanns", "Eckart Zitzler"], + link=Link(url="https://doi.org/10.1109/CEC.2002.1007032"), + ) + ], + implementations={"impl_pymoo"}, +) + +#! - name: WFG +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2005.861417 +#! implementation: https://pymoo.org/problems/many/wfg.html +#! source (real-world/artificial): '' +#! textual description: '' +things["suite_wfg"] = Suite( + name="WFG", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="A review of multiobjective test problems and a scalable test problem toolkit", + authors=["Simon Huband", "Philip Hingston", "Luigi Barone", "Lyndon While"], + link=Link(url="https://doi.org/10.1109/TEVC.2005.861417"), + ) + ], + implementations={"impl_pymoo"}, +) + +#! - name: CDMP +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'yes' +#! dynamic: '?' +#! noise: '?' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3321707.3321878 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: implementation unknown. +things["suite_cdmp"] = Suite( + name="CDMP", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + constraints=[Constraint(hard="yes")], + dynamic_type={"unknown"}, + noise_type={"unknown"}, + references=[ + Reference( + title="CDMP benchmark", + authors=[], + link=Link(url="https://doi.org/10.1145/3321707.3321878"), + ) + ], +) + +#! - name: SDP +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'yes' +#! noise: '?' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TCYB.2019.2896021 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: implementation unknown. +things["suite_sdp"] = Suite( + name="SDP", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + dynamic_type={"dynamic"}, + noise_type={"unknown"}, + references=[ + Reference( + title="SDP dynamic multi-objective benchmark", + authors=[], + link=Link(url="https://doi.org/10.1109/TCYB.2019.2896021"), + ) + ], +) + +#! - name: MaOP +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: '?' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1016/j.swevo.2019.02.003 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: implementation unknown. +things["suite_maop"] = Suite( + name="MaOP", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + noise_type={"unknown"}, + references=[ + Reference( + title="MaOP benchmark", + authors=[], + link=Link(url="https://doi.org/10.1016/j.swevo.2019.02.003"), + ) + ], +) + +#! - name: BP +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: '?' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/CEC.2019.8790277 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: implementation unknown. +things["suite_bp"] = Suite( + name="BP", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + noise_type={"unknown"}, + references=[ + Reference( + title="BP benchmark", + authors=[], + link=Link(url="https://doi.org/10.1109/CEC.2019.8790277"), + ) + ], +) + +#! - name: GPD +#! suite/generator/single: generator +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: optional +#! dynamic: 'no' +#! noise: optional +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1016/j.asoc.2020.106139 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: implementation unknown. +things["gen_gpd"] = Generator( + name="GPD", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + constraints=[Constraint(hard="some")], + noise_type={"optional"}, + references=[ + Reference( + title="GPD generator", + authors=[], + link=Link(url="https://doi.org/10.1016/j.asoc.2020.106139"), + ) + ], +) + +#! - name: ETMOF +#! suite/generator/single: suite +#! objectives: 2-50 +#! dimensionality: 25-10000 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'yes' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.48550/arXiv.2110.08033 +#! implementation: https://github.com/songbai-liu/etmo +#! source (real-world/artificial): '' +#! textual description: '' +things["impl_etmof"] = Implementation( + name="ETMOF", + description="Evolutionary many-task optimization framework", + links=[Link(type="repository", url="https://github.com/songbai-liu/etmo")], +) +things["suite_etmof"] = Suite( + name="ETMOF", + objectives=set(range(2, 51)), + variables=[Variable(type="continuous", dim=ValueRange(min=25, max=10000))], + dynamic_type={"dynamic"}, + references=[ + Reference( + title="Evolutionary many-task optimization framework", + authors=[], + link=Link(url="https://doi.org/10.48550/arXiv.2110.08033"), + ) + ], + implementations={"impl_etmof"}, +) + +#! - name: MMOPP +#! suite/generator/single: suite +#! objectives: 2-7 +#! dimensionality: '?' +#! variable type: '?' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412 +#! implementation: http://www5.zzu.edu.cn/ecilab/info/1036/1251.htm +#! source (real-world/artificial): '' +#! textual description: '' +things["impl_mmopp"] = Implementation( + name="MMOPP", + description="ECI lab distribution page for MMOPP", + links=[Link(type="website", url="http://www5.zzu.edu.cn/ecilab/info/1036/1251.htm")], +) +# FIXME: variable type and dimensionality unknown ("?"). +things["suite_mmopp"] = Suite( + name="MMOPP", + objectives=set(range(2, 8)), + variables=[Variable(type="unknown")], + constraints=[Constraint(hard="yes")], + modality={"multimodal"}, + references=[ + Reference( + title="MMOPP technical report", + authors=[], + link=Link( + url="http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412" + ), + ) + ], + implementations={"impl_mmopp"}, +) + +#! - name: CFD +#! suite/generator/single: suite +#! objectives: 1-2 +#! dimensionality: scalable +#! variable type: '?' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1007/978-3-319-99259-4_24 +#! implementation: https://bitbucket.org/arahat/cfd-test-problem-suite +#! source (real-world/artificial): real world +#! textual description: expensive evaluations 30s-15m +things["impl_cfd"] = Implementation( + name="CFD test problem suite", + description="Expensive real-world CFD-based test problems", + evaluation_time=["30s", "15m"], + links=[Link(type="repository", url="https://bitbucket.org/arahat/cfd-test-problem-suite")], +) +# FIXME: variable type unknown. +things["suite_cfd"] = Suite( + name="CFD", + description="expensive evaluations 30s-15m", + objectives={1, 2}, + variables=[Variable(type="unknown", dim=ValueRange(min=1))], + constraints=[Constraint(hard="yes")], + source={"real-world"}, + references=[ + Reference( + title="CFD test problem suite", + authors=[], + link=Link(url="https://doi.org/10.1007/978-3-319-99259-4_24"), + ) + ], + implementations={"impl_cfd"}, +) + +#! - name: GBEA +#! suite/generator/single: suite +#! objectives: 1-2 +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'yes' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3321707.3321805 +#! implementation: 'https://github.com/ttusar/coco-gbea' +#! source (real-world/artificial): real world +#! textual description: 'expensive evaluations 5s-35s, RW-GAN-Mario and TopTrumps are part of GBEA' +things["impl_gbea"] = Implementation( + name="coco-gbea", + description="Game-Benchmark for Evolutionary Algorithms (COCO fork)", + evaluation_time=["5 seconds", "34 seconds"], + links=[Link(type="repository", url="https://github.com/ttusar/coco-gbea")], +) +things["suite_gbea"] = Suite( + name="GBEA", + description="expensive evaluations 5s-35s, RW-GAN-Mario and TopTrumps are part of GBEA", + objectives={1, 2}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + noise_type={"noisy"}, + modality={"multimodal"}, + source={"real-world"}, + references=[ + Reference( + title="Game benchmark for evolutionary algorithms", + authors=[], + link=Link(url="https://doi.org/10.1145/3321707.3321805"), + ) + ], + implementations={"impl_gbea"}, +) + +#! - name: Car structure +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: 144-222 +#! variable type: discrete +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3205651.3205702 +#! implementation: http://ladse.eng.isas.jaxa.jp/benchmark/ +#! source (real-world/artificial): real world +#! textual description: 54 constraints +things["impl_car_structure"] = Implementation( + name="Car-structure benchmark", + description="JAXA LADSE benchmark problems", + links=[Link(type="website", url="http://ladse.eng.isas.jaxa.jp/benchmark/")], +) +# FIXME: "discrete" has no direct VariableType - using integer. +things["suite_car_structure"] = Suite( + name="Car structure", + description="54 constraints", + objectives={2}, + variables=[Variable(type="integer", dim=ValueRange(min=144, max=222))], + constraints=[Constraint(hard="yes", number=54)], + source={"real-world"}, + references=[ + Reference( + title="Car structure design benchmark", + authors=[], + link=Link(url="https://doi.org/10.1145/3205651.3205702"), + ) + ], + implementations={"impl_car_structure"}, +) + +#! - name: EMO2017 +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: 4-24 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/ +#! implementation: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/downloads/realworld-problems-bbcomp-EMO-2017.zip +#! source (real-world/artificial): real world +#! textual description: '' +things["impl_emo2017"] = Implementation( + name="EMO 2017 real-world problems", + description="BBComp EMO-2017 real-world problem archive", + links=[ + Link( + type="download", + url="https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/downloads/realworld-problems-bbcomp-EMO-2017.zip", + ) + ], +) +things["suite_emo2017"] = Suite( + name="EMO2017", + objectives={2}, + variables=[Variable(type="continuous", dim=ValueRange(min=4, max=24))], + source={"real-world"}, + references=[ + Reference( + title="BBComp EMO 2017", + authors=[], + link=Link(url="https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/"), + ) + ], + implementations={"impl_emo2017"}, +) + +#! - name: JSEC2019 +#! suite/generator/single: single +#! objectives: 1-5 +#! dimensionality: '32' +#! variable type: continuous +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html +#! implementation: http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html +#! source (real-world/artificial): real world +#! textual description: expensive evaluations 3s; 22 constraints +things["impl_jsec2019"] = Implementation( + name="JSEC 2019 competition", + description="JPNSEC EC-Symposium 2019 competition problem", + evaluation_time=["3s"], + links=[ + Link( + type="website", + url="http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html", + ) + ], +) +things["fn_jsec2019"] = Problem( + name="JSEC2019", + description="expensive evaluations 3s; 22 constraints", + objectives={1, 2, 3, 4, 5}, + variables=[Variable(type="continuous", dim=32)], + constraints=[Constraint(hard="yes", number=22)], + source={"real-world"}, + references=[ + Reference( + title="JPNSEC EC-Symposium 2019 competition", + authors=[], + link=Link( + url="http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html" + ), + ) + ], + implementations={"impl_jsec2019"}, +) + +#! - name: RE +#! suite/generator/single: suite +#! objectives: 2-9 +#! dimensionality: 2-7 +#! variable type: continuous;integer;mixed +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1016/j.asoc.2020.106078 +#! implementation: https://github.com/ryojitanabe/reproblems +#! source (real-world/artificial): real world like +#! textual description: '' +things["suite_re"] = Suite( + name="RE", + objectives=set(range(2, 10)), + variables=[ + Variable(type="continuous", dim=ValueRange(min=2, max=7)), + Variable(type="integer", dim=ValueRange(min=2, max=7)), + ], + source={"real-world-like"}, + references=[ + Reference( + title="Easy-to-evaluate real-world multi-objective optimization problems", + authors=["Ryoji Tanabe", "Hisao Ishibuchi"], + link=Link(url="https://doi.org/10.1016/j.asoc.2020.106078"), + ) + ], + implementations={"impl_reproblems"}, +) + +#! - name: CRE +#! suite/generator/single: suite +#! objectives: 2-5 +#! dimensionality: 3-7 +#! variable type: continuous;integer;mixed +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1016/j.asoc.2020.106078 +#! implementation: https://github.com/ryojitanabe/reproblems +#! source (real-world/artificial): real world like +#! textual description: '' +things["suite_cre"] = Suite( + name="CRE", + objectives={2, 3, 4, 5}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=3, max=7)), + Variable(type="integer", dim=ValueRange(min=3, max=7)), + ], + constraints=[Constraint(hard="yes")], + source={"real-world-like"}, + references=[ + Reference( + title="Easy-to-evaluate real-world multi-objective optimization problems", + authors=["Ryoji Tanabe", "Hisao Ishibuchi"], + link=Link(url="https://doi.org/10.1016/j.asoc.2020.106078"), + ) + ], + implementations={"impl_reproblems"}, +) + +#! - name: Radar waveform +#! suite/generator/single: single +#! objectives: '9' +#! dimensionality: 4-12 +#! variable type: integer +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1007/978-3-540-70928-2_53 +#! implementation: http://code.evanhughes.org/ +#! source (real-world/artificial): real world +#! textual description: '' +things["impl_radar_waveform"] = Implementation( + name="Evan Hughes radar waveform code", + description="Radar waveform design reference implementation", + links=[Link(type="website", url="http://code.evanhughes.org/")], +) +things["fn_radar_waveform"] = Problem( + name="Radar waveform", + objectives={9}, + variables=[Variable(type="integer", dim=ValueRange(min=4, max=12))], + constraints=[Constraint(hard="yes")], + source={"real-world"}, + references=[ + Reference( + title="Radar waveform design", + authors=[], + link=Link(url="https://doi.org/10.1007/978-3-540-70928-2_53"), + ) + ], + implementations={"impl_radar_waveform"}, +) + +#! - name: MF2 +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 1-n +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'yes' +#! reference: https://doi.org/10.21105/joss.02049 +#! implementation: https://github.com/sjvrijn/mf2 +#! source (real-world/artificial): '' +#! textual description: '' +things["impl_mf2"] = Implementation( + name="mf2", + description="Multi-fidelity test function collection", + language="Python", + links=[Link(type="repository", url="https://github.com/sjvrijn/mf2")], +) +things["suite_mf2"] = Suite( + name="MF2", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + fidelity_levels={1, 2}, + references=[ + Reference( + title="mf2: a collection of multi-fidelity benchmark functions in Python", + authors=[], + link=Link(url="https://doi.org/10.21105/joss.02049"), + ) + ], + implementations={"impl_mf2"}, +) + +#! - name: AMVOP +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: mixed continuous+ordinal+categorical+both +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2013.2281531 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: implementation unknown. "ordinal" not representable, using integer+categorical+continuous. +things["suite_amvop"] = Suite( + name="AMVOP", + objectives={1}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="integer", dim=ValueRange(min=1)), + Variable(type="categorical", dim=ValueRange(min=1)), + ], + modality={"multimodal"}, + references=[ + Reference( + title="AMVOP", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281531"), + ) + ], +) + +#! - name: RWMVOP +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous;mixed continuous+ordinal+categorical+both +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2013.2281531 +#! implementation: '?' +#! source (real-world/artificial): real world +#! textual description: '' +# FIXME: implementation unknown. +things["suite_rwmvop"] = Suite( + name="RWMVOP", + objectives={1}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="integer", dim=ValueRange(min=1)), + Variable(type="categorical", dim=ValueRange(min=1)), + ], + constraints=[Constraint(hard="yes")], + source={"real-world"}, + references=[ + Reference( + title="RWMVOP", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281531"), + ) + ], +) + +#! - name: SBOX-COST +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.48550/arXiv.2305.12221 +#! implementation: https://github.com/IOHprofiler/IOHexperimenter/ +#! source (real-world/artificial): '' +#! textual description: problems from BBOB but allows instances with the optimum close to the +#! boundary +things["suite_sbox_cost"] = Suite( + name="SBOX-COST", + description="problems from BBOB but allows instances with the optimum close to the boundary", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + references=[ + Reference( + title="SBOX-COST", + authors=[], + link=Link(url="https://doi.org/10.48550/arXiv.2305.12221"), + ) + ], + implementations={"impl_iohexperimenter"}, +) + +#! - name: "\u03C1MNK-Landscapes" +#! suite/generator/single: generator +#! objectives: scalable +#! dimensionality: scalable +#! variable type: binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1016/j.ejor.2012.12.019 +#! implementation: https://gitlab.com/aliefooghe/mocobench/ +#! source (real-world/artificial): '' +#! textual description: tunable variable and objective dimensions; tunable multimodality and +#! correlation between objectives +things["gen_rho_mnk_landscapes"] = Generator( + name="ρMNK-Landscapes", + description="tunable variable and objective dimensions; tunable multimodality and correlation between objectives", + # FIXME: original "scalable" - truncated to 1..10. + objectives=set(range(1, 11)), + variables=[Variable(type="binary", dim=ValueRange(min=1))], + modality={"multimodal"}, + references=[ + Reference( + title="On the design of multi-objective evolutionary algorithms based on NK-landscapes", + authors=[], + link=Link(url="https://doi.org/10.1016/j.ejor.2012.12.019"), + ) + ], + implementations={"impl_mocobench"}, +) + +#! - name: mUBQP +#! suite/generator/single: generator +#! objectives: scalable +#! dimensionality: scalable +#! variable type: binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: yes (quadratic) +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1016/j.asoc.2013.11.008 +#! implementation: https://gitlab.com/aliefooghe/mocobench/ +#! source (real-world/artificial): '' +#! textual description: tunable variable and objective dimensions; tunable density and correlation +#! between objectives +things["gen_mubqp"] = Generator( + name="mUBQP", + description="tunable variable and objective dimensions; tunable density and correlation between objectives", + # FIXME: original "scalable" - truncated to 1..10. + objectives=set(range(1, 11)), + variables=[Variable(type="binary", dim=ValueRange(min=1))], + modality={"multimodal", "quadratic"}, + references=[ + Reference( + title="mUBQP benchmark", + authors=[], + link=Link(url="https://doi.org/10.1016/j.asoc.2013.11.008"), + ) + ], + implementations={"impl_mocobench"}, +) + +#! - name: "\u03C1mTSP" +#! suite/generator/single: generator +#! objectives: scalable +#! dimensionality: scalable +#! variable type: permutations +#! constraints: no (apart from being permutations) +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: yes (quadratic) +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1007/978-3-319-45823-6_40 +#! implementation: https://gitlab.com/aliefooghe/mocobench/ +#! source (real-world/artificial): '' +#! textual description: tunable variable and objective dimensions; tunable instance type (euclidian/random); +#! tunable correlation between objectives +# FIXME: "permutations" has no direct VariableType; constraints are implicit permutations. +things["gen_rho_mtsp"] = Generator( + name="ρmTSP", + description="tunable variable and objective dimensions; tunable instance type (euclidean/random); tunable correlation between objectives", + # FIXME: original "scalable" - truncated to 1..10. + objectives=set(range(1, 11)), + variables=[Variable(type="unknown", dim=ValueRange(min=1))], + modality={"multimodal", "quadratic"}, + references=[ + Reference( + title="On the impact of multi-objective scalability for the ρmTSP", + authors=[], + link=Link(url="https://doi.org/10.1007/978-3-319-45823-6_40"), + ) + ], + implementations={"impl_mocobench"}, +) + +#! - name: CEC2015-DMOO +#! suite/generator/single: suite +#! objectives: 2-3 +#! dimensionality: '?' +#! variable type: continuous +#! constraints: '?' +#! dynamic: 'yes' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: Benchmark Functions for CEC 2015 Special Session and Competition on Dynamic +#! Multi-objective Optimization +#! implementation: '' +#! source (real-world/artificial): '' +#! textual description: '' +# FIXME: reference is a title-only string; implementation unavailable; dimensionality unknown. +things["suite_cec2015_dmoo"] = Suite( + name="CEC2015-DMOO", + objectives={2, 3}, + variables=[Variable(type="continuous")], + constraints=[Constraint(hard="?")], + dynamic_type={"dynamic"}, + references=[ + Reference( + title="Benchmark Functions for CEC 2015 Special Session and Competition on Dynamic Multi-objective Optimization", + authors=[], + ) + ], +) + +#! - name: Ealain +#! suite/generator/single: generator +#! objectives: 1+ +#! dimensionality: scalable +#! variable type: continuous,binary,integer +#! constraints: optional +#! dynamic: optional +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: optional +#! reference: https://doi.org/10.1145/3638530.3654299 +#! implementation: https://github.com/qrenau/Ealain +#! source (real-world/artificial): Real-world-like +#! textual description: Real-world-like, easily extensible to increase complexity +things["impl_ealain"] = Implementation( + name="Ealain", + description="Real-world-like extensible benchmark problem generator", + links=[Link(type="repository", url="https://github.com/qrenau/Ealain")], +) +things["gen_ealain"] = Generator( + name="Ealain", + description="Real-world-like, easily extensible to increase complexity", + # FIXME: original "1+" - truncated to 1..10. + objectives=set(range(1, 11)), + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + Variable(type="integer", dim=ValueRange(min=1)), + ], + constraints=[Constraint(hard="some")], + dynamic_type={"optional"}, + fidelity_levels={1, 2}, + source={"real-world-like"}, + references=[ + Reference( + title="Ealain", + authors=[], + link=Link(url="https://doi.org/10.1145/3638530.3654299"), + ) + ], + implementations={"impl_ealain"}, +) + +#! - name: MA-BBOB +#! suite/generator/single: generator +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3673908 +#! implementation: https://github.com/IOHprofiler/IOHexperimenter/blob/master/example/Competitions/MA-BBOB/Example_MABBOB.ipynb +#! source (real-world/artificial): artificial +#! textual description: Generator that creates affine combinations of BBOB functions +things["impl_ma_bbob"] = Implementation( + name="MA-BBOB (IOHexperimenter)", + description="Example notebook for MA-BBOB in IOHexperimenter", + links=[ + Link( + type="example", + url="https://github.com/IOHprofiler/IOHexperimenter/blob/master/example/Competitions/MA-BBOB/Example_MABBOB.ipynb", + ) + ], +) +things["gen_ma_bbob"] = Generator( + name="MA-BBOB", + description="Generator that creates affine combinations of BBOB functions", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + source={"artificial"}, + references=[ + Reference( + title="MA-BBOB", + authors=[], + link=Link(url="https://doi.org/10.1145/3673908"), + ) + ], + implementations={"impl_ma_bbob", "impl_iohexperimenter"}, +) + +#! - name: MPM2 +#! suite/generator/single: generator +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf +#! implementation: https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py +#! source (real-world/artificial): '' +#! textual description: nonlinear nonseparable nonsymmetric; scalable in terms of time to evaluate +#! the objective function +things["impl_mpm2"] = Implementation( + name="MPM2 (smoof)", + description="Python implementation of MPM2 distributed with smoof", + language="Python", + links=[ + Link( + type="source", + url="https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py", + ) + ], +) +things["gen_mpm2"] = Generator( + name="MPM2", + description="nonlinear nonseparable nonsymmetric; scalable in terms of time to evaluate the objective function", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + references=[ + Reference( + title="MPM2 technical report TR15-01", + authors=[], + link=Link(url="https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf"), + ) + ], + implementations={"impl_mpm2"}, +) + +#! - name: Convex DTLZ2 +#! suite/generator/single: single +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2013.2281535 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of DTLZ2 with a convex Pareto front (instead of concave) +# FIXME: implementation unknown. +things["fn_convex_dtlz2"] = Problem( + name="Convex DTLZ2", + description="Variant of DTLZ2 with a convex Pareto front (instead of concave)", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Convex DTLZ2", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281535"), + ) + ], +) + +#! - name: Inverted DTLZ1 +#! suite/generator/single: single +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2013.2281534 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of DTLZ1 with an inverted Pareto front +# FIXME: implementation unknown. +things["fn_inverted_dtlz1"] = Problem( + name="Inverted DTLZ1", + description="Variant of DTLZ1 with an inverted Pareto front", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Inverted DTLZ1", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281534"), + ) + ], +) + +#! - name: Minus DTLZ +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2016.2587749 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of DTLZ that minimises the inverse of the base DTLZ functions +# FIXME: implementation unknown. +things["suite_minus_dtlz"] = Suite( + name="Minus DTLZ", + description="Variant of DTLZ that minimises the inverse of the base DTLZ functions", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Minus DTLZ / Minus WFG", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2016.2587749"), + ) + ], +) + +#! - name: Minus WFG +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2016.2587749 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of WFG that minimises the inverse of the base WFG functions +# FIXME: implementation unknown. +things["suite_minus_wfg"] = Suite( + name="Minus WFG", + description="Variant of WFG that minimises the inverse of the base WFG functions", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Minus DTLZ / Minus WFG", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2016.2587749"), + ) + ], +) + +#! - name: L1-ZDT +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: scalable +#! variable type: continuous;binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/1143997.1144179 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of ZDT with linkages between variables within one of two groups +#! but not between variables in a different group; Linear recombination operators +#! can potentially take advantage of the problem structure +# FIXME: implementation unknown. +things["suite_l1_zdt"] = Suite( + name="L1-ZDT", + description="Variant of ZDT with linkages between variables within groups", + objectives={2}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + references=[ + Reference( + title="Linkage ZDT/DTLZ variants", + authors=[], + link=Link(url="https://doi.org/10.1145/1143997.1144179"), + ) + ], +) + +#! - name: L2-ZDT +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: scalable +#! variable type: continuous;binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/1143997.1144179 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of ZDT with linkages between all variables; Linear recombination +#! operators can potentially take advantage of the problem structure +# FIXME: implementation unknown. +things["suite_l2_zdt"] = Suite( + name="L2-ZDT", + description="Variant of ZDT with linkages between all variables", + objectives={2}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + references=[ + Reference( + title="Linkage ZDT/DTLZ variants", + authors=[], + link=Link(url="https://doi.org/10.1145/1143997.1144179"), + ) + ], +) + +#! - name: L3-ZDT +#! suite/generator/single: suite +#! objectives: '2' +#! dimensionality: scalable +#! variable type: continuous;binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/1143997.1144179 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of L2-ZDT using a mapping to prevent linear recombination operators +#! from potentially taking advantage of the problem structure +# FIXME: implementation unknown. +things["suite_l3_zdt"] = Suite( + name="L3-ZDT", + description="Variant of L2-ZDT with anti-linkage mapping", + objectives={2}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + references=[ + Reference( + title="Linkage ZDT/DTLZ variants", + authors=[], + link=Link(url="https://doi.org/10.1145/1143997.1144179"), + ) + ], +) + +#! - name: L2-DTLZ +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/1143997.1144179 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of DTLZ2 and DTLZ3 with linkages between all variables; Linear +#! recombination operators can potentially take advantage of the problem structure +# FIXME: implementation unknown. +things["suite_l2_dtlz"] = Suite( + name="L2-DTLZ", + description="Variant of DTLZ2/DTLZ3 with linkages between all variables", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Linkage ZDT/DTLZ variants", + authors=[], + link=Link(url="https://doi.org/10.1145/1143997.1144179"), + ) + ], +) + +#! - name: L3-DTLZ +#! suite/generator/single: suite +#! objectives: 2+ +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/1143997.1144179 +#! implementation: '?' +#! source (real-world/artificial): '' +#! textual description: Variant of L2-DTLZ using a mapping to prevent linear recombination operators +#! from potentially taking advantage of the problem structure +# FIXME: implementation unknown. +things["suite_l3_dtlz"] = Suite( + name="L3-DTLZ", + description="Variant of L2-DTLZ with anti-linkage mapping", + # FIXME: original "2+" - truncated to 2..10. + objectives=set(range(2, 11)), + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + references=[ + Reference( + title="Linkage ZDT/DTLZ variants", + authors=[], + link=Link(url="https://doi.org/10.1145/1143997.1144179"), + ) + ], +) + +#! - name: CEC2018 DT - CEC2018 Competition on Dynamic Multiobjective Optimisation +#! suite/generator/single: suite +#! objectives: 2 or 3 +#! dimensionality: scalable? +#! variable type: '?' +#! constraints: 'no' +#! dynamic: 'yes' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf +#! implementation: https://pymoo.org/problems/dynamic/df.html +#! source (real-world/artificial): artificial +#! textual description: '14 problems. Time-dependent: Pareto front/Pareto set geometry; +#! irregular Pareto front shapes; variable-linkage; number of disconnected Pareto +#! front segments; etc.' +# FIXME: variable type unknown. +things["suite_cec2018_dt"] = Suite( + name="CEC2018 DT", + long_name="CEC2018 Competition on Dynamic Multiobjective Optimisation", + description="14 problems. Time-dependent: Pareto front/Pareto set geometry; irregular Pareto front shapes; variable-linkage; number of disconnected Pareto front segments; etc.", + objectives={2, 3}, + variables=[Variable(type="unknown", dim=ValueRange(min=1))], + dynamic_type={"dynamic"}, + source={"artificial"}, + references=[ + Reference( + title="CEC2018 DMOP Competition TR", + authors=[], + link=Link(url="https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf"), + ) + ], + implementations={"impl_pymoo"}, +) + +#! - name: MODAct - multiobjective design of actuators +#! suite/generator/single: suite +#! objectives: 2 3 4 or 5 +#! dimensionality: '20' +#! variable type: mixed; integer and continuous +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/TEVC.2020.3020046 +#! implementation: https://pymoo.org/problems/constrained/modact.html +#! source (real-world/artificial): real-world +#! textual description: Realistic Constrained Multi-Objective Optimization Benchmark +#! Problems from Design. Need the https://github.com/epfl-lamd/modact package installed; evaluation +#! times around 20ms +things["impl_modact"] = Implementation( + name="modact", + description="EPFL-LAMD modact package", + evaluation_time=["20ms"], + links=[Link(type="repository", url="https://github.com/epfl-lamd/modact")], +) +things["suite_modact"] = Suite( + name="MODAct", + long_name="multiobjective design of actuators", + description="Realistic Constrained Multi-Objective Optimization Benchmark Problems from Design.", + objectives={2, 3, 4, 5}, + variables=[ + Variable(type="continuous", dim=20), + Variable(type="integer", dim=20), + ], + constraints=[Constraint(hard="yes")], + source={"real-world"}, + references=[ + Reference( + title="MODAct", + authors=[], + link=Link(url="https://doi.org/10.1109/TEVC.2020.3020046"), + ) + ], + implementations={"impl_modact", "impl_pymoo"}, +) + +#! - name: IOHClustering +#! suite/generator/single: suite; generator +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no ' +#! reference: https://arxiv.org/pdf/2505.09233 +#! implementation: https://github.com/IOHprofiler/IOHClustering +#! source (real-world/artificial): artificial, but based on real data +#! textual description: 'Set of benchmark problems from clustering: optimization task +#! is selecting cluster centers for a given set of data, with the number of clusters +#! defining problem dimensionality. Includes both a suite and a generator. Based on ML clustering datasets' +things["impl_iohclustering"] = Implementation( + name="IOHClustering", + description="Clustering-based optimization benchmark built on ML datasets", + links=[Link(type="repository", url="https://github.com/IOHprofiler/IOHClustering")], +) +things["suite_iohclustering"] = Suite( + name="IOHClustering", + description="Set of benchmark problems from clustering: optimization task is selecting cluster centers for a given set of data.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + source={"artificial-from-real-data"}, + references=[ + Reference( + title="IOHClustering", + authors=[], + link=Link(url="https://arxiv.org/pdf/2505.09233"), + ) + ], + implementations={"impl_iohclustering"}, +) +things["gen_iohclustering"] = Generator( + name="IOHClustering", + description="Generator counterpart of the IOHClustering suite.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + modality={"multimodal"}, + source={"artificial-from-real-data"}, + references=[ + Reference( + title="IOHClustering", + authors=[], + link=Link(url="https://arxiv.org/pdf/2505.09233"), + ) + ], + implementations={"impl_iohclustering"}, +) + +#! - name: GNBG-II +#! suite/generator/single: suite; generator +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 +#! implementation: https://github.com/rohitsalgotra/GNBG-II +#! source (real-world/artificial): artificial +#! textual description: Generalized Numerical Benchmark Generator (version 2). Also in IOH https://github.com/IOHprofiler/IOHGNBG +things["impl_gnbg_ii"] = Implementation( + name="GNBG-II", + description="Generalized Numerical Benchmark Generator version 2", + links=[Link(type="repository", url="https://github.com/rohitsalgotra/GNBG-II")], +) +things["impl_iohgnbg"] = Implementation( + name="IOHGNBG", + description="IOHprofiler version of GNBG", + links=[Link(type="repository", url="https://github.com/IOHprofiler/IOHGNBG")], +) +things["suite_gnbg_ii"] = Suite( + name="GNBG-II", + description="Generalized Numerical Benchmark Generator (version 2). Also available in IOH.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="GNBG-II", + authors=[], + link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3712255.3734271"), + ) + ], + implementations={"impl_gnbg_ii", "impl_iohgnbg"}, +) +things["gen_gnbg_ii"] = Generator( + name="GNBG-II", + description="Generator counterpart of GNBG-II.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="GNBG-II", + authors=[], + link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3712255.3734271"), + ) + ], + implementations={"impl_gnbg_ii", "impl_iohgnbg"}, +) + +#! - name: GNBG +#! suite/generator/single: suite; generator +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://arxiv.org/abs/2312.07083 +#! implementation: https://github.com/Danial-Yazdani/GNBG-Generator +#! source (real-world/artificial): artificial +#! textual description: Generalized Numerical Benchmark Generator +things["impl_gnbg"] = Implementation( + name="GNBG Generator", + description="Generalized Numerical Benchmark Generator", + links=[Link(type="repository", url="https://github.com/Danial-Yazdani/GNBG-Generator")], +) +things["suite_gnbg"] = Suite( + name="GNBG", + description="Generalized Numerical Benchmark Generator", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="GNBG", + authors=[], + link=Link(url="https://arxiv.org/abs/2312.07083"), + ) + ], + implementations={"impl_gnbg"}, +) +things["gen_gnbg"] = Generator( + name="GNBG", + description="Generator counterpart of GNBG.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="GNBG", + authors=[], + link=Link(url="https://arxiv.org/abs/2312.07083"), + ) + ], + implementations={"impl_gnbg"}, +) + +#! - name: DynamicBinVal +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: binary +#! constraints: 'no' +#! dynamic: 'yes' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://arxiv.org/pdf/2404.15837 +#! implementation: https://github.com/IOHprofiler/IOHexperimenter +#! source (real-world/artificial): artificial +#! textual description: Four versions of the dynamic binary value problem +things["suite_dynamicbinval"] = Suite( + name="DynamicBinVal", + description="Four versions of the dynamic binary value problem", + objectives={1}, + variables=[Variable(type="binary", dim=ValueRange(min=1))], + dynamic_type={"dynamic"}, + source={"artificial"}, + references=[ + Reference( + title="DynamicBinVal", + authors=[], + link=Link(url="https://arxiv.org/pdf/2404.15837"), + ) + ], + implementations={"impl_iohexperimenter"}, +) + +#! - name: PBO +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://dl.acm.org/doi/pdf/10.1145/3319619.3326810 +#! implementation: https://github.com/IOHprofiler/IOHexperimenter +#! source (real-world/artificial): artificial +#! textual description: Suite of 25 binary optimization problems +things["suite_pbo"] = Suite( + name="PBO", + description="Suite of 25 binary optimization problems", + objectives={1}, + variables=[Variable(type="binary", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="PBO benchmarks", + authors=[], + link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3319619.3326810"), + ) + ], + implementations={"impl_iohexperimenter"}, +) + +#! - name: W-model +#! suite/generator/single: generator +#! objectives: '1' +#! dimensionality: scalable +#! variable type: binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://dl.acm.org/doi/abs/10.1145/3205651.3208240?casa_token=S4U_Pi9f6MwAAAAA:U9ztNTPwmupT8K3GamWZfBL7-8fqjxPtr_kprv51vdwA-REsp0EyOFGa99BtbANb0XbqyrVg795hIw +#! implementation: https://github.com/thomasWeise/BBDOB_W_Model +#! source (real-world/artificial): artificial +#! textual description: Tunable generator for binary optimization based on several +#! difficulty features +things["impl_wmodel"] = Implementation( + name="BBDOB W-Model", + description="Tunable generator for binary optimization", + links=[Link(type="repository", url="https://github.com/thomasWeise/BBDOB_W_Model")], +) +things["gen_wmodel"] = Generator( + name="W-model", + description="Tunable generator for binary optimization based on several difficulty features", + objectives={1}, + variables=[Variable(type="binary", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="W-model", + authors=[], + link=Link( + url="https://dl.acm.org/doi/abs/10.1145/3205651.3208240" + ), + ) + ], + implementations={"impl_wmodel"}, +) + +#! - name: Submodular Optimitzation +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: binary +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181 +#! implementation: https://github.com/IOHprofiler/IOHexperimenter +#! source (real-world/artificial): artificial +#! textual description: set of graph-based submodular optimization problems from 4 +#! problem types +things["suite_submodular"] = Suite( + name="Submodular Optimization", + description="set of graph-based submodular optimization problems from 4 problem types", + objectives={1}, + variables=[Variable(type="binary", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="Submodular optimization benchmark", + authors=[], + link=Link(url="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181"), + ) + ], + implementations={"impl_iohexperimenter"}, +) + +#! - name: CEC2013 +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://peerj.com/articles/cs-2671/CEC2013.pdf +#! implementation: https://github.com/P-N-Suganthan/CEC2013 +#! source (real-world/artificial): artificial +#! textual description: suite used for cec2013 competition. Also in IOH https://github.com/IOHprofiler/IOHexperimenter +things["impl_cec2013"] = Implementation( + name="CEC2013 reference code", + description="Suganthan's reference implementation", + links=[Link(type="repository", url="https://github.com/P-N-Suganthan/CEC2013")], +) +things["suite_cec2013"] = Suite( + name="CEC2013", + description="suite used for cec2013 competition. Also in IOH.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="CEC2013 definitions", + authors=[], + link=Link(url="https://peerj.com/articles/cs-2671/CEC2013.pdf"), + ) + ], + implementations={"impl_cec2013", "impl_iohexperimenter"}, +) + +#! - name: CEC2022 +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: scalable +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '?' +#! multi-fidelity: 'no' +#! reference: https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf +#! implementation: https://github.com/P-N-Suganthan/2022-SO-BO +#! source (real-world/artificial): artificial +#! textual description: suite used for cec2022 competition. Also in IOH https://github.com/IOHprofiler/IOHexperimenter +things["impl_cec2022"] = Implementation( + name="CEC2022 reference code", + description="Suganthan's reference implementation", + links=[Link(type="repository", url="https://github.com/P-N-Suganthan/2022-SO-BO")], +) +things["suite_cec2022"] = Suite( + name="CEC2022", + description="suite used for cec2022 competition. Also in IOH.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + source={"artificial"}, + references=[ + Reference( + title="CEC2022 TR", + authors=[], + link=Link(url="https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf"), + ) + ], + implementations={"impl_cec2022", "impl_iohexperimenter"}, +) + +#! - name: Onemax+Sphere / Zeromax+Sphere +#! suite/generator/single: single +#! objectives: '2' +#! dimensionality: scalable +#! variable type: binary and continuous;mixed; +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3449726.3459521 +#! implementation: +#! source (real-world/artificial): 'artificial' +#! textual description: '' +# FIXME: no implementation provided. +things["fn_onemax_sphere_zeromax_sphere"] = Problem( + name="Onemax+Sphere / Zeromax+Sphere", + objectives={2}, + variables=[ + Variable(type="binary", dim=ValueRange(min=1)), + Variable(type="continuous", dim=ValueRange(min=1)), + ], + source={"artificial"}, + references=[ + Reference( + title="Onemax+Sphere / Zeromax+Sphere", + authors=[], + link=Link(url="https://doi.org/10.1145/3449726.3459521"), + ) + ], +) + +#! - name: Onemax+Sphere / DeceptiveTrap+RotatedEllipsoid +#! suite/generator/single: single +#! objectives: '2' +#! dimensionality: scalable +#! variable type: binary and continuous;mixed; +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3449726.3459521 +#! implementation: +#! source (real-world/artificial): 'artificial' +#! textual description: '' +# FIXME: no implementation provided. +things["fn_onemax_sphere_deceptive_rotell"] = Problem( + name="Onemax+Sphere / DeceptiveTrap+RotatedEllipsoid", + objectives={2}, + variables=[ + Variable(type="binary", dim=ValueRange(min=1)), + Variable(type="continuous", dim=ValueRange(min=1)), + ], + source={"artificial"}, + references=[ + Reference( + title="Mixed-variable multi-objective test problems", + authors=[], + link=Link(url="https://doi.org/10.1145/3449726.3459521"), + ) + ], +) + +#! - name: InverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoid +#! suite/generator/single: single +#! objectives: '2' +#! dimensionality: scalable +#! variable type: binary and continuous;mixed; +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: '' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1145/3449726.3459521 +#! implementation: +#! source (real-world/artificial): 'artificial' +#! textual description: '' +# FIXME: no implementation provided. +things["fn_invdeceptive_deceptive_rotell"] = Problem( + name="InverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoid", + objectives={2}, + variables=[ + Variable(type="binary", dim=ValueRange(min=1)), + Variable(type="continuous", dim=ValueRange(min=1)), + ], + source={"artificial"}, + references=[ + Reference( + title="Mixed-variable multi-objective test problems", + authors=[], + link=Link(url="https://doi.org/10.1145/3449726.3459521"), + ) + ], +) + +#! - name: PorkchopPlotInterplanetaryTrajectory +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 2 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1109/CEC65147.2025.11042973 +#! implementation: https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world +#! source (real-world/artificial): 'real-world' +#! textual description: '' +things["impl_transfer_rf_bbob_rw"] = Implementation( + name="Transfer Random Forests BBOB Real-world", + description="Real-world BBOB-like problem implementations (Porkchop, KinematicsRobotArm)", + links=[ + Link( + type="repository", + url="https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world", + ) + ], +) +things["suite_porkchop"] = Suite( + name="PorkchopPlotInterplanetaryTrajectory", + objectives={1}, + variables=[Variable(type="continuous", dim=2)], + modality={"multimodal"}, + source={"real-world"}, + references=[ + Reference( + title="Porkchop plot interplanetary trajectory benchmark", + authors=[], + link=Link(url="https://doi.org/10.1109/CEC65147.2025.11042973"), + ) + ], + implementations={"impl_transfer_rf_bbob_rw"}, +) + +#! - name: KinematicsRobotArm +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 21 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'no' +#! multi-fidelity: 'no' +#! reference: https://doi.org/10.1023/A:1013258808932 +#! implementation: https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world +#! source (real-world/artificial): 'real-world' +#! textual description: '' +things["suite_kinematics_robotarm"] = Suite( + name="KinematicsRobotArm", + objectives={1}, + variables=[Variable(type="continuous", dim=21)], + modality={"unimodal"}, + source={"real-world"}, + references=[ + Reference( + title="Kinematics of a robot arm", + authors=[], + link=Link(url="https://doi.org/10.1023/A:1013258808932"), + ) + ], + implementations={"impl_transfer_rf_bbob_rw"}, +) + +#! - name: VehicleDynamics +#! suite/generator/single: suite +#! objectives: '1' +#! dimensionality: 2 +#! variable type: continuous +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! reference: https://www.scitepress.org/Papers/2023/121580/121580.pdf +#! implementation: https://zenodo.org/records/8307853 +#! source (real-world/artificial): 'real-world' +#! textual description: '' +things["impl_vehicle_dynamics"] = Implementation( + name="VehicleDynamics (Zenodo)", + description="Zenodo archive for the vehicle dynamics benchmark", + links=[Link(type="archive", url="https://zenodo.org/records/8307853")], +) +things["suite_vehicle_dynamics"] = Suite( + name="VehicleDynamics", + objectives={1}, + variables=[Variable(type="continuous", dim=2)], + modality={"multimodal"}, + source={"real-world"}, + references=[ + Reference( + title="VehicleDynamics benchmark", + authors=[], + link=Link(url="https://www.scitepress.org/Papers/2023/121580/121580.pdf"), + ) + ], + implementations={"impl_vehicle_dynamics"}, +) + +#! - name: MECHBench +#! suite/generator/single: Problem Suite +#! variable type: Continuous +#! dimensionality: scalable' +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Real-World Application +#! implementation: https://github.com/BayesOptApp/MECHBench +#! textual description: This is a set of problems with inspiration from Structural +#! Mechanics Design Optimization. The suite comprises three physical models, from +#! which the user may define different kind of problems which impact the final design +#! output. +#! reference: https://arxiv.org/abs/2511.10821 +#! other info: +#! partial evaluations: 'no' +#! full name: MECHBench +#! constraint properties: Hard Constraints +#! number of constraints: 1 or 2 +#! description of multimodality: Unstructured or non isotropic multimodality +#! key challenges / characteristics: Embeds physical simulations and is flexible +#! and modular +#! scientific motivation: Bridge the black-box optimization techniques to a Mechanical +#! Design Problem which require these kinds of algorithms +#! limitations: The models do not include fracture or damage mechanics, just plasticity. +#! implementation languages: Python +#! approximate evaluation time: Times -> from 1 minute to 7 minutes +things["impl_mechbench"] = Implementation( + name="MECHBench", + description="Structural mechanics design optimization benchmark", + language="Python", + evaluation_time=["1 minute", "7 minutes"], + links=[Link(type="repository", url="https://github.com/BayesOptApp/MECHBench")], +) +things["suite_mechbench"] = Suite( + name="MECHBench", + long_name="MECHBench", + description="Set of problems inspired by Structural Mechanics Design Optimization. Embeds physical simulations (plasticity only, no fracture/damage). Unstructured/non-isotropic multimodality.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + constraints=[Constraint(hard="yes", number={1, 2})], + modality={"multimodal"}, + allows_partial_evaluation="no", + source={"real-world"}, + references=[ + Reference( + title="MECHBench", + authors=[], + link=Link(url="https://arxiv.org/abs/2511.10821"), + ) + ], + implementations={"impl_mechbench"}, +) + +#! - name: EXPObench +#! suite/generator/single: Problem Suite +#! variable type: Continuous, Integer, Categorical, Conditional +#! dimensionality: 10 to 135 +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'yes' +#! multimodal: Unknown +#! multi-fidelity: 'no' +#! source (real-world/artificial): Real-World Application +#! implementation: https://github.com/AlgTUDelft/ExpensiveOptimBenchmark +#! textual description: Wind farm layout optimization, gas filter design, pipe shape +#! optimization, hyperparameter tuning, and hospital simulation +#! reference: https://doi.org/10.1016/j.asoc.2023.110744 +#! other info: +#! partial evaluations: 'no' +#! full name: EXPensive Optimization benchmark library +#! constraint properties: Hard Constraints, Soft Constraints, Box Constraints, only +#! box constraints implemented, others appear as penalty in objective +#! number of constraints: 2 per variable (box), other constraints unknown (simulator +#! fails) +#! form of noise model: real-life (unknown) +#! type of noise space: Observational +#! key challenges / characteristics: Expensive objectives +#! scientific motivation: Address the lack of real-life expensive benchmarks +#! limitations: single-objective only, constraints are handled naively (penalty in +#! objective), no parallelization +#! implementation languages: Python +#! approximate evaluation time: 2 to 80 seconds +# FIXME: "Conditional" variable type has no schema representation; box number expressed as 2 per variable cannot be encoded. +things["impl_expobench"] = Implementation( + name="EXPObench", + description="EXPensive Optimization benchmark library (wind farm layout, gas filter design, pipe shape, hyperparameter tuning, hospital simulation)", + language="Python", + evaluation_time=["2 seconds", "80 seconds"], + links=[Link(type="repository", url="https://github.com/AlgTUDelft/ExpensiveOptimBenchmark")], +) +things["suite_expobench"] = Suite( + name="EXPObench", + long_name="EXPensive Optimization benchmark library", + description="Wind farm layout optimization, gas filter design, pipe shape optimization, hyperparameter tuning, and hospital simulation", + objectives={1}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=10, max=135)), + Variable(type="integer", dim=ValueRange(min=10, max=135)), + Variable(type="categorical", dim=ValueRange(min=10, max=135)), + ], + constraints=[ + Constraint(type="box", hard="yes"), + Constraint(hard="some"), + ], + noise_type={"observational", "real-life"}, + allows_partial_evaluation="no", + source={"real-world"}, + references=[ + Reference( + title="EXPObench", + authors=[], + link=Link(url="https://doi.org/10.1016/j.asoc.2023.110744"), + ) + ], + implementations={"impl_expobench"}, +) + +#! - name: Gasoline direct injection engine design +#! suite/generator/single: Single Problem +#! variable type: Continuous, Ordinal +#! dimensionality: '7' +#! objectives: '2' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: Unknown +#! multi-fidelity: 'yes' +#! source (real-world/artificial): Real-World Application +#! implementation: https://doi.org/10.1016/j.ejor.2022.08.032 +#! textual description: ... +#! other info: +#! partial evaluations: Unknown +#! constraint properties: Hard Constraints, Soft Constraints +#! number of constraints: '5' +#! key challenges / characteristics: Expensive +#! limitations: Proprietary +#! implementation languages: Matlab Simulink and Wave RT co-simulation +# FIXME: "Ordinal" variable type not in schema; falling back to integer. +things["impl_gasoline"] = Implementation( + name="Gasoline direct injection engine design", + description="Proprietary Matlab Simulink + Wave RT co-simulation", + language="Matlab Simulink / Wave RT", + links=[Link(type="paper", url="https://doi.org/10.1016/j.ejor.2022.08.032")], +) +things["fn_gasoline"] = Problem( + name="Gasoline direct injection engine design", + description="Multi-objective optimization to minimize fuel consumption and NOx emissions over a two-minute dynamic duty cycle, subject to five constraints (turbine inlet temperature, knock occurrences, peak cylinder pressure, peak cylinder pressure rise, total work). Seven decision variables cover hardware choices and engine control parameters.", + objectives={2}, + variables=[ + Variable(type="continuous", dim=7), + Variable(type="integer", dim=7), + ], + constraints=[Constraint(hard="yes", number=5)], + fidelity_levels={1, 2}, + source={"real-world"}, + references=[ + Reference( + title="Gasoline direct injection engine design", + authors=[], + link=Link(url="https://doi.org/10.1016/j.ejor.2022.08.032"), + ) + ], + implementations={"impl_gasoline"}, +) + +#! - name: BEACON +#! suite/generator/single: Generator +#! variable type: Continuous +#! dimensionality: scalable +#! objectives: '2' +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Artificially Generated +#! implementation: https://github.com/Stebbet/BEACON/ +#! textual description: Generator for bi-objective benchmark problems with explicitly +#! controlled correlations in continuous spaces. +#! reference: https://dl.acm.org/doi/10.1145/3712255.3734303 +#! other info: +#! partial evaluations: 'no' +#! full name: Continuous Bi-objective Benchmark problems with Explicit Adjustable +#! COrrelatioN control +#! constraint properties: Box Constraints +#! number of constraints: '0' +#! description of multimodality: Random +#! key challenges / characteristics: Multimodal, different correlations among objectives +#! scientific motivation: Controlled correlation among objectives +#! limitations: No analytical Pareto front, only bi-objective +#! implementation languages: Python +#! approximate evaluation time: Negligible +things["impl_beacon"] = Implementation( + name="BEACON", + description="Continuous Bi-objective Benchmark with Explicit Adjustable COrrelatioN control", + language="Python", + evaluation_time=["negligible"], + links=[Link(type="repository", url="https://github.com/Stebbet/BEACON/")], +) +things["gen_beacon"] = Generator( + name="BEACON", + long_name="Continuous Bi-objective Benchmark problems with Explicit Adjustable COrrelatioN control", + description="Generator for bi-objective benchmark problems with explicitly controlled correlations in continuous spaces. Multimodal with random structure.", + objectives={2}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + constraints=[Constraint(type="box", hard="yes", number=0)], + modality={"multimodal"}, + allows_partial_evaluation="no", + source={"artificial"}, + references=[ + Reference( + title="BEACON", + authors=[], + link=Link(url="https://dl.acm.org/doi/10.1145/3712255.3734303"), + ) + ], + implementations={"impl_beacon"}, +) + +#! - name: TulipaEnergy +#! suite/generator/single: Problem Suite +#! variable type: Continuous +#! dimensionality: scalable +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'yes' +#! multimodal: 'no' +#! multi-fidelity: 'yes' +#! source (real-world/artificial): Real-World Application +#! implementation: https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/ +#! textual description: Determine the optimal investment and operation decisions for +#! different types of assets in the energy system ... minimizing loss of load. +#! reference: See https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references +#! other info: +#! partial evaluations: Unknown +#! full name: TulipaEnergyModel.jl +#! constraint properties: Hard Constraints, Soft Constraints +#! number of constraints: millions +#! type of dynamicism: none +#! form of noise model: depends on input — still working on stochastic inputs +#! type of noise space: Parameter +#! key challenges / characteristics: modeled as a potentially very large linear program +#! scientific motivation: new techniques for solving large whitebox linear optimization problems +#! limitations: not yet stochastic +#! implementation languages: Julia / JMP +#! approximate evaluation time: from minutes to hours +#! links to usage examples: https://github.com/TulipaEnergy/Tulipa-OBZ-CaseStudy +# FIXME: "number of constraints: millions" cannot be expressed precisely. +things["impl_tulipa"] = Implementation( + name="TulipaEnergyModel.jl", + description="Large linear program for optimal investment and operation of energy systems", + language="Julia / JuMP", + evaluation_time=["minutes", "hours"], + links=[ + Link(type="website", url="https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/"), + Link(type="example", url="https://github.com/TulipaEnergy/Tulipa-OBZ-CaseStudy"), + ], +) +things["suite_tulipa_energy"] = Suite( + name="TulipaEnergy", + long_name="TulipaEnergyModel.jl", + description="Determine the optimal investment and operation decisions for different assets in the energy system (production, consumption, conversion, storage, transport) while minimizing loss of load. Modelled as a potentially very large linear program with multiple fidelity levels.", + objectives={1}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + constraints=[Constraint(hard="yes"), Constraint(hard="some")], + noise_type={"parameter"}, + modality={"unimodal"}, + fidelity_levels={1, 2}, + source={"real-world"}, + references=[ + Reference( + title="TulipaEnergyModel.jl scientific references", + authors=[], + link=Link( + url="https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references" + ), + ) + ], + implementations={"impl_tulipa"}, +) + +#! - name: ATO +#! suite/generator/single: Single Problem +#! variable type: Continuous +#! dimensionality: '10' +#! objectives: '2' +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'no' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Real-World Application +#! implementation: '-' +#! textual description: Parameters of the Modules of the Automatic Train Operation +#! should be optimized. The parameters are continuous with different ranges. There +#! are two objectives (minimizing energy consumption, minimizing driving duration. +#! other info: +#! partial evaluations: 'no' +# FIXME: no implementation available. +things["fn_ato"] = Problem( + name="ATO", + description="Parameters of the Modules of the Automatic Train Operation are optimized; two objectives: minimizing energy consumption and minimizing driving duration.", + objectives={2}, + variables=[Variable(type="continuous", dim=10)], + modality={"unimodal"}, + allows_partial_evaluation="no", + source={"real-world"}, +) + +#! - name: Brachytherapy treatment planning +#! suite/generator/single: Problem Suite +#! variable type: Continuous +#! dimensionality: 100-500 +#! objectives: 2-3 +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'yes' +#! source (real-world/artificial): Real-World Application +#! textual description: Treatment planning for internal radiation therapy +#! reference: https://www.sciencedirect.com/science/article/pii/S1538472123016781 +#! other info: +#! partial evaluations: 'yes' +#! full name: Brachytherapy treatment planning +#! constraint properties: Hard Constraints +#! number of constraints: scalable +#! key challenges / characteristics: Multi-objective; aggregated objectives +#! limitations: No public source code +# FIXME: no public source code; no implementation URL. +things["suite_brachytherapy"] = Suite( + name="Brachytherapy treatment planning", + long_name="Brachytherapy treatment planning", + description="Treatment planning for internal radiation therapy. Multi-objective with aggregated objectives; no public source code.", + objectives={2, 3}, + variables=[Variable(type="continuous", dim=ValueRange(min=100, max=500))], + constraints=[Constraint(hard="yes", number=ValueRange(min=1))], + modality={"multimodal"}, + fidelity_levels={1, 2}, + allows_partial_evaluation="yes", + source={"real-world"}, + references=[ + Reference( + title="Brachytherapy treatment planning", + authors=[], + link=Link(url="https://www.sciencedirect.com/science/article/pii/S1538472123016781"), + ) + ], +) + +#! - name: FleetOpt +#! suite/generator/single: Single Problem +#! variable type: Integer +#! dimensionality: 'Upper level: 54; lower level: 13208' +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: Unknown +#! multi-fidelity: 'no' +#! source (real-world/artificial): Real-World Application +#! implementation: 'Not public: was done for real client with their private data' +#! textual description: Healthcare organisation in the UK ... +#! reference: https://dl.acm.org/doi/abs/10.1145/3638530.3664137 +#! other info: +#! partial evaluations: 'yes' +# FIXME: bilevel dimensionality (upper 54 / lower 13208) expressed as {54, 13208}; impl not public. +things["fn_fleetopt"] = Problem( + name="FleetOpt", + description="UK healthcare organisation fleet optimisation: reduce the fleet of non-emergency healthcare trip vehicles while still ensuring all trips can be covered. Bilevel: upper level 54 vars, lower level 13208 vars.", + objectives={1}, + variables=[Variable(type="integer", dim={54, 13208})], + constraints=[Constraint(hard="yes")], + allows_partial_evaluation="yes", + source={"real-world"}, + references=[ + Reference( + title="FleetOpt", + authors=[], + link=Link(url="https://dl.acm.org/doi/abs/10.1145/3638530.3664137"), + ) + ], +) + +#! - name: Building spatial design +#! suite/generator/single: Single Problem +#! variable type: Continuous, Boolean +#! dimensionality: scalable depending on problem size (e.g. 90 for) +#! objectives: '2' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: Unknown +#! multi-fidelity: 'no' +#! source (real-world/artificial): Real-World Application +#! implementation: https://github.com/TUe-excellent-buildings/BSO-toolbox +#! textual description: 'Optimise the spatial layout of a building to: minimise energy +#! consumption for climate control, and minimise the strain on the structure' +#! reference: https://hdl.handle.net/1887/81789 +#! other info: +#! partial evaluations: 'no' +#! constraint properties: Hard Constraints, Box Constraints, Permutation Constraints +#! number of constraints: 2065 (as example, depends on problem size) +#! implementation languages: C++ +#! approximate evaluation time: Roughly 1 second per evaluation for the smallest +#! considered design, and roughly 40 seconds for the larger designs we considered. +# FIXME: Permutation Constraints not representable in ConstraintType; using multiple Constraint objects. +things["impl_bso_toolbox"] = Implementation( + name="BSO-toolbox", + description="Building Spatial Design toolbox (TU/e)", + language="C++", + evaluation_time=["1 second", "40 seconds"], + links=[Link(type="repository", url="https://github.com/TUe-excellent-buildings/BSO-toolbox")], +) +things["fn_building_spatial"] = Problem( + name="Building spatial design", + description="Optimise the spatial layout of a building to minimise energy consumption for climate control and minimise the strain on the structure. Many hard constraints; mixed-variable (continuous+binary); expensive evaluations.", + objectives={2}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + constraints=[ + Constraint(hard="yes"), + Constraint(type="box", hard="yes"), + ], + allows_partial_evaluation="no", + source={"real-world"}, + references=[ + Reference( + title="Building spatial design", + authors=[], + link=Link(url="https://hdl.handle.net/1887/81789"), + ) + ], + implementations={"impl_bso_toolbox"}, +) + +#! - name: Electric Motor Design Optimization +#! suite/generator/single: Single Problem +#! variable type: Continuous, Integer +#! dimensionality: '13' +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'yes' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Real-World Application +#! implementation: Implementation not freely available +#! textual description: The goal is to find a design of a synchronous electric motor +#! for power steering systems that minimizes costs and satisfies all constraints. +#! reference: https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf (paper in Slovene) +#! other info: +#! partial evaluations: 'no' +#! full name: Electric Motor Design Optimization +#! constraint properties: Hard Constraints, Soft Constraints, Box Constraints +#! number of constraints: '12' +#! description of multimodality: Constraints are multimodal +#! key challenges / characteristics: Time-consuming solution evaluation, highly-constrained +#! scientific motivation: Challenging to find good solutions in a limited time +#! limitations: 'Unavailability ...' +#! implementation languages: Python +#! approximate evaluation time: 8 minutes +#! general: This is not an available problem, but could be interesting to show to +#! researchers which difficulties appear in real-world problems +things["impl_emdo"] = Implementation( + name="Electric Motor Design Optimization", + description="Not publicly available", + language="Python", + evaluation_time=["8 minutes"], +) +things["fn_emdo"] = Problem( + name="Electric Motor Design Optimization", + long_name="Electric Motor Design Optimization", + description="""# Goal +Find a design of a synchronous electric motor for power steering systems that minimizes costs and satisfies all constraints. + +# Motivation +Challenging to find good solutions in a limited time. + +# Key Challenges +* Time-consuming solution evaluation +* Highly-constrained problem +* Constraints are multimodal + +This is not an available problem, but could be interesting to show to researchers which difficulties appear in real-world problems.""", + objectives={1}, + variables=[ + Variable(type="continuous", dim=13), + Variable(type="integer", dim=13), + ], + constraints=[ + Constraint(hard="yes", number=12), + Constraint(hard="some"), + Constraint(type="box", hard="yes"), + ], + noise_type={"noisy"}, + modality={"multimodal"}, + allows_partial_evaluation="no", + source={"real-world"}, + references=[ + Reference( + title="A Multi-Step Evaluation Process in Electric Motor Design", + authors=["Tea Tušar", "Peter Korošec", "Bogdan Filipič"], + link=Link(url="https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf"), + ) + ], + implementations={"impl_emdo"}, +) + +#! - name: BONO-Bench +#! suite/generator/single: Generator +#! variable type: Continuous +#! dimensionality: scalable +#! objectives: '2' +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Artificially Generated +#! implementation: https://github.com/schaepermeier/bonobench +#! textual description: Bi-objective problem generator and suite with scalable continuous +#! decision space. Features complex problem properties (different types of multimodality +#! and challenges in decision and objective space) as well as Pareto front approximations +#! with error guarantees for the hypervolume and exact R2 indicators. +#! other info: +#! partial evaluations: 'no' +#! full name: Bi-objective Numerical Optimization Benchmark (BONO-Bench) +#! constraint properties: Box Constraints +#! implementation languages: Python +things["impl_bonobench"] = Implementation( + name="BONO-Bench", + description="Bi-objective Numerical Optimization Benchmark (BONO-Bench)", + language="Python", + links=[Link(type="repository", url="https://github.com/schaepermeier/bonobench")], +) +things["gen_bono_bench"] = Generator( + name="BONO-Bench", + long_name="Bi-objective Numerical Optimization Benchmark", + description="Bi-objective problem generator and suite with scalable continuous decision space. Features complex problem properties and Pareto front approximations with error guarantees for the hypervolume and exact R2 indicators.", + objectives={2}, + variables=[Variable(type="continuous", dim=ValueRange(min=1))], + constraints=[Constraint(type="box", hard="yes")], + modality={"multimodal"}, + allows_partial_evaluation="no", + source={"artificial"}, + implementations={"impl_bonobench"}, +) + +#! - name: RandOptGen +#! suite/generator/single: Generator +#! variable type: Continuous, Integer, Boolean +#! dimensionality: scalable +#! objectives: scalable +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Artificially Generated +#! implementation: https://github.com/MALEO-research-group/RandOptGen +#! textual description: 'RandOptGen: A Unified Random Problem Generator for Single-and +#! Multi-Objective Optimization Problems with Mixed-Variable Input Spaces' +#! other info: +#! partial evaluations: 'no' +#! full name: RandOptGen +#! implementation languages: Python +#! approximate evaluation time: milliseconds +#! links to usage examples: https://doi.org/10.1145/3712256.3726478 +things["impl_randoptgen"] = Implementation( + name="RandOptGen", + description="Unified Random Problem Generator for Single- and Multi-Objective Optimization with Mixed-Variable Input Spaces", + language="Python", + evaluation_time=["milliseconds"], + links=[ + Link(type="repository", url="https://github.com/MALEO-research-group/RandOptGen"), + Link(type="example", url="https://doi.org/10.1145/3712256.3726478"), + ], +) +things["gen_randoptgen"] = Generator( + name="RandOptGen", + long_name="RandOptGen", + description="A Unified Random Problem Generator for Single- and Multi-Objective Optimization Problems with Mixed-Variable Input Spaces.", + # FIXME: original "scalable" - truncated to 1..10. + objectives=set(range(1, 11)), + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="integer", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + modality={"multimodal"}, + allows_partial_evaluation="no", + source={"artificial"}, + implementations={"impl_randoptgen"}, +) + +#! - name: CUTEr +#! suite/generator/single: Problem Suite +#! variable type: Continuous, Integer, Boolean +#! dimensionality: scalable +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: Unknown +#! multi-fidelity: 'no' +#! source (real-world/artificial): Artificially Generated +#! implementation: Not Found +#! textual description: A constrained and unconstrained testing environment +#! reference: https://dl.acm.org/doi/10.1145/962437.962439 +#! other info: +#! partial evaluations: 'no' +# FIXME: implementation not found. +things["suite_cuter"] = Suite( + name="CUTEr", + description="A constrained and unconstrained testing environment.", + objectives={1}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="integer", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + constraints=[Constraint(hard="yes")], + allows_partial_evaluation="no", + source={"artificial"}, + references=[ + Reference( + title="CUTEr", + authors=[], + link=Link(url="https://dl.acm.org/doi/10.1145/962437.962439"), + ) + ], +) + +#! - name: CUTEst +#! suite/generator/single: Problem Suite +#! variable type: Continuous, Integer, Boolean +#! dimensionality: scalable +#! objectives: '1' +#! constraints: 'yes' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: 'yes' +#! multi-fidelity: 'no' +#! source (real-world/artificial): Artificially Generated +#! implementation: https://github.com/jfowkes/pycutest +#! textual description: The Constrained and Unconstrained Testing Environment with +#! safe threads (CUTEst) for optimization software +#! reference: https://link.springer.com/article/10.1007/s10589-014-9687-3 +#! other info: +#! partial evaluations: 'no' +#! full name: 'Constrained and Unconstrained Testing Environment with safe threads ' +#! constraint properties: Soft Constraints, Box Constraints +#! number of constraints: scalable +#! implementation languages: Python, C++, Fortran +#! general: 'Python implementation: https://github.com/jfowkes/pycutest' +things["impl_pycutest"] = Implementation( + name="pycutest", + description="Python interface to CUTEst", + language="Python / C++ / Fortran", + links=[Link(type="repository", url="https://github.com/jfowkes/pycutest")], +) +things["suite_cutest"] = Suite( + name="CUTEst", + long_name="Constrained and Unconstrained Testing Environment with safe threads", + description="CUTEst for optimization software", + objectives={1}, + variables=[ + Variable(type="continuous", dim=ValueRange(min=1)), + Variable(type="integer", dim=ValueRange(min=1)), + Variable(type="binary", dim=ValueRange(min=1)), + ], + constraints=[ + Constraint(hard="some", number=ValueRange(min=1)), + Constraint(type="box", hard="yes"), + ], + modality={"multimodal"}, + allows_partial_evaluation="no", + source={"artificial"}, + references=[ + Reference( + title="CUTEst", + authors=[], + link=Link(url="https://link.springer.com/article/10.1007/s10589-014-9687-3"), + ) + ], + implementations={"impl_pycutest"}, +) + +#! - name: PUBOi +#! suite/generator/single: Generator +#! variable type: Boolean +#! dimensionality: scalable +#! objectives: '1' +#! constraints: 'no' +#! dynamic: 'no' +#! noise: 'no' +#! multimodal: Unknown +#! multi-fidelity: 'no' +#! source (real-world/artificial): Artificially Generated +#! implementation: https://gitlab.com/verel/pubo-importance-benchmark +#! textual description: A benchmark in which variable importance is tunable, based +#! on the Walsh function +#! reference: https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12 +#! other info: +#! partial evaluations: 'no' +#! full name: Polynomial Unconstrained Binary Optimization +#! key challenges / characteristics: Tunable variable importance +#! implementation languages: Python, C++ +things["impl_puboi"] = Implementation( + name="PUBO Importance Benchmark", + description="A benchmark in which variable importance is tunable, based on the Walsh function", + language="Python / C++", + links=[Link(type="repository", url="https://gitlab.com/verel/pubo-importance-benchmark")], +) + +things["gen_puboi"] = Generator( + name="PUBOi", + long_name="Polynomial Unconstrained Binary Optimization with tunable importance", + description="A benchmark in which variable importance is tunable, based on the Walsh function.", + objectives={1}, + variables=[Variable(type="binary", dim=ValueRange(min=1))], + allows_partial_evaluation="no", + source={"artificial"}, + references=[ + Reference( + title="PUBOi", + authors=[], + link=Link(url="https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12"), + ) + ], + implementations={"impl_puboi"}, +) + + +library = Library(things) + +# Make sure model is really valid +Library.model_validate(library) + +if __name__ == "__main__": + with open("problems.yaml", "w") as fd: + fd.write(to_yaml_str(library)) diff --git a/problems.yaml b/problems.yaml index 4328038..ea3b7af 100644 --- a/problems.yaml +++ b/problems.yaml @@ -1,1264 +1,3597 @@ -- name: BBOB - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1080/10556788.2020.1808977 - implementation: https://github.com/numbbo/coco - source (real-world/artificial): '' - textual description: '' -- name: BBOB-biobj - suite/generator/single: suite - objectives: '2' - dimensionality: 2-40 - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.48550/arXiv.1604.00359 - implementation: https://github.com/numbbo/coco - source (real-world/artificial): '' - textual description: '' -- name: BBOB-noisy - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'yes' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://hal.inria.fr/inria-00369466 - implementation: https://web.archive.org/web/20210416065610/https://coco.gforge.inria.fr/doku.php?id=downloads - source (real-world/artificial): '' - textual description: '' -- name: BBOB-largescale - suite/generator/single: suite - objectives: '1' - dimensionality: 20-640 - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.48550/arXiv.1903.06396 - implementation: https://github.com/numbbo/coco - source (real-world/artificial): '' - textual description: '' -- name: BBOB-mixint - suite/generator/single: suite - objectives: '1' - dimensionality: 5-160 - variable type: integer;continuous;mixed - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3321707.3321868 - implementation: https://github.com/numbbo/coco - source (real-world/artificial): '' - textual description: '' -- name: BBOB-biobj-mixint - suite/generator/single: suite - objectives: '2' - dimensionality: 5-160 - variable type: integer;continuous;mixed - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3321707.3321868 - implementation: https://github.com/numbbo/coco - source (real-world/artificial): '' - textual description: '' -- name: BBOB-constrained - suite/generator/single: suite - objectives: '1' - dimensionality: 2-40 - variable type: continuous - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: http://numbbo.github.io/coco-doc/bbob-constrained/ - implementation: https://github.com/numbbo/coco - source (real-world/artificial): '' - textual description: '' -- name: MOrepo - suite/generator/single: suite - objectives: '2' - dimensionality: '?' - variable type: combinatorial - constraints: '?' - dynamic: '?' - noise: '?' - multimodal: '?' - multi-fidelity: 'no' - reference: '' - implementation: https://github.com/MCDMSociety/MOrepo - source (real-world/artificial): '' - textual description: '' -- name: ZDT - suite/generator/single: suite - objectives: '2' - dimensionality: scalable - variable type: continuous;binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1162/106365600568202 - implementation: https://github.com/anyoptimization/pymoo - source (real-world/artificial): '' - textual description: '' -- name: DTLZ - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/CEC.2002.1007032 - implementation: https://pymoo.org/problems/many/dtlz.html - source (real-world/artificial): '' - textual description: '' -- name: WFG - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2005.861417 - implementation: https://pymoo.org/problems/many/wfg.html - source (real-world/artificial): '' - textual description: '' -- name: CDMP - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'yes' - dynamic: '?' - noise: '?' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3321707.3321878 - implementation: '?' - source (real-world/artificial): '' - textual description: '' -- name: SDP - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'yes' - noise: '?' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TCYB.2019.2896021 - implementation: '?' - source (real-world/artificial): '' - textual description: '' -- name: MaOP - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: '?' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1016/j.swevo.2019.02.003 - implementation: '?' - source (real-world/artificial): '' - textual description: '' -- name: BP - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: '?' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/CEC.2019.8790277 - implementation: '?' - source (real-world/artificial): '' - textual description: '' -- name: GPD - suite/generator/single: generator - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: optional - dynamic: 'no' - noise: optional - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1016/j.asoc.2020.106139 - implementation: '?' - source (real-world/artificial): '' - textual description: '' -- name: ETMOF - suite/generator/single: suite - objectives: 2-50 - dimensionality: 25-10000 - variable type: continuous - constraints: 'no' - dynamic: 'yes' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.48550/arXiv.2110.08033 - implementation: https://github.com/songbai-liu/etmo - source (real-world/artificial): '' - textual description: '' -- name: MMOPP - suite/generator/single: suite - objectives: 2-7 - dimensionality: '?' - variable type: '?' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412 - implementation: http://www5.zzu.edu.cn/ecilab/info/1036/1251.htm - source (real-world/artificial): '' - textual description: '' -- name: CFD - suite/generator/single: suite - objectives: 1-2 - dimensionality: scalable - variable type: '?' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1007/978-3-319-99259-4_24 - implementation: https://bitbucket.org/arahat/cfd-test-problem-suite - source (real-world/artificial): real world - textual description: expensive evaluations 30s-15m -- name: GBEA - suite/generator/single: suite - objectives: 1-2 - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'yes' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3321707.3321805 - implementation: 'https://github.com/ttusar/coco-gbea' - source (real-world/artificial): real world - textual description: 'expensive evaluations 5s-35s, RW-GAN-Mario and TopTrumps are part of GBEA' -- name: Car structure - suite/generator/single: suite - objectives: '2' - dimensionality: 144-222 - variable type: discrete - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3205651.3205702 - implementation: http://ladse.eng.isas.jaxa.jp/benchmark/ - source (real-world/artificial): real world - textual description: 54 constraints -- name: EMO2017 - suite/generator/single: suite - objectives: '2' - dimensionality: 4-24 - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/ - implementation: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/downloads/realworld-problems-bbcomp-EMO-2017.zip - source (real-world/artificial): real world - textual description: '' -- name: JSEC2019 - suite/generator/single: single - objectives: 1-5 - dimensionality: '32' - variable type: continuous - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html - implementation: http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html - source (real-world/artificial): real world - textual description: expensive evaluations 3s; 22 constraints -- name: RE - suite/generator/single: suite - objectives: 2-9 - dimensionality: 2-7 - variable type: continuous;integer;mixed - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1016/j.asoc.2020.106078 - implementation: https://github.com/ryojitanabe/reproblems - source (real-world/artificial): real world like - textual description: '' -- name: CRE - suite/generator/single: suite - objectives: 2-5 - dimensionality: 3-7 - variable type: continuous;integer;mixed - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1016/j.asoc.2020.106078 - implementation: https://github.com/ryojitanabe/reproblems - source (real-world/artificial): real world like - textual description: '' -- name: Radar waveform - suite/generator/single: single - objectives: '9' - dimensionality: 4-12 - variable type: integer - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1007/978-3-540-70928-2_53 - implementation: http://code.evanhughes.org/ - source (real-world/artificial): real world - textual description: '' -- name: MF2 - suite/generator/single: suite - objectives: '1' - dimensionality: 1-n - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'yes' - reference: https://doi.org/10.21105/joss.02049 - implementation: https://github.com/sjvrijn/mf2 - source (real-world/artificial): '' - textual description: '' -- name: AMVOP - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: mixed continuous+ordinal+categorical+both - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2013.2281531 - implementation: '?' - source (real-world/artificial): '' - textual description: '' -- name: RWMVOP - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: continuous;mixed continuous+ordinal+categorical+both - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2013.2281531 - implementation: '?' - source (real-world/artificial): real world - textual description: '' -- name: SBOX-COST - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.48550/arXiv.2305.12221 - implementation: https://github.com/IOHprofiler/IOHexperimenter/ - source (real-world/artificial): '' - textual description: problems from BBOB but allows instances with the optimum close to the - boundary -- name: "\u03C1MNK-Landscapes" - suite/generator/single: generator - objectives: scalable - dimensionality: scalable - variable type: binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1016/j.ejor.2012.12.019 - implementation: https://gitlab.com/aliefooghe/mocobench/ - source (real-world/artificial): '' - textual description: tunable variable and objective dimensions; tunable multimodality and +fn_ato: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: 'Parameters of the Modules of the Automatic Train Operation are optimized; + two objectives: minimizing energy consumption and minimizing driving duration.' + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: + - unimodal + name: ATO + noise_type: null + objectives: + - 2 + references: null + source: + - real-world + tags: null + type: problem + variables: + - dim: 10 + type: continuous +fn_building_spatial: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + - equality: null + hard: yes + number: null + type: box + description: Optimise the spatial layout of a building to minimise energy + consumption for climate control and minimise the strain on the structure. + Many hard constraints; mixed-variable (continuous+binary); expensive + evaluations. + dynamic_type: null + evaluation_time: + - 1 second + - 40 seconds + fidelity_levels: null + implementations: + - impl_bso_toolbox + instances: null + long_name: null + modality: null + name: Building spatial design + noise_type: null + objectives: + - 2 + references: + - authors: [] + link: + type: null + url: https://hdl.handle.net/1887/81789 + title: Building spatial design + source: + - real-world + tags: null + type: problem + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +fn_convex_dtlz2: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of DTLZ2 with a convex Pareto front (instead of concave) + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: null + name: Convex DTLZ2 + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281535 + title: Convex DTLZ2 + source: null + tags: null + type: problem + variables: + - dim: + max: null + min: 1 + type: continuous +fn_emdo: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: 12 + type: unknown + - equality: null + hard: yes + number: null + type: box + - equality: null + hard: some + number: null + type: unknown + description: "# Goal\nFind a design of a synchronous electric motor for power steering + systems that minimizes costs and satisfies all constraints.\n\n# Motivation\n\ + Challenging to find good solutions in a limited time.\n\n# Key Challenges\n* Time-consuming + solution evaluation\n* Highly-constrained problem\n* Constraints are multimodal\n\ + \nThis is not an available problem, but could be interesting to show to researchers + which difficulties appear in real-world problems." + dynamic_type: null + evaluation_time: + - 8 minutes + fidelity_levels: null + implementations: + - impl_emdo + instances: null + long_name: Electric Motor Design Optimization + modality: + - multimodal + name: Electric Motor Design Optimization + noise_type: + - noisy + objectives: + - 1 + references: + - authors: + - Tea Tušar + - Peter Korošec + - Bogdan Filipič + link: + type: null + url: https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf + title: A Multi-Step Evaluation Process in Electric Motor Design + source: + - real-world + tags: null + type: problem + variables: + - dim: 13 + type: continuous + - dim: 13 + type: integer +fn_fleetopt: + allows_partial_evaluation: yes + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: 'UK healthcare organisation fleet optimisation: reduce the fleet of + non-emergency healthcare trip vehicles while still ensuring all trips can be covered. + Bilevel: upper level 54 vars, lower level 13208 vars.' + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: null + name: FleetOpt + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/abs/10.1145/3638530.3664137 + title: FleetOpt + source: + - real-world + tags: null + type: problem + variables: + - dim: + - 13208 + - 54 + type: integer +fn_gasoline: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: 5 + type: unknown + description: Multi-objective optimization to minimize fuel consumption and NOx + emissions over a two-minute dynamic duty cycle, subject to five constraints + (turbine inlet temperature, knock occurrences, peak cylinder pressure, peak + cylinder pressure rise, total work). Seven decision variables cover hardware + choices and engine control parameters. + dynamic_type: null + evaluation_time: [] + fidelity_levels: + - 1 + - 2 + implementations: + - impl_gasoline + instances: null + long_name: null + modality: null + name: Gasoline direct injection engine design + noise_type: null + objectives: + - 2 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1016/j.ejor.2022.08.032 + title: Gasoline direct injection engine design + source: + - real-world + tags: null + type: problem + variables: + - dim: 7 + type: integer + - dim: 7 + type: continuous +fn_invdeceptive_deceptive_rotell: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: null + name: InverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoid + noise_type: null + objectives: + - 2 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3449726.3459521 + title: Mixed-variable multi-objective test problems + source: + - artificial + tags: null + type: problem + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +fn_inverted_dtlz1: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of DTLZ1 with an inverted Pareto front + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: null + name: Inverted DTLZ1 + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281534 + title: Inverted DTLZ1 + source: null + tags: null + type: problem + variables: + - dim: + max: null + min: 1 + type: continuous +fn_jsec2019: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: 22 + type: unknown + description: expensive evaluations 3s; 22 constraints + dynamic_type: null + evaluation_time: + - 3s + fidelity_levels: null + implementations: + - impl_jsec2019 + instances: null + long_name: null + modality: null + name: JSEC2019 + noise_type: null + objectives: + - 1 + - 2 + - 3 + - 4 + - 5 + references: + - authors: [] + link: + type: null + url: + http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html + title: JPNSEC EC-Symposium 2019 competition + source: + - real-world + tags: null + type: problem + variables: + - dim: 32 + type: continuous +fn_onemax_sphere_deceptive_rotell: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: null + name: Onemax+Sphere / DeceptiveTrap+RotatedEllipsoid + noise_type: null + objectives: + - 2 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3449726.3459521 + title: Mixed-variable multi-objective test problems + source: + - artificial + tags: null + type: problem + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +fn_onemax_sphere_zeromax_sphere: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + instances: null + long_name: null + modality: null + name: Onemax+Sphere / Zeromax+Sphere + noise_type: null + objectives: + - 2 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3449726.3459521 + title: Onemax+Sphere / Zeromax+Sphere + source: + - artificial + tags: null + type: problem + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +fn_radar_waveform: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: null + dynamic_type: null + evaluation_time: [] + fidelity_levels: null + implementations: + - impl_radar_waveform + instances: null + long_name: null + modality: null + name: Radar waveform + noise_type: null + objectives: + - 9 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1007/978-3-540-70928-2_53 + title: Radar waveform design + source: + - real-world + tags: null + type: problem + variables: + - dim: + max: 12 + min: 4 + type: integer +gen_beacon: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: 0 + type: box + description: Generator for bi-objective benchmark problems with explicitly + controlled correlations in continuous spaces. Multimodal with random + structure. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_beacon + long_name: Continuous Bi-objective Benchmark problems with Explicit Adjustable + COrrelatioN control + modality: + - multimodal + name: BEACON + noise_type: null + objectives: + - 2 + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/10.1145/3712255.3734303 + title: BEACON + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_bono_bench: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: box + description: Bi-objective problem generator and suite with scalable continuous + decision space. Features complex problem properties and Pareto front + approximations with error guarantees for the hypervolume and exact R2 + indicators. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_bonobench + long_name: Bi-objective Numerical Optimization Benchmark + modality: + - multimodal + name: BONO-Bench + noise_type: null + objectives: + - 2 + references: null + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_ealain: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: some + number: null + type: unknown + description: Real-world-like, easily extensible to increase complexity + dynamic_type: + - optional + evaluation_time: null + fidelity_levels: + - 1 + - 2 + implementations: + - impl_ealain + long_name: null + modality: null + name: Ealain + noise_type: null + objectives: + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3638530.3654299 + title: Ealain + source: + - real-world-like + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: integer + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +gen_gnbg: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Generator counterpart of GNBG. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_gnbg + long_name: null + modality: null + name: GNBG + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://arxiv.org/abs/2312.07083 + title: GNBG + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_gnbg_ii: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Generator counterpart of GNBG-II. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_gnbg_ii + - impl_iohgnbg + long_name: null + modality: null + name: GNBG-II + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 + title: GNBG-II + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_gpd: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: some + number: null + type: unknown + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: GPD + noise_type: + - optional + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1016/j.asoc.2020.106139 + title: GPD generator + source: null + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_iohclustering: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Generator counterpart of the IOHClustering suite. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_iohclustering + long_name: null + modality: + - multimodal + name: IOHClustering + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://arxiv.org/pdf/2505.09233 + title: IOHClustering + source: + - artificial-from-real-data + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_ma_bbob: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Generator that creates affine combinations of BBOB functions + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_ma_bbob + - impl_iohexperimenter + long_name: null + modality: + - multimodal + name: MA-BBOB + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3673908 + title: MA-BBOB + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_mpm2: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: nonlinear nonseparable nonsymmetric; scalable in terms of time to + evaluate the objective function + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_mpm2 + long_name: null + modality: + - multimodal + name: MPM2 + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf + title: MPM2 technical report TR15-01 + source: null + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: continuous +gen_mubqp: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: tunable variable and objective dimensions; tunable density and correlation between objectives -- name: mUBQP - suite/generator/single: generator - objectives: scalable - dimensionality: scalable - variable type: binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: yes (quadratic) - multi-fidelity: 'no' - reference: https://doi.org/10.1016/j.asoc.2013.11.008 - implementation: https://gitlab.com/aliefooghe/mocobench/ - source (real-world/artificial): '' - textual description: tunable variable and objective dimensions; tunable density and correlation - between objectives -- name: "\u03C1mTSP" - suite/generator/single: generator - objectives: scalable - dimensionality: scalable - variable type: permutations - constraints: no (apart from being permutations) - dynamic: 'no' - noise: 'no' - multimodal: yes (quadratic) - multi-fidelity: 'no' - reference: https://doi.org/10.1007/978-3-319-45823-6_40 - implementation: https://gitlab.com/aliefooghe/mocobench/ - source (real-world/artificial): '' - textual description: tunable variable and objective dimensions; tunable instance type (euclidian/random); - tunable correlation between objectives -- name: CEC2015-DMOO - suite/generator/single: suite - objectives: 2-3 - dimensionality: '?' - variable type: continuous - constraints: '?' - dynamic: 'yes' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: Benchmark Functions for CEC 2015 Special Session and Competition on Dynamic - Multi-objective Optimization - implementation: '' - source (real-world/artificial): '' - textual description: '' -- name: Ealain - suite/generator/single: generator - objectives: 1+ - dimensionality: scalable - variable type: continuous,binary,integer - constraints: optional - dynamic: optional - noise: 'no' - multimodal: '?' - multi-fidelity: optional - reference: https://doi.org/10.1145/3638530.3654299 - implementation: https://github.com/qrenau/Ealain - source (real-world/artificial): Real-world-like - textual description: Real-world-like, easily extensible to increase complexity -- name: MA-BBOB - suite/generator/single: generator - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3673908 - implementation: https://github.com/IOHprofiler/IOHexperimenter/blob/master/example/Competitions/MA-BBOB/Example_MABBOB.ipynb - source (real-world/artificial): artificial - textual description: Generator that creates affine combinations of BBOB functions -- name: MPM2 - suite/generator/single: generator - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf - implementation: https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py - source (real-world/artificial): '' - textual description: nonlinear nonseparable nonsymmetric; scalable in terms of time to evaluate - the objective function -- name: Convex DTLZ2 - suite/generator/single: single - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2013.2281535 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of DTLZ2 with a convex Pareto front (instead of concave) -- name: Inverted DTLZ1 - suite/generator/single: single - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2013.2281534 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of DTLZ1 with an inverted Pareto front -- name: Minus DTLZ - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2016.2587749 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of DTLZ that minimises the inverse of the base DTLZ functions -- name: Minus WFG - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2016.2587749 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of WFG that minimises the inverse of the base WFG functions -- name: L1-ZDT - suite/generator/single: suite - objectives: '2' - dimensionality: scalable - variable type: continuous;binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/1143997.1144179 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of ZDT with linkages between variables within one of two groups - but not between variables in a different group; Linear recombination operators - can potentially take advantage of the problem structure -- name: L2-ZDT - suite/generator/single: suite - objectives: '2' - dimensionality: scalable - variable type: continuous;binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/1143997.1144179 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of ZDT with linkages between all variables; Linear recombination - operators can potentially take advantage of the problem structure -- name: L3-ZDT - suite/generator/single: suite - objectives: '2' - dimensionality: scalable - variable type: continuous;binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/1143997.1144179 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of L2-ZDT using a mapping to prevent linear recombination operators - from potentially taking advantage of the problem structure -- name: L2-DTLZ - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/1143997.1144179 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of DTLZ2 and DTLZ3 with linkages between all variables; Linear - recombination operators can potentially take advantage of the problem structure -- name: L3-DTLZ - suite/generator/single: suite - objectives: 2+ - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/1143997.1144179 - implementation: '?' - source (real-world/artificial): '' - textual description: Variant of L2-DTLZ using a mapping to prevent linear recombination operators - from potentially taking advantage of the problem structure -- name: CEC2018 DT - CEC2018 Competition on Dynamic Multiobjective Optimisation - suite/generator/single: suite - objectives: 2 or 3 - dimensionality: scalable? - variable type: '?' - constraints: 'no' - dynamic: 'yes' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf - implementation: https://pymoo.org/problems/dynamic/df.html - source (real-world/artificial): artificial - textual description: '14 problems. Time-dependent: Pareto front/Pareto set geometry; - irregular Pareto front shapes; variable-linkage; number of disconnected Pareto - front segments; etc.' -- name: MODAct - multiobjective design of actuators - suite/generator/single: suite - objectives: 2 3 4 or 5 - dimensionality: '20' - variable type: mixed; integer and continuous - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/TEVC.2020.3020046 - implementation: https://pymoo.org/problems/constrained/modact.html - source (real-world/artificial): real-world - textual description: Realistic Constrained Multi-Objective Optimization Benchmark - Problems from Design. Need the https://github.com/epfl-lamd/modact package installed; evaluation - times around 20ms -- name: IOHClustering - suite/generator/single: suite; generator - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no ' - reference: https://arxiv.org/pdf/2505.09233 - implementation: https://github.com/IOHprofiler/IOHClustering - source (real-world/artificial): artificial, but based on real data - textual description: 'Set of benchmark problems from clustering: optimization task - is selecting cluster centers for a given set of data, with the number of clusters - defining problem dimensionality. Includes both a suite and a generator. Based on ML clustering datasets' -- name: GNBG-II - suite/generator/single: suite; generator - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 - implementation: https://github.com/rohitsalgotra/GNBG-II - source (real-world/artificial): artificial - textual description: Generalized Numerical Benchmark Generator (version 2). Also in IOH https://github.com/IOHprofiler/IOHGNBG -- name: GNBG - suite/generator/single: suite; generator - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://arxiv.org/abs/2312.07083 - implementation: https://github.com/Danial-Yazdani/GNBG-Generator - source (real-world/artificial): artificial - textual description: Generalized Numerical Benchmark Generator -- name: DynamicBinVal - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: binary - constraints: 'no' - dynamic: 'yes' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://arxiv.org/pdf/2404.15837 - implementation: https://github.com/IOHprofiler/IOHexperimenter - source (real-world/artificial): artificial - textual description: Four versions of the dynamic binary value problem -- name: PBO - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://dl.acm.org/doi/pdf/10.1145/3319619.3326810 - implementation: https://github.com/IOHprofiler/IOHexperimenter - source (real-world/artificial): artificial - textual description: Suite of 25 binary optimization problems -- name: W-model - suite/generator/single: generator - objectives: '1' - dimensionality: scalable - variable type: binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://dl.acm.org/doi/abs/10.1145/3205651.3208240?casa_token=S4U_Pi9f6MwAAAAA:U9ztNTPwmupT8K3GamWZfBL7-8fqjxPtr_kprv51vdwA-REsp0EyOFGa99BtbANb0XbqyrVg795hIw - implementation: https://github.com/thomasWeise/BBDOB_W_Model - source (real-world/artificial): artificial - textual description: Tunable generator for binary optimization based on several + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_mocobench + long_name: null + modality: + - quadratic + - multimodal + name: mUBQP + noise_type: null + objectives: + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1016/j.asoc.2013.11.008 + title: mUBQP benchmark + source: null + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: binary +gen_puboi: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: A benchmark in which variable importance is tunable, based on the + Walsh function. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_puboi + long_name: Polynomial Unconstrained Binary Optimization with tunable + importance + modality: null + name: PUBOi + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12 + title: PUBOi + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: binary +gen_randoptgen: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: A Unified Random Problem Generator for Single- and + Multi-Objective Optimization Problems with Mixed-Variable Input Spaces. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_randoptgen + long_name: RandOptGen + modality: + - multimodal + name: RandOptGen + noise_type: null + objectives: + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: null + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: integer + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +gen_rho_mnk_landscapes: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: tunable variable and objective dimensions; tunable multimodality + and correlation between objectives + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_mocobench + long_name: null + modality: + - multimodal + name: ρMNK-Landscapes + noise_type: null + objectives: + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1016/j.ejor.2012.12.019 + title: On the design of multi-objective evolutionary algorithms based on + NK-landscapes + source: null + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: binary +gen_rho_mtsp: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: tunable variable and objective dimensions; tunable instance type + (euclidean/random); tunable correlation between objectives + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_mocobench + long_name: null + modality: + - quadratic + - multimodal + name: ρmTSP + noise_type: null + objectives: + - 1 + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1007/978-3-319-45823-6_40 + title: On the impact of multi-objective scalability for the ρmTSP + source: null + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: unknown +gen_wmodel: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Tunable generator for binary optimization based on several difficulty features -- name: Submodular Optimitzation - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: binary - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181 - implementation: https://github.com/IOHprofiler/IOHexperimenter - source (real-world/artificial): artificial - textual description: set of graph-based submodular optimization problems from 4 - problem types -- name: CEC2013 - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://peerj.com/articles/cs-2671/CEC2013.pdf - implementation: https://github.com/P-N-Suganthan/CEC2013 - source (real-world/artificial): artificial - textual description: suite used for cec2013 competition. Also in IOH https://github.com/IOHprofiler/IOHexperimenter -- name: CEC2022 - suite/generator/single: suite - objectives: '1' - dimensionality: scalable - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '?' - multi-fidelity: 'no' - reference: https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf - implementation: https://github.com/P-N-Suganthan/2022-SO-BO - source (real-world/artificial): artificial - textual description: suite used for cec2022 competition. Also in IOH https://github.com/IOHprofiler/IOHexperimenter -- name: Onemax+Sphere / Zeromax+Sphere - suite/generator/single: single - objectives: '2' - dimensionality: scalable - variable type: binary and continuous;mixed; - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3449726.3459521 - implementation: - source (real-world/artificial): 'artificial' - textual description: '' -- name: Onemax+Sphere / DeceptiveTrap+RotatedEllipsoid - suite/generator/single: single - objectives: '2' - dimensionality: scalable - variable type: binary and continuous;mixed; - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3449726.3459521 - implementation: - source (real-world/artificial): 'artificial' - textual description: '' -- name: InverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoid - suite/generator/single: single - objectives: '2' - dimensionality: scalable - variable type: binary and continuous;mixed; - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: '' - multi-fidelity: 'no' - reference: https://doi.org/10.1145/3449726.3459521 - implementation: - source (real-world/artificial): 'artificial' - textual description: '' -- name: PorkchopPlotInterplanetaryTrajectory - suite/generator/single: suite - objectives: '1' - dimensionality: 2 - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://doi.org/10.1109/CEC65147.2025.11042973 - implementation: https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world - source (real-world/artificial): 'real-world' - textual description: '' -- name: KinematicsRobotArm - suite/generator/single: suite - objectives: '1' - dimensionality: 21 - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'no' - multi-fidelity: 'no' - reference: https://doi.org/10.1023/A:1013258808932 - implementation: https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world - source (real-world/artificial): 'real-world' - textual description: '' -- name: VehicleDynamics - suite/generator/single: suite - objectives: '1' - dimensionality: 2 - variable type: continuous - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - reference: https://www.scitepress.org/Papers/2023/121580/121580.pdf - implementation: https://zenodo.org/records/8307853 - source (real-world/artificial): 'real-world' - textual description: '' -- name: MECHBench - suite/generator/single: Problem Suite - variable type: Continuous - dimensionality: scalable' - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - source (real-world/artificial): Real-World Application - implementation: https://github.com/BayesOptApp/MECHBench - textual description: This is a set of problems with inspiration from Structural - Mechanics Design Optimization. The suite comprises three physical models, from - which the user may define different kind of problems which impact the final design - output. - reference: https://arxiv.org/abs/2511.10821 - other info: - partial evaluations: 'no' - full name: MECHBench - constraint properties: Hard Constraints - number of constraints: 1 or 2 - description of multimodality: Unstructured or non isotropic multimodality - key challenges / characteristics: Embeds physical simulations and is flexible - and modular - scientific motivation: Bridge the black-box optimization techniques to a Mechanical - Design Problem which require these kinds of algorithms - limitations: The models do not include fracture or damage mechanics, just plasticity. - implementation languages: Python - approximate evaluation time: Times -> from 1 minute to 7 minutes -- name: EXPObench - suite/generator/single: Problem Suite - variable type: Continuous, Integer, Categorical, Conditional - dimensionality: 10 to 135 - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'yes' - multimodal: Unknown - multi-fidelity: 'no' - source (real-world/artificial): Real-World Application - implementation: https://github.com/AlgTUDelft/ExpensiveOptimBenchmark - textual description: Wind farm layout optimization, gas filter design, pipe shape + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_wmodel + long_name: null + modality: null + name: W-model + noise_type: null + objectives: + - 1 + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/abs/10.1145/3205651.3208240 + title: W-model + source: + - artificial + tags: null + type: generator + variables: + - dim: + max: null + min: 1 + type: binary +impl_beacon: + description: Continuous Bi-objective Benchmark with Explicit Adjustable + COrrelatioN control + evaluation_time: + - negligible + language: Python + links: + - type: repository + url: https://github.com/Stebbet/BEACON/ + name: BEACON + requirements: null + type: implementation +impl_bonobench: + description: Bi-objective Numerical Optimization Benchmark (BONO-Bench) + evaluation_time: null + language: Python + links: + - type: repository + url: https://github.com/schaepermeier/bonobench + name: BONO-Bench + requirements: null + type: implementation +impl_bso_toolbox: + description: Building Spatial Design toolbox (TU/e) + evaluation_time: + - 1 second + - 40 seconds + language: C++ + links: + - type: repository + url: https://github.com/TUe-excellent-buildings/BSO-toolbox + name: BSO-toolbox + requirements: null + type: implementation +impl_car_structure: + description: JAXA LADSE benchmark problems + evaluation_time: null + language: null + links: + - type: website + url: http://ladse.eng.isas.jaxa.jp/benchmark/ + name: Car-structure benchmark + requirements: null + type: implementation +impl_cec2013: + description: Suganthan's reference implementation + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/P-N-Suganthan/CEC2013 + name: CEC2013 reference code + requirements: null + type: implementation +impl_cec2022: + description: Suganthan's reference implementation + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/P-N-Suganthan/2022-SO-BO + name: CEC2022 reference code + requirements: null + type: implementation +impl_cfd: + description: Expensive real-world CFD-based test problems + evaluation_time: + - 15m + - 30s + language: null + links: + - type: repository + url: https://bitbucket.org/arahat/cfd-test-problem-suite + name: CFD test problem suite + requirements: null + type: implementation +impl_coco: + description: 'Comparing Continuous Optimizers: black-box optimization benchmarking + platform' + evaluation_time: null + language: C/Python + links: + - type: repository + url: https://github.com/numbbo/coco + name: COCO framework + requirements: null + type: implementation +impl_coco_legacy: + description: Archived COCO download page that hosted the bbob-noisy suite + evaluation_time: null + language: C/Python + links: + - type: archive + url: + https://web.archive.org/web/20210416065610/https://coco.gforge.inria.fr/doku.php?id=downloads + name: COCO legacy (bbob-noisy) + requirements: null + type: implementation +impl_ealain: + description: Real-world-like extensible benchmark problem generator + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/qrenau/Ealain + name: Ealain + requirements: null + type: implementation +impl_emdo: + description: Not publicly available + evaluation_time: + - 8 minutes + language: Python + links: null + name: Electric Motor Design Optimization + requirements: null + type: implementation +impl_emo2017: + description: BBComp EMO-2017 real-world problem archive + evaluation_time: null + language: null + links: + - type: download + url: + https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/downloads/realworld-problems-bbcomp-EMO-2017.zip + name: EMO 2017 real-world problems + requirements: null + type: implementation +impl_etmof: + description: Evolutionary many-task optimization framework + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/songbai-liu/etmo + name: ETMOF + requirements: null + type: implementation +impl_expobench: + description: EXPensive Optimization benchmark library (wind farm layout, gas + filter design, pipe shape, hyperparameter tuning, hospital simulation) + evaluation_time: + - 80 seconds + - 2 seconds + language: Python + links: + - type: repository + url: https://github.com/AlgTUDelft/ExpensiveOptimBenchmark + name: EXPObench + requirements: null + type: implementation +impl_gasoline: + description: Proprietary Matlab Simulink + Wave RT co-simulation + evaluation_time: null + language: Matlab Simulink / Wave RT + links: + - type: paper + url: https://doi.org/10.1016/j.ejor.2022.08.032 + name: Gasoline direct injection engine design + requirements: null + type: implementation +impl_gbea: + description: Game-Benchmark for Evolutionary Algorithms (COCO fork) + evaluation_time: + - 34 seconds + - 5 seconds + language: null + links: + - type: repository + url: https://github.com/ttusar/coco-gbea + name: coco-gbea + requirements: null + type: implementation +impl_gnbg: + description: Generalized Numerical Benchmark Generator + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/Danial-Yazdani/GNBG-Generator + name: GNBG Generator + requirements: null + type: implementation +impl_gnbg_ii: + description: Generalized Numerical Benchmark Generator version 2 + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/rohitsalgotra/GNBG-II + name: GNBG-II + requirements: null + type: implementation +impl_iohclustering: + description: Clustering-based optimization benchmark built on ML datasets + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/IOHprofiler/IOHClustering + name: IOHClustering + requirements: null + type: implementation +impl_iohexperimenter: + description: IOHprofiler experimenter framework + evaluation_time: null + language: C++/Python + links: + - type: repository + url: https://github.com/IOHprofiler/IOHexperimenter + name: IOHexperimenter + requirements: null + type: implementation +impl_iohgnbg: + description: IOHprofiler version of GNBG + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/IOHprofiler/IOHGNBG + name: IOHGNBG + requirements: null + type: implementation +impl_jsec2019: + description: JPNSEC EC-Symposium 2019 competition problem + evaluation_time: + - 3s + language: null + links: + - type: website + url: + http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html + name: JSEC 2019 competition + requirements: null + type: implementation +impl_ma_bbob: + description: Example notebook for MA-BBOB in IOHexperimenter + evaluation_time: null + language: null + links: + - type: example + url: + https://github.com/IOHprofiler/IOHexperimenter/blob/master/example/Competitions/MA-BBOB/Example_MABBOB.ipynb + name: MA-BBOB (IOHexperimenter) + requirements: null + type: implementation +impl_mechbench: + description: Structural mechanics design optimization benchmark + evaluation_time: + - 1 minute + - 7 minutes + language: Python + links: + - type: repository + url: https://github.com/BayesOptApp/MECHBench + name: MECHBench + requirements: null + type: implementation +impl_mf2: + description: Multi-fidelity test function collection + evaluation_time: null + language: Python + links: + - type: repository + url: https://github.com/sjvrijn/mf2 + name: mf2 + requirements: null + type: implementation +impl_mmopp: + description: ECI lab distribution page for MMOPP + evaluation_time: null + language: null + links: + - type: website + url: http://www5.zzu.edu.cn/ecilab/info/1036/1251.htm + name: MMOPP + requirements: null + type: implementation +impl_mocobench: + description: Multi-objective combinatorial optimization benchmark + evaluation_time: null + language: C++ + links: + - type: repository + url: https://gitlab.com/aliefooghe/mocobench/ + name: mocobench + requirements: null + type: implementation +impl_modact: + description: EPFL-LAMD modact package + evaluation_time: + - 20ms + language: null + links: + - type: repository + url: https://github.com/epfl-lamd/modact + name: modact + requirements: null + type: implementation +impl_morepo: + description: Multi-objective optimisation problem repository + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/MCDMSociety/MOrepo + name: MOrepo + requirements: null + type: implementation +impl_mpm2: + description: Python implementation of MPM2 distributed with smoof + evaluation_time: null + language: Python + links: + - type: source + url: https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py + name: MPM2 (smoof) + requirements: null + type: implementation +impl_puboi: + description: A benchmark in which variable importance is tunable, based on the + Walsh function + evaluation_time: null + language: Python / C++ + links: + - type: repository + url: https://gitlab.com/verel/pubo-importance-benchmark + name: PUBO Importance Benchmark + requirements: null + type: implementation +impl_pycutest: + description: Python interface to CUTEst + evaluation_time: null + language: Python / C++ / Fortran + links: + - type: repository + url: https://github.com/jfowkes/pycutest + name: pycutest + requirements: null + type: implementation +impl_pymoo: + description: Multi-objective optimization in Python + evaluation_time: null + language: Python + links: + - type: repository + url: https://github.com/anyoptimization/pymoo + name: pymoo + requirements: null + type: implementation +impl_radar_waveform: + description: Radar waveform design reference implementation + evaluation_time: null + language: null + links: + - type: website + url: http://code.evanhughes.org/ + name: Evan Hughes radar waveform code + requirements: null + type: implementation +impl_randoptgen: + description: Unified Random Problem Generator for Single- and Multi-Objective + Optimization with Mixed-Variable Input Spaces + evaluation_time: + - milliseconds + language: Python + links: + - type: repository + url: https://github.com/MALEO-research-group/RandOptGen + - type: example + url: https://doi.org/10.1145/3712256.3726478 + name: RandOptGen + requirements: null + type: implementation +impl_reproblems: + description: Real-world inspired multi-objective optimization problem suite + evaluation_time: null + language: Python + links: + - type: repository + url: https://github.com/ryojitanabe/reproblems + name: reproblems + requirements: null + type: implementation +impl_transfer_rf_bbob_rw: + description: Real-world BBOB-like problem implementations (Porkchop, + KinematicsRobotArm) + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/ShuaiqunPan/Transfer_Random_forests_BBOB_Real_world + name: Transfer Random Forests BBOB Real-world + requirements: null + type: implementation +impl_tulipa: + description: Large linear program for optimal investment and operation of + energy systems + evaluation_time: + - hours + - minutes + language: Julia / JuMP + links: + - type: website + url: https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/ + - type: example + url: https://github.com/TulipaEnergy/Tulipa-OBZ-CaseStudy + name: TulipaEnergyModel.jl + requirements: null + type: implementation +impl_vehicle_dynamics: + description: Zenodo archive for the vehicle dynamics benchmark + evaluation_time: null + language: null + links: + - type: archive + url: https://zenodo.org/records/8307853 + name: VehicleDynamics (Zenodo) + requirements: null + type: implementation +impl_wmodel: + description: Tunable generator for binary optimization + evaluation_time: null + language: null + links: + - type: repository + url: https://github.com/thomasWeise/BBDOB_W_Model + name: BBDOB W-Model + requirements: null + type: implementation +suite_amvop: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: + - multimodal + name: AMVOP + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281531 + title: AMVOP + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: categorical + - dim: + max: null + min: 1 + type: integer + - dim: + max: null + min: 1 + type: continuous +suite_bbob: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco + long_name: null + modality: + - multimodal + name: BBOB + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1080/10556788.2020.1808977 + title: 'COCO: a platform for comparing continuous optimizers in a black-box setting' + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_bbob_biobj: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco + long_name: null + modality: + - multimodal + name: BBOB-biobj + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.48550/arXiv.1604.00359 + title: BBOB bi-objective test suite + source: null + tags: null + type: suite + variables: + - dim: + max: 40 + min: 2 + type: continuous +suite_bbob_biobj_mixint: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco + long_name: null + modality: + - multimodal + name: BBOB-biobj-mixint + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3321707.3321868 + title: BBOB bi-objective mixed-integer test suite + source: null + tags: null + type: suite + variables: + - dim: + max: 160 + min: 5 + type: integer + - dim: + max: 160 + min: 5 + type: continuous +suite_bbob_constrained: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco + long_name: null + modality: + - multimodal + name: BBOB-constrained + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: http://numbbo.github.io/coco-doc/bbob-constrained/ + title: bbob-constrained documentation + source: null + tags: null + type: suite + variables: + - dim: + max: 40 + min: 2 + type: continuous +suite_bbob_largescale: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco + long_name: null + modality: + - multimodal + name: BBOB-largescale + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.48550/arXiv.1903.06396 + title: BBOB large-scale test suite + source: null + tags: null + type: suite + variables: + - dim: + max: 640 + min: 20 + type: continuous +suite_bbob_mixint: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco + long_name: null + modality: + - multimodal + name: BBOB-mixint + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3321707.3321868 + title: BBOB mixed-integer test suite + source: null + tags: null + type: suite + variables: + - dim: + max: 160 + min: 5 + type: integer + - dim: + max: 160 + min: 5 + type: continuous +suite_bbob_noisy: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_coco_legacy + long_name: null + modality: + - multimodal + name: BBOB-noisy + noise_type: + - noisy + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://hal.inria.fr/inria-00369466 + title: 'Real-parameter black-box optimization benchmarking: noisy functions definitions' + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_bp: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: BP + noise_type: + - unknown + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/CEC.2019.8790277 + title: BP benchmark + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_brachytherapy: + allows_partial_evaluation: yes + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: + max: null + min: 1 + type: unknown + description: Treatment planning for internal radiation therapy. + Multi-objective with aggregated objectives; no public source code. + dynamic_type: null + evaluation_time: null + fidelity_levels: + - 1 + - 2 + implementations: null + long_name: Brachytherapy treatment planning + modality: + - multimodal + name: Brachytherapy treatment planning + noise_type: null + objectives: + - 2 + - 3 + problems: null + references: + - authors: [] + link: + type: null + url: https://www.sciencedirect.com/science/article/pii/S1538472123016781 + title: Brachytherapy treatment planning + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: 500 + min: 100 + type: continuous +suite_car_structure: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: 54 + type: unknown + description: 54 constraints + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_car_structure + long_name: null + modality: null + name: Car structure + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3205651.3205702 + title: Car structure design benchmark + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: 222 + min: 144 + type: integer +suite_cdmp: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: null + dynamic_type: + - unknown + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: CDMP + noise_type: + - unknown + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3321707.3321878 + title: CDMP benchmark + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_cec2013: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: suite used for cec2013 competition. Also in IOH. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_cec2013 + - impl_iohexperimenter + long_name: null + modality: null + name: CEC2013 + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://peerj.com/articles/cs-2671/CEC2013.pdf + title: CEC2013 definitions + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_cec2015_dmoo: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: '?' + number: null + type: unknown + description: null + dynamic_type: + - dynamic + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: CEC2015-DMOO + noise_type: null + objectives: + - 2 + - 3 + problems: null + references: + - authors: [] + link: null + title: Benchmark Functions for CEC 2015 Special Session and Competition on + Dynamic Multi-objective Optimization + source: null + tags: null + type: suite + variables: + - dim: 0 + type: continuous +suite_cec2018_dt: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: '14 problems. Time-dependent: Pareto front/Pareto set geometry; irregular + Pareto front shapes; variable-linkage; number of disconnected Pareto front segments; + etc.' + dynamic_type: + - dynamic + evaluation_time: null + fidelity_levels: null + implementations: + - impl_pymoo + long_name: CEC2018 Competition on Dynamic Multiobjective Optimisation + modality: null + name: CEC2018 DT + noise_type: null + objectives: + - 2 + - 3 + problems: null + references: + - authors: [] + link: + type: null + url: + https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf + title: CEC2018 DMOP Competition TR + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: unknown +suite_cec2022: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: suite used for cec2022 competition. Also in IOH. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_cec2022 + - impl_iohexperimenter + long_name: null + modality: null + name: CEC2022 + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: + https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf + title: CEC2022 TR + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_cfd: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: expensive evaluations 30s-15m + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_cfd + long_name: null + modality: null + name: CFD + noise_type: null + objectives: + - 1 + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1007/978-3-319-99259-4_24 + title: CFD test problem suite + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: unknown +suite_cre: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_reproblems + long_name: null + modality: null + name: CRE + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + problems: null + references: + - authors: + - Ryoji Tanabe + - Hisao Ishibuchi + link: + type: null + url: https://doi.org/10.1016/j.asoc.2020.106078 + title: Easy-to-evaluate real-world multi-objective optimization problems + source: + - real-world-like + tags: null + type: suite + variables: + - dim: + max: 7 + min: 3 + type: continuous + - dim: + max: 7 + min: 3 + type: integer +suite_cuter: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: A constrained and unconstrained testing environment. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: CUTEr + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/10.1145/962437.962439 + title: CUTEr + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: integer + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +suite_cutest: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: box + - equality: null + hard: some + number: + max: null + min: 1 + type: unknown + description: CUTEst for optimization software + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_pycutest + long_name: Constrained and Unconstrained Testing Environment with safe threads + modality: + - multimodal + name: CUTEst + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://link.springer.com/article/10.1007/s10589-014-9687-3 + title: CUTEst + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: integer + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +suite_dtlz: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_pymoo + long_name: null + modality: null + name: DTLZ + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: + - Kalyanmoy Deb + - Lothar Thiele + - Marco Laumanns + - Eckart Zitzler + link: + type: null + url: https://doi.org/10.1109/CEC.2002.1007032 + title: Scalable multi-objective optimization test problems + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_dynamicbinval: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Four versions of the dynamic binary value problem + dynamic_type: + - dynamic + evaluation_time: null + fidelity_levels: null + implementations: + - impl_iohexperimenter + long_name: null + modality: null + name: DynamicBinVal + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://arxiv.org/pdf/2404.15837 + title: DynamicBinVal + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary +suite_emo2017: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_emo2017 + long_name: null + modality: null + name: EMO2017 + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/ + title: BBComp EMO 2017 + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: 24 + min: 4 + type: continuous +suite_etmof: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: + - dynamic + evaluation_time: null + fidelity_levels: null + implementations: + - impl_etmof + long_name: null + modality: null + name: ETMOF + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + - 11 + - 12 + - 13 + - 14 + - 15 + - 16 + - 17 + - 18 + - 19 + - 20 + - 21 + - 22 + - 23 + - 24 + - 25 + - 26 + - 27 + - 28 + - 29 + - 30 + - 31 + - 32 + - 33 + - 34 + - 35 + - 36 + - 37 + - 38 + - 39 + - 40 + - 41 + - 42 + - 43 + - 44 + - 45 + - 46 + - 47 + - 48 + - 49 + - 50 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.48550/arXiv.2110.08033 + title: Evolutionary many-task optimization framework + source: null + tags: null + type: suite + variables: + - dim: + max: 10000 + min: 25 + type: continuous +suite_expobench: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: box + - equality: null + hard: some + number: null + type: unknown + description: Wind farm layout optimization, gas filter design, pipe shape optimization, hyperparameter tuning, and hospital simulation - reference: https://doi.org/10.1016/j.asoc.2023.110744 - other info: - partial evaluations: 'no' - full name: EXPensive Optimization benchmark library - constraint properties: Hard Constraints, Soft Constraints, Box Constraints, only - box constraints implemented, others appear as penalty in objective - number of constraints: 2 per variable (box), other constraints unknown (simulator - fails) - form of noise model: real-life (unknown) - type of noise space: Observational - key challenges / characteristics: Expensive objectives - scientific motivation: Address the lack of real-life expensive benchmarks - limitations: single-objective only, constraints are handled naively (penalty in - objective), no parallelization - implementation languages: Python - approximate evaluation time: 2 to 80 seconds -- name: Gasoline direct injection engine design - suite/generator/single: Single Problem - variable type: Continuous, Ordinal - dimensionality: '7' - objectives: '2' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: Unknown - multi-fidelity: 'yes' - source (real-world/artificial): Real-World Application - implementation: https://doi.org/10.1016/j.ejor.2022.08.032 - textual description: 'A multi-objective optimization problem seeking to minimize - fuel consumption and NOx emissions over a two-minute dynamic duty cycle, subject - to five constraints (turbine inlet temperature, number of knock occurrences, peak - cylinder pressure, peak cylinder pressure rise, total work). Seven decision variables - are defined: four define the hardware choices of cylinder compression ratio, turbo - machinery and EGR cooler sizing; three relate to control variables that parameterise - the engine control logic.' - other info: - partial evaluations: Unknown - constraint properties: Hard Constraints, Soft Constraints - number of constraints: '5' - key challenges / characteristics: Expensive - limitations: Proprietary - implementation languages: Matlab Simulink and Wave RT co-simulation -- name: BEACON - suite/generator/single: Generator - variable type: Continuous - dimensionality: scalable - objectives: '2' - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - source (real-world/artificial): Artificially Generated - implementation: https://github.com/Stebbet/BEACON/ - textual description: Generator for bi-objective benchmark problems with explicitly - controlled correlations in continuous spaces. - reference: https://dl.acm.org/doi/10.1145/3712255.3734303 - other info: - partial evaluations: 'no' - full name: Continuous Bi-objective Benchmark problems with Explicit Adjustable - COrrelatioN control - constraint properties: Box Constraints - number of constraints: '0' - description of multimodality: Random - key challenges / characteristics: Multimodal, different correlations among objectives - scientific motivation: Controlled correlation among objectives - limitations: No analytical Pareto front, only bi-objective - implementation languages: Python - approximate evaluation time: Negligible -- name: TulipaEnergy - suite/generator/single: Problem Suite - variable type: Continuous - dimensionality: scalable - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'yes' - multimodal: 'no' - multi-fidelity: 'yes' - source (real-world/artificial): Real-World Application - implementation: https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/ - textual description: Determine the optimal investment and operation decisions for - different types of assets in the energy system (production, consumption, conversion, - storage, and transport), while minimizing loss of load. - reference: See https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references - other info: - partial evaluations: Unknown - full name: TulipaEnergyModel.jl - constraint properties: Hard Constraints, Soft Constraints - number of constraints: millions - type of dynamicism: none - form of noise model: "depends on input \u2014 still working on stochastic inputs" - type of noise space: Parameter - key challenges / characteristics: modeled as a potentially very large linear program, - different fidelities possible - scientific motivation: new techniques for solving large whitebox linear optimization - problems - limitations: not yet stochastic - implementation languages: Julia / JMP - approximate evaluation time: from minutes to hours - links to usage examples: https://github.com/TulipaEnergy/Tulipa-OBZ-CaseStudy -- name: ATO - suite/generator/single: Single Problem - variable type: Continuous - dimensionality: '10' - objectives: '2' - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'no' - multi-fidelity: 'no' - source (real-world/artificial): Real-World Application - implementation: '-' - textual description: Parameters of the Modules of the Automatic Train Operation - should be optimized. The parameters are continuous with different ranges. There - are two objectives (minimizing energy consumption, minimizing driving duration. - other info: - partial evaluations: 'no' -- name: Brachytherapy treatment planning - suite/generator/single: Problem Suite - variable type: Continuous - dimensionality: 100-500 - objectives: 2-3 - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'yes' - source (real-world/artificial): Real-World Application - textual description: Treatment planning for internal radiation therapy - reference: https://www.sciencedirect.com/science/article/pii/S1538472123016781 - other info: - partial evaluations: 'yes' - full name: Brachytherapy treatment planning - constraint properties: Hard Constraints - number of constraints: scalable - key challenges / characteristics: Multi-objective; aggregated objectives - limitations: No public source code -- name: FleetOpt - suite/generator/single: Single Problem - variable type: Integer - dimensionality: 'Upper level: 54; lower level: 13208' - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: Unknown - multi-fidelity: 'no' - source (real-world/artificial): Real-World Application - implementation: 'Not public: was done for real client with their private data' - textual description: 'Healthcare organisation in the UK provided data about their - current fleet of vehicles to conduct non-emergency heathcare trips in the Argyll - and Bute region of Scotland, UK. They also provided historical data about the - trips the vehicles took and about the bases which the vehicles return to. The - aim is to reduce the existing fleet of vehicles while still ensuring all trips - can be covered. Moving a vehicle from one base to another to help cover trips - is OK as long as the original base can still cover its trips. Link to paper with - more details: https://dl.acm.org/doi/abs/10.1145/3638530.3664137' - reference: https://dl.acm.org/doi/abs/10.1145/3638530.3664137 - other info: - partial evaluations: 'yes' -- name: Building spatial design - suite/generator/single: Single Problem - variable type: Continuous, Boolean - dimensionality: scalable depending on problem size (e.g. 90 for) - objectives: '2' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: Unknown - multi-fidelity: 'no' - source (real-world/artificial): Real-World Application - implementation: https://github.com/TUe-excellent-buildings/BSO-toolbox - textual description: 'Optimise the spatial layout of a building to: minimise energy - consumption for climate control, and minimise the strain on the structure' - reference: https://hdl.handle.net/1887/81789 - other info: - partial evaluations: 'no' - full name: Building spatial design - constraint properties: Hard Constraints, Box Constraints, Permutation Constraints - number of constraints: 2065 (as example, depends on problem size) - key challenges / characteristics: Many hard constraints (simulator cannot evaluate - the solution if these are violated); Mixed-variable search space (continuous - + binary); Multiple objectives; (Somewhat) expensive solution evaluations - implementation languages: C++ - approximate evaluation time: Roughly 1 second per evaluation for the smallest - considered design, and roughly 40 seconds for the larger designs we considered. - (Even the larger designs we considered are still relatively small for the considered - problem.) -- name: Electric Motor Design Optimization - suite/generator/single: Single Problem - variable type: Continuous, Integer - dimensionality: '13' - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'yes' - multimodal: 'yes' - multi-fidelity: 'no' - source (real-world/artificial): Real-World Application - implementation: Implementation not freely available - textual description: The goal is to find a design of a synchronous electric motor - for power steering systems that minimizes costs and satisfies all constraints. - reference: https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf (paper in Slovene) - other info: - partial evaluations: 'no' - full name: Electric Motor Design Optimization - constraint properties: Hard Constraints, Soft Constraints, Box Constraints - number of constraints: '12' - description of multimodality: Constraints are multimodal - key challenges / characteristics: Time-consuming solution evaluation, highly-constrained - problem - scientific motivation: Challenging to find good solutions in a limited time - limitations: 'Unavailability, even if available, it wouldn''t be helpful to use - for benchmarking due taking a long time to evaluate a single solution ' - implementation languages: Python - approximate evaluation time: 8 minutes - general: This is not an available problem, but could be interesting to show to - researchers which difficulties appear in real-world problems -- name: BONO-Bench - suite/generator/single: Generator - variable type: Continuous - dimensionality: scalable - objectives: '2' - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - source (real-world/artificial): Artificially Generated - implementation: https://github.com/schaepermeier/bonobench - textual description: Bi-objective problem generator and suite with scalable continuous - decision space. Features complex problem properties (different types of multimodality - and challenges in decision and objective space) as well as Pareto front approximations - with error guarantees for the hypervolume and exact R2 indicators. - other info: - partial evaluations: 'no' - full name: Bi-objective Numerical Optimization Benchmark (BONO-Bench) - constraint properties: Box Constraints - implementation languages: Python -- name: RandOptGen - suite/generator/single: Generator - variable type: Continuous, Integer, Boolean - dimensionality: scalable - objectives: scalable - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - source (real-world/artificial): Artificially Generated - implementation: https://github.com/MALEO-research-group/RandOptGen - textual description: 'RandOptGen: A Unified Random Problem Generator for Single-and - Multi-Objective Optimization Problems with Mixed-Variable Input Spaces' - other info: - partial evaluations: 'no' - full name: RandOptGen - implementation languages: Python - approximate evaluation time: milliseconds - links to usage examples: https://doi.org/10.1145/3712256.3726478 -- name: CUTEr - suite/generator/single: Problem Suite - variable type: Continuous, Integer, Boolean - dimensionality: scalable - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: Unknown - multi-fidelity: 'no' - source (real-world/artificial): Artificially Generated - implementation: Not Found - textual description: A constrained and unconstrained testing environment - reference: https://dl.acm.org/doi/10.1145/962437.962439 - other info: - partial evaluations: 'no' -- name: CUTEst - suite/generator/single: Problem Suite - variable type: Continuous, Integer, Boolean - dimensionality: scalable - objectives: '1' - constraints: 'yes' - dynamic: 'no' - noise: 'no' - multimodal: 'yes' - multi-fidelity: 'no' - source (real-world/artificial): Artificially Generated - implementation: https://github.com/jfowkes/pycutest - textual description: The Constrained and Unconstrained Testing Environment with - safe threads (CUTEst) for optimization software - reference: https://link.springer.com/article/10.1007/s10589-014-9687-3 - other info: - partial evaluations: 'no' - full name: 'Constrained and Unconstrained Testing Environment with safe threads ' - constraint properties: Soft Constraints, Box Constraints - number of constraints: scalable - implementation languages: Python, C++, Fortran - general: 'Python implementation: https://github.com/jfowkes/pycutest' -- name: PUBOi - suite/generator/single: Generator - variable type: Boolean - dimensionality: scalable - objectives: '1' - constraints: 'no' - dynamic: 'no' - noise: 'no' - multimodal: Unknown - multi-fidelity: 'no' - source (real-world/artificial): Artificially Generated - implementation: https://gitlab.com/verel/pubo-importance-benchmark - textual description: A benchmark in which variable importance is tunable, based - on the Walsh function - reference: https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12 - other info: - partial evaluations: 'no' - full name: Polynomial Unconstrained Binary Optimization - key challenges / characteristics: Tunable variable importance - implementation languages: Python, C++ + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_expobench + long_name: EXPensive Optimization benchmark library + modality: null + name: EXPObench + noise_type: + - real-life + - observational + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1016/j.asoc.2023.110744 + title: EXPObench + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: 135 + min: 10 + type: categorical + - dim: + max: 135 + min: 10 + type: integer + - dim: + max: 135 + min: 10 + type: continuous +suite_gbea: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: expensive evaluations 5s-35s, RW-GAN-Mario and TopTrumps are part + of GBEA + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_gbea + long_name: null + modality: + - multimodal + name: GBEA + noise_type: + - noisy + objectives: + - 1 + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/3321707.3321805 + title: Game benchmark for evolutionary algorithms + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_gnbg: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Generalized Numerical Benchmark Generator + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_gnbg + long_name: null + modality: null + name: GNBG + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://arxiv.org/abs/2312.07083 + title: GNBG + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_gnbg_ii: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Generalized Numerical Benchmark Generator (version 2). Also + available in IOH. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_gnbg_ii + - impl_iohgnbg + long_name: null + modality: null + name: GNBG-II + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 + title: GNBG-II + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_iohclustering: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: 'Set of benchmark problems from clustering: optimization task is selecting + cluster centers for a given set of data.' + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_iohclustering + long_name: null + modality: + - multimodal + name: IOHClustering + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://arxiv.org/pdf/2505.09233 + title: IOHClustering + source: + - artificial-from-real-data + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_kinematics_robotarm: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_transfer_rf_bbob_rw + long_name: null + modality: + - unimodal + name: KinematicsRobotArm + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1023/A:1013258808932 + title: Kinematics of a robot arm + source: + - real-world + tags: null + type: suite + variables: + - dim: 21 + type: continuous +suite_l1_zdt: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of ZDT with linkages between variables within groups + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: L1-ZDT + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/1143997.1144179 + title: Linkage ZDT/DTLZ variants + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +suite_l2_dtlz: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of DTLZ2/DTLZ3 with linkages between all variables + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: L2-DTLZ + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/1143997.1144179 + title: Linkage ZDT/DTLZ variants + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_l2_zdt: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of ZDT with linkages between all variables + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: L2-ZDT + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/1143997.1144179 + title: Linkage ZDT/DTLZ variants + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +suite_l3_dtlz: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of L2-DTLZ with anti-linkage mapping + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: L3-DTLZ + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/1143997.1144179 + title: Linkage ZDT/DTLZ variants + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_l3_zdt: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of L2-ZDT with anti-linkage mapping + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: L3-ZDT + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1145/1143997.1144179 + title: Linkage ZDT/DTLZ variants + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous +suite_maop: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: MaOP + noise_type: + - unknown + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1016/j.swevo.2019.02.003 + title: MaOP benchmark + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_mechbench: + allows_partial_evaluation: no + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: + - 1 + - 2 + type: unknown + description: Set of problems inspired by Structural Mechanics Design + Optimization. Embeds physical simulations (plasticity only, no + fracture/damage). Unstructured/non-isotropic multimodality. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_mechbench + long_name: MECHBench + modality: + - multimodal + name: MECHBench + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://arxiv.org/abs/2511.10821 + title: MECHBench + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_mf2: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: + - 1 + - 2 + implementations: + - impl_mf2 + long_name: null + modality: null + name: MF2 + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.21105/joss.02049 + title: 'mf2: a collection of multi-fidelity benchmark functions in Python' + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_minus_dtlz: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of DTLZ that minimises the inverse of the base DTLZ + functions + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: Minus DTLZ + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2016.2587749 + title: Minus DTLZ / Minus WFG + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_minus_wfg: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Variant of WFG that minimises the inverse of the base WFG + functions + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: Minus WFG + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2016.2587749 + title: Minus DTLZ / Minus WFG + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_mmopp: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_mmopp + long_name: null + modality: + - multimodal + name: MMOPP + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + problems: null + references: + - authors: [] + link: + type: null + url: + http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412 + title: MMOPP technical report + source: null + tags: null + type: suite + variables: + - dim: 0 + type: unknown +suite_modact: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: Realistic Constrained Multi-Objective Optimization Benchmark + Problems from Design. + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_modact + - impl_pymoo + long_name: multiobjective design of actuators + modality: null + name: MODAct + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2020.3020046 + title: MODAct + source: + - real-world + tags: null + type: suite + variables: + - dim: 20 + type: continuous + - dim: 20 + type: integer +suite_morepo: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: '?' + number: null + type: unknown + description: null + dynamic_type: + - unknown + evaluation_time: null + fidelity_levels: null + implementations: + - impl_morepo + long_name: null + modality: null + name: MOrepo + noise_type: + - unknown + objectives: + - 2 + problems: null + references: null + source: null + tags: null + type: suite + variables: + - dim: 0 + type: unknown +suite_pbo: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: Suite of 25 binary optimization problems + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_iohexperimenter + long_name: null + modality: null + name: PBO + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://dl.acm.org/doi/pdf/10.1145/3319619.3326810 + title: PBO benchmarks + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary +suite_porkchop: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_transfer_rf_bbob_rw + long_name: null + modality: + - multimodal + name: PorkchopPlotInterplanetaryTrajectory + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/CEC65147.2025.11042973 + title: Porkchop plot interplanetary trajectory benchmark + source: + - real-world + tags: null + type: suite + variables: + - dim: 2 + type: continuous +suite_re: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_reproblems + long_name: null + modality: null + name: RE + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + problems: null + references: + - authors: + - Ryoji Tanabe + - Hisao Ishibuchi + link: + type: null + url: https://doi.org/10.1016/j.asoc.2020.106078 + title: Easy-to-evaluate real-world multi-objective optimization problems + source: + - real-world-like + tags: null + type: suite + variables: + - dim: + max: 7 + min: 2 + type: continuous + - dim: + max: 7 + min: 2 + type: integer +suite_rwmvop: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: RWMVOP + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281531 + title: RWMVOP + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: categorical + - dim: + max: null + min: 1 + type: integer + - dim: + max: null + min: 1 + type: continuous +suite_sbox_cost: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: problems from BBOB but allows instances with the optimum close to + the boundary + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_iohexperimenter + long_name: null + modality: + - multimodal + name: SBOX-COST + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.48550/arXiv.2305.12221 + title: SBOX-COST + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_sdp: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: + - dynamic + evaluation_time: null + fidelity_levels: null + implementations: null + long_name: null + modality: null + name: SDP + noise_type: + - unknown + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: [] + link: + type: null + url: https://doi.org/10.1109/TCYB.2019.2896021 + title: SDP dynamic multi-objective benchmark + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_submodular: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: set of graph-based submodular optimization problems from 4 + problem types + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_iohexperimenter + long_name: null + modality: null + name: Submodular Optimization + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181 + title: Submodular optimization benchmark + source: + - artificial + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary +suite_tulipa_energy: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: + - equality: null + hard: yes + number: null + type: unknown + - equality: null + hard: some + number: null + type: unknown + description: Determine the optimal investment and operation decisions for + different assets in the energy system (production, consumption, conversion, + storage, transport) while minimizing loss of load. Modelled as a potentially + very large linear program with multiple fidelity levels. + dynamic_type: null + evaluation_time: null + fidelity_levels: + - 1 + - 2 + implementations: + - impl_tulipa + long_name: TulipaEnergyModel.jl + modality: + - unimodal + name: TulipaEnergy + noise_type: + - parameter + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: + https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references + title: TulipaEnergyModel.jl scientific references + source: + - real-world + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_vehicle_dynamics: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_vehicle_dynamics + long_name: null + modality: + - multimodal + name: VehicleDynamics + noise_type: null + objectives: + - 1 + problems: null + references: + - authors: [] + link: + type: null + url: https://www.scitepress.org/Papers/2023/121580/121580.pdf + title: VehicleDynamics benchmark + source: + - real-world + tags: null + type: suite + variables: + - dim: 2 + type: continuous +suite_wfg: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_pymoo + long_name: null + modality: null + name: WFG + noise_type: null + objectives: + - 2 + - 3 + - 4 + - 5 + - 6 + - 7 + - 8 + - 9 + - 10 + problems: null + references: + - authors: + - Simon Huband + - Philip Hingston + - Luigi Barone + - Lyndon While + link: + type: null + url: https://doi.org/10.1109/TEVC.2005.861417 + title: A review of multiobjective test problems and a scalable test problem + toolkit + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: continuous +suite_zdt: + allows_partial_evaluation: null + can_evaluate_objectives_independently: null + code_examples: null + constraints: null + description: null + dynamic_type: null + evaluation_time: null + fidelity_levels: null + implementations: + - impl_pymoo + long_name: null + modality: null + name: ZDT + noise_type: null + objectives: + - 2 + problems: null + references: + - authors: + - Eckart Zitzler + - Kalyanmoy Deb + - Lothar Thiele + link: + type: null + url: https://doi.org/10.1162/106365600568202 + title: 'Comparison of multiobjective evolutionary algorithms: empirical results' + source: null + tags: null + type: suite + variables: + - dim: + max: null + min: 1 + type: binary + - dim: + max: null + min: 1 + type: continuous From f34d46fea11b95ebf135113c537a7b03a2529546 Mon Sep 17 00:00:00 2001 From: Olaf Mersmann Date: Wed, 22 Apr 2026 22:28:50 +0200 Subject: [PATCH 5/6] fix: Make references top level objects --- SCHEMA.md | 57 ++- examples/cobi.py | 14 +- examples/emdo.py | 16 +- examples/problems.py | 888 +++++++++++++------------------- problems.yaml | 1057 ++++++++++++++++++++++----------------- src/opltools/schema.py | 21 +- tests/test_library.py | 30 ++ tests/test_reference.py | 3 +- 8 files changed, 1040 insertions(+), 1046 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index ae9c18b..60bf7b1 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -22,6 +22,7 @@ Three design choices shape everything below: - [Library](#library) - [Thing types](#thing-types) - [Implementation](#implementation) + - [Reference](#reference) - [ProblemLike](#problemlike) (shared fields) - [Problem](#problem) - [Suite](#suite) @@ -29,7 +30,7 @@ Three design choices shape everything below: - [Shared building blocks](#shared-building-blocks) - [Variable](#variable) / [VariableType](#variabletype) - [Constraint](#constraint) / [ConstraintType](#constrainttype) - - [Reference](#reference) / [Link](#link) + - [Link](#link) - [ValueRange](#valuerange) - [YesNoSome](#yesnosome) @@ -43,26 +44,33 @@ IDs are free-form but must be unique and the convention is to add a prefix marki | Prefix | Type | |---------|------------------| | `impl_` | Implementation | +| `ref_` | Reference | | `fn_` | Problem | | `suite_`| Suite | | `gen_` | Generator | -On load the library validates that every ID referenced by a suite (`problems`) or problem (`implementations`) exists and has the correct type. Suites also have their `fidelity_levels` auto-populated from their problems. +On load the library validates that every ID referenced by a suite (`problems`), problem (`implementations`), or any [ProblemLike](#problemlike) (`references`) exists and has the correct type. Suites also have their `fidelity_levels` auto-populated from their problems. ```yaml impl_coco: type: implementation name: COCO description: Comparing Continuous Optimisers +ref_bbob: + type: reference + title: "Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup" + link: {type: url, url: "https://numbbo.github.io/coco/"} fn_sphere: type: problem name: Sphere objectives: [1] implementations: [impl_coco] + references: [ref_bbob] suite_bbob: type: suite name: BBOB problems: [fn_sphere] + references: [ref_bbob] ``` --- @@ -112,6 +120,32 @@ impl_py_cocoex: - {type: package, url: https://pypi.org/project/coco-experiment/} ``` +### Reference + +A bibliographic pointer stored at the top level of the [Library](#library) so that a single reference can be shared by multiple problems, suites, generators, and implementations. +Requires either a `title` or a `link` and optionally a list of `authors`. + +| Field | Type | Notes | +|-----------|------------------------|------------------------------------------| +| `title` | str? | required if `link` is not given | +| `authors` | list of str? | | +| `link` | [Link](#link)? | required if `title` is not given | + +```yaml +ref_honey_badger: + type: reference + title: "Honey Badger Algorithm: New metaheuristic algorithm for solving optimization problems." + authors: + - Fatma A. Hashim + - Essam H. Houssein + - Kashif Hussain + - Mai S. Mabrouk + - Walid Al-Atabany + link: {type: doi, url: "https://doi.org/10.1016/j.matcom.2021.08.013"} +``` + +[ProblemLike](#problemlike) entities refer to a reference by its ID in the `references` field. + ### ProblemLike Fields shared by [Problem](#problem), [Suite](#suite), and [Generator](#generator). @@ -123,7 +157,7 @@ The schema deliberately puts most descriptive fields here so suites can be chara | `long_name` | str? | | | `description` | str? (markdown) | longer prose | | `tags` | set of str? | free-form keywords | -| `references` | set of [Reference](#reference)? | | +| `references` | set of IDs? | must resolve to [Reference](#reference)s | | `implementations` | set of IDs? | must resolve to [Implementation](#implementation)s | | `objectives` | set of int? | e.g. `{1}`, `{2, 3}` — **not** a ValueRange | | `variables` | set of [Variable](#variable)? | | @@ -252,23 +286,6 @@ constraints: `box | linear | function | unknown`. `function` covers non-linear/black-box constraints. -### Reference - -Bibliographic pointer. -Requires either a `title` or a `link` and optionally a list of `authors`. - -```yaml -references: - - title: "Honey Badger Algorithm: New metaheuristic algorithm for solving optimization problems." - authors: - - Fatma A. Hashim - - Essam H. Houssein - - Kashif Hussain - - Mai S. Mabrouk - - Walid Al-Atabany - link: {type: doi, url: "https://doi.org/10.1016/j.matcom.2021.08.013"] -``` - ### Link `{type?: str, url: str}`. diff --git a/examples/cobi.py b/examples/cobi.py index 2e534e4..571bba8 100644 --- a/examples/cobi.py +++ b/examples/cobi.py @@ -20,6 +20,12 @@ requirements="https://github.com/numbbo/cobi-problem-generator/blob/main/requirements.txt", ) +things["cobi_ref"] = Reference( + title="Pareto Set Characterization in Constrained Multiobjective Optimization and the COBI Problem Generator", + authors=["Anne Auger", "Dimo Brockhoff", "Luka Opravš", "Tea Tušar"], + link={"type": "arxiv", "url": "https://arxiv.org/abs/2604.09131"}, +) + things["cobi_problem"] = Generator( name="COBI Problem", description="Generator of COnstrained BI-objective optimization problems", @@ -32,13 +38,7 @@ "multi-peak", "convex-quadratic", }, - references=[ - Reference( - title="Pareto Set Characterization in Constrained Multiobjective Optimization and the COBI Problem Generator", - authors=["Anne Auger", "Dimo Brockhoff", "Luka Opravš", "Tea Tušar"], - link={"type": "arxiv", "url": "https://arxiv.org/abs/2604.09131"}, - ) - ], + references={"cobi_ref"}, objectives={2}, variables=[Variable(type="continuous", dim={"min": 1})], implementations={"cobi_impl"}, diff --git a/examples/emdo.py b/examples/emdo.py index 7a4504d..4f51f06 100644 --- a/examples/emdo.py +++ b/examples/emdo.py @@ -1,4 +1,4 @@ -from opltools import Library, Problem, Implementation +from opltools import Library, Problem, Implementation, Reference from pydantic_yaml import to_yaml_str #! - name: Electric Motor Design Optimization @@ -39,6 +39,11 @@ language="python", evaluation_time=["8 minutes"] ), + "ref_emdo": Reference( + title="A Multi-Step Evaluation Process in Electric Motor Design", + authors=["Tea Tušar", "Peter Korošec", "Bogdan Filipič"], + link={"url": "https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf"}, + ), "fn_emdo": Problem( name="Electric Motor Design Optimization", description="""# Goal @@ -67,14 +72,7 @@ noise_type=["yes"], fidelity_levels=[1], source=["real-world"], - references=[ - { - "title": "A Multi-Step Evaluation Process in Electric Motor Design", - "lang": "sj", - "authors": ["Tea Tušar", "Peter Korošec", "Bogdan Filipič"], - "link": {"url": "https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf"} - } - ], + references={"ref_emdo"}, implementations=["impl_emdo"] ) }) diff --git a/examples/problems.py b/examples/problems.py index 0a2d4ea..7011a89 100644 --- a/examples/problems.py +++ b/examples/problems.py @@ -81,6 +81,271 @@ ) +# ===================================================================== +# Shared references (reused by multiple YAML entries). +# ===================================================================== + +things["ref_coco_a_platform_for"] = Reference( + title="COCO: a platform for comparing continuous optimizers in a black-box setting", + link=Link(url="https://doi.org/10.1080/10556788.2020.1808977"), +) +things["ref_bbob_bi_objective_test_suite"] = Reference( + title="BBOB bi-objective test suite", + link=Link(url="https://doi.org/10.48550/arXiv.1604.00359"), +) +things["ref_real_parameter_black_box"] = Reference( + title="Real-parameter black-box optimization benchmarking: noisy functions definitions", + link=Link(url="https://hal.inria.fr/inria-00369466"), +) +things["ref_bbob_large_scale_test_suite"] = Reference( + title="BBOB large-scale test suite", + link=Link(url="https://doi.org/10.48550/arXiv.1903.06396"), +) +things["ref_bbob_mixed_integer_test_suite"] = Reference( + title="BBOB bi-objective mixed-integer test suite", + link=Link(url="https://doi.org/10.1145/3321707.3321868"), +) +things["ref_bbob_constrained_documentation"] = Reference( + title="bbob-constrained documentation", + link=Link(url="http://numbbo.github.io/coco-doc/bbob-constrained/"), +) +things["ref_comparison_of_multiobjective_evolutionary"] = Reference( + title="Comparison of multiobjective evolutionary algorithms: empirical results", + authors=["Eckart Zitzler", "Kalyanmoy Deb", "Lothar Thiele"], + link=Link(url="https://doi.org/10.1162/106365600568202"), +) +things["ref_scalable_multi_objective_optimization"] = Reference( + title="Scalable multi-objective optimization test problems", + authors=["Kalyanmoy Deb", "Lothar Thiele", "Marco Laumanns", "Eckart Zitzler"], + link=Link(url="https://doi.org/10.1109/CEC.2002.1007032"), +) +things["ref_a_review_of_multiobjective"] = Reference( + title="A review of multiobjective test problems and a scalable test problem toolkit", + authors=["Simon Huband", "Philip Hingston", "Luigi Barone", "Lyndon While"], + link=Link(url="https://doi.org/10.1109/TEVC.2005.861417"), +) +things["ref_cdmp_benchmark"] = Reference( + title="CDMP benchmark", + link=Link(url="https://doi.org/10.1145/3321707.3321878"), +) +things["ref_sdp_dynamic_multi_objective_benchmark"] = Reference( + title="SDP dynamic multi-objective benchmark", + link=Link(url="https://doi.org/10.1109/TCYB.2019.2896021"), +) +things["ref_maop_benchmark"] = Reference( + title="MaOP benchmark", + link=Link(url="https://doi.org/10.1016/j.swevo.2019.02.003"), +) +things["ref_bp_benchmark"] = Reference( + title="BP benchmark", + link=Link(url="https://doi.org/10.1109/CEC.2019.8790277"), +) +things["ref_gpd_generator"] = Reference( + title="GPD generator", + link=Link(url="https://doi.org/10.1016/j.asoc.2020.106139"), +) +things["ref_evolutionary_many_task_optimization"] = Reference( + title="Evolutionary many-task optimization framework", + link=Link(url="https://doi.org/10.48550/arXiv.2110.08033"), +) +things["ref_mmopp_technical_report"] = Reference( + title="MMOPP technical report", + link=Link(url="http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412"), +) +things["ref_cfd_test_problem_suite"] = Reference( + title="CFD test problem suite", + link=Link(url="https://doi.org/10.1007/978-3-319-99259-4_24"), +) +things["ref_game_benchmark_for_evolutionary"] = Reference( + title="Game benchmark for evolutionary algorithms", + link=Link(url="https://doi.org/10.1145/3321707.3321805"), +) +things["ref_car_structure_design_benchmark"] = Reference( + title="Car structure design benchmark", + link=Link(url="https://doi.org/10.1145/3205651.3205702"), +) +things["ref_bbcomp_emo_2017"] = Reference( + title="BBComp EMO 2017", + link=Link(url="https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/"), +) +things["ref_jpnsec_ec_symposium_2019_competition"] = Reference( + title="JPNSEC EC-Symposium 2019 competition", + link=Link(url="http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html"), +) +things["ref_easy_to_evaluate_real"] = Reference( + title="Easy-to-evaluate real-world multi-objective optimization problems", + authors=["Ryoji Tanabe", "Hisao Ishibuchi"], + link=Link(url="https://doi.org/10.1016/j.asoc.2020.106078"), +) +things["ref_radar_waveform_design"] = Reference( + title="Radar waveform design", + link=Link(url="https://doi.org/10.1007/978-3-540-70928-2_53"), +) +things["ref_mf2_a_collection_of"] = Reference( + title="mf2: a collection of multi-fidelity benchmark functions in Python", + link=Link(url="https://doi.org/10.21105/joss.02049"), +) +things["ref_amvop"] = Reference( + title="RWMVOP", + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281531"), +) +things["ref_sbox_cost"] = Reference( + title="SBOX-COST", + link=Link(url="https://doi.org/10.48550/arXiv.2305.12221"), +) +things["ref_on_the_design_of"] = Reference( + title="On the design of multi-objective evolutionary algorithms based on NK-landscapes", + link=Link(url="https://doi.org/10.1016/j.ejor.2012.12.019"), +) +things["ref_mubqp_benchmark"] = Reference( + title="mUBQP benchmark", + link=Link(url="https://doi.org/10.1016/j.asoc.2013.11.008"), +) +things["ref_on_the_impact_of"] = Reference( + title="On the impact of multi-objective scalability for the ρmTSP", + link=Link(url="https://doi.org/10.1007/978-3-319-45823-6_40"), +) +things["ref_benchmark_functions_for_cec"] = Reference( + title="Benchmark Functions for CEC 2015 Special Session and Competition on Dynamic Multi-objective Optimization", +) +things["ref_ealain"] = Reference( + title="Ealain", + link=Link(url="https://doi.org/10.1145/3638530.3654299"), +) +things["ref_ma_bbob"] = Reference( + title="MA-BBOB", + link=Link(url="https://doi.org/10.1145/3673908"), +) +things["ref_mpm2_technical_report_tr15_01"] = Reference( + title="MPM2 technical report TR15-01", + link=Link(url="https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf"), +) +things["ref_convex_dtlz2"] = Reference( + title="Convex DTLZ2", + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281535"), +) +things["ref_inverted_dtlz1"] = Reference( + title="Inverted DTLZ1", + link=Link(url="https://doi.org/10.1109/TEVC.2013.2281534"), +) +things["ref_minus_dtlz_minus_wfg"] = Reference( + title="Minus DTLZ / Minus WFG", + link=Link(url="https://doi.org/10.1109/TEVC.2016.2587749"), +) +things["ref_linkage_zdt_dtlz_variants"] = Reference( + title="Linkage ZDT/DTLZ variants", + link=Link(url="https://doi.org/10.1145/1143997.1144179"), +) +things["ref_cec2018_dmop_competition_tr"] = Reference( + title="CEC2018 DMOP Competition TR", + link=Link(url="https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf"), +) +things["ref_modact"] = Reference( + title="MODAct", + link=Link(url="https://doi.org/10.1109/TEVC.2020.3020046"), +) +things["ref_iohclustering"] = Reference( + title="IOHClustering", + link=Link(url="https://arxiv.org/pdf/2505.09233"), +) +things["ref_gnbg_ii"] = Reference( + title="GNBG-II", + link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3712255.3734271"), +) +things["ref_gnbg"] = Reference( + title="GNBG", + link=Link(url="https://arxiv.org/abs/2312.07083"), +) +things["ref_dynamicbinval"] = Reference( + title="DynamicBinVal", + link=Link(url="https://arxiv.org/pdf/2404.15837"), +) +things["ref_pbo_benchmarks"] = Reference( + title="PBO benchmarks", + link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3319619.3326810"), +) +things["ref_w_model"] = Reference( + title="W-model", + link=Link(url="https://dl.acm.org/doi/abs/10.1145/3205651.3208240"), +) +things["ref_submodular_optimization_benchmark"] = Reference( + title="Submodular optimization benchmark", + link=Link(url="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181"), +) +things["ref_cec2013_definitions"] = Reference( + title="CEC2013 definitions", + link=Link(url="https://peerj.com/articles/cs-2671/CEC2013.pdf"), +) +things["ref_cec2022_tr"] = Reference( + title="CEC2022 TR", + link=Link(url="https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf"), +) +things["ref_onemax_sphere_zeromax_sphere"] = Reference( + title="Mixed-variable multi-objective test problems", + link=Link(url="https://doi.org/10.1145/3449726.3459521"), +) +things["ref_porkchop_plot_interplanetary_trajectory"] = Reference( + title="Porkchop plot interplanetary trajectory benchmark", + link=Link(url="https://doi.org/10.1109/CEC65147.2025.11042973"), +) +things["ref_kinematics_of_a_robot_arm"] = Reference( + title="Kinematics of a robot arm", + link=Link(url="https://doi.org/10.1023/A:1013258808932"), +) +things["ref_vehicledynamics_benchmark"] = Reference( + title="VehicleDynamics benchmark", + link=Link(url="https://www.scitepress.org/Papers/2023/121580/121580.pdf"), +) +things["ref_mechbench"] = Reference( + title="MECHBench", + link=Link(url="https://arxiv.org/abs/2511.10821"), +) +things["ref_expobench"] = Reference( + title="EXPObench", + link=Link(url="https://doi.org/10.1016/j.asoc.2023.110744"), +) +things["ref_gasoline_direct_injection_engine_design"] = Reference( + title="Gasoline direct injection engine design", + link=Link(url="https://doi.org/10.1016/j.ejor.2022.08.032"), +) +things["ref_beacon"] = Reference( + title="BEACON", + link=Link(url="https://dl.acm.org/doi/10.1145/3712255.3734303"), +) +things["ref_tulipaenergymodel_jl_scientific_references"] = Reference( + title="TulipaEnergyModel.jl scientific references", + link=Link(url="https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references"), +) +things["ref_brachytherapy_treatment_planning"] = Reference( + title="Brachytherapy treatment planning", + link=Link(url="https://www.sciencedirect.com/science/article/pii/S1538472123016781"), +) +things["ref_fleetopt"] = Reference( + title="FleetOpt", + link=Link(url="https://dl.acm.org/doi/abs/10.1145/3638530.3664137"), +) +things["ref_building_spatial_design"] = Reference( + title="Building spatial design", + link=Link(url="https://hdl.handle.net/1887/81789"), +) +things["ref_a_multi_step_evaluation"] = Reference( + title="A Multi-Step Evaluation Process in Electric Motor Design", + authors=["Tea Tušar", "Peter Korošec", "Bogdan Filipič"], + link=Link(url="https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf"), +) +things["ref_cuter"] = Reference( + title="CUTEr", + link=Link(url="https://dl.acm.org/doi/10.1145/962437.962439"), +) +things["ref_cutest"] = Reference( + title="CUTEst", + link=Link(url="https://link.springer.com/article/10.1007/s10589-014-9687-3"), +) +things["ref_puboi"] = Reference( + title="PUBOi", + link=Link(url="https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12"), +) + # ===================================================================== # Entries # ===================================================================== @@ -104,13 +369,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, - references=[ - Reference( - title="COCO: a platform for comparing continuous optimizers in a black-box setting", - authors=[], - link=Link(url="https://doi.org/10.1080/10556788.2020.1808977"), - ) - ], + references={"ref_coco_a_platform_for"}, implementations={"impl_coco"}, ) @@ -133,13 +392,7 @@ objectives={2}, variables=[Variable(type="continuous", dim=ValueRange(min=2, max=40))], modality={"multimodal"}, - references=[ - Reference( - title="BBOB bi-objective test suite", - authors=[], - link=Link(url="https://doi.org/10.48550/arXiv.1604.00359"), - ) - ], + references={"ref_bbob_bi_objective_test_suite"}, implementations={"impl_coco"}, ) @@ -163,13 +416,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, noise_type={"noisy"}, - references=[ - Reference( - title="Real-parameter black-box optimization benchmarking: noisy functions definitions", - authors=[], - link=Link(url="https://hal.inria.fr/inria-00369466"), - ) - ], + references={"ref_real_parameter_black_box"}, implementations={"impl_coco_legacy"}, ) @@ -192,13 +439,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=20, max=640))], modality={"multimodal"}, - references=[ - Reference( - title="BBOB large-scale test suite", - authors=[], - link=Link(url="https://doi.org/10.48550/arXiv.1903.06396"), - ) - ], + references={"ref_bbob_large_scale_test_suite"}, implementations={"impl_coco"}, ) @@ -224,13 +465,7 @@ Variable(type="integer", dim=ValueRange(min=5, max=160)), ], modality={"multimodal"}, - references=[ - Reference( - title="BBOB mixed-integer test suite", - authors=[], - link=Link(url="https://doi.org/10.1145/3321707.3321868"), - ) - ], + references={"ref_bbob_mixed_integer_test_suite"}, implementations={"impl_coco"}, ) @@ -256,13 +491,7 @@ Variable(type="integer", dim=ValueRange(min=5, max=160)), ], modality={"multimodal"}, - references=[ - Reference( - title="BBOB bi-objective mixed-integer test suite", - authors=[], - link=Link(url="https://doi.org/10.1145/3321707.3321868"), - ) - ], + references={"ref_bbob_mixed_integer_test_suite"}, implementations={"impl_coco"}, ) @@ -286,13 +515,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=2, max=40))], constraints=[Constraint(hard="yes")], modality={"multimodal"}, - references=[ - Reference( - title="bbob-constrained documentation", - authors=[], - link=Link(url="http://numbbo.github.io/coco-doc/bbob-constrained/"), - ) - ], + references={"ref_bbob_constrained_documentation"}, implementations={"impl_coco"}, ) @@ -347,13 +570,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), Variable(type="binary", dim=ValueRange(min=1)), ], - references=[ - Reference( - title="Comparison of multiobjective evolutionary algorithms: empirical results", - authors=["Eckart Zitzler", "Kalyanmoy Deb", "Lothar Thiele"], - link=Link(url="https://doi.org/10.1162/106365600568202"), - ) - ], + references={"ref_comparison_of_multiobjective_evolutionary"}, implementations={"impl_pymoo"}, ) @@ -376,13 +593,7 @@ # FIXME: original "2+" - schema requires set[int]; truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Scalable multi-objective optimization test problems", - authors=["Kalyanmoy Deb", "Lothar Thiele", "Marco Laumanns", "Eckart Zitzler"], - link=Link(url="https://doi.org/10.1109/CEC.2002.1007032"), - ) - ], + references={"ref_scalable_multi_objective_optimization"}, implementations={"impl_pymoo"}, ) @@ -405,13 +616,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="A review of multiobjective test problems and a scalable test problem toolkit", - authors=["Simon Huband", "Philip Hingston", "Luigi Barone", "Lyndon While"], - link=Link(url="https://doi.org/10.1109/TEVC.2005.861417"), - ) - ], + references={"ref_a_review_of_multiobjective"}, implementations={"impl_pymoo"}, ) @@ -438,13 +643,7 @@ constraints=[Constraint(hard="yes")], dynamic_type={"unknown"}, noise_type={"unknown"}, - references=[ - Reference( - title="CDMP benchmark", - authors=[], - link=Link(url="https://doi.org/10.1145/3321707.3321878"), - ) - ], + references={"ref_cdmp_benchmark"}, ) #! - name: SDP @@ -469,13 +668,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=1))], dynamic_type={"dynamic"}, noise_type={"unknown"}, - references=[ - Reference( - title="SDP dynamic multi-objective benchmark", - authors=[], - link=Link(url="https://doi.org/10.1109/TCYB.2019.2896021"), - ) - ], + references={"ref_sdp_dynamic_multi_objective_benchmark"}, ) #! - name: MaOP @@ -499,13 +692,7 @@ objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], noise_type={"unknown"}, - references=[ - Reference( - title="MaOP benchmark", - authors=[], - link=Link(url="https://doi.org/10.1016/j.swevo.2019.02.003"), - ) - ], + references={"ref_maop_benchmark"}, ) #! - name: BP @@ -529,13 +716,7 @@ objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], noise_type={"unknown"}, - references=[ - Reference( - title="BP benchmark", - authors=[], - link=Link(url="https://doi.org/10.1109/CEC.2019.8790277"), - ) - ], + references={"ref_bp_benchmark"}, ) #! - name: GPD @@ -560,13 +741,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=1))], constraints=[Constraint(hard="some")], noise_type={"optional"}, - references=[ - Reference( - title="GPD generator", - authors=[], - link=Link(url="https://doi.org/10.1016/j.asoc.2020.106139"), - ) - ], + references={"ref_gpd_generator"}, ) #! - name: ETMOF @@ -593,13 +768,7 @@ objectives=set(range(2, 51)), variables=[Variable(type="continuous", dim=ValueRange(min=25, max=10000))], dynamic_type={"dynamic"}, - references=[ - Reference( - title="Evolutionary many-task optimization framework", - authors=[], - link=Link(url="https://doi.org/10.48550/arXiv.2110.08033"), - ) - ], + references={"ref_evolutionary_many_task_optimization"}, implementations={"impl_etmof"}, ) @@ -629,15 +798,7 @@ variables=[Variable(type="unknown")], constraints=[Constraint(hard="yes")], modality={"multimodal"}, - references=[ - Reference( - title="MMOPP technical report", - authors=[], - link=Link( - url="http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412" - ), - ) - ], + references={"ref_mmopp_technical_report"}, implementations={"impl_mmopp"}, ) @@ -669,13 +830,7 @@ variables=[Variable(type="unknown", dim=ValueRange(min=1))], constraints=[Constraint(hard="yes")], source={"real-world"}, - references=[ - Reference( - title="CFD test problem suite", - authors=[], - link=Link(url="https://doi.org/10.1007/978-3-319-99259-4_24"), - ) - ], + references={"ref_cfd_test_problem_suite"}, implementations={"impl_cfd"}, ) @@ -707,13 +862,7 @@ noise_type={"noisy"}, modality={"multimodal"}, source={"real-world"}, - references=[ - Reference( - title="Game benchmark for evolutionary algorithms", - authors=[], - link=Link(url="https://doi.org/10.1145/3321707.3321805"), - ) - ], + references={"ref_game_benchmark_for_evolutionary"}, implementations={"impl_gbea"}, ) @@ -744,13 +893,7 @@ variables=[Variable(type="integer", dim=ValueRange(min=144, max=222))], constraints=[Constraint(hard="yes", number=54)], source={"real-world"}, - references=[ - Reference( - title="Car structure design benchmark", - authors=[], - link=Link(url="https://doi.org/10.1145/3205651.3205702"), - ) - ], + references={"ref_car_structure_design_benchmark"}, implementations={"impl_car_structure"}, ) @@ -783,13 +926,7 @@ objectives={2}, variables=[Variable(type="continuous", dim=ValueRange(min=4, max=24))], source={"real-world"}, - references=[ - Reference( - title="BBComp EMO 2017", - authors=[], - link=Link(url="https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/"), - ) - ], + references={"ref_bbcomp_emo_2017"}, implementations={"impl_emo2017"}, ) @@ -825,15 +962,7 @@ variables=[Variable(type="continuous", dim=32)], constraints=[Constraint(hard="yes", number=22)], source={"real-world"}, - references=[ - Reference( - title="JPNSEC EC-Symposium 2019 competition", - authors=[], - link=Link( - url="http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html" - ), - ) - ], + references={"ref_jpnsec_ec_symposium_2019_competition"}, implementations={"impl_jsec2019"}, ) @@ -859,13 +988,7 @@ Variable(type="integer", dim=ValueRange(min=2, max=7)), ], source={"real-world-like"}, - references=[ - Reference( - title="Easy-to-evaluate real-world multi-objective optimization problems", - authors=["Ryoji Tanabe", "Hisao Ishibuchi"], - link=Link(url="https://doi.org/10.1016/j.asoc.2020.106078"), - ) - ], + references={"ref_easy_to_evaluate_real"}, implementations={"impl_reproblems"}, ) @@ -892,13 +1015,7 @@ ], constraints=[Constraint(hard="yes")], source={"real-world-like"}, - references=[ - Reference( - title="Easy-to-evaluate real-world multi-objective optimization problems", - authors=["Ryoji Tanabe", "Hisao Ishibuchi"], - link=Link(url="https://doi.org/10.1016/j.asoc.2020.106078"), - ) - ], + references={"ref_easy_to_evaluate_real"}, implementations={"impl_reproblems"}, ) @@ -927,13 +1044,7 @@ variables=[Variable(type="integer", dim=ValueRange(min=4, max=12))], constraints=[Constraint(hard="yes")], source={"real-world"}, - references=[ - Reference( - title="Radar waveform design", - authors=[], - link=Link(url="https://doi.org/10.1007/978-3-540-70928-2_53"), - ) - ], + references={"ref_radar_waveform_design"}, implementations={"impl_radar_waveform"}, ) @@ -962,13 +1073,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], fidelity_levels={1, 2}, - references=[ - Reference( - title="mf2: a collection of multi-fidelity benchmark functions in Python", - authors=[], - link=Link(url="https://doi.org/10.21105/joss.02049"), - ) - ], + references={"ref_mf2_a_collection_of"}, implementations={"impl_mf2"}, ) @@ -996,13 +1101,7 @@ Variable(type="categorical", dim=ValueRange(min=1)), ], modality={"multimodal"}, - references=[ - Reference( - title="AMVOP", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2013.2281531"), - ) - ], + references={"ref_amvop"}, ) #! - name: RWMVOP @@ -1030,13 +1129,7 @@ ], constraints=[Constraint(hard="yes")], source={"real-world"}, - references=[ - Reference( - title="RWMVOP", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2013.2281531"), - ) - ], + references={"ref_amvop"}, ) #! - name: SBOX-COST @@ -1060,13 +1153,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, - references=[ - Reference( - title="SBOX-COST", - authors=[], - link=Link(url="https://doi.org/10.48550/arXiv.2305.12221"), - ) - ], + references={"ref_sbox_cost"}, implementations={"impl_iohexperimenter"}, ) @@ -1092,13 +1179,7 @@ objectives=set(range(1, 11)), variables=[Variable(type="binary", dim=ValueRange(min=1))], modality={"multimodal"}, - references=[ - Reference( - title="On the design of multi-objective evolutionary algorithms based on NK-landscapes", - authors=[], - link=Link(url="https://doi.org/10.1016/j.ejor.2012.12.019"), - ) - ], + references={"ref_on_the_design_of"}, implementations={"impl_mocobench"}, ) @@ -1124,13 +1205,7 @@ objectives=set(range(1, 11)), variables=[Variable(type="binary", dim=ValueRange(min=1))], modality={"multimodal", "quadratic"}, - references=[ - Reference( - title="mUBQP benchmark", - authors=[], - link=Link(url="https://doi.org/10.1016/j.asoc.2013.11.008"), - ) - ], + references={"ref_mubqp_benchmark"}, implementations={"impl_mocobench"}, ) @@ -1157,13 +1232,7 @@ objectives=set(range(1, 11)), variables=[Variable(type="unknown", dim=ValueRange(min=1))], modality={"multimodal", "quadratic"}, - references=[ - Reference( - title="On the impact of multi-objective scalability for the ρmTSP", - authors=[], - link=Link(url="https://doi.org/10.1007/978-3-319-45823-6_40"), - ) - ], + references={"ref_on_the_impact_of"}, implementations={"impl_mocobench"}, ) @@ -1189,12 +1258,7 @@ variables=[Variable(type="continuous")], constraints=[Constraint(hard="?")], dynamic_type={"dynamic"}, - references=[ - Reference( - title="Benchmark Functions for CEC 2015 Special Session and Competition on Dynamic Multi-objective Optimization", - authors=[], - ) - ], + references={"ref_benchmark_functions_for_cec"}, ) #! - name: Ealain @@ -1230,13 +1294,7 @@ dynamic_type={"optional"}, fidelity_levels={1, 2}, source={"real-world-like"}, - references=[ - Reference( - title="Ealain", - authors=[], - link=Link(url="https://doi.org/10.1145/3638530.3654299"), - ) - ], + references={"ref_ealain"}, implementations={"impl_ealain"}, ) @@ -1271,13 +1329,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, source={"artificial"}, - references=[ - Reference( - title="MA-BBOB", - authors=[], - link=Link(url="https://doi.org/10.1145/3673908"), - ) - ], + references={"ref_ma_bbob"}, implementations={"impl_ma_bbob", "impl_iohexperimenter"}, ) @@ -1313,13 +1365,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, - references=[ - Reference( - title="MPM2 technical report TR15-01", - authors=[], - link=Link(url="https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf"), - ) - ], + references={"ref_mpm2_technical_report_tr15_01"}, implementations={"impl_mpm2"}, ) @@ -1344,13 +1390,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Convex DTLZ2", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2013.2281535"), - ) - ], + references={"ref_convex_dtlz2"}, ) #! - name: Inverted DTLZ1 @@ -1374,13 +1414,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Inverted DTLZ1", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2013.2281534"), - ) - ], + references={"ref_inverted_dtlz1"}, ) #! - name: Minus DTLZ @@ -1404,13 +1438,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Minus DTLZ / Minus WFG", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2016.2587749"), - ) - ], + references={"ref_minus_dtlz_minus_wfg"}, ) #! - name: Minus WFG @@ -1434,13 +1462,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Minus DTLZ / Minus WFG", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2016.2587749"), - ) - ], + references={"ref_minus_dtlz_minus_wfg"}, ) #! - name: L1-ZDT @@ -1468,13 +1490,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), Variable(type="binary", dim=ValueRange(min=1)), ], - references=[ - Reference( - title="Linkage ZDT/DTLZ variants", - authors=[], - link=Link(url="https://doi.org/10.1145/1143997.1144179"), - ) - ], + references={"ref_linkage_zdt_dtlz_variants"}, ) #! - name: L2-ZDT @@ -1501,13 +1517,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), Variable(type="binary", dim=ValueRange(min=1)), ], - references=[ - Reference( - title="Linkage ZDT/DTLZ variants", - authors=[], - link=Link(url="https://doi.org/10.1145/1143997.1144179"), - ) - ], + references={"ref_linkage_zdt_dtlz_variants"}, ) #! - name: L3-ZDT @@ -1534,13 +1544,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), Variable(type="binary", dim=ValueRange(min=1)), ], - references=[ - Reference( - title="Linkage ZDT/DTLZ variants", - authors=[], - link=Link(url="https://doi.org/10.1145/1143997.1144179"), - ) - ], + references={"ref_linkage_zdt_dtlz_variants"}, ) #! - name: L2-DTLZ @@ -1565,13 +1569,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Linkage ZDT/DTLZ variants", - authors=[], - link=Link(url="https://doi.org/10.1145/1143997.1144179"), - ) - ], + references={"ref_linkage_zdt_dtlz_variants"}, ) #! - name: L3-DTLZ @@ -1596,13 +1594,7 @@ # FIXME: original "2+" - truncated to 2..10. objectives=set(range(2, 11)), variables=[Variable(type="continuous", dim=ValueRange(min=1))], - references=[ - Reference( - title="Linkage ZDT/DTLZ variants", - authors=[], - link=Link(url="https://doi.org/10.1145/1143997.1144179"), - ) - ], + references={"ref_linkage_zdt_dtlz_variants"}, ) #! - name: CEC2018 DT - CEC2018 Competition on Dynamic Multiobjective Optimisation @@ -1630,13 +1622,7 @@ variables=[Variable(type="unknown", dim=ValueRange(min=1))], dynamic_type={"dynamic"}, source={"artificial"}, - references=[ - Reference( - title="CEC2018 DMOP Competition TR", - authors=[], - link=Link(url="https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf"), - ) - ], + references={"ref_cec2018_dmop_competition_tr"}, implementations={"impl_pymoo"}, ) @@ -1673,13 +1659,7 @@ ], constraints=[Constraint(hard="yes")], source={"real-world"}, - references=[ - Reference( - title="MODAct", - authors=[], - link=Link(url="https://doi.org/10.1109/TEVC.2020.3020046"), - ) - ], + references={"ref_modact"}, implementations={"impl_modact", "impl_pymoo"}, ) @@ -1711,13 +1691,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, source={"artificial-from-real-data"}, - references=[ - Reference( - title="IOHClustering", - authors=[], - link=Link(url="https://arxiv.org/pdf/2505.09233"), - ) - ], + references={"ref_iohclustering"}, implementations={"impl_iohclustering"}, ) things["gen_iohclustering"] = Generator( @@ -1727,13 +1701,7 @@ variables=[Variable(type="continuous", dim=ValueRange(min=1))], modality={"multimodal"}, source={"artificial-from-real-data"}, - references=[ - Reference( - title="IOHClustering", - authors=[], - link=Link(url="https://arxiv.org/pdf/2505.09233"), - ) - ], + references={"ref_iohclustering"}, implementations={"impl_iohclustering"}, ) @@ -1767,13 +1735,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="GNBG-II", - authors=[], - link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3712255.3734271"), - ) - ], + references={"ref_gnbg_ii"}, implementations={"impl_gnbg_ii", "impl_iohgnbg"}, ) things["gen_gnbg_ii"] = Generator( @@ -1782,13 +1744,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="GNBG-II", - authors=[], - link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3712255.3734271"), - ) - ], + references={"ref_gnbg_ii"}, implementations={"impl_gnbg_ii", "impl_iohgnbg"}, ) @@ -1817,13 +1773,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="GNBG", - authors=[], - link=Link(url="https://arxiv.org/abs/2312.07083"), - ) - ], + references={"ref_gnbg"}, implementations={"impl_gnbg"}, ) things["gen_gnbg"] = Generator( @@ -1832,13 +1782,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="GNBG", - authors=[], - link=Link(url="https://arxiv.org/abs/2312.07083"), - ) - ], + references={"ref_gnbg"}, implementations={"impl_gnbg"}, ) @@ -1863,13 +1807,7 @@ variables=[Variable(type="binary", dim=ValueRange(min=1))], dynamic_type={"dynamic"}, source={"artificial"}, - references=[ - Reference( - title="DynamicBinVal", - authors=[], - link=Link(url="https://arxiv.org/pdf/2404.15837"), - ) - ], + references={"ref_dynamicbinval"}, implementations={"impl_iohexperimenter"}, ) @@ -1893,13 +1831,7 @@ objectives={1}, variables=[Variable(type="binary", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="PBO benchmarks", - authors=[], - link=Link(url="https://dl.acm.org/doi/pdf/10.1145/3319619.3326810"), - ) - ], + references={"ref_pbo_benchmarks"}, implementations={"impl_iohexperimenter"}, ) @@ -1929,15 +1861,7 @@ objectives={1}, variables=[Variable(type="binary", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="W-model", - authors=[], - link=Link( - url="https://dl.acm.org/doi/abs/10.1145/3205651.3208240" - ), - ) - ], + references={"ref_w_model"}, implementations={"impl_wmodel"}, ) @@ -1962,13 +1886,7 @@ objectives={1}, variables=[Variable(type="binary", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="Submodular optimization benchmark", - authors=[], - link=Link(url="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181"), - ) - ], + references={"ref_submodular_optimization_benchmark"}, implementations={"impl_iohexperimenter"}, ) @@ -1997,13 +1915,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="CEC2013 definitions", - authors=[], - link=Link(url="https://peerj.com/articles/cs-2671/CEC2013.pdf"), - ) - ], + references={"ref_cec2013_definitions"}, implementations={"impl_cec2013", "impl_iohexperimenter"}, ) @@ -2032,13 +1944,7 @@ objectives={1}, variables=[Variable(type="continuous", dim=ValueRange(min=1))], source={"artificial"}, - references=[ - Reference( - title="CEC2022 TR", - authors=[], - link=Link(url="https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf"), - ) - ], + references={"ref_cec2022_tr"}, implementations={"impl_cec2022", "impl_iohexperimenter"}, ) @@ -2065,13 +1971,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), ], source={"artificial"}, - references=[ - Reference( - title="Onemax+Sphere / Zeromax+Sphere", - authors=[], - link=Link(url="https://doi.org/10.1145/3449726.3459521"), - ) - ], + references={"ref_onemax_sphere_zeromax_sphere"}, ) #! - name: Onemax+Sphere / DeceptiveTrap+RotatedEllipsoid @@ -2097,13 +1997,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), ], source={"artificial"}, - references=[ - Reference( - title="Mixed-variable multi-objective test problems", - authors=[], - link=Link(url="https://doi.org/10.1145/3449726.3459521"), - ) - ], + references={"ref_onemax_sphere_zeromax_sphere"}, ) #! - name: InverseDeceptiveTrap+RotatedEllipsoid / DeceptiveTrap+RotatedEllipsoid @@ -2129,13 +2023,7 @@ Variable(type="continuous", dim=ValueRange(min=1)), ], source={"artificial"}, - references=[ - Reference( - title="Mixed-variable multi-objective test problems", - authors=[], - link=Link(url="https://doi.org/10.1145/3449726.3459521"), - ) - ], + references={"ref_onemax_sphere_zeromax_sphere"}, ) #! - name: PorkchopPlotInterplanetaryTrajectory @@ -2168,13 +2056,7 @@ variables=[Variable(type="continuous", dim=2)], modality={"multimodal"}, source={"real-world"}, - references=[ - Reference( - title="Porkchop plot interplanetary trajectory benchmark", - authors=[], - link=Link(url="https://doi.org/10.1109/CEC65147.2025.11042973"), - ) - ], + references={"ref_porkchop_plot_interplanetary_trajectory"}, implementations={"impl_transfer_rf_bbob_rw"}, ) @@ -2198,13 +2080,7 @@ variables=[Variable(type="continuous", dim=21)], modality={"unimodal"}, source={"real-world"}, - references=[ - Reference( - title="Kinematics of a robot arm", - authors=[], - link=Link(url="https://doi.org/10.1023/A:1013258808932"), - ) - ], + references={"ref_kinematics_of_a_robot_arm"}, implementations={"impl_transfer_rf_bbob_rw"}, ) @@ -2233,13 +2109,7 @@ variables=[Variable(type="continuous", dim=2)], modality={"multimodal"}, source={"real-world"}, - references=[ - Reference( - title="VehicleDynamics benchmark", - authors=[], - link=Link(url="https://www.scitepress.org/Papers/2023/121580/121580.pdf"), - ) - ], + references={"ref_vehicledynamics_benchmark"}, implementations={"impl_vehicle_dynamics"}, ) @@ -2290,13 +2160,7 @@ modality={"multimodal"}, allows_partial_evaluation="no", source={"real-world"}, - references=[ - Reference( - title="MECHBench", - authors=[], - link=Link(url="https://arxiv.org/abs/2511.10821"), - ) - ], + references={"ref_mechbench"}, implementations={"impl_mechbench"}, ) @@ -2355,13 +2219,7 @@ noise_type={"observational", "real-life"}, allows_partial_evaluation="no", source={"real-world"}, - references=[ - Reference( - title="EXPObench", - authors=[], - link=Link(url="https://doi.org/10.1016/j.asoc.2023.110744"), - ) - ], + references={"ref_expobench"}, implementations={"impl_expobench"}, ) @@ -2403,13 +2261,7 @@ constraints=[Constraint(hard="yes", number=5)], fidelity_levels={1, 2}, source={"real-world"}, - references=[ - Reference( - title="Gasoline direct injection engine design", - authors=[], - link=Link(url="https://doi.org/10.1016/j.ejor.2022.08.032"), - ) - ], + references={"ref_gasoline_direct_injection_engine_design"}, implementations={"impl_gasoline"}, ) @@ -2457,13 +2309,7 @@ modality={"multimodal"}, allows_partial_evaluation="no", source={"artificial"}, - references=[ - Reference( - title="BEACON", - authors=[], - link=Link(url="https://dl.acm.org/doi/10.1145/3712255.3734303"), - ) - ], + references={"ref_beacon"}, implementations={"impl_beacon"}, ) @@ -2518,15 +2364,7 @@ modality={"unimodal"}, fidelity_levels={1, 2}, source={"real-world"}, - references=[ - Reference( - title="TulipaEnergyModel.jl scientific references", - authors=[], - link=Link( - url="https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references" - ), - ) - ], + references={"ref_tulipaenergymodel_jl_scientific_references"}, implementations={"impl_tulipa"}, ) @@ -2590,13 +2428,7 @@ fidelity_levels={1, 2}, allows_partial_evaluation="yes", source={"real-world"}, - references=[ - Reference( - title="Brachytherapy treatment planning", - authors=[], - link=Link(url="https://www.sciencedirect.com/science/article/pii/S1538472123016781"), - ) - ], + references={"ref_brachytherapy_treatment_planning"}, ) #! - name: FleetOpt @@ -2624,13 +2456,7 @@ constraints=[Constraint(hard="yes")], allows_partial_evaluation="yes", source={"real-world"}, - references=[ - Reference( - title="FleetOpt", - authors=[], - link=Link(url="https://dl.acm.org/doi/abs/10.1145/3638530.3664137"), - ) - ], + references={"ref_fleetopt"}, ) #! - name: Building spatial design @@ -2677,13 +2503,7 @@ ], allows_partial_evaluation="no", source={"real-world"}, - references=[ - Reference( - title="Building spatial design", - authors=[], - link=Link(url="https://hdl.handle.net/1887/81789"), - ) - ], + references={"ref_building_spatial_design"}, implementations={"impl_bso_toolbox"}, ) @@ -2750,13 +2570,7 @@ modality={"multimodal"}, allows_partial_evaluation="no", source={"real-world"}, - references=[ - Reference( - title="A Multi-Step Evaluation Process in Electric Motor Design", - authors=["Tea Tušar", "Peter Korošec", "Bogdan Filipič"], - link=Link(url="https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf"), - ) - ], + references={"ref_a_multi_step_evaluation"}, implementations={"impl_emdo"}, ) @@ -2876,13 +2690,7 @@ constraints=[Constraint(hard="yes")], allows_partial_evaluation="no", source={"artificial"}, - references=[ - Reference( - title="CUTEr", - authors=[], - link=Link(url="https://dl.acm.org/doi/10.1145/962437.962439"), - ) - ], + references={"ref_cuter"}, ) #! - name: CUTEst @@ -2930,13 +2738,7 @@ modality={"multimodal"}, allows_partial_evaluation="no", source={"artificial"}, - references=[ - Reference( - title="CUTEst", - authors=[], - link=Link(url="https://link.springer.com/article/10.1007/s10589-014-9687-3"), - ) - ], + references={"ref_cutest"}, implementations={"impl_pycutest"}, ) @@ -2975,13 +2777,7 @@ variables=[Variable(type="binary", dim=ValueRange(min=1))], allows_partial_evaluation="no", source={"artificial"}, - references=[ - Reference( - title="PUBOi", - authors=[], - link=Link(url="https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12"), - ) - ], + references={"ref_puboi"}, implementations={"impl_puboi"}, ) diff --git a/problems.yaml b/problems.yaml index ea3b7af..c348956 100644 --- a/problems.yaml +++ b/problems.yaml @@ -44,8 +44,8 @@ fn_building_spatial: evaluations. dynamic_type: null evaluation_time: - - 1 second - 40 seconds + - 1 second fidelity_levels: null implementations: - impl_bso_toolbox @@ -57,11 +57,7 @@ fn_building_spatial: objectives: - 2 references: - - authors: [] - link: - type: null - url: https://hdl.handle.net/1887/81789 - title: Building spatial design + - ref_building_spatial_design source: - real-world tags: null @@ -70,11 +66,11 @@ fn_building_spatial: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary fn_convex_dtlz2: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -101,11 +97,7 @@ fn_convex_dtlz2: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2013.2281535 - title: Convex DTLZ2 + - ref_convex_dtlz2 source: null tags: null type: problem @@ -153,23 +145,16 @@ fn_emdo: objectives: - 1 references: - - authors: - - Tea Tušar - - Peter Korošec - - Bogdan Filipič - link: - type: null - url: https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf - title: A Multi-Step Evaluation Process in Electric Motor Design + - ref_a_multi_step_evaluation source: - real-world tags: null type: problem variables: - - dim: 13 - type: continuous - dim: 13 type: integer + - dim: 13 + type: continuous fn_fleetopt: allows_partial_evaluation: yes can_evaluate_objectives_independently: null @@ -194,11 +179,7 @@ fn_fleetopt: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/abs/10.1145/3638530.3664137 - title: FleetOpt + - ref_fleetopt source: - real-world tags: null @@ -237,20 +218,16 @@ fn_gasoline: objectives: - 2 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1016/j.ejor.2022.08.032 - title: Gasoline direct injection engine design + - ref_gasoline_direct_injection_engine_design source: - real-world tags: null type: problem variables: - - dim: 7 - type: integer - dim: 7 type: continuous + - dim: 7 + type: integer fn_invdeceptive_deceptive_rotell: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -269,11 +246,7 @@ fn_invdeceptive_deceptive_rotell: objectives: - 2 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3449726.3459521 - title: Mixed-variable multi-objective test problems + - ref_onemax_sphere_zeromax_sphere source: - artificial tags: null @@ -282,11 +255,11 @@ fn_invdeceptive_deceptive_rotell: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary fn_inverted_dtlz1: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -313,11 +286,7 @@ fn_inverted_dtlz1: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2013.2281534 - title: Inverted DTLZ1 + - ref_inverted_dtlz1 source: null tags: null type: problem @@ -354,12 +323,7 @@ fn_jsec2019: - 4 - 5 references: - - authors: [] - link: - type: null - url: - http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html - title: JPNSEC EC-Symposium 2019 competition + - ref_jpnsec_ec_symposium_2019_competition source: - real-world tags: null @@ -385,11 +349,7 @@ fn_onemax_sphere_deceptive_rotell: objectives: - 2 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3449726.3459521 - title: Mixed-variable multi-objective test problems + - ref_onemax_sphere_zeromax_sphere source: - artificial tags: null @@ -398,11 +358,11 @@ fn_onemax_sphere_deceptive_rotell: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary fn_onemax_sphere_zeromax_sphere: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -421,11 +381,7 @@ fn_onemax_sphere_zeromax_sphere: objectives: - 2 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3449726.3459521 - title: Onemax+Sphere / Zeromax+Sphere + - ref_onemax_sphere_zeromax_sphere source: - artificial tags: null @@ -434,11 +390,11 @@ fn_onemax_sphere_zeromax_sphere: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary fn_radar_waveform: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -462,11 +418,7 @@ fn_radar_waveform: objectives: - 9 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1007/978-3-540-70928-2_53 - title: Radar waveform design + - ref_radar_waveform_design source: - real-world tags: null @@ -502,11 +454,7 @@ gen_beacon: objectives: - 2 references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/10.1145/3712255.3734303 - title: BEACON + - ref_beacon source: - artificial tags: null @@ -585,11 +533,7 @@ gen_ealain: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3638530.3654299 - title: Ealain + - ref_ealain source: - real-world-like tags: null @@ -598,15 +542,15 @@ gen_ealain: - dim: max: null min: 1 - type: integer + type: continuous - dim: max: null min: 1 - type: binary + type: integer - dim: max: null min: 1 - type: continuous + type: binary gen_gnbg: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -625,11 +569,7 @@ gen_gnbg: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://arxiv.org/abs/2312.07083 - title: GNBG + - ref_gnbg source: - artificial tags: null @@ -649,8 +589,8 @@ gen_gnbg_ii: evaluation_time: null fidelity_levels: null implementations: - - impl_gnbg_ii - impl_iohgnbg + - impl_gnbg_ii long_name: null modality: null name: GNBG-II @@ -658,11 +598,7 @@ gen_gnbg_ii: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 - title: GNBG-II + - ref_gnbg_ii source: - artificial tags: null @@ -702,11 +638,7 @@ gen_gpd: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1016/j.asoc.2020.106139 - title: GPD generator + - ref_gpd_generator source: null tags: null type: generator @@ -734,11 +666,7 @@ gen_iohclustering: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://arxiv.org/pdf/2505.09233 - title: IOHClustering + - ref_iohclustering source: - artificial-from-real-data tags: null @@ -758,8 +686,8 @@ gen_ma_bbob: evaluation_time: null fidelity_levels: null implementations: - - impl_ma_bbob - impl_iohexperimenter + - impl_ma_bbob long_name: null modality: - multimodal @@ -768,11 +696,7 @@ gen_ma_bbob: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3673908 - title: MA-BBOB + - ref_ma_bbob source: - artificial tags: null @@ -802,11 +726,7 @@ gen_mpm2: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf - title: MPM2 technical report TR15-01 + - ref_mpm2_technical_report_tr15_01 source: null tags: null type: generator @@ -845,11 +765,7 @@ gen_mubqp: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1016/j.asoc.2013.11.008 - title: mUBQP benchmark + - ref_mubqp_benchmark source: null tags: null type: generator @@ -878,11 +794,7 @@ gen_puboi: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12 - title: PUBOi + - ref_puboi source: - artificial tags: null @@ -929,15 +841,15 @@ gen_randoptgen: - dim: max: null min: 1 - type: integer + type: continuous - dim: max: null min: 1 - type: binary + type: integer - dim: max: null min: 1 - type: continuous + type: binary gen_rho_mnk_landscapes: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -967,12 +879,7 @@ gen_rho_mnk_landscapes: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1016/j.ejor.2012.12.019 - title: On the design of multi-objective evolutionary algorithms based on - NK-landscapes + - ref_on_the_design_of source: null tags: null type: generator @@ -1011,11 +918,7 @@ gen_rho_mtsp: - 9 - 10 references: - - authors: [] - link: - type: null - url: https://doi.org/10.1007/978-3-319-45823-6_40 - title: On the impact of multi-objective scalability for the ρmTSP + - ref_on_the_impact_of source: null tags: null type: generator @@ -1043,11 +946,7 @@ gen_wmodel: objectives: - 1 references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/abs/10.1145/3205651.3208240 - title: W-model + - ref_w_model source: - artificial tags: null @@ -1082,8 +981,8 @@ impl_bonobench: impl_bso_toolbox: description: Building Spatial Design toolbox (TU/e) evaluation_time: - - 1 second - 40 seconds + - 1 second language: C++ links: - type: repository @@ -1199,8 +1098,8 @@ impl_expobench: description: EXPensive Optimization benchmark library (wind farm layout, gas filter design, pipe shape, hyperparameter tuning, hospital simulation) evaluation_time: - - 80 seconds - 2 seconds + - 80 seconds language: Python links: - type: repository @@ -1221,8 +1120,8 @@ impl_gasoline: impl_gbea: description: Game-Benchmark for Evolutionary Algorithms (COCO fork) evaluation_time: - - 34 seconds - 5 seconds + - 34 seconds language: null links: - type: repository @@ -1306,8 +1205,8 @@ impl_ma_bbob: impl_mechbench: description: Structural mechanics design optimization benchmark evaluation_time: - - 1 minute - 7 minutes + - 1 minute language: Python links: - type: repository @@ -1487,6 +1386,475 @@ impl_wmodel: name: BBDOB W-Model requirements: null type: implementation +ref_a_multi_step_evaluation: + authors: + - Tea Tušar + - Peter Korošec + - Bogdan Filipič + link: + type: null + url: https://dis.ijs.si/tea/Publications/Tusar23Multistep.pdf + title: A Multi-Step Evaluation Process in Electric Motor Design + type: reference +ref_a_review_of_multiobjective: + authors: + - Simon Huband + - Philip Hingston + - Luigi Barone + - Lyndon While + link: + type: null + url: https://doi.org/10.1109/TEVC.2005.861417 + title: A review of multiobjective test problems and a scalable test problem + toolkit + type: reference +ref_amvop: + authors: null + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281531 + title: RWMVOP + type: reference +ref_bbcomp_emo_2017: + authors: null + link: + type: null + url: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/ + title: BBComp EMO 2017 + type: reference +ref_bbob_bi_objective_test_suite: + authors: null + link: + type: null + url: https://doi.org/10.48550/arXiv.1604.00359 + title: BBOB bi-objective test suite + type: reference +ref_bbob_constrained_documentation: + authors: null + link: + type: null + url: http://numbbo.github.io/coco-doc/bbob-constrained/ + title: bbob-constrained documentation + type: reference +ref_bbob_large_scale_test_suite: + authors: null + link: + type: null + url: https://doi.org/10.48550/arXiv.1903.06396 + title: BBOB large-scale test suite + type: reference +ref_bbob_mixed_integer_test_suite: + authors: null + link: + type: null + url: https://doi.org/10.1145/3321707.3321868 + title: BBOB bi-objective mixed-integer test suite + type: reference +ref_beacon: + authors: null + link: + type: null + url: https://dl.acm.org/doi/10.1145/3712255.3734303 + title: BEACON + type: reference +ref_benchmark_functions_for_cec: + authors: null + link: null + title: Benchmark Functions for CEC 2015 Special Session and Competition on + Dynamic Multi-objective Optimization + type: reference +ref_bp_benchmark: + authors: null + link: + type: null + url: https://doi.org/10.1109/CEC.2019.8790277 + title: BP benchmark + type: reference +ref_brachytherapy_treatment_planning: + authors: null + link: + type: null + url: https://www.sciencedirect.com/science/article/pii/S1538472123016781 + title: Brachytherapy treatment planning + type: reference +ref_building_spatial_design: + authors: null + link: + type: null + url: https://hdl.handle.net/1887/81789 + title: Building spatial design + type: reference +ref_car_structure_design_benchmark: + authors: null + link: + type: null + url: https://doi.org/10.1145/3205651.3205702 + title: Car structure design benchmark + type: reference +ref_cdmp_benchmark: + authors: null + link: + type: null + url: https://doi.org/10.1145/3321707.3321878 + title: CDMP benchmark + type: reference +ref_cec2013_definitions: + authors: null + link: + type: null + url: https://peerj.com/articles/cs-2671/CEC2013.pdf + title: CEC2013 definitions + type: reference +ref_cec2018_dmop_competition_tr: + authors: null + link: + type: null + url: + https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf + title: CEC2018 DMOP Competition TR + type: reference +ref_cec2022_tr: + authors: null + link: + type: null + url: https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf + title: CEC2022 TR + type: reference +ref_cfd_test_problem_suite: + authors: null + link: + type: null + url: https://doi.org/10.1007/978-3-319-99259-4_24 + title: CFD test problem suite + type: reference +ref_coco_a_platform_for: + authors: null + link: + type: null + url: https://doi.org/10.1080/10556788.2020.1808977 + title: 'COCO: a platform for comparing continuous optimizers in a black-box setting' + type: reference +ref_comparison_of_multiobjective_evolutionary: + authors: + - Eckart Zitzler + - Kalyanmoy Deb + - Lothar Thiele + link: + type: null + url: https://doi.org/10.1162/106365600568202 + title: 'Comparison of multiobjective evolutionary algorithms: empirical results' + type: reference +ref_convex_dtlz2: + authors: null + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281535 + title: Convex DTLZ2 + type: reference +ref_cuter: + authors: null + link: + type: null + url: https://dl.acm.org/doi/10.1145/962437.962439 + title: CUTEr + type: reference +ref_cutest: + authors: null + link: + type: null + url: https://link.springer.com/article/10.1007/s10589-014-9687-3 + title: CUTEst + type: reference +ref_dynamicbinval: + authors: null + link: + type: null + url: https://arxiv.org/pdf/2404.15837 + title: DynamicBinVal + type: reference +ref_ealain: + authors: null + link: + type: null + url: https://doi.org/10.1145/3638530.3654299 + title: Ealain + type: reference +ref_easy_to_evaluate_real: + authors: + - Ryoji Tanabe + - Hisao Ishibuchi + link: + type: null + url: https://doi.org/10.1016/j.asoc.2020.106078 + title: Easy-to-evaluate real-world multi-objective optimization problems + type: reference +ref_evolutionary_many_task_optimization: + authors: null + link: + type: null + url: https://doi.org/10.48550/arXiv.2110.08033 + title: Evolutionary many-task optimization framework + type: reference +ref_expobench: + authors: null + link: + type: null + url: https://doi.org/10.1016/j.asoc.2023.110744 + title: EXPObench + type: reference +ref_fleetopt: + authors: null + link: + type: null + url: https://dl.acm.org/doi/abs/10.1145/3638530.3664137 + title: FleetOpt + type: reference +ref_game_benchmark_for_evolutionary: + authors: null + link: + type: null + url: https://doi.org/10.1145/3321707.3321805 + title: Game benchmark for evolutionary algorithms + type: reference +ref_gasoline_direct_injection_engine_design: + authors: null + link: + type: null + url: https://doi.org/10.1016/j.ejor.2022.08.032 + title: Gasoline direct injection engine design + type: reference +ref_gnbg: + authors: null + link: + type: null + url: https://arxiv.org/abs/2312.07083 + title: GNBG + type: reference +ref_gnbg_ii: + authors: null + link: + type: null + url: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 + title: GNBG-II + type: reference +ref_gpd_generator: + authors: null + link: + type: null + url: https://doi.org/10.1016/j.asoc.2020.106139 + title: GPD generator + type: reference +ref_inverted_dtlz1: + authors: null + link: + type: null + url: https://doi.org/10.1109/TEVC.2013.2281534 + title: Inverted DTLZ1 + type: reference +ref_iohclustering: + authors: null + link: + type: null + url: https://arxiv.org/pdf/2505.09233 + title: IOHClustering + type: reference +ref_jpnsec_ec_symposium_2019_competition: + authors: null + link: + type: null + url: + http://www.jpnsec.org/files/competition2019/EC-Symposium-2019-Competition-English.html + title: JPNSEC EC-Symposium 2019 competition + type: reference +ref_kinematics_of_a_robot_arm: + authors: null + link: + type: null + url: https://doi.org/10.1023/A:1013258808932 + title: Kinematics of a robot arm + type: reference +ref_linkage_zdt_dtlz_variants: + authors: null + link: + type: null + url: https://doi.org/10.1145/1143997.1144179 + title: Linkage ZDT/DTLZ variants + type: reference +ref_ma_bbob: + authors: null + link: + type: null + url: https://doi.org/10.1145/3673908 + title: MA-BBOB + type: reference +ref_maop_benchmark: + authors: null + link: + type: null + url: https://doi.org/10.1016/j.swevo.2019.02.003 + title: MaOP benchmark + type: reference +ref_mechbench: + authors: null + link: + type: null + url: https://arxiv.org/abs/2511.10821 + title: MECHBench + type: reference +ref_mf2_a_collection_of: + authors: null + link: + type: null + url: https://doi.org/10.21105/joss.02049 + title: 'mf2: a collection of multi-fidelity benchmark functions in Python' + type: reference +ref_minus_dtlz_minus_wfg: + authors: null + link: + type: null + url: https://doi.org/10.1109/TEVC.2016.2587749 + title: Minus DTLZ / Minus WFG + type: reference +ref_mmopp_technical_report: + authors: null + link: + type: null + url: + http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412 + title: MMOPP technical report + type: reference +ref_modact: + authors: null + link: + type: null + url: https://doi.org/10.1109/TEVC.2020.3020046 + title: MODAct + type: reference +ref_mpm2_technical_report_tr15_01: + authors: null + link: + type: null + url: https://ls11-www.cs.tu-dortmund.de/_media/techreports/tr15-01.pdf + title: MPM2 technical report TR15-01 + type: reference +ref_mubqp_benchmark: + authors: null + link: + type: null + url: https://doi.org/10.1016/j.asoc.2013.11.008 + title: mUBQP benchmark + type: reference +ref_on_the_design_of: + authors: null + link: + type: null + url: https://doi.org/10.1016/j.ejor.2012.12.019 + title: On the design of multi-objective evolutionary algorithms based on + NK-landscapes + type: reference +ref_on_the_impact_of: + authors: null + link: + type: null + url: https://doi.org/10.1007/978-3-319-45823-6_40 + title: On the impact of multi-objective scalability for the ρmTSP + type: reference +ref_onemax_sphere_zeromax_sphere: + authors: null + link: + type: null + url: https://doi.org/10.1145/3449726.3459521 + title: Mixed-variable multi-objective test problems + type: reference +ref_pbo_benchmarks: + authors: null + link: + type: null + url: https://dl.acm.org/doi/pdf/10.1145/3319619.3326810 + title: PBO benchmarks + type: reference +ref_porkchop_plot_interplanetary_trajectory: + authors: null + link: + type: null + url: https://doi.org/10.1109/CEC65147.2025.11042973 + title: Porkchop plot interplanetary trajectory benchmark + type: reference +ref_puboi: + authors: null + link: + type: null + url: https://link.springer.com/chapter/10.1007/978-3-031-04148-8_12 + title: PUBOi + type: reference +ref_radar_waveform_design: + authors: null + link: + type: null + url: https://doi.org/10.1007/978-3-540-70928-2_53 + title: Radar waveform design + type: reference +ref_real_parameter_black_box: + authors: null + link: + type: null + url: https://hal.inria.fr/inria-00369466 + title: 'Real-parameter black-box optimization benchmarking: noisy functions definitions' + type: reference +ref_sbox_cost: + authors: null + link: + type: null + url: https://doi.org/10.48550/arXiv.2305.12221 + title: SBOX-COST + type: reference +ref_scalable_multi_objective_optimization: + authors: + - Kalyanmoy Deb + - Lothar Thiele + - Marco Laumanns + - Eckart Zitzler + link: + type: null + url: https://doi.org/10.1109/CEC.2002.1007032 + title: Scalable multi-objective optimization test problems + type: reference +ref_sdp_dynamic_multi_objective_benchmark: + authors: null + link: + type: null + url: https://doi.org/10.1109/TCYB.2019.2896021 + title: SDP dynamic multi-objective benchmark + type: reference +ref_submodular_optimization_benchmark: + authors: null + link: + type: null + url: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181 + title: Submodular optimization benchmark + type: reference +ref_tulipaenergymodel_jl_scientific_references: + authors: null + link: + type: null + url: + https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references + title: TulipaEnergyModel.jl scientific references + type: reference +ref_vehicledynamics_benchmark: + authors: null + link: + type: null + url: https://www.scitepress.org/Papers/2023/121580/121580.pdf + title: VehicleDynamics benchmark + type: reference +ref_w_model: + authors: null + link: + type: null + url: https://dl.acm.org/doi/abs/10.1145/3205651.3208240 + title: W-model + type: reference suite_amvop: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -1506,11 +1874,7 @@ suite_amvop: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2013.2281531 - title: AMVOP + - ref_amvop source: null tags: null type: suite @@ -1518,7 +1882,7 @@ suite_amvop: - dim: max: null min: 1 - type: categorical + type: continuous - dim: max: null min: 1 @@ -1526,7 +1890,7 @@ suite_amvop: - dim: max: null min: 1 - type: continuous + type: categorical suite_bbob: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -1547,11 +1911,7 @@ suite_bbob: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1080/10556788.2020.1808977 - title: 'COCO: a platform for comparing continuous optimizers in a black-box setting' + - ref_coco_a_platform_for source: null tags: null type: suite @@ -1580,11 +1940,7 @@ suite_bbob_biobj: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.48550/arXiv.1604.00359 - title: BBOB bi-objective test suite + - ref_bbob_bi_objective_test_suite source: null tags: null type: suite @@ -1613,11 +1969,7 @@ suite_bbob_biobj_mixint: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3321707.3321868 - title: BBOB bi-objective mixed-integer test suite + - ref_bbob_mixed_integer_test_suite source: null tags: null type: suite @@ -1625,11 +1977,11 @@ suite_bbob_biobj_mixint: - dim: max: 160 min: 5 - type: integer + type: continuous - dim: max: 160 min: 5 - type: continuous + type: integer suite_bbob_constrained: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -1654,11 +2006,7 @@ suite_bbob_constrained: - 1 problems: null references: - - authors: [] - link: - type: null - url: http://numbbo.github.io/coco-doc/bbob-constrained/ - title: bbob-constrained documentation + - ref_bbob_constrained_documentation source: null tags: null type: suite @@ -1687,11 +2035,7 @@ suite_bbob_largescale: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.48550/arXiv.1903.06396 - title: BBOB large-scale test suite + - ref_bbob_large_scale_test_suite source: null tags: null type: suite @@ -1720,11 +2064,7 @@ suite_bbob_mixint: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3321707.3321868 - title: BBOB mixed-integer test suite + - ref_bbob_mixed_integer_test_suite source: null tags: null type: suite @@ -1732,11 +2072,11 @@ suite_bbob_mixint: - dim: max: 160 min: 5 - type: integer + type: continuous - dim: max: 160 min: 5 - type: continuous + type: integer suite_bbob_noisy: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -1758,11 +2098,7 @@ suite_bbob_noisy: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://hal.inria.fr/inria-00369466 - title: 'Real-parameter black-box optimization benchmarking: noisy functions definitions' + - ref_real_parameter_black_box source: null tags: null type: suite @@ -1798,11 +2134,7 @@ suite_bp: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/CEC.2019.8790277 - title: BP benchmark + - ref_bp_benchmark source: null tags: null type: suite @@ -1840,11 +2172,7 @@ suite_brachytherapy: - 3 problems: null references: - - authors: [] - link: - type: null - url: https://www.sciencedirect.com/science/article/pii/S1538472123016781 - title: Brachytherapy treatment planning + - ref_brachytherapy_treatment_planning source: - real-world tags: null @@ -1877,11 +2205,7 @@ suite_car_structure: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3205651.3205702 - title: Car structure design benchmark + - ref_car_structure_design_benchmark source: - real-world tags: null @@ -1923,11 +2247,7 @@ suite_cdmp: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3321707.3321878 - title: CDMP benchmark + - ref_cdmp_benchmark source: null tags: null type: suite @@ -1946,8 +2266,8 @@ suite_cec2013: evaluation_time: null fidelity_levels: null implementations: - - impl_cec2013 - impl_iohexperimenter + - impl_cec2013 long_name: null modality: null name: CEC2013 @@ -1956,11 +2276,7 @@ suite_cec2013: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://peerj.com/articles/cs-2671/CEC2013.pdf - title: CEC2013 definitions + - ref_cec2013_definitions source: - artificial tags: null @@ -1994,10 +2310,7 @@ suite_cec2015_dmoo: - 3 problems: null references: - - authors: [] - link: null - title: Benchmark Functions for CEC 2015 Special Session and Competition on - Dynamic Multi-objective Optimization + - ref_benchmark_functions_for_cec source: null tags: null type: suite @@ -2027,12 +2340,7 @@ suite_cec2018_dt: - 3 problems: null references: - - authors: [] - link: - type: null - url: - https://www.academia.edu/download/94499025/TR-CEC2018-DMOP-Competition.pdf - title: CEC2018 DMOP Competition TR + - ref_cec2018_dmop_competition_tr source: - artificial tags: null @@ -2062,12 +2370,7 @@ suite_cec2022: - 1 problems: null references: - - authors: [] - link: - type: null - url: - https://github.com/P-N-Suganthan/2022-SO-BO/blob/main/CEC2022%20TR.pdf - title: CEC2022 TR + - ref_cec2022_tr source: - artificial tags: null @@ -2101,11 +2404,7 @@ suite_cfd: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1007/978-3-319-99259-4_24 - title: CFD test problem suite + - ref_cfd_test_problem_suite source: - real-world tags: null @@ -2141,13 +2440,7 @@ suite_cre: - 5 problems: null references: - - authors: - - Ryoji Tanabe - - Hisao Ishibuchi - link: - type: null - url: https://doi.org/10.1016/j.asoc.2020.106078 - title: Easy-to-evaluate real-world multi-objective optimization problems + - ref_easy_to_evaluate_real source: - real-world-like tags: null @@ -2183,11 +2476,7 @@ suite_cuter: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/10.1145/962437.962439 - title: CUTEr + - ref_cuter source: - artificial tags: null @@ -2196,15 +2485,15 @@ suite_cuter: - dim: max: null min: 1 - type: integer + type: continuous - dim: max: null min: 1 - type: binary + type: integer - dim: max: null min: 1 - type: continuous + type: binary suite_cutest: allows_partial_evaluation: no can_evaluate_objectives_independently: null @@ -2235,11 +2524,7 @@ suite_cutest: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://link.springer.com/article/10.1007/s10589-014-9687-3 - title: CUTEst + - ref_cutest source: - artificial tags: null @@ -2248,15 +2533,15 @@ suite_cutest: - dim: max: null min: 1 - type: integer + type: continuous - dim: max: null min: 1 - type: binary + type: integer - dim: max: null min: 1 - type: continuous + type: binary suite_dtlz: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2284,15 +2569,7 @@ suite_dtlz: - 10 problems: null references: - - authors: - - Kalyanmoy Deb - - Lothar Thiele - - Marco Laumanns - - Eckart Zitzler - link: - type: null - url: https://doi.org/10.1109/CEC.2002.1007032 - title: Scalable multi-objective optimization test problems + - ref_scalable_multi_objective_optimization source: null tags: null type: suite @@ -2321,11 +2598,7 @@ suite_dynamicbinval: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://arxiv.org/pdf/2404.15837 - title: DynamicBinVal + - ref_dynamicbinval source: - artificial tags: null @@ -2354,11 +2627,7 @@ suite_emo2017: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://www.ini.rub.de/PEOPLE/glasmtbl/projects/bbcomp/ - title: BBComp EMO 2017 + - ref_bbcomp_emo_2017 source: - real-world tags: null @@ -2436,11 +2705,7 @@ suite_etmof: - 50 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.48550/arXiv.2110.08033 - title: Evolutionary many-task optimization framework + - ref_evolutionary_many_task_optimization source: null tags: null type: suite @@ -2479,11 +2744,7 @@ suite_expobench: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1016/j.asoc.2023.110744 - title: EXPObench + - ref_expobench source: - real-world tags: null @@ -2492,7 +2753,7 @@ suite_expobench: - dim: max: 135 min: 10 - type: categorical + type: continuous - dim: max: 135 min: 10 @@ -2500,7 +2761,7 @@ suite_expobench: - dim: max: 135 min: 10 - type: continuous + type: categorical suite_gbea: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2524,11 +2785,7 @@ suite_gbea: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/3321707.3321805 - title: Game benchmark for evolutionary algorithms + - ref_game_benchmark_for_evolutionary source: - real-world tags: null @@ -2557,11 +2814,7 @@ suite_gnbg: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://arxiv.org/abs/2312.07083 - title: GNBG + - ref_gnbg source: - artificial tags: null @@ -2582,8 +2835,8 @@ suite_gnbg_ii: evaluation_time: null fidelity_levels: null implementations: - - impl_gnbg_ii - impl_iohgnbg + - impl_gnbg_ii long_name: null modality: null name: GNBG-II @@ -2592,11 +2845,7 @@ suite_gnbg_ii: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/pdf/10.1145/3712255.3734271 - title: GNBG-II + - ref_gnbg_ii source: - artificial tags: null @@ -2627,11 +2876,7 @@ suite_iohclustering: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://arxiv.org/pdf/2505.09233 - title: IOHClustering + - ref_iohclustering source: - artificial-from-real-data tags: null @@ -2661,11 +2906,7 @@ suite_kinematics_robotarm: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1023/A:1013258808932 - title: Kinematics of a robot arm + - ref_kinematics_of_a_robot_arm source: - real-world tags: null @@ -2691,11 +2932,7 @@ suite_l1_zdt: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/1143997.1144179 - title: Linkage ZDT/DTLZ variants + - ref_linkage_zdt_dtlz_variants source: null tags: null type: suite @@ -2703,11 +2940,11 @@ suite_l1_zdt: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary suite_l2_dtlz: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2734,11 +2971,7 @@ suite_l2_dtlz: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/1143997.1144179 - title: Linkage ZDT/DTLZ variants + - ref_linkage_zdt_dtlz_variants source: null tags: null type: suite @@ -2765,11 +2998,7 @@ suite_l2_zdt: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/1143997.1144179 - title: Linkage ZDT/DTLZ variants + - ref_linkage_zdt_dtlz_variants source: null tags: null type: suite @@ -2777,11 +3006,11 @@ suite_l2_zdt: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary suite_l3_dtlz: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2808,11 +3037,7 @@ suite_l3_dtlz: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/1143997.1144179 - title: Linkage ZDT/DTLZ variants + - ref_linkage_zdt_dtlz_variants source: null tags: null type: suite @@ -2839,11 +3064,7 @@ suite_l3_zdt: - 2 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1145/1143997.1144179 - title: Linkage ZDT/DTLZ variants + - ref_linkage_zdt_dtlz_variants source: null tags: null type: suite @@ -2851,11 +3072,11 @@ suite_l3_zdt: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary suite_maop: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2883,11 +3104,7 @@ suite_maop: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1016/j.swevo.2019.02.003 - title: MaOP benchmark + - ref_maop_benchmark source: null tags: null type: suite @@ -2924,11 +3141,7 @@ suite_mechbench: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://arxiv.org/abs/2511.10821 - title: MECHBench + - ref_mechbench source: - real-world tags: null @@ -2959,11 +3172,7 @@ suite_mf2: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.21105/joss.02049 - title: 'mf2: a collection of multi-fidelity benchmark functions in Python' + - ref_mf2_a_collection_of source: null tags: null type: suite @@ -2999,11 +3208,7 @@ suite_minus_dtlz: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2016.2587749 - title: Minus DTLZ / Minus WFG + - ref_minus_dtlz_minus_wfg source: null tags: null type: suite @@ -3039,11 +3244,7 @@ suite_minus_wfg: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2016.2587749 - title: Minus DTLZ / Minus WFG + - ref_minus_dtlz_minus_wfg source: null tags: null type: suite @@ -3081,12 +3282,7 @@ suite_mmopp: - 7 problems: null references: - - authors: [] - link: - type: null - url: - http://www5.zzu.edu.cn/system/_content/download.jsp?urltype=news.DownloadAttachUrl&owner=1327567121&wbfileid=4764412 - title: MMOPP technical report + - ref_mmopp_technical_report source: null tags: null type: suite @@ -3121,11 +3317,7 @@ suite_modact: - 5 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2020.3020046 - title: MODAct + - ref_modact source: - real-world tags: null @@ -3185,11 +3377,7 @@ suite_pbo: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://dl.acm.org/doi/pdf/10.1145/3319619.3326810 - title: PBO benchmarks + - ref_pbo_benchmarks source: - artificial tags: null @@ -3219,11 +3407,7 @@ suite_porkchop: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/CEC65147.2025.11042973 - title: Porkchop plot interplanetary trajectory benchmark + - ref_porkchop_plot_interplanetary_trajectory source: - real-world tags: null @@ -3257,13 +3441,7 @@ suite_re: - 9 problems: null references: - - authors: - - Ryoji Tanabe - - Hisao Ishibuchi - link: - type: null - url: https://doi.org/10.1016/j.asoc.2020.106078 - title: Easy-to-evaluate real-world multi-objective optimization problems + - ref_easy_to_evaluate_real source: - real-world-like tags: null @@ -3299,11 +3477,7 @@ suite_rwmvop: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TEVC.2013.2281531 - title: RWMVOP + - ref_amvop source: - real-world tags: null @@ -3312,7 +3486,7 @@ suite_rwmvop: - dim: max: null min: 1 - type: categorical + type: continuous - dim: max: null min: 1 @@ -3320,7 +3494,7 @@ suite_rwmvop: - dim: max: null min: 1 - type: continuous + type: categorical suite_sbox_cost: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -3342,11 +3516,7 @@ suite_sbox_cost: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.48550/arXiv.2305.12221 - title: SBOX-COST + - ref_sbox_cost source: null tags: null type: suite @@ -3383,11 +3553,7 @@ suite_sdp: - 10 problems: null references: - - authors: [] - link: - type: null - url: https://doi.org/10.1109/TCYB.2019.2896021 - title: SDP dynamic multi-objective benchmark + - ref_sdp_dynamic_multi_objective_benchmark source: null tags: null type: suite @@ -3416,11 +3582,7 @@ suite_submodular: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10254181 - title: Submodular optimization benchmark + - ref_submodular_optimization_benchmark source: - artificial tags: null @@ -3464,12 +3626,7 @@ suite_tulipa_energy: - 1 problems: null references: - - authors: [] - link: - type: null - url: - https://tulipaenergy.github.io/TulipaEnergyModel.jl/stable/40-scientific-foundation/45-scientific-references - title: TulipaEnergyModel.jl scientific references + - ref_tulipaenergymodel_jl_scientific_references source: - real-world tags: null @@ -3499,11 +3656,7 @@ suite_vehicle_dynamics: - 1 problems: null references: - - authors: [] - link: - type: null - url: https://www.scitepress.org/Papers/2023/121580/121580.pdf - title: VehicleDynamics benchmark + - ref_vehicledynamics_benchmark source: - real-world tags: null @@ -3538,16 +3691,7 @@ suite_wfg: - 10 problems: null references: - - authors: - - Simon Huband - - Philip Hingston - - Luigi Barone - - Lyndon While - link: - type: null - url: https://doi.org/10.1109/TEVC.2005.861417 - title: A review of multiobjective test problems and a scalable test problem - toolkit + - ref_a_review_of_multiobjective source: null tags: null type: suite @@ -3575,14 +3719,7 @@ suite_zdt: - 2 problems: null references: - - authors: - - Eckart Zitzler - - Kalyanmoy Deb - - Lothar Thiele - link: - type: null - url: https://doi.org/10.1162/106365600568202 - title: 'Comparison of multiobjective evolutionary algorithms: empirical results' + - ref_comparison_of_multiobjective_evolutionary source: null tags: null type: suite @@ -3590,8 +3727,8 @@ suite_zdt: - dim: max: null min: 1 - type: binary + type: continuous - dim: max: null min: 1 - type: continuous + type: binary diff --git a/src/opltools/schema.py b/src/opltools/schema.py index a2c5b5b..f4adc54 100644 --- a/src/opltools/schema.py +++ b/src/opltools/schema.py @@ -12,6 +12,7 @@ class OPLType(Enum): suite = "suite" generator = "generator" implementation = "implementation" + reference = "reference" class Link(BaseModel): @@ -73,7 +74,8 @@ def __hash__(self): return hash((self.type, self.hard, self.equality, number)) -class Reference(BaseModel): +class Reference(Thing): + type: OPLType = OPLType.reference title: str | None = None authors: list[str] | None = None link: Link | None = None @@ -108,7 +110,7 @@ class ProblemLike(Thing): long_name: str | None = None description: str | None = None tags: set[str] | None = None - references: set[Reference] | None = None + references: set[str] | None = None implementations: set[str] | None = None objectives: set[int] | None = None variables: set[Variable] | None = None @@ -142,7 +144,7 @@ class Generator(ProblemLike): class Library(RootModel): - root: dict[str, Problem | Generator | Suite | Implementation] = {} + root: dict[str, Problem | Generator | Suite | Implementation | Reference] = {} def _check_id_references(self, ids, type: OPLType) -> None: for id in ids: @@ -174,6 +176,19 @@ def _percolate_set(self, thing: Any, children: set | None, property: str): @model_validator(mode="after") def _validate(self) -> Self: + # Check reference ids on all ProblemLike things + for id, thing in self.root.items(): + if isinstance(thing, ProblemLike) and thing.references: + for ref_id in thing.references: + if ref_id not in self.root: + raise ValueError( + f"{thing.type.name} {id} references reference with undefined id '{ref_id}'." + ) + if self.root[ref_id].type != OPLType.reference: + raise ValueError( + f"{thing.type.name} {id} references reference with id '{ref_id}' but id is a {self.root[ref_id].type.name}." + ) + # First check and fixup all problems for id, thing in self.root.items(): if isinstance(thing, Problem) and thing.implementations: diff --git a/tests/test_library.py b/tests/test_library.py index 542bc7b..c525eb3 100644 --- a/tests/test_library.py +++ b/tests/test_library.py @@ -6,6 +6,7 @@ Implementation, Library, Problem, + Reference, Suite, ) @@ -81,6 +82,35 @@ def test_fixup_fidelity_all_problems_without_levels(self): }) assert lib.root["s1"].fidelity_levels == set() + def test_reference_as_top_level_thing(self): + lib = Library(root={ + "ref1": Reference(title="A paper"), + "p1": Problem(name="P1", references={"ref1"}), + }) + assert isinstance(lib.root["ref1"], Reference) + assert lib.root["p1"].references == {"ref1"} + + def test_problem_references_missing_reference(self): + with pytest.raises(ValidationError, match="undefined id"): + Library(root={ + "p1": Problem(name="P1", references={"does-not-exist"}), + }) + + def test_problem_references_non_reference(self): + with pytest.raises(ValidationError, match="but id is a"): + Library(root={ + "p1": Problem(name="P1"), + "p2": Problem(name="P2", references={"p1"}), + }) + + def test_suite_references_a_reference(self): + lib = Library(root={ + "ref1": Reference(title="A paper"), + "p1": Problem(name="P1"), + "s1": Suite(name="S1", problems={"p1"}, references={"ref1"}), + }) + assert lib.root["s1"].references == {"ref1"} + def test_fixup_evaluation_time_percolates_from_implementation_to_suite(self): lib = Library(root={ "impl1": Implementation( diff --git a/tests/test_reference.py b/tests/test_reference.py index 0d388a6..f3af998 100644 --- a/tests/test_reference.py +++ b/tests/test_reference.py @@ -1,12 +1,13 @@ import pytest from pydantic import ValidationError -from opltools.schema import Link, Reference +from opltools.schema import Link, OPLType, Reference class TestReference: def test_only_author(self): ref = Reference(title="A paper") + assert ref.type is OPLType.reference assert ref.title == "A paper" assert ref.authors is None assert ref.link is None From 58fce936dc99c20d49294388e605c94950820552 Mon Sep 17 00:00:00 2001 From: Olaf Mersmann Date: Wed, 22 Apr 2026 23:30:22 +0200 Subject: [PATCH 6/6] fix: Change unknown value of YesNoSome to "unknown" The diff for problems.yaml is larger than it needs to be because I regenerted the file using examples/problems.py and the order in which the sets are serialized is not stable. Relevant are only the changes made in the examples/problems.py, the yaml file is just the serialization of the problem definitions. Fixes #178 --- SCHEMA.md | 2 +- examples/problems.py | 4 +- problems.yaml | 138 +++++++++++++++++++------------------- src/opltools/yesnosome.py | 2 +- tests/test_yesnosome.py | 2 +- 5 files changed, 74 insertions(+), 74 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index 60bf7b1..f5d6f18 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -318,5 +318,5 @@ For example only some constraints might hard but we don't know the exact number ```yaml constraints: [{type: box, hard: some}] -allows_partial_evaluation: "?" +allows_partial_evaluation: "unknown" ``` diff --git a/examples/problems.py b/examples/problems.py index 7011a89..af82020 100644 --- a/examples/problems.py +++ b/examples/problems.py @@ -543,7 +543,7 @@ name="MOrepo", objectives={2}, variables=[Variable(type="unknown")], - constraints=[Constraint(hard="?")], + constraints=[Constraint(hard="unknown")], dynamic_type={"unknown"}, noise_type={"unknown"}, implementations={"impl_morepo"}, @@ -1256,7 +1256,7 @@ name="CEC2015-DMOO", objectives={2, 3}, variables=[Variable(type="continuous")], - constraints=[Constraint(hard="?")], + constraints=[Constraint(hard="unknown")], dynamic_type={"dynamic"}, references={"ref_benchmark_functions_for_cec"}, ) diff --git a/problems.yaml b/problems.yaml index c348956..3c291b7 100644 --- a/problems.yaml +++ b/problems.yaml @@ -66,11 +66,11 @@ fn_building_spatial: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous fn_convex_dtlz2: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -111,6 +111,10 @@ fn_emdo: can_evaluate_objectives_independently: null code_examples: null constraints: + - equality: null + hard: some + number: null + type: unknown - equality: null hard: yes number: 12 @@ -119,10 +123,6 @@ fn_emdo: hard: yes number: null type: box - - equality: null - hard: some - number: null - type: unknown description: "# Goal\nFind a design of a synchronous electric motor for power steering systems that minimizes costs and satisfies all constraints.\n\n# Motivation\n\ Challenging to find good solutions in a limited time.\n\n# Key Challenges\n* Time-consuming @@ -151,10 +151,10 @@ fn_emdo: tags: null type: problem variables: - - dim: 13 - type: integer - dim: 13 type: continuous + - dim: 13 + type: integer fn_fleetopt: allows_partial_evaluation: yes can_evaluate_objectives_independently: null @@ -224,10 +224,10 @@ fn_gasoline: tags: null type: problem variables: - - dim: 7 - type: continuous - dim: 7 type: integer + - dim: 7 + type: continuous fn_invdeceptive_deceptive_rotell: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -255,11 +255,11 @@ fn_invdeceptive_deceptive_rotell: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous fn_inverted_dtlz1: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -358,11 +358,11 @@ fn_onemax_sphere_deceptive_rotell: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous fn_onemax_sphere_zeromax_sphere: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -390,11 +390,11 @@ fn_onemax_sphere_zeromax_sphere: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous fn_radar_waveform: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -542,15 +542,15 @@ gen_ealain: - dim: max: null min: 1 - type: continuous + type: integer - dim: max: null min: 1 - type: integer + type: binary - dim: max: null min: 1 - type: binary + type: continuous gen_gnbg: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -686,8 +686,8 @@ gen_ma_bbob: evaluation_time: null fidelity_levels: null implementations: - - impl_iohexperimenter - impl_ma_bbob + - impl_iohexperimenter long_name: null modality: - multimodal @@ -749,8 +749,8 @@ gen_mubqp: - impl_mocobench long_name: null modality: - - quadratic - multimodal + - quadratic name: mUBQP noise_type: null objectives: @@ -841,15 +841,15 @@ gen_randoptgen: - dim: max: null min: 1 - type: continuous + type: integer - dim: max: null min: 1 - type: integer + type: binary - dim: max: null min: 1 - type: binary + type: continuous gen_rho_mnk_landscapes: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -902,8 +902,8 @@ gen_rho_mtsp: - impl_mocobench long_name: null modality: - - quadratic - multimodal + - quadratic name: ρmTSP noise_type: null objectives: @@ -1205,8 +1205,8 @@ impl_ma_bbob: impl_mechbench: description: Structural mechanics design optimization benchmark evaluation_time: - - 7 minutes - 1 minute + - 7 minutes language: Python links: - type: repository @@ -1882,15 +1882,15 @@ suite_amvop: - dim: max: null min: 1 - type: continuous + type: integer - dim: max: null min: 1 - type: integer + type: categorical - dim: max: null min: 1 - type: categorical + type: continuous suite_bbob: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -1977,11 +1977,11 @@ suite_bbob_biobj_mixint: - dim: max: 160 min: 5 - type: continuous + type: integer - dim: max: 160 min: 5 - type: integer + type: continuous suite_bbob_constrained: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2072,11 +2072,11 @@ suite_bbob_mixint: - dim: max: 160 min: 5 - type: continuous + type: integer - dim: max: 160 min: 5 - type: integer + type: continuous suite_bbob_noisy: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2266,8 +2266,8 @@ suite_cec2013: evaluation_time: null fidelity_levels: null implementations: - - impl_iohexperimenter - impl_cec2013 + - impl_iohexperimenter long_name: null modality: null name: CEC2013 @@ -2292,7 +2292,7 @@ suite_cec2015_dmoo: code_examples: null constraints: - equality: null - hard: '?' + hard: unknown number: null type: unknown description: null @@ -2485,30 +2485,30 @@ suite_cuter: - dim: max: null min: 1 - type: continuous + type: integer - dim: max: null min: 1 - type: integer + type: binary - dim: max: null min: 1 - type: binary + type: continuous suite_cutest: allows_partial_evaluation: no can_evaluate_objectives_independently: null code_examples: null constraints: - - equality: null - hard: yes - number: null - type: box - equality: null hard: some number: max: null min: 1 type: unknown + - equality: null + hard: yes + number: null + type: box description: CUTEst for optimization software dynamic_type: null evaluation_time: null @@ -2533,15 +2533,15 @@ suite_cutest: - dim: max: null min: 1 - type: continuous + type: integer - dim: max: null min: 1 - type: integer + type: binary - dim: max: null min: 1 - type: binary + type: continuous suite_dtlz: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2719,14 +2719,14 @@ suite_expobench: can_evaluate_objectives_independently: null code_examples: null constraints: - - equality: null - hard: yes - number: null - type: box - equality: null hard: some number: null type: unknown + - equality: null + hard: yes + number: null + type: box description: Wind farm layout optimization, gas filter design, pipe shape optimization, hyperparameter tuning, and hospital simulation dynamic_type: null @@ -2738,8 +2738,8 @@ suite_expobench: modality: null name: EXPObench noise_type: - - real-life - observational + - real-life objectives: - 1 problems: null @@ -2753,15 +2753,15 @@ suite_expobench: - dim: max: 135 min: 10 - type: continuous + type: integer - dim: max: 135 min: 10 - type: integer + type: categorical - dim: max: 135 min: 10 - type: categorical + type: continuous suite_gbea: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -2940,11 +2940,11 @@ suite_l1_zdt: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous suite_l2_dtlz: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -3006,11 +3006,11 @@ suite_l2_zdt: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous suite_l3_dtlz: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -3072,11 +3072,11 @@ suite_l3_zdt: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous suite_maop: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -3333,7 +3333,7 @@ suite_morepo: code_examples: null constraints: - equality: null - hard: '?' + hard: unknown number: null type: unknown description: null @@ -3450,11 +3450,11 @@ suite_re: - dim: max: 7 min: 2 - type: continuous + type: integer - dim: max: 7 min: 2 - type: integer + type: continuous suite_rwmvop: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -3486,15 +3486,15 @@ suite_rwmvop: - dim: max: null min: 1 - type: continuous + type: integer - dim: max: null min: 1 - type: integer + type: categorical - dim: max: null min: 1 - type: categorical + type: continuous suite_sbox_cost: allows_partial_evaluation: null can_evaluate_objectives_independently: null @@ -3598,11 +3598,11 @@ suite_tulipa_energy: code_examples: null constraints: - equality: null - hard: yes + hard: some number: null type: unknown - equality: null - hard: some + hard: yes number: null type: unknown description: Determine the optimal investment and operation decisions for @@ -3727,8 +3727,8 @@ suite_zdt: - dim: max: null min: 1 - type: continuous + type: binary - dim: max: null min: 1 - type: binary + type: continuous diff --git a/src/opltools/yesnosome.py b/src/opltools/yesnosome.py index e10b24a..722a8ea 100644 --- a/src/opltools/yesnosome.py +++ b/src/opltools/yesnosome.py @@ -5,7 +5,7 @@ class YesNoSome(Enum): yes = "yes" no = "no" some = "some" - unknown = "?" + unknown = "unknown" def union( diff --git a/tests/test_yesnosome.py b/tests/test_yesnosome.py index 7667cd4..dfb23bb 100644 --- a/tests/test_yesnosome.py +++ b/tests/test_yesnosome.py @@ -9,7 +9,7 @@ def test_from_string(self): assert YesNoSome("yes") == YesNoSome.yes assert YesNoSome("no") == YesNoSome.no assert YesNoSome("some") == YesNoSome.some - assert YesNoSome("?") == YesNoSome.unknown + assert YesNoSome("unknown") == YesNoSome.unknown def test_bad_string(self): with pytest.raises(ValueError):