From 16091f9b96e41df657f93d4043d58a77bbda3907 Mon Sep 17 00:00:00 2001 From: Pieter Gijsbers Date: Sun, 6 Oct 2024 22:26:43 +0200 Subject: [PATCH] Website/generator (#618) * Crude start substituting header and footer * Start making generator for framework gallery * Generate framework gallery * Generate desktop header without icons * Add icons * Add mobile navigation bar * Add source * Add parsing of paper information * Towards generating the frameworks page * Switch to using string Templates * Add data for first few frameworks * Add GAMA, H2O, FLAML * Fix typos * Improve typing, efficiency * Read template from main instead of inside function * Separate file loading from parsing * Fix bug that would invert mobile and desktop navigation * Add summaries and documentation links * Sort frameworks alphabetically * Add TPOT paper * Change expected location of stylesheet * Do not rebuild mkdocs on website change * Add generate static pages step --- .github/workflows/build_deploy_docs.yml | 10 +- docs/website/img/icons/book-open.svg | 4 + docs/website/img/icons/file-chart-bar.svg | 4 + docs/website/img/icons/file-lines.svg | 4 + docs/website/img/icons/github.svg | 4 + docs/website/img/icons/heart.svg | 4 + docs/website/img/icons/home.svg | 4 + docs/website/img/icons/messages.svg | 5 + docs/website/img/icons/swatchbook.svg | 4 + docs/website/scripts/generate_index.py | 138 +++++++++ docs/website/scripts/generate_navigation.py | 55 ++++ docs/website/scripts/navigation.toml | 41 +++ docs/website/scripts/official_frameworks.toml | 263 ++++++++++++++++++ docs/website/scripts/templates/footer.html | 9 + .../framework_card_paper_template.html | 23 ++ .../templates/framework_card_template.html | 51 ++++ .../templates/frameworks_template.html | 38 +++ .../scripts/templates/index_template.html | 240 ++++++++++++++++ 18 files changed, 899 insertions(+), 2 deletions(-) create mode 100644 docs/website/img/icons/book-open.svg create mode 100644 docs/website/img/icons/file-chart-bar.svg create mode 100644 docs/website/img/icons/file-lines.svg create mode 100644 docs/website/img/icons/github.svg create mode 100644 docs/website/img/icons/heart.svg create mode 100644 docs/website/img/icons/home.svg create mode 100644 docs/website/img/icons/messages.svg create mode 100644 docs/website/img/icons/swatchbook.svg create mode 100644 docs/website/scripts/generate_index.py create mode 100644 docs/website/scripts/generate_navigation.py create mode 100644 docs/website/scripts/navigation.toml create mode 100644 docs/website/scripts/official_frameworks.toml create mode 100644 docs/website/scripts/templates/footer.html create mode 100644 docs/website/scripts/templates/framework_card_paper_template.html create mode 100644 docs/website/scripts/templates/framework_card_template.html create mode 100644 docs/website/scripts/templates/frameworks_template.html create mode 100644 docs/website/scripts/templates/index_template.html diff --git a/.github/workflows/build_deploy_docs.yml b/.github/workflows/build_deploy_docs.yml index ddaedfa59..e406ed79d 100644 --- a/.github/workflows/build_deploy_docs.yml +++ b/.github/workflows/build_deploy_docs.yml @@ -23,7 +23,7 @@ concurrency: cancel-in-progress: false jobs: - # Single deploy job since we're just deploying + # TODO: Break up into separate jobs: build website, build docs, deploy deploy: environment: name: github-pages @@ -49,13 +49,19 @@ jobs: sed -i "s@WEBSITE@${WEBSITE_URL}@g" mkdocs.yml - uses: actions/setup-python@v4 with: - python-version: '3.11' + python-version: '3.12' - name: Install MkDocs run: | python -m pip install mkdocs-material - name: Build MkDocs Pages run: | mkdocs build --site-dir site/docs + - name: Build Website Pages + run: | + cd ./site/scripts + python generate_index.py + cp index.html ../ + cp frameworks.html ../ - name: Upload artifact uses: actions/upload-pages-artifact@v2 with: diff --git a/docs/website/img/icons/book-open.svg b/docs/website/img/icons/book-open.svg new file mode 100644 index 000000000..d68fb2ea9 --- /dev/null +++ b/docs/website/img/icons/book-open.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/img/icons/file-chart-bar.svg b/docs/website/img/icons/file-chart-bar.svg new file mode 100644 index 000000000..143e4a056 --- /dev/null +++ b/docs/website/img/icons/file-chart-bar.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/img/icons/file-lines.svg b/docs/website/img/icons/file-lines.svg new file mode 100644 index 000000000..fd8f942c3 --- /dev/null +++ b/docs/website/img/icons/file-lines.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/img/icons/github.svg b/docs/website/img/icons/github.svg new file mode 100644 index 000000000..e2369e8c5 --- /dev/null +++ b/docs/website/img/icons/github.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/img/icons/heart.svg b/docs/website/img/icons/heart.svg new file mode 100644 index 000000000..197a7086b --- /dev/null +++ b/docs/website/img/icons/heart.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/img/icons/home.svg b/docs/website/img/icons/home.svg new file mode 100644 index 000000000..2a4fe0672 --- /dev/null +++ b/docs/website/img/icons/home.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/img/icons/messages.svg b/docs/website/img/icons/messages.svg new file mode 100644 index 000000000..67dc316e6 --- /dev/null +++ b/docs/website/img/icons/messages.svg @@ -0,0 +1,5 @@ + + diff --git a/docs/website/img/icons/swatchbook.svg b/docs/website/img/icons/swatchbook.svg new file mode 100644 index 000000000..fb8a332b7 --- /dev/null +++ b/docs/website/img/icons/swatchbook.svg @@ -0,0 +1,4 @@ + + diff --git a/docs/website/scripts/generate_index.py b/docs/website/scripts/generate_index.py new file mode 100644 index 000000000..d1528bd41 --- /dev/null +++ b/docs/website/scripts/generate_index.py @@ -0,0 +1,138 @@ +from pathlib import Path +from string import Template + +import tomllib +from typing import NamedTuple, Sequence, Iterable + +from generate_navigation import generate_navigation + + +class Paper(NamedTuple): + title: str + abstract: str + pdf: str + venue: str + year: int + authors: str + arxiv: str | None = None + + +class Framework(NamedTuple): + name: str + repository: str + icon: str + summary: str + papers: Sequence[Paper] + documentation: str = None + + +def load_framework_definitions(definition_file: Path = Path("official_frameworks.toml")) -> list[Framework]: + with definition_file.open("rb") as fh: + frameworks = tomllib.load(fh)["frameworks"] + frameworks = parse_frameworks(frameworks) + return sorted(frameworks, key=lambda fw: fw.name.lower()) + +def parse_frameworks(framework_descriptions): + return [ + Framework( + **{attr: val for attr, val in fw.items() if attr != "papers"}, + papers=tuple(Paper(**paper) for paper in fw.get("papers", [])), + ) + for fw in framework_descriptions + ] + + +def load_navigation() -> str: + return generate_navigation() + + +def load_footer() -> str: + with open("templates/footer.html", "r") as f: + return f.read() + + +def generate_framework_gallery(frameworks: Iterable[Framework]) -> str: + template = Template(""" + + """) + framework_icon_html = [ + template.substitute(fw._asdict()) + for fw in frameworks + ] + return "\n".join(framework_icon_html) + + +def generate_main_page(frameworks: Iterable[Framework], template: Template) -> str: + header = load_navigation() + footer = load_footer() + + framework_gallery = generate_framework_gallery(frameworks) + return template.substitute( + **dict( + navigation=header, + footer=footer, + framework_gallery=framework_gallery, + ) + ) + + +def generate_framework_list( + frameworks: Sequence[Framework], + framework_card_template: Template, + framework_paper_template: Template, +) -> str: + framework_cards = [] + for framework in frameworks: + paper_list = "\n".join( + framework_paper_template.substitute(paper._asdict()) + for paper in framework.papers + ) + framework_card = framework_card_template.substitute( + framework._asdict() | {"paper_list": paper_list}, + ) + framework_cards.append(framework_card) + + return "\n".join(framework_cards) + + +def generate_framework_page(frameworks: Sequence[Framework]) -> str: + navigation = load_navigation() + footer = load_footer() + + with open("templates/frameworks_template.html", "r") as f: + main_content = Template(f.read()) + + with open("templates/framework_card_template.html", "r") as f: + framework_card_template = Template(f.read()) + + with open("templates/framework_card_paper_template.html") as f: + framework_paper_template = Template(f.read()) + + framework_cards = generate_framework_list( + frameworks, + framework_card_template, + framework_paper_template, + + ) + + main_content = main_content.substitute(**dict( + navigation=navigation, + footer=footer, + framework_cards=framework_cards, + ) + ) + + return main_content + + +if __name__ == "__main__": + main_page_template = Template(Path("templates/index_template.html").read_text()) + frameworks = load_framework_definitions() + + main_html = generate_main_page(frameworks, main_page_template) + Path("index.html").write_text(main_html) + + framework_html = generate_framework_page(frameworks) + Path("frameworks.html").write_text(framework_html) diff --git a/docs/website/scripts/generate_navigation.py b/docs/website/scripts/generate_navigation.py new file mode 100644 index 000000000..55dd2e5e3 --- /dev/null +++ b/docs/website/scripts/generate_navigation.py @@ -0,0 +1,55 @@ +from string import Template + +import tomllib +from pathlib import Path +from typing import NamedTuple, Iterable + + +class NavigationItem(NamedTuple): + name: str + url: str + icon: str + icon_only: bool = False + + +def generate_navigation_for(items: Iterable[NavigationItem], mobile: bool = False) -> str: + # TODO: Add outlink icon + item_template = Template("${name_or_icon}") + html_items = [] + for item in items: + nav_html = generate_nav_item_html(item, item_template, mobile) + html_items.append(nav_html) + + items_html = "\n".join(html_items) + classes = "mobile nav-mobile" if mobile else "desktop" + return f""" + + """ + + +def generate_nav_item_html(item, item_template, mobile: bool): + requires_icon = mobile or item.icon_only + if not requires_icon: + return item_template.substitute(url=item.url, name_or_icon=item.name) + + with open(item.icon, "r") as fh: + icon = fh.read() + return item_template.substitute(url=item.url, name_or_icon=icon) + + +def load_navigation_definitions(configuration_file: Path = Path("navigation.toml")) -> list[NavigationItem]: + with configuration_file.open("rb") as fh: + items = tomllib.load(fh)["pages"] + return [NavigationItem(**item) for item in items] + + +def generate_navigation() -> str: + items = load_navigation_definitions() + desktop_navigation = generate_navigation_for(items) + mobile_navigation = generate_navigation_for(items, mobile=True) + return desktop_navigation + mobile_navigation + + + diff --git a/docs/website/scripts/navigation.toml b/docs/website/scripts/navigation.toml new file mode 100644 index 000000000..990f31be0 --- /dev/null +++ b/docs/website/scripts/navigation.toml @@ -0,0 +1,41 @@ +[[pages]] +name = "home" # Text displayed in desktop version of navbar header +icon = "../img/icons/home.svg" # relative path to an (.svg) icon that's displayed for the mobile version +url = "index.html" # (relative) link to the page it should link to. External links will have a little icon showing it's an out-link. +icon_only = true # If set, never show the text of `name` but show the icon even in desktop mode (default: false). + +[[pages]] +name = "results" +icon = "../img/icons/file-chart-bar.svg" +url = "results.html" + +[[pages]] +name = "frameworks" +icon = "../img/icons/swatchbook.svg" +url = "frameworks.html" + +[[pages]] +name = "papers" +icon = "../img/icons/file-lines.svg" +url = "papers.html" + +[[pages]] +name = "user guide" +icon = "../img/icons/book-open.svg" +url = "user_guide.html" + +[[pages]] +name = "contributing" +icon = "../img/icons/heart.svg" +url = "contributing.html" + +[[pages]] +name = "discussion" +icon = "../img/icons/messages.svg" +url = "https://github.com/openml/automlbenchmark/discussions/216" + +[[pages]] +name = "Github" +icon = "../img/icons/github.svg" +url = "https://github.com/openml/automlbenchmark/" +icon_only = true \ No newline at end of file diff --git a/docs/website/scripts/official_frameworks.toml b/docs/website/scripts/official_frameworks.toml new file mode 100644 index 000000000..d36e4e304 --- /dev/null +++ b/docs/website/scripts/official_frameworks.toml @@ -0,0 +1,263 @@ +# Please note that order matters: https://toml.io/en/v1.0.0#array-of-tables + +[[frameworks]] +name = "GAMA" +icon = "img/logos/GAMA-icon.png" # URL or relative path +repository = "https://github.com/openml-labs/gama" +documentation="https://openml-labs.github.io/gama/master/" +summary = """ +GAMA is developed for AutoML research and features a flexible AutoML +pipeline, which makes it easy to develop and evaluate new AutoML +components. GAMA's benchmarking configuration features evolutionary +optimization and ensemble construction. +""" + +[[frameworks.papers]] +title = "GAMA: A General Automated Machine Learning Assistant" +abstract = """ +The General Automated Machine learning Assistant (GAMA) is a +modular AutoML system developed to empower users to track and +control how AutoML algorithms search for optimal machine +learning pipelines, and facilitate AutoML research itself. In +contrast to current, often black-box systems, GAMA allows users +to plug in different AutoML and post-processing techniques, logs +and visualizes the search process, and supports easy +benchmarking. It currently features three AutoML search +algorithms, two model post-processing steps, and is designed to +allow for more components to be added. +""" +year = 2021 +venue = "Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Cham, 2020." +pdf ="https://arxiv.org/pdf/2007.04911.pdf" +arxiv = "https://link.springer.com/chapter/10.1007/978-3-030-67670-4_39" +authors = "Pieter Gijsbers, Joaquin Vanschoren" + +[[frameworks]] +name = "TPOT" +icon = "img/logos/tpot.jpeg" +repository ="https://github.com/EpistasisLab/tpot" +documentation="http://epistasislab.github.io/tpot/" +summary = "TPOT is a Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming. It has a focus on optimizing models for biomedical data." + +[[frameworks.papers]] +title="Automating biomedical data science through tree-based pipeline optimization" +authors="Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A. Lavender, La Creis Kidd, and Jason H. Moore" +abstract=""" +Automated machine learning (AutoML) systems are helpful data science assistants designed to scan data for novel features, select appropriate supervised learning models and optimize their parameters. For this purpose, Tree-based Pipeline Optimization Tool (TPOT) was developed using strongly typed genetic programing (GP) to recommend an optimized analysis pipeline for the data scientist’s prediction problem. However, like other AutoML systems, TPOT may reach computational resource limits when working on big data such as whole-genome expression data. +""" +year=2016 +venue="Applications of Evolutionary Computation, pages 123-137" +pdf="https://proceedings.mlsys.org/paper/2021/file/92cc227532d17e56e07902b254dfad10-Paper.pdf" +arxiv="https://arxiv.org/pdf/1601.07925" + +[[frameworks]] +name = "AutoGluon" +repository = "https://github.com/awslabs/autogluon" +documentation = "https://auto.gluon.ai/stable/index.html" +icon = "img/logos/autogluon.png" +summary = """ +AutoGluon enables easy-to-use and easy-to-extend AutoML with a focus on automated +stack ensembling, deep learning, and real-world applications spanning image, text, +and tabular data. +""" + +[[frameworks.papers]] +title="AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data" +authors="Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, Alexander Smola" +abstract=""" +We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only +a single line of Python to train highly accurate machine learning models on an +unprocessed tabular dataset such as a CSV file. Unlike existing AutoML frameworks +that primarily focus on model/hyperparameter selection, AutoGluon-Tabular succeeds +by ensembling multiple models and stacking them in multiple layers. Experiments +reveal that our multi-layer combination of many models offers better use of +allocated training time than seeking out the best. A second contribution is an +extensive evaluation of public and commercial AutoML platforms including TPOT, H2O, +AutoWEKA, auto-sklearn, AutoGluon, and Google AutoML Tables. Tests on a suite of 50 +classification and regression tasks from Kaggle and the OpenML AutoML Benchmark +reveal that AutoGluon is faster, more robust, and much more accurate. We find that +AutoGluon often even outperforms the best-in-hindsight combination of all of its +competitors. In two popular Kaggle competitions, AutoGluon beat 99% of the +participating data scientists after merely 4h of training on the raw data. +""" +year=2020 +venue="ICML 2020 AutoML Workshop" +pdf="https://arxiv.org/pdf/2003.06505.pdf" +arxiv="https://arxiv.org/pdf/2003.06505" + +[[frameworks]] +name = "auto-sklearn" +repository = "https://github.com/automl/auto-sklearn" +documentation="https://automl.github.io/auto-sklearn/master/#" +icon = "img/logos/auto-sklearn.png" +summary = """ +Auto-sklearn is an automated machine learning toolkit and a drop-in replacement +for a scikit-learn estimator. Auto-sklearn frees a machine learning user from +algorithm selection and hyperparameter tuning. It leverages recent advantages in +Bayesian optimization, meta-learning and ensemble construction. +""" + +[[frameworks.papers]] +title="Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning" +authors="Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, Frank Hutter" +abstract=""" +Automated Machine Learning (AutoML) supports practitioners and +researchers with the tedious task of designing machine learning +pipelines and has recently achieved substantial success. In this +paper we introduce new AutoML approaches motivated by our +winning submission to the second ChaLearn AutoML challenge. We +develop PoSH Auto-sklearn, which enables AutoML systems to work +well on large datasets under rigid time limits using a new, +simple and meta-feature-free meta-learning technique and employs +a successful bandit strategy for budget allocation. However, +PoSH Auto-sklearn introduces even more ways of running AutoML +and might make it harder for users to set it up correctly. +Therefore, we also go one step further and study the design +space of AutoML itself, proposing a solution towards truly +hands-free AutoML. Together, these changes give rise to the next +generation of our AutoML system, Auto-sklearn 2.0. We verify +the improvements by these additions in a large experimental +study on 39 AutoML benchmark datasets and conclude the paper by +comparing to other popular AutoML frameworks and Auto-sklearn +1.0, reducing the relative error by up to a factor of 4.5, and +yielding a performance in 10 minutes that is substantially +better than what Auto-sklearn 1.0 achieves within an hour. +""" +year=2021 +venue="arXiv" +pdf="https://arxiv.org/pdf/2007.04074.pdf" +arxiv="https://arxiv.org/abs/2007.04074" + +[[frameworks.papers]] +title="Efficient and Robust Automated Machine Learning" +authors="Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, Frank Hutter" +year=2015 +venue="Advances in Neural Information Processing Systems 28 (NIPS 2015)" +pdf="https://papers.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf" +arxiv="https://papers.neurips.cc/paper/2015/hash/11d0e6287202fced83f79975ec59a3a6-Abstract.html" +abstract=""" +The success of machine learning in a broad range of applications +has led to an ever-growing demand for machine learning systems +that can be used off the shelf by non-experts. To be effective +in practice, such systems need to automatically choose a good +algorithm and feature preprocessing steps for a new dataset at +hand, and also set their respective hyperparameters. Recent work +has started to tackle this automated machine learning (AutoML) +problem with the help of efficient Bayesian optimization +methods. In this work we introduce a robust new AutoML system +based on scikit-learn (using 15 classifiers, 14 feature +preprocessing methods, and 4 data preprocessing methods, giving +rise to a structured hypothesis space with 110 hyperparameters). +This system, which we dub auto-sklearn, improves on existing +AutoML methods by automatically taking into account past +performance on similar datasets, and by constructing ensembles +from the models evaluated during the optimization. Our system +won the first phase of the ongoing ChaLearn AutoML challenge, +and our comprehensive analysis on over 100 diverse datasets +shows that it substantially outperforms the previous state of +the art in AutoML. We also demonstrate the performance gains due +to each of our contributions and derive insights into the +effectiveness of the individual components of auto-sklearn. +""" + +[[frameworks]] +name = "mljar-supervised" +repository = "https://github.com/mljar/mljar-supervised" +documentation = "https://mljar.com/" +icon ="img/logos/mljar.png" +summary = "The mljar-supervised is an Automated Machine Learning Python package that works with tabular data. It is designed to save time for a data scientist. It abstracts the common way to preprocess the data, construct the machine learning models, and perform hyper-parameters tuning to find the best model 🏆. It is no black-box as you can see exactly how the ML pipeline is constructed (with a detailed Markdown report for each ML model)." + +[[frameworks]] +name = "h2o" +repository = "https://github.com/h2oai/h2o-3" +documentation="https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html" +icon = "img/logos/h2o-automl-logo.jpeg" +summary = """ +H2O's AutoML can be used for automating the machine learning +workflow, which includes automatic training and tuning of many +models within a user-specified time-limit. H2O offers a number of +model explainability methods that apply to AutoML objects (groups of +models), as well as individual models (e.g. leader model). +Explanations can be generated automatically with a single function +call, providing a simple interface to exploring and explaining the +AutoML models. +""" + +[[frameworks.papers]] +title="H2O AutoML: Scalable Automatic Machine Learning" +authors="Erin LeDell and Sébastien Poirier" +abstract=""" +H2O is an open source, distributed machine learning platform +designed to scale to very large datasets, with APIs in R, +Python, Java and Scala. We present H2O AutoML, a highly +scalable, fully-automated, supervised learning algorithm which +automates the process of training a large selection of +candidate models and stacked ensembles within a single function. +The result of the AutoML run is a “leaderboard”: a ranked list +of models, all of which can be easily exported for use in a +production environment. Models in the leader- board can be +ranked by numerous model performance metrics or other model +attributes such as training time or average per-row prediction +speed. The H2O AutoML algorithm relies on the efficient training +of H2O machine learning algorithms to produce a large number +of models in a short amount of time. H2O AutoML uses a +combination of fast random search and stacked ensembles to +achieve results competitive with, and often better than, other +frameworks which rely on more complex model tuning techniques +such as Bayesian optimization or genetic algorithms. H2O AutoML +trains a variety of algorithms (e.g. GBMs, Random Forests, +Deep Neural Networks, GLMs), yielding a healthy amount of +diversity across candidate models, which can be exploited by +stacked ensembles to produce a powerful final model. The +effectiveness of this technique is reflected in the OpenML +AutoML Benchmark, which compares the performance of several of +the most well known, open source AutoML systems across a number +of datasets. +""" +year=2020 +venue="ICML 2020 AutoML Workshop" +pdf="https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf" + + +[[frameworks]] +name = "Light AutoML" +repository = "https://github.com/sb-ai-lab/LightAutoML" +documentation = "https://developers.sber.ru/portal/products/lightautoml" +icon = "img/logos/LightAutoML_logo_small.png" +summary = "LightAutoML is open-source Python library aimed at automated machine learning. It is designed to be lightweight and efficient for various tasks with tabular, text data." + +[[frameworks]] +name = "FLAML" +repository = "https://github.com/microsoft/FLAML" +documentation="https://microsoft.github.io/FLAML/" +icon = "img/logos/flaml.svg" +summary = """ +FLAML is a lightweight Python library that finds accurate machine +learning models automatically, efficiently and economically. It +frees users from selecting learners and hyperparameters for each +learner. +""" + +[[frameworks.papers]] +title="FLAML: A Fast and Lightweight AutoML Librar" +authors="Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu" +abstract=""" +We study the problem of using low computational cost to automate +the choices of learners and hyperparameters for an ad-hoc +training dataset and error metric, by conducting trials of +different configurations on the given training data. We +investigate the joint impact of multiple factors on both trial +cost and model error, and propose several design guidelines. +Following them, we build a fast and lightweight library FLAML +which optimizes for low computational resource in finding +accurate models. FLAML integrates several simple but effective +search strategies into an adaptive system. It significantly +outperforms top-ranked AutoML libraries on a large open source +AutoML benchmark under equal, or sometimes orders of magnitude +smaller budget constraints. +""" +year=2021 +venue="MLSys Conference 2021" +pdf="https://proceedings.mlsys.org/paper/2021/file/92cc227532d17e56e07902b254dfad10-Paper.pdf" +arxiv="" + diff --git a/docs/website/scripts/templates/footer.html b/docs/website/scripts/templates/footer.html new file mode 100644 index 000000000..f3f7ed867 --- /dev/null +++ b/docs/website/scripts/templates/footer.html @@ -0,0 +1,9 @@ + diff --git a/docs/website/scripts/templates/framework_card_paper_template.html b/docs/website/scripts/templates/framework_card_paper_template.html new file mode 100644 index 000000000..86d0c22a2 --- /dev/null +++ b/docs/website/scripts/templates/framework_card_paper_template.html @@ -0,0 +1,23 @@ +
+

+ ${title} +

+
+ ${authors} +
+
+ ${abstract} +
+
+
+ ${year} +
${venue}
+
+ PDF + arxiv +
+
diff --git a/docs/website/scripts/templates/framework_card_template.html b/docs/website/scripts/templates/framework_card_template.html new file mode 100644 index 000000000..43c65d5e0 --- /dev/null +++ b/docs/website/scripts/templates/framework_card_template.html @@ -0,0 +1,51 @@ +
+
+ +

${name}

+
+ + 📖 +
+
+
+ ${summary} +
+ +
+ ${paper_list} +
+ +
diff --git a/docs/website/scripts/templates/frameworks_template.html b/docs/website/scripts/templates/frameworks_template.html new file mode 100644 index 000000000..e7d783d9d --- /dev/null +++ b/docs/website/scripts/templates/frameworks_template.html @@ -0,0 +1,38 @@ + + + + AMLB: Frameworks + + + + + + + + ${navigation} +
+
+

AutoML Frameworks

+ There is more to an AutoML system than just its performance. This page + contains more information about the integrated AutoML frameworks, + including links to their papers, repositories + , and + documentation 📖. Summaries taken directly from the respective + documentation pages. Want to integrate your own framework? + Adding your own framework + is relatively simple. +
+
+ ${framework_cards} +
+
+ + ${footer} + diff --git a/docs/website/scripts/templates/index_template.html b/docs/website/scripts/templates/index_template.html new file mode 100644 index 000000000..cbe415f02 --- /dev/null +++ b/docs/website/scripts/templates/index_template.html @@ -0,0 +1,240 @@ + + + + AMLB: an AutoML Benchmark + + + + + + + + ${navigation} +
+
+
AMLB
+
An AutoML Benchmark
+
+ Comparing different AutoML frameworks is notoriously challenging. AMLB + is an open and extensible benchmark that follows best practices and + avoids common mistakes when comparing AutoML frameworks. +
+
+
+
+
+

Easy to Use

+
+ You can run an entire benchmark with a single command! The AutoML + benchmark tool automates the installation of the AutoML framework, + the experimental setup, and executing the experiment. +
+
+ > python runbenchmark.py autosklearn openml/s/269 1h8c +
+
+ + + + + Installation guide + +
+
+

Visualize Results

+
+ The results can be visualized with our + interactive visualization tool + or one of our + notebooks. This includes critical difference diagrams, + scaled performance plots, and more! +
+ +
+ + + + + Results + +
+
+

Easy to Extend

+
+ Adding a framework + and + adding a dataset + is easy. These extensions can be kept completely private, or + shared with the community. For datasets, it is even possible to + work with + OpenML + tasks and suites directly! +
+ + + + + + + + +
+ + + + + Extending the benchmark + +
+
+
+
+
+
+

📄 Paper

+
+ Our + JMLR paper introduces the benchmark. It includes an in-depth discussion of the different design + decisions and its limitations, as well as a multi-faceted analysis + of results from large scale comparison across 9 frameworks on more + than 100 tasks conducted in 2023. +
+
+
+

🧑‍💻 Code

+
+ The entire benchmark tool is open source and developed on + Github. The Github discussion board and issue trackers are the main way + for us to interact with the community. +
+
+
+ +
+
+

AutoML Frameworks

+
+
+ Many AutoML frameworks are already integrated with the AutoML + benchmark tool and + adding more is easy. + We have more information about the different frameworks on our + framework overview page. The icons below + link directly to their respective Github repositories. +
+
+ ${framework_gallery} +
+
+
+
+
+

Community

+
+ We welcome any contributions to the AutoML benchmark. Our goal is to + provide the best benchmark tools for AutoML research and we can't do + that on our own. Contributions are appreciated in many forms, + including feedback on the benchmark design, feature requests, bug + reports, code and documentation contributions, and more. Why not stop + by on our + welcome board + and let us know what got you interested in the benchmark? +
+
+
+ + ${footer} +