Add PerpetualBooster

[deadsoul44]

Add PerpetualBooster as an additional algorithm.

https://github.com/perpetual-ml/perpetual

It does not need hyperparameter tuning and supports multi-output and multi-class cases.

I can create a pull request if you are willing to review and accept.

[PGijsbers]

I think it's interesting, but I am planning to try and add a feature soon that allows having integration scripts in separate independent repositories. I propose I'll leave another message here when I have something experimental going. Perhaps it would be interesting to try out?

[deadsoul44]

It will be really helpful to benchmark our algorithm. I am waiting for it.

[PGijsbers]

You can always do local integration for yourself if you just want to use the benchmark with your framework. There is no need to have it included in this codebase for that.

[deadsoul44]

I compared PerpetualBooster against AutGluon (BQ), which is the number one framework in the benchmark, and got some promising results in local tests on small and medium tasks. I have some questions.

• All tasks are classification tasks in small, medium, large yml files. Where are regression tasks?
• I want to run the benchmark with only PerpetualBooster on AWS to compare the results against the rest of the frameworks. What is the default EC2 instance type? What is the correct command to run on AWS? I don't want to make a mistake due to costs.
• Are you willing to review and merge a pull request to include PerpetualBooster in the repo and website if the results are good enough?
• The default metrics for classification are AUC and LogLoss. But I think F1 score is a better metric because frameworks can overfit to logloss especially. Is it possible to include F1 as a default metric or as an additional metric?

P.s. I checked the repo and website before asking these. Thanks in advance.

[deadsoul44]

Answering my own first two questions after reading the paper :)
https://jmlr.org/papers/volume25/22-0493/22-0493.pdf

• www.openml.org/s/269 for regression and www.openml.org/s/271 for classification.
• 1h8c_gp3 for one hour, 4h8c_gp3 for 4 hour

Correct me if I am wrong.