Automated Smoke Test #368
Automated Smoke Test #368
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
LGTM. Do we care about testing with other python versions? |
That's a great question! I think the idea here is to make sure we can communicate with the SUTs correctly, not to comprehensively test the operation of ModelBench, so I believe we probably don't care. But I'm open to being wrong! @wpietri , thoughts? |
|
Definitely a good question. Yes my intent here was about 10% smoke test for our CLI and 90% making sure that there are no external surprises. Given our other test coverage, I think we're good on Python version shenanigans, so let's try it without. But if reality surprises us, we should look at the question again. |
|
Makes sense to me! |
Theoretically, this runs a tiny benchmark every day at 12:00 UTC/8:00 EST. I say theoretically because there's apparently no way to test it if it's not on the main branch. I've tested the benchmark run itself and it works, but the scheduler is anybody's guess.