Skip to content
This repository has been archived by the owner on Oct 2, 2024. It is now read-only.

Issues: mlcommons/modelgauge

63 Open 135 Closed
63 Open 135 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Add --top-p and --temp to CLI
#592 opened Sep 30, 2024 by bkorycki
New file organization for safe v1
#586 opened Sep 26, 2024 by bkorycki
Tests get newest dependency versions
#585 opened Sep 26, 2024 by bkorycki
Unite hugging face secrets
#575 opened Sep 17, 2024 by bkorycki
@rogthefrog
4
Improve separation of concerns in tests
#570 opened Sep 13, 2024 by rogthefrog
1
Automatically start/stop huggingface endpoints
#568 opened Sep 13, 2024 by bkorycki
New reference models
#567 opened Sep 13, 2024 by wpietri
Safetests use first batch of real v1.0 prompts
#564 opened Sep 12, 2024 by bkorycki
Redo modelgauge homepage to match current notions
#556 opened Sep 9, 2024 by wpietri
Ensure modelgauge works with llamaguard by default
#552 opened Sep 5, 2024 by dhosterman
Include gemma 9b in SUT list (HF)
#551 opened Sep 5, 2024 by bollacker
Ensemble annotator in v1.0 test
#547 opened Sep 4, 2024 by bkorycki
Package ensemble annotator code
#531 opened Aug 27, 2024 by rogthefrog
@rogthefrog
4
Report *accurate* num items scored in test result + add num items that *were not* scored
#526 opened Aug 22, 2024 by bkorycki
add demo evaluator to modelgauge test
#525 opened Aug 22, 2024 by wpietri
Post-0.5, pre-1.0 cleanup
#490 opened Aug 1, 2024 by wpietri
Make extra sure a model dev can add their own SUT and run a benchmark
#488 opened Jul 30, 2024 by wpietri
Make some plugins default
#486 opened Jul 29, 2024 by dhosterman
Add caching to annotator pipeline
#482 opened Jul 24, 2024 by bkorycki
Llamaguard returns is_safe logprob in annotation
#476 opened Jul 17, 2024 by bkorycki
5
CLI for prompt->sut->annotator->output
#474 opened Jul 16, 2024 by bollacker
run-prompts caches answers
#471 opened Jul 12, 2024 by wpietri
Update safetests to use modellab files
#467 opened Jul 10, 2024 by dhosterman
Evaluator runner
#464 opened Jul 8, 2024 by bkorycki
Add records for annotator prompt inputs
#463 opened Jul 2, 2024 by tsunamit
ProTip! Type g i on any issue or pull request to go back to the issue listing page.