How is TorchBench applied to testing new versions of PyTorch?

[HLH13297997663]

Hello, may I ask what tasks will be used for end-to-end testing before the release of the new version of PyTorch?
Will the test focus on the consistency of metrics between the previous and subsequent versions, such as the loss of training tasks, iteration speed, etc

[FindHao]

You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).

you may check this paper for more details. https://arxiv.org/abs/2304.14226

[HLH13297997663]

You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).

you may check this paper for more details. https://arxiv.org/abs/2304.14226

非常感谢您的回复，还想问一下，当pytorch的训练功能有组件更新或者添加了新功能，是如何进行回归测试的？

[FindHao]

You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).
you may check this paper for more details. https://arxiv.org/abs/2304.14226

非常感谢您的回复，还想问一下，当pytorch的训练功能有组件更新或者添加了新功能，是如何进行回归测试的？

If the new features are enabled by default and used in models included in torchbench, the ci will do the testing naturally. Since we use the nightly pytorch, if there are some updates on already applied testings, they will be tested automatically.

[FindHao]

You can install nightly pytorch for testing, but the default benchmark only focus on sinlge iteration's computensive part(forward, backward, and optimizer phases).
you may check this paper for more details. https://arxiv.org/abs/2304.14226

非常感谢您的回复，还想问一下，当pytorch的训练功能有组件更新或者添加了新功能，是如何进行回归测试的？

If the new features are enabled by default and used in models included in torchbench, the ci will do the testing naturally. Since we use the nightly pytorch, if there are some updates on already applied testings, they will be tested automatically.

For other new features, the PR authors are usually required to write unit tests in pytorch repo.