Copilot Metrics - suggestions for next release

[mirkus90]

Select Topic Area

Product Feedback

Body

Hello! I am working with a couple of customers helping them to measure and evaluate the impact of Copilot within their organizations.
I've seen the new GA release and the additional metrics, which is great! However, I have a couple of suggestions for additional metrics to be exposed.

1. What we see missing in the metrics is a breakdown of the specific actions users can request in VSCode, such as "Explain," "Fix," "Generate Docs," and "Generate Tests."

These actions are available in the UI, and if it’s possible tracking them (of course still in anonymized and aggregated way, exactly like for the suggestions and the acceptances) could offer a more granular view of usage. This would enable a more precise assessment of Copilot’s impact - like determining whether it’s being used more for documentation, unit tests, or code explanation.

2. Another additional insight we would like to see is about the kind of use of the Copilot Chat. It could be interesting to see what the users ask into the chat, to understand on which kind of tasks Copilot helps them. To aggregate the asks and anonymize the data, instead of returning the actual questions, there could be a simple NLP system which clusters them by intents and returns the top intents.

As we are moving very quickly with GitHub Copilot adoptions for many customers, and a lot of them wants to measure the effectiveness of it, I think having those metrics simplifies the work of creating productivity dashboards a lot (we are leveraging the public dashboard accelerators), and allows the customers to measure the direct impact of GitHub Copilot on their development processes and justify the ROI (e.g. amount of docs/comments/unit tests generate with and without Copilot in each sprint, or the time the devs spend to analyze an unknown code from zero to hero, with and without GitHub Copilot).

Free to discuss about this!

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐