GitHub has released a report on the performance of programmers when using GitHub Copilot, revealing that overall, utilizing Copilot results in more accurate and higher quality code, with fewer bugs.
In this study, programmers with at least 5 years of experience were divided into two random groups: one group of 104 individuals used Copilot, while the other group of 98 individuals did not. They were tasked with writing a Web API based on instructions, followed by code correctness assessments through test suite runs and code quality evaluations.
The rate of code passing the test suite without using Copilot was only 39.2%, whereas the Copilot group achieved a pass rate of 60.8%, indicating that Copilot significantly improves the ability for code to adhere to specifications.
Subsequently, a review was conducted on the code of 25 programmers who wrote code that passed all tests, randomly selected from all individuals. It was found that the Copilot group had fewer errors, with code that is easier to read, maintain, and understand. Despite the minor differences compared to working without passing the test suite, the Copilot group’s higher scores are of significant importance. The overall higher code quality led to a higher rate of code acceptance during patch submissions, with the Copilot group achieving 29.9%, as opposed to the non-Copilot group at 24.5%.
This controlled study clearly demonstrates that using Copilot enhances the overall quality of code.
Source: GitHub Blog
TLDR: The utilization of GitHub Copilot by programmers with at least 5 years of experience resulted in more accurate and high-quality code, with fewer errors, showcasing its ability to elevate code standards significantly.
Leave a Comment