Expert Python programmers saw the most benefit
From mid-June, 2023 through the end of July that year, the Melbourne-based ANZ Bank conducted an internal trial of GitHub Copilot that involved 100 of the firm's 5,000 engineers.
ANZ Bank sought to conduct its own evaluation, citing the potential benefit of AI on productivity while also acknowledging that the technology"raises inherent risks, uncertainties and unintentional consequences regarding intellectual property, data security and privacy.""Prior to starting the experiment, risks related to intellectual property, data security and privacy were assessed in conjunction with ANZ’s legal and security teams to arrive at a set of guidelines," it said.
Both of these results were deemed statistically significant. As for security, the experiment was inconclusive. "They felt it helped them review and understand existing code, create documentation, and test their code; they felt it allowed them to spend less time debugging their code and reduced their overall development time; and they felt the suggestions it provided were somewhat helpful, and aligned well with their project's coding standards," the report says.