GPT-4 is expected to massively improve productivity and job satisfaction for engineers and scientists, but access is limited and involves risk. GitHub Copilot offers a route to GPT-4 access that meets a higher infosec bar. This article makes a business justification for GitHub Copilot, then attempts to subsequently argue for Copilot X from the justification of Copilot. Copilot X adoption, in turn, can assist in driving GPT-4 adoption across job families and tools.
GitHub’s research shows that developers wrote code 55% faster with Copilot and “60–75% of users reported they feel more fulfilled with their job”
Some developer factors influence benefits. Here (p6) is a factor analysis.
Outside of GitHub, one startup VP says it helped him save 10% of the time he'd spend coding, and “He now requires his team of three engineers to use the tool because he quickly found it saved him plenty of time by eliminating rote work.”
Copilot is built on Codex from OpenAI. According to OpenAI, “OpenAI Codex is most capable in Python”
Justifications for Copilot X given Copilot approval:
Continued ROI justification. It costs nothing additional and has substantial upside with an estimated additional 55% completion speed over the legacy version of Copilot, plus additional employee satisfaction.
Copilot X has entirely new capabilities, detailed here.
Codex achieves a “pass@1” 33.5% of the time, but GPT-4 achieves this 67% of the time for Python coding tasks in the HumanEval standard benchmark. This is virtually an exact doubling, thus the above point estimate. This “pass@1” metric is directly tied to developers acceptance of tool suggestions, and other research tells us that “the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers’ perception of productivity.”
Copilot X is just a new version of the tool. Getting an early look at new versions is generally good practice for any tool or library.
Other related articles:
[Source] “ChatGPT substantially raises average productivity: time taken decreases by 0.8 SDs and output quality rises by 0.4 SDs. Inequality between workers decreases...Exposure to ChatGPT increases job satisfaction” for “mid-level professional writing tasks”