For the first time, artificial intelligence has achieved what a human programmer does—develop correct algorithms and write the corresponding computer code for new complex tasks. Reported in the journal Science, the artificial intelligence system AlphaCode from DeepMind has reached a performance level allowing it to surpass half of the participants in an online programming contest.
The Presence of AI in Various Fields
Whether it’s a language assistant in mobile apps, a scientific aide, or a support system in vehicles or industrial facilities, adaptive algorithms and AI systems are already present in many areas of our lives. Meanwhile, the machine brain is mastering strategic games and data analysis, as well as tasks long considered beyond its capabilities: AI is bluffing in poker, assessing opponents’ reactions in negotiations, and developing new mathematical theorems.
AI and Programming
How realistic are science fiction scenarios where artificial intelligence itself becomes the creator of new algorithms and AI systems? Until now, systems could generate small code fragments or simple programs based on existing schemes and rules. “But to write a complete program, you must understand the task, develop the correct solution algorithm, and then write the code to implement this algorithm,” explained Yujia Li and his colleagues at DeepMind, a subsidiary of Google. If the task is presented not in a computer-representative form but in the form of complex natural language texts, it is even more challenging for the machine. Such tasks are set, for example, at programming competitions, where human programmers must convert them into computer code. However, until now, AI systems based on neural networks could not solve them.
That was the case until now—the DeepMind team has for the first time developed an artificial intelligence system capable of programming well enough to participate in coding competitions. “As far as we know, no computer system has ever achieved such performance in programming competitions,” say Li and his team.
Their system, AlphaCode, correctly solved an average of 34% of the competitive tasks on the Codeforces platform, thus demonstrating better results than half of the human participants.
Interestingly, AlphaCode achieved this not thanks to pre-set programming rules. The artificial intelligence system developed its capabilities solely as a result of training on competitive tasks and their solutions from previous years. These data are usually published by programming platforms, thus they are available for participants’ preparation. The AI system learned to program using only these datasets.
AlphaCode’s Learning and Performance
For its training, AlphaCode analyzed 715 gigabytes of code fragments in various programming languages from the GitHub platform. This unstructured collection of scripts did not contain problem descriptions, so it was suitable only for familiarizing with typical structures and syntactic elements. Then, the encoder module of the AlphaCode Transformer received half of the corresponding script, and the decoder had to learn to correctly complete the other half.
In the next phase of training, the pre-trained AI system received about 13,500 old competition tasks from the Codeforces platform, both correct and incorrect solutions. By evaluating this data, AlphaCode “learned” which features distinguish correct programming solutions from incorrect ones.
After completing this stage, the evaluation—competition—took place. For this, AlphaCode was given tasks formulated in December 2021 on the Codeforces platform for ten competitions with more than 5,000 participants. The AI system initially created millions of potential code solutions for each task. Then, a sub-module checked the viability of these versions and selected ten potentially suitable solutions for each task.
Assessment and Results
The Li team then uploaded these code solutions to the Codeforces platform, where they were evaluated. Based on these evaluations, the researchers determined where AlphaCode would have ranked among the contestant participants.
The result: the code solutions by AlphaCode turned out to be better than those of 54.3% of human participants. “This performance is equivalent to that of a novice programmer who has been training for a year,” explained Li and his colleagues. Like the human participants, AlphaCode more often correctly solved easier tasks than complex ones.
This is the first instance where artificial intelligence was able to compete with human programmers in an open programming contest. As a result, AlphaCode made it to the cover of Science. The DeepMind team considers their AI system a promising approach for future programmers: “The relatively simple architecture demonstrates unique potential to perform the analysis necessary to solve complex problems using code,” say Li and his colleagues.
Expert Opinion
Artificial intelligence researcher Zico Kolter, from a university not involved in the study, shares a similar view: “Regardless of how much AlphaCode ‘understands’ what it is doing, the system demonstrates extraordinarily good results when it comes to solving complex programming tasks,” wrote the scientist in an accompanying commentary.
Kolter believes AlphaCode is the first example of a model capable of learning to code without knowing the rules. However, the process is still very complex: AlphaCode required 175 megawatt-hours of energy for training and analysis, as well as computational power equivalent to 2,149 days of continuous operation at one petaflop per second. “Thus, our work on code generation systems still needs improvement,” acknowledge Li and his team.
