Mozilla to use machine learning to find code bugs before they ship

In a bid to cut the number of coding errors made in its Firefox browser, Mozilla is deploying Clever-Commit, a machine-learning-driven coding assistant developed in conjunction with game developer Ubisoft.

Clever-Commit analyzes code changes as developers commit them to the Firefox codebase. It compares them to all the code it has seen before to see if they look similar to code that the system knows to be buggy.

If the assistant thinks that a commit looks suspicious, it warns the developer. Presuming its analysis is correct, it means that the bug can be fixed before it gets committed into the source repository. Clever-Commit can even suggest fixes for the bugs that it finds. Initially, Mozilla plans to use Clever-Commit during code reviews, and in time this will expand to other phases of development, too. It works with all three of the languages that Mozilla uses for Firefox: C++, JavaScript, and Rust.

The tool builds on work by Ubisoft La Forge, Ubisoft’s research lab. Last year, Ubisoft presented the Commit-Assistant, based on research called CLEVER, a system for finding bugs and suggesting fixes. That system found some 60-70 percent of buggy commits, though it also had a false positive rate of 30 percent. Even though this false positive rate is quite high, users of this system nonetheless felt that it was worthwhile, thanks to the time saved when it did correctly identify a bug.

Mozilla is anticipating a similar level of success once Clever-Commit is fully integrated into its development processes, with three to four bugs out of five being caught before they’re committed. This is much cheaper and easier than catching them after they’ve gone into production.

This use of machine learning to make inferences from a large codebase is not exclusive to Mozilla and Ubisoft. Last year, Microsoft introduced a feature called IntelliCode in Visual Studio. IntelliCode has examined some of the most popular public GitHub repositories (more than 2,000 projects each with more than 100 stars) to figure out best coding practices. The initial uses of IntelliCode are conservative; it provides smarter IntelliSense completion suggestions by knowing which suggestions are more likely to be used, based on how other projects use the same APIs. It’s also being used to detect coding styles and whitespace usage so that it can format code in a manner consistent with the rest of a project.

IntelliCode, like Clever-Commit, can be used to find bugs. As an example, Microsoft has shown it detecting improperly used variables, in which a developer subtracted an coordinate from a  coordinate when they actually intended to find the difference between two coordinates.