The fire between the Linux kernel community and the University of Minnesota (UMN) is being put out. Thanks to an ill-thought-out Linux security project, two UMN graduate students tried to insert deliberately buggy patches into Linux. Greg Kroah-Hartman, the well-respected Linux kernel maintainer for the Linux stable branch, responded by banning not only them but any UMN-connected developers from contributing to the Linux kernel. Now, UMN has addressed the Linux kernel developer’s community’s concerns. And, in a message to the Linux Kernel Mailing List (LKML), the Linux Foundation Technical Advisory Board (TAB) and volunteer senior Linux kernel maintainers and developers have reported on what they found when they closely and thoroughly examined patches from UMN academics.
First things first: 435 commits coming from UMN-associated developers were re-reviewed. “The huge majority of the reviewed commits were found to be correct.” Of the rest, 39 commits were incorrect and in need of fixing; 25 had already been fixed by later commits; 12 no longer mattered; 9 had been made before the guilty research group existed and one commit has been removed by its author’s request.
Five deliberately corrupt changes had been submitted to the LKML. “These changes were submitted using two fake identities, which is against the documented requirements for how to contribute code to the Linux kernel. The University appears to have allowed researchers to use fake identities when agreeing to the ‘Developers Certificate of Origin,’ a legal statement about the work being submitted.”
However, unlike what was claimed by the researchers, Qiushi Wu and Aditya Pakki, and their graduate advisor, Kangjie Lu, an assistant professor in the UMN Computer Science & Engineering Department’s paper, “On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits,” aka “Hypocrite Commits” the TAB reported in clear detail that “All patch submissions that were invalid were caught, or ignored, by the Linux kernel developers and maintainers. Our patch-review processes worked as intended when confronted with these malicious patches.”
Still, although no new attacks had been found, the kernel developers felt this enormous review had to be done. As Kroah-Hartman told me, “We were required to be thorough.” That’s because there was also the possibility, no matter how small, that deliberately corrupt code had been placed in the program.
In the meantime, the UMN had responded favorably to most of the Linux Foundation TAB’s requests. Later the UMN gave full disclosure to the Linux community about who did what and how the Hypocrite Commits project was conducted.
Looking ahead, the Linux community wants to work with the UMN again if the school improves “the quality of the changes that are proposed for inclusion into the kernel.”
On the Linux side, the TAB members wrote: the “TAB, working with researchers, will create a document explaining best practices for all research groups to follow when working with the kernel (and open-source projects in general).”
Specifically, with UMN, since trust has been lost, the TAB asks that UMN, as many companies and other research organizations do:
designate a set of experienced internal developers to review and provide feedback on proposed kernel changes before those changes are submitted publicly. This review catches obvious mistakes and relieves the community of the need to repeatedly remind developers of elementary practices like adherence to coding standards and thorough testing of patches. It results in a higher-quality patch stream that will encounter fewer problems in the kernel community.
Until that’s done, “patches from UMN will continue to find a chilly reception.”