DéjàVu: a map of code duplicates on GitHub

CV Lopes, P Maj, P Martins, V Saini, D Yang… - Proceedings of the …, 2017 - dl.acm.org
Proceedings of the ACM on Programming Languages, 2017dl.acm.org
Previous studies have shown that there is a non-trivial amount of duplication in source code.
This paper analyzes a corpus of 4.5 million non-fork projects hosted on GitHub representing
over 428 million files written in Java, C++, Python, and JavaScript. We found that this corpus
has a mere 85 million unique files. In other words, 70% of the code on GitHub consists of
clones of previously created files. There is considerable variation between language
ecosystems. JavaScript has the highest rate of file duplication, only 6% of the files are …
Previous studies have shown that there is a non-trivial amount of duplication in source code. This paper analyzes a corpus of 4.5 million non-fork projects hosted on GitHub representing over 428 million files written in Java, C++, Python, and JavaScript. We found that this corpus has a mere 85 million unique files. In other words, 70% of the code on GitHub consists of clones of previously created files. There is considerable variation between language ecosystems. JavaScript has the highest rate of file duplication, only 6% of the files are distinct. Java, on the other hand, has the least duplication, 60% of files are distinct. Lastly, a project-level analysis shows that between 9% and 31% of the projects contain at least 80% of files that can be found elsewhere. These rates of duplication have implications for systems built on open source software as well as for researchers interested in analyzing large code bases. As a concrete artifact of this study, we have created DéjàVu, a publicly available map of code duplicates in GitHub repositories.
ACM Digital Library