cedricrupb/ASE22-neural-bug-detectors-comparable-to-developers: ASE22 Artifact for "Are Neural Bug Detectors Comparable to Software Developers on Variable Misuse bugs?"

This artifact includes all results related to our study "Are Neural Bug Detectors Comparable to Software Developers on Variable Misuse bugs?".

In our study, we compared the performance of developers in identifying variable misuse bugs with the performance of neural bug detectors. For estimating the developer performance, we performed a survey with over 100 developers. The answers collected in this survey are provided in this artifact in a machine readable format. How to facilitate the answers in your own project (reuse) is described in our quick start guide.

In addition, we included all necessary scripts to replicate our evaluation results in this artifact. We describe them in the following.

Content The artifact includes the following components:

• Web UI: The developer survey was performed online in the browser of the participants. For this, we created a custom web interface tailored for our study task. We included both the implementation of the frontend (website) and backend implementation (buisness logic and database) in this artifact. Therefore, it is not only possible to replicate our survey with same interface and a new group of participants but it is also possible to extend the interface for future studies. (Folder: webui/)

• Neural bug detectors: We evaluate the performance of the developers against two neural bug detectors. In this artifact, we include the bug detectors (implementation + trained models) and the evaluation script used for producing our results. Besides the replication of our bug detector evaluation, the detectors can also be used in future projects for detecting variable misuse bugs in Java methods. (Folder: models/)

• Analysis scripts: After collecting the raw results from the developers and neural bug detectors, we performed several analysis to gain insights how developers and bug detectors compare on the variable misuse task. We include all analysis steps in form of Jupyter notebooks in the artifact. With this, it is possible to reproduce all the figures of our paper. (Folder: notebooks/)

We also provide a more detailed overview of the project structure below.

USAGE: The collection of participant answers are provided in a machine readable format and can be easily reused outside of this artifact. To replicate our results, we recommend the use of our virtual machine which is available here. However, the artifact also contains the necessary content to manually install all the components which makes it simpler to facilitate the components of this artifact.

UPDATE: This version improves the description of the installation process.