Reflecting on the Use of the Policy-Process-Product Theory in Empirical Software Engineering
Authors: 
Kelechi G. Kalu, Taylor R. Schorlemmer, Sophie Chen, Kyle A. Robinson, Erik Kocinare, 
James C. DavisAuthors Info & Claims
Pages 2112 - 2116
Abstract
The primary theory of software engineering is that an organization’s Policies and Processes influence the quality of its Products. We call this the PPP Theory. Although empirical software engineering research has grown common, it is unclear whether researchers are trying to evaluate the PPP Theory. To assess this, we analyzed half (33) of the empirical works published over the last two years in three prominent software engineering conferences. In this sample, 70% focus on policies/processes or products, not both. Only 33% provided measurements relating policy/process and products. We make four recommendations: (1) Use PPP Theory in study design; (2) Study feedback relationships; (3) Diversify the studied feed-forward relationships; and (4) Disentangle policy and process. Let us remember that research results are in the context of, and with respect to, the relationship between software products, processes, and policies.
References
[1]
ACM SIGSOFT. 2021. ACM SIGSOFT Empirical Standards. https://github.com/acmsigsoft/EmpiricalStandards Accessed: May 3, 2023
[2]
Paschal C. Amusuo, Aishwarya Sharma, Siddharth R. Rao, Abbey Vincent, and James C. Davis. 2022. Reflections on software failure analysis. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 1615–1620. isbn:978-1-4503-9413-0 https://doi.org/10.1145/3540250.3560879
[3]
Dharun Anandayuvaraj and James C Davis. 2022. Reflecting on Recurring Failures in IoT Development. In 37th IEEE/ACM International Conference on Automated Software Engineering. IEEE/ACM, 1–5.
[4]
Stephen J. Ball. 2015. What is policy? 21 years later: reflections on the possibilities of policy research. Discourse: Studies in the Cultural Politics of Education, 36, 3 (2015), May, 306–313. issn:0159-6306, 1469-3739 https://doi.org/10.1080/01596306.2015.1015279
[5]
Victor R. Basili. 2006. The Role of Empirical Study in Software Engineering. In 2006 30th Annual IEEE/NASA Software Engineering Workshop. IEEE, Columbia, MD, USA. 3–6. isbn:978-0-7695-2624-9 https://doi.org/10.1109/SEW.2006.34
[6]
Victor R. Basili, Forrest Shull, and Filippo Lanubile. 1999. Building Knowledge through Families of Experiments. IEEE Transactions on Software Engineering, 25, 04 (1999), July, 456–473. issn:0098-5589 https://doi.org/10.1109/32.799939 Publisher: IEEE Computer Society
[7]
Pierre Bourque, Richard E. Fairley, and IEEE Computer Society. 2023. Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 4.0 (4th ed.). IEEE Computer Society Press, Washington, DC, USA. isbn:0769551661 https://doi.org/10.1109/9781118823096
[8]
Susan S. Brilliant, John C. Knight, and Nancy G. Leveson. 1990. Analysis of faults in an N-version software experiment. IEEE Transactions on software engineering, 16, 2 (1990), 238–247.
[9]
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20, 1 (1960), 37–46.
[10]
H.K. Colebatch. 2018. Introduction to the Handbook on Policy, Process and Governing. In Handbook on Policy, Process and Governing. Edward Elgar Publishing, 1–13. isbn:978-1-78471-487-1 https://doi.org/10.4337/9781784714871.00005
[11]
W Alec Cram, Jeffrey G Proudfoot, and John D’arcy. 2017. Organizational information security policies: a review and research framework. European Journal of Information Systems, 26 (2017), 605–641.
[12]
Luca Di Grazia and Michael Pradel. 2022. The evolution of type annotations in python: an empirical study. In ESEC/FSE (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA. 209–220. isbn:978-1-4503-9413-0 https://doi.org/10.1145/3540250.3549114
[13]
Bogdan Dit, Evan Moritz, Mario Linares-Vasquez, and Denys Poshyvanyk. 2013. Supporting and Accelerating Reproducible Research in Software Maintenance Using TraceLab Component Library. In 2013 IEEE International Conference on Software Maintenance. IEEE, Eindhoven, Netherlands. 330–339. isbn:978-0-7695-4981-1 https://doi.org/10.1109/ICSM.2013.44
[14]
Naranker Dulay, Emil Lupu, Morris Sloman, and N. Damianou. 2001. A policy deployment model for the Ponder language. IEEE Press., Seattle. isbn:978-0-7803-6719-7 https://doi.org/10.1109/INM.2001.918064 Pages: 543
[15]
Jesús M. González-Barahona and Gregorio Robles. 2012. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering, 17, 1-2 (2012), Feb., 75–89. issn:1382-3256, 1573-7616 https://doi.org/10.1007/s10664-011-9181-9
[16]
Hao He, Runzhi He, Haiqiao Gu, and Minghui Zhou. 2021. A large-scale empirical study on Java library migrations: prevalence, trends, and rationales. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA. 478–490. isbn:978-1-4503-8562-6 https://doi.org/10.1145/3468264.3468571
[17]
Chris Hobbs. 2016. Software Development Standards. In Embedded Software Development for Safety-Critical Systems (2nd ed.), Chris Hobbs (Ed.). Elsevier, 65–91. isbn:978-0-08-100599-5
[18]
W.S. Humphrey. 1996. Using a defined and measured Personal Software Process. IEEE Software, 13, 3 (1996), May, 77–88. issn:1937-4194 https://doi.org/10.1109/52.493023
[19]
IEEE. 2014. IEEE Standard for Software Quality Assurance Processes. IEEE Standards Association. https://ieeexplore.ieee.org/document/6838485 IEEE Std 730-2014
[20]
ISO. 2015. Quality management systems – Requirements. International Organization for Standardization. https://www.iso.org/standard/62085.html ISO Standard 9001
[21]
ISO/IEC 12207. 2017. Systems and software engineering–Software life cycle processes. https://www.iso.org/standard/63711.html
[22]
Özgür Kafali, Jasmine Jones, Megan Petruso, Laurie Williams, and Munindar P. Singh. 2017. How Good Is a Security Policy against Real Breaches? A HIPAA Case Study. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). 530–540. https://doi.org/10.1109/ICSE.2017.55
[23]
S. Keshav. 2007. How to Read a Paper. SIGCOMM Comput. Commun. Rev., 37, 3 (2007), jul, 83–84. issn:0146-4833 https://doi.org/10.1145/1273445.1273458
[24]
B.A. Kitchenham, S.L. Pfleeger, L.M. Pickard, P.W. Jones, D.C. Hoaglin, K. El Emam, and J. Rosenberg. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28, 8 (2002), Aug., 721–734. issn:1939-3520 https://doi.org/10.1109/TSE.2002.1027796
[25]
Lech Madeyski and Barbara Kitchenham. 2017. Would wider adoption of reproducible research be beneficial for empirical software engineering research? Journal of Intelligent & Fuzzy Systems, 32, 2 (2017), Jan., 1509–1521. issn:10641246, 18758967 https://doi.org/10.3233/JIFS-169146
[26]
MISRA. 2013. MISRA C 2012: Guidelines for the Use of the C Language in Critical Systems: March 2013. Motor Industry Software Research Association.
[27]
Emerson Murphy-Hill, Gail C. Murphy, and William G. Griswold. 2010. Understanding context: creating a lasting impact in experimental software engineering research. In FSE/SDP workshop on Future of SWEng research (FoSER ’10). Association for Computing Machinery, New York, NY, USA. 255–258. isbn:978-1-4503-0427-6 https://doi.org/10.1145/1882362.1882415
[28]
Meiyappan Nagappan, Thomas Zimmermann, and Christian Bird. 2013. Diversity in software engineering research. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, Saint Petersburg Russia. 466–476. isbn:978-1-4503-2237-9 https://doi.org/10.1145/2491411.2491415
[29]
Prasad Naldurg and Roy H. Campbell. 2003. Modeling insecurity: policy engineering for survivability. In ACM workshop on Survivable and self-regenerative systems (SSRS ’03). Association for Computing Machinery, New York, NY, USA. 91–98. isbn:978-1-58113-784-2 https://doi.org/10.1145/1036921.1036931
[30]
Zachary Newman, John Speed Meyers, and Santiago Torres-Arias. 2022. Sigstore: software signing for everybody. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 2353–2367.
[31]
NIST. [n. d.]. NIST SP 800-12: Chapter 5 - Computer Security Policy. https://csrc.nist.rip/publications/nistpubs/800-12/800-12-html/chapter5.html
[32]
Chinenye Okafor, Taylor R Schorlemmer, Santiago Torres-Arias, and James C Davis. 2022. Sok: Analysis of software supply chain security by establishing secure design properties. In Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses. 15–24.
[33]
Alan J Perlis, Frederick Sayward, and Mary Shaw. 1981. Software metrics: an analysis and evaluation. 5, Mit Press.
[34]
Kai Petersen and Claes Wohlin. 2009. Context in industrial software engineering research. In 2009 3rd International Symposium on Empirical Software Engineering and Measurement. 401–404. https://doi.org/10.1109/ESEM.2009.5316010
[35]
Roger S Pressman. 2014. Software engineering: a practitioner’s approach. McGraw-Hill Education.
[36]
Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021. A comprehensive study of deep learning compiler bugs. In ESEC/FSE (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA. 968–980. isbn:978-1-4503-8562-6 https://doi.org/10.1145/3468264.3468591
[37]
Janet Siegmund, Norbert Siegmund, and Sven Apel. 2015. Views on Internal and External Validity in Empirical Software Engineering. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. 1, 9–19. https://doi.org/10.1109/ICSE.2015.24 ISSN: 1558-1225
[38]
Morris Sloman. 1994. Policy driven management for distributed systems. Journal of Network and Systems Management, 2, 4 (1994), Dec., 333–360. issn:1064-7570, 1573-7705 https://doi.org/10.1007/BF02283186
[39]
Ian Sommerville. 2011. Software engineering (9th ed ed.). Pearson, Boston. isbn:978-0-13-703515-1 978-0-13-705346-9 OCLC: ocn462909026
[40]
Murugiah Souppaya, Karen Scarfone, and Donna Dodson. 2022. Secure Software Development Framework (SSDF) version 1.1 recommendations for mitigating the risk of software vulnerabilities. National Institute of Standards and Technology (U.S.), Gaithersburg, MD. https://doi.org/10.6028/NIST.SP.800-218
[41]
René Wies. 1995. Using a Classification of Management Policies for Policy Specification and Policy Transformation. In Integrated Network Management IV, Adarshpal S. Sethi, Yves Raynaud, and Fabienne Faure-Vincent (Eds.). Springer US, Boston, MA. 44–56. isbn:978-1-4757-5517-6 978-0-387-34890-2 https://doi.org/10.1007/978-0-387-34890-2_4
[42]
Claes Wohlin, Martin Höst, and Anders Wesslén. 1999. Can the personal software process be used for empirical studies. In Proceedings ICSE Workshop on Empirical Studies of Software Development and Evolution.
[43]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg. isbn:978-3-642-29043-5 https://doi.org/10.1007/978-3-642-29044-2
[44]
C. Wohlin and A. Wesslen. 1998. Understanding software defect detection in the Personal Software Process. In Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257). 49–58. issn:1071-9458 https://doi.org/10.1109/ISSRE.1998.730773
[45]
Simon Xu. 2017. Empirical research methods for software engineering: Keynote address. In 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA). 1–1. https://doi.org/10.1109/SERA.2017.7965698
Index Terms
- Reflecting on the Use of the Policy-Process-Product Theory in Empirical Software Engineering
Recommendations
Issues in applying empirical software engineering to software architecture
ECSA'07: Proceedings of the First European conference on Software ArchitectureEmpirical software engineering focuses on the evaluation of software engineering technologies, such as processes and tools, by comparing related sets of data. It has contributed a valuable body of knowledge in several areas such as Software Economics ...
Evidence-Based Software Engineering for Practitioners
Software engineers might make incorrect decisions about adopting new techniques if they donýt consider scientific evidence about the techniquesý efficacy. Procedures used for evidence-based medicine can also apply to software engineering. Such evidence-...
Benchmarking as Empirical Standard in Software Engineering Research
EASE '21: Proceedings of the 25th International Conference on Evaluation and Assessment in Software EngineeringIn empirical software engineering, benchmarks can be used for comparing different methods, techniques and tools. However, the recent ACM SIGSOFT Empirical Standards for Software Engineering Research do not include an explicit checklist for ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2023
2215 pages
ISBN:9798400703270
DOI:10.1145/3611643
- General Chair:
- Satish Chandra,
- Program Chairs:
- Kelly Blincoe,
- Paolo Tonella
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 November 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
ESEC/FSE '23
Sponsor:
ESEC/FSE '23: 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
December 3 - 9, 2023
CA, San Francisco, USA
Acceptance Rates
Overall Acceptance Rate 112 of 543 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 114Total Downloads
- Downloads (Last 12 months)114
- Downloads (Last 6 weeks)20
Reflects downloads up to 05 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in