Speculative Distributed Simulation of Very Large Spiking Neural Networks

Arnon Amir, Pallab Datta, William P Risk, Andrew S Cassidy, Jeffrey A Kusnitz, Steve K Esser, Alexander Andreopoulos, Theodore M Wong, Myron Flickner, Rodrigo Alvarez-Icaza, Emmett McQuinn, Ben Shaw, Norm Pass, and Dharmendra S Modha. 2013. Cognitive computing programming paradigm: A Corelet Language for composing networks of neurosynaptic cores. In Proceedings of the The 2013 International Joint Conference on Neural Networks(IJCNN). IEEE, Piscataway, NJ, USA, 1–10. https://doi.org/10.1109/IJCNN.2013.6707078

Rajagopal Ananthanarayanan, Steven K Esser, Horst D Simon, and Dharmendra S Modha. 2009. The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (Portland, Oregon) (SC). ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/1654059.1654124

Peter D Barnes, Christopher D Carothers, David R Jefferson, and Justin M LaPre. 2013. Warp speed: executing time warp on 1,966,080 cores. In Proceedings of the 1st ACM SIGSIM Conference on Principles of Advanced Discrete Simulation (Montréal, Québec, Canada) (SIGSIM PADS ’13). ACM, New York, NY, USA, 327–336. https://doi.org/10.1145/2486092.2486134

Trevor Bekolay, James Bergstra, Eric Hunsberger, Travis DeWolf, Terrence C Stewart, Daniel Rasmussen, Xuan Choo, Aaron Russell Voelker, and Chris Eliasmith. 2014. Nengo: a Python tool for building large-scale functional brain models. Frontiers in neuroinformatics 7 (2014), 13. https://doi.org/10.3389/fninf.2013.00048

T Binzegger. 2004. A Quantitative Map of the Circuit of Cat Primary Visual Cortex. Journal of Neuroscience 24 (2004), 8441–8453. https://doi.org/10.1523/JNEUROSCI.1400-04.2004

Romain Brette, Michelle Rudolph, Ted Carnevale, Michael Hines, David Beeman, James M Bower, Markus Diesmann, Abigail Morrison, Philip H Goodman, Frederick C Harris, Jr, Milind Zirpe, Thomas Natschläger, Dejan Pecevski, Bard Ermentrout, Mikael Djurfeldt, Anders Lansner, Olivier Rochel, Thierry Vieville, Eilif Muller, Andrew P Davison, Sami El Boustani, and Alain Destexhe. 2007. Simulation of networks of spiking neurons: a review of tools and strategies. Journal of computational neuroscience 23, 3 (Dec. 2007), 349–398. https://doi.org/10.1007/s10827-007-0038-6

Kristofor D Carlson, Michael Beyeler, Nikil Dutt, and Jeffrey L Krichmar. 2014. GPGPU accelerated simulation and parameter tuning for neuromorphic applications. In Proceedings of the 19th Asia and South Pacific Design Automation Conference(ASP-DAC). IEEE, Piscataway, NJ, USA, 570–577. https://doi.org/10.1109/ASPDAC.2014.6742952

Nicholas T Carnevale and Michael L Hines. 2006. The NEURON Book. Cambridge University Press, Cambridge, UK. https://doi.org/10.1017/CBO9780511541612

Christopher D Carothers, Kalyan S Perumalla, and Richard M Fujimoto. 1999. Efficient Optimistic Parallel Simulations Using Reverse Computation. ACM Transactions on Modeling and Computer Simulation 9 (1999), 224–253. https://doi.org/10.1145/347823.347828

Andrew S Cassidy, Rodrigo Alvarez-Icaza, Filipp Akopyan, Jun Sawada, John V Arthur, Paul A Merolla, Pallab Datta, Marc Gonzalez Tallada, Brian Taba, Alexander Andreopoulos, Arnon Amir, Steven K Esser, Jeff Kusnitz, Rathinakumar Appuswamy, Chuck Haymes, Bernard Brezzo, Roger Moussalli, Ralph Bellofatto, Christian Baks, Michael Mastro, Kai Schleupen, Charles E Cox, Ken Inoue, Steve Millman, Nabil Imam, Emmett Mcquinn, Yutaka Y Nakamura, Ivan Vo, Chen Guok, Don Nguyen, Scott Lekuch, Sameh Asaad, Daniel Friedman, Bryan L Jackson, Myron D Flickner, William P Risk, Rajit Manohar, and Dharmendra S Modha. 2014. Real-Time Scalable Cortical Computing at 46 Giga-Synaptic OPS/Watt with  100 × Speedup in Time-to-Solution and  100,000 × Reduction in Energy-to-Solution. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(SC). IEEE, Piscataway, NJ, USA, 27–38. https://doi.org/10.1109/SC.2014.8

Andrew S Cassidy, Jun Sawada, Paul Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Filipp Akopyan, Bryan L Jackson, and Dharmendra S Modha. 2016. TrueNorth: A High-Performance, Low-Power Neurosynaptic Processor for Multi-Sensory Perception, Action, and Cognition. Technical Report. Almaden Research Center, IBM Research.

Kit Cheung, Simon R Schultz, and Wayne Luk. 2016. NeuroFlow: A General Purpose Spiking Neural Network Simulation Platform using Customizable Processors. Frontiers in neuroscience 9 (Jan. 2016), 1–15. https://doi.org/10.3389/fnins.2015.00516

Ting-Shuo Chou, Hirak J Kashyap, Jinwei Xing, Stanislav Listopad, Emily L Rounds, Michael Beyeler, Nikil Dutt, and Jeffrey L Krichmar. 2018. CARLsim 4: An Open Source Library for Large Scale, Biologically Detailed Spiking Neural Network Simulation using Heterogeneous Clusters. In Proceedings of the 2018 International Joint Conference on Neural Networks(IJCNN). IEEE, Piscataway, NJ, USA, 1–8. https://doi.org/10.1109/IJCNN.2018.8489326

Davide Cingolani, Alessandro Pellegrini, and Francesco Quaglia. 2017. Transparently Mixing Undo Logs and Software Reversibility for State Recovery in Optimistic PDES. ACM Transactions on Modeling and Computer Simulation 27, 2 (May 2017), 1–26. https://doi.org/10.1145/3077583

Andreas K Fidjeland, Etienne B Roesch, Murray P Shanahan, and Wayne Luk. 2009. NeMo: A Platform for Neural Modelling of Spiking Neurons Using GPUs. In Proceedings of the 20th IEEE International Conference on Application-specific Systems, Architectures and Processors(ASAP). IEEE, Piscataway, NJ, USA, 137–144. https://doi.org/10.1109/ASAP.2009.24

Richard M Fujimoto. 1990. Parallel Discrete Event Simulation. Commun. ACM 33, 10 (Oct. 1990), 30–53. https://doi.org/10.1145/84537.84545

Richard M Fujimoto. 1990. Performance of Time Warp Under Synthetic Workloads. In Proceedings of the SCS Multiconference on Distributed Simulation, David Nicol (Ed.). Society for Computer Simulation International, San Diego, CA, USA, 23–28.

Marc-Oliver Gewaltig and Markus Diesmann. 2007. NEST (NEural Simulation Tool). Vol. 2. Scholarpedia, Chapter 4. https://doi.org/10.4249/scholarpedia.1430

Samanwoy Ghosh-Dastidar and Hojjat Adeli. 2009. Spiking neural networks. International journal of neural systems 19 (2009), 295–308. https://doi.org/10.1142/S0129065709002002

A Goldberg, John A P Sekar, and Jonathan R Karr. 2020. Exact Parallelization of the Stochastic Simulation Algorithm for Scalable Simulation of Large Biochemical Networks. (2020). arxiv:2005.05295 [q-bio.MN]

Samuel Greengard. 2020. Neuromorphic chips take shape. Commun. ACM 63, 8 (July 2020), 9–11. https://doi.org/10.1145/3403960

William Gropp. 2012. MPI 3 and Beyond: Why MPI Is Successful and What Challenges It Faces. In Recent Advances in the Message Passing Interface, Jesper Larsson Träff, Siegfried Benkner, and Jack J Dongarra (Eds.). Lecture Notes in Computer Science, Vol. 7490. Springer International Publishing, Berlin Heidelberg, Germany, 1–9. https://doi.org/10.1007/978-3-642-33518-1_1

Alexander Hanuschkin, Susanne Kunkel, Moritz Helias, Abigail Morrison, and Markus Diesmann. 2010. A general and efficient method for incorporating precise spike times in globally time-driven simulations. Frontiers in neuroinformatics 4 (Oct. 2010), 1–19. https://doi.org/10.3389/fninf.2010.00113

Suzana Herculano-Houzel. 2012. The remarkable, yet not extraordinary, human brain as a scaled-up primate brain and its associated cost. Proceedings of the National Academy of Sciences of the United States of America 109, Supplement 1 (June 2012), 10661–10668. https://doi.org/10.1073/pnas.1201895109

Suzana Herculano-Houzel and Jon H Kaas. 2011. Gorilla and Orangutan Brains Conform to the Primate Cellular Scaling Rules: Implications for Human Evolution. Brain, behavior and evolution 77 (2011), 33–44. https://doi.org/10.1159/000322729

Roger V Hoang, Devyani Tanna, Laurence C Jayet Bray, Sergiu M Dascalu, and Frederick C Harris. 2013. A novel CPU/GPU simulation environment for large-scale biologically realistic neural modeling. Frontiers in neuroinformatics 7 (2013), 10. https://doi.org/10.3389/fninf.2013.00019

David R Jefferson. 1985. Virtual Time. ACM Transactions on Programming Languages and Systems 7, 3 (July 1985), 404–425. https://doi.org/10.1145/3916.3988

Susanne Kunkel, Maximilian Schmidt, Jochen M Eppler, Hans E Plesser, Gen Masumoto, Jun Igarashi, Shin Ishii, Tomoki Fukai, Abigail Morrison, Markus Diesmann, and Moritz Helias. 2014. Spiking network simulation code for petascale computers. Frontiers in neuroinformatics 8 (Oct. 2014), 78. https://doi.org/10.3389/fninf.2014.00078

Henry Markram. 2006. The blue brain project. Nature reviews. Neuroscience 7, 2 (Feb. 2006), 153–160. https://doi.org/10.1038/nrn1848

Cyrille Mascart, Gilles Scarella, Patricia Reynaud-Bouret, and Alexandre Muzy. 2021. Scalability of large neural network simulations via activity tracking with time asynchrony and procedural connectivity. (June 2021). https://doi.org/10.1101/2021.06.12.448096

Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E Granger, Richard P Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J Curry, Andy R Terrel, Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. 2017. SymPy: symbolic computing in Python. PeerJ Computer Science 3 (Jan. 2017), 1–27. https://doi.org/10.7717/peerj-cs.103

Kirill Minkovich, Corey M Thibeault, Michael John O’Brien, Aleksey Nogin, Youngkwan Cho, and Narayan Srinivasa. 2014. HRLSim: a high performance spiking neural network simulator for GPGPU clusters. IEEE transactions on neural networks and learning systems 25, 2 (Feb. 2014), 316–331. https://doi.org/10.1109/TNNLS.2013.2276056

Quang Anh Pham Nguyen, Philipp Andelfinger, Wentong Cai, and Alois Knoll. 2019. Transitioning Spiking Neural Network Simulators to Heterogeneous Hardware. In Proceedings of the 2019 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation(Chicago, IL, USA) (SIGSIM-PADS). ACM, New York, NY, USA, 115–126. https://doi.org/10.1145/3316480.3322893

David M Nicol. 1993. The cost of conservative synchronization in parallel discrete event simulations. J. ACM 40, 2 (April 1993), 304–333. https://doi.org/10.1145/151261.151266

Daniele M Papetti, Simone Spolaor, Daniela Besozzi, Paolo Cazzaniga, Marco Antoniotti, and Marco S Nobile. 2020. On the automatic calibration of fully analogical spiking neuromorphic chips. In Proceedings of the 2020 International Joint Conference on Neural Networks(IJCNN). IEEE, Piscataway, NJ, USA, 1–8. https://doi.org/10.1109/IJCNN48605.2020.9206654

Dejan Pecevski. 2009. PCSIM: A Parallel Simulation Environment for Neural Circuits Fully Integrated with Python. Frontiers in neuroinformatics 3 (2009), 15. https://doi.org/10.3389/neuro.11.011.2009

Alessandro Pellegrini, Roberto Vitali, and Francesco Quaglia. 2012. The ROme OpTimistic Simulator: Core Internals and Programming Model. In Proceedings of the 4th International ICST Conference on Simulation Tools and Techniques(SIMUTOOLS). ICST, Brussels, Belgium, 96–98. https://doi.org/10.4108/icst.simutools.2011.245551

Matthew D Pickett, Gilberto Medeiros-Ribeiro, and R Stanley Williams. 2013. A scalable neuristor built with Mott memristors. Nature materials 12(2013), 114–117. https://doi.org/10.1038/nmat3510

Mark Plagge, Christopher D Carothers, Elsa Gonsiorowski, and Neil Mcglohon. 2018. NeMo: A Massively Parallel Discrete-Event Simulation Model for Neuromorphic Architectures. ACM Transactions on Modeling and Computer Simulation 28 (2018), 1–25. https://doi.org/10.1145/3186317

Chi-Sang Poon and Kuan Zhou. 2011. Neuromorphic Silicon Neurons and Large-Scale Neural Networks: Challenges and Opportunities. Frontiers in neuroscience 5 (2011), 108. https://doi.org/10.3389/fnins.2011.00108

Tobias C Potjans and Markus Diesmann. 2014. The Cell-Type Specific Cortical Microcircuit: Relating Structure and Activity in a Full-Scale Spiking Network Model. Cerebral cortex 24(2014), 785–806. https://doi.org/10.1093/cercor/bhs358

Bruno R Preiss, Wayne M Loucks, and Ian D Macintyre. 1994. Effects of the Checkpoint Interval on Time and Space in Time Warp. ACM Transactions on Modeling and Computer Simulation 4 (1994), 223–253. https://doi.org/10.1145/189443.189444

Francesco Quaglia and Andrea Santoro. 2003. Non-Blocking Checkpointing for Optimistic Parallel Simulation: Description and an Implementation. IEEE Transactions on Parallel and Distributed Systems 14 (2003), 593–610.

Sanjiv Shah and Mark Bull. 2006. OpenMP. In Proceedings of the 2006 ACM/IEEE conference on Supercomputing (Tampa, Florida) (SC). ACM, New York, NY, USA, 13. https://doi.org/10.1145/1188455.1188469

Renan O Shimoura, Nilton L Kamiji, Rodrigo F O Pena, Vinicius L Cordeiro, Cesar C Ceballos, Romaro Cecilia, and Antonio C Roque. 2018. [RE] The Cell-Type Specific Cortical Microcircuit: Relating Structure And Activity In A Full-Scale Spiking Network Model. Zenodo 34(2018), 1537–1557. https://doi.org/10.5281/ZENODO.1244116

Donald L Snyder and Michael I Miller. 2012. Random Point Processes in Time and Space(second ed.). Springer, New York, NY, USA. https://doi.org/10.1007/978-1-4612-3166-0

Athul Sripad, Giovanny Sanchez, Mireya Zapata, Vito Pirrone, Taho Dorta, Salvatore Cambria, Albert Marti, Karthikeyan Krishnamourthy, and Jordi Madrenas. 2018. SNAVA—A real-time multi-FPGA multi-model spiking neural network simulation architecture. Neural networks: the official journal of the International Neural Network Society 97 (Jan. 2018), 28–45. https://doi.org/10.1016/j.neunet.2017.09.011

Marcel Stimberg, Romain Brette, and Dan F M Goodman. 2019. Brian 2, an intuitive and efficient neural simulator. eLife 8, e47314 (Aug. 2019), e47314. https://doi.org/10.7554/eLife.47314

Gianluca Susi, Pilar Garcés, Emanuele Paracone, Alessandro Cristini, Mario Salerno, Fernando Maestú, and Ernesto Pereda. 2021. FNS allows efficient event-driven spiking neural network simulations based on a neuron model supporting spike latency. Scientific reports 11, 1 (June 2021), 12160. https://doi.org/10.1038/s41598-021-91513-8

Tim P Vogels and L F Abbott. 2005. Signal Propagation and Logic Gating in Networks of Integrate-and-Fire Neurons. The Journal of neuroscience: the official journal of the Society for Neuroscience 25, 46 (Nov. 2005), 10786–10795. https://doi.org/10.1523/JNEUROSCI.3508-05.2005

Runchun M Wang, Chetan S Thakur, and André van Schaik. 2018. An FPGA-Based Massively Parallel Neuromorphic Cortex Simulator. Frontiers in neuroscience 12 (April 2018), 213. https://doi.org/10.3389/fnins.2018.00213

Esin Yavuz, James Turner, and Thomas Nowotny. 2016. GeNN: a code generation framework for accelerated brain simulations. Scientific reports 6 (Jan. 2016), 18854. https://doi.org/10.1038/srep18854