Skip to main content
Springer Nature Link
Log in

A mulsemedia framework for delivering sensory effects to heterogeneous systems

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Technological advances in computing have allowed multimedia systems to create more immersive experiences for users. Beyond the traditional senses of sight and hearing, researchers have observed that the use of smell, taste, and touch in such systems is becoming increasingly well-received, leading to a new category of multimedia systems called mulsemedia—multiple sensorial media—systems. In parallel, these systems introduce heterogeneous technologies to deliver different sensory effects such as lighting, wind, vibration, and smell, under varied conditions and restrictions. This new paradigm shift poses many challenges, mainly related to mulsemedia integration, delay, responsiveness, sensory effects intensities, wearable and other heterogeneous devices for delivering sensory effects, and remote delivery of mulsemedia components. In addition, new approaches to interacting with multimedia applications have emerged such as multi-touch interfaces, voice processing, and brain–computer interfaces, giving rise to new kinds of complex interactive systems. In this article, we underpin fundamental challenges for delivering multisensory effects to heterogeneous systems. We propose an interoperable mulsemedia framework for coping with these challenges, meeting the emerging requirements. It is achieved through the evolution of an open distributed mulsemedia system. We changed its core following architectural and design patterns to accommodate different profiles of communication, connectivity, and sensory effects metadata standard according to the need of mulsemedia applications and devices available in the user’s environment. The results include case studies where the framework has been duly applied.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Bulgaria)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. SEMP available at http://sourceforge.net/projects/semediaplayer/.

  2. PlaySEM SE Video Player available at https://github.com/estevaosaleme/PlaySEM_SEVideoPlayer.

  3. PlaySEM SER 1.0.0 available at https://github.com/estevaosaleme/PlaySEM_SERenderer/releases/tag/1.0.0.

  4. PlaySEM SER 1.1.0 available at https://github.com/estevaosaleme/PlaySEM_SERenderer/releases/tag/1.1.0.

  5. libNiFalcon available at https://github.com/libnifalcon.

  6. JTouchTool available at https://github.com/IanJohnArcher/JTouchToolkit.

  7. Haptik Library available at http://sirslab.dii.unisi.it/haptiklibrary/.

  8. CHAI 3D available at https://github.com/chai3d/chai3d.

  9. HAPI (H3DAPI) available at http://www.h3dapi.org/.

  10. OpenHaptics available at https://www.3dsystems.com/haptics-devices/openhaptics.

  11. Immersion's TouchSense SDK available at https://www.immersion.com/technology/#touchsense-technology.

  12. P4A Haptic Toolkit available at https://github.com/NickKaklanis/WebHapticModule.

  13. JHaptic library available at https://github.com/guari/jhaptic.

  14. Haptics.js available at http://www.hapticsjs.org/.

  15. SimHaptics available at https://github.com/filipposanfilippo/SimHaptics.

  16. Haptlet available at http://crgallacher.com/haply-project-open-source-haptics/.

  17. Hajukone available at https://github.com/davidmcgookin/Haju.

  18. inScent available at https://www.uni-ulm.de/?inscent.

  19. Gustometer available at https://github.com/antocanna88/gustometer.

  20. PlaySEM SER 2.0.0 available at https://github.com/estevaosaleme/PlaySEM_SERenderer/releases/tag/2.0.0.

References

  1. Adelstein, B.D., Lee, T.G., Ellis, S.R.: Head tracking latency in virtual environments: psychophysics and a model. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 47, pp. 2083–2087. SAGE Publications Sage, Los Angeles, CA (2003)

  2. Balzarotti, N., Baud-Bovy, G.: Hpge: an haptic plugin for game engines. In: International Conference on Games and Learning Alliance, pp. 330–339. Springer (2018)

  3. Banks, A., Gupta, R.: MQTT Version 3.1.1, OASIS Standard. http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html (2014). Online Accessed 19 October 2017

  4. Bartocci, S., Betti, S., Marcone, G., Tabacchiera, M., Zanuccoli, F., Chiari, A.: A novel multimedia-multisensorial 4d platform. In: 2015 AEIT International Annual Conference (AEIT), pp. 1–6 (2015). https://doi.org/10.1109/aeit.2015.7415215

  5. Broy, M.: The ‘grand challenge’ in informatics: engineering software-intensive systems. Computer 39(10), 72–80 (2006). https://doi.org/10.1109/MC.2006.358

    Article  Google Scholar 

  6. Buschmann, F., Henney, K., Schmidt, D.: Pattern-Oriented Software Architecture: A Pattern Language for Distributed Computing. Wiley, Oxford (2007)

    Google Scholar 

  7. Canna, A., Prinster, A., Fratello, M., Puglia, L., Magliulo, M., Cantone, E., Pirozzi, M.A., Salle, F.D., Esposito, F.: A low-cost open-architecture taste delivery system for gustatory FMRI and BCI experiments. J. Neurosci. Methods 311, 1–12 (2019). https://doi.org/10.1016/j.jneumeth.2018.10.003

    Article  Google Scholar 

  8. Choi, B., Lee, E.S., Yoon, K.: Streaming media with sensory effect. In: Information Science and Applications (ICISA), 2011 International Conference on, pp. 1–6 (2011). https://doi.org/10.1109/icisa.2011.5772390

  9. Cho, H.Y.: Event-Based control of 4D effects using MPEG RoSE. Master’s thesis, School of Mechanical, Aerospace and Systems Engineering. Korea Advanced Institute of Science and Technology. Master’s Thesis (2010)

  10. Comsa, I., Trestian, R., Ghinea, G.: 360° Mulsemedia experience over next generation wireless networks—a reinforcement learning approach. In: 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6 (2018). https://doi.org/10.1109/qomex.2018.8463409

  11. Conti, F.: The chai libraries. Tech. rep. (2003)

  12. Covaci, A., Zou, L., Tal, I., Muntean, G.M., Ghinea, G.: Is multimedia multisensorial?—a review of mulsemedia systems. ACM Comput. Surv. 51(5), 91:1–91:35 (2018). https://doi.org/10.1145/3233774

    Article  Google Scholar 

  13. Dobbelstein, D., Rukzio, E., Herrdum, S.: Demonstration of inscent: a wearable olfactory display as an amplification for mobile notifications. In: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, UbiComp ‘17, pp. 229–232. ACM, New York, NY, USA (2017). https://doi.org/10.1145/3123024.3123185

  14. European Committee for Interoperable Systems: Interoperability (2018). http://www.ecis.eu/ecis-interoperability/

  15. Fette, I., Melnikov, A.: The WebSocket Protocol. RFC 6455. https://tools.ietf.org/html/rfc6455 (2011). Online Accessed 19 Oct 2017

  16. Gallacher, C., Mohtat, A., Ding, S., Kövecses, J.: Toward open-source portable haptic displays with visual-force-tactile feedback colocation. In: Haptics Symposium (HAPTICS), 2016 IEEE, pp. 65–71. IEEE (2016)

  17. Galster, M., Avgeriou, P.: Chapter 6—supporting variability through agility to achieve adaptable architectures. In: Babar, M.A., Brown, A.W., Mistrik, I. (eds.) Agile Software Architecture, pp. 139–159. Morgan Kaufmann, Boston (2014). https://doi.org/10.1016/b978-0-12-407772-0.00005-8

    Chapter  Google Scholar 

  18. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Longman Publishing Co. Inc, Boston (1995)

    MATH  Google Scholar 

  19. Ghinea, G., Timmerer, C., Lin, W., Gulliver, S.R.: Mulsemedia: state of the art, perspectives, and challenges. ACM Trans. Multimed. Comput. Commun. Appl. 11(1s), 17:1–17:23 (2014). https://doi.org/10.1145/2617994

    Google Scholar 

  20. GitHut: GitHut—A Small Place to Discover Languages in GitHub (2019). https://githut.info/

  21. Groher, I., Weinreich, R.: Supporting variability management in architecture design and implementation. In: 2013 46th Hawaii International Conference on System Sciences, pp. 4995–5004 (2013). https://doi.org/10.1109/hicss.2013.505

  22. Howell, M.J., Herrera, N.S., Moore, A.G., McMahan, R.P.: A reproducible olfactory display for exploring olfaction in immersive media experiences. Multimed. Tools Appl. 75(20), 12311–12330 (2016). https://doi.org/10.1007/s11042-015-2971-0

    Article  Google Scholar 

  23. ISO/IEC/IEEE Systems and software engineering: ISO/IEC/IEEE Systems and software engineering—architecture description. ISO/IEC/IEEE 42010:2011(E) (Revision of ISO/IEC 42010:2007 and IEEE Std 1471-2000), pp. 1–46 (2011). https://doi.org/10.1109/ieeestd.2011.6129467

  24. Itkowitz, B., Handley, J., Zhu, W.: The openhaptics/spl trade/toolkit: a library for adding 3d touch/spl trade/navigation and haptics to graphics applications. In: First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference, pp. 590–591 (2005). https://doi.org/10.1109/whc.2005.133

  25. Jalal, L., Anedda, M., Popescu, V., Murroni, M.: Qoe assessment for broadcasting multi sensorial media in smart home scenario. In: 2018 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pp. 1–5 (2018). https://doi.org/10.1109/bmsb.2018.8436875

  26. Kaklanis, N., Votis, K., Tzovaras, D.: Adding haptic feedback to web applications towards improving end-users’ cognitive capabilities. In: 2015 6th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), pp. 245–249 (2015). https://doi.org/10.1109/coginfocom.2015.7390599

  27. Karunanayaka, K., Johari, N., Hariri, S., Camelia, H., Bielawski, K.S., Cheok, A.D.: New thermal taste actuation technology for future multisensory virtual reality and internet. IEEE Trans. Vis. Comput. Graph. 24(4), 1496–1505 (2018). https://doi.org/10.1109/TVCG.2018.2794073

    Article  Google Scholar 

  28. Kim, J.R., Osgouei, R.H., Choi, S.: Effects of visual and haptic latency on touchscreen interaction: a case study using painting task. In: World Haptics Conference (WHC), 2017 IEEE, pp. 159–164. IEEE (2017)

  29. Kim, S.K., Joo, Y.S.: Sensible media simulation in an automobile application and human responses to sensory effects. ETRI J. 35(6), 1001–1010 (2014). https://doi.org/10.4218/etrij.13.2013.0038

    Article  Google Scholar 

  30. Kolsanov, A., Nazaryan, A., Ivaschenko, A., Kuzmin, A.: Intelligent SDK for 3D surgery simulation. In: 2016 6th International Conference—Cloud System and Big Data Engineering (Confluence), pp. 384–387 (2016). https://doi.org/10.1109/confluence.2016.7508148

  31. Kovatsch, M., Lanter, M., Shelby, Z.: Californium: scalable cloud services for the internet of things with coap. In: 2014 International Conference on the Internet of Things (IOT), pp. 1–6 (2014). https://doi.org/10.1109/iot.2014.7030106

  32. Lin, Y., Yang, M., Lin, Y.: Low-cost 4D experience theater using home appliances. IEEE Trans. Multimed. (2018). https://doi.org/10.1109/tmm.2018.2876043

    Google Scholar 

  33. Luque, F.P., Galloso, I., Feijoo, C., Martín, C.A., Cisneros, G.: Integration of multisensorial stimuli and multimodal interaction in a hybrid 3dtv system. ACM Trans. Multimed. Comput. Commun. Appl. 11(1s), 16:1–16:22 (2014). https://doi.org/10.1145/2617992

    Article  Google Scholar 

  34. Martin, S., Hillier, N.: Characterisation of the novint falcon haptic device for application as a robot manipulator. In: Australasian Conference on Robotics and Automation (ACRA), pp. 291–292. Citeseer (2009)

  35. McGookin, D., Escobar, D.: Hajukone: developing an open source olfactory device. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1721–1728. ACM (2016)

  36. Mistrik, I., Galster, M., Maxim, B.R.: Software Engineering for Variability Intensive Systems: Foundations and Applications. CRC Press, Boca Raton (2019)

    Book  Google Scholar 

  37. Monks, J., Olaru, A., Tal, I., Muntean, G.M.: Quality of experience assessment of 3d video synchronised with multisensorial media components. In: 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), pp. 1–6 (2017). https://doi.org/10.1109/bmsb.2017.7986129

  38. Murray, N., Ademoye, O.A., Ghinea, G., Muntean, G.M.: A tutorial for olfaction-based multisensorial media application design and evaluation. ACM Comp. Surv. 50(5), 67:1–67:30 (2017). https://doi.org/10.1145/3108243

    Article  Google Scholar 

  39. Murray, N., Lee, B., Qiao, Y., Muntean, G.M.: The impact of scent type on olfaction-enhanced multimedia quality of experience. IEEE Trans. Syst. Man Cybern. Syst. 47(9), 2503–2515 (2017). https://doi.org/10.1109/TSMC.2016.2531654

    Google Scholar 

  40. Nakamura, H., Miyashita, H.: Development and evaluation of interactive system for synchronizing electric taste and visual content. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ‘12, pp. 517–520. ACM, New York, NY, USA (2012). https://doi.org/10.1145/2207676.2207747

  41. Pimentel, V., Nickerson, B.G.: Communicating and displaying real-time data with websocket. IEEE Internet Comput. 16(4), 45–53 (2012). https://doi.org/10.1109/MIC.2012.64

    Article  Google Scholar 

  42. Pressman, R., Maxim, B.: Software Engineering: A Practitioner’s Approach, 8th edn. McGraw-Hill Science, New York (2014)

    Google Scholar 

  43. Rainer, B., Waltl, M., Cheng, E., Shujau, M., Timmerer, C., Davis, S., Burnett, I., Ritz, C., Hellwagner, H.: Investigating the impact of sensory effects on the quality of experience and emotional response in web videos. In: 2012 Fourth International Workshop on Quality of Multimedia Experience, pp. 278–283 (2012). https://doi.org/10.1109/qomex.2012.6263842

  44. Ranasinghe, N., Do, E.Y.L.: Digital lollipop: studying electrical stimulation on the human tongue to simulate taste sensations. ACM Trans. Multimed. Comput. Commum. Appl. 13(1), 5 (2017)

    Google Scholar 

  45. Ranasinghe, N., Nguyen, T.N.T., Liangkun, Y., Lin, L.Y., Tolley, D., Do, E.Y.L.: Vocktail: a virtual cocktail for pairing digital taste, smell, and color sensations. In: Proceedings of the 2017 ACM on Multimedia Conference, pp. 1139–1147. ACM (2017)

  46. Saleme, E.B., Celestrini, J.R., Santos, C.A.S.: Time evaluation for the integration of a gestural interactive application with a distributed mulsemedia platform. In: Proceedings of the 8th ACM on Multimedia Systems Conference—MMSys’17, pp. 308–314. ACM Press, New York, New York, USA (2017). https://doi.org/10.1145/3083187.3084013

  47. Saleme, E.B., Covaci, A., Mesfin, G., Santos, C.A.S., Ghinea, G.: Mulsemedia DIY: a survey of devices and a tutorial for building your own mulsemedia environment. ACM Comput. Surv. (2019). https://doi.org/10.1145/3319853

    Google Scholar 

  48. Saleme, E.B., Santos, C.A.S., Falbo, R.A., Ghinea, G., Andres, F.: Towards a reference ontology on mulsemedia systems. In: Proceedings of the 10th International Conference on Management of Digital EcoSystems, MEDES ‘18, pp. 23–30. ACM, New York, NY, USA (2018). https://doi.org/10.1145/3281375.3281378

  49. Saleme, E.B., Santos, C.A.S., Ghinea, G.: Coping with the challenges of delivering multiple sensorial media. IEEE MultiMed. (2018). https://doi.org/10.1109/mmul.2018.2873565

    Google Scholar 

  50. Saleme, E.B., Santos, C.A.S., Ghinea, G.: Improving response time interval in networked event-based mulsemedia systems. In: Proceedings of the 9th ACM Multimedia Systems Conference, MMSys ‘18, pp. 216–224. ACM, New York, NY, USA (2018). https://doi.org/10.1145/3204949.3204965

  51. Saleme, E.B., Santos, C.A.S.: PlaySEM: a platform for rendering MulSeMedia compatible with MPEG-V. In: Proceedings of the 21 st Brazilian Symposium on Multimedia and the Web—WebMedia ‘15, pp. 145–148. ACM Press, New York, New York, USA (2015). https://doi.org/10.1145/2820426.2820450

  52. Sanfilippo, F., Weustink, P.B.T., Pettersen, K.Y.: A coupling library for the force dimension haptic devices and the 20-sim modelling and simulation environment. In: IECON 2015—41st Annual Conference of the IEEE Industrial Electronics Society, pp. 000168–000173 (2015). https://doi.org/10.1109/iecon.2015.7392094

  53. Santos, C.A.S., Neto, A.N.R., Saleme, E.B.: An event driven approach for integrating multi-sensory effects to interactive environments. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 981–986. IEEE (2015). https://doi.org/10.1109/smc.2015.178

  54. Shelby, Z., Hartke, K., Bormann, C.: The constrained application protocol (CoAP). RFC 7252. https://tools.ietf.org/html/rfc7252 (2014). Online Accessed 19 Oct 2017

  55. Sommerville, I.: Software engineering, 10th edn. Pearson, London (2015)

    MATH  Google Scholar 

  56. Suk, C.B., Hyun, J.S., Yong, L.H.: Sensory effect metadata for SMMD media service. In: Perry, M., Sasaki, H., Ehmann, M., Bellot, G.O., Dini, O. (eds.) ICIW, pp. 649–654. IEEE Computer Society, Washington D.C. (2009). https://doi.org/10.1109/iciw.2009.104

    Google Scholar 

  57. Sulema, Y.: Asampl: Programming language for mulsemedia data processing based on algebraic system of aggregates. In: Auer, M.E., Tsiatsos, T. (eds.) Interactive Mobile Communication Technologies and Learning, pp. 431–442. Springer, Cham (2018)

    Chapter  Google Scholar 

  58. UPnP Forum: UPnP Device Architecture 1.0. http://upnp.org/specs/arch/UPnP-arch-DeviceArchitecture-v1.0.pdf (2008). Online Accessed 05 Apr 2018

  59. Vi, C.T., Marzo, A., Ablart, D., Memoli, G., Subramanian, S., Drinkwater, B., Obrist, M.: Tastyfloats: a contactless food delivery system. In: Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces, ISS ‘17, pp. 161–170. ACM, New York, NY, USA (2017). https://doi.org/10.1145/3132272.3134123

  60. Waltl, M., Rainer, B., Timmerer, C., Hellwagner, H.: An end-to-end tool chain for sensory experience based on mpeg-v. Image Commun. 28(2), 136–150 (2013). https://doi.org/10.1016/j.image.2012.10.009

    Google Scholar 

  61. Waltl, M., Timmerer, C., Hellwagner, H.: A test-bed for quality of multimedia experience evaluation of sensory effects. In: 2009 International Workshop on Quality of Multimedia Experience, pp. 145–150 (2009). https://doi.org/10.1109/qomex.2009.5246962

  62. Waltl, M., Timmerer, C., Hellwagner, H.: Improving the quality of multimedia experience through sensory effects. In: 2010 Second International Workshop on Quality of Multimedia Experience (QoMEX), pp. 124–129 (2010). https://doi.org/10.1109/qomex.2010.5517704

  63. Wegner, P.: Interoperability. ACM Comput. Surv. 28(1), 285–287 (1996). https://doi.org/10.1145/234313.234424

    Article  MathSciNet  Google Scholar 

  64. Yoon, K., Kim, S.K., Han, J.J., Han, S., Preda, M.: MPEG-V: Bridging the Virtual and Real World, 1st edn. Academic Press, London (2015)

    Google Scholar 

  65. Yoon, K.: End-to-end framework for 4-d broadcasting based on mpeg-v standard. Image Commun. 28(2), 127–135 (2013). https://doi.org/10.1016/j.image.2012.10.008

    Google Scholar 

  66. Yuan, Z., Bi, T., Muntean, G.M., Ghinea, G.: Perceived synchronization of mulsemedia services. IEEE Trans. Multimed. 17(7), 957–966 (2015). https://doi.org/10.1109/TMM.2015.2431915

    Article  Google Scholar 

  67. Yuan, Z., Chen, S., Ghinea, G., Muntean, G.M.: User quality of experience of mulsemedia applications. ACM Trans. Multimed. Comput. Commun. Appl. 11(1s), 15:1–15:19 (2014). https://doi.org/10.1145/2661329

    Article  Google Scholar 

  68. Yuan, Z., Ghinea, G., Muntean, G.M.: Beyond multimedia adaptation: quality of experience-aware multi-sensorial media delivery. IEEE Trans. Multimed. 17(1), 104–117 (2015). https://doi.org/10.1109/TMM.2014.2371240

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Federal University of Espírito Santo, Av. Fernando Ferrari, 514, Vitória, ES, 29075-910, Brazil

    Estêvão Bissoli Saleme & Celso A. S. Santos

  2. Brunel University London, Wilfred Brown Building 215, Kingston Lane, Uxbridge, Middlesex, UB8 3PH, UK

    Gheorghita Ghinea

Authors
  1. Estêvão Bissoli Saleme
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Celso A. S. Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gheorghita Ghinea
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Estêvão Bissoli Saleme.

Additional information

Communicated by A. Ulrich Mauthe.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This study was financed in part by the CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), Finance Codes 88881.187844/2018-01 and 88882.317673/2019-01. Prof. G. Ghinea gratefully acknowledges funding from the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 688503 for the NEWTON project (http://www.newtonproject.eu). E. B. Saleme additionally acknowledges support from the Federal Institute of Espírito Santo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saleme, E.B., Santos, C.A.S. & Ghinea, G. A mulsemedia framework for delivering sensory effects to heterogeneous systems. Multimedia Systems 25, 421–447 (2019). https://doi.org/10.1007/s00530-019-00618-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-019-00618-8

Keywords

Search

Navigation