TUIO.org

TUIO 1.1 Protocol Specification

Introduction

Since the publication of the TUIO [1] protocol specification to the public domain, apart from its initial implementation within reacTIVision [2], it has been adopted by several other projects related to tangible and multi-touch interaction, such as the NUI group community and several other tangible interaction platforms within academic and commercial contexts. This document describes an incremental and backwards compatible update of the original TUIO specification, defining a third additional blob profile for the geometric description of untagged tangible objects. This update also introduces an additional source message, which allows the identification of the TUIO source and as a consequence the multiplexing of various TUIO sources on the client side. This moderate update, fully preserving the message format of the original TUIO 1.0 specification, should provide an upgrade path towards the more radical changes that will be introduced with the future TUIO 2.0 specification.

Reference

For academic publications we suggest to cite the original TUIO paper instead of this web page:
Kaltenbrunner, M. & Bovermann, T. & Bencina, R. & Costanza, E. "TUIO - A Protocol for Table-Top Tangible User Interfaces", Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation (GW 2005), Vannes (France)

General Observations

This protocol definition is an attempt to provide a general and versatile communication interface between tangible tabletop controller interfaces and underlying application layers. It was designed to meet the needs of tabletop interactive multi-touch surfaces, where the user is able to manipulate a set of objects and draw gestures onto the table surface with the finger tips. The objects are tracked by a sensor system and can be identified and located in position and orientation on the table surface. A second cursor profile, representing for example finger touches, does not have a unique ID and doesn't provide rotation information compared to the object profile. A third blob profile describes the approximate extension of untagged tangible objects or touch regions with an oriented bounding box, represented by an elliptical approximation including its angle and area.

The TUIO protocol is encoded using the Open Sound Control format, which provides an efficient binary encoding method for the transmission of arbitrary controller data. Therefore the TUIO messages can be basically transmitted through any channel that is supported by an actual OSC implementation. The default transport method for the TUIO protocol is the encapsulation of the binary OSC bundle data within UDP packets sent to the default TUIO port number 3333. This default transport method is usually referred as TUIO/UDP, and most implementations are based on this method due to its simplicity and speed when sent over a local or wide area network. Since OSC is not directly bound to a dedicated transport method, alternative transport channels such as TCP can be employed to transmit the OSC encoded TUIO data. As introduced with the TUIO 1.1 implementations, there are already several alternative transport methods available, such as TUIO/TCP and TUIO/FLC (flash local connection via shared memory) to interface with Adobe Flash applications for example.

Implementation Details

The TUIO protocol defines two main classes of messages: SET messages and ALIVE messages. SET messages are used to communicate information about an object's state such as position, orientation, and other recognized states. ALIVE messages indicate the current set of objects present on the surface using a list of unique Session IDs.

To avoid possible errors evolving out of packet loss, no explicit ADD or REMOVE messages are included in the TUIO protocol. The receiver deduces object lifetimes by examining the difference between sequential ALIVE messages.

In addition to SET and ALIVE messages, FSEQ messages are defined to uniquely tag each update step with a unique frame sequence ID. An optional SOURCE message identifies the TUIO source in order to allow source multiplexing on the client side. To summarize:

• Object attributes are sent after each state change using a SET message
• The client deduces object addition and removal from SET and ALIVE messages
• On object removal an updated ALIVE message is sent
• FSEQ messages associate a unique frame id with a bundle of SET and ALIVE messages
• An optional SOURCE message identifies the source application and address

Efficiency & Reliability

In order to provide low latency communication our implementation of the TUIO protocol uses UDP transport. When using UDP the possibility exists that some packets will be lost. Therefore, our implementation of the TUIO protocol includes redundant information to correct possible lost packets, while maintaining an efficient usage of the channel. An alternative TCP connection would assure the secure transport but at the cost of higher latency.

For efficiency reasons set messages are packed into a bundle to completely use the space provided by a UDP packet. Each bundle also includes a redundant alive message to allow for the possibility of packet loss. For larger object sets, a bundle with a series of set messages, each including an alive message, are transmitted. When the surface is quiescent, alive messages are sent at a fixed rate dependent on the channel quality, for example once every second, to ensure that the receiver eventually acquires a consistent view of the set of alive objects.

The state of each alive but unchanged object is periodically resent with additional set messages. This redundant information is resent at a lower rate, and includes only a subset of the unchanged objects at each update. The subset is continuously cycled so that each object is periodically addressed.

Finally, each packet is marked with a frame sequence ID (fseq) message: an incrementing number that is the same for all packets containing data acquired at the same time. This allows the client to maintain consistency by identifying and dropping out-of-order packets. To summarize:

• each bundle starts with a SOURCE message to identify the TUIO source
• each bundle of SET messages includes an ALIVE message containing the session IDs of all currently active TUIO components
• multiple SET messages can be bundled to fully utilize UDP packet capacity
• each bundle is concluded with a frame sequence FSEQ message
• when the surface is quiescent the ALIVE and FSEQ messages are resent periodically
• the state of a cycling subset of alive but unchanged objects can be continuously resent via redundant SET messages

Message & Bundle Format

Since TUIO is implemented using Open Sound Control (OSC) [4] it follows its general syntax. A TUIO client implementation therefore has to make use of an appropriate OSC library such as oscpack or liblo, and has to listen to the following message types and bundle structure, which in this example are referring to the two-dimensional Cursor profile.

  /tuio/2Dcur source application@address
  /tuio/2Dcur alive s_id0 ... s_idN
  /tuio/2Dcur set s_id x_pos y_pos x_vel y_vel m_accel
  /tuio/2Dcur fseq f_id

A typical TUIO bundle will contain an initial ALIVE message, followed by an arbitrary number of SET messages that can fit into the actual bundle capacity and a concluding FSEQ message. A minimal TUIO bundle needs to contain at least the compulsory ALIVE and FSEQ messages. The FSEQ frame ID is incremented for each delivered bundle, while redundant bundles can be marked using the frame sequence ID -1.
The optional source message can be transmitted to allow the multiplexing of several TUIO trackers on the client side. The application@address argument is a single string that specifies the application name and any unique source address (IP, host name, MAC address). If sent on localhost, the second address part can be omitted, hence any string without the @ separator implicitly comes from localhost.

Profiles

We define a set of profiles, which allow the transmission of cursor, object and blob descriptors within the context two dimensional surfaces and special cases also in the 3D space above the table surface. If these predefined profiles don't meet a system's requirements TUIO also permits the definition of free form custom profiles, which allow a user defined set of parameters to be transmitted in a similar manner. Most current TUIO implementations concentrate on the 2D surface profile, although there already exist some partial implementations of the additional 3D profiles. Please note that the following textual representation illustrates the syntax of the individual TUIO messages. An actual OSC implementation encodes these messages in a compact binary form, as specified in the data type table below.

2D Interactive Surface

/tuio/2Dobj set s i x y a X Y A m r
/tuio/2Dcur set s x y X Y m
/tuio/2Dblb set s x y a w h f X Y A m r

2.5D Interactive Surface

/tuio/25Dobj set s i x y z a X Y Z A m r
/tuio/25Dcur set s x y z X Y Z m
/tuio/25Dblb set s x y z a w h f X Y Z A m r

3D Interactive Surface

/tuio/3Dobj set s i x y z a b c X Y Z A B C m r
/tuio/3Dcur set s x y z X Y Z m
/tuio/3Dblb set s x y z a b c w h d v X Y Z A B C m r

custom profile

/tuio/_[formatString]
/tuio/_sixyP set s i x y 0.57

For the custom profile, the attributes of the set message can be assigned in a user defined format, since it carries the message format within its name, in a format similar to the OSC header.

Session ID

The Session ID is a unique identifier for each individual TUIO object, cursor or blob, which is maintained during its individual presence during a session. This unique ID allows to distinguish various objects with the same class ID or to identify individual unmarked blobs or cursors. The session ID usually is unique for all profiles, although using the same session ID within the cursor and blob profile for example indicates a reference to the same individual instance. This allows the additional transmission of the cursor or object geometry using the blob profile if desired.

Object ID

The tangible object components can be distinguished by a dedicated Class ID, which corresponds to a unique identifier that can be associated with a tangible object. This can be for example achieved by tagging the objects with a fiducial marker symbol when used within a computer vision environment. Alternatively the object class can be also derived from its shape, color or any other canonical property. In combination with the Session ID also multiple instances of an Object component with the same Class ID can be distinguished when used simultaneously in a session.

Attributes

The attributes defined in this section reflect the object properties we considered important for an interactive surface interface. Some of these attributes (ID, position and angle) are retrieved directly by the sensor. Others (speed, acceleration) are derived from these primary attributes using timing information. Computing these attributes on the low level side of a tangible user interface system is more reliable, since the necessary timing information does not need to be available to clients.

x = sensor_x / sensor_width
y = sensor_y / sensor_height 

X = (sensor_dx / sensor_width) / dt
Y = (sensor_dy / sensor_height) / dt
m = (speed - last_speed) / dt

A = ((a - last_a) / 2*PI) / dt
r = (A - last_A) / dt