Eyesweb Siempre User Manual
description
Transcript of Eyesweb Siempre User Manual
EyesWeb XMI 5.3.0 – SIEMPRE Library
December 22, 2012
Part I
Description
1
The EyesWeb SIEMPRE Library includes a collection of software modules and patcheswhich have been specifically designed and developed to support the requirements from theEU ICT Siempre project. In particular, the SIEMPRE Library integrates the EyesWebplatform with modules which are required for a distributed multimodal data recording,playback, analysis.
• The SMPTE decoder block received an audio signal in inputs and decodes the SMPTEtimecode contained in the signal. See SMPTE time code and the references therin fora description of the SMPTE encoding schema.
• The SMPTE encoder block generates an audio signal with an encoded time code.
• The Wave File Writer block writes timeseries in the binary format chosen for theSIEMPRE recordings. The chosen binary format uses a small subset of the Broadcast
Wave Format specifications.
• The DeckLink input block add support for the DeckLink family of framegrabbers,which are used in the Casa Paganini setup (see Acquisition setup at Casa Paganini)for synchronized audio/video input from High Definition (HD) videocameras.
• The Qualisys SMPTE decoder block, which decodes the SMPTE timecode as packedin 32 bits by the Qualisys Track Manager software. This block, toghether with theOSC support already available in EyesWeb, allows integration between EyesWeb andthe Qualisys system. This provides the possibility to analyze, in real-time, the datatracked by the Qualisys Motion Tracking system. For the aim of the SIEMPREproject, the integration is particularly useful to be able to synchronize with the Qual-isys Software to visualize the recorded data.
• The Qualisys SMPTE encoder block, which encodes an SMPTE timecode to a packed32 bits integer number with the same coding schema as the Qualisys Track Manager
software.
• The Timecode generator block, which generates a timecode locked to an audio inputsignal; it may be useful for data playback.
• The Qualisys RT Protocol Decoder block, which get the complete data packet, assent by the Qualisys Track Manager software, and decodes all of the data in it. It letyou to get all the data in a compact way; e.g., all the tracked markers are received inan unique labelled set, instead of being received by separate OSC blocks.
• The Qualisys TSV Reader block, which reads the data exported by the Qualisys
Track Manager software in the TSV (Tab Separated Values) format.
In the following chapters, some examples are described, which show the overall infras-tructure used for many of the recordings which happens in the EU ICT Siempre project
and the related EyesWeb patches.
2
Chapter 1
Example 1: String Quartet recording
The aim is to record audio, video, motion capture, and biometric-sensors data of perfor-mances of a String Quartet. The recording setup includes a Qualisys Motion Trackingsystem, one or more HD videocameras, ambient microphones, instrument microphones,wearable biometric sensors. The recording happens on distinct PCs, and the synchro-nization of the data on the computers is guaranteed by timestamping each sample with areference SMPTE clock which is received by all computers (see Figure 1.1). In the depictedexample, three computers are used, however, it is straighforward to integrate new computerand devices in the recording infrastructure.
The three computers which are visible in the setup perform different operations:
• The black computer in the middle (A) records the Motion Tracking data. The samecomputer also operates as the master clock generator, i.e., it generates the SMPTEsignal which is propoagated to the other computers.
• The lower-left computer (B) records audio/video data coming from the HD camera(JVC GY-HD-251 in this case).
• The lower-right computer (C) records audio data from the instruments.
EyesWeb and the SIEMPRE Library are used for several purposes in this setup.In computer (A) a patch to generate the SMPTE signal on multiple audio channels is
used. The patch si visible in Figure 1.2.In computer (B) a patch to save audio/video data from the HD camera is used. The
patch si visible in Figure 1.3.In computer (C) a patch to save audio data from the microphones on the instruments
is used. The patch si visible in Figure 1.4.
3
Figure 1.1: The acquisition setup installed at Casa Paganini, InfoMus Lab, Genova, Italy.
4
Figure 1.2: The smpte.eywx patch generates an Smpte signal.
Figure 1.3: A patch to save the HD audio/video signal.
Figure 1.4: A patch to save the multichannel audio signal from the microphones on theinstruments.
5
Chapter 2
Qualisys integration
The SIEMPRE project is based on the use of the Qualisys Motion Tracking system toanalyze the movements of the performers. EyesWeb has been extended in order to integratewith Qualisys and receive the tracking data in real-time. Figure 2.1 shows an example of anEyesWeb patch receiving motion tracking data in real-time. The Z coordinate of a markerplaced on the top of a bow is received by the EyesWeb patch and plotted (see the graph inthe lower part of the patch). Moreover, the patch also receives the SMPTE timecode fromQualisys; this timecode allows for synchronization of the processed data.
Another interesting integration example is given in Figure 2.2. In this example, theSmpte audio signal is generated by EyesWeb basing on the value received, via OSC, fromthe Qualisys RTM software. This mechanism may be used, for instance, to listen to therecorded audio tracks in sync with the playback of the recorded motion-tracking data.
As a matter of facts, many audio editors1 allow for using the Smpte value as the referenceto which all the audio data is synchronized. When the audio data is recorded, it is times-tamped with the current Smpte value; when the same Smpte value is received in input, thecorresponding audio data is played. Thus, by regenerating the Smpte signal, EyesWeb cancontrol the playback of the recorded data in synch with the playback of the motion trackingdata in the Qualisys RTM software. To use this patch, open a QTM file2 and process itwith the command Run real time processing on file..., which enables the Qualisys real timeprotocol (i.e., data streaming via OSC). Then, run the patch; you should see the sameSmpte value in both the status bar of the Qualisys software and in the EyesWeb displayin the top of the patch (the one with a black background and red digits). The maximumallowed difference between these values can be tuned by means of the slider. However, notethat imposing a small threshold may cause the generated smpte signal to be resynched tothe value streamed by Qualisys too many times. This can cause degradation of the qualityof the played audio signal as the generated Smpte signal contains discontinuities.
The example in Figure 2.3 shows how the data received by the QTM software can bevisualized in EyesWeb by means of the 3d display. In this patch, the data is receivedby the OSC block using the Raw Mode; i.e., te data is not actually decoded and eachreceived packet is sent, as is, to the Qualisys RT Protocol Decoder block. This blockprovides, as output, two labelled sets, one for the idenditified markers and another one forthe unidentified markers. The identified markers are sent to the BoneLines3DMaker block,
1The setup at Casa Paganini uses Adobe Audition as the audio editor2Of course, the file must have been recorded with Smpte enabled
6
Figure 2.1: The qualisys rt.eywx patch shows the integration between EyesWeb and theQualisys RTM software.
7
Figure 2.2: The qualisys controlled smpte.eywx patch regenerates an audio Smpte time-code based on the Smpte valued received by Qualisys via OSC.
8
which adds bones among markers to form some skeleton structures. These structures arefinally added to an OpenGl 3d world and visualized to the user. Note the difference with thepatch shown in Figure 2.1: in that patch an OSC receiver is used to get a single coordinate(the Z) of a single marker. That aproach makes sense when you need to show a smallnumber of markers and you know their labels in advance (as the OSC receiver needs thename of the marker as the OSC tag; however, when the number of marker increases, as itis in the String Quartet case, that approach is unfeasible due to the large number of blockswhich would be required. To overcame this problem, the Qualisys Visualization patch usesthe RT Protocol decoder block: this block receives a raw packet from the OSC block, anddecodes all the data in it (the packet is composed of many OSC bundles). The data is sentto the output pins; in particular two output pins provide the labelled set of identified andunidentified markers. Thus, the data can be easily managed with standard EyesWeb blocksworking with geometric labelled sets, or it can be visualized to the user as it happens inthis patch.
Figure 2.3: The qualisys visualization.eywx patch shows the visualization, in EyesWeb,of the 3D mocap data.
9
Chapter 3
Mocap data playback and analysis
Once the mocap data is recorded with the Qualisys software (or with any other mocapsystem), the data can be playback and analyzed in EyesWeb. This chaptershows someexample patches, from simple ones which simply playback the mocap data, to more complexones, where data from different sources is playback in synch in EyesWeb. Note that in thischapter we are not going to use the Qualisys software anymore: we suppose the data hasbeen recorder, may be labelled, and exported in some readile format. Most examples willuse the TSV format (Tab Separated Values) or the C3D format. Thus, you do not needto have the Qualisys software to test these patches; you can use the example files providedwith EyesWeb.
This first simple example, visible in Figure 3.1, demonstrates the playback of a TSV(tab Separated Values) file as exported by Qualisys. The TSV player can be controlled byfeeding it with an external clock signal. In this case the clock is extracted from an audiosignal.
The example in Figure 3.2 shows a simple processing of the data coming from a QualisysTSV file. In this case the data of each single performer is extracted, and, for each subsetof markers, we compute the bounding parallelepiped and draw it. Moreover, we extractthe three markers of the head, compute their baricenter, and then we connect the obtainedfour head baricenters through bones. The resulting structure is visualized in the 3d display,together with the performers skeletons and bounding parallelepipeds.
10
Figure 3.1: The qualisys tsv player.eywx patch shows the visualization, in EyesWeb, ofthe 3D mocap data exported by Qualisys in TSV format.
11
Figure 3.2: The qualisys tsv processing.eywx patch shows a simple processing of datacoming from a Qualisys TSV file.
12
Part II
Reference
13
Chapter 4
SIEMPRE Catalog
4.1 Blocks
4.1.1 Qualisys6DTSVReader
bitmap
class name Qualisys6DTSVReadercatalog name SIEMPREcatalog id SIEMPREclass id Qualisys6DTSVReader
Read TSV file from Qualisys. The file must contains at least markers name. In the TSVexport settings window check: - Include TSV header - Export time data for every frame -Export column headers - Include TSV header
Outputs
OutputDoubleLabelledSetid output labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The labelled set of 3d points double, containing points from qualisys.
14
OutputUnidentifiedDoubleLabelledSetid output unidentify labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The labelled set of 3d points double, containing unidentified points from qualisys.
OutputFrameNumberid output frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*
Extracted data’s frame number.
SMPTEid output smptetype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
Extracted data’s SMPTE.
Parameters
FileNameid file nametype Kernel, String datatype (Kernel Catalog).type id kernel, string
layout
Filename,MustExist=true,SaveMode=false,OverwritePrompt=true,Filter=”TSV files (*.tsv)—*.tsv——”
File name
15
ActivationModeid read modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:BangLineTimePolling
domain [ 0, 4 )Activation mode:Bang: external clock connected to the input, is used to activate the block.On line number: external integer value connected to the input (which is interpreted as
the line number of the file) is used to activate the block.On time: external time value connected to the input (which is interpreted as one of the
time columns of the file) is used to activate the block.Polling: an internal signal is used to activate the block. If the file has at least one time
column the signal is generated according to the offset of consecutive times. If there is notime, Eyesweb automatically activates the block.
LoopModeid eof modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:LoopStopPauseNoOutput
domain [ 0, 4 )Loop mode.It works only if ActivationMode is set to Polling mode. Loop: activates circular loop
mode.Stop: not implemented yet.Pause: generates the current value of file. If the input value of line or time is out of
bound it generates first or last value of file.NoOutput: output is not generated.
UseSMPTEid use SMPTEtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
It specifies if the block use SMPTE or time column to read the file.
16
4.1.2 QualisysC3DReader
bitmap
class name QualisysC3DReadercatalog name SIEMPREcatalog id SIEMPREclass id QualisysC3DReader
Read C3D file format. The file follows C3D specifications(www.C3D.org) Actually theblock doesn’t read analod data.
Outputs
OutputDoubleLabelledSetid output labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The labelled set of 3d points double, containing points from qualisys.
OutputUnidentifiedDoubleLabelledSetid output unidentify labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The labelled set of 3d points double, containing unidentified points from qualisys.
17
Parameters
FileNameid file nametype Kernel, String datatype (Kernel Catalog).type id kernel, string
layout
Filename,MustExist=true,SaveMode=false,OverwritePrompt=true,Filter=”C3D files (*.C3D)—*.C3D——”
File name
ActivationModeid read modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:BangLineTimePolling
domain [ 0, 4 )Activation mode:Bang: external clock connected to the input, is used to activate the block.On line number: external integer value connected to the input (which is interpreted as
the line number of the file) is used to activate the block.On time: external time value connected to the input (which is interpreted as one of the
time columns of the file) is used to activate the block.Polling: an internal signal is used to activate the block. If the file has at least one time
column the signal is generated according to the offset of consecutive times. If there is notime, Eyesweb automatically activates the block.
LoopModeid eof modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:LoopStopPauseNoOutput
domain [ 0, 4 )Loop mode.Loop: activates circular loop mode.Stop: not implemented yet.
18
Pause: generates the current value of file. If the input value of line or time is out ofbound it generates first or last value of file.
NoOutput: output is not generated.
19
4.1.3 QualisysRTProtocolDecoder
bitmap
class name QualisysRTProtocolDecodercatalog name SIEMPREcatalog id SIEMPREclass id qualisys rtprotocol decoderauthors Paolo Coletta
Decodes the Raw Data received from the Qualisys QTM software via the OSC RealTimeprotocol.
Details The OSC Realtime protocol of QTM can also be decoded by using ”standard”EyesWeb OSC blocks. However, receiving data about all the markers is not strainghforward,as the number unidentified QTM trajectories is time-varying. This block gets all such dataas once, building labelled-sets as output
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
RawDataid inputtype Base, RawDatatype id base, raw data
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
The Raw Data received from the Qualisys QTM software via the RT protocol
Outputs
Timestampid timestamp outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
The time elapsed since the ecordng was started
20
SMTE timecodeid smpte outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
The value of SMPTE as received by QTM
Frame numberid frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*
The frame number QTM
Unidentified markersid unidentified markerstype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The set of markers which were not recognized by QTM as belonging to an AIM model
Identified markersid identified markerstype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The set of markers which were recognized by QTM as belonging to an AIM model
Parameters
TimeCode Framerateid frame ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:CustomATSC24/film (24fps)NSTC (29.97fps)PAL (25fps)ATSC30 (30fps)
domain [ 0, 5 )
21
The framerate of the SMPTE timecode, i.e., how many timecode words are available ina second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected
Custom Framerateid custom frame ratetype Kernel, Double datatype (Kernel Catalog).type id kernel, double
The custom framerate of the SMPTE timecode; this parameter is used only when Time-Code Framerate is set to Custom
Sampling rateid subframe sampling ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )
The sampling rate of the data in the source system. Since the SMPTE is expressed alsousing frames and subframes, the sampling rate of the date is required to correctly interpretthe frame and subframe fields.
22
4.1.4 QualisysSmpteDecoder
bitmap
class name QualisysSmpteDecodercatalog name SIEMPREcatalog id SIEMPREclass id qualisys smpte decoderauthors Paolo Coletta
Decodes an SMPTE timecode from an integer number. The SMPTE is assumed to beencoded with the same format used in the Qualisys RT protocol (see details).
Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: MinutesBit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
Encoded SMPTE timecodeid inputtype Kernel, Int datatype (Kernel Catalog).type id kernel, int
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
The encoded SMPTE timecode as provided by the Qualisys RT protocolo. Usuallyobtained by means of the OSC block
Outputs
Decoded timecodeid outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
The decoded timecode
23
Parameters
TimeCode Framerateid frame ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:CustomATSC24/film (24fps)NSTC (29.97fps)PAL (25fps)ATSC30 (30fps)
domain [ 0, 5 )The framerate of the SMPTE timecode, i.e., how many timecode words are available in
a second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected
Custom Framerateid custom frame ratetype Kernel, Double datatype (Kernel Catalog).type id kernel, double
The custom framerate of the SMPTE timecode; this parameter is used only when Time-Code Framerate is set to Custom
Sampling rateid subframe sampling ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )
The sampling rate of the data in the source system. Since the SMPTE is expressed alsousing frames and subframes, the sampling rate of the date is required to correctly interpretthe frame and subframe fields.
24
4.1.5 QualisysSmpteEncoder
bitmap
class name QualisysSmpteEncodercatalog name SIEMPREcatalog id SIEMPREclass id qualisys smpte encoderauthors Paolo Coletta
Encodes an SMPTE timecode to an integer number, to make it suitable, for instance, tobe sent via OSC. The encoding protocol is the one used by the Qualisys RT communicationprotocol (see details).
Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: MinutesBit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
Timecodeid inputtype Kernel, Time datatype (Kernel Catalog).type id kernel, time
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
Timecode to be encoded to a packed SMPTE integer
Outputs
Encoded timecodeid outputtype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*
The encoded timecode
25
Parameters
TimeCode Framerateid frame ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:CustomATSC24/film (24fps)NSTC (29.97fps)PAL (25fps)ATSC30 (30fps)
domain [ 0, 5 )The framerate of the SMPTE timecode, i.e., how many timecode words are available in
a second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected
Custom Framerateid custom frame ratetype Kernel, Double datatype (Kernel Catalog).type id kernel, double
The custom framerate of the SMPTE timecode; this parameter is used only when Time-Code Framerate is set to Custom
Sampling rateid subframe sampling ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )
The sampling rate of the data in the source system. Since the SMPTE is expressed alsousing frames and subframes, the sampling rate of the date is required to correctly interpretthe frame and subframe fields.
26
4.1.6 QualisysTSVReader
bitmap
class name QualisysTSVReadercatalog name SIEMPREcatalog id SIEMPREclass id QualisysTSVReader
Read TSV file from Qualisys. The file must contains at least markers name. In the TSVexport settings window check: - Include TSV header - Export time data for every frame -Export column headers - Include TSV header
Outputs
OutputDoubleLabelledSetid output labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The labelled set of 3d points double, containing points from qualisys.
OutputUnidentifiedDoubleLabelledSetid output unidentify labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d doubleinplace id *no*inherited id *no*
The labelled set of 3d points double, containing unidentified points from qualisys.
OutputFrameNumberid output frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intinplace id *no*inherited id *no*
Extracted data’s frame number.
27
SMPTEid output smptetype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
Extracted data’s SMPTE.
Parameters
FileNameid file nametype Kernel, String datatype (Kernel Catalog).type id kernel, string
layout
Filename,MustExist=true,SaveMode=false,OverwritePrompt=true,Filter=”TSV files (*.tsv)—*.tsv——”
File name
ActivationModeid read modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:BangLineTimePollingFrame
domain [ 0, 5 )Activation mode:Bang: external clock connected to the input, is used to activate the block.On line number: external integer value connected to the input (which is interpreted as
the line number of the file) is used to activate the block.On time: external time value connected to the input (which is interpreted as one of the
time columns of the file) is used to activate the block.Polling: an internal signal is used to activate the block. If the file has at least one time
column the signal is generated according to the offset of consecutive times. If there is notime, Eyesweb automatically activates the block.
On frame number: external integer value connected to the input (which is interpretedas the frame number of the file) is used to activate the block.
28
MediaTimeModeid media timetype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
It specifies if the block set time column of the file as media time. If not, interpolatedtime is used to set the value. If the input is not a time , this parameter is ignored and theblock take the column value in the file ad default.
LoopModeid eof modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:LoopStopPauseNoOutput
domain [ 0, 4 )Loop mode.It works only if ActivationMode is set to Polling mode. Loop: activates circular loop
mode.Stop: not implemented yet.Pause: generates the current value of file. If the input value of line or time is out of
bound it generates first or last value of file.NoOutput: output is not generated.
UseSMPTEid use SMPTEtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
It specifies if the block use SMPTE or time column to read the file.
29
4.1.7 QualisysTSVWriter
bitmap
class name QualisysTSVWritercatalog name SIEMPREcatalog id SIEMPREclass id QualisysTSVWriter
Writes to file a geometric labelled set of 3d points. Points coordinates are written astab-separated values, using the same format of the Qualisys software.
Notes By default, the block writes the data in a Qualisys compatible format. Tothis aim, an header is written in the file which is needed by Qualisys to correctly read thedata; the header is needed even if you plan to read the generated file through the EyesWebQualisysTsvreader block. However, the user can decide to omit such header; this makesthe resulting file not compatible with the Qualisys software, but easier to be imported inthird parties software (e.g., Calc, Excel, Matlab). In this case, since the column names arenot written: it is up to the user to remember the meaning of each column
’Labels mode’ parameter: the column number of the generated file must uniquely iden-tify a marker (label), otherwise reading back the file will be impossible. However, the labelsavailable in the input dataset might change during time (e.g., missing markers when captur-ing, new markers available, etc), thus, the problem arises to decide which labels (and hencewhich columns) will be written to file. The user can select among six possible solutions,through the ’Labels mode’ parameter: Custom: the user can specify the number of labelsand, for each label, its value. Note that the set of user-specified labels is furtherly processedby the block to get rid of empty or duplicate labels
Inputs
InputGeometricLabelledSet3DDoubleid input labelled settype Base, BaseGeometricLabelledSet3Doubletype id base, base geometric labelled set 3d double
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
30
Input Geometric Labelled Set of 3D double points. This must be an homogeneous listof 3d geometric points.
Parameters
Filenameid file nametype Kernel, String datatype (Kernel Catalog).type id kernel, string
layout
Filename,MustExist=true,SaveMode=true,OverwritePrompt=true,Filter=”TSV files (*.tsv)—*.tsv—All files (*.*)—*.*——”
The name of the output file
Input typeid input typetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layoutCombo Box:DoubleInt
domain [ 0, 2 )Specifies if the input geometric labelled set is double or int.
Write time columnid write timetype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
Specifies if the block writes the time column on the output file. The value of the timecolumn is extracted from the media time of the input data
Write smpte columnid write smptetype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
Specifies if the block writes the smpte column on the output file. If set to true, anadditional input will be available; the value of the smpte is received through this input
Write frame number columnid write frame numbertype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
Specifies whether to block writes the frame number column on the output file
31
First frame numberid first frame numbertype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )
Specifies the value of the first frame number. By default, frames start at 1, but adifferent value can be specified.
Write headerid write headertype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
Specifies whether the block writes the file header or not. See the block notes for details
NumOfCamerasid num of cameratype Kernel, Int datatype (Kernel Catalog).type id kernel, int
Number of cameras. It’s in the Qualisys TSV header.
Frequencyid frequencytype Kernel, Int datatype (Kernel Catalog).type id kernel, int
Frequency. It’s in the Qualisys TSV header.
Number of analog channelsid num of analogtype Kernel, Int datatype (Kernel Catalog).type id kernel, int
Number of analog channels. This value is not used actually by EyesWeb when readingthe TSV, however, it is used by the Qualisys software and must match the number ofcolumns in the corresponding * a.tsv file.
Analog frequencyid analog freqtype Kernel, Int datatype (Kernel Catalog).type id kernel, int
Analog frequency. This value is not actually used by EyesWeb when reading the TSV,however, it is used by the Qualisys software and refers to the sampling frequency of theanalog data which is available in the corresponding * a.tsv file.
Descriptionid descriptiontype Kernel, String datatype (Kernel Catalog).type id kernel, string
32
A free-text description to be wrote in the TSV header
TimeStampid time stamptype Kernel, String datatype (Kernel Catalog).type id kernel, string
A string representing the timestamp (date and time) of the recording session (e.g.,Friday, December 21st, 2012
Labels modeid labels modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:CustomFirst non-empty frameFirst frameKinect Microsoft SDKKinect OpenNIFile
domain [ 0, 6 )Specifies how the block decides which labels are written to file. See the block notes for
details
33
4.1.8 SmpteDecoder
bitmap
class name SmpteDecodercatalog name SIEMPREcatalog id SIEMPREclass id smpte decoderauthors Paolo Coletta
Decodes an SMPTE timecode from the audio signal.
Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), withminor changes to build under Win32. The derived source code is available at https://svn.infomus.org/public/libltcsmpte/trunk
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
Audio Streamid inputtype Base, PCMAudioBuffertype id base, pcm audio buffer
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
The input audio stream which should contain the SMPTE timecode track
Outputs
Timecodeid outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
The decoded timecode
34
Parameters
TimeCode Trackid time code tracktype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 0, +infinity )
The zero-based index of the timecode track. If it is greater than the number of availablechannels, the last channel will be used.
Output correction modeid output correction modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:NoneOutput offsetBuffer start
domain [ 0, 3 )In general the SMPTE timecode might be not aligned with the audio buffer. This
parameter specifies how to cope with this issue.None: the error caused by this misalignemnt is neglectedOutput offset: an addition output is added to the block. The output specifies where
the SMPTE timecode started with respecte to first sample of the audio buffer. The offsetmight be negative
Buffer start: the output timeoeis adjusted such that it can be interpreted as the timecodeassigned to the first sample of the buffer
Enable Output Lockedid enable output lockedtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
Enable/disable an output providing info about whether the SMPTE signale has beedlocked
TimeCode Framerateid frame ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:CustomATSC24/film (24fps)NSTC (29.97fps)PAL (25fps)ATSC30 (30fps)
domain [ 0, 5 )
35
The framerate of the SMPTE timecode, i.e., how many timecode words are available ina second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected
Custom Framerateid custom frame ratetype Kernel, Double datatype (Kernel Catalog).type id kernel, double
The custom framerate of the SMPTE timecode; this parameter is used only when Time-Code Framerate is set to Custom
36
4.1.9 SmpteEncoder
bitmap
class name SmpteEncodercatalog name SIEMPREcatalog id SIEMPREclass id smpte encoderauthors Paolo Coletta
Encodes an SMPTE timecode to an audio signal.
Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), withminor changes to build under Win32. The derived source code is available at https://svn.infomus.org/public/libltcsmpte/trunk
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
Audio clockid inputtype Base, Audio clocktype id base, audio clock
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
The input audio clock
Outputs
Outputid outputtype Base, PCMAudioBuffertype id base, pcm audio bufferinplace id *no*inherited id *no*
The encoded timecode
37
Parameters
TimeCodeid time codetype Kernel, Time datatype (Kernel Catalog).type id kernel, time
The initial timeocde.
Resetid resettype Kernel, Trigger datatype (Kernel Catalog).type id kernel, trigger
Reset the encoder to the initial timecode value
TimeCode Framerateid frame ratetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:CustomATSC24/film (24fps)NSTC (29.97fps)PAL (25fps)ATSC30 (30fps)
domain [ 0, 5 )The framerate of the SMPTE timecode, i.e., how many timecode words are available in
a second. If Custom is selected a custom value can be specified in the Custom Framerateparameter, otherwise one standard value can be selected
Custom Framerateid custom frame ratetype Kernel, Double datatype (Kernel Catalog).type id kernel, double
The custom framerate of the SMPTE timecode; this parameter is used only when Time-Code Framerate is set to Custom
Enable timecode outputid enable timecode outputtype Kernel, Bool datatype (Kernel Catalog).type id kernel, bool
If this parameter is true, an additional output is added to the block which providesthe interpolated timecode of the begining of the audiobuffer (i.e., the timecode virtuallyassigned to sample 0 of the buffer)
38
4.1.10 TimeCodeGenerator
bitmap
class name TimeCodeGeneratorcatalog name SIEMPREcatalog id SIEMPREclass id timecode generatorauthors Paolo Coletta
Generatae a timecode starting locked to the input audio clock signal.
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
Audio clockid inputtype Base, Audio clocktype id base, audio clock
requiredrequired for initializationrequired for execution
read only/read write read onlyreferred as inplace *no*referred as inherited *no*
The input audio clock
Outputs
Timecode outputid timecode outputtype Kernel, Time datatype (Kernel Catalog).type id kernel, timeinplace id *no*inherited id *no*
The timecode virtually assigned to the sample 0 of the output buffer
Parameters
TimeCodeid time codetype Kernel, Time datatype (Kernel Catalog).type id kernel, time
39
The initial timeocde.
Resetid resettype Kernel, Trigger datatype (Kernel Catalog).type id kernel, trigger
Reset the encoder to the initial timecode value
40
4.1.11 WaveFileWriter
bitmap
class name WaveFileWritercatalog name SIEMPREcatalog id SIEMPREclass id wave file writerauthors Paolo Coletta
Write the input time series as a wav file.
Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interactionand Entrainment using Music PeRformance Experimentation), 2010-2012
Inputs
TimeSeriesid inputtype Kernel, Generic datatypetype id kernel, generic datatype
requiredrequired for initializationrequired for execution
read only/read write read writereferred as inplace *no*referred as inherited *no*
Time series to be saved to file
Required interfacesKernel, StaticTimeSeries
Parameters
Filenameid param filenametype Kernel, String datatype (Kernel Catalog).type id kernel, string
layout
Filename,MustExist=true,SaveMode=true,OverwritePrompt=true,Filter=”Wave files (*wav)—*.wav—All files (*.*)—*.*—All files (*.*)—*.*——”
Name of the wav file. Use an empty name to stop the recording
41
Formatid param formattype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:32bit float (float)64bit float (Double)Normalized 32bit float (float)Normalized 64bit float (Double)
domain [ 0, 4 )Defines the format used to write the sample into the file : ’float’ each sample is writed
as 32bit floating value; ’double’ each sample is writed as 32bit floating value;
Titleid param titletype Kernel, String datatype (Kernel Catalog).type id kernel, string
Title tag. It will be mapped to the ’INAM’ Exif tag
Datetimeid param datetimetype Kernel, String datatype (Kernel Catalog).type id kernel, string
Datetime tag. It will be mapped to the ’ICRD’ Exif tag
Commentid param commenttype Kernel, String datatype (Kernel Catalog).type id kernel, string
Comment tag. Free text that will be mapped to the ’ICMT’ Exif tag
Timecodeid param timecodetype Kernel, String datatype (Kernel Catalog).type id kernel, string
Timecode tag. It will be mapped to the ’ISMP’ Exif tag.
AddMetaDataid add metadata infotype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:No metadataUsing InfoChunkUsing iXml
domain [ 0, 3 )
42
Defines the format used to write metadata into the file.
43
4.1.12 DeckLinkInput
bitmap
class name DeckLinkInputcatalog name DeckLinkcatalog id decklinkclass id decklink inputauthors Paolo Coletta
Acquire synchronized audio/video from a DeckLink card.
Notes With the partial support of the EU ICT Project 250026 - SIEMPRE (SocialInteraction and Entrainment using Music PeRformance Experimentation), 2010-2012
Outputs
Outputid video outputtype Base, Imagetype id base, imageinplace id *no*inherited id *no*
AudioOutputid audio outputtype Base, PCMAudioBuffertype id base, pcm audio bufferinplace id *no*inherited id *no*
Parameters
Device Indexid device indextype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 0, +infinity )
The zero-base index of the Decklink card, in the case that more than one is installed inthe system
44
InputModeid input modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layoutCombo Box:Audio/VideoVideo only
domain [ 0, 2 )Specifies whether to capture audio, or video, or both
Video Modeid video modetype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layout
Combo Box:NTSC 720x486 30/1.001fps interlacedNTSC 720x486 30/1.001fps interlaced (3:2 pulldown)PAL 720x576 25fps interlacedNTSC 720x486 30/1.001fps progressivePAL 720x576 25fps progressiveHD 1920x1080 24/1.001fps interlacedHD 1920x1080 24fps interlacedHD 1920x1080 25fps progressiveHD 1920x1080 30/1.001fps progressiveHD 1920x1080 30fps progressiveHD 1920x1080 25fps interlacedHD 1920x1080 30/1.001fps interlacedHD 1920x1080 30/1.001fps interlacedHD 1920x1080 50fps progressiveHD 1920x1080 60/1.001fps progressiveHD 1920x1080 60fps progressiveHD 1280x720 50fps progressiveHD 1280x720 60/1.001fps progressiveHD 1280x720 24/1.001fps progressive2K 2048x1556 60fps interlaced2K 2048x1556 60fps interlaced2K 2048x1556 60fps interlaced
domain [ 0, 22 )Select the resolution and framerate of the video stream
45
Pixel Formatid pixel formattype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layoutCombo Box:UYVY 4:2:2 packedBGRA 4:4:4 raw
domain [ 0, 2 )Select the pixel format of the video stream
Num Audio Channelsid num channelstype Kernel, Int datatype (Kernel Catalog).type id kernel, intdomain [ 1, +infinity )
Specifies the number of audio channels to capture
Audio Sample Formatid sample formattype Kernel, Int datatype (Kernel Catalog).type id kernel, int
layoutCombo Box:16bits signed32bits signed
domain [ 0, 2 )Specifies the format of each sample of audio
46
Part III
Appendices
47
Appendix A
SIEMPRE Binary Format
A.1 SIEMPRE – Binary file format
Monodimensional timeseries will be saved or converted to time-aligned Broadcast WaveFormat compatible files (i.e., extended WAVE files).
The SIEMPRE project will exploit subset of the Broadcast Wave Format standard. Inparticular it will fix the number of channels to 1 (monodimensional data), the format toWAVE FORMAT IEEE FLOAT (0x03), and the bits per sample to 32.
For a general description of Broadcast Wave Format and Wave files you may refer tothe Microsoft documentation or start from the following links:
• Audio File Format Specifications
• SonicSpot
A.1.1 Format description
The WAVE (sub-)format used for SIEMPRE has the following structure:
Field Length ContentChunkID 4 “RIFF”
ChunkSize 4Size, in bytes, of the RIFF chunk. Should beequal to the length, in bytes, of the file minus 8.
WaveID 4 “WAVE”
Wave chunks . . .
Some chunks. Each chunk is identified by afourCC (four characters identifier) followed bythe size, in bytes, of the chunk (the size doesnot include the FourCC and the size itself). Twomandatory chunks are the “fmt” chunk and the“data” chunk, which contains the characteristicsof the file (sample rate, num channels, samplesize, etc.) and the sample data respectively.
48
The fmt chunk and the data chunks have the following structure:
Fmt ChunkField Length Content
ChunkID 4 “fmt ”
ChunkSize 4For the SIEMPRE case the value is 16, a genericreader should be prepared to face different sizestoo.
FormatTag 2For the SIEMPRE case the value isWAVE FORMAT IEEE FLOAT, i.e., 0x03.
NumChannels 2For the SIEMPRE case the value is fixed to 1,i.e., monodimensional channels
NumSamplesPerSeq 4Sampling rate (e.g., somthing around 250.0 forkinematical sensors)
AvgBytesPerSec 4Data rate, in bytes, of the file. Can be computedas NumSamplesPerSeq * BlockAlignment
BlockAlignment 2
Size of a block (a sample for each channel). Canbe computed as NumChannels * (BitsPerSample/ 8). For the SIEMPRE case NumChannels isfixed to 1, BitsPerSample is fixed to 32, thusBlockAlignement is fixed to 4.
BitsPerSample 2Number of bits for each sample. For the SIEM-PRE case this is fixed to 32 (single precisionfloating point numbers)
Data ChunkField Length Content
ChunkID 4 “data”
ChunkSize 4For the SIEMPRE case the value is 4 (size ofsample) multiplied by the number of samples
Data . . . This is the actal data: sequence of float numbersin the SIEMPRE case
Padding bytes 0 or 1In the SIEMPRE case this is not needed as datais aligned. In the general case this is only neededif data is not aligned to a 16bits boundary
49
A.1.2 Application compatibility
The files produced by the Wave File Writer block have been tested to load in the followingapplications:
• Players
– VLC media player
– Windows Media Player
– Quick Time player
• Music Software
– Audacity
– Sony Sound Force
– Sony Vegas Pro
• Computing environments
– Matlab: does not load additional info about the file. This is a general limitationfor the WAVE format support in Matlab.
50