Home › Support › How to export a nuiCapture Session

How to export a nuiCapture Session

How to export a nuiCapture Session

A nuiCapture Session (NCS) can be exported by first selecting ‘Open Capture Session(s)…’ from the ‘File’ menu or toolbar. After loading the nuiCapture Session(s) and modifying the corresponding export settings, click the ‘Export’ toggle button to begin the export process. Click the toggle button again to cancel the export process.

1: Adding/removing nuiCapture Sessions for export

Under the ‘Capture Sessions’ panel, click on the buttons labeled ‘Add Capture Session(s)’ and ‘Remove Capture Session’ to add or remove an NCS file from the export queue, respectively. Multiple NCS files can be added concurrently.

2: Export settings

After adding a nuiCapture session to the export queue, the ‘Properties’ panel is automatically populated with the default export settings of the Kinect sensors captured during the session. The capture settings of a sensor are stored in a nuiCapture Video (NCV) file. A NCS file stores the path of the NCV file of each sensor captured during the session. When loading a NCS file, the system will prompt the user to manually select the location of a NCV file that is not found. Alternatively, paths within a NCS file can be modified by opening the NCS file with a standard text editor and manually changing the paths.

Each row in the ‘Sensor Configuration’ table represents a Kinect sensor captured during the session. The ‘Properties’ panel contains the export settings for a selected sensor. A disabled setting signifies that the associated data was not captured. Follow these steps to modify the export settings of a nuiCapture session:

  1. Check the checkbox labeled ‘Export’ in the ‘Properties -> General’ panel to enable the export of the captured data of the selected Kinect sensor.
  2. The export frame rate can be changed from the default by adjusting the ‘Frame Rate’ slider.
  3. The start and end frame indices for export can be modified by entering appropriate numerical values into the text boxes labeled ‘Export Start Frame’ and ‘Export End Frame’, respectively. Clear both text boxes to reset to default.
  4. Select a path to which the exported data will be written to by clicking the ‘Output Path’ ‘Browse…’ button in the ‘Properties -> General’ panel.
  5. Check the checkbox labeled ‘Export’ in the ‘Properties -> Depth’ panel to enable the export of depth image data.
  6. Check the checkbox labeled ‘Export’ in the ‘Properties -> Color’ panel to enable the export of color image data.
  7. Check the checkbox labeled ‘Export’ in the ‘Properties -> Skeleton’ panel to enable the export of skeleton data.
  8. Check the checkbox labeled ‘Export’ in the ‘Properties -> Face’ panel to enable the export of facial data.
  9. Check the checkbox labeled ‘Export’ in the ‘Properties -> Audio’ panel to enable the export of audio data.
  10. Check the checkbox labeled ‘AVI’ in the ‘Properties -> Export Formats’ panel to render an AVI movie file of each selected visual data type (skeletal and facial data are rendered in a single video).
  11. Check the checkbox labeled ‘BVH’ in the ‘Properties -> Export Formats’ panel to output BVH files containing the skeletal and/or facial data. Select between an Autodesk 3DSMax- and MotionBuilder-compliant format.
  12. Check the checkbox labeled ‘EXR’ in the ‘Properties -> Export Formats’ panel to output OpenEXR image files containing registered point cloud, color, and depth validity (alpha map) data.
  13. Check the checkbox labeled ‘FBX’ in the ‘Properties -> Export Formats’ panel to output FBX files containing the skeletal and/or facial data.
  14. Check the checkbox labeled ‘MAT’ in the ‘Properties -> Export Formats’ panel to output MAT files (for use with Matlab version 5 and above) containing the selected visual data.
  15. Check the checkbox labeled ‘WAV’ in the ‘Properties -> Export Formats’ panel to output a WAV file containing the audio data. The WAV file will be included in generated AVI files.

3: Exporting AVI files

Click the ‘Options…’ button in the ‘AVI’ section under the ‘Properties -> Export Formats’ panel to select the compression format of the AVI files. The AVI rendering is performed using VirtualDub.

4: Skeletal Joint and Bone Smoothing Parameters

To enable skeletal joint and bone smoothing, check the checkbox labeled ‘Enable’ in the respective parameter section. A description of each parameter is provided in the following:

  • Smoothing – The amount of smoothing to apply to the joint position/bone orientation.
  • Correction – The amount of correction to apply to the joint position/bone orientation.
  • Prediction – The number of future frames to predict the joint positions/bone orientations for.
  • Jitter Radius – The deviation distance in meters that defines jitter.
  • Maximum Deviation Radius – The maximum radius in meters that filtered joint positions/bone orientations can deviate from the raw data.

Face model diagram

A diagram of the face model is provided in the following document:
View the face model diagram

5: BVH file description

BVH is a common file format for storing hierarchical skeletal joint information. Currently, the BVH formats for both AutoDesk 3D Studio Max and AutoDesk MotionBuilder are supported. To facilitate use with third party software, a single frame containing a default T-stance is prepended to the motion data. Bone lengths do not reflect the captured skeletal data; a template is used to generate the bone lengths.

6: OpenEXR file description

OpenEXR is a high dynamic-range (HDR) image file format capable of storing 16-bit (half) floating-point values. The channel types and their formats will depend on the visual data types selected for export. If depth data is selected for export, the image file will contain the following four channels:

  • X – contains the x-component of the depth point cloud data in millimeters. The depth data is centered at the origin and thus will contain negative values. The pixel values are stored as 16-bit (half) floating-point values.
  • Y – contains the y-component of the depth point cloud data in millimeters. The depth data is centered at the origin and thus will contain negative values. The pixel values are stored as 16-bit (half) floating-point values.
  • Z – contains the z-component of the depth point cloud data in millimeters. The pixel values are stored as 16-bit (half) floating-point values.
  • A – contains the pixel-wise depth validity data (can be used as an alpha map). A pixel value is set to 1 (fully opaque) if the depth is valid (i.e., within the limits of the selected depth range) and 0 (fully transparent), otherwise. The pixel values are stored as 16-bit (half) floating-point values.

If color data is selected for export, the image file will contain the following three channels:

  • R – the red channel of the color image data. The pixel values are stored as 16-bit (half) floating-point values and are normalized to the range 0 to 1.
  • G – the green channel of the color image data. The pixel values are stored as 16-bit (half) floating-point values and are normalized to the range 0 to 1.
  • B – the blue channel of the color image data. The pixel values are stored as 16-bit (half) floating-point values and are normalized to the range 0 to 1.

If both depth and color data are selected for export, the ‘R’, ‘G’, and ‘B’ channels will have the same dimensions as the depth image data. This is to allow for registered color and depth images. If depth data is not selected for export, the ‘R’, ‘G’, and ‘B’ channels will have the same dimensions as the original color image data.

7: Reading the generated MAT files

A generated MAT file stores the selected visual data in five optional structures named ‘DepthFrame’, ‘ColorFrame’, ‘SkeletonFrame’, ‘SkeletonToColorFrameMapping’, and ‘DepthFrameToColorFrameMapping’. The timestamp (in milliseconds) of the frame relative to the start of the capture session and the frame number are also provided. The following subsections will describe each of these structures.

7.1: DepthFrame

A DepthFrame structure is present in the MAT file when depth data is selected for export. The format of a DepthFrame structure is given below.

  • DepthFrame
    • DepthData
    • Valid
    • PlayerIndex
  • The DepthData structure represents the point cloud. Its format will be described in Section 7.1.1.
  • The Valid matrix represents the validity of each pixel of the depth image. A pixel value of 1 (true) denotes a valid value for the corresponding pixel of the depth image, and 0 (false) an invalid pixel value.
  • The PlayerIndex matrix represents the player index of each pixel of the depth image. A non-zero pixel value signifies that a tracked subject occupies the pixel, and a value of 0 denotes that no tracked subject occupies the pixel.
6.1.1: DepthData

The DepthData structure represents the point cloud. The format of a DepthData structure is given below.

  • DepthData
    • X
    • Y
    • Depth
  • The X matrix contains the pixel-wise x-component of the point cloud.
  • The Y matrix contains the pixel-wise y-component of the point cloud.
  • The Depth matrix contains the pixel-wise z-component of the point cloud.
7.2: ColorFrame

A ColorFrame structure is present in the MAT file when color data is selected for export. The format of a ColorFrame structure is given below.

  • ColorFrame
    • ColorData
  • The ColorData matrix represents the RGB color image.
7.3: SkeletonFrame

A SkeletonFrame structure is present in the MAT file when skeleton data is selected for export. The format of a SkeletonFrame structure is given below.

  • SkeletonFrame
    • FloorClipPlane
    • TrackingMode
    • Skeletons
  • The FloorClipPlane structure contains the coefficients of an estimated floor plane equation. Its format will be described in Section 7.3.1. The general plane equation is:

    Ax + By + Cz + D = 0

    where:

    A = FloorClipPlane.X
    B = FloorClipPlane.Y
    C = FloorClipPlane.Z
    D = FloorClipPlane.W

  • The TrackingMode determines the number of joints being tracked by the skeleton pipeline. The ‘Standing’ and ‘Seated’ tracking modes indicate that 20 joints (full skeleton) and 10 joints (upper-body skeleton) are tracked, respectively.
  • The Skeletons array is comprised of Skeleton structures, which will be described in Section 7.3.2.
7.3.1: FloorClipPlane

The FloorClipPlane structure contains the coefficients of an estimated floor plane equation. The format of a FloorClipPlane structure is given below.

  • FloorClipPlane
    • X
    • Y
    • Z
    • W
  • The X value represents the coefficient A in the general plane equation.
  • The Y value represents the coefficient B in the general plane equation.
  • The Z value represents the coefficient C in the general plane equation.
  • The W value represents the coefficient D in the general plane equation.
7.3.2: Skeleton

An array of Skeleton structures is contained within a Skeletons array. It contains the tracking status, joint positions, and bone orientations comprising a skeleton. The format of a Skeleton structure is given below.

  • Skeleton
    • TrackingState
    • Position
    • Joints
    • BoneOrientations
    • ClippedEdges
    • TrackingId
  • The TrackingState represents the tracking state of the skeleton. A TrackingState of ‘Tracked’ signifies that all joint positions are tracked. A TrackingState of ‘PositionOnly’ indicates that the overall position of the skeleton is tracked but not the individual joint positions. A TrackingState of ‘NotTracked’ means that the skeleton is not tracked.
  • The Position structure represents the global position of a tracked subject. The Position structure is described in detail in Section 7.3.2.1.
  • The Joints array is comprised of Joint structures, which will be described in Section 7.3.2.2. The order of the joints within the Joints array is as follows:
    1. HipCenter
    2. Spine
    3. ShoulderCenter
    4. Head
    5. ShoulderLeft
    6. ElbowLeft
    7. WristLeft
    8. HandLeft
    9. ShoulderRight
    10. ElbowRight
    11. WristRight
    12. HandRight
    13. HipLeft
    14. KneeLeft
    15. AnkleLeft
    16. FootLeft
    17. HipRight
    18. KneeRight
    19. AnkleRight
    20. FootRight
  • The BoneOrientations array is comprised of BoneOrientation structures, which will be described in Section 7.3.2.3.
  • The ClippedEdges structure indicates the edges of the field of view that are clipping the skeleton. Its format will be described in Section 7.3.2.4.
  • The TrackingId represents the tracking id of the skeleton
7.3.2.1: Position

The Position structure represents the global position of a tracked subject.

  • Position
    • X
    • Y
    • Z
  • The X value represents the x-component of the subject’s global position.
  • The Y value represents the y-component of the subject’s global position.
  • The Z value represents the z-component of the subject’s global position.
7.3.2.2: Joint

The Joints array is comprised of an array of Joint structures. The format of the Joint structure is given below.

  • Joint
    • JointType
    • Position
  • The JointType string indicates the joint type.
  • The Position structure represents the location of a tracked skeletal joint.
7.3.2.2.1: Position

The Position structure represents the position of a tracked joint. The format of the Position structure is given as follows.

  • Position
    • X
    • Y
    • Z
  • The X value represents the x-component of the joint location.
  • The Y value represents the y-component of the joint location.
  • The Z value represents the z-component of the joint location.
7.3.2.3: BoneOrientation

The Bone Orientation structure contains the orientations of skeletal bones in terms of both absolute and relative transformations. Bones are implicitly defined by the start and end joints identified in a BoneOrientation structure. The format of the BoneOrientation structure is given below.

  • BoneOrientation
    • StartJoint
    • EndJoint
    • HierarchicalRotation
    • AbsoluteRotation
  • The StartJoint string represents the start joint of the bone.
  • The EndJoint string represents the end joint of the bone.
  • The HierarchicalRotation structure provides the amount of rotation in 3D space from the parent bone to the child. This information tells us how much we need to rotate in the 3D space the direction of the bone relative to the parent. This is equivalent to considering the rotation of the reference Cartesian axis in the parent bone object space to the child bone object space, considering that the bone lays on the Y axis of its object space. In the hierarchical definition, the rotation of the Hip Center joint provides the absolute orientation of the player in camera space coordinates. This assumes that the player object space has the origin at the Hip Center joint, the Y axis is upright, the X axes is to the left and the Z axis faces the camera. The format of the HierarchicalRotation structure will be described in Section 7.3.2.3.1. The top-down bone hierarchy is given below.
    HipCenter
    Spine HipLeft HipRight
    ShoulderCenter KneeLeft KneeRight
    ShoulderLeft Head ShoulderRight AnkleLeft AnkleRight
    ElbowLeft ElbowRight FootLeft FootRight
    WristLeft WristRight
    HandLeft HandRight
  • The AbsoluteRotation structure provides the orientation of a bone in the 3D camera space. The orientation of a bone is relative to the child joint and the hip center still contains the orientation of the player. Its format will be described in Section 7.3.2.3.3.
7.3.2.3.1: HierarchicalRotation

The HierarchicalRotation structure provides the amount of rotation in 3D space from the parent bone to the child. The format of the HierarchicalRotation structure is given below.

  • HierarchicalRotation
    • Matrix
    • Quaternion
  • Matrix represents the relative rotation matrix from the parent bone to the child.
  • The Quaternion structure represents the relative quaternion transformation from the parent bone to the child. Its format will be described in Section 7.3.2.3.2.
7.3.2.3.2: Quaternion

The Quaternion structure represents a rotation transformation. Its format is described below.

  • Quaternion
    • X
    • Y
    • Z
    • W
  • The X value represents the x-component of the quaternion.
  • The Y value represents the y-component of the quaternion.
  • The Z value represents the z-component of the quaternion.
  • The W value represents the w-component of the quaternion.
7.3.2.3.3: AbsoluteRotation

The AbsoluteRotation structure provides the orientation of a bone in the 3D camera space. The format of the AbsoluteRotation structure is given below.

  • AbsoluteRotation
    • Matrix
    • Quaternion
  • Matrix represents the rotation of the given bone in the 3D camera space.
  • The Quaternion structure represents the quaternion transformation of the given bone in the 3D camera space. Its format was described in Section 7.3.2.3.2
7.3.2.4: ClippedEdges

The ClippedEdges structure indicates the edges of the field of view that are clipping the skeleton. The format of the BoneOrientation structure is given below.

  • ClippedEdges
    • None
    • Right
    • Left
    • Top
    • Bottom
  • The None value is set to true if none of the edges are clipping the skeleton.
  • The Right value is set to true if the right edge is clipping the skeleton.
  • The Left value is set to true if the left edge is clipping the skeleton.
  • The Top value is set to true if the top edge is clipping the skeleton.
  • The Bottom value is set to true if the bottom edge is clipping the skeleton.
7.4: FaceFrame

A FaceFrame structure is present in the MAT file when facial data is selected for export. The format of a FaceFrame structure is given below.

  • FaceFrame
    • Faces
  • The Faces array is comprised of Face structures, which will be described in Section 7.4.1.
7.4.1: Face

An array of Face structures is contained within a Faces array. It contains the tracking status, facial feature positions, and animation units comprising a face. The format of a Face structure is given below.

  • Face
    • Rect
    • Shape3D
    • ColorToDepthSpaceTranslation
    • ProjectedShape3D
    • AnimationUnitCoefficients
    • Rotation
    • Translation
    • TrackSuccessful
    • TrackingId
  • The Rect structure represents the rectangle in the color image frame of the bounding box around the tracked face. The Rect structure is further described in Section 7.4.1.1.
  • The ColorToDepthSpaceTranslation structure provides an approximate global translation from the color space to the depth space. This translation must be applied to each facial feature point to transform it from the color space to the depth space. The ColorToDepthSpaceTranslation structure is further described in Section 7.4.1.2.
  • The Shape3D structure array contains the 3D facial feature locations in the color space. The Shape3D structure array is further described in Section 7.4.1.3. The order of the facial features within the Shape3D array is as follows:

    • View the face model diagram

    1. TopSkull
    2. TopLeftForehead
    3. MiddleOfForehead1
    4. MidpointBetweenEyebrows1
    5. MiddleUpperEdgeOfNoseBone1
    6. NoseTip1
    7. BottomMiddleEdgeOfNose1
    8. MiddleTopDipUpperLip
    9. MiddleBottomDipLowerLip
    10. AboveChin1
    11. BottomOfChin1
    12. OneHalfTopLeftOfSkull
    13. TopLeftOfSkull
    14. LeftBorderBetweenHairAndForehead
    15. LeftSideOfSkull
    16. LeftOfLeftEyebrow
    17. MiddleTopOfLeftEyebrow
    18. RightOfLeftEyebrow
    19. MiddleBottomOfLeftEyebrow
    20. AboveMidUpperLeftEyelid
    21. OuterCornerOfLeftEye
    22. MiddleTopLeftEyelid
    23. MiddleBottomLeftEyelid
    24. InnerCornerLeftEye
    25. UnderMidBottomLeftEyelid
    26. LeftNoseBorder
    27. LeftNostrilOuterBorder
    28. LeftCheekBone
    29. LowerInnerContactPointBetweenLeftEarAndFace
    30. UpperInnerContactPointBetweenLeftEarAndFace
    31. LeftSideOfChin
    32. OutsideLeftCornerMouth
    33. LeftOfChin
    34. LeftTopDipUpperLip
    35. TopRightForehead
    36. MiddleOfForehead2
    37. MidpointBetweenEyebrows2
    38. MiddleUpperEdgeOfNoseBone2
    39. NoseTip2
    40. BottomMiddleEdgeOfNose2
    41. MiddleTopLowerLip
    42. Undefined1
    43. Undefined2
    44. Undefined3
    45. OneHalfTopRightOfSkull
    46. TopRightOfSkull
    47. RightBorderBetweenHairAndForehead
    48. RightSideOfSkull
    49. RightOfRightEyebrow
    50. MiddleTopOfRightEyebrow
    51. LeftOfRightEyebrow
    52. MiddleBottomOfRightEyebrow
    53. AboveMidUpperRightEyelid
    54. OuterCornerOfRightEye
    55. MiddleTopRightEyelid
    56. MiddleBottomRightEyelid
    57. InnerCornerRightEye
    58. UnderMidBottomRightEyelid
    59. RightNoseBorder
    60. RightNostrilOuterBorder
    61. RightCheekBone
    62. LowerInnerContactPointBetweenRightEarAndFace
    63. UpperInnerContactPointBetweenRightEarAndFace
    64. RightSideOfChin
    65. OutsideRightCornerMouth
    66. RightOfChin
    67. RightTopDipUpperLip
    68. OuterTopLeftPupil
    69. OuterBottomLeftPupil
    70. OuterTopRightPupil
    71. OuterBottomRightPupil
    72. InnerTopLeftPupil
    73. InnerBottomLeftPupil
    74. InnerTopRightPupil
    75. InnerBottomRightPupil
    76. LeftSideOfNoseTip
    77. RightSideOfNoseTip
    78. LeftUpperEdgeOfNoseBone
    79. RightUpperEdgeOfNoseBone
    80. LeftTopUpperLip
    81. RightTopUpperLip
    82. LeftBottomUpperLip
    83. RightBottomUpperLip
    84. LeftTopLowerLip
    85. RightTopLowerLip
    86. LeftBottomLowerLip
    87. RightBottomLowerLip
    88. MiddleBottomUpperLip
    89. LeftCornerMouth
    90. RightCornerMouth
    91. BottomOfLeftCheek
    92. BottomOfRightCheek
    93. LeftLowerEdgeOfNoseBone
    94. RightLowerEdgeOfNoseBone
    95. NoseBump
    96. AboveThreeFourthLeftEyelid
    97. AboveThreeFourthRightEyelid
    98. ThreeFourthTopLeftEyelid
    99. ThreeFourthTopRightEyelid
    100. ThreeFourthBottomLeftEyelid
    101. ThreeFourthBottomRightEyelid
    102. BelowThreeFourthLeftEyelid
    103. BelowThreeFourthRightEyelid
    104. AboveOneFourthLeftEyelid
    105. AboveOneFourthRightEyelid
    106. OneFourthTopLeftEyelid
    107. OneFourthTopRightEyelid
    108. OneFourthBottomLeftEyelid
    109. OneFourthBottomRightEyelid
    110. OneFourthBottomLeftOuterEyelid
    111. OneFourthBottomRightOuterEyelid
    112. LeftNostrilMiddleBorder
    113. RightNostrilMiddleBorder
    114. LowerOuterContactPointBetweenRightEarAndFace
    115. UpperOuterContactPointBetweenRightEarAndFace
    116. UpperRightSideOfHead
    117. RightSideOfHead
    118. LowerOuterContactPointBetweenLeftEarAndFace
    119. UpperOuterContactPointBetweenLeftEarAndFace
    120. UpperLeftSideOfHead
    121. LeftSideOfHead
  • The ProjectedShape3D structure array represents the 2D projection of the facial feature locations contained in the Shape3D structure to the color image. The ProjectedShape3D structure array is further described in Section 7.4.1.4.
  • The AnimationUnitCoefficients structure provides the coefficients of a set of animation units. The AnimationUnitCoefficients structure is further described in Section 7.4.1.5.
  • The Rotation structure provides the head pose angles. The angles are expressed in degrees, with values ranging from -180 degrees to +180 degrees. The Rotation structure is further described in Section 7.4.1.6.
  • The Translation structure provides the 3D head position in meters. The Translation structure is further described in Section 7.4.1.7.
  • The TrackSuccessful value represents the tracking state of the face. A TrackSuccessful value of 1 (true) signifies that the face was tracked successfully and 0 (false), otherwise.
  • The TrackingId represents the tracking id of the skeleton of the subject that the face data belongs to.
7.4.1.1: Rect

The Rect structure represents the rectangle in the color image frame of the bounding box around the tracked face. The format of the Rect structure is given below.

  • Rect
    • Width
    • Height
    • Left
    • Top
    • Right
    • Bottom
  • The Width value represents the width of the rectangle.
  • The Height value represents the height of the rectangle.
  • The Left value represents the x-coordinate of the left side of the rectangle.
  • The Top value represents the y-coordinate of the top side of the rectangle.
  • The Right value represents the x-coordinate of the right side of the rectangle.
  • The Bottom value represents the y-coordinate of the bottom side of the rectangle.
7.4.1.2: ColorToDepthSpaceTranslation

ColorToDepthSpaceTranslation structure provides an approximate global translation from the color space to the depth space. This translation must be applied to each facial feature point to transform it from the color space to the depth space. The format of the ColorToDepthSpaceTranslation structure is given below.

  • ColorToDepthSpaceTranslation
    • X
    • Y
    • Z
  • The X value represents the x-component of the translation in meters.
  • The Y value represents the y-component of the translation in meters.
  • The Z value represents the z-component of the translation in meters.
7.4.1.3: Shape3D

The Shape3D structure array contains the 3D facial feature locations in the color space. The format of the Shape3D structure is given below.

  • Shape3D
    • X
    • Y
    • Z
  • The X value represents the x-component of the 3D facial feature point location in meters.
  • The Y value represents the y-component of the 3D facial feature point location in meters.
  • The Z value represents the z-component of the 3D facial feature point location in meters.
7.4.1.4: ProjectedShape3D

The ProjectedShape3D structure array represents the 2D projection of the facial feature locations contained in the Shape3D structure to the color image. The format of the ProjectedShape3D structure is given below.

  • ProjectedShape3D
    • X
    • Y
  • The X value represents the x-component of the 2D facial feature point location in pixels.
  • The Y value represents the y-component of the 2D facial feature point location in pixels.
7.4.1.5: AnimationUnitCoefficients

The AnimationUnitCoefficients structure provides the coefficients of a set of action units from the Facial Action Coding System (FACS) (P. Ekman and W. V. Friesen, 1977). The format of the AnimationUnitCoefficients structure is given below.

  • AnimationUnitCoefficients
    • AU0
    • AU1
    • AU2
    • AU3
    • AU4
    • AU5
  • The AU0 (AU10 in FACS) value represents the Upper Lip Raiser action unit. A value of 0 indicates a neutral expression with covering teeth. A value of 1 indicates that the teeth are shown fully. A value of -1 indicates the lip is pushed down maximally.
  • The AU1 (AU26/27 in FACS) value represents the Jaw Lowerer action unit. Values of 0 and -1 indicate a closed jaw. A value of 1 indicates a fully open jaw.
  • The AU2 (AU20 in FACS) value represents the Lip Stretcher action unit. A value of 0 indicates a neutral expression. Values of 1 and -1 indicate fully stretched (joker’s smile) and fully rounded lips (kissing mouth), respectively. A value of -0.5 indicates rounded lips (pout).
  • The AU3 (AU4 in FACS) value represents the Brow Lowerer action unit. A value of 0 indicates a neutral expression. Values of 1 and -1 indicate a maximally lowered and raised brow, respectively.
  • The AU4 (AU13/15 in FACS) value represents the Lip Corner Depressor action unit. A value of 0 indicates a neutral expression. Values of 1 and -1 indicate a very sad frown and very happy smile, respectively.
  • The AU5 (AU2 in FACS) value represents the Outer Brow Raiser action unit. A value of 0 indicates a neutral expression. A value of 1 indicates brows that are raised as in an expression of deep surprise. A value of -1 indicates brows that are fully lowered as in a very sad face.
7.4.1.6: Rotation

The Rotation structure provides the head pose angles. The angles are expressed in degrees, with values ranging from -180 degrees to +180 degrees. The format of the Rotation structure is given below.

  • Rotation
    • X
    • Y
    • Z
  • The X value represents the head rotation about the x axis (pitch) in degrees.
  • The Y value represents the head rotation about the y axis (yaw) in degrees.
  • The Z value represents the head rotation about the z axis (roll) in degrees.
7.4.1.7: Translation

The Translation structure provides the 3D head position in meters. The format of the Translation structure is given below.

  • Translation
    • X
    • Y
    • Z
  • The X value represents the x-component of the head position in meters.
  • The Y value represents the y-component of the head position in meters.
  • The Z value represents the z-component of the head position in meters.
7.5: DepthFrameToColorFrameMapping

A DepthFrameToColorFrameMapping structure is present in a MAT file when both the depth and color modalities are selected for export. It represents the mapping from the depth frame space to the color frame space to perform texture mapping. The format of a DepthFrameToColorFrameMapping structure is given below.

  • DepthFrameToColorFrameMapping
    • X
    • Y
  • The X element represents the x-component of the mapping.
  • The Y element represents the y-component of the mapping.
7.6: SkeletonFrameToColorFrameMapping

A SkeletonFrameToColorFrameMapping structure is present in a MAT file when both the skeleton and color modalities are selected for export. It represents the mapping from the skeleton frame space to the color frame space. The format of a SkeletonFrameToColorFrameMapping structure is given below.

  • SkeletonFrameToColorFrameMapping
    • X
    • Y
  • The X element represents the x-component of the mapping.
  • The Y element represents the y-component of the mapping.
7.7: FaceTriangle

A FaceTriangles structure array is provided when face data is selected for export. This structure is constant and valid for all frames. It represents the triangles comprising the Candide-3 face model mesh. The format of a FaceTriangles structure array is given below.

  • FaceTriangle
    • First
    • Second
    • Third
  • The First element represents the index into Shape3D and ProjectedShape3D corresponding to the first point of a face mesh triangle.
  • The Second element represents the index into Shape3D and ProjectedShape3D corresponding to the second point of a face mesh triangle.
  • The Third element represents the index into Shape3D and ProjectedShape3D corresponding to the third point of a face mesh triangle.