Person Identifier

The person identifier MUST be a unique ID (typically, a UUID) permanently associated with a unique person. This person ID is normally assigned by a node able to perform person identification (face recognition node, voice recognition node, sound source localisation name, identification based on physical features like height/age/gender, person identification based on pre-defined features like the colour of the clothes, etc.) This ID is meant to be persistent so that the robot can recognize people across encounters/sessions. Nodes providing person IDs MAY serialise these IDs to a permanent storage, for them to persist across robot reboots.

When meaningful (see section Person frame), a TF frame MUST be associated to the person ID and named person_<personID>. Due to the importance of the head in human-robot interaction, the person_<personID> frame is expected to be placed as close as possible to the head of the human. If neither the face nor the skeleton are tracked, the person_<personID> frame might be located to the last known position of the human, or removed altogether if no meaningful estimate of the human location is available. See section Person frame for details regarding the person_<personID> frame.

At any given time, the list of currently-seen persons is published under the /humans/persons/tracked topic as hri_msgs/IdsList messages, and the list of all known persons (ie, persons that have been seen and recognized at least once in the past) under /humans/persons/known.

Face Identifier

The face identifier MUST be a unique ID that identifies a detected face. This ID is typically generated by the face detector/head pose estimator upon face detection.

Importantly, this ID is not persistent: once a face is lost (for instance, the person goes out of frame), its ID is not valid nor meaningful anymore. To cater for a broad range of applications (where re-identification might not be always necessary), there is no expectation that the face detector will attempt to recognise the face and re-assign the same face ID if the person reappears.

There is a one-to-one relationship between this face ID and the estimated 6D pose of the head. If the node publishes a head pose estimation, the ROS TF frame MUST be named face_<faceID> (see section Face and Gaze Frames for the face frame conventions).

At any given time, the list of tracked faces SHOULD be published under the /humans/faces/tracked topic as hri_msgs/IdsList messages.

Body Identifier

Similarly to the face identifier, the body identifier MUST be a unique ID, associated to a person’s skeleton. It is normally created by a skeleton tracker upon detection of a skeleton.

Like the face ID, the body ID is not persistent, and is valid only as long as the specific skeleton is tracked by the skeleton tracker which initially detected it.

The corresponding TF frame is body_<bodyID>, and TF frames associated with each of the body parts of the person, MUST suffixed with the same ID (see section Body frames).

At any given time, the list of tracked bodies SHOULD be published under the /humans/bodies/tracked topic as hri_msgs/IdsList messages.

Voice Identifier

Likewise, a speech separation node MUST assign a unique, non-persistent, ID for each detected voice. Tracked voices SHOULD be published under the /humans/voices/tracked topic as hri_msgs/IdsList messages.

Identifier Matching

Associations between IDs (for instance to denote that a given voice belongs to a given person, or a given face to a given body) are expressed by publishing hri_msgs/IdsMatch messages on the /humans/candidate_matches topic. The hri_msgs/IdsMatch message MAY include a confidence level.

A typical implementation will have several specialised nodes publishing candidate matches on /humans/candidate_matches (e.g. a face recognition node providing matches between faces and persons; a voice recognition node providing matches between voices and persons) and one 'person manager' node collecting the candidates, and publishing the most likely associations between a person ID and its face/body/voice IDs under the /humans/persons/ namespace.

Identifier Syntax

Identifiers can be arbitrary, as long as they are unique. We recommend to follow standard syntaxic rules so that they are also valid indentifiers in mainstream programming languages (eg, start with a letter).

Note that using people’s names as identifier is possible, but not generally recommended as collisions are likely.

Faces

The list of currently detected faces (list of face IDs) is published under /humans/faces/tracked (as a hri_msgs/IdsList message).

For each detected face, a namespace /humans/faces/<faceID>/ is created (eg /humans/faces/bf3d/).

The following subtopics MAY then be available, depending on available detectors:

Bodies

The list of currently detected bodies (list of body IDs) is published under /humans/bodies/tracked (as a hri_msgs/IdsList message).

For each detected body, a namespace /humans/bodies/<bodyID>/ is created. The following subtopics MAY then be available, depending on available detectors:

3D body poses SHOULD be exposed via TF frames. This is discussed in Section Kinematic Model and Coordinate Frames.

Voices

The list of currently detected voices (list of voice IDs) is published under /humans/voices/tracked (as a hri_msgs/IdsList message).

For each detected voice, a namespace /humans/voices/<voiceID>/ is created.

The following subtopics MAY then be available, depending on available detectors:

Persons

The list of currently tracked persons (list of person IDs) is published under /humans/persons/tracked (as a hri_msgs/IdsList message).

The list of known persons (either actively tracked, or known but not tracked anymore) is published under /humans/persons/known (as a hri_msgs/IdsList message).

For each detected person, a namespace /humans/persons/<personID>/ is created.

The following subtopics MAY then be available, depending on available detectors, and whether or not the person has yet been matched to a face/body/voice:

Interactions

Finally, the namespace /humans/interactions exposes topics where group-level interactions are published when detected.

See section Group-level Interactions for details.

Illustrative Example

You run a node your_face_detector_node. This node detects two faces, and publishes the corresponding regions of interest and cropped faces. The node effectively advertises and publishes onto the following topics:

> rostopic list
/humans/faces/23bd5/roi     # hri_msgs/NormalizedRegionOfInterest2D
/humans/faces/23bd5/cropped # sensor_msgs/Image
/humans/faces/b092e/roi     # hri_msgs/NormalizedRegionOfInterest2D
/humans/faces/b092e/cropped # sensor_msgs/Image

You start an additional node to recognise expressions: your_expression_classifier_node. The node subscribes to the /humans/faces/<faceID>/cropped topics and publishes expressions for each faces under the same namespace:

> rostopic list
/humans/faces/23bd5/roi
/humans/faces/23bd5/cropped
/humans/faces/23bd5/expression # hri_msgs/Expression
/humans/faces/b092e/roi
/humans/faces/b092e/cropped
/humans/faces/b092e/expression # hri_msgs/Expression

You then launch your_body_tracker_node. It detects one body:

> rostopic list
/humans/faces/23bd5/...
/humans/faces/b092e/...
/humans/bodies/67dd1/roi     # hri_msgs/NormalizedRegionOfInterest2D
/humans/bodies/67dd1/cropped # sensor_msgs/Image

In addition, you start a 2D/3D pose estimator your_skeleton_estimator_node. The 2D skeleton can be published under the same body namespace, and the 3D skeleton is published as a joint state. The joint state can then be converted into TF frames using eg a URDF model of the human, alongside a robot_state_publisher:

> rostopic list
/humans/faces/23bd5/...
/humans/faces/b092e/...
/humans/bodies/67dd1/roi
/humans/bodies/67dd1/cropped
/humans/bodies/67dd1/skeleton2d # hri_msgs/Skeleton2D
/humans/bodies/67dd1/joint_states # sensor_msgs/JointState


> xacro ws/human_description/urdf/human-tpl.xacro id:=67dd1 height:=1.7 > body-67dd1.urdf
> rosparam set human_description_67dd1 -t body-67dd1.urdf
> rosrun robot_state_publisher robot_state_publisher joint_states:=/humans/bodies/67dd1/joint_states robot_description:=human_description_67dd1

So far, faces and bodies are detected, but they are not yet 'unified' as a person.

First, we need a stable way to associate a face to a person. This would typically require a node for facial recognition. Such a node would subscribe to each of the detected faces' /cropped subtopics, and publish candidate matches on the /humans/candidate_matches topic, using a hri_msgs/IdsMatch message. For instance:

> rostopic echo /humans/candidate_matches
face_id: "23bd5"
body_id: ''
voice_id: ''
person_id: "76c0c"
confidence: 0.73
---

In that example, the person ID 76c0c is created and assigned by the face recognition node itself.

Finally, you would need a your_person_manager_node to publish the /humans/persons/76c0c/ subtopics based on the candidate matches:

> rostopic list
/humans/faces/23bd5/...
/humans/faces/b092e/...
/humans/bodies/67dd1/...
/humans/persons/76c0c/face_id

In this simple example, only the /face_id subtopic would be advertised (with a latched message pointing to the face ID 23bd5). In practice, additional information could be gathered by the your_person_manager_node to expose eg soft biometrics, engagement, etc. Similarly, the association between the person and its body would have to be performed by a dedicated node.

Overall, six independent nodes are combined to implement this pipeline:

This possible pipeline is only for illustration purposes: depending on each specific pipeline implementations, some of these nodes might be merged or on the contrary, further divided into smaller nodes. For instance, one might choose to integrate together the face recogniser node and the person manager.

Note as well that in order to build a complete perception pipeline for HRI, additional nodes would be needed, for instance for voice processing.

Kinematic Model of the Human

The main 15 links defined on the human body are presented in the above diagram. Frames orientations and naming are based on REP-103 and REP-120. Right: render of the reference URDF model of the human body in rviz.

The following diagram presents all the link (boxes) and joints (arrows) in the recommended human kinematic model.

In practice, each of these links and joints must be suffixed with the corresponding <bodyID>, as several skeletons might be present at the same time.

A parametric URDF model of humans is available in the human_description package. It SHOULD be used to instantiate at run-time new human URDF model, adjusted for the e.g. height of the detected persons. The person's joint state (published under /humans/bodies/<bodyID>/joint_states) can then be used with eg a robot_state_publisher node <http://wiki.ros.org/robot_state_publisher> to publish the body's TF frames.

When generated, the URDF models of the humans should be loaded on the ROS parameter server under /human_description_<bodyID>.

Face and Gaze Frames

• Head pose estimation nodes MUST publish the head 6D pose as a TF frame named face_<faceID> where <faceID> stands for the unique face identifier.
• the parent of this frame is the sensor frame used to estimate the face pose.
• The origin of the frame must be the sellion (defined as the deepest midline point of the angle formed between the nose and forehead. It can generally be approximated to the mid point of line connecting the two eyes).
• The x axis is expected to point forward (ie, out of the face), the z axis is expected to point toward the scalp (ie, up when the person is standing vertically).
• Any other facial landmark published as a TF frame must be parented to the head TF frame. It should be suffixed with the same _<faceID>.

In addition, the person's gaze direction MUST be published as a gaze_<faceID> frame, collocated with the face_<faceID> frame, and with its z axis aligned with the estimated gaze vector, x right, and y down ('optical frame' convention).

If gaze is not estimated beyond general head orientation, the gaze_<faceID>'s z axis will be colinear with the face_<faceID>'s x axis.

Finally, nodes performing attention estimation MAY publish a frame focus_<faceID> representing the estimated focus of attention of the person.

Body Frames

• The body frame is named body_<bodyID> where <bodyID> stands for the unique skeleton identifier.
• The origin of the frame is located at the mid point of the line connecting the hips.
• the parent of this frame is the sensor frame used to estimate the body pose.
• The x axis is expected to point forward (ie, out of the body), the z axis is expected to point toward the head (ie, up when the person is standing vertically).
• The other skeleton points published as TF frames must be parented to the root body_<bodyID> frame, and all be suffixed with the same _<bodyID>. Section Kinematic Model of the Human lists the recommended names of body links and body joints.
• if the skeleton tracker provide an estimate of the head pose, it might publish a frame named head_<bodyID>. It is the joint responsibility of the face tracker and skeleton tracker to ensure that face_<faceID> and head_<bodyID> are consistent with each other.

Voice Frame

• Sound source localisation algorithms can broadcast estimated TF frames for detected voices. These frames should be named voice_<voiceID>.
• The orientation of the frame is meaningless, and should be ignored.

Person Frame

The person_<personID> frame has a slightly more complex semantic and must be interpreted in conjunction with the person's location_confidence value (see Persons topics).

We can distinguish three cases:

1. The human is currently being tracked (ie personID is set, and at least one of faceID or bodyID is set). In this case, location_confidence MUST be set to 1 and:

• when a face ID is also defined, the person_<personID> frame must be collocated with the face_<faceID> frame.
• when a body ID is defined (ie the skeleton is being tracked), the person_<personID> frame must be collocated with the skeleton frame the closest to the head.
• if both the face and body IDs are defined, the person_<personID> frame must be collocated with the face_<faceID> frame.

2. The human is not seen, but has been previously seen. In this case, location_confidence MUST be set to a value < 1 and a person_<personID> TF frame MUST be published as long as location_confidence > 0. Simple implementations might choose to set location_confidence = 0.5 as soon as the person is not actively seen anymore, continuously broadcast the last known location. More advanced implementations might slowly decrease location_confidence over time to represent the fact that the human might have walked away, for instance.
3. The human is known, but has never been seen before. In this case, location_confidence MUST be set to 0, and no TF frame should be broadcast.

Representation of Groups

When detected, group-level interactions are published on the /humans/interactions/groups, using the hri_msgs/Group.msg message type.

Each group is defined by a unique group ID, and a list of person IDs. (groups can only be defined between persons).

Representation of gazing behaviours

Social gazing (eg, gazing between people) is represented as hri_msgs/Gaze.msg messages, published on the /humans/interactions/gazing topic.

Each Gaze.msg messages contain a sender and a receiver that MUST be known persons. Note that the relationship is not symmetrical: "A gazes at B" does not imply "B gazes at A". As such, mutual gaze will lead to two messages being published.

If one or the other of the sender and receiver IDs are not set, the robot is assumed to respectively originate or be the target of the gaze.

Nodes publishing gazing information are expected to continuously publish gaze messages, until the person is not gazing at the target anymore.

Contributors

Antonio Andriella, Lorenzo Ferrini, Youssef Mohamed, Andres Ramirez-Duque

(alphabetic order)

Funding

This work has been primarily funded by PAL Robotics, with the Bristol Robotics Lab/University of the West of England funding the initial research.

In addition, the work leading to this REP has received funding from the European Union through the H2020 SPRING project (grant agreement 871245), and the ACCIÓ Tecniospring TALBOT project.