Building a world-class fingerprint sensor using data insights (part 3)

Recap (parts 1 & 2)

In part 1, I covered the importance of using data insights in helping to create a world-class fingerprint sensor solution. In that journey, five different data areas were relied on -specifically:

1. BigQuery data
2. User trial observations
3. User trial surveys
4. Key Performance Indicators (KPIs)
5. Customer sentiment

In part 2, I began covering BigQuery data insights that looked into Unlock Activity, Unlock Efficiency, and Poor Quality touch tracking. This article will look into Transaction Failures, Finger Enrollments, and Unlocking Speed reports. I'll wrap up with some notes about consolidated reports and the actual SQL queries behind the reports.

Transaction Failures

When using PIN unlock, it's pretty clear what constitutes a failed entry attempt. The user enters a number sequence, hits the enter key, and if that number doesn't match the registered PIN - that is a failed attempt. When you've reached some system defined number of consecutive failed attempts, the phone could do anything from locking you out for a period of time before the next allowed PIN entry, to erasing & factory resetting your phone. The penalty was generally dictated by which carrier the phone was being sold through.

When using fingerprint unlock, what should constitute a failed entry attempt and when should the penalty kick in? After all, we know from the last article that there are incidental touches where the sensor registers unlock attempts that were truly not deliberate attempts by the user.

This is where the concept of transaction failures comes in. Here are the key details that came out in the solution:

• A Transaction Failure is defined as five consecutive failed touches where the matcher failed to match the captured image against any enrolled finger.
• When a Transaction Failure is encountered, the phone would temporarily disarm the fingerprint sensor for 30 seconds, where the user could either wait to reattempt with the fingerprint sensor, or immediately try their backup unlock method, which could be PIN, password, or pattern unlock.
• Any system penalties, like erasing & factory resetting your phone, would be tied to reaching some count of consecutive transaction failures rather than consecutive failed touches.

Transaction failures are pretty bad user experiences, so it was important to know how often it was occurring for a user, above and beyond their Unlock Efficiency metrics. Some examples of what could cause transaction failures:

• User enrolled with one finger and tried unlocking with another finger, not realizing they were not enrolled. This actually happened more than you would think, especially during user trials where people unenrolled/re-enrolled lots over many software releases.
• Fingerprints were worn down through physical activity and stopped being recognized. In our user trial population, we had a guitar player and a car mechanic enthusiast who encountered this.
• Weather changes in dryness and humidity do influence fingerprint features. Enrolled fingerprints in one condition may stop being recognized when weather changes influenced fingerprint features later on, even with the application of template updating. (I'll touch upon template updating in the next section).
• Foreign materials like lotion, food grease, or sweat could temporarily affect unlock performance.
• Not all users - their technique and deliberateness - are created equal and for some of those users they could be the key reason for the bad performance.

Below screenshot is an example of Transaction Failure report (first box) and Finger Enrollment report (second box); we'll cover enrollments next. Ultimately, Transaction Failure (TF) report included the count of transaction failures encountered grouped by device (barcode), software (bldid) release, and date.

Finger Enrollments

Number enrolled impacts

When designing a fingerprint unlock solution, how many fingers should you allow to be enrolled? Overall, we chose allowing up to five fingers for a front-facing sensor. In general, these are some factors to consider:

• Sensor placement - A sensor on the back of the phone may only comfortably accommodate the index finger, whereas front or even side sensor placement may accept thumb and index, or creatively other fingers.
• Left/right hand - Whatever finger(s) are natural to enroll, you could see people enrolling those finger(s) from both hands.
• You - Technically, you can have your significant other, friend, etc... enroll their fingers on your phone too.
• System performance - The more fingers you allowed to enroll, the more system resources (e.g. memory) was required, which was constrained in the secure processor where the critical enrollment & matching fingerprint software executed.

In terms of system performance, beyond memory, how did enrolled finger count affect matcher performance and why was this important to track? Here are some key insights:

• With more fingers enrolled, the matcher software checks take longer to fail when an unrecognized fingerprint is presented, false rejection or otherwise, because there are simply more enrolled fingers to check against.
• In an effort to make successful matching go as fast as possible, the fingerprint sensor vendor was asked to implement an algorithm that would compare the fingerprint against the last matched finger first. For example, if you unlocked with your right index finger last time, then try matching against the right index finger first, before checking against any other enrolled fingers.

Template updating

A bit of nomenclature first. When enrolling a fingerprint, none of the captured images are ever stored. Rather, typically the spatial coordinates of features found in your print are extracted from the image and stored in a file called a template.

Take a look at the size of your thumb print. Now, take a look at the size of the fingerprint sensor on your phone (if it still has one!). You'll of course notice right away that your print is larger than the touch sensor size.

If you can remember back to when you enrolled your finger, the phone user experience may have asked you to touch the sensor over and over, guiding you to touch various areas of your finger to best capture as much of your fingerprint as possible.

Enrollment is a trade-off experience:

• On the one hand, you want an enrolled finger template to cover as much of the fingerprint as possible, constructed from good images, so that matching performance is high. This means the enrollment process should collect lots of prints.
• On the other hand, users don't want to endure a lengthy process to enroll, as learned from user trial surveys and observations. Additionally, as we discussed previously, people's fingerprints do change over time, which negates the benefit of enrollments that collect a lot of prints up front.

How does one address this? Enter the concept of Template Updating. Template updating allows the fingerprint sensor solution to expand or replace in the template new or better information from a high quality match. Here were the rules - any time a fingerprint match:

1. Introduced new or better coverage of a fingerprint but still captured "enough" of the fingerprint found in an existing template AND
2. Represented a high quality image AND
3. Resulted in a very high confidence match score from the matcher

THEN, the template was expanded with this additional fingerprint feature data. With people unlocking their phones 50 times a day, there was ample opportunity to apply template updating to improve match performance and account for fingerprint changes due to weather, wear, etc... So, it was very important to track how much time elapsed between fingerprint enrollment and usage, because FRR would expect to drop as the template quality continued to improve.

The maximum number of prints that could contribute to the template was experimented with during development. Observations showed that increasing the max prints up to a certain point did help drive down FRR for the population of users.

The Fingerprint Enrollment report (second box below), helped capture the following key information:

1. The software release (bldfngrprnt) in which the enrollment was done. During rapid prototyping, different software releases deployed different template structures and so it was important to know under which release enrollment happened to make sense of the results, or to request re-enrollment to generate valid results going forward.
2. The time elapsed since enrollment (tstamp, num of days). This provided a quick view of how much time template updating had been given to do its thing.
3. The number of entries (rows) in the table for a device effectively showed how many fingers were enrolled.

Unlock Speed

Speed metrics

Besides accuracy, unlock speed influences user perception of how good your fingerprint sensor solution works. A blazingly fast unlock experience feels magical whereas a slow one becomes quickly annoying. Overall, the goal was to: (1) make the solution faster than the most popular alternative on previous generation phones - i.e. PIN unlock and (2) be faster than all competitor phones.

Unlock speed was measured from sensor wake-on-touch (WOT) to when the phone got to the home screen in idle state, where you could begin interacting with the phone. This end-to-end process was broken down into the following measurements:

1. Capture time (milliseconds) - the amount of time between WOT and the sensor capturing/cleaning an image of the fingerprint that could then be processed by the matcher.
2. Match time (milliseconds) - the amount of time required by the matcher to declare a match (or no match).
3. Capture match time (milliseconds) - simply the sum of the two above metrics, capture and match time. This represented the amount of time consumed by the fingerprint sensor vendor's software and hardware solution.
4. Homescreen time (milliseconds) - the amount of time when the fingerprint sensor system notifies the Android keyguard - lock screen components to when the phone is fully unlocked and has reached the home screen in an idle state.

Thus, the full end-to-end user experience of unlock speed encompassed the sum of 3. and 4. above.

• To get improvements in capture and match time, we worked externally with the sensor vendor to drive speed up optimizations.
• To get improvements in home screen rendering time, we worked with internal Motorola teams on how to optimize the graphics & display drivers, speeding up animation effects, and the like.

Outlier metrics

The previous set of metrics were average values. Though, the system did experience variability and it was also important to look at outliers. The following metrics were derived:

1. Capture outliers and Capture percent - the count and percentage of occurrences where the capture process exceeded 300 ms.
2. Match outliers and Match percent - the count and percentage of occurrences where the match process exceeded 100 ms.
3. Homescreen outliers and Homescreen percent - the count and percentage of occurrences where the amount of time to get to home screen idle exceeded 750 ms.

In reality, this data would have better been processed and viewed graphically as a distribution. However, it did serve as a quick check of unlock speed performance outliers which could trigger another level of investigation if required.

Below is an example of Unlock Speed and Outliers report --

Consolidated Reports

While each of the individual reports provided useful insights in their respective areas, what was extremely powerful was the ability to generate a consolidated report for a single user.

There were many times where an internal user would reach out and provide feedback on an issue with fingerprint usage. A key step in helping that user would be to run a consolidated report, where I could select their device, and if necessary, narrow down on software release and/or calendar date(s), and pull up a summary of unlock activity, unlock efficiency, poor quality touches, transaction failures, unlock speed and outliers. Seeing all the data together provided the big picture and helped confirm or refute certain claims very quickly.

BigQuery SQL

Behind all the report tables were the SQL queries to extract and package the data. As with most non-trivial queries, the process of creating generally involved iterative development and validation, to make sure the data extraction and presentation followed intentions.

Below shows all the SQL queries I wrote to process the BigQuery data into these reports --

# Unlock Activity

SELECT
  barcode,
  bldid,
  caldate,
  reject_total,
  success_total,
  (reject_total   success_total) AS unlock_attempts_total,
  ROUND(100 * (reject_total / (reject_total   success_total)), 2) AS reject_percent
FROM
(
  SELECT
    barcode,
    bldid,
    DATE(devicetime) AS caldate,
    SUM(INTEGER(JSON_EXTRACT_SCALAR(event, '$.rj_cnt'))) AS reject_total,
    COUNT(*) AS success_total
  FROM TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
  WHERE tag = "MOT_FPS" AND segid = "FPS_UNLOCK" AND JSON_EXTRACT_SCALAR(event, '$.result') = "2"
      AND ((barcode CONTAINS "NVTV4D0098"))
  GROUP BY barcode, caldate, bldid
)
ORDER BY barcode, caldate DESC

 
# Unlock Efficiency

SELECT
  barcode,
  bldid,
  reject_cnt,
  ROUND(percent_raw * 100, 2) AS percent
FROM (
  SELECT
    barcode,
    bldid,
    reject_cnt,
    RATIO_TO_REPORT(touches) OVER (PARTITION BY barcode, bldid ORDER BY bldid DESC) AS percent_raw
  FROM (
    SELECT
      barcode,
      bldid,
      IFNULL(reject_cnt, 0) AS reject_cnt,
      COUNT(reject_cnt) AS touches
    FROM (
      SELECT
        barcode,
        bldid,
        INTEGER(JSON_EXTRACT_SCALAR(event, '$.rj_cnt')) AS reject_cnt
      FROM
        TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
      WHERE tag = "MOT_FPS" AND segid = "FPS_UNLOCK" AND JSON_EXTRACT_SCALAR(event, '$.result') = "2"
        AND ((barcode CONTAINS "NVTV4D0098"))
    )
    GROUP BY barcode, bldid, reject_cnt
    ORDER BY barcode, bldid, reject_cnt ASC
  )
)
 

# Poor Quality Touches

SELECT
  barcode,
  bldid,
  caldate,
  IFNULL(INTEGER(SUM(incidental_touch_cnt)), 0) AS total_incidental,
  SUM(finger_moved_too_fast_cnt) AS total_finger_moved_too_fast
FROM (
  SELECT
    barcode,
    bldid,
    DATE(devicetime) AS caldate,
    INTEGER(JSON_EXTRACT_SCALAR(event, '$.quality')) >> 16 AS incidental_touch_cnt,
      CASE
      WHEN INTEGER(JSON_EXTRACT_SCALAR(event, '$.result')) = 5 THEN 1 ELSE 0
    END AS finger_moved_too_fast_cnt
  FROM
    TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
  WHERE
    tag = "MOT_FPS" AND segid = "FPS_UNLOCK"
    AND ((barcode CONTAINS "NVTV4D0098"))
)
GROUP BY barcode, caldate, bldid
ORDER BY barcode, caldate DESC


# Transactional Failures

SELECT
  barcode,
  bldid,
  caldate,
  COUNT(reject_count) AS count_tf
FROM (
  SELECT
    barcode,
    bldid,
    DATE(devicetime) AS caldate,
    INTEGER(JSON_EXTRACT_SCALAR(event, '$.rj_cnt')) AS reject_count
  FROM
    TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
  WHERE
    tag = "MOT_FPS" AND segid = "FPS_UNLOCK" AND INTEGER(JSON_EXTRACT_SCALAR(event, '$.rj_cnt')) = 5
    AND ((barcode CONTAINS "NVTV4D0098"))
)
GROUP BY barcode, caldate, bldid
ORDER BY barcode, caldate DESC


# Finger Enrollments

SELECT
  barcode,
  bldfngrprnt,
  devicetime AS tstamp,
  DATEDIFF(CURRENT_DATE(), DATE(devicetime)) AS num_of_days
FROM
  TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
WHERE
  tag = "MOT_FPS" AND segid = "FPS_ENROLL"
  AND ((barcode CONTAINS "NVTV4D0098"))
ORDER BY barcode, tstamp DESC
 

# Unlock Speed

SELECT
  barcode,
  bldid,
  ROUND(AVG(capture_time_ms), 2) AS avg_capture_time_ms,
  ROUND(AVG(match_time_ms), 2) AS avg_match_time_ms,
  ROUND(AVG(capture_time_ms   match_time_ms), 2) AS avg_capture_match_time_ms,
  ROUND(AVG(homescreen_time_ms), 2) AS avg_homescreen_time_ms
FROM (
  SELECT
    barcode,
    bldid,
    INTEGER(JSON_EXTRACT_SCALAR(event, '$.cap_time')) AS capture_time_ms,
    INTEGER(JSON_EXTRACT_SCALAR(event, '$.mt_time')) AS match_time_ms,
    INTEGER(JSON_EXTRACT_SCALAR(event, '$.time2hm')) AS homescreen_time_ms,
  FROM TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
  WHERE tag = "MOT_FPS" AND segid = "FPS_UNLOCK"
    AND ((barcode CONTAINS "NVTV4D0098"))
)
GROUP BY barcode, bldid
ORDER BY barcode, bldid DESC
 

# Unlock Speed Outliers

SELECT
  barcode,
  bldid,
  capture_outliers,
  ROUND((capture_outliers / capture_total) * 100, 2) AS capture_percent,
  match_outliers,
  ROUND((match_outliers / match_total) * 100, 2) AS match_percent,
  homescreen_outliers,
  ROUND((homescreen_outliers / homescreen_total) * 100, 2) AS homescreen_percent
FROM (
  SELECT
    barcode,
    bldid,
    COUNT(capture_time_ms) AS capture_total,
    COUNT(match_time_ms) AS match_total,
    COUNT(homescreen_time_ms) AS homescreen_total,
    SUM(CASE WHEN capture_time_ms > 300 THEN 1 ELSE 0 END) AS capture_outliers,
    SUM(CASE WHEN match_time_ms > 100 THEN 1 ELSE 0 END) AS match_outliers, 
    SUM(CASE WHEN homescreen_time_ms > 750 THEN 1 ELSE 0 END) AS homescreen_outliers
  FROM (
    SELECT
      barcode,
      bldid,
      INTEGER(JSON_EXTRACT_SCALAR(event, '$.cap_time')) AS capture_time_ms,
      INTEGER(JSON_EXTRACT_SCALAR(event, '$.mt_time')) AS match_time_ms,
      INTEGER(JSON_EXTRACT_SCALAR(event, '$.time2hm')) AS homescreen_time_ms,
    FROM TABLE_QUERY([mot-foundations:d_devicestats], 'table_id BETWEEN "201606195000" AND "201607180000"')
    WHERE tag = "MOT_FPS" AND segid = "FPS_UNLOCK"
      AND ((barcode CONTAINS "NVTV4D0098"))
  )
  GROUP BY barcode, bldid
  ORDER BY barcode, bldid DESC

)

Coming up next

In the next article, I'll go over User trial observations. Because fingerprint sensor use is tied to humans, it was important to leverage direct observations, while generating measureable insights via data collected in the process.