Evaluation of Biometric Systems - scati.greyc.fr · Evaluation of biometric systems ... Touch...

Evaluation of Biometric

Systems

Christophe Rosenberger

2

GREYC research lab

Evaluation: a love story

Evaluation of biometric systems

Quality of biometric templates

Conclusions & perspectives

Outline

3

Le pôle TES et le sans-contact

Laboratory staff: 7 CNRS researchers 25 Full professors 11 Associate professors 56 Assistant professors 79 PhD students 17 permanent staff 30 Engineers and post-doc

Research Group in Computer science, Automatics, Image processing and Electronics of Caen

Research topics: Electronics Image processing Algorithmic Document analysis Multi-agents Robotics navigation Automatics Computer security Natural language processing Biometrics Cryptography

GREYC Lab

4

E-transactions (© E-secure Transactions Cluster)

E-payment & biometrics research unit

GREYC Lab

5

Le pôle TES et le sans-contact

Biometrics: Operational authentication that respects the privacy of users

Biometric authentication (palm veins, keystroke dynamics…) Evaluation of biometric systems (acceptability, security…) Protection of biometrics (cancelable biometrics, smartcards…)

Research topics

GREYC Keystroke Keystroke dynamics

authentication

6

Research topics

Biometric systems

Keystroke dynamics

Touch screen interaction Hand shape, palm vein

Face

Finger Knuckle Print

Fingerprint

Signature dynamics

Iris

7

GREYC research lab





Outline

8


PhD thesis: 1996-1999

« Adaptive segmentation: application to hyperspectral images »

Application: Algae detection

Questions:

How to choose the best segmentation method for a type of region ?

How to validate my results ?

9


Supervised evaluation: relevance

Use of an a priori information:

Synthetic

Easy to obtain (by construction),

Not always representative.

Expert

Dedicated to an application,

Costly.

Image

processing

algorithm

Reference

10


Image

processing

algorithm

Statistical parameters Example: PSNR, EQM…

Advantage

Easy computation

Automatic evaluation

Drawback

Difference with an expert jugment

Unsupervised evaluation: consistency

11

Which one is the

best ?

Segmentation 1

Segmentation 2

Segmentation 3

Original image


12


Paradox

Compression: PSNR known as a poor metric but still used

Segmentation : many metrics exist but not very often used

Interpretation : needs to use a benchmark (not easy), very few metrics for the evaluation of a single result

….

Still a long way to go

13

GREYC research lab





Outline

14

Context

Biometrics: high level application with well defined tasks

Multiple organizations:

• BIOAPI consortium: development a biometric API standard (1998)

• National committee 37 (AFNOR): normalisation (2004)

Three important aspects:

• Performance

• Security

• Usability

Evaluation

15

Evaluation

M. El-Abed, R. Giot, B. Hemery, J.-J. Schwartzmann, C. Rosenberger, "Towards the Security Evaluation of Biometric

Authentication Systems", International Conference on Security

Science and Technology (ICSST) 2011

16

Evaluation

General process

In order to quantify the efficiency of a biometric system, we generally use two databases:

1-Learning database: used for the enrolment of individuals (can use different captures for the model definition);

2-Testing database: used for verification or identification with captures of known individuals (impostors and genuine users).

Learning

database

Test ing

database

17

Performance evaluation

EER

FRR

FAR

FAR : False Acceptance Rate

FRR : False Rejection Rate

EER : Equal Error Rate

ROC curve: FRR vs FAR

AUC : Area under the curve

Evaluation

18

Evaluation

Data for the tests:

Real users: time consuming, operators are needed

Benchmark databases:

• http://biosecure.it-sudparis.eu/AB/

• http://www4.comp.polyu.edu.hk/~biometrics/

• http://www.cbsr.ia.ac.cn:8080/iapr_home.jsp

• http://www.nist.gov/itl/biometrics/

Large datasets allowing a relative comparison

Different formats, naming conventions

Lots of computations (scenarios, size of the database…)

Sometimes not free

19

J. Mahier, B. Hemery, M. El Abed, M. El-Allam, M. Bouhaddaoui, C. Rosenberger. « Computation EvaBio: A Tool for Performance Evaluation in Biometrics », International Journal of Automated Identification Technology (IJAIT), 2011.

Evaluation

20

M. El-Abed, P. Lacharme, and C. Rosenberger, "Security EvaBio: An Analysis Tool for the Security Evaluation of Biometric

Authentication Systems", the 5th IAPR/IEEE International

Conference on Biometrics (ICB), New Delhi, India, p. 1-6, 2012.

Evaluation

Security audit

21

Evaluation

Usability analysis

M. El Abed, R. Giot, B. Hemery, C. Rosenberger, « Evaluation of Biometric Systems : A Study of Users' Acceptance and Satisfaction »

Inderscience International Journal of Biometrics (IJBM), pages 1-27,

2011.

22

Evaluation

Update strategies analysis

R. Giot, B. Dorizzi, C. Rosenberger, "Analysis of Template Update Strategies for Keystroke Dynamics", SSCI 2011 CIBIM - 2011 IEEE

Workshop on Computational Intelligence in Biometrics and Identity

Management 2011

Performance analysis of an update strategy method for keystroke dynamics considering attacks

23

Evaluation

Evaluation of cancelable biometric systems

R. Belguechi, E. Cherrier, M. El Abed and C. Rosenberger, "Evaluation of Cancelable Biometric Systems : Application to

Finger-Knuckle-Prints", IEEE International Conference on Hand-

based Biometrics, 2011

24

GREYC research lab





Outline

25

Quality of biometric data

Objective Quantifying the quality of a biometric raw data to optimize the performance of biometric systems

Patrick Grother, Elham Tabassi, "Performance of Biometric Quality

Measures", IEEE Transactions on Pattern Analysis and Machine

Intelligence archive, Volume 29 Issue 4, April 2007

26


A generic approach

Image quality

Pattern-based quality

Multi-class SVM Quality

M. EL-Abed, B. Hemery, C. Charrier, and C. Rosenberger, "Evaluation de la qualité de données biométriques", RNTI journal,

special issue on "Qualité des Données et des Connaissances", p.

1-22, 2011.

Image quality: blind evaluation (without any reference)

Pattern-based quality: is there any interesting information in the image for the recognition ?

27


M. Saad, A. C. Bovik and C. Charrier, “A DCT statistics-based blind image quality index,” IEEE Signal Processing Letters, 2010.

Image quality

BLIINDS [2] is a NR-IQ index combining three kinds of information:

Contrast distortion (v1)

Structure distortion (v2)

Anisotropy orientation (v3 & v4)

BLIINDS is entirely based on a DCT framework

It uses local DCT patch of size 17*17 to calculate the three information

Local contrast (pooled -> v1)

Kurtosis of the non-DCT coefficients (pooled -> v2)

2 measures based on Renyi Entropy (pooled -> v3 & v4)

For each patch k

28


Pattern-based quality

The Scale-Invariant Feature Transform (SIFT)

SIFT

(X, Y) Scale Descriptor (128-

elements)

1) Number of keypoints detected from the image

2) DC coefficient of the m-by-n descriptors matrix (m = number of the keypoints detected and n = 128)

3) and 4) mean and standard deviation of scales

D. G. Lowe, “Distinctive image features from scale-invariant keypoints”. International journal of computer vision, 2004.

29


Benchmark databases

Database Individuals/samples

FACES94 152 / 20

ENSIB 100 / 40

FERET 725 / average of 11

AR 120 / 26

30


Synthetic alterations

An example of alterations for a reference image from FACES94. From

left to right, reference image then alteration level 1, 2 and 3

a) Blurring alteration

b) Gaussian noise alteration

c) Resize alteration

31


EER evolution

EE

R (

%)

1 2 3 5 6 7 8 9 4 10

Good Fair Poor Very poor

32


The predicted EER /real EER values for each quality set

Database Good Fair Poor Very poor

FACES94 0.4744

0.2936

0.6843

0.5131

1.8078

1.661

5.7983

5.044

ENSIB 10.6397

10.413

13.2912

13.4

16.5495

16.53

17.787

17.774

FERET 31.88

26

32.23

30.187

32.52

32.12

34.37

33.757

P. Grother and E. Tabassi. Performance of biometric quality measures. IEEE Transactions on Pattern Analysis and Machine

Intelligence (PAMI), p. 531-543, 2007.

Validation

A good metric should be able to predict the performance of the biometric system based on the quality category

33


M. El Abed, R. Giot, B. Hemery, C. Charrier, C. Rosenberger, "A SVM-Based Model for the evaluation of biometric sample quality" SSCI 2011 CIBIM -

2011 IEEE Workshop on Computational Intelligence in Biometrics and

Identity Management 2011.

Method Good Fair Poor Very poor

Contribution 0.869 0.828 0.797 0.626

NFIQ 0.82 0.698 0.632 0.64

Kolmogorov-Smirnov (KS) test of the genuine and impostor scores distribution

Comparison with NFIQ

Use the FVC2002 Db2 database (100 individuals, 8 samples)

Kolmogorov-Smirnov test on the intraclass and interclass distributions

34


Applications

Optimization of the enrolment process

Comparison of biometric sensors

Use quality information in soft biometrics approaches

Use quality as a weighting factor in multibiometrics

35

GREYC research lab





Outline

36

Conclusion

Evaluation of biometric systems: a major trend

• Transfer to industry

• Make the users confident in the technology

• Facilitate the work for researchers

Perspectives:

• Certification process for biometric systems ?

• “Biometrics Alliance Initiative”

37

http://www.epaymentbiometrics.ensicaen.fr/

Evaluation of Biometric Systems - scati.greyc.fr · Evaluation of biometric systems ... Touch...

Documents

Transcript of Evaluation of Biometric Systems - scati.greyc.fr · Evaluation of biometric systems ... Touch...