3D Feature Extraction of 3D Human Facials Scans

(1)

3D Feature Extraction of 3D Human Facials Scans

Aslak Søgaard - s093112 & Søren Bohøj - s093118

Kongens Lyngby 2012 IMM-BSC-2012-0020

(2)

Building 321, DK-2800 Kongens Lyngby, Denmark Phone +45 45253351, Fax +45 45882673

reception@imm.dtu.dk www.imm.dtu.dk

(3)

Abstract

Studies indicate a correlation between mental illnesses and the human genome.

If mental illnesses can be determined by the genome, it is possible that these gene abnormalities could result in alterations of the facial morphology, i.e. cause birth defects.

This thesis presents a database of 3D facial scans, which has been annotated manually in respect to a 73-landmark scheme. The extracted mouth features has been aligned by the generalized Procrustes analysis and statistically analysed by the principal component analysis. In the principal component space, the K-means clustering algorithm is used to divide the population into groups of similar characteristic mouth features.

The results from the clustering showed a differentiation of the population based upon the mouth features. The method described in this thesis hereby presents a possibility of dividing people in groups based upon their shared characteristic mouth features.

Keywords: Principal Component Analysis, Generalized Procrustes Analysis, K-means Clustering, 3D Facial Scans, 3D Feature Extraction, 73-Landmark Scheme.

(4)

(5)

Resum´ e

Undersøgelser viser en sammenhæng imellem psykiske sygdomme og det men- neskelige genom. Hvis psykiske sygdomme kan være bestemt af genomet, er der mulighed for at gen-abnormaliteter kan resultere i ændringer af ansigtets morfologi, dvs. foresage medfødte defekter.

Denne afhandling præsenterer en database af 3D ansigts-scanninger, som er blevet annoteret manuelt i forhold til en 73-punkts protokol. Udtrækning af mundens karakteristika er blevet bragt i samme refererings ramme af den gen- eraliserede Procrustes analyse og analyseret statistisk ved principal komponent analyse. I rummet udgjort af principal komponenterne, er K-means clustering algoritmen anvendt til at opdele populationen i grupper med similære mund karakteristika.

Resultaterne fra denne gruppering viste en differentiering af befolkningen baseret p˚a mundens karakteristika. Metoden beskrevet i denne afhandling præsenterer dermed en mulighed for at opdele mennesker i grupper baseret p˚a deres fælles mund karakteristika.

Nøgleord: Principal Komponent Analyse, Generaliseret Procrustes Analyse, K-means Clustering, 3D Ansigts-Scanninger, 3D Karakteristika Udtrækning, 73-Punkts Protokol.

(6)

(7)

Preface

This thesis is prepared at the Section of Image Analysis, at the Department of Informatics and Mathematical Modelling, IMM, located at the Technical University of Denmark, DTU, as partial fulfilment of the requirements for the acquiring of the degree Bachelor of Science in Engineering, B.Sc.Eng.

The thesis deals with statistical analysis of the 3D data and the possibility of clustering a given population of peoples 3D facial scans. There will be used GPA, PCA and the K-means clustering.

It is assumed that the reader has a basic knowledge within the field of linear algebra, statistics and image analysis.

Lyngby, June 2012

Aslak Søgaard & Søren Bohøj

[Aslak email: asse.116@hotmail.com]

[Søren email: soeren@bohoej.dk]

(8)

(9)

Acknowledgements

We would like to thank the following people for the assistance and support of making this thesis.

First and foremost, we would like to thank our supervisor Rasmus R. Paulsen for his support throughout this thesis. It has been a great experience to be a part of the process in the analysis of images. We would also like to thank for his regularly proof-reading during the writing of this thesis.

We thank our co-supervisor Jens Fagertun for his support throughout this thesis and always having time to spare. He has been a great help during the process of making the MATLAB-scripts used in the data analysis and by proof-reading a great part of the manuscript of this thesis.

We thank Thomas Hansen from Psykiatrisk Center Sct. Hans, Retspsykiatrisk afd. R, for the opportunity of collecting our own data samples and for ideas for future work.

We would like to thank the group of image project group, with Rasmus R.

Paulsen as supervisor, for their consultation when obstacles have occurred. We also thank them for the time spend on answering questions and partitioning in productive conversations, which has been of great help.

(10)

(11)

Chapter 1

Introduction

Diagnosis of mental illnesses and syndromes is a challenging area for physicians and a task only suitable for the experienced physicians. Experienced physicians are able to diagnose patients almost merely by identifying facial characteristics associated with specific dysmorphic¹ syndromes [5]. These facial dysmorphologies along with personal interaction with the patient are used in diagnosing the mental disorders. It takes a lot of practise to master these skills but unfortu- nately, due to the limited exposure to these kinds of disorders during training, the techniques are learned on the job instead. It would be of great help if inexpe- rienced physicians could acquire these skills during their training. Fortunately, 3 dimensional (3D) facial scans could be the solution to this problem.

The facial characteristics of a person is filled with information. Take for instance Fig. 1.1, where four mouths, which all differ more or less from each other, are shown. As one can see, a lot of geometric information is available within the features of the mouth, information which can be captured by 3D facial scans. With focus only on the mouth, we intend to explore the possibilities for dividing a population into groups. By this we hope that the people in each group share some sort of mouth characteristics, based upon 3D feature extraction. Feature extraction is used in various fields and studies, as in [3], where feature extraction is used in the field of facial recognition. In order to work with large databases in facial recognition, it is important that it is automatic.

1Abnormal structural features or birth defects

(16)

A way of doing so is to make an algorithm which automatically sets predefined anatomical correct placed landmarks on the facial scans. This is perfected by

”teaching” the algorithm where to annotate these landmarks by performing a lot of manual annotation. During the process of this thesis, a great deal of time has been used to do just that. By this, we have been contributing with valuable data to the studies regarding [3]. In addition, a thorough description on how to annotate correctly has been made in order to ensure consensus when annotation is performed by different people. This annotation guide can be found in appendix A.

Figure 1.1: Four different mouths show the diversity among people.

There are studies indicating a linkage between mental illnesses and the human genome. The field of biological psychiatry has been around since the late 1980s and is a field of medicine. It is an approach to psychiatry in the pursuit of understanding mental disorders in terms of the biological functions of the ner- vous system [6]. In the late 1980s, research groups discovered genes which were associated with the mental disorders manic depression and schizophrenia [10].

These studies are, how ever, complicated by several factors. One example, is the fact that psychiatric diagnosis relies on the opinion of the physician based on the interaction with the patient. Several known disorders are characterised by facial dysmorphologies, where Williams syndrome and Noonan syndrome are two examples of such [5].

We will during this thesis extract 3D facial features from 3D facial scans and process the data using various mathematical tools, such as generalized Pro- crustes analysis (GPA) and principal component analysis (PCA). In the end we will perform a cluster analysis, using the K-means clustering method, to divide the dataset in to a number of groups and describe the results so that it is humanly understandable. Through analyses of the extracted facial features we wish to determine whether this new knowledge, gained from the clustering, would be helpful in applications like the ones already mentioned and others to be discussed.

(17)

1.1 Mathematical Notations 3

1.1 Mathematical Notations

The reader is expected to be familiar with linear algebra. The mathematical notations are listed below and will be shown throughout this thesis as follows:

Scalars are represented by a lower-case Latin or Greek letter.

x (1.1)

Vectors are denoted with lower-case non-italic Latin or Greek letters. Column vectors are used in this thesis.

x = [x₁,x₂,· · · ,x_n]^T (1.2)

In this thesis, matrices are denoted with capital non-italic Latin or Greek letters.

X =

a d c b

(1.3)

Objects which occur in groups are presented in vectors with curly braces.

{a, d, c, b} (1.4)

The indexing of a matrix is represented with a row-column subscript of a scalar or a vector.

Axy = Ax, x = [x,y] (1.5)

Of a specific dataset, the mean vector is denoted with lower-case non-italic Latin or Greek letters with an overhead bar.

x (1.6)

(18)

(19)

Chapter 2

Previous Work

3D facial analysis is an area with great potential as a tool in the process of diagnosing a mental illness. In [5],

”...the potential of computer-based models of 3D facial morphology to assist in dysmorphology training...”

Hammond, P. [4]

is described. Average faces of groups of children with various syndromes (Noo- nan, William, velocardiofacial and fragile X) and of a control group with no known genetic disorder are being used in demonstrating dysmorphology. Even though the efficacy of this model, in helping trainees recognising dysmorphic syndromes, has yet to be tested it surely shows potential. For an overview of [5], see [4].

There are many tools used within the area of statistical analysis. One, is the principal component analysis (PCA) which was derived by Pearson in 1902 [8].

PCA was developed as a basic statistical tool used to transform data and was also used in [4].

PCA can be applied on all sorts of data such as geographical or image data. In Daultry [1], a definition of PCA is presented and applied, as an example, on a

(20)

geographical set of data. PCA is basically a rotation of axes in a coordinate system and is sometimes useful for looking at data from another perspective.

Think for example about the difference between looking at an object at a normal perspective and looking at it from a birds perspective. From the birds perspective we tend to get a more overall perception of the geographical features of the object. In this example we rotate the given axes and that is basically the concept of PCA.

Another tool, which we will use before performing PCA on our data, is the generalized Procrustes analysis (GPA). In Greek mythology, Procrustes was an innkeeper with an iron bed in which he invited passers to sleep in. The passers never fitted the bed so Procrustes would either stretch or cut off their limbs in order for them to do so. In shape analysis, it is easiest to compare multiple shapes of objects when they are in the same frame of reference. So just like the bandit Procrustes, we sometimes want to ”stretch” our data in order to make a proper comparison of two or more shapes of objects. GPA is mostly used for aligning the shapes by translating, scaling and rotating them into a chosen mean shape. This is essentially what GPA does. GPA can be used to compare all sorts of things, whether it is shapes of objects or peoples personal opinion of which ice cream they like the most, GPA can align the data. One shape or one persons opinion in an entire survey is therefore referred to as one configuration.

GPA is a developed version of the original Procustes analysis (PA) and first introduced by Kristof and Wingersky in 1971 and later popularized by Gower in 1975 [11]. The original PA was first introduced by Green in 1952 and Hurley and Cattel in 1962 and is not applicable to more than two configurations [11].

GPA, on the other hand, is applicable to as many configurations as desired.

This makes it possible for us to compare our data in the best possible way by first aligning the data with GPA and then reducing the number of dimensions with PCA.

After preprocessing our data with GPA and PCA, we want to do a cluster analysis of the data. By this, our intention is to get an idea of how unique the mouth features makes us as individuals. A cluster analysis is basically a method to divide a dataset into smaller groups of data points, where the points in each group is more similar to the other points within the same group than with any other point in the set. By this logic, if we get as many groups as we have got points then every point is unique because each point is not similar to any other point in the set. On the other hand, if we only get a very small number of groups then all the points are somehow similar to each other and are nowhere near unique. This tells us that the greater the number of groups we get, the more the mouth features of a person will differentiate that person from other people and vice versa. There exists numerous of different methods of clustering but one of these which we will focus on is the K-means clustering method. The K-means clustering method dates back to James MacQueen in 1967, although

(21)

7

the standard algorithm first was described by Stuart Lloyd in 1957 [7]. Basically, it is a method for dividing data points intokclusters (groups), wherekis chosen by the user to be any integer, as long as kis lesser than the number of points in the dataset. By randomly choosing k ”means”¹ amongst a dataset, it is possible to assign each point in the set to the nearest ”mean” and thereby divide the points into kclusters. The current estimated ”means” is constantly repositioned so that we in the end get the best possible fit.

A more comprehensive description of the K-means algorithm can be found in the Chapter 3.

1Randomly picked points assigned as a mean value.

(22)

(23)

Chapter 3

Theory

In this chapter we will go through some mathematical operations which have been used in this thesis. These operations are not general knowledge for students at our own level which is why we will include a short introduction to them here.

In the end of this chapter, we will go through an example where we use the described operations on a small set of data.

3.1 GPA

As mentioned earlier, GPA can be used to align data in such a way that the comparison of multiple configurations has the best possible basis. In this thesis we will only use GPA to align the multiple shapes of objects that is our data.

When performing GPA, there is a number of steps to be followed and they are as follows:

1. Define a shape of reference.

2. Align the remaining shapes to the reference shape.

3. Rescale the new mean shape of the aligned shapes.

(24)

4. Compare the calculated mean shape with the reference mean. If they are different continue back to step 2. Otherwise the true mean shape has been found.

If the data consists of a certain number of shapes, the first thing to do is to define a shape of reference. This can be done easily by simple choosing the first shape in the set as the reference shape. Move all shapes to a common centre (i.e. possibly the origin). Now the shapes are being scaled and rotated to fit the reference shape best possible. Afterwards a new mean is calculated from the aligned shapes and compared to the reference mean. If the calculated mean has changed more than a certain threshold, then step 2-4 is repeated.

3.2 PCA

PCA is a linear transformation which can be used in dimension reduction of a specific dataset. It rotates the axes of the original coordinate system so the new axes will be defined by the variance in the data. The first axis will be parallel to the greatest variance of the data, the second axis will be parallel to the second greatest, and so forth. All the new axes is orthogonal to each other and therefore uncorrelated. PCA is very difficult to illustrate in dimensions greater than 3. In Fig. 3.1 you can see an example of a PCA of a 2-dimensional dataset. From the PCA, we get a set of values, or scores, which are the values of the data points after the transformation. The number of values assigned to each score in the new coordinate system is the same as the number of principal components.

Figure 3.1: PCA of a 2-dimensional dataset depicted with the two new rotated axes PCA1 and PCA2 [3].

(25)

3.2 PCA 11

3.2.1 Mathematical Approach to PCA

Given a dataset

X = [x1,x2, ...,xn], (3.1) where xn is a column vector representing each patient. This can also be written as

X =







x1,1 x1,2 · · · x1,n

x2,1 x2,2 · · · x2,n

... ... . .. ... xd,1 xd,2 · · · xd,n







, (3.2)

where nis the number of samples and the number of variables determines the dimensiond.

The mean of X is used and represented by Mean{X}= x =

Pn i=1x1,i

n ,

Pn i=1x2,i

n , ..., Pn

i=1xd,i

n

, (3.3)

where x is a column vector consisting of the mean of each variable. ˆX is the existing dataset with a new origin and is calculated by subtracting x from X

X = [xˆ 1−x,x2−x,· · ·,xn−x], (3.4) which again can be written as

X =ˆ







x1,1−x1 x1,2−x1 · · · x1,n−x1

x2,1−x2 x2,2−x2 · · · x2,n−x2

... ... . .. ... xd,1−xd xd,2−xd · · · xd,n−xd







(3.5)

The new origin is where the principal components will intersect and define the new coordinate system. The new situation is shown in Fig. 3.2.

(26)

Figure 3.2: A new origin is defined by a PCA of a 2-dimensional dataset.

The covariance matrix ΣXˆ is now calculated by ΣXˆ = 1

n

X ˆˆX^T, (3.6)

which is the general method. If the number of dimensions exceeds the number of samples it is more economical to calculate the covariance matrix as follows:

ΣXˆ = 1 n

Xˆ^TXˆ (3.7)

This method is a better choice when a dimension reduction is needed because the result would be an×nmatrix instead of ad×dmatrix.

The Eigenvectors Φ_X_ˆ of the covariance matrix represents the new axes which are called the principal components.

ΣXˆΦXˆ = ΦXˆΛXˆ (3.8)

Here ΛXˆ is a diagonal matrix with Eigenvaluesλdin the diagonal. There will be min(d, n) non zero Eigenvalues which will be arranged in either a descending or an ascending order. In this thesis a descending order is used. λ1is the greatest Eigenvalue and therefore represents the greatest variance i.e. the first principal component.

Λ_X_ˆ =







λ₁ 0 · · · 0 0 λ₂ · · · 0 ... ... . .. ... 0 0 · · · λ_d







(3.9)

(27)

3.3 K-means Clustering 13

The sum of all Eigenvalues will be equal to min(d, n).

min(d, n) =

d

X

i=1

λi=T r{Λ} (3.10)

The corresponding Eigenvectors Φ_X_ˆ to Λ_X_ˆ is therefore arranged in the same order withφ₁ as the first Eigenvector corresponding to the highest Eigenvalue λ₁.

ΦXˆ = [φ1, φ2,· · ·, φd] (3.11) For more details, on the mathematical approach to PCA, see [1].

3.3 K-means Clustering

K-means clustering is, as mentioned, an iterative method used to divide a number of data points intokclusters, wherekis chosen by the user. The algorithm chooses k mean values randomly amongst the points and each point is then appointed to the nearest mean. All points associated with a mean value j¹, is then called clusterj. Whenk clusters have been chosen, the current estimated mean value is then repositioned so that the sum of squared distances from all the points in a particular cluster is minimized, i.e. the centroid² of that particular cluster is appointed the new estimated mean value. Now, all the points are again appointed to the nearest of the new estimated mean values and a new set of clusters are created. This goes on until no further changes of the estimated means occur and can be described as the four steps of the K-means algorithm as follows:

1. kmean values is chosen randomly amongst the points.

2. Each data point is appointed to the nearest mean value andkclusters are created.

3. The centroids of thesekclusters becomes the new estimated mean value.

4. Step 2 and 3 are repeated until no further changes occur.

1j= 1,· · ·, k

2Center of gravity of a given clusterj.

(28)

When no further changes occur, it means that convergence of the mean values has been reached and the output will be the best possible grouping of the data points with that particular value ofk. An issue of importance to be considered is of course the choice ofk, which if chosen badly, will result in a bad definition of the clusters. In the end it is up to the user to choose ak which can always be re-evaluated if it has been found inaccurate.

3.4 Example

Through this example we will show how GPA, PCA and the K-means clustering will be used in this thesis. The data used in this example is 19 images taken of Aslak which will be sufficient in order to show the usage of these three methods.

We will show the impact of GPA on both the entire facial features and the mouth features, but only perform PCA and K-means clustering on the mouth features. We start with the raw data, i.e. the 19 digital images. The features of the 19 faces and 19 mouths extracted from the digital images can be seen before GPA in Fig. 3.3 and Fig. 3.4, respectively.

Figure 3.3: 19 faces from Aslak before GPA.

(29)

3.4 Example 15

Figure 3.4: 19 mouths from Aslak before GPA.

It is easy to see the facial characteristics in Fig. 3.3, but with different people and more images it would undoubtedly be more difficult to see. In the following, the effects of GPA is shown on the faces and the mouths in Fig. 3.5 and Fig.

3.6, respectively. See that the points are aligned and become more comparable.

Figure 3.5: 19 faces from Aslak after GPA.

(30)

Figure 3.6: 19 mouths from Aslak after GPA, were the mean mouth shape is marked by the red line.

Now we want to perform PCA of the aligned data. By the use of MATLABs own function princomp, we can see the different principal components which account for the total variance in the data set. The first three principal components are shown in a 3D space in Fig. 3.7 and the two first principal components are shown in a plane in Fig. 3.8, respectively. Please note that in order to visualize this correctly, one would have to see it in as many dimensions as there are principal components.

Figure 3.7: The first three principal components shown in a 3D space.

(31)

3.4 Example 17

Figure 3.8: The first two principal components shown in a plane.

Another way of displaying the importance of each principal component can be seen in Fig. 3.9. This shows that the six first principal components accounts for over 95% of the total variance in the data set.

Figure 3.9: The first six principal components and how much variance they account for.

In Fig. 3.10, the PCA scores corresponding to a certain mouth is plotted with their respective shapes. We see the mouth shapes in a PCA space and only with the first two principal components as axis. This includes only 19 pictures of one person but with varying facial expressions, which explains the outliers in Fig.

3.10.

In an example like this, it is possible to do some sort of visual clustering but with a greater number of data points it becomes very difficult and highly unlikely to

(32)

Figure 3.10: The PCA scores and corresponding mouth shapes shown in a PCA plane.

do better than the computer. For example, one could say that 5 clusters should suffice and, by visual clustering, be created like in Fig. 3.11.

Figure 3.11: A visual clustering with 5 clusters.

Now, we can use the computer to do a K-means clustering on the same scores as before and the result can be seen in Fig. 3.12. Here we see that the computer chooses the clusters differently.

With a greater number of points, one could say that the choice ofkis very difficult to make. This is one problematic issue regarding the K-means clustering.

But, after a K-means clustering one can always look and decide whether or not it looks realistic. If not, a newkcan be chosen and the analysis can be resumed.

(33)

3.4 Example 19

Figure 3.12: A K-means clustering with 5 clusters. Here the centroids are represented byN

.

Or as we intend, one can do the K-means clustering with several numbers ofk and afterwards choose the most appropriate clustering.

(34)

(35)

Chapter 4

Materials, Set-Up and Method

Through this chapter we will provide the reader with information regarding the whole set-up of the equipment used in the process of collecting the data. All data has been collected at Glostrup Blodbank located at Glostrup Hospital.

The facial imaging system is made by Candf ield and the model is VECTRA M3. In addition to the data which we received from Glostrup Blodbank, we took pictures of our selves and they can be seen in appendix B.

4.1 Hardware and Software

The hardware used in the VECTRA M3 system is actually rather simple equipment. Six high resolution cameras are used to shoot three different pictures at three different angles simultaneously, thus capturing the depth of an object and making the visualization of that object in 3D possible. The six cameras are positioned on a tripod in pairs of two, so that two cameras are shooting head on in a slightly downwards direction and the other two pairs are shooting slightly from either side in an upwards direction. A picture of the set-up can be seen in Fig. 4.1, although it should be noted that the computer screen and table is not a part of the originally set up from Candf ield. The software used is the

(36)

standard MIRROR software installed by Candf ield from the beginning. The user only has to take the pictures and the software completes the 3D rendering by itself. If the reader is more interested in the specifics of these products see [9].

(37)

4.1 Hardware and Software 23

Figure 4.1: TheCandfield VECTRA M3 set-up.

(38)

4.2 Procedure

The patient is positioned on a stool and is faced directly towards the centre camera pair in such a way that the patient is able to look him/her self in the eyes in a mirror placed below the centre camera pair as can be seen in Fig. 4.1.

In this way the patient is at the right height in order for the 3D capturing to work. The person controlling the imaging software is in charge of taken the pictures and is able to see a preview of the three pictures at the same time prior to taking them. On the preview screen, there are lines displayed which are used to position the patient correctly. In the following, you can see the three images taken by each of the three camera pairs in Fig. 4.2 - 4.4 and the finished 3D reconstructed image in Fig. 4.5.

Figure 4.2: Picture taken from the front.

(39)

4.2 Procedure 25

Figure 4.3: Picture taken from the right side.

(40)

Figure 4.4: Picture taken from the left side.

(41)

4.2 Procedure 27

Figure 4.5: The 3D reconstructed image made from the three pictures.

(42)

4.3 Data Processing

After the data has been stored as 3D facial scans it is time to process them.

This was done in several steps as part of the analysis that we wanted to do. We were given a rather large dataset consisting of 620 facial scans from Glostrup Blodbank. All of these, including the images we obtained of our selves, were annotated by us according to the 73-landmark scheme. Information of the scheme and the method regarding correct annotation is presented in appendix A. The overall process can be described in a number of steps as follows:

1. Loading the obtained data into MATLAB.

2. Extracting the facial features as points by the 73-landmark scheme.

3. The points defining the mouth features are placed in a matrix and ready for the GPA algorithm.

4. After the GPA, the mean mouth is subtracted from the GPA matrix.

5. When the mean is subtracted, the data is ready for a PCA.

6. The PCA returns the Eigenvalues and Eigenvectors of the dataset and the scores are shown in a scatter plot, which presents the opportunity for an statistical analysis. This means that we are able to see how much each mouth vary from one another.

7. From this scatter plot a K-means clustering is performed with various values ofk.

The list of source-code files can be found in appendix D.

(43)

Chapter 5

Data

In the following, a dataset consisting of a total 622 facial scans is presented.

One facial scan each of Søren and Aslak is presented in Fig. 5.1 and Fig. 5.2, respectively, and all scans can be found in appendix B. In addition to these 2 facial scans, a set of 620 facial scans from Glostrup Blodbank is being used in this thesis but are not presented due to confidentiality reasons.

Figure 5.1: The third facial scan of Søren taken at Glostrup Hospital.

(44)

Figure 5.2: The third facial scan of Aslak taken at Glostrup Hospital.

5.1 Raw Data

In this section the raw data is plotted as can be seen in Fig. 5.3 and Fig. 5.4, where the raw landmarks are shown for the face and mouth, respectively. The faces and the extracted mouths are clearly not aligned.

Figure 5.3: The 622 facial scans as raw data equal to a number of 622×73 points.

(45)

5.2 GPA 31

Figure 5.4: The 622 extracted mouth shapes as raw data.

5.2 GPA

In this section the results from performing a GPA on the data from Fig. 5.3 and Fig. 5.4 is displayed. The effect is clearly visual as can be seen in Fig. 5.5 and Fig. 5.6, which show the faces and mouths, respectively.

Figure 5.5: The result of performing GPA on the data.

(46)

Figure 5.6: The result of performing GPA on the extracted mouth data.

5.3 PCA

In this section the results from performing PCA of the data are shown. In Fig.

5.7¹, the mean mouth shape and the mean mouth shape ±σ along the first principal component (P C1) are plotted.²

Figure 5.7: M ean=Red,M ean−σ₁·P C₁=Greenand M ean+σ₁·P C₁= Blue.

The PCA scores, which are the coordinates for the mouths in the PCA space,

1σ1 is the standard deviation alongP C1.

2σis the standard deviation.

(47)

5.3 PCA 33

are shown in a two dimensional scatter plot including the corresponding mouth shape in Fig. 5.8.

Figure 5.8: The 622 PCA scores including their corresponding mouth shapes in a PCA plane.

(48)

5.3.1 Principal Component Variance

The total variance explained by the principal components are shown in Fig. 5.9.

This helps in the process of determining the number of principal components, which should be taken in to account, when the data analysis is performed. We choose only to use the first three principal components and the total variance explained by them is 59.43 %.

Figure 5.9: The variance explained by the Principal components in %.

(49)

5.4 K-means Clustering

The K-means clustering method is used to divide the data in groups of 3, 6, 9, 12, 15 and 18. Only the scenarios where k= 3 andk= 6 are shown here. The other scenarios can be found in appendix C. We have chosen these values because we believe the data is then best described so it is humanly understandable and this will be discussed further later on.

For every value of k we getk clusters each represented by k centroids. These centroids are each translated into a mouth shape according to their position in the PCA space. Fork= 3, the K-means clustering and the centroids from the clustering are shown in Fig. 5.10 and Fig. 5.11, respectively. The same is shown fork= 6 in Fig. 5.12 and Fig. 5.13. In Fig. 5.14, three randomly picked mouth shapes from cluster 1 are displayed. The randomly picked mouth shapes from clusters, where k = 3 and k= 6, can be found in appendix E. Every point in the scatter plot represents a mouth shape consisting of ten points, each with three coordinates, which defines the circumference of the mouth. We only see the mouth shapes in 2 dimensions(2D) but keep in mind that they are in fact 3D shapes.

(50)

Figure 5.10: K-means clustering, withk= 3, performed on the 622 facial scans.

The centroids are marked byN .

Figure 5.11: The 3 centroids from Fig. 5.10.

(51)

Figure 5.12: K-means clustering, withk= 6, performed on the 622 facial scans.

The centroids are marked by N .

Figure 5.13: The 6 centroids from Fig. 5.12.

(52)

Figure 5.14: Three randomly picked mouth shapes from the cluster represented by the unnatural centroid.

(53)

5.4.1 Centroid Classification

In order to appoint a mouth shape to a certain cluster, one calculates the eu- clidean distance to all centroids and the closest centroid is the one to which a mouth is appointed. In Fig. 5.15 - 5.18, the centroid closest to the mouth shapes are shown for Søren and Aslak. The two mouth shapes are extracted from the scans shown in Fig. 5.1 and Fig. 5.2.

Figure 5.15: Sørens mouth shape and the centroid closest to him, whenk= 3.

Figure 5.16: Aslaks mouth shape and the centroid closest to him, whenk= 3.

(54)

Figure 5.17: Sørens mouth shape and the centroid closest to him, whenk= 6.

Figure 5.18: Aslaks mouth shape and the centroid closest to him, whenk= 6.

(55)

Chapter 6

Discussion

In the following we will comment and discuss the results of our analyses and mainly focus on the results from the K-means clustering analysis.

6.1 K-means Clustering

From the PCA we see that the total amount of variance explained by the first three components is almost 60 %. Of course, this is just a consequence of our choice of numbers of principal components. In addition to this, we only display the clusters in two dimensions because it appears messy and non intuitive when showing a 3D space on a 2D paper. In 2D it becomes much simpler and as one can see the clusters are well defined as long as one keep in mind that the scatter plot is a 3D point cloud projected into the plane. Due to this, there will be times where points from different clusters are mixed together, but in reality (3D) they are clearly in different cluster.

As mentioned earlier, all points in the scatter plot represent a specific 3D mouth shape. This means that we do not see the third dimension (z) of the mouth shapes. Therefore, we cannot get a clear visualization of the relative position of the ten points to each other. Here, the ten points are the ones defining each mouth shape.

(56)

With this clustering method, as mentioned earlier, we plan to determine whether we are able to interpret our data and describe it in such a way that it is humanly understandable. It is a question of finding the optimum number of clusters, i.e.

the value ofk, by which we get a clear differentiation of a population. By this we mean, dividing the population into well defined groups where each group is somehow different from all other groups in respect to the mouth features and are represented by a common shape, i.e. the centroids. Keep in mind, the centroids representing a cluster are synthetic created shapes and are defined by the point in space to which they are assigned. They are placed at the point clouds center of gravity which makes them the average shape from a given cluster.

Earlier we saw that with a smaller dataset and with relatively great variance, it was possible to do some sort of visual clustering in 2D. How ever, in regards to the used dataset, we do not find it possible nor logical to try and perform such a clustering.

6.1.1 Choice of k

The values of k used in this thesis are, as mentioned, 3, 6, 9, 12, 15 and 18.

The reason for focusing merely on the scenarios when k = 3,6 is that when k = 9,12,15,18, the centroids become too similar in shape. It then becomes very difficult to assign a person to a single cluster without further and more comprehensive data analysis. In addition, whenk= 9,12,15,18, it seemed that the possibility of the centroids approaching an ”illegal”¹shape got higher.

6.2 Cluster Classification

When we look at the centroids from the K-means clustering wherek= 3 in Fig.

5.11, we see that they differ most from each other by the curvature of the upper lip. The second and third centroid have a rather flat Cupids bow, where the first centroid have a very distinct Cupids bow. The second and third centroid appears to be similar in shape even though they clearly are far from each other, as can be seen on Fig. 5.10.

By looking at the K-means clustering where k = 6, we see that the centroids from cluster 2, 3 and 5 are much alike. Centroid 1, 4 and 6 stands alone, where centroid 1 looks to have the most illegal (unnatural) shape of them all. As can be seen on Fig. 5.12, the clusters 2, 3, 4, 5 and 6 seem relatively dense with the

1A shape that does not seem humanly plausible.

(57)

6.3 Reflection 43

exception of a few outliers. Cluster 1 (red), how ever, is spread out widely which makes it more likely that the centroid takes on an illegal shape. Here meaning, in a widely spread and low density cluster, there are large variance between the points. Thereby, when making a linear combination between extremity points, the result will mostly end up more extreme than for a low variance and high density cluster. Looking at the three randomly picked mouth shapes from this distinct cluster in Fig. 5.14, we, of course, see three natural shapes but the variance between them are bigger than with any of the other clusters. This can be seen by comparing the mouth shapes in appendix E. It seems as though there is some sort of relationship between the density of points in a cluster and the possibility of the centroid taking on a humanly plausible shape. The greater the variance in a cluster is, the greater the distance will get between each real shape, i.e. the centroid will most likely be placed at an illegal shape. This is a pattern which can be seen through all of the additional K-means clustering scenarios, wherek= 9,12,15,18, and can be seen in appendix C.

6.2.1 Classification of our own Mouth Shapes

As can be seen in Fig. 5.15, 5.16, 5.17 and 5.18, we see that the shape of our own mouths are in the same cluster at both values of k. Comparing the 2D shapes, one sees that the mouth of Aslak has a more round circumference with a rather flat Cupids bow and the mouth of Søren has a more edgy circumference with a more marked Cupids bow. To determine whether this classification seems plausible, we compare the shapes with each centroid of each cluster. Here, we see that the choice of centroid withk= 3 is a plausible classification. Even though it is difficult to distinguish between centroid 2 and 3, we find it reasonable that our mouth shapes lies within cluster 2. By looking at the three randomly picked mouth shapes from each cluster in appendix E, Fig. E.1, we see the same characteristics as we do in Fig. 5.15 and Fig. 5.16. Doing the same fork= 6, we see that there seems to be other centroids which appears to be more appropriate choices based on the 2D shape. Even though, by looking at the randomly picked mouth shapes in appendix E, Fig. E.2, we still find it a plausible classification due to the same reasons as before.

6.3 Reflection

In general, the majority of the mouth shapes within the clusters will share some of the same characteristics as the representing centroids. In each case there will be outliers which will not share the same characteristics as the centroids but in

(58)

general the mouth shapes will resemble the centroid in one or the other way.

The main reason for some of the mouths and centroids having the same shape, is that each point in the mouth has three coordinates. On 2D we do not see the variation in the third dimension. Because of this, we cannot differentiate between the mouth shapes degree of ”bulkiness”². Some shapes look alike in 2D but has different degrees of bulkiness, which makes it difficult to describe the clusters and differentiate them from each other. We do not get a visualization of the depth of the shapes as explained earlier in this chapter. This is a factor which is necessary to take into account and makes this clustering a bit uncertain in its definition of groups. If you removed the the third dimension from the beginning of the analysis, you would be free of this problem but of course you would lose a lot of the 3D information. If ones interest only lies within the 2D shape of the mouth this would be the way to do it, but this has not been the approach in this thesis.

Another reason for some of the shape similarities between the different groups, can be found at the periphery of the clusters. Here, shapes close to the dividing line between clusters will obviously resemble some of the shapes right across the line in a different cluster. In addition, if these periphery shapes are relatively far from their respective centroids, there will properly be an even greater similarity between the borderline shapes than between the shapes and their respective centroids. This is of course an inevitable consequence of any form of clustering within a relatively compact dataset.

Judging from Fig. 5.10, we believe that the performed clustering is fairly reasonable, because almost all points are within well defined lines of each cluster.

It is almost like having a big cake and dividing it in 3 even pieces, where each piece is a 3D point cloud of mouth shapes. Even though we do not visualize the 3D shapes, we still get a division of the population where each group is more or less different from each other. With 6 clusters, as seen in in Fig. 5.12, we do not see the same kind of structure which makes it difficult to describe the differences. Further more, the outliers have less bad influence on the clustering when k= 3, because the relationship between ”normal” points and outliers is greater than withk= 6.

Considering all of this, the variance explained by the first three principal components, could possibly be increased through a fully automated annotation algorithm. If so, this could result in a more precise and true clustering and the elimination of the humanly bias aspect of annotation.

To fully succeed in dividing the population into well defined groups and to make

2The volume and density of the lips

(59)

6.4 Future Work and Possible Applications 45

the result humanly understandable, one would have to consider the aspect of displaying the mouth shapes as a 3D surface. By this, one would properly have greater success in both explaining and visualizing the differences between the clusters.

6.4 Future Work and Possible Applications

The method described in this thesis needs further work in order for it to be a robust clustering approach. It would be interesting to examine the use of other statistical tools and/or clustering methods. Instead of PCA, independent component analysis (ICA) could be a candidate in helping to explain the dataset.

Other clustering methods could involve non centroid based clustering methods.

To increase the level of details of the mouth features, one could try and incor- porate the points which defines whether the lips are closed or open. By this, one would be able to get more detailed information from each mouth regarding the thickness of the lips. This could be of interest when looking at syndromes where the lips suffers from great dysmorphologies.

We believe that this method of clustering could be helpful within some phases of facial recognition. Imagine a data base of the entire Caucasian race as 3D facial scans, divided inkclusters, and it is wanted to match a random person with one from that database. One extracts the wanted facial features and calculates the distances to each centroid. The closest centroid then indicates which cluster to examine first and if this fails, the second closest centroid would indicate which cluster to examine next, and so on. By this, one can eliminate a great number of people before the actual facial recognition process.

In [5], they use average faces of people with specific syndromes that effects the facial morphology to train physicians in noticing these dysmorphologies. From these average faces, one could assign the 3D shape of the mouth as a form of centroid. This would, of course, require more work on our method but if the entire mouth surface within the landmarks where detailed extracted with clear lip boundaries, we see a possible usage in clustering a great number of people at the same time. By a potential clustering of large populations, one could calculate the distance from each mouth shape to the various centroids, thereby determining the risk of having one of these syndromes. This would, how ever, only apply to the syndromes where the morphology of the mouth are effected but the procedure could apply to all areas of the face.

Otherwise, this clustering method could be used in various statistical analyses of the population and could be altered to extract other features within the

(60)

73-landmark scheme.

(61)

Chapter 7

Conclusion

As scientists are investigating the correlation between mental illnesses and the human genome, the possibility of diagnosing patients using digital images in- creases. It is already known that some syndromes causes alterations in the facial morphology so the possibility of discovering new relations are very plausible.

The objectives of this thesis were to: Extract facial features from 3D facial scans, concentrating on the features of the mouth; perform statistical analyses of the dataset, such as GPA and PCA; perform a clustering analysis, thereby dividing the population into groups and describing the results so that it was humanly understandable.

Using the 73-landmark scheme, which is described in appendix A, we manually annotated the 3D facial scans provided from Glostrup Blodbank, thereby extracting the facial features. In Chapter 5, this thesis presents an overview of the results from the statistical analyses and the K-means clustering. The success of dividing the population into groups of different mouth features, i.e.

the clustering, is described and discussed in Chapter 6.

Even though the clustering of the data could have been better visualized by the use of 3D shapes, we still find the last objective satisfying. If we were to start from the beginning with the knowledge we have now, we would change the method for presenting the results. Here, we would try and present the mouth

(62)

shapes in 3D, as discussed, which would make the visual differentiation of the clusters easier and more understandable. In addition, we would look into the possibility of getting more of the variance explained, for example by working with a higher number of principal components.

The method presented in this thesis could be used to cluster other facial features, because it could easily be adapted to extract other facial features within the 73- landmark scheme. In addition, this method could be used in other applications, as discussed in Chapter 6.

In this thesis, a method which could be used as an assisting clustering tool in the process of digital detection of facial dysmorphologies¹ has been presented.

A small contribution towards the hope of a future digital diagnostic tool for mental illnesses has been made.

1As a tool for assisting the work in [5]

(63)

Chapter 8

Additional Note

During the aftermath of this thesis, we discovered that we have misled the reader about the number of principal components which were used in this thesis.

Instead of three principal components, as we have written during this report, we actually use all thirty principal components. This means that we look at the results with 100 % variance explained, instead of the 60 % we claim we do.

How ever, this does not change our conclusion other than we would not like to get more variance explained.

(64)

(65)

List of Figures

1.1 Four different mouths show the diversity among people. . . 2

3.1 PCA of a 2-dimensional dataset depicted with the two new rotated axes PCA1 and PCA2 [3]. . . 10 3.2 A new origin is defined by a PCA of a 2-dimensional dataset. . . 11 3.3 19 faces from Aslak before GPA. . . 14 3.4 19 mouths from Aslak before GPA. . . 15 3.5 19 faces from Aslak after GPA. . . 15 3.6 19 mouths from Aslak after GPA, were the mean mouth shape is

marked by the red line. . . 16 3.7 The first three principal components shown in a 3D space. . . 16 3.8 The first two principal components shown in a plane. . . 17 3.9 The first six principal components and how much variance they

account for. . . 17 3.10 The PCA scores and corresponding mouth shapes shown in a

PCA plane. . . 18

(66)

3.11 A visual clustering with 5 clusters. . . 18 3.12 A K-means clustering with 5 clusters. Here the centroids are

represented byN

. . . 19

4.1 TheCandfield VECTRA M3 set-up. . . 23 4.2 Picture taken from the front. . . 24 4.3 Picture taken from the right side. . . 25 4.4 Picture taken from the left side. . . 26 4.5 The 3D reconstructed image made from the three pictures. . . . 27

5.1 The third facial scan of Søren taken at Glostrup Hospital. . . 29 5.2 The third facial scan of Aslak taken at Glostrup Hospital. . . 30 5.3 The 622 facial scans as raw data equal to a number of 622×73

points. . . 30 5.4 The 622 extracted mouth shapes as raw data. . . 31 5.5 The result of performing GPA on the data. . . 31 5.6 The result of performing GPA on the extracted mouth data. . . . 32 5.7 M ean=Red,M ean−σ1·P C1=GreenandM ean+σ1·P C1=

Blue. . . 32 5.8 The 622 PCA scores including their corresponding mouth shapes

in a PCA plane. . . 33 5.9 The variance explained by the Principal components in %. . . 34 5.10 K-means clustering, with k = 3, performed on the 622 facial

scans. The centroids are marked byN

. . . 36 5.11 The 3 centroids from Fig. 5.10. . . 36

(67)

LIST OF FIGURES 53

5.12 K-means clustering, with k = 6, performed on the 622 facial scans. The centroids are marked byN. . . 37 5.13 The 6 centroids from Fig. 5.12. . . 37 5.14 Three randomly picked mouth shapes from the cluster repre-

sented by the unnatural centroid. . . 38 5.15 Sørens mouth shape and the centroid closest to him, whenk= 3. 39 5.16 Aslaks mouth shape and the centroid closest to him, whenk= 3. 39 5.17 Sørens mouth shape and the centroid closest to him, whenk= 6. 40 5.18 Aslaks mouth shape and the centroid closest to him, whenk= 6. 40

A.1 Eyebrows and eyes from Aslak where the numbers indicate where the individual points must be placed. . . 63 A.2 Eyebrows and eyes from Søren displayed without the indicating

numbers. . . 63 A.3 Eyes from Aslak with correct annotation. . . 64 A.4 Eyes from Søren with correct annotation. . . 65 A.5 Mouth from Aslak with correct annotation displayed with numbers. 66 A.6 Mouth from Søren with correct annotation displayed without

numbers. . . 66 A.7 Nose from Aslak with correct annotation displayed with numbers. 67 A.8 Nose from Søren with correct annotation displayed without num-

bers. . . 67 A.9 Jawline from Aslak with correct annotation displayed with numbers. 68 A.10 Jawline from Søren with correct annotation displayed without

numbers. . . 68 A.11 Aslak with correct manually annotation. . . 69

(68)

A.12 Aslak with automatic annotation. . . 70 A.13 Søren with correct manually annotation. . . 70 A.14 Søren with automatic annotation. . . 71

B.1 The 1 - 9 facial scans of Søren taken at Glostrup Hospital. . . 74 B.2 The 10 - 19 facial scans of Søren taken at Glostrup Hospital. . . 75 B.3 The 20 - 26 facial scans of Søren taken at Glostrup Hospital. . . 76 B.4 The 1 - 9 facial scans of Aslak taken at Glostrup Hospital. . . 77 B.5 The 10 - 19 facial scans of Aslak taken at Glostrup Hospital. . . 78 B.6 The 20 - 24 facial scans of Aslak taken at Glostrup Hospital. . . 79

C.1 K-means performed on the 622 facial scans, where k= 9. . . 82 C.2 Centroids from the K-means clustering, where k= 9 . . . 83 C.3 K-means performed on the 622 facial scans, where k= 12. . . 84 C.4 Centroids from the K-means clustering, where k= 12 . . . 85 C.5 K-means performed on the 622 facial scans, where k= 15. . . 86 C.6 Centroids from the K-means clustering, where k= 15 . . . 87 C.7 K-means performed on the 622 facial scans, where k= 18. . . 88 C.8 Centroids 1 - 9 from the K-means clustering, wherek= 18 . . . . 89 C.9 Centroids 10 - 18 from the K-means clustering, wherek= 18 . . 90

E.1 Randomly picked mouth shapes from the three clusters fork= 3 96 E.2 Randomly picked mouth shapes from the six clusters fork= 6 . 97

(69)

List of Tables

A.1 The facial features are defined by the 73 annotation landmarks[2]. 61

(70)

(71)

List of Symbols and Acronyms

Symbols

N Centroid from the K-means clustering Λ Matrix containing eigenvalues

λ Eigenvalue φ Eigenvector

ΦXˆ Matrix containing eigenvectors ΣXˆ Covariance Matrix

X Matrix

x Vector

X^T Transposed matrix of X Xˆ New matrix made by(X−x)

x Mean of X

σ Standard Deviation from PCA σ² Variance from PCA

d Number of dimensions

k Number of mean values in K-means

(72)

n Number of samples

x Scalar

Acronyms

2D Two Dimensional/Dimensions 3D Three Dimensional/Dimensions Fig. Figure

GPA General Procrustes Analysis ICA Independent Component Analysis OPA Ordinary Procrustes Analysis PA Procrustes Analysis

PC Principal Component

(73)

Bibliography

[1] S. Daultrey. Principal components analysis. Geo Abstracts Ltd Norwich, 1976.

[2] J. Fagertun and M. B. Stegmann. The IMM frontal face database. Technical report, Informatics and Mathematical Modelling, Technical University of Denmark, DTU, Richard Petersens Plads, Building 321, DK-2800 Kgs.

Lyngby, 2005.

[3] Jens Fagertun. Face recognition. Master’s thesis, 2005.

[4] P. Hammond et al. The use of 3d face shape modelling in dysmorphology.

Archives of disease in childhood, 92(12):1120, 2007.

[5] P. Hammond, T.J. Hutton, J.E. Allanson, L.E. Campbell, R. Hennekam, S. Holden, M.A. Patton, A. Shaw, I.K. Temple, M. Trotter, et al. 3d analysis of facial morphology. American Journal of Medical Genetics Part A, 126(4):339–348, 2004.

[6] http://en.wikipedia.org/wiki/Biological_psychiatry.

[7] http://en.wikipedia.org/wiki/K-means_clustering.

[8] http//en.wikipedia.org/wiki/Principal_component_

analysis.

[9] http://www.canfieldsci.com/imaging_systems/facial_

systems/VECTRA_M3_Imaging_System.html.

[10] http://www.minddisorders.com/Flu-Inv/

Genetic-factors-and-mental-disorders.html#b.

(74)

[11] http://www.opp.nl/uk/software/senstools-for-windows/

files/gpa-all.pdf.

(75)

Appendix A

Annotation Guide to a 73-Landmark Scheme

A.1 Annotation Scheme

In this thesis we have used the 73-landmark scheme for annotation in order to add further data to the IMM Frontal Face Database [2]. In [2], a 73-landmark scheme is presented and described. However, we will in the following guide try to give a more detailed description regarding the placement of these landmarks.

Facial Features Annotation Landmarks

Right eyebrow 1 – 8

Left eyebrow 9 – 16

Right eye 17 – 24

Left eye 25 – 32

Right eye pupil 33

Left eye pupil 34

Nose 35 – 46

Mouth 47 – 62

Jaw 63 – 73

Table A.1: The facial features are defined by the 73 annotation landmarks[2].

(76)

A.2 Annotation

It is important to understand the concept of annotation correctly in order to place the landmarks at the correct anatomical position. Therefore, it is necessary to be advised by someone to whom annotation is familiar. In that way the landmarks can be placed with minimum variance compared to the correct anatomical position. We present in the following a guide which covers all aspects and should make one able to annotate correctly.

A.2.1 Guidelines

Some areas, regarding the face, are difficult to annotate the same time over and over again. Some of the areas we have had trouble with are:

1. The Eyebrows

2. The Eyes (Pupils and Iris) 3. The Mouth

A.2.1.1 The Eyebrows

Regarding annotation of the eyebrows, it is a good idea to start by marking the two ends before surrounding the rest of the hairline. Point 5 and 13 should be placed in the ends but a lot of people have many thin hairs between the two brows. If so, try and make a small circle so the outline of the circle covers the thin hair and then place the point in the centre of that circle. At point 1 and 9, at the other end of the brow, it is easier to locate the end points. In general, try and place the points which surround the brows in such an order that point 2 and 8, 3 and 7, 4 and 6, 14 and 12, 15 and 11 and 16 and 10 are placed on the same vertical line. Try and place the points equidistant between each other as can be seen in Fig. A.1 and Fig. A.2.

(77)

A.2 Annotation 63

Figure A.1: Eyebrows and eyes from Aslak where the numbers indicate where the individual points must be placed.

Figure A.2: Eyebrows and eyes from Søren displayed without the indicating numbers.

(78)

A.2.1.2 The Eyes (Pupils and Iris)

Regarding the eyes, it is usually easier to locate the right places because of the contrast between the white eye and the skin colour. Start by adjusting the circles around the iris’ and the pupils. This makes the remaining points easier to place. Point 21 and 29 should be placed at the medial canthus of the patients right and left eye, respectively. Point 17 and 25 are also easy to place at the lateral canthus. When placing the rest of the points on the eyelid margin, try again and place them in the same order as with the eyebrows, i.e. place point 18 and 24, 19 and 23, 20 and 22, 30 and 28, 31 and 27 and 32 and 26 on the same vertical line. Make sure that all points lies on the eyelid margin and try placing them as shown on Fig. A.1 and Fig. A.2 so point 18 and 24, 20 and 22, 30 and 28 and 32 and 26 is aligned with the edge of the iris. Place point 33 and 34 in each of their pupil centre and try to align them with point 19 and 23 and 31 and 27, respectively. In Fig. A.3 and Fig. A.4 you can see close ups of the pupils and iris annotation.

Figure A.3: Eyes from Aslak with correct annotation.

(79)

A.2 Annotation 65

Figure A.4: Eyes from Søren with correct annotation.

(80)

A.2.1.3 The Mouth

The mouth is actually quite easy to annotate but in some situations it can be difficult to distinguish between the colour of the lip and the colour of the skin.

Point 47 and 53 are easy to locate and should be placed in the corners of the mouth. Point 49 to 51 marks the Cupids bow and are also fairly easy to locate.

Point 48 and 52 should be placed equidistant between point 47 and 49 and 53 and 51, respectively. Point 54 to 56 should also be placed equidistant between each other. Point 57 to 62 describes whether the mouth is closed or open. If open, place point 58 and 61 in the middle of the lower and upper edge of the lip, respectively. Point 57, 59, 60 and 62 should be placed as can be seen on Fig. A.5 and Fig. A.6.

Figure A.5: Mouth from Aslak with correct annotation displayed with numbers.

Figure A.6: Mouth from Søren with correct annotation displayed without numbers.

(81)

A.2 Annotation 67

A.2.2 Annotation of the Remaining Landmarks

A.2.2.1 The Nose

The nose is quite easy to annotate but it is difficult to describe the anatomical placements. Therefore, pay extra attention to Fig. A.7 and Fig. A.8 in this section. Point 35 and 45 should be placed on either side of the nose on the same horizontal line as the medial canthus and where the nose bone connects to the cheek bone. Point 36 and 44 should be placed as can be seen in Fig. A.7 right at the notch. Point 37 and 43 should be placed at the broadest point along the nose flip. Point 38 and 42 should also be placed at the broadest point where the nostrils connects to the face as it can be seen in Fig. A.7 and Fig. A.8. Point 39 and 41 should be placed in the middle of the nostrils and at the edge. Point 40 marks the spot where the nasal columella connects to the face and point 46 marks the tip of the nose.

Figure A.7: Nose from Aslak with correct annotation displayed with numbers.

Figure A.8: Nose from Søren with correct annotation displayed without numbers.

(82)

A.2.2.2 The Jawline

Regarding the jawline, start by placing the two outer points 63 and 73 at each of their ear flip. Then, place point 68 at the middle of the chin. In this way it is very easy to place the rest of the points along the jawline, but try again to place them equidistant between each other as can be seen in Fig. A.9 and Fig.

A.10.

Figure A.9: Jawline from Aslak with correct annotation displayed with numbers.

Figure A.10: Jawline from Søren with correct annotation displayed without numbers.

(83)

A.2 Annotation 69

A.2.2.3 Concluding Remarks

In general, make sure that all points are placed correctly, i.e. make sure that point 2 is not placed at the location of point 8 and vice versa. In the beginning it is a good idea to confirm with the figures until you feel confident enough to annotate without them. It is very important that the annotation is being done in the same way regardless of whom performs the annotation. In Fig. A.11 and Fig. A.12 you can see the manually annotation and automatically annotation of Aslak, respectively. In Fig. A.13 and Fig. A.14 you can see the manually annotation and automatically annotation of Søren, respectively.

Figure A.11: Aslak with correct manually annotation.

(84)

Figure A.12: Aslak with automatic annotation.

Figure A.13: Søren with correct manually annotation.

(85)

A.2 Annotation 71

Figure A.14: Søren with automatic annotation.

(86)

(87)

Appendix B

All Scans of Søren and Aslak

This appendix shows all of the scans that were made of Søren and Aslak at Glostrup Hospital.

(88)

(a) Søren Frontal 1 (b) Søren Frontal 2 (c) Søren Frontal 3

(d) Søren Frontal 4 (e) Søren Frontal 5 (f) Søren Frontal 6

(g) Søren Frontal 7 (h) Søren Frontal 8 (i) Søren Frontal 9

Figure B.1: The 1 - 9 facial scans of Søren taken at Glostrup Hospital.

(89)

75

(a) Søren Frontal 10 (b) Søren Frontal 12 (c) Søren Frontal 13

(d) Søren Frontal 14 (e) Søren Frontal 15 (f) Søren Frontal 16

(g) Søren Frontal 17 (h) Søren Frontal 18 (i) Søren Frontal 19

Figure B.2: The 10 - 19 facial scans of Søren taken at Glostrup Hospital.

3D Feature Extraction of 3D Human Facials Scans