http://bytefish.de

Getting a Huawei E352s-5 to work with Linux

Philipp Wagner — Sat, 27 Apr 2013 16:18:00 +0200

getting the Huawei E352s-5 to work with linux

Yes, I am alive and well! This is just a quick note on how to get a Huawei E352s-5 working under Linux. Today I've upgraded my contract to a new mobile data plan and I got a Huawei stick with it, which seems to be the latest model T-Mobile sells you in Germany (and now you know I am at the monopolist). If you buy hardware without researching the internet thoroughly, you'll almost always need some effort to get it working with Linux. Especially if you aren't on the latest distribution. I am running an Ubuntu 10.10 for years and I've been quite happy with it ever since. As expected, the modem isn't recognized by Ubuntu 10.10. It's not a big deal, since Huawei modems are pretty common.

switching into modem mode

If you have inserted your SIM card, plugged in the Huawei E352s-5 stick and your distribution recognized it... you can stop reading here. But if you are on an old distribution like I am, you'll probably need to use usb_modeswitch to get it working. First of all, let's see what dmesg says after inserting the stick:

philipp@mango:~$ dmesg
...
[   48.220072] usb 2-5: new high speed USB device using ehci_hcd and address 4
[   48.362048] scsi5 : usb-storage 2-5:1.4
[   48.367660] scsi6 : usb-storage 2-5:1.5
[   49.365716] scsi 5:0:0:0: CD-ROM            HUAWEI   Mass Storage     2.31 PQ: 0 ANSI: 2
[   49.368325] sr1: scsi-1 drive
[   49.368854] sr 5:0:0:0: Attached scsi CD-ROM sr1
[   49.369011] sr 5:0:0:0: Attached scsi generic sg2 type 5
[   49.372964] scsi 6:0:0:0: Direct-Access     HUAWEI   SD Storage       2.31 PQ: 0 ANSI: 2
[   49.374988] sd 6:0:0:0: Attached scsi generic sg3 type 0
[   49.380995] sd 6:0:0:0: [sdb] Attached SCSI removable disk
[   49.511962] ISO 9660 Extensions: Microsoft Joliet Level 1
[   49.542715] ISOFS: changing to secondary root

So the device seems to register as an USB Mass Storage device, running lsusb shows us the the Vendor and Product ID:

philipp@mango:~$ lsusb
...
Bus 002 Device 008: ID 12d1:14fe Huawei Technologies Co., Ltd.

Great! So let's see how to switch the device into its Modem mode. There is a nice open source project called usb_modeswitch, which does the job. The usb_modeswitch coming with my Ubuntu 10.10 is rather outdated, so I'll build it from source. You'll need libusb-dev to build it, so if you haven't installed it yet you can simply install it via apt-get (or your distributions package manager). usb_modeswitch seems to play fine with most of the libusb-dev versions:

sudo apt-get install libusb-dev

Next we'll download, unpack and install the most recent versions of usb_modeswitch and its usb_modeswitch-data, available at:

http://www.draisberghof.de/usb_modeswitch

Installing is easy, I guess you are familiar with the commands (if not comment below):

# Download sources:
wget http://www.draisberghof.de/usb_modeswitch/usb-modeswitch-1.2.5.tar.bz2
# Extract the source:
tar xjf usb-modeswitch-1.2.5.tar.bz2
# Change directory:
cd usb-modeswitch-1.2.5
# Install it:
sudo make install

Do the same for the usb_modeswitch-data, which contains a lot of udev scripts, that might cover your hardware already:

wget http://www.draisberghof.de/usb_modeswitch/usb-modeswitch-data-20121109.tar.bz2
tar xjf usb-modeswitch-data-20121109.tar.bz2
cd usb-modeswitch-data-20121109
sudo make install

The new usb_modeswitch overrides the old version, so there's no need to take care about old installations! With usb_modeswitch it's easy to switch the Huawei E352-s into the Modem mode. A quick research reveals the switching command, I guess it's the packet Windows sends to the device in order to switch modes. I really don't feel the need to sniff it by myself (you have to run the commands as root):

usb_modeswitch -v 12d1 -p 14fe -M '55534243123456780000000000000011062000000100000000000000000000'

Now running lsusb reveals a new Product ID:

philipp@mango:~$ lsusb
...
Bus 002 Device 022: ID 12d1:1506 Huawei Technologies Co., Ltd

Using the option module should register the modem:

modprobe option
echo "12d1 1506" > /sys/bus/usb-serial/drivers/option1/new_id

Running dmesg (or looking at /var/log/messages if you prefer) reveals:

philipp@mango:~$ dmesg
...
[ 1471.905063] usb 2-5: new high speed USB device using ehci_hcd and address 12
[ 1472.039036] option 2-5:1.0: GSM modem (1-port) converter detected
[ 1472.039214] usb 2-5: GSM modem (1-port) converter now attached to ttyUSB0
[ 1472.039374] option 2-5:1.1: GSM modem (1-port) converter detected
[ 1472.039522] usb 2-5: GSM modem (1-port) converter now attached to ttyUSB1
[ 1472.039647] option 2-5:1.2: GSM modem (1-port) converter detected
[ 1472.039751] usb 2-5: GSM modem (1-port) converter now attached to ttyUSB2
[ 1472.039893] option 2-5:1.3: GSM modem (1-port) converter detected
[ 1472.039997] usb 2-5: GSM modem (1-port) converter now attached to ttyUSB3
[ 1472.040173] option 2-5:1.4: GSM modem (1-port) converter detected
[ 1472.040268] usb 2-5: GSM modem (1-port) converter now attached to ttyUSB4
[ 1472.040404] option 2-5:1.5: GSM modem (1-port) converter detected
[ 1472.040496] usb 2-5: GSM modem (1-port) converter now attached to ttyUSB5

The modem should be usable by now and your network-manager should ask you for the PIN to unlock the device. Congratulations!

udev rules

If you don't want to impress people by remembering these cryptic lines, you could write two udev rules to execute these commands whenever the device is added to the USB subsystem. Store it to /etc/udev/rules.d/70-huawei_e352.rules for example.

Filename /etc/udev/rules.d/70-huawei_e352.rules:

ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="14fe", RUN+="/usr/sbin/usb_modeswitch -v 12d1 -p 14fe -M '55534243123456780000000000000011062000000100000000000000000000'"
ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="14fe", RUN+="/bin/bash -c 'modprobe option && echo 12d1 1506 > /sys/bus/usb-serial/drivers/option1/new_id'"

And reload the rules:

udevadm control --reload-rules

And that's it basically! Your stick should now be put into Modem mode automatically, whenever you plug it in.

final note

I have noticed, that my Ubuntu 10.10 modem-manager sometimes has problems to recognize the device after reattaching it. This happens especially after restoring my system from hibernation. Restarting the network-manager service doesn't help here either. I know it doesn't sound great, but killing the modem-manager process works great for me:

philipp@mango:~$ ps ax | grep modem-manager
 3914 ?        S      0:00 /usr/sbin/modem-manager

philipp@mango:~$ sudo kill 3914

It is automatically restarted by the network-manager and the magic happens. If I have some time, I might take a look at it. But I guess it has been fixed in recent versions or doesn't apply to the Unity-based Ubuntu distributions at all.

useful links

http://www.draisberghof.de/usb_modeswitch/

Website Update

Philipp Wagner — Wed, 13 Mar 2013 20:10:00 +0100

Website Update

http://bytefish.de has been built on DokuWiki for years and I absolutely loved it. But I don't have the time to keep the installation in sync with latest security fixes and more recently some of the plugins began to act strange. It's finally time to make a change, so these pages are now built by Pelican, which is a static website generator. Disqus is used for the comments, but I can't import comments from the old system - I am sorry.

Most of my private work is open source, so are the sources and content of this page:

https://github.com/bytefish/bytefish.de

The theme is fit to my needs and contains some hardcoded stuff, so it is probably not useful for you without putting some work into. Most of the existing blog posts and wiki pages have been converted and should be available under the same URLs. I have omitted a few articles, that are not relevant anymore and I had to change some URLs in the wiki pages. Therefore some of the wiki pages have been turned into articles, early drafts will be added to /pages in the future. You can get a list of tags for the articles at /tags, see I am not a huge fan of tag clouds so I have simply ordered the list by the number of articles.

I'll probably extend some functionality, but right now I am quite happy. If you find something is missing in here, please do not hesitate to send me a mail!

Validating Algorithms

Philipp Wagner — Sat, 08 Sep 2012 09:02:00 +0200

Validating Algorithms

This post was originally written for the OpenCV QA forum. I post it here, because I think it's a great example of how Open Source projects make your life easy.

introduction

Actually validating algorithms is a very interesting topic and it's really not that hard. In this post I'll show how to validate your algorithms (I'll take the FaceRecognizer in this example). As always in my posts I will show it with a full source code example, because I think it's much easier to explain stuff by code.

So whenever people tell me "my algorithm performs bad", I ask them:

What is bad actually?
Did you rate this by looking at one sample?
What was your image data?
How do you split between training and test data?
What is your metric?
[...]

My hope is, that this post will clear up some confusion and show how easy it is to validate algorithms. Because what I have learned from experimenting with computer vision and machine learning algorithms is:

Without a proper validation it's all about chasing ghosts. You really, really need figures to talk about.

All code in this post is put under BSD License, so feel free to use it for your projects.

validating algorithms

One of the most important tasks of any computer vision project is to acquire image data. You need to get the same image data as you expect in production, so you won't have any bad experiences when going live. A very practical example: If you want to recognize faces in the wild, then it isn't useful to validate your algorithms on images taken in a very controlled scenario. Get as much data as possible, because Data is king.

Once you have got some data and you have written your algorithm, it comes to evaluating it. There are several strategies for validating, but I think you should start with a simple Cross Validation and go on from there, for informations on Cross Validation see:

Wikipedia on Cross-Validation

Instead of implementing it all by ourself, we'll make use of scikit-learn a great Open Source project:

https://github.com/scikit-learn

It comes with a very good documentation and tutorials for validating algorithms:

http://scikit-learn.org/stable/tutorial/statistical_inference/index.html

So the plan is the following:

Write a function to read some image data.
Wrap the cv2.FaceRecognizer into a scikit-learn estimator.
Estimate the performance of our cv2.FaceRecognizer with a given validation and metric.
Profit!!

Getting the image data right

First I'd like to write some words on the image data to be read, because questions on this almost always pop up. For sake of simplicity I have assumed in the example, that the images (the faces, persons you want to recognize) are given in folders. One folder per person. So imagine I have a folder (a dataset) called dataset1, with the subfolders person1, person2 and so on:

philipp@mango:~/facerec/data/dataset1$ tree -L 2 | head -n 20
.
|-- person1
|   |-- 1.jpg
|   |-- 2.jpg
|   |-- 3.jpg
|   |-- 4.jpg
|-- person2
|   |-- 1.jpg
|   |-- 2.jpg
|   |-- 3.jpg
|   |-- 4.jpg

[...]

One of the public available datasets, that is already coming in such a folder structure is the AT&T Facedatabase, available at:

http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html

Once unpacked it is going to look like this (on my filesystem it is unpacked to /home/philipp/facerec/data/at/, your path is different!):

philipp@mango:~/facerec/data/at$ tree .
.
|-- README
|-- s1
|   |-- 1.pgm
|   |-- 2.pgm
 ...
|-- s2
|   |-- 1.pgm
|   |-- 2.pgm
 ...
|-- s3
|   |-- 1.pgm
|   |-- 2.pgm
 ...

40 directories, 401 files

putting it together

So first of all we'll define a method read_images for reading in the image data and labels:

import os
import sys
import cv2
import numpy as np

def read_images(path, sz=None):
    """Reads the images in a given folder, resizes images on the fly if size is given.

    Args:
        path: Path to a folder with subfolders representing the subjects (persons).
        sz: A tuple with the size Resizes 

    Returns:
        A list [X,y]

            X: The images, which is a Python list of numpy arrays.
            y: The corresponding labels (the unique number of the subject, person) in a Python list.
    """
    c = 0
    X,y = [], []
    for dirname, dirnames, filenames in os.walk(path):
        for subdirname in dirnames:
            subject_path = os.path.join(dirname, subdirname)
            for filename in os.listdir(subject_path):
                try:
                    im = cv2.imread(os.path.join(subject_path, filename), cv2.IMREAD_GRAYSCALE)
                    # resize to given size (if given)
                    if (sz is not None):
                        im = cv2.resize(im, sz)
                    X.append(np.asarray(im, dtype=np.uint8))
                    y.append(c)
                except IOError, (errno, strerror):
                    print "I/O error({0}): {1}".format(errno, strerror)
                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
            c = c+1
    return [X,y]

Reading in the image data then becomes as easy as calling:

[X,y] = read_images("/path/to/some/folder")

Because some algorithms (for example Eigenfaces, Fisherfaces) require your images to be of equal size, I added a second parameter sz. By passing the tuple sz, all of the images get resized. So the following call will resize all images in /path/to/some/folder to 100x100 pixels, while loading:

[X,y] = read_images("/path/to/some/folder", (100,100))

All classifiers in scikit-learn are derived from a BaseEstimator, which is supposed to have a fit and predict method. The fit method gets a list of samples X and corresponding labels y, so it's trivial to map to the train method of the cv2.FaceRecognizer. The predict method also gets a list of samples and corresponding labels, but this time we'll need to return the predictions for each sample:

from sklearn.base import BaseEstimator

class FaceRecognizerModel(BaseEstimator):

    def __init__(self):
        self.model = cv2.createEigenFaceRecognizer()

    def fit(self, X, y):
        self.model.train(X,y)

    def predict(self, T):
        return [self.model.predict(T[i]) for i in range(0, T.shape[0])]

You can then choose between a large range of validation methods and metrics to test the cv2.FaceRecognizer with. You can find the available cross validation algorithms in sklearn.cross_validation:

Leave-One-Out cross validation
K-Folds cross validation
Stratified K-Folds cross validation
Leave-One-Label-Out cross-validation
Random sampling with replacement cross-validation
[...]

For estimating the recognition rate of the cv2.FaceRecognizer I suggest using a Stratified Cross Validation. You may ask why anyone needs the other Cross Validation methods. Imagine you want to perform emotion recognition with your algorithm. What happens if your training set has images of the person you test your algorithm with? You will probably find the closest match to the person, but not the emotion. In these cases you should perform a subject-independent cross validation.

Creating a Stratified k-Fold Cross Validation Iterator is very simple with scikit-learn:

from sklearn import cross_validation as cval
# Then we create a 10-fold cross validation iterator:
cv = cval.StratifiedKFold(y, 10)

And there's a wide range of metrics we can choose from. For now I only want to know the precision of the model, so we import the callable function sklearn.metrics.precision_score:

from sklearn.metrics import precision_score

Now we'll only need to create our estimator and pass the estimator, X, y, precision_score and cv to sklearn.cross_validation.cross_val_score, which calculates the cross validation scores for us:

# Now we'll create a classifier, note we wrap it up in the 
# FaceRecognizerModel we have defined in this file. This is 
# done, so we can use it in the awesome scikit-learn library:
estimator = FaceRecognizerModel()
# And getting the precision_scores is then as easy as writing:
precision_scores = cval.cross_val_score(estimator, X, y, score_func=precision_score, cv=cv)

There's a large amount of metrics available, feel free to choose another one:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/metrics.py

So let's put all this in a script!

validation.py

# Author: Philipp Wagner <bytefish@gmx.de>
# Released to public domain under terms of the BSD Simplified license.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#   * Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
#   * Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#   * Neither the name of the organization nor the names of its contributors
#     may be used to endorse or promote products derived from this software
#     without specific prior written permission.
#
#   See <http://www.opensource.org/licenses/bsd-license>

import os
import sys
import cv2
import numpy as np

from sklearn import cross_validation as cval
from sklearn.base import BaseEstimator
from sklearn.metrics import precision_score

def read_images(path, sz=None):
    """Reads the images in a given folder, resizes images on the fly if size is given.

    Args:
        path: Path to a folder with subfolders representing the subjects (persons).
        sz: A tuple with the size Resizes 

    Returns:
        A list [X,y]

            X: The images, which is a Python list of numpy arrays.
            y: The corresponding labels (the unique number of the subject, person) in a Python list.
    """
    c = 0
    X,y = [], []
    for dirname, dirnames, filenames in os.walk(path):
        for subdirname in dirnames:
            subject_path = os.path.join(dirname, subdirname)
            for filename in os.listdir(subject_path):
                try:
                    im = cv2.imread(os.path.join(subject_path, filename), cv2.IMREAD_GRAYSCALE)
                    # resize to given size (if given)
                    if (sz is not None):
                        im = cv2.resize(im, sz)
                    X.append(np.asarray(im, dtype=np.uint8))
                    y.append(c)
                except IOError, (errno, strerror):
                    print "I/O error({0}): {1}".format(errno, strerror)
                except:
                    print "Unexpected error:", sys.exc_info()[0]
                    raise
            c = c+1
    return [X,y]

class FaceRecognizerModel(BaseEstimator):

    def __init__(self):
        self.model = cv2.createFisherFaceRecognizer()

    def fit(self, X, y):
        self.model.train(X,y)

    def predict(self, T):
        return [self.model.predict(T[i]) for i in range(0, T.shape[0])]

if __name__ == "__main__":
    # You'll need at least some images to perform the validation on:
    if len(sys.argv) < 2:
        print "USAGE: facerec_demo.py </path/to/images> [</path/to/store/images/at>]"
        sys.exit()
    # Read the images and corresponding labels into X and y.
    [X,y] = read_images(sys.argv[1])
    # Convert labels to 32bit integers. This is a workaround for 64bit machines,
    # because the labels will truncated else. This is fixed in recent OpenCV
    # revisions already, I just leave it here for people on older revisions.
    #
    # Thanks to Leo Dirac for reporting:
    y = np.asarray(y, dtype=np.int32)
    # Then we create a 10-fold cross validation iterator:
    cv = cval.StratifiedKFold(y, 10)
    # Now we'll create a classifier, note we wrap it up in the 
    # FaceRecognizerModel we have defined in this file. This is 
    # done, so we can use it in the awesome scikit-learn library:
    estimator = FaceRecognizerModel()
    # And getting the precision_scores is then as easy as writing:
    precision_scores = cval.cross_val_score(estimator, X, y, score_func=precision_score, cv=cv)
    # Let's print them:
    print precision_scores

running the script

The above script will print out the precision scores for the Fisherfaces method. You simply need to call the script with the image folder:

philipp@mango:~/src/python$ python validation.py /home/philipp/facerec/data/at

Precision Scores:
[ 1.          0.85        0.925       0.9625      1.          0.9625
  0.8875      0.93333333  0.9625      0.925     ]

conclusion

The conclusion is, that... Open Source projects make your life really easy! There's much to enhance for the script. You probably want to add some logging, see which fold you are in for example. But it's a start for evaluating any metric you want, just read through the scikit-learn tutorials and adapt it to the above script. I encourage everybody to play around with OpenCV Python and scikit-learn, because interfacing these two great projects is really, really easy as you can see!

OpenCV FaceRecognizer documentation

Philipp Wagner — Tue, 26 Jun 2012 01:38:00 +0200

OpenCV FaceRecognizer documentation

I've just finished the documentation for the new cv::FaceRecognizer, which I have contributed to OpenCV 2.4. All the code and this documentation is going to be included in OpenCV 2.4.2. I hope you like it.

As of Revision 8779 OpenCV now comes with a thorough documentation for cv::FaceRecognizer. The HTML version of the documentation is online available at:

http://docs.opencv.org/trunk/modules/contrib/doc/facerec/index.html

There are several highlights:

Fisherfaces

Philipp Wagner — Sun, 03 Jun 2012 09:03:00 +0200

Fisherfaces

Some time ago I have written a post on Linear Discriminant Analysis, a statistical method often used for dimensionality reduction and classification. It was invented by the great statistician Sir R. A. Fisher, who successfully used it for classifying flowers in his 1936 paper "The use of multiple measurements in taxonomic problems" (The famous Iris Data Set is still available at the UCI Machine Learning Repository.). But why do we need another dimensionality reduction method, if the Principal Component Analysis (PCA) did such a good job? Well, the PCA finds a linear combination of features that maximizes the total variance in data. While this is clearly a powerful way to represent data, it doesn't consider any classes and so a lot of discriminative information may be lost when throwing some components away. This can yield bad results, especially when it comes to classification. In order to find a combination of features that separates best between classes the Linear Discriminant Analysis instead maximizes the ration of between-classes to within-classes scatter. The idea is, that same classes should cluster tightly together.

This was also recognized by Belhumeur, Hespanha and Kriegman and so they applied a Discriminant Analysis to face recognition in their paper "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection" (1997). The Eigenfaces approach by Pentland and Turk as described in "Eigenfaces for Recognition" (1991) was a revolutionary one, but the original paper already discusses the negative effects of images with changes in background, light and perspective. So on datasets with differences in the setup, the Principal Component Analysis is likely to find the wrong components for classification and can perform poorly.

You seldomly see something explained with code and examples, so I thought I'll change that. And there are already so many libraries, why not add another one? I have prepared a Python and GNU Octave/MATLAB version. The code is available in my github repository at:

https://www.github.com/bytefish/facerec

I've put all the code under a BSD License, so feel free to use it for your projects. While this post is based on the very first version number 0.1 of the framework, it still outlines the concepts in detail. You won't have any troubles to reproduce the results with the latest sources, I've added a comprehensive README to the project.

There's also a much simpler implementation in the documents over at:

You'll find all code shown in these documents in the projects github repository:

https://github.com/bytefish/facerecognition_guide

Algorithmic Description

The Fisherfaces algorithm we are going to implement basically goes like this:

Construct the Imagematrix X with each column representing an image. Each image is a assigned to a class in the corresponding class vector C.
Project X into the (N-c)-dimensional subspace as P with the rotation matrix WPca identified by a Principal Component Analysis, where
- N is the number of samples in X
- c is unique number of classes (length(unique(C)))
Calculate the between-classes scatter of the projection P as Sb = \sum_{i=1}^{c} N_i*(mean_i - mean)*(mean_i - mean)^T, where
- mean is the total mean of P
- mean_i is the mean of class i in P
- N_i is the number of samples for class i
Calculate the within-classes scatter of P as Sw = \sum_{i=1}^{c} \sum_{x_k \in X_i} (x_k - mean_i) * (x_k - mean_i)^T, where
- X_i are the samples of class i
- x_k is a sample of X_i
- mean_i is the mean of class i in P
Apply a standard Linear Discriminant Analysis and maximize the ratio of the determinant of between-class scatter and within-class scatter. The solution is given by the set of generalized eigenvectors Wfld of Sb and Sw corresponding to their eigenvalue. The rank of Sb is atmost (c-1), so there are only (c-1) non-zero eigenvalues, cut off the rest.
Finally obtain the Fisherfaces by W = WPca * Wfld.

For a detailed description of the algorithms used in this post, please refer to the paper in Further Reading.

Implementation Details

Enough talk, show me the code! I will focus on the Python version, because it's a lot cleaner. Don't get me wrong, most of the functions are also implemented in GNU Octave and it's also easy to use. Aside from architectural aspects, translating the core algorithms from Octave to Python was almost trivial. Thanks again to NumPy and matplotlib, which make Python feel like GNU Octave/MATLAB. This is great due to the fact, that OpenCV uses NumPy arrays since OpenCV2.2. That makes it possible to integrate NumPy projects into an OpenCV project without any hassle -- rapid prototyping comes for free!

Since the Fisherfaces method performs a Principal Component Analysis and Linear Discriminant Analysis on the data, all we need are two classes to perform a PCA and LDA. The Fisherfaces method then essentially boils down to (comments stripped off):

class Fisherfaces(Model):

    def __init__(self, X=None, y=None, num_components=None, k=1):
        Model.__init__(self, name="Fisherfaces")
        self.k = k
        self.num_components = num_components
        try:
            self.compute(X,y)
        except:
            pass


    def compute(self, X, y):
        """ Compute the Fisherfaces as described in [BHK1997]: Wpcafld = Wpca * Wfld.

        Args:
            X [dim x num_data] input data
            y [1 x num_data] classes
        """
        n = len(y)
        c = len(np.unique(y))
        pca = PCA(X, n-c)
        lda = LDA(pca.project(X), y, self.num_components)
        self._eigenvectors = pca.W*lda.W
        self.num_components = lda.num_components
        self.P = self.project(X)
        self.y = y

    def predict(self, X):
        Q = self.project(X)
        return NearestNeighbor.predict(self.P, Q, self.y, k=self.k)

    def project(self, X):
        return np.dot(self._eigenvectors.T, X)

    def reconstruct(self, X):
        return np.dot(self._eigenvectors, X)

All models in the code derive from a generic Model((Which does absolutely nothing, I just thought it would make a nice OOP hierarchy...)) and implement a compute and predict method, which are used to estimate the performance of a classifier with a cross validation. Classes for performing a leave-one-out and k-fold cross validation are implemented. I only measure the recognition rate of a classifier, so something for your use case might be missing. Feel free to add your measures in the validation module.

The filereader module only has a FilesystemReader right now. It loads images in a given path, assuming a folder structure of <path>/<subject>/<image> for reading in the dataset. There are some limitations, which will be fixed in later versions. For now you must ensure, that all images have equal size and the image format is supported by PIL (same applies for ImageMagick in GNU Octave).

Last but not least, there's a visualization module. I don't feel afraid to say: it feels very, very hardcoded. Plotting and subplotting eigenvectors is supported, so is comparing accuracy (+ standard deviation) vs. parameters for a bunch of validators (errorbar plot). Absolutely everything regarding the style of a plot is missing, so it will take a lot more iterations to create smoother plotting facilities. If you want to quickly create some plots, go for the Octave version for now.

Experiments

Everything is implemented? So let the experiments begin! The last face recognition experiment I have posted was performed on the AT&T Face database. Now if you spider through some literature you will notice, that it's a rather easy dataset. Images are taken in a very controlled environment with only slightly changes in perspective and light. You won't see any great improvements with a Discriminant Analysis, because it's hard to beat the 96% accuracy we got with Eigenfaces.

That's why I will validate the algorithms on the Yale Facedatabase A. You can download it from the UCSD Computer Vision Group:

http://vision.ucsd.edu/content/yale-face-database

The second experiment is something I wanted to do for a long time: we will see how faces of celebrities cluster together and I can finally project my face to where it belongs (hopefully I belong to the Brad Pitt cluster)! You'll see dramatic improvements with a Linear Discriminant Analysis in this experiment.

Validation scheme

My experimental validation scheme is very easy. I will do n runs for each experiment and each run is validated with a k-fold Cross-validation on a shuffled training set. From these n runs the average accuracy and standard deviation is reported. In literature you'll often find the results of a Leave-One-Out Cross Validation, so I'll also report it. For a good writeup on cross validation based model selection, see Andrew Moore's slides on Cross-validation for preventing and detecting overfitting.

k-fold Cross Validation

Cross-... what? Imagine I want to estimate the error of my face recognition algorithm. I have captured images of people over a long period of time and now I am building a training and test set from these images. Somehow I have managed to put all happy looking people in the training set, while all people in my test set are terribly sad with their head hanging down somewhere. Should I trust the results I get from this evaluation? Probably not, because it's a really, really unfortunate split of my available data. A cross-validation avoids such splits, because you build k (non-overlapping) training and test datasets from your (shuffled) original dataset. So what's a fold then? It's best explained with an example.

For a dataset D with 3 classes {c0,c1,c2} each having 4 observations {o0,o1,o2,o3}, a 4-fold cross validation produces the following four folds F:

(F1)

    o0 o1 o2 o3 
c0 | A  B  B  B |
c1 | A  B  B  B |
c2 | A  B  B  B |

A = {D[c0][o0], D[c1][o0], D[c2][o0]}
B = D\A

(F2)

    o0 o1 o2 o3 
c0 | B  A  B  B |
c1 | B  A  B  B |
c2 | B  A  B  B |

A = {D[c0][o1], D[c1][o1], D[c2][o1]}
B = D\A

(F3)

    o0 o1 o2 o3 
c0 | B  B  A  B |
c1 | B  B  A  B |
c2 | B  B  A  B |

A = {D[c0][o2], D[c1][o2], D[c2][o2]}
B = D\A

(F4)

    o0 o1 o2 o3 
c0 | B  B  B  A |
c1 | B  B  B  A |
c2 | B  B  B  A |

A = {D[c0][o3], D[c1][o3], D[c2][o3]}
B = D\A

From these folds the accuracy and the standard deviation can be calculated. Please have a look at the Python or GNU Octave implementations if you have troubles implementing it. It's pretty self-explaining.

Leave One Out Cross Validation

A Leave-one-out cross-validation (LOOCV) is the degenerate case of a k-fold cross-validation, where k is chosen to be the number of observations in the dataset. Simply put, you use one observation for testing your model and the remaining data for training the model. This results in as many experiments as observations available, which may be too expensive for large datasets.

For the same example as above you would end up with the following 12 folds F in a LOOCV:

(F1)

    o0 o1 o2 o3 
c0 | A  B  B  B |
c1 | B  B  B  B |
c2 | B  B  B  B |

A = {D[c0][o0]}
B = D\A

(F2)

    o0 o1 o2 o3 
c0 | B  A  B  B |
c1 | B  B  B  B |
c2 | B  B  B  B |

A = {D[c0][o1]}
B = D\A

(...)

(F12)
    o0 o1 o2 o3 
c0 | B  B  B  B |
c1 | B  B  B  B |
c2 | B  B  B  A |

A = {D[c2][o3]}
B = D\A

Yale Facedatabase A

The first database I will evaluate is the Yale Facedatabase A. It consists of 15 people (14 male, 1 female) each with 11 grayscale images sized 320x243px. There are changes in the light conditions (center light, left light, right light), background and in facial expressions (happy, normal, sad, sleepy, surprised, wink) and glasses (glasses, no-glasses).

Data Preprocessing

Unfortunately the images in this database are not aligned, so the faces may be at different positions in the image. For the Eigenfaces and Fisherfaces method the images need to be preprocessed, since both methods are sensible to rotation and scale. You don't want to do this manually, so I have written a function crop.m/crop_face.py for this, which has the following description:

function crop(filename, eye0, eye1, top, left, dsize)
    %% Rotates an image around the eyes, resizes to destination size.
    %%
    %% Assumes a filename is structured as ".../subject/image.extension", so that
    %% images are saved in a the folder (relative to where you call the script):
    %%  "crop_<subject>/crop_<filename>"
    %%
    %% Args:
    %%  filename: Image to preprocess.
    %%  eye0: [x,y] position of the left eye
    %%  eye1: [x,y] position of the right eye
    %%  top: percentage of offset above the eyes
    %%  left: percentage of horizontal offset 
    %%  dsize: [width, height] of destination filesize
    %%
    %% Example:
    %% crop("/path/to/file", [300, 100], [380, 110], 0.3, 0.2, [70,70])
    %% 
    %% Leaves 21px offset above the eyes (0.3*70px) and 28px horizontal
    %% offset. 20% of the image to the left eye, 20% to the right eye
    %% (2*0.2*70=28px).

I specify the dimensions of the cropped image to be (100,130) for this database. top is the vertical offset, which specifies how much of the image is above the eyes, 40% seems to be a good value for these images. left specifies the horizontal offset, how much space is left from an eye to the border, 30% is a good value. The image is then rotated by the eye positions. If it's not desired to align all images at the eyes, scale or rotate them, please adjust the script to your needs. batch_crop.m is a simple script to crop images given by their filename and associated eye coordinates. Feel free to translate it to Python.

After having preprocessed the images (coordinates for this database are given here) you'll end up with subsets for each person like this (function used to create the gallery of images):

The preprocessed dataset then has to be stored in a hierarchy similiar to this:

philipp@mango:~/python/facerec$ tree /home/philipp/facerec/data/yalefaces
/home/philipp/facerec/data/yalefaces
|-- s01
|   |-- crop_subject01.centerlight
|   |-- crop_subject01.glasses
|   |-- crop_subject01.happy
|       [...]
|-- s02
|   |-- crop_subject02.centerlight
|   |-- crop_subject02.glasses
|   |-- crop_subject02.happy
|       [...]

15 directories, 165 files

Eigenfaces

I want to do face recognition on this dataset. The Eigenfaces method did such a good job on the AT&T Facedatabase, so how does it work for the Yalefaces? Let's find out and import the models, validation and filereader:

>>> from facerec.models import *
>>> from facerec.validation import *
>>> from facerec.filereader import *

Read in the Yale Facedatabase A:

>>> dataset = FilesystemReader("/home/philipp/facerec/data/yalefaces")

Create the Eigenfaces model, we'll take the 50 principal components in this example:

>>> eigenface = Eigenfaces(num_components=50) # take 50 principal components

We can measure the performance of this classifier with a Leave-One Out Cross validation:

>>> cv0 = LeaveOneOutCrossValidation(eigenface)
>>> cv0.validate(dataset.data,dataset.classes,print_debug=True)
>>> cv0
Leave-One-Out Cross Validation (model=Eigenfaces, runs=1, accuracy=81.82%, tp=135, fp=30, tn=0, fn=0)

So the model has a recognition rate of 81.82%. Note that a Leave-One-Out Cross validation can be computationally expensive for large datasets. In this example we are computing 165 models, with a computation time of 3 seconds per model this validation will take circa 8 minutes to finish. It's often more useful to perform a k-fold cross validation instead, because a Leave One Out Cross validation can also act weird at times (see the slides at Introduction to Pattern Analysis: Cross Validation). A 5-fold cross validation is good for this database:

>>> cv1= KFoldCrossValidation(eigenfaces, k=5) # perform a 5-fold cv
>>> for i in xrange(10):
...     cv1.validate(dataset.data,dataset.classes, print_debug=True)

The model will be validated 50 times in total, so the loop will take 2.5 minutes to finish. If you want to log the validation results per fold (so you can get the standard deviation of a single k-fold cross-validation run for example) add results_per_fold=True when initializing the class (be careful, the figures may be misleading for very small folds).

Some coffees later the validation has finished. Before reporting the final figure I'd like to show you some methods common to all validation objects:

>>> cv1.results
array([[123,  27,   0,   0],
       [123,  27,   0,   0],
       [121,  29,   0,   0],
       [119,  31,   0,   0],
       [120,  30,   0,   0],
       [121,  29,   0,   0],
       [122,  28,   0,   0],
       [124,  26,   0,   0],
       [120,  30,   0,   0],
       [121,  29,   0,   0]])
>>> cv1.tp
array([123, 123, 121, 119, 120, 121, 122, 124, 120, 121])
>>> cv1.accuracy
0.80933333333333335
>>> cv1.std_accuracy
0.0099777436361914683

And to get an overview you can simply get the representation of the Validation object:

>>> cv1 # or simply get the representation
k-Fold Cross Validation (model=Eigenfaces, k=5, runs=10, accuracy=80.93%, std(accuracy)=1.00%, tp=1214, fp=286, tn=0, fn=0)

With 80.93%+-1.00% accuracy the Eigenfaces algorithms performs OK on this dataset, I wouldn't say it performs bad. But can we do better, by taking less or more principal components? We can write a small script to evaluate that. I will perform a 5-fold cross-validation on the Eigenfaces model with [10, 25, 40, 55, 70, 85, 100] components:

parameter = range(10,101,15)
results = []
for i in parameter:
    cv = KFoldCrossValidation(Eigenfaces(num_components=i),k=5)
    cv.validate(dataset.data,dataset.classes)
    results.append(cv)

And we can see, that it doesn't really better the recognition rate:

>>> for result in results:
...     print "%d components: %.2f%%+-%.2f%%" % (result.model.num_components, result.accuracy*100, result.std_accuracy*100)
... 
10 components: 73.33%+-0.00%
25 components: 80.00%+-0.00%
40 components: 80.67%+-0.00%
55 components: 81.33%+-0.00%
70 components: 82.67%+-0.00%
85 components: 81.33%+-0.00%
100 components: 82.00%+-0.00%

Maybe we can get a better understanding of the method if we look at the Eigenfaces. Computing the model is necessary, because in the validation the model is cleared.

>>> eigenfaces = Eigenfaces(dataset.data,dataset.classes,num_components=100) # model will now be computed

and then plot the Eigenfaces:

>>> from facerec.visual import *
>>> PlotSubspace.plot_weight(model=eigenfaces, num_components=16, dataset=dataset, title="Eigenfaces", rows=4, cols=4, filename="16_eigenfaces.png")

Almost everywhere you see grayscaled Eigenfaces, which are really hard to explain. With a colored heatmap instead you can easily see the distribution of grayscale intensities, where 0 codes a dark blue and 255 is a dark red:

Just by looking at the first 16 Eigenfaces we can see, that they don't really describe facial features to distinguish between persons, but find components to reconstruct from. The fourth Eigenface for example describes the left-light, while the principal component 6 describes the right-light. This makes sense if you think of the Principal Component Analysis not as a classification method, but more as a compression method.

Let's see what I mean. By computing the Eigenfaces, we have projected the database onto the first hundred components as identified by a Principal Component Analysis. This effectively reduced the dimensionality of the images to 100. We can now try to reconstruct the real faces from these 100 components:

>>> proj1 = eigenfaces.P[:,17].copy()
>>> I = eigenfaces.reconstruct(proj1)

... and plot it:

>>> PlotBasic.plot_grayscale(I, "eigenface_reconstructed_face.png", (dataset.width, dataset.height))

... and we should see a face again:

Fisherfaces

Time for the Fisherfaces method! If you haven't done already, import the modules and load the data:

>>> from facerec.models import *
>>> from facerec.validation import *
>>> from facerec.filereader import *

Read the dataset:

>>> dataset = FilesystemReader("/home/philipp/facerec/data/yalefaces")

create the model (default parameters are ok):

>>> fisherfaces = Fisherfaces()

And then perform a Leave-One-Out Cross-Validation:

>>> cv2 = LeaveOneOutCrossValidation(fisherfaces)
>>> cv2.validate(dataset.data,dataset.classes, print_debug=True)
>>> cv2
Leave-One-Out Cross Validation (runs=1, accuracy=96.36%, tp=159, fp=6, tn=0, fn=0)

... which yields an accuracy of 96.36%. Performing 10 runs of a 5-fold cross validation supports this:

>>> cv3 = KFoldCrossValidation(fisherfaces, k=5) # perform a 5-fold cv
>>> for i in xrange(10):
...     cv3.validate(X,y,print_debug=True)
>>> cv3
k-Fold Cross Validation (k=5, runs=10, accuracy=96.80%, std(accuracy)=1.63%, tp=1452, fp=48, tn=0, fn=0, model=Fisherfaces (num_components=14))

Allthough the standard deviation is slightly higher for the Fisherfaces, with 96.80%+-1.63% it performs much better than the Eigenfaces method. You can also see that the faces were reduced to only 14 components (equals number of subjects - 1). Let's have a look at the components identified by the Fisherfaces method:

>>> fisherface = Fisherfaces(dataset.data,dataset.classes) # initialize and compute, or call compute explicitly

Import the visual module and plot the faces:

>>> from facerec.visual import *
>>> PlotSubspace.plot_weight(model=fisherface, num_components=14, dataset=dataset, filename="16_fisherfaces.png", rows=4, cols=4)

The Fisherfaces are a bit harder to explain, because they identify regions of a face that separate faces best from each other. None of them seems to encode particular light settings, at least it's not as obvious as in the Eigenfaces method:

I could only guess which component describes which features. So I leave the interpretation up to the reader. What we lose with the Fisherfaces method for sure, is the ability to reconstruct faces. If I want to reconstruct face number 17, just like in the Eigenfaces section:

>>> proj2 = fisherface.P[:,17].copy()
>>> I2 = fisherface.reconstruct(proj2)

... and plot it:

>>> PlotBasic.plot_grayscale(I2, "fisherface_reconstructed_face.png", (dataset.width, dataset.height))

We get back a picture similar to:

Conclusion

On this dataset the Fisherface method performed much better, than the Eigenfaces method did. While it yields a much better recognition performance, the ability to reconstruct faces from the Eigenspace is lost. So in conclusion, I think both methods complement each other. It would be a good idea to keep the PCA object around (already computed in the Fisherfaces method) in order to use its reconstruction capabilities.

Celebrities Dataset

Some years ago a lot of people told me, that I am looking just like Brad Pitt. Or did I tell it everyone after seeing Troy for the fifth time? Whatever -- sadly times have changed. I am losing my hair and now I am exponentially progressing towards a Sir Patrick Stewart or a bald Britney Spears. Now armed with the algorithms we've discussed, I can finally clear the confusion and find out which celebrity I resemble the most! This last experiment turned into a weekend project itself, but I had a lot of fun writing it and I hope you have fun reading it.

My celebrity dataset consists of the following faces:

I have chosen the images to be frontal face with a similiar perspective and light settings. Since the algorithms are sensible towards rotation and scale, the images have been preprocessed to equal scale of 70x70px and centered eyes. Here is my Clooney-Set for example:

I hope you understand that I can't share the dataset with you. Some of the photos have a public license, but most of the photos have an unclear license. From my experience all I can tell you is: finding images of George Clooney took less than a minute, but it was really, really hard finding good images of Johnny Depp or Arnold Schwarzenegger.

Recognition

Let's see how both methods perform on the celebrities dataset.

Import the modules:

>>> from facerec.models import *
>>> from facerec.validation import *
>>> from facerec.filereader import *
>>> from facerec.visual import *

Read in the database:

>>> dataset = FilesystemReader("/home/philipp/facerec/data/c1")

The Eigenfaces method with a Leave-One-Out cross validation:

>>> eigenface = Eigenfaces(num_components=100)
>>> cv0 = LeaveOneOutCrossValidation(eigenface)
>>> cv0.validate(dataset.data,dataset.classes,print_debug=True)
>>> cv0
Leave-One-Out Cross Validation (model=Eigenfaces, runs=1, accuracy=44.00%, tp=44,
 fp=56, tn=0, fn=0)

Only achieves 44% recognition rate! A 5-fold cross validation:

>>> cv1 = KFoldCrossValidation(eigenface, k=5)
>>> for i in xrange(10):
...     cv1.validate(dataset.data,dataset.classes,print_debug=True)
>>> cv1
k-Fold Cross Validation (model=Eigenfaces, k=5, runs=10, accuracy=43.30%,
 std(accuracy)=2.33%, tp=433, fp=567, tn=0, fn=0)

Shows that the recognition rate of 43.30%+-2.33% is a bit better than guessing, still it's too bad to decide between celebrities.

The Fisherface method instead:

>>> fisherface = Fisherfaces()

... achieves a 93% recognition rate with a leave one out strategy:

>>> cv2 = LeaveOneOutCrossValidation(fisherface)
>>> cv2.validate(dataset.data,dataset.classes, print_debug=True)
Leave-One-Out Cross Validation (model=Fisherfaces, runs=1, accuracy=93.00%, tp=93,
 fp=7, tn=0, fn=0)

And with 5-fold cross validation:

>>> cv3 = KFoldCrossValidation(fisherface,k=5)
>>> for i in xrange(10):
...     cv3.validate(dataset.data,dataset.classes,print_debug=True)
>>> cv3
k-Fold Cross Validation (model=Fisherfaces (num_components=9), k=5, runs=10, 
accuracy=90.50%, std(accuracy)=1.36%, tp=905, fp=95, tn=0, fn=0)

... it has a recognition rate of 90.50%+-1.36%. So it's a good classificator to decide between our celebrities. Now let's have a look at the components found by both methods!

Compute both models (necessary because the validation empties the model):

>>> eigenface = Eigenfaces(dataset.data,dataset.classes,num_components=100)
>>> fisherface = Fisherfaces(dataset.data,dataset.classes)

The Eigenfaces...

>>> PlotSubspace.plot_weight(model=eigenface, num_components=16, dataset=dataset, title="Celebrity Eigenface", rows=4, cols=4, filename="16_celebrity_eigenfaces.png")

... again encode light in the images, see the second Eigenface for example:

While the Fisherfaces identify regions:

>>> PlotSubspace.plot_weight(model=fisherface, num_components=9, dataset=dataset, title="Celebrity Fisherface", rows=3, cols=3, filename="9_celebrity_fisherfaces.png")

... where you can decide between the persons. The first component for example describes the eyes, while other components describe mouth-ness or eye-browness of a face:

By using two and three components, we can see how the faces are distributed (using GNU Octave, see fisherfaces_example.m). I'll use just one example per class for a better overview:

octave:1> addpath (genpath ("."));
octave:2> [X y width height names] = read_images("/home/philipp/facerec/data/c1");
octave:3> fisherface = fisherfaces(X,y);
octave:4> figure; hold on
octave:5> for i = [1:10]
> C = findclasses(fisherface.y, i)
> text(fisherface.P(1,C(1)), fisherface.P(2,C(1)), num2str(i));
> endfor
octave:6> print -dpng "-S640,480" celebrity_facespace.png
octave:7> names
names =

{
  [1,1] = crop_angelina_jolie
  [1,2] = crop_arnold_schwarzenegger
  [1,3] = crop_brad_pitt
  [1,4] = crop_george_clooney
  [1,5] = crop_johnny_depp
  [1,6] = crop_justin_timberlake
  [1,7] = crop_katy_perry
  [1,8] = crop_keanu_reeves
  [1,9] = crop_patrick_stewart
  [1,10] = crop_tom_cruise
}

Brad Pitt (3), Johnny Depp (5) and Tom Cruise (10) seem to have a unique face, as they are far away from all other faces. Surprisingly George Clooney (4) and Justin Timberlake (6) are near to each other. Arnold Schwarzenegger (2), Katy Perry (7), Keanu Reeves (8) and Patrick Stewart (9) are also close to each other. Angelina Jolie (1) is right between those two clusters; with George Clooney (4) and Keanu Reeves (8) as closest match:

Who Am I?

Now let's finally answer the question why I wrote this post at all. Who do I resemble the most? I'll use a frontal photo of myself and crop it, just like I did with the other faces:

Then I compute the model:

>>> fisherface = Fisherfaces(dataset.data,dataset.classes)

Read in my face:

>>> Q = FilesystemReader.read_image("/home/philipp/crop_me.png")

And find the closest match:

>>> prediction = fisherface.predict(Q)
>>> print dataset.className(prediction)
...
crop_patrick_stewart

I knew it. The closest match to my face is Patrick Stewart. Taking the 5 nearest neighbors, doesn't change a thing... it's still Patrick. Damn, I knew including him would change everything -- and I am not even close to Brad Pitt!

If I project my face on the first 2 components I would actually resemble Arnie the most:

But including the third dimension (projection on second and third component) brings me closer to Patrick Stewart:

Just because the Eigenfaces algorithm can't classify the persons doesn't mean we can't use it here aswell. It did a great job at reconstructing a projected vector, so if I project my face into the facespace... and reconstruct it... my face should be the mixture of those faces.

Let's see what I look like:

>>> eigenface = Eigenfaces(num_components=100)
>>> eigenface.compute(dataset.data,dataset.classes)
>>> proj = eigenface.project(Q)
>>> rec = eigenface.reconstruct(proj)
>>> PlotBasic.greyscale(rec, "philipp_celebrity.png")

Sorry it's a bit small, because my training images are only 70x70 pixels:

Definitely some Schwarzenegger in there!

Working with the GNU Octave version

The GNU Octave version is just as simple to use as the Python version. I think there's no need to explain every function argument in detail, the code is short and it's all commented in the definition files already. Let's go and fire up Octave. But before you start to do anything, you have to add the path of definition files to the GNU Octave search path. Since GNU Octave 3 you can use addpath(genpath("/path/to/folder/")) to add all definition files in a folder and subfolder:

addpath (genpath ("."));

This makes all function definition in the subfolders available to GNU Octave (see the examples in facerec) and now you can read in the dataset:

% load data
[X y width height names] = read_images("/home/philipp/facerec/data/yalefaces_recognition");

from this we can learn the Eigenfaces with 100 principal components:

eigenface = eigenfaces(X,y,100);

If we want to see the classes in 2D:

%% 2D plot of projection (add the classes you want)
figure; hold on;
for i = findclasses(eigenface.y, [1,2,3])
    text(eigenface.P(1,i), eigenface.P(2,i), num2str(eigenface.y(i)));
endfor

I adopted the KISS principle for the validation. The key idea is the following: the validation gets two function handles fun_train, which learns a model and fun_predict, which tests the learned model. Inside the validation a data array and corresponding classes are passed to the training function; and fun_predict gets a model and test instance. What to do if you need to add parameters to one of the calls? Bind them! I want to learn eigenfaces with 100 components and the prediction should use a 1 nearest neighbor for the classification:

fun_eigenface = @(X,y) eigenfaces(X,y,100);
fun_predict = @(model, Xtest) eigenfaces_predict(model, Xtest, 1)

Easy huh? Performing validations can now be done with:

% perform a Leave-One-Out Cross Validation
cv0 = LeaveOneOutCV(X,y,fun_eigenface, fun_predict, 1)
% perform a 10-fold cross validation
cv1 = KFoldCV(X,y,10,fun_eigenface, fun_predict,1)
% perform a 3-fold cross validation
cv2 = KFoldCV(X,y,3,fun_eigenface, fun_predict,1)

From these results you can gather the same statistics I have used in the Python version. I hope you acknowledge that this version is just as easy to use as the Python version. There are some additional parameters for the functions, so please read the function definition files or the examples to understand those. For further questions, simply comment below.

Conclusion and Lessons learned

In this post I have presented you a simple framework to evaluate your face recognition algorithms with Python and GNU Octave. The Python version will see some updates in the future and its usage might change a little, because some things are too complicated right now. I don't know if maintaining the GNU Octave version pays off, because I'd like to use it for prototyping algorithms. And after all, you have been answered one of most important questions in universe: my face resembles Patrick Stewart.

There are a few lessons I've learnt in this project and I'd like to share. If you are working with large datasets like with hundreds of face images, be sure to always choose the appropriate type for your data: a grayscale image for example only takes values in a range of [0,255] and fits into an unsigned byte. It would be a horrible waste of memory to take floats, which is the default NumPy datatype for arrays and matrices. And this applies to every other dataset as well. If you know which values your inputs can take: choose the appropriate type.

Working with NumPy arrays

One thing you have to get used to when starting with NumPy is working with its arrays and matrices. In NumPy it's a rule of thumb to take NumPy arrays for everything, so if you use one of NumPys built-in functions you can be pretty sure that an array is returned. And there are certain differences to a NumPy matrix you should know about. I've run into the following problem: I wanted to subtract the column mean from each column vector of an array A. My first version was like this:

>>> A = np.array([[2,2],[2,7]])
>>> A - A.mean(axis=1)
array([[ 0. , -2.5],
       [ 0. ,  2.5]])
>>> A.mean(axis=1)
array([ 2. ,  4.5])

Which is obvliously wrong and it is because A.mean(axis=1) returns a row instead of a column, as you can see in the listing. You can avoid that with:

>>> A - A.mean(axis=1).reshape(-1,1)
array([[ 0. ,  0. ],
       [-2.5,  2.5]])
>>> # or you set the metadata information to a matrix
>>> A = np.asmatrix(A)
>>> A - A.mean(axis=1)
matrix([[ 0. ,  0. ],
        [-2.5,  2.5]])

So if you want to treat an array like a matrix, then simply call numpy.asmatrix at some point in your code. Next thing you'll need to understand is related to array slicing and indexing, which is a great thing in NumPy. If you want to just slice the first column off an array, be sure to always make a copy of the sliced data instead of just assigning a variable to the slice. This applies if you don't go for speed, the situation might change if you are looping through data a million times. See this little example. Let's start the interactive python shell and import numpy:

>>> import numpy as np

With pmap we can now see how much memory the python process currently consumes:

philipp@mango:~$ ps ax | grep python
27241 pts/7    S+     0:00 python
philipp@mango:~$ pmap -x 27241
[...]
b783c000       0     524     524 rw---    [ anon ]
b78d7000       0       8       8 rw---    [ anon ]
bf979000       0     172     172 rw---    [ stack ]
-------- ------- ------- ------- -------
total kB   26436       -       -       -

26 MB is a pretty good value (rounded up). Now let's create a huge NumPy matrix with 9000*9000*64 bit:

>>> huge = np.zeros((9000,9000),dtype=np.float64)

The process should now occupy 26 MB + 618 MB, making it 644 MB in total. pmap verifies:

philipp@mango:~$ pmap -x 27241
[...]
b783c000       0     524     524 rw---    [ anon ]
b78d7000       0       8       8 rw---    [ anon ]
bf979000       0     172     172 rw---    [ stack ]
-------- ------- ------- ------- -------
total kB  659252       -       -       -

The process now has 644 MB. What happens if you attempt to slice the array? Take off the first column and assign it to huge again:

>>> huge = huge[:,0]
>>> huge.shape
(9000,)

Wonderful, and the memory?

philipp@mango:~$ pmap -x 27241
[...]
b783c000       0     524     524 rw---    [ anon ]
b78d7000       0       8       8 rw---    [ anon ]
bf979000       0     172     172 rw---    [ stack ]
-------- ------- ------- ------- -------
total kB  659252       -       -       -

Still 644 MB! Why? Because NumPy on one hand stores your raw array data (called a data buffer in the NumPy internals document and on the other hand stores information about the raw data. If you slice a column and assign it to a variable, you basically just create a new metadata object (with the specific information about shape etc.), but it's still a view on the same data buffer. So all your data still resides in memory. This makes slicing and indexing of NumPy arrays and matrices superfast, but it's something you should definitely know about.

If speed is not crucial to you and you care about your bytes, be sure to make a copy:

>>> huge = huge[0:,0].copy() # version (1)
>>> huge = np.array(huge[0:,0], copy=True) # or version (2)

... and the memory magically shrinks to 26 MB again.:

philipp@mango:~$ pmap -x 27241
[...]
b783c000       0     524     524 rw---    [ anon ]
b78d7000       0       8       8 rw---    [ anon ]
bf979000       0     172     172 rw---    [ stack ]
-------- ------- ------- ------- -------
total kB   26436       -       -       -

OOP in GNU Octave

I didn't go for OOP in GNU Octave, because even simple examples like myclass.m:

function b = myclass(a)
  b.a = a;
  b = class (b, "myclass");
end

fail with GNU Octave, version 3.2.4:

$ octave
octave:0> x = myclass(1)
error: class: invalid call from outside class constructor
error: called from:
error:   myclass at line 3, column 5

From what I have read the error seems to be a regression bug and it's probably fixed in the latest stable release. However, the implementation of OOP in GNU Octave still seems to be experimental and I don't think everybody is on latest releases. And to be honest, I've seldomly (read almost never) seen object oriented code with either MATLAB or Octave, so I don't really feel guilty about not using it.

Extracting Contours with OpenCV

Philipp Wagner — Sat, 26 May 2012 13:37:00 +0200

Extracting Contours with OpenCV

Yesterday I was asked how to extract a contour from a given image in OpenCV. Here is an example. Imagine we got this tasty apple and we want to put it in another image (with a green background):

One solution is to first detect the edges of the apple with a Canny filter, then find the contours with OpenCV's findContours and create a mask with drawContours:

Finally copy the masked original image to the new image, which means only the areas of the contours will be copied... and you are done. You end up with a tasty apple on a green background:

In code this translates to:

#include "cv.h"
#include "highgui.h"

using namespace cv;
using namespace std;

int main() {

    // read in the apple (change path to the file)
    Mat img0 = imread("/home/philipp/img/apple.jpg", 1);

    Mat img1;
    cvtColor(img0, img1, CV_RGB2GRAY);

    // apply your filter
    Canny(img1, img1, 100, 200);

    // find the contours
    vector< vector<Point> > contours;
    findContours(img1, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE);

    // you could also reuse img1 here
    Mat mask = Mat::zeros(img1.rows, img1.cols, CV_8UC1);

    // CV_FILLED fills the connected components found
    drawContours(mask, contours, -1, Scalar(255), CV_FILLED);

    /*
     Before drawing all contours you could also decide
     to only draw the contour of the largest connected component
     found. Here's some commented out code how to do that:
    */

//    vector<double> areas(contours.size());
//    for(int i = 0; i < contours.size(); i++)
//        areas[i] = contourArea(Mat(contours[i]));
//    double max;
//    Point maxPosition;
//    minMaxLoc(Mat(areas),0,&max,0,&maxPosition);
//    drawContours(mask, contours, maxPosition.y, Scalar(1), CV_FILLED);

    // let's create a new image now
    Mat crop(img0.rows, img0.cols, CV_8UC3);

    // set background to green
    crop.setTo(Scalar(0,255,0));

    // and copy the magic apple
    img0.copyTo(crop, mask);

    // normalize so imwrite(...)/imshow(...) shows the mask correctly!
    normalize(mask.clone(), mask, 0.0, 255.0, CV_MINMAX, CV_8UC1);

    // show the images
    imshow("original", img0);
    imshow("mask", mask);
    imshow("canny", img1);
    imshow("cropped", crop);

    imwrite("/home/philipp/img/apple_canny.jpg", img1);
    imwrite("/home/philipp/img/apple_mask.jpg", mask);
    imwrite("/home/philipp/img/apple_cropped.jpg", crop);

    waitKey();
    return 0;
}

OpenCV, Microsoft Visual Studio and libfacerec

Philipp Wagner — Thu, 19 Apr 2012 01:13:00 +0200

OpenCV, Microsoft Visual Studio and libfacerec

I've got a lot of mails from people, who have problems to use OpenCV with Microsoft Visual Studio 2008/2010. While I think it's unbelievably easy by using CMake, I am going to explain how to build the libfacerec demo without using CMake. I have illustrated each step with screenshots and detailed explanation, so it's easy for you to follow. I am going to use Microsoft Windows 7, Microsoft Visual Studio 2010 and OpenCV 2.3.1 in this tutorial.

Setting up OpenCV

First of all you'll need OpenCV. For this tutorial I suggest to download the OpenCV Superpack 2.3.1, which is available as OpenCV-2.3.1-win-superpack.exe at:

http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/

There are a lot of other ways to install OpenCV, so please consult the docs if you don't like the suggested one. The Superpack itself is not an installer, but a self-extracting archive. It contains the pre-built OpenCV library for VC9 (Microsoft Visual Studio 2008), VC10 (Microsoft Visual Studio 2010) and MinGW to mention a few. You'll only need the VC10 binaries for this tutorial.

Double-click the OpenCV-2.3.1-win-superpack.exe and select a folder to extract to. I'll choose D:\projects in this example:

The OpenCV superpack is no installer, so you'll need to manually add the OpenCV libraries to the DLL search path of Windows. The search paths of Windows are given in the PATH environment variable, which is a list of the directories to search, each separated by a semicolon.

Windows Vista and Windows 7 users

From the Desktop (or Start Menu), right-click My Computer and click Properties.
Click Advanced System Settings link in the left column.
In the System Properties window click the Environment Variables button.
In the Environment Variables window, highlight the Path variable in the Systems Variable section and click the Edit button. In the PATH variable each search path is separated with a semicolon.
At the end of the line add the following directories:
- D:\projects\opencv\build\x86\vc10\bin
- D:\projects\opencv\build\common\tbb\ia32\vc10

So you have to append a line like this to the existing PATH:

;D:\projects\opencv\build\x86\vc10\bin;D:\projects\opencv\build\common\tbb\ia32\vc10

Windows 2000 and Windows XP users

From the Desktop, right-click My Computer and click Properties.
In the System Properties window, click on the Advanced tab.
In the Advanced section, click the Environment Variables button.
In the Environment Variables window, highlight the Path variable in the Systems Variable section and click the Edit button. In the PATH variable each search path is separated with a semicolon.
At the end of the line add the following directories:
- D:\projects\opencv\build\x86\vc10\bin
- D:\projects\opencv\build\common\tbb\ia32\vc10

So you have to append a line like this to the existing PATH:

;D:\projects\opencv\build\x86\vc10\bin;D:\projects\opencv\build\common\tbb\ia32\vc10

Getting libfacerec

Next you'll need to download libfacerec, which is available from:

http://www.github.com/bytefish/libfacerec

If you don't know how to clone a github repository with git, then you can download the source from:

https://github.com/bytefish/libfacerec/downloads (as zip, as tar)

Then extract the zip or tar.gz file to a folder of your choice, I am using D:\downloads\libfacerec in this tutorial. If you are using an other folder, then just keep it consistent.

Setting up Visual Studio

You'll now learn how to configure a Microsoft Visual Studio 10 C++ project with OpenCV. Parts might resemble the post by karlphilipp, which also explains how to configure OpenCV 2.3 on Visual Studio 2010. I won't reinvent the wheel, so you probably want to read hist post at:

http://stackoverflow.com/a/7014918/513875

The difference is, that I've decided to use pre-built Visual Studio 2010 (VC 10) libraries from the superpack. And I mention some pitfalls you'll encounter when compiling libfacerec.

Create an Empty project

Start Visual Studio and from the main menu select "File -> New Project...". Then create an "Empty Project", name it libfacerec and store it in D:\projects:

Adding the libfacerec headers and sources

First of all the libfacerec headers and sources are added to our solution. For this copy the libfacerec header files from your extracted archive in D:\downloads\libfacerec\include to where your the solution is located in D:\projects\libfacerec\libfacerec. Also copy the source files from D:\downloads\libfacerec\src to D:\projects\libfacerec\libfacerec.

Now add the files to the Microsoft Visual Studio solution. In the Solution Explorer right click on Header Files and select Add -> Existing Item...", then select all .hpp files you've just copied. Do the same for the source files and right click on Source Files and select all .cpp files.

Your project now looks like this:

Additional Include Directories

Your project doesn't compile yet, because we didn't configure OpenCV. The first step is to set the additional include directories for our project. Go to the Project Properties by either right click on your project and select Properties or by pressing Alt + F7, in the new window do the following:

From the Configuration box (combo box in the top left), select All Configurations
Open Configuration Properties -> C/C++ -> General and edit the field Additional Include Directories to add these 3 include paths:
- D:\projects\opencv\build\include
- D:\projects\opencv\build\include\opencv
- D:\projects\opencv\build\include\opencv2

Additional Library Directories and Additional Dependencies

Then we need to add the libraries to build against. Again in the Project Properties Window do the following:

Add the path of the OpenCV libraries by Configuration Properties -> Linker -> General and on the Additional Library Directories field add:
- D:\projects\opencv\build\x86\vc10\lib

Then go to Configuration Properties -> Linker -> Input and in the Additional Dependencies field add:

opencv_highgui231.lib
opencv_core231.lib
opencv_imgproc231.lib

Build libfacerec

Finally libfacerec can be built. We are linking against the libraries:

opencv_core231.lib
opencv_highgui231.lib
opencv_imgproc231.lib

Those libraries have no Debug symbols, so we build libfacerec with the Release Configuration:

Then build the project and its executable by simply selecting Build -> Build Solution (or F7) from the Menu. And you should see:

Congratulations!

Running the Demo

Now to the Eigenfaces Demo. You need some data to make the examples work, sorry but I really can't include those face databases in my repository. First of all this. In the demo I have decided to read the images from a very simple CSV file. Why? Because it's the simplest platform-independent approach I can think of. However, if you know a simpler solution please ping me about it. Basically all the CSV file needs to contain are lines composed of a filename followed by a ; followed by the label (as integer number), making up a line like this:

/path/to/image.ext;0

Think of the label as the subject (the person) this image belongs to, so same subjects (persons) should have the same label. So let's make up an example. Download the AT&T Facedatabase from AT&T Facedatabase and the corresponding CSV file from at.txt, which looks like this (file is without ... of course):

./at/s1/1.pgm;0
./at/s1/2.pgm;0
...
./at/s2/1.pgm;1
./at/s2/2.pgm;1
...
./at/s40/1.pgm;39
./at/s40/2.pgm;39

Imagine I have extracted the files to D:/data/at and have downloaded the CSV file to D:/data/at.txt. Then I would simply Search & Replace ./ with D:/data/. You can do that in an editor of your choice, every sufficiently advanced editor can do this. Once you have a CSV file with valid filenames and labels, you can run the demo by with the path to the CSV file as parameter:

D:\projects\libfacerec\Release\libfacerec.exe D:/data/at.txt

and you should see (Note I have switched to the Eigenfaces model in the demo. Latest version in trunk is Fisherfaces!):

Conclusion

So I can finally answer the question of all the mails I got: Yes, I am pretty sure all this also works with Microsoft Visual Studio.

Aligning face images

Philipp Wagner — Tue, 10 Apr 2012 21:58:00 +0200

Aligning face images

Some people have asked me how I've aligned the face images in my articles. Giving people my ImageMagick hack is embarrassing for me, so I've decided to rewrite it into a Python script. You don't need to copy and paste it, as the script comes with the source folder of my Guide on Face Recognition: crop_face.py.

The code is really easy to use. To scale, rotate and crop the face image you just need to call CropFace(image, eye_left, eye_right, offset_pct, dest_sz), where:

eye_left is the position of the left eye
eye_right is the position of the right eye
offset_pct is the percent of the image you want to keep next to the eyes (horizontal, vertical direction)
dest_sz is the size of the output image

If you are using the same offset_pct and dest_sz for your images, they are all aligned at the eyes.

example

Imagine we are given this photo of Arnold Schwarzenegger, which is under a Public Domain license. The (x,y)-position of the eyes are approximately (252,364) for the left and (420,366) for the right eye. Now you only need to define the horizontal offset, vertical offset and the size your scaled, rotated & cropped face should have. Here are some examples:

Configuration	Cropped, Scaled, Rotated Face
0.1 (10%), 0.1 (10%), (200,200)
0.2 (20%), 0.2 (20%), (200,200)
0.3 (30%), 0.3 (30%), (200,200)
0.2 (20%), 0.2 (20%), (70,70)

crop_face.py

#!/usr/bin/env python
# Software License Agreement (BSD License)
#
# Copyright (c) 2012, Philipp Wagner
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above
#    copyright notice, this list of conditions and the following
#    disclaimer in the documentation and/or other materials provided
#    with the distribution.
#  * Neither the name of the author(s) nor the names of its
#    contributors may be used to endorse or promote products derived
#    from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

import sys, math, Image

def Distance(p1,p2):
  dx = p2[0] - p1[0]
  dy = p2[1] - p1[1]
  return math.sqrt(dx*dx+dy*dy)

def ScaleRotateTranslate(image, angle, center = None, new_center = None, scale = None, resample=Image.BICUBIC):
  if (scale is None) and (center is None):
    return image.rotate(angle=angle, resample=resample)
  nx,ny = x,y = center
  sx=sy=1.0
  if new_center:
    (nx,ny) = new_center
  if scale:
    (sx,sy) = (scale, scale)
  cosine = math.cos(angle)
  sine = math.sin(angle)
  a = cosine/sx
  b = sine/sx
  c = x-nx*a-ny*b
  d = -sine/sy
  e = cosine/sy
  f = y-nx*d-ny*e
  return image.transform(image.size, Image.AFFINE, (a,b,c,d,e,f), resample=resample)

def CropFace(image, eye_left=(0,0), eye_right=(0,0), offset_pct=(0.2,0.2), dest_sz = (70,70)):
  # calculate offsets in original image
  offset_h = math.floor(float(offset_pct[0])*dest_sz[0])
  offset_v = math.floor(float(offset_pct[1])*dest_sz[1])
  # get the direction
  eye_direction = (eye_right[0] - eye_left[0], eye_right[1] - eye_left[1])
  # calc rotation angle in radians
  rotation = -math.atan2(float(eye_direction[1]),float(eye_direction[0]))
  # distance between them
  dist = Distance(eye_left, eye_right)
  # calculate the reference eye-width
  reference = dest_sz[0] - 2.0*offset_h
  # scale factor
  scale = float(dist)/float(reference)
  # rotate original around the left eye
  image = ScaleRotateTranslate(image, center=eye_left, angle=rotation)
  # crop the rotated image
  crop_xy = (eye_left[0] - scale*offset_h, eye_left[1] - scale*offset_v)
  crop_size = (dest_sz[0]*scale, dest_sz[1]*scale)
  image = image.crop((int(crop_xy[0]), int(crop_xy[1]), int(crop_xy[0]+crop_size[0]), int(crop_xy[1]+crop_size[1])))
  # resize it
  image = image.resize(dest_sz, Image.ANTIALIAS)
  return image

if __name__ == "__main__":
  image =  Image.open("arnie.jpg")
  CropFace(image, eye_left=(252,364), eye_right=(420,366), offset_pct=(0.1,0.1), dest_sz=(200,200)).save("arnie_10_10_200_200.jpg")
  CropFace(image, eye_left=(252,364), eye_right=(420,366), offset_pct=(0.2,0.2), dest_sz=(200,200)).save("arnie_20_20_200_200.jpg")
  CropFace(image, eye_left=(252,364), eye_right=(420,366), offset_pct=(0.3,0.3), dest_sz=(200,200)).save("arnie_30_30_200_200.jpg")
  CropFace(image, eye_left=(252,364), eye_right=(420,366), offset_pct=(0.2,0.2)).save("arnie_20_20_70_70.jpg")

Colormaps in OpenCV

Philipp Wagner — Fri, 25 Nov 2011 12:23:00 +0100

Colormaps in OpenCV

introduction

It's a fact, that human perception isn't built for observing fine changes in grayscale images. Human sensors (eyes and stuff) are more sensitive to observing changes between colors, so you often need to recolor your grayscale images to get a clue about them. Yesterday I needed to visualize some data in OpenCV. And while it's easy to apply a colormap in MATLAB, OpenCV doesn't come with predefined colormaps. I don't have much knowledge about colormaps and I am totally off when it comes to choosing colors anyway (my former art teacher will acknowledge), but there are people with a lot more experience. One page I've found is the blog of Matteo Niccoli, who shares some colormaps on his FEX page, along with a lot of interesting links.

In the end I simply took the interpolation steps from the GNU Octave colormaps and turned them into C++ classes. The source code is on github:

https://github.com/bytefish/colormaps-opencv

Please note: This code has been contributed to OpenCV and is available in the contrib module since OpenCV 2.4. Simply use cv::applyColorMap, the same way as described in this article (Official documentation).

using the code

You only need cv::applyColorMap to apply a colormap on a given image. The signature of cv::applyColorMap is:

void applyColorMap(InputArray src, OutputArray dst, int colormap)

The following code example reads an image (given as a command line argument), applies a Jet colormap on it and shows the result:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include "colormap.hpp"

using namespace cv;

int main(int argc, const char *argv[]) {
    // Get the path to the image, if it was given
    // if no arguments were given.
    string filename;
    if (argc > 1) {
        filename = string(argv[1]);
    }
    // The following lines show how to apply a colormap on a given image
    // and show it with cv::imshow example with an image. An exception is
    // thrown if the path to the image is invalid.
    if(!filename.empty()) {
        Mat img0 = imread(filename);
        // Throw an exception, if the image can't be read:
        if(img0.empty()) {
            CV_Error(CV_StsBadArg, "Sample image is empty. Please adjust your path, so it points to a valid input image!");
        }
        // Holds the colormap version of the image:
        Mat cm_img0;
        // Apply the colormap:
        applyColorMap(img0, cm_img0, COLORMAP_JET);
        // Show the result:
        imshow("cm_img0", cm_img0);
        waitKey(0);
    }

    return 0;
}

The available colormaps are:

enum
{
    COLORMAP_AUTUMN = 0,
    COLORMAP_BONE = 1,
    COLORMAP_JET = 2,
    COLORMAP_WINTER = 3,
    COLORMAP_RAINBOW = 4,
    COLORMAP_OCEAN = 5,
    COLORMAP_SUMMER = 6,
    COLORMAP_SPRING = 7,
    COLORMAP_COOL = 8,
    COLORMAP_HSV = 9,
    COLORMAP_PINK = 10,
    COLORMAP_HOT = 11
}

And here are the scales corresponding to each of the maps, so you know what they look like:

Name	Scale
Autumn
Bone
Cool
Hot
HSV
Jet
Ocean
Pink
Rainbow
Spring
Summer
Winter

Demo

The demo coming with the cv::applyColorMap shows how to use the colormaps and how I've created the scales you have seen:

https://github.com/bytefish/colormaps-opencv/blob/master/src/main.cpp

If you want to apply a Jet colormap on a given image, then call the demo with:

./cm_demo /path/to/your/image.ext

Or if you just want to generate the color scales, then call the demo with:

./cm_demo

Note: Replace ./cm_demo with cm_demo.exe if you are running Windows!

Building the demo

The project comes as a CMake project, so building it is as simple as:

philipp@mango:~/some/dir/colormaps-opencv$ mkdir build
philipp@mango:~/some/dir/colormaps-opencv$ cd build
philipp@mango:~/some/dir/colormaps-opencv/build$ cmake ..
philipp@mango:~/some/dir/colormaps-opencv/build$ make
philipp@mango:~/some/dir/colormaps-opencv/build$ ./cm_demo path/to/your/image.ext

Or if you use MinGW on Windows:

C:\some\dir\colormaps-opencv> mkdir build
C:\some\dir\colormaps-opencv> cd build
C:\some\dir\colormaps-opencv\build> cmake -G "MinGW Makefiles" ..
C:\some\dir\colormaps-opencv\build> mingw32-make
C:\some\dir\colormaps-opencv\build> cm_demo.exe /path/to/your/image.ext

Or if you are using Visual Studio, you can generate yourself a solution like this (please adjust to your version):

C:\some\dir\colormaps-opencv> mkdir build
C:\some\dir\colormaps-opencv> cd build
C:\some\dir\colormaps-opencv\build> cmake -G "Visual Studio 9 2008" ..

Skin Color Thresholding with OpenCV

Philipp Wagner — Tue, 15 Nov 2011 18:30:00 +0100

Skin Color Thresholding with OpenCV

This snippet implements common Skin Color Thresholding rules taken from:

Nusirwan Anwar bin Abdul Rahman, Kit Chong Wei and John See. RGB-H-CbCr Skin Colour Model for Human Face Detection. (Online available)

Works good for my test data:

/*
 * Copyright (c) 2011. Philipp Wagner <bytefish[at]gmx[dot]de>.
 * Released to public domain under terms of the BSD Simplified license.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *   * Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *   * Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *   * Neither the name of the organization nor the names of its contributors
 *     may be used to endorse or promote products derived from this software
 *     without specific prior written permission.
 *
 *   See <http:www.opensource.org/licenses/bsd-license>
 */

#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <iostream>

using namespace cv;

using std::cout;
using std::endl;

bool R1(int R, int G, int B) {
    bool e1 = (R>95) && (G>40) && (B>20) && ((max(R,max(G,B)) - min(R, min(G,B)))>15) && (abs(R-G)>15) && (R>G) && (R>B);
    bool e2 = (R>220) && (G>210) && (B>170) && (abs(R-G)<=15) && (R>B) && (G>B);
    return (e1||e2);
}

bool R2(float Y, float Cr, float Cb) {
    bool e3 = Cr <= 1.5862*Cb+20;
    bool e4 = Cr >= 0.3448*Cb+76.2069;
    bool e5 = Cr >= -4.5652*Cb+234.5652;
    bool e6 = Cr <= -1.15*Cb+301.75;
    bool e7 = Cr <= -2.2857*Cb+432.85;
    return e3 && e4 && e5 && e6 && e7;
}

bool R3(float H, float S, float V) {
    return (H<25) || (H > 230);
}

Mat GetSkin(Mat const &src) {
    // allocate the result matrix
    Mat dst = src.clone();

    Vec3b cwhite = Vec3b::all(255);
    Vec3b cblack = Vec3b::all(0);

    Mat src_ycrcb, src_hsv;
    // OpenCV scales the YCrCb components, so that they
    // cover the whole value range of [0,255], so there's
    // no need to scale the values:
    cvtColor(src, src_ycrcb, CV_BGR2YCrCb);
    // OpenCV scales the Hue Channel to [0,180] for
    // 8bit images, so make sure we are operating on
    // the full spectrum from [0,360] by using floating
    // point precision:
    src.convertTo(src_hsv, CV_32FC3);
    cvtColor(src_hsv, src_hsv, CV_BGR2HSV);
    // Now scale the values between [0,255]:
    normalize(src_hsv, src_hsv, 0.0, 255.0, NORM_MINMAX, CV_32FC3);

    for(int i = 0; i < src.rows; i++) {
        for(int j = 0; j < src.cols; j++) {

            Vec3b pix_bgr = src.ptr<Vec3b>(i)[j];
            int B = pix_bgr.val[0];
            int G = pix_bgr.val[1];
            int R = pix_bgr.val[2];
            // apply rgb rule
            bool a = R1(R,G,B);

            Vec3b pix_ycrcb = src_ycrcb.ptr<Vec3b>(i)[j];
            int Y = pix_ycrcb.val[0];
            int Cr = pix_ycrcb.val[1];
            int Cb = pix_ycrcb.val[2];
            // apply ycrcb rule
            bool b = R2(Y,Cr,Cb);

            Vec3f pix_hsv = src_hsv.ptr<Vec3f>(i)[j];
            float H = pix_hsv.val[0];
            float S = pix_hsv.val[1];
            float V = pix_hsv.val[2];
            // apply hsv rule
            bool c = R3(H,S,V);

            if(!(a&&b&&c))
                dst.ptr<Vec3b>(i)[j] = cblack;
        }
    }
    return dst;
}


int main(int argc, const char *argv[]) {
    // Get filename to the source image:
    if (argc != 2) {
        cout << "usage: " << argv[0] << " <image.ext>" << endl;
        exit(1);
    }
    // Load image & get skin proportions:
    Mat image = imread(argv[1]);
    Mat skin = GetSkin(image);

    // Show the results:
    namedWindow("original");
    namedWindow("skin");

    imshow("original", image);
    imshow("skin", skin);

    waitKey(0);
    return 0;
}

http://bytefish.de

Getting a Huawei E352s-5 to work with Linux

getting the Huawei E352s-5 to work with linux

switching into modem mode

udev rules

final note

useful links

Website Update

Website Update

Validating Algorithms

Validating Algorithms

introduction

validating algorithms

Getting the image data right

putting it together

validation.py

running the script

conclusion

OpenCV FaceRecognizer documentation

OpenCV FaceRecognizer documentation

Fisherfaces

Fisherfaces

Algorithmic Description

Implementation Details

Experiments

Validation scheme

k-fold Cross Validation

Leave One Out Cross Validation

Yale Facedatabase A

Data Preprocessing

Eigenfaces

Fisherfaces

Conclusion

Celebrities Dataset

Recognition

Who Am I?

Working with the GNU Octave version

Conclusion and Lessons learned

Working with NumPy arrays

OOP in GNU Octave

Further reading

Extracting Contours with OpenCV

Extracting Contours with OpenCV

OpenCV, Microsoft Visual Studio and libfacerec

OpenCV, Microsoft Visual Studio and libfacerec

Setting up OpenCV

Windows Vista and Windows 7 users

Windows 2000 and Windows XP users

Getting libfacerec

Setting up Visual Studio

Create an Empty project

Adding the libfacerec headers and sources

Additional Include Directories

Additional Library Directories and Additional Dependencies

Build libfacerec

Running the Demo

Conclusion

Aligning face images

Aligning face images

example

crop_face.py

Colormaps in OpenCV

Colormaps in OpenCV

introduction

using the code

Demo

Building the demo

Skin Color Thresholding with OpenCV

Skin Color Thresholding with OpenCV