EmgCV Haar Classifiers Face and Facial Parts Detection

Hello folks..
after so many days of absence; im back with another emguCV project... Its about using haar classifier or commonly known as Haar Cascades in emguCV.

Those who doesn't kn ow much about haar classifiers well Essentially it's a variation on a visual Neural Network recognizer that breaks up the image into pieces that contain a specific feature (nose, mouth, etc) to improve accuracy over a full image neural network that might be confused by the object of interest's orientation or position within the image.

To know more about haar classifiers; follow that link after all who wants to get into all these definitions and bla bla bla ... :P at least I don't.. :P lets do some practical work man..!!!

I've borken the whole project in following steps

• emguCv
• Set Variables
• Haar Cascade Files
• Capute Video

1. emguCV

EmguCV is an openCV wrapper for .Net enviornment; right now, its only available for C#. You can download it from emguCV website. After downloading, include emguCV.dll in your project reference and add following references in your form

 using Emgu.CV;  
 using Emgu.Util;  
 using Emgu.CV.Structure;  
 using Emgu.CV.CvEnum;  
 using Emgu.CV.UI;  

2. Set Variables

Now set the private variables that we'll use throughout the project

   public partial class frmCapture : Form  
   {  
     private Capture cap; // used to caprute image from device  
     private HaarCascade haar, LEye, REeye, Mouth, Nose, hand; //haar classifiers for different parts  
     private Bitmap bmp, leftE, rightE, faceCrop, Mouthb; //bitmaps  
     private Bitmap facee; //bitmap of face  
     private Image<Gray, byte> MouthImg, LeyeImg, ReyeImg, Noseb; //cropped areas of face  
     private Rectangle CropArea;   
     public frmCapture()  
     {  
       InitializeComponent();  
     }  

3. Haar Cascade Files

You have to load haar cascade files now; they're nothing but simple xml files but pretty long having data about every facial part

 private void frmCapture_Load(object sender, EventArgs e)  
     {  
       cap = new Capture(0);  // device captutre; 0 is for default device
       haar = new HaarCascade("haarcascade_frontalface_alt.xml");   //haar cascade for face
       LEye = new HaarCascade("ojoI.xml");     //haar cascade for left eye
       REeye = new HaarCascade("ojoD.xml");     //haar cascade for right eye
       Mouth = new HaarCascade("Mouth.xml");     //haar cascade for mouth
       Nose = new HaarCascade("Nariz.xml");     //haar cascade for nose
       hand = new HaarCascade("A_gest.xml");     //haar cascade for hand
     }  

4. Capture Video

For video capturing; im using conventional technique of placing my code in timer control's timer_tick function

  //I've used timer control for continious detection.  
     private void timer1_Tick(object sender, EventArgs e)  
     {  
       //use "using" for better memory performance.  
       using (Image<Bgr, byte> nextframe = cap.QueryFrame()) //cap.QueryFrame() automatically detects default capture device.  
       {  
         if (nextframe != null)  
         {  
           Image<Gray, byte> grayframe = nextframe.Convert<Gray, byte>();  
           grayframe._EqualizeHist();  
           //Detect Face  
           var faces = grayframe.DetectHaarCascade(haar, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING | HAAR_DETECTION_TYPE.FIND_BIGGEST_OBJECT, new Size(nextframe.Width / 8, nextframe.Height / 8))[0];   
           var nose = grayframe.DetectHaarCascade(Nose, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(nextframe.Width / 24, nextframe.Height / 12))[0];  
           var hands = grayframe.DetectHaarCascade(hand, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(24,24))[0];  
           facee = new Bitmap(nextframe.Width, nextframe.Height);  
           facee = nextframe.ToBitmap();  
           foreach (var han in hands)  
           {  
             nextframe.Draw(han.rect, new Bgr(0, 10, 10), 3);  
           }  
           foreach (var face in faces)  
           {  
             nextframe.Draw(face.rect, new Bgr(0, double.MaxValue, 0), 3);  
             CropArea = new Rectangle(face.rect.X, face.rect.Y, face.rect.Width, face.rect.Height); //Crop face from image  
             faceCrop = facee.Clone(CropArea, facee.PixelFormat);  
           }  
           if (faceCrop != null)  
           {  
             Image<Bgr, byte> facePic = new Image<Bgr, byte>(faceCrop);  
             Image<Gray, byte> grayframeface = facePic.Convert<Gray, byte>();  
             //Detect Mouth  
             // it is detected from lower part of cropped face image  
             CropArea = new Rectangle(0, faceCrop.Height/2, faceCrop.Width, faceCrop.Height / 2);  
             bmp = new Bitmap(faceCrop.Width, faceCrop.Height / 2);  
             bmp = faceCrop.Clone(CropArea, faceCrop.PixelFormat);  
             Image<Bgr, byte> mouthpic = new Image<Bgr, byte>(bmp);  
             Image<Gray, byte> grayframemouth = mouthpic.Convert<Gray, byte>();  
             var mouth = grayframemouth.DetectHaarCascade(Mouth, 1.4, 4, HAAR_DETECTION_TYPE.SCALE_IMAGE | HAAR_DETECTION_TYPE.FIND_BIGGEST_OBJECT, new Size(grayframemouth.Width / 4, grayframemouth.Height / 4))[0];  
             foreach (var mou in mouth)  
             {  
                 grayframemouth.Draw(mou.rect,new Gray(), 2);  
                 CropArea = new Rectangle(mou.rect.X, mou.rect.Y, mou.rect.Width, mou.rect.Height);  
                 MouthImg = grayframemouth.Copy(CropArea);  
             }  
             //Detect Right Eye  
             // It is detected from upper right cropped part of face cropped image  
             CropArea = new Rectangle(0, 0, faceCrop.Width / 2, faceCrop.Height / 2);  
             bmp = new Bitmap(faceCrop.Width / 2, faceCrop.Height / 2);  
             bmp = faceCrop.Clone(CropArea, faceCrop.PixelFormat);  
             Image<Bgr, byte> ReyePic = new Image<Bgr, byte>(bmp);  
             Image<Gray, byte> greyreyepic = ReyePic.Convert<Gray, byte>();  
             var righteye = greyreyepic.DetectHaarCascade(REeye, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(ReyePic.Width/8,ReyePic.Height/8))[0];  
             foreach (var eyer in righteye)  
             {  
                 greyreyepic.Draw(eyer.rect, new Gray(), 2);  
                 CropArea = new Rectangle(eyer.rect.X, eyer.rect.Y, eyer.rect.Width, eyer.rect.Height);  
                 // greyreyepic.ROI = CropArea;  
                 ReyeImg = greyreyepic.Copy(CropArea);  
             }  
             //Detect Left Eye  
             CropArea = new Rectangle(faceCrop.Width / 2, 0, faceCrop.Width / 2, faceCrop.Height / 2);  
             bmp = new Bitmap(faceCrop.Width / 2, faceCrop.Height / 2);  
             bmp = faceCrop.Clone(CropArea, faceCrop.PixelFormat);  
             Image<Bgr,byte> LeyePic = new Image<Bgr, byte>(bmp);  
             Image<Gray, byte> greyleyepic = LeyePic.Convert<Gray, byte>();  
             var lefteye = greyleyepic.DetectHaarCascade(LEye, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(greyleyepic.Width / 8, greyleyepic.Height / 8))[0];  
           foreach (var eyel in lefteye)  
           {  
             greyreyepic.Draw(eyel.rect, new Gray(), 2);  
             CropArea = new Rectangle(eyel.rect.X, eyel.rect.Y, eyel.rect.Width, eyel.rect.Height);  
            //  greyleyepic.ROI = CropArea;  
             LeyeImg = greyleyepic.Copy(CropArea);  
           }  
             //Detect Nose  
           foreach (var nou in nose)  
           {  
             nextframe.Draw(nou.rect,new Bgr(Color.Black), 2);  
           }  
          MainVideo.Image = nextframe.ToBitmap(); ;  
          if (grayframeface != null)  
          {  
            piccropface.Image = grayframeface.ToBitmap();  
          }  
          bmp.Dispose();  
          if (grayframemouth != null)  
          {  
            cropmouth.Image = grayframemouth.ToBitmap();  
          }  
          if (greyreyepic != null)  
          {  
            piccropreye.Image = greyreyepic.ToBitmap();  
          }  
          if (greyleyepic != null)  
          {  
            piccropleye.Image = greyleyepic.ToBitmap();  
          }  
          if (MouthImg != null)  
          {  
            // picmouth.Image = Mouthb;  
            picmouth.Image = MouthImg.ToBitmap();  
          }  
          if (ReyeImg != null)  
          {  
            //   picrighteye.Image = rightE;  
            picrighteye.Image = ReyeImg.ToBitmap();  
          }  
          if (LeyeImg != null)  
          {  
            //  piclefteye.Image = leftE;  
            piclefteye.Image = LeyeImg.ToBitmap();  
          }  
          if (Noseb != null)  
          {  
            picnose.Image = Noseb.ToBitmap();  
          }  
         }  
         }  
       }  
 }  

Now let me tell you guys how the code actually work.. Al tough the code is self explanatory but still few things has to be discussed. First we capture and image from camera,convert it into gray scale and equalize histogram for better performance. Now see the code below

 var faces = grayframe.DetectHaarCascade(haar, 1.4, 4, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING | HAAR_DETECTION_TYPE.FIND_BIGGEST_OBJECT, new Size(nextframe.Width / 8, nextframe.Height / 8))[0];   

The above code loads haar classifiers from our xml file and matches the gray scale image with it. We can also pass our properties like expected width & height of our face in image that'll make out face detection more precise and faster. We do similar things for nose, mouth and eyes..

To make visible the part of image where a face is detected; we draw a rectangle on that part of image with the code below

 foreach (var face in faces)  
           {  
             nextframe.Draw(face.rect, new Bgr(0, double.MaxValue, 0), 3);  
             CropArea = new Rectangle(face.rect.X, face.rect.Y, face.rect.Width, face.rect.Height); //Crop face from image  
             faceCrop = facee.Clone(CropArea, facee.PixelFormat);  
           }  

After we've find the face; we crop it from image and perform the haar classifier matching for nose, eyes and mouth on that part.. Father, we've also divide our face cropped image in three parts; upper left part will specifically detect for left eye; upper right part will detect for right eye while the whole lower part will be used to detect mouth.. This'll make our code run faster..

5. Screen Mockup

Last but not the least.. haar classifiers are best for working with static images. They doesn't give good result with live pattern matching. So if your'e working with static images; it'll work like magic, but for live videos etc i'd prefer CAMSHIFT technique.. Your guys can check out my blog on CAMSHIFT here.


Don't forget to comment.. Suggestions are most welcome :)

Jazak Allah..