Nimble

Using the nimble approach to grocery detection uses what is called non parametric density estimation to determine the probability that a detected key point is in the image. Given a mosaic as below:



We compute what looks like a heat map for each class:

Mosaicing

So as to deal with the computation time we have applied optical flow to make sure that we only process a very small set of new pixels at each time step. This drastically reduces computation time, and as you can see still produces a good set of features.



Next step: start applying some detection algorithms to these points.



As you can see I got the old cascade working on it, just i forgot to make it output its final surf homographies, still works thou.

Full system!!

So far all this has been about running and training a bunch of different algorithms for the hope that we could one day apply them to actually detecting groceries.



Note that the video is not actually running in real time.

Data Update

So I worked with my code some more, got a few more promising results from HAAR, but still nothing amazing. The results can be seen here

Update

For the last 2 weeks i have been refactoring my code and getting it to run en mass on all the different grozi images. I have finished my first full run of all the data, it all runs on many machines and can be done in around 6 hrs!! The first pass through the data is promising for surf, but rather depressing for HAAR . I found a bug in my code that probably makes it very unlikely for it to detect anything. Thus, I am in the process of retraining. As far as the SURF data goes, its pretty promising, in general its very good for objects that have a clear number of really strong features. though some items that are rather small or are easily deformed are not nearly as easily detected.
http://grozi.calit2.net/grozi.html

After the data is done training, I intend on moving forward on building out the final system. This will basically involve incorporating the first level classifier that I am getting from the project in cse 254 and then running detection on that. Assuming that we get reasonable performance with haar for an item it would be best to cascade from chm->haar-> sift as the total time to detection would be something near 1 sec using all 3 approaches.

Meanwhile, I am chugging along with my project in yoav's class in trying to determine the presence or absence of items. Go there to see more of which direction we are going with that.

Week 2 update

Progress report:
As the project currently stands I have integrated into my framework 2 Algorithms for detecting objects: SURF and Ada boost using HAAR-like features, i am also well on my way to applying FERNS.

HAAR

The entire HAAR algorithm was already implemented in Open CV 2.0, all that was needed was to provide the training data. I made a small modification to the application, taking a hint from face recognition, and implemented sub classifiers that are capable of detecting object specific items (like the tide logo alone, or another area of interest for the color histogram).













The algorithm runs very very fast, it takes mere milliseconds to detect, but its not terribly
accurate














It still detects tide reasonably well, rejecting only a handful of frames (less than 20%).

SURF

SURF is a keypoint detector similar to SIFT. I use open CV's implementation to extract keypoints and then feed them into the fast linear approximate nearest neighbor library to speed up detection. A keypoint is recognized if the distance between the 2 nearest neighbors is less than .6. This technique was traditionally applied to SIFT features, and works quite well to the SURF features.

The algorithm runs really slow(ironically), but has a FP rate of 0.0 on the data set I'm using.




Another neat thing is that most of the time you get a good homography to go along with it. This could surely help with tracking.




FERNS

I am also currently trying to apply Ferns, essentially another keypoint detector using simplified randomized trees. All the following images are from their demo software.


It performs well enough in my tests, but i have to mess with the algorithm to get it working in my framework














Note that this algorithm runs fast, but has some problems. Its detection phase runs fast, but still slows down to about 2 fps in detection. In the below pictures, the circles represent features picked out by the algorithm, darker circles represent recognized keypoints, lighter are not recognized.















Also notice that the homography isn't always as fantastic as with surf,
















Still on the whole it shows promise for being fast and pretty accurate.

Introduction

See my project proposal and the course website
also useful:
main grozi site
previous paper on the grozi 120 dataset

Overall the goal of this project is to develop a system capable of effectively detecting an object in a scene from the grocery store, given a set list of items. more to come soon!