UAV Targetting System

Final Presentation

2008-03-17T10:10:00.001-07:00

Presentation

2008-03-12T12:09:00.000-07:00

Location

2008-03-10T11:05:00.000-07:00

I have also over the weekend tried to implement a location based tool that allows me to visualise where the image targets found think that they actually are. I simply took the best matching window and plotted a red square around it to ensure that the chosen location can be visualised. also the Co-ordinates of this location have been taken from the center of the chosen window and printed to the output file.

To get the distance from the center we take arc tan of the field of view times the altitude to get the distance the whole image covers...then on a pixel distance we can decide how far from the center which will be the gps location of the plane (approx).

Image location boxes can be viewed below:

Blurred Test Solves Several Cases

2008-03-10T10:12:00.000-07:00

Testing the blurred images again i found a human error in my code due to reusing code from elsewhere in the program. It is now fixed and actually shows a good improvement in that all images are classified as Present if they truly are present. I have yet to see a test/sample case where the images have been incorrectly classified.

Also with the new blurred image code the images are now actually picking up previous blurred images too. Showing that a careful selection of varying conditions from illumination to blurred tests will give results that a usable throughout the system. Ideally a larger set of training data would be used rather than the minimal 12 training images currently working with, yet this gives reasonable results to the minimum degree of accuracy required...100% coverage of all target images, yet still allows through a few similar images that are not truly targets (i.e true negatives).



2: 0.119070 Present - Normal
3: 0.128092 Present - Normal
4: 0.082883 Present - Normal
5: 0.131328 Present - Normal
6: 0.072161 Present - Dark
7: 0.053662 Present - Normal
8: 0.159959 Present - Normal
9: 1.000000 Empty
10: 0.113700 Present - Normal
11: 0.156434 Present - Normal
12: 1.000000 Empty
13: 0.045220 Present - Normal
14: 0.077655 Present - Blurred Test        //Old Blurry image now found
15: 0.099331 Present - Blurred Test        //Old Blurry image now found
16: 1.000000 Empty
17: 1.000000 Empty
18: 1.000000 Empty
19: 0.181530 Present - Normal
20: 0.187497 Present - Normal
21: 0.183492 Present - Normal
22: 0.154783 Present - Normal              //Non blurry version of image
23: 0.140616 Present - Normal              //Non blurry version of image
24: 0.163971 Present - Normal              //New Blurry image found as a normal instead
25: 0.095063 Present - Blurred Test        //New Blurry image found

Blurred images Results

2008-03-10T09:48:00.001-07:00

The blurred image results are quite bizarre. All the cases of blurred targets are found as well as the base cases too. These blurred images though are not linked to the blurred test images however but are classified by the light and normal image sets....very peculiar so i am now currently trying to debug the cases and see what the results are just for the blurred cases and what the return value is in this situation.

(Trying to add pictures but google has a network error)...

Ahh i solved it another way...apparently google died this morning.

Improvements To Illumination Variation

2008-03-05T14:22:00.000-08:00

I have made some changes to the base images that are used for the illumination variation, however i have found that the system required some tweaking in the range of what is classified as a target as often i have found that the avc difference in chi squared between the tests is too strick, to loosen this i have simply drawn a boundary of either the average chi squared or 0.1 which is the larger.

By doing so i get a better classification result for the images.

AVC: 0.191358 AVD: 0.032299 AVL: 0.014278


2: 0.109081 Present - Normal
3: 0.045300 Present - Light
4: 0.022777 Present - Light
5: 0.058292 Present - Light
6: 0.083343 Present - Dark
7: 0.030796 Present - Light
8: 0.063477 Present - Light
9: 1.000000 Empty                      //Actually Present just extremely bright.
10: 0.060175 Present - Light
11: 0.167384 Present - Normal
12: 1.000000 Empty                     //Actually Present just extremely dark.
13: 0.156815 Present - Normal
14: 1.000000 Empty                     //Actually Present images are blurred.
15: 1.000000 Empty                     //Actually Present images are bluured.
16: 1.000000 Empty
17: 1.000000 Empty
18: 1.000000 Empty
19: 0.187468 Present - Normal          //Actually Empty image of bag strap
20: 0.171188 Present - Normal          //Actually Empty image of chemistry book
21: 0.186641 Present - Normal          //Actually Empty image of penny

I see this as a massive improvement. I was always dubious about blurring and now i can see clearly that this is an issue i have still to fix but i hope that with a test of such an image with no blurring it is picked up. This would prove to me that the blurring is the issue and not the classification of that model/style of image.

Immediate Possibilities:

Test Blur Theory Against Non Blurry Images

Check Dark Images for Improvements

Target Location

Varying Illuimnation - Color Constancy Issues

2008-03-05T13:36:00.001-08:00

After spending most of the weekend reading color constancy papers, and finding out things that seem to be quite damaging to my methods i have come across one great big artifact of them. When trying to give my system samples of varying lighting conditions and by artificially creating these as has been done before i have found that the samples although taken under similar conditions with the same camera in virtually the same place in the room have massively different values in the Hue channel thus ruining my results. However i do believe this is due to poor selection of target images, yet it does highlight that if this image had appeared in the live data set rather than the training one it would be misclassified. This is the issue in general with color constancy that human perception does not necesaarily match the machine representation of the image.

The images below are my samples for dark targets of a bottletop. Clearly they all contain something circular but it is hard to see, possibly red etc but not a clear view.

Yet in hue channels we can clearly see that there is a massive difference in these images as the right hand column is completely variant. This i believe to be color constancy or rather the lack of color constancy in effect.

Draft Report

2008-03-03T13:27:00.000-08:00

Draft Report:

The report in its draft stages can be found here...

I lied...

2008-02-26T22:53:00.000-08:00

One image fails with random chance...no idea why. It makes little to no sense to me. All the others look to be consistent.

HSV Based Histogram Analysis of Targets Gives CORRECT Separation

2008-02-26T21:03:00.000-08:00

As the title says, it is working, with the images running against a viable target i am able to separate the actual data perfectly for my sample case. The actual data, believe it or not, personally not keeping my hopes too high is being gathered this weekend. Apaarently the camera fell off the gimball on the flight last time...trust Aeronautical Engineers and you get burned.

Output from my Matlab Code:

1: 0.152674 Empty
2: 0.208643 Empty
3: 0.166598 Empty
4: 0.005949 Present
5: 0.000491 Present
6: 0.138655 Empty
7: 0.188792 Empty
8: 0.203062 Empty
9: 0.977584 Empty
10: 0.177460 Empty
11: 0.136012 Empty
12: 0.204911 Empty
13: 0.111032 Empty

Images 4 and 5 are the 2 with the little red coke lids. At the moment i am partitioning the data based on, if it is a target it must return a chi squared less than 0.1. I have also run a sample on a bunch of targets to get a viable chi squared value and then doubled this limit and compared it against a variety of samples.

2: 0.087182 Present //actual target clear
3: 0.131061 Present //actual target clear
4: 0.114681 Present //actual target clear
5: 0.136188 Present //actual target on different but similar background
6: 0.250049 Empty //actual target on different and dark background
7: 0.065181 Present //actual target clear
8: 0.110614 Present //actual target clear
9: 0.396452 Empty //actual target with flash so very bright
10: 0.117868 Present //actual target clear
11: 0.094940 Present //actual target clear
12: 0.291048 Empty //actual target in the very dark
13: 0.094446 Present //actual target clear
14: 0.282812 Empty //actual target on a table with alot of light
15: 0.230164 Empty //actual target on table with alot of light and blurred
16: 0.738673 Empty //empty image
17: 0.249815 Empty //part of a pizza box
18: 0.275440 Empty //empty image
19: 0.151039 Present //bag strap
20: 0.167269 Present //chemistry book
21: 0.157746 Present //circular penny

Better than before

2008-02-15T11:49:00.000-08:00

I have now updated the histogram to basically select the best chi value it finds from an image. If it is above a certain threshold then i can assume it has something in that portion of the image. However, this does work in the sense that all target values are passed through however some none targets seem to be creeping, i am going to play with the threshold level so that it works dynamically to maybe improve the efficiency of the program.

Currently the images 1-6 are TARGETS and 7-13 are EMPTY. The output from my matlab code gives the following results with no improvements or tweaking to the thresholds;

1: 0.037861 Present
2: 0.257812 Present
3: 0.033023 Present
4: 0.067213 Present
5: 0.094324 Present
6: 0.306998 Present
7: 0.015638 Empty
8: 0.015694 Empty
9: 0.949423 Present
10: 0.015573 Empty
11: 0.010490 Empty
12: 0.030843 Present
13: 0.012793 Empty

Brute Force Histograms

2008-02-14T18:13:00.001-08:00

I have improved a worked out the few obvious bugs with my former "histograms". Now as you can clearly see they are much more correct, and i have also altered my code to allow overlapping windows but they must differ by a window size of at least 1/4 this can be altered but i think for a start it is a good heuristic. It gives a decent coverage however the random generator doesn't seem to like the corners very much, i am not sure why as when i tested this at the start it gave me a variety of numbers from the minimum allowed to the maximum in the range, so i would expect them now maybe its simply a bad case?

The chi squared results though are lower than i had hoped, as you can see from these results they do not have a particular one which shines out as having or containing any useful information in fact the value is quite low and the margins not overly noticeable.

Chi Squared for sample image, larger values bolded.


0.0152         0.0146         0.0097         0.0097
0.0139         0.0143         0.0847         0.0050
0.0107         0.0022         0.0131         0.0086
0.0147         0.0125         0.0031         0.0152
0.0378         0.0108         0.0059         0.0035
0.0134         0.0511         0.0535         0.0062
0.0152         0.0052         0.0141         0.0929
0.0152         0.0470         0.0031         0.0102
0.0045         0.0029         0.0037         0.0071
0.0030         0.0038         0.0073         0.0105
0.0180         0.0050         0.0142         0.0061
0.0025         0.0074         0.0031         0.0138

Histogram Comparisons In Matlab

2008-02-13T14:47:00.000-08:00

I have made some noticeable progress since Monday in that i have devised a system that finds me squares of images depending on a density and window size and for each creates a histogram and compares against a blank one. However i ahve had issues normalising these so that they equal 1. Some issue with the fact matlab doesn't properly bin the data but rather ranges it...explained better by an example image. It is infact the way matlab bins these histograms that ruins my chi squared algorithm.

Integral Histograms

2008-02-11T15:52:00.000-08:00

Integral Histograms:

It has been decided that rgb is actually a useless colorspace for this project so after much playing around HSV has been chosen as an alternative. See bottom of post for examples of the HSI images...

After issues in OpenCv trying to reprogram again in Matlab. Can't really see the difference here, but the last bin is about 1%. Which may not seem substantial but here is the image the histogram was created from.

HSI Image Examples:

Histogram Errors...

2008-02-04T11:09:00.000-08:00

I have debugged my code and compared the histogram outputs to several other methods, i.e matlabs built in histogram functions, plus one histogram function i wrote for CSE166. They all give the save values as what i have in OpenCV so i think that this must just be an artifact of the image?

Meeting With AUVSI Leader Scheduled

2008-01-30T15:17:00.000-08:00

Meeting: I have eventually managed to tie down the leader of the payload section of the AUVSI to a meeting with me on Monday. He has sent me links to some of the papers they used previously. Such as a paper: here it seems to be quite useful from the images displayed, however i cannot find source code to sue based on this and the maths looks quite rigorous. I plan on viewing this over the weekend as well as securing the histogram analysis. I would like to get the previous code from the team to find out exactly how they interacted the system together as i am wary of building up an entire system in one direction to find it is difficult to combine with the system they will be using.

To Complete:
- method for histogram standardization (windows of images)
- classifier based on distance from training data mean and std (distance based classifier)
- read paper in entirety and try and find source code
- attempt to find reasonable method for location the targets within images

Colourspace Histograms

2008-01-30T10:33:00.000-08:00

Colourspace Histogram's: I have managed to create a solution for appending images in open CV that lets me now display the histograms for different colour planes on one image. The results are shown below:

HSI

LAB

RGB

HSI

LAB

RGB

HSI

LAB

RGB

Idea's for Future: Do the same comparisons for hsi and lab with the training set and see if they have any useful comparisions with the chi squared results or if they ahve any cleaner results that may be used. I.e replicate rgb functionality in hsi and lab and compare classification results.

Problems Ran into this Week: so far i've not been able to figure out a way of only creating histograms on a portion of an image, using the set window size. However i think today i may have stumbled upon something called setImageROI which operates only on a selected region of an image, however to do this over an image in the same way as a convolution would be very computational expensive and this first stage is supposed to be a quick check of a present target or not...and even before this inclusion it is already quite slow with the multiple operations being computed.

Colourspace Histograms Created

2008-01-28T20:39:00.001-08:00

The varying colourspace histograms have been created, however, i can only display them as individual histograms for each plane (ie r,g,b or l,a,b or h,s,i) seperately. This is difficult to visualise the properties of each i have found, therefore i wish to find a way of appending images together using OpenCv before publishing the results here.

If i do not find a way by wednesday i shall manually do it, or simply submit the images as they are simply to show the results, but if possible i would like to have the images appended together so that RGB, LAB and HSI display together rather than each individual colourspace.

Classification Results

2008-01-27T20:40:00.000-08:00

Results: So far the results have been mixed. As we can see from the images below, a couple of positive images get classified into my "gray area" of dubious images. These are image that have traits of both target and empty images. However, the thresholding levels of the classification are hardcoded. I would like this to be more dynamic and autonomous, possibly using some form of Bayes or distance based classifier. Yet this would require gaining data for a positive set as well as just negative (empty images) or would need to calculate the distance of comparisions from the center of the negative class and threshold the classification based on some distance from this center point. Options to be discussed in meeting on Monday 28th Jan.

I have an idea or rather a hope that the images have been classified badly here because of the variety of the lighting offered. I imagine that in real conditions such a sharp difference in lighting would not be present. I shall attempt to get some more realistic data which has less variation in lighting.

Images:

Fake Sample Data

2008-01-27T19:19:00.000-08:00

Sample Data: The sample data i was hoping to have recieved by now hasn't happened due to problems with the engineering side of the UAV. Therefore i have created some basic shapes myself using my carpet and several objects.

I am still trying to sort out something with the UCSD AUVSI team to get hold of some real data whether it be this years lats years or whatever. At least something to offer the system that is truly relative to the project rather than similar prepared images.

Images:

Update on Histogram

2008-01-25T15:05:00.000-08:00

Found the bug in my code. However histogram results are more correct they do not show the results i had hoped.

Image With Target: 0.342762
Bright Green Empty: 0.333333
Image With Target: 0.334144
Bright Green Empty: 0.333333
dark Green Empty: 0.333333

As you can see the fact a target is present seems to have little improvement to the chi squared value. I was hoping to see a large jump in the difference. A possible direction i hope to take is to use the most common pixel colouring for each chanel and compare with that to boost different values.

Code Available Here: CODE

Histogram Issues

2008-01-24T21:43:00.000-08:00

Histogram Trouble:

The last few days i've been having some trouble getting histograms to do what i want in OpenCV. However after some building of my own functions specific to my needs i have what i think is conceptually a good setup. However, the results do not agree.

I start by having several labelled images of empty views. I then compose an average image, this takes all the values from each pixel (r,g,b) and averages over all test images. Thus creating an average empty image via supervision.

My intention was then to compare all new images against this average image, to do this i wanted to compare to the average image's histogram rather than he pixels directly, using chi squared. However this has some interesting results, but not good ones.

An empty image that is dark in comparision to the average empty image will be given a high chi squared return, i.e. looks very different, yet one that has the same background as the average image but contains a reasonably large target recieves a low score i.e similar.

Chi Value: 0.017876

Image with Target

Chi Value: 0.333333

Blank Image

Average Image

Still working on debug info any possible hints for improvement here would be helpful.

OpenCV

2008-01-22T17:26:00.000-08:00

Currently trying to get OpenCV to work with Visual Studio....having issues. Apparently there are missing .dll's. And the FAQ's response does not work as it should. I have a friend in ECE who has it compiled on his laptop and has it working with Visual Studio so am going to beg for some help.

Tracking Methods

2008-01-16T14:36:00.000-08:00

Open CV: http://www.intel.com/technology/computing/opencv/

This is a C based library that can be used directly in C code to track moving objects, from first viewing of the project it seems to find regions of interest and track them. The idea behind this is using a filter over an image and taking the gradient values at each window, trying to detect possible regions of interest the video shows an example of a labeled result. This is quite noisy however and should i use it i would need to ensure parameters are set to help specifcy this for the application to remove or at least lower the effects of noise in the image.

Videos: Input and Output

Use: I intend to use this if possible to help give me possible locations for the targets and therefore create bounding boxes of the image so that processing does not need to be calculated for an entire image.

============================================================

Kalman Filters: Kalman filters offer a way of predicting possible error changes from measured data to give us a discription of a true result using probabilty and knowledge of the movement or abilities of the object being tracked. A former lecturer of mine Andrew Calway (University of Bristol) showed me a case study in the case of tracking objects using Kalman filters with a moving camcorder. It can achieve this by finding the object in one location to start with and given possible information about movement uses probability to look in certain expected positions for the target in question, it also considers error margins in these calculations. Thus saving searching an entire image for a target if it has been previously identified.

"The purpose of a Kalman filter is to estimate the state of a system from meaurements which contain random errors. An example is estimating the position and velocity of a satelite from radar data."

Benefits: A further benefit of Kalman filters is that it seems to have a toolbox available directly in matlab currently. This is a major bonus as Matlab will be essential for some fo the high powered matrix operations i will doubtlessly have to perform for varying image processing/segmentation and classification issues related with the project.

Papers: Video Object Tracking Using Region Split And Merge And A Kalman Filter Tracking Algorithm

Video:

Use: I propose to use a kalman filter technique to improve the accuracy of the bounding box of the target found using opencv and be able to predict where the target may occur in the next image. By doing so i can reduce computayional overhead and hopefully lower processing costs too. Thus giving me a smaller problem to work with which is localized to a bounding box of the image.

Sample Data

2008-01-14T11:55:00.000-08:00

Full flight images (from 2006) can be found at: http://art1.mae.ncsu.edu/album.php?albumid=35

Current Situation

2008-01-06T09:33:00.000-08:00

Current Situation as of (06/01/08):
- Chosen project in UAV competition UCSD team
- Cleared from University of Bristol (UoB) to take CSE190a
- Writing draft proposal for CSE190 by wed 9th Jan:

finding relevant papers related to target recognition
deciding on limits/aims of a one quarter project in this field
finding as much information on the system used by the UCSD team for the UAV