Intro to Object Detection

About

This is the notes for Improving Deep Neural Networks taught by Andrew Ng at Coursera. Here, lesson notes and assignments will be provided in order to enhance my comprehension about Neural Networks. You can view my github for the programming assignment.

Content

This week we learnt about some CNN applications: object detections. There are several strategies to implement this task:

  • Object Localization
  • Landmark Detections
  • Sliding Windows Detections
  • Bounding Box Prediction
  • IOU
  • Non-Maximum Suppresion
  • Anchor Boxes
  • YOLO Algorithm

Now, let’s talk about them one by one~

Object Localization

You can find that all of the tasks we’ve done before are just about classification but not localization. Given a picture, just output 0/1 to classify if this image has a cat. But in this chapter we’d like to do not only classification, but also object localization which is to say, give the position of the detected object. In order to acommplish that, we are going to output 4 more values: bx,by,bh,bw. All of these four values depict the precise position of the object in the image. Thus, we are going to define a new format of output y, a (1+4+n_classes)*1 vector.

Landmark Detection

I don’t know what does this strange landmark matter. As Andrew says, we use landmark detection to help us know what’t the movement of someone currently. But he didn’t cover the details… Let’s omitted it.

Sliding Windows Detections

In the lessons in Machine learning, Andrew has already covered some details about slide windows detections in maybe PCA?(mentioned the automatic driving chapter anyway). Suppose you have a window to detect the object you want. Then you just slide this window to check if the object exists here. To detect different objects of the different size, various sizes of windows are needed. When it comes to the implementation of sliding windows, you need to change the orignal output, a single number into a vector. Furthermore, just iterate over the whole windows position is much computational-expensive. To handle with this shortcoming, we choose concolution implementation of sliding windows, which is to convolute the whole image into the final output, decrease much repeated computation.

Bounding Box Detection

In the previou part, we talked about sliding window dectection. Here we introduce another method used in YOLO algorithm: Bounding box. The main idea is like this: spilt the whole image into 3*3 grid cells,(or 19*19,just for explanation). Then for each grid cell, we implement the object detection on it. And we only detect the object whose center is in this grid cell.

Intersection over union(IOU)

For the purpose of building an evaluation of object localization, we use intersection over union. Generally speaking, IoU is a measure of the overlap between two bounding boxes.

Non-max Suppression

Say we have detect that there are two cars in the picture. But several windows are pointed to the same car. Additional objects should be omitted. Thus, non-max suppression is a good idea. Here is the non-max suppression algorithm:

Anchor Box

If we want to detect multiple object in one position, say a woman is in front of a car, we want to detect both car and woman, the past bounding box detection can not work well. Therefore, someone put forward the anchor box. With two anchor boxes, each object in the training image is assigned to grid cell that contains object’s midpoint and anchor box for the grid cell with highest IoU. Also, the correspoinding shape of output y should change like this:

YOLO Algorithm

We will go through this algorithm in this assignment, so let’s just skip it :)

Assignment Autonomous driving - Car detection

This assignment is to implement YOLO model to detect the cars in the road. At the end of the assignment, you’ll have a well trained model to detect the cars in the road! Let’s begin.

The Whole Process

First, we encoding the preprocessed image into a deep CNN, then get a encoding matrix.

Figure : Encoding architecture for YOLO

Since we are using 5 anchor boxes, each of the 19 x19 cells thus encodes information about 5 boxes. Anchor boxes are defined only by their width and height.

For simplicity, we will flatten the last two last dimensions of the shape (19, 19, 5, 85) encoding. So the output of the Deep CNN is (19, 19, 425).

Figure : Flattening the last two last dimensions

Now, for each box (of each cell) we will compute the following elementwise product and extract a probability that the box contains a certain class.

Figure : Find the class detected by each box

Filtering with a threshold on class scores

As the steps mentioned above, you can write the belowing code implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
def yolo_filter_boxes(box_confidence, boxes, box_class_probs, threshold = .6):
"""Filters YOLO boxes by thresholding on object and class confidence.

Arguments:
box_confidence -- tensor of shape (19, 19, 5, 1)
boxes -- tensor of shape (19, 19, 5, 4)
box_class_probs -- tensor of shape (19, 19, 5, 80)
threshold -- real value, if [ highest class probability score < threshold], then get rid of the corresponding box

Returns:
scores -- tensor of shape (None,), containing the class probability score for selected boxes
boxes -- tensor of shape (None, 4), containing (b_x, b_y, b_h, b_w) coordinates of selected boxes
classes -- tensor of shape (None,), containing the index of the class detected by the selected boxes

Note: "None" is here because you don't know the exact number of selected boxes, as it depends on the threshold.
For example, the actual output size of scores would be (10,) if there are 10 boxes.
"""

# Step 1: Compute box scores
### START CODE HERE ### (≈ 1 line)
box_scores = box_confidence*box_class_probs
### END CODE HERE ###

# Step 2: Find the box_classes thanks to the max box_scores, keep track of the corresponding score
### START CODE HERE ### (≈ 2 lines)
box_classes = K.argmax(box_scores, axis=-1)
box_class_scores = K.max(box_scores,axis=-1)

# box_classes和box_class_scores: (19, 19, 5)
### END CODE HERE ###

# Step 3: Create a filtering mask based on "box_class_scores" by using "threshold". The mask should have the
# same dimension as box_class_scores, and be True for the boxes you want to keep (with probability >= threshold)
### START CODE HERE ### (≈ 1 line)
filtering_mask = box_class_scores>=threshold
### END CODE HERE ###

# Step 4: Apply the mask to scores, boxes and classes
### START CODE HERE ### (≈ 3 lines)
# 过滤掉score低于threshold的box
scores = tf.boolean_mask(box_class_scores,filtering_mask)
boxes = tf.boolean_mask(boxes,filtering_mask)
classes = tf.boolean_mask(box_classes,filtering_mask)
### END CODE HERE ###

return scores, boxes, classes

Non-max Suppression

Here we only implement IoU: Non-max suppression uses the very important function called “Intersection over Union”, or IoU.

Figure : Definition of “Intersection over Union”.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
def iou(box1, box2):
"""Implement the intersection over union (IoU) between box1 and box2

Arguments:
box1 -- first box, list object with coordinates (x1, y1, x2, y2)
box2 -- second box, list object with coordinates (x1, y1, x2, y2)
"""

# Calculate the (y1, x1, y2, x2) coordinates of the intersection of box1 and box2. Calculate its Area.
### START CODE HERE ### (≈ 5 lines)
xi1 = max(box1[0],box2[0])
yi1 = max(box1[1],box2[1])
xi2 = min(box1[2],box2[2])
yi2 = min(box1[3],box2[3])
inter_area = (xi2-xi1)*(yi2-yi1)
### END CODE HERE ###

# Calculate the Union area by using Formula: Union(A,B) = A + B - Inter(A,B)
### START CODE HERE ### (≈ 3 lines)
box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
union_area = box1_area + box2_area - inter_area
### END CODE HERE ###

# compute the IoU
### START CODE HERE ### (≈ 1 line)
iou = inter_area / union_area
### END CODE HERE ###

return iou

You are now ready to implement non-max suppression. The key steps are:

  1. Select the box that has the highest score.
  2. Compute its overlap with all other boxes, and remove boxes that overlap it more than iou_threshold.
  3. Go back to step 1 and iterate until there’s no more boxes with a lower score than the current selected box.

This will remove all boxes that have a large overlap with the selected boxes. Only the “best” boxes remain.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
def yolo_non_max_suppression(scores, boxes, classes, max_boxes = 10, iou_threshold = 0.5):
"""
Applies Non-max suppression (NMS) to set of boxes

Arguments:
scores -- tensor of shape (None,), output of yolo_filter_boxes()
boxes -- tensor of shape (None, 4), output of yolo_filter_boxes() that have been scaled to the image size (see later)
classes -- tensor of shape (None,), output of yolo_filter_boxes()
max_boxes -- integer, maximum number of predicted boxes you'd like
iou_threshold -- real value, "intersection over union" threshold used for NMS filtering

Returns:
scores -- tensor of shape (, None), predicted score for each box
boxes -- tensor of shape (4, None), predicted box coordinates
classes -- tensor of shape (, None), predicted class for each box

Note: The "None" dimension of the output tensors has obviously to be less than max_boxes. Note also that this
function will transpose the shapes of scores, boxes, classes. This is made for convenience.
"""

max_boxes_tensor = K.variable(max_boxes, dtype='int32') # tensor to be used in tf.image.non_max_suppression()
K.get_session().run(tf.variables_initializer([max_boxes_tensor])) # initialize variable max_boxes_tensor

# Use tf.image.non_max_suppression() to get the list of indices corresponding to boxes you keep
### START CODE HERE ### (≈ 1 line)
nms_indices = tf.image.non_max_suppression(boxes,scores,iou_threshold=iou_threshold,max_output_size=max_boxes)
### END CODE HERE ###
#print('nms_indices:' + str(nms_indices.eval()))
# Use K.gather() to select only nms_indices from scores, boxes and classes
### START CODE HERE ### (≈ 3 lines)
scores = K.gather(scores,nms_indices)
boxes = K.gather(boxes,nms_indices)
classes = K.gather(classes,nms_indices)
### END CODE HERE ###

return scores, boxes, classes

Wrapping up the filtering

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def yolo_eval(yolo_outputs, image_shape = (720., 1280.), max_boxes=10, score_threshold=.6, iou_threshold=.5):
"""
Converts the output of YOLO encoding (a lot of boxes) to your predicted boxes along with their scores, box coordinates and classes.

Arguments:
yolo_outputs -- output of the encoding model (for image_shape of (608, 608, 3)), contains 4 tensors:
box_confidence: tensor of shape (None, 19, 19, 5, 1)
box_xy: tensor of shape (None, 19, 19, 5, 2)
box_wh: tensor of shape (None, 19, 19, 5, 2)
box_class_probs: tensor of shape (None, 19, 19, 5, 80)
image_shape -- tensor of shape (2,) containing the input shape, in this notebook we use (608., 608.) (has to be float32 dtype)
max_boxes -- integer, maximum number of predicted boxes you'd like
score_threshold -- real value, if [ highest class probability score < threshold], then get rid of the corresponding box
iou_threshold -- real value, "intersection over union" threshold used for NMS filtering

Returns:
scores -- tensor of shape (None, ), predicted score for each box
boxes -- tensor of shape (None, 4), predicted box coordinates
classes -- tensor of shape (None,), predicted class for each box
"""

### START CODE HERE ###

# Retrieve outputs of the YOLO model (≈1 line)
box_confidence, box_xy, box_wh, box_class_probs = yolo_outputs

# Convert boxes to be ready for filtering functions
boxes = yolo_boxes_to_corners(box_xy, box_wh)

# Use one of the functions you've implemented to perform Score-filtering with a threshold of score_threshold (≈1 line)
scores, boxes, classes = yolo_filter_boxes(box_confidence, boxes, box_class_probs, score_threshold)

# Scale boxes back to original image shape.
boxes = scale_boxes(boxes, image_shape)

# Use one of the functions you've implemented to perform Non-max suppression with a threshold of iou_threshold (≈1 line)
scores, boxes, classes = yolo_non_max_suppression(scores, boxes, classes, max_boxes, iou_threshold)

### END CODE HERE ###

return scores, boxes, classes

Summary

After running your model, you’ve seen your well-predicted model on image detection.

In the next lesson, we’ll learn about face recognition and NN style tranformation.