This is the paper proposed RetinaNet and also the focal loss function to better train object detection models. Object detection models are in two camps, there are two-stage proposal-driven models such as R-CNN, and one-stage detector such that YOLO and SSD. The paper claimed that the prior result on one-stage model is not as good as two-stage because of class imbalance during training.

In two-stage models, the first stage generates a sparse set of candidate proposals and the second stage classifies the proposals into classes. One-stage detectors, however, need to evaluate 10K to 100K candidate locations per image but only a few locations contain objects, whereas two-stage models would reduce this to 1K to 2K with 1:3 ratio of positive to negative proposals. The easy negatives can overwhelm training and lead to degenerate models. Therefore the solution is to down-weight easy examples (a.k.a. inliers) such that their contribution to total loss is small even if their number is large.

The loss function proposed is as follows:

\[\begin{aligned} p_t &= \begin{cases} p & \text{if }y=1\\ 1-p & \text{otherwise} \end{cases}\\ \text{FL}(p_t) &= -(1-p_t)^\gamma \log(p_t) \end{aligned}\]

Which \(p\) is the predicted probability in classification and \(y\) is the one-hot target (0 or 1). Therefore \(p_t\) is the probability of truth, which ideally should be 1 if \(y=1\) and 0 otherwise. The cross entropy for a sample is therefore \(-\log(p_t)\), but the log function decays too gently as \(p_t\) approaches 1. The modulating factor \((1-p_t)^\gamma\) is to make the loss function decays faster. It degenerates into the simple cross entropy at \(\gamma=0\) and the paper suggested \(\gamma=2\) while it is found relatively robust in \(\gamma\in[0.5,5]\). This helps to make loss closer to zero once \(p_t\) is large enough (i.e. ,\(p_t>0.5\)).

To test this proposal, the paper suggested a RetinaNet model. It combines a backbone network with two task-specific subnetworks, such that:

  • The backbone, such as ResNet, connects to feature pyramid network (FPN). The ResNet backbone provides a bottom-up pathway to produce feature maps, and FPN is the top-down pathway to combine feature maps of different resolution into a multiscale feature pyramid
    • RetinaNet use ResNet feature maps \(C_3\) to \(C_5\) with FPN to produce pyramid levels \(P_3\) to \(P_7\), which \(P_k\) has resolution \(1/2^k\) of the input
    • ResNet residual stage \(C_3\) to \(C_5\) contributes to \(P_3\) to \(P_5\) using FPN top-down pathway; but \(P_6\) is obtained via a 3×3 stride-2 conv on \(P_5\), and \(P_7\) is from ReLU then another 3×3 stride-2 conv on \(P_6\)
    • All pyramid levels have \(C=256\) channels
  • Anchor boxes similar to that of RPN are produced from the FPN, with areas of 32² to 512² on \(P_3\) to \(P_7\) and aspect ratios 1:2, 1:1, 2:1. Optionally at each layer, anchor boxes are scaled to the ratio of \(2^0, 2^{1/3}, 2^{2/3}\) of its original size. In total \(A=3\times 3=9\) anchors are produced at each spatial location.
    • Each anchor is assigned a length \(K\) one-hot vector as classification target and length \(4K\) vector as bbox regression target
    • anchor label: compare bbox to ground-truth using IoU, positive samples for \(>0.5\) and background for \(\in[0,0.4)\)
  • Classification subnet: Take the input feature map of \(C=256\) channels at each pyramid level \(P_k\), then
    • pass through 4× of: 3×3 conv with \(C\) channels, then ReLU
    • then 3×3 conv with \(KA\) channels, then sigmoid activation to produce binary classification of each anchor
  • Regression subnet: Parallel to and same design as classification subnet, except the output is length \(4A\) and with linear activation
    • output length is \(4A\) instead of \(4KA\): the bounding box regression is class agnostic but found to be equally effective
  • Model output \(A\) anchors per pyramid level and spatial location, but to trade-off for speed, only decode the bbox of the top-scoring 1K predictions
    • at training, focal loss applied to all 100K anchors per image, normalized by the number of anchors assigned to a ground-truth box
    • detector confidence threshold: 0.05
    • then run non-maximum suppression, with threshold of 0.5
  • New layers in the model are initialized with:
    • bias are zeroed, except the final conv layer for classification to use \(b=-\log\frac{1-\pi}{\pi}\) with \(\pi=0.01\) as the initial probability of finding a foreground class
    • weights are Gaussian with \(\sigma=0.01\)

The network is trained with a minibatch size of 16 images, 90K steps, weight decay of 10e-4, momentum of 0.9, and initial learning rate of \(10^{-2}\) but tuned down to \(10^{-3}\) at step 60K and further to \(10^{-4}\) at step 80K. The overall loss function is the sum of focal loss for classification and smoothed L1 loss for regression.

Bibliographic data

   title = "Focal Loss for Dense Object Detection",
   author = "Tsung-Yi Lin and Priya Goyal and Ross Girshick and Kaiming He and Piotr Dollar",
   booktitle = "Proc CVPR",
   pages = "2980--2988",
   year = "2017",
   note = "arXiv:1708.02002",