OpenCV

OpenCV Tracking 物体追踪算法

关于OpenCV中物体追踪算法的笔记

Posted by YuCong on November 10, 2020

本文主要含OpenCV Tracking API的使用方法记录与物体追踪相关算法的研究。

Tracking API

Created 2020.11.10 by Cong Yu; Last modified: 2022.09.05-v1.1.3

Contact: windmillyucong@163.com

Copyleft! 2022 Cong Yu. Some rights reserved.

References

0. Intro

0.1 头文件

1

#include <opencv2/tracking.hpp>

0.2 Class

opencv tracking API 提供了很多种 tracker：

单个物体追踪算法

img

多个物体追踪

1. Usage

1.1 Track

代码片段示例

track的使用方法非常简单：

create()构造，init()初始化，然后update()更新bbx即可。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

class MyTracker {
 public:
  explicit MyTracker(std::string tracker_type = "CSRT")
      : tracker_type_(std::move(tracker_type)) {}
  ~MyTracker() = default;

  void InitTracker(cv::Mat image, const cv::Rect &box);
  bool Track(cv::Mat image);
  std::shared_ptr<const cv::Rect> GetBBox() const { return roi_; };

 private:
  cv::Ptr<cv::Tracker> tracker_;
  std::string tracker_type_;
  std::shared_ptr<cv::Rect> roi_;
};

void MyTracker::InitTracker(cv::Mat image, const cv::Rect &box) {
  if (tracker_type_ == "MOSSE") {
    tracker_ = cv::TrackerMOSSE::create();
  } else if (tracker_type_ == "KCF") {
    tracker_ = cv::TrackerKCF::create();
  } else {
    tracker_ = cv::TrackerCSRT::create();
  }
  tracker_->init(image, box);
}

bool MyTracker::Track(cv::Mat image) {
  if (!tracker_ /*|| !roi_*/) {
    return false;
  }

  cv::Rect2d bbox;
  if (tracker_->update(image, bbox)) {
    roi_ = std::make_shared<cv::Rect>(bbox);
    LOG(ERROR) << "TACK Success!" << bbox;
    return true;
  } else {
    LOG(ERROR) << "TACK FAILED!";
    return false;
  }
}

init()

方法原型：

1
2
3
4
5
6

bool cv::legacy::Tracker::init	(	InputArray 	image,
const Rect2d & 	boundingBox 
)	

Python:
cv.legacy.Tracker.init(	image, boundingBox	) ->	retval

Initialize the tracker with a known bounding box that surrounded the target.

  • Parameters
    • image: The initial frame
    • boundingBox: The initial bounding box

  • Returns

    True if initialization went succesfully, false otherwise

update()

方法原型：

1
2
3
4
5
6

bool cv::legacy::Tracker::update	(	InputArray 	image,
Rect2d & 	boundingBox 
)

Python:
cv.legacy.Tracker.update(	image	) ->	retval, boundingBox

Update the tracker, find the new most likely bounding box for the target.

  • Parameters
    • image: [in] The current frame.
    • boundingBox: [out] The bounding box that represent the new target location, if true was returned, not modified otherwise
  • Returns
    • True means that target was located
    • and false means that tracker cannot locate target in current frame.
    • Note, that latter does not imply that tracker has failed, maybe target is indeed missing from the frame (say, out of sight)

注意：

  • 在实际使用中，可以不必每次对整张图像update。而是将上一次追踪出的物体的框扩大一下，然后对图像裁切一部分roi，然后再交给update函数更新，这样可以或许可以更快速？//todo(congyu) 待验证

1.2 代码设计: detect & track

在实际使用中，只有track是无法实现一个完整的物体检测与追踪功能的。一方面：track只管追踪bbx，而最初的bbx怎么获得呢？需要用户给定，或者由用户设计一个detect方法来做物体检测。另一方面：可能出现某几帧track失败的情况，此时需要重新进行detect，以重新确定track的目标。所以

一个完整的物体检测与追踪算法的实现是：

  1. 用户设计Detect()方法，输入image，输出bbx [[cv/2020-11-09-detect]]

  2. 然后将bbx交给tracker 做init()

  3. 然后每次tracker 做update()

  4. 并判断update()是否失败，失败时就需要重新Detect()

完整的逻辑示例如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

#include <gflags/gflags.h>
#include <glog/logging.h>
#include <memory>
#include <thread>

#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/tracking.hpp>

class MyDetector {
 public:
  MyDetector() {}
  ~MyDetector() {}

  void Init() {
    detect_and_track_thread_running_ = true;
    detect_and_track_thread_ =
        std::thread(&MyDetector::DetectAndTrackThread, this);

    // balabala
    // ...
  }

  bool Detect(const cv::Mat& src) {
    // balabala
    // ...
    return true;
  }

  bool Track(const cv::Mat& src) {
    // balabala
    // ...
    return true;
  }

 private:
  void DetectAndTrackThread() {
    // init
    bool need_detect = true;
	// thread while
    while (detect_and_track_thread_running_) {
      if (need_detect) {
        std::vector<cv::Rect> bounding_boxes;
        if (Detect(src_)) {
          need_detect = false;
          LOG(ERROR) << "detect succeeded!";
        } else {
          LOG(ERROR) << "detect failed! keep trying...";
        }
      } else {
        if (Track(src_)) {
          LOG(ERROR) << "tracking succeeded!";
        } else {
          need_detect = true;
          LOG(ERROR) << "tracking failed! redo detect...";
        }
      }
    }
    std::this_thread::sleep_for(std::chrono::milliseconds(80));
  }

  bool detect_and_track_thread_running_ = false;
  std::thread detect_and_track_thread_;
  cv::Mat src_;
};

void TestDetectAndTracking() {
  MyDetector my_detector_;
  my_detector_.Init();
}

int main(int argc, char* argv[]) {
  google::InitGoogleLogging(argv[0]);
  google::InstallFailureSignalHandler();
  google::ParseCommandLineFlags(&argc, &argv, false);

  TestDetectAndTracking();
  return 0;
}

那么问题来了：有了detect()，为什么不每次都在while循环中直接使用detece()? 因为detect()通常为物体检测算法，可能是常用的数字图像处理方法，也可能是DL的方法，运算速度往往大于track。

1.3 MultiTracker

基本上MultiTracker就是将Tracker丢在一个vector里面，然后封装出一个统一的接口。

add()

1
2
3
4

bool cv::legacy::MultiTracker::add	(	std::vector< Ptr< legacy::Tracker > > 	newTrackers,
InputArray 	image,
std::vector< Rect2d > 	boundingBox 
)

Add a set of objects to be tracked.

Parameters

  • newTrackers: list of tracking algorithms to be used
  • image: input image
  • boundingBox: list of the tracked objects

update()

1
2
3

bool cv::legacy::MultiTracker::update	(	InputArray 	image,
std::vector< Rect2d > & 	boundingBox 
)

Update the current tracking status.

Parameters

  • image: input image
  • boundingBox: the tracking result, represent a list of ROIs of the tracked objects.

代码片段

使用方法同样非常简单：

  1. create()生成一个cv::MultiTracker

    1

      cv::Ptr<cv::MultiTracker> multi_tracker_ = cv::MultiTracker::create();

  2. 然后生成多个cv::Tracker

    1

      cv::Ptr<cv::Tracker> track_circle_ = cv::TrackerKCF::create();

  3. 使用add()添加tracker

    1

      multi_tracker_->add(track_circle_, src, circle_roi);

  4. 之后，在while循环中使用update()更新即可

    1
2

        std::vector<cv::Rect2d> bounding_boxes;
    multi_tracker_->update(src, bounding_boxes);

追踪效果：

MultiTracker

Fig1.  MultiTracker

全部代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

#include <random>

#include <gflags/gflags.h>
#include <glog/logging.h>

#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/tracking.hpp>

inline double RandomDouble(const double min, const double max) {
  std::default_random_engine random_engine(std::random_device{}());
  std::uniform_real_distribution<> random_double(min, max);
  return random_double(random_engine);
}

const cv::Size kSrcSize = cv::Size(640, 480);

inline cv::Point RandomMove(const cv::Point &point) {
  const int kRange = 10;
  auto point_new = point;
  point_new.x = point_new.x > (kSrcSize.width - kRange)
                    ? (kSrcSize.width - kRange)
                    : point_new.x;
  point_new.y = point_new.y > (kSrcSize.height - kRange)
                    ? (kSrcSize.height - kRange)
                    : point_new.y;
  point_new.x = point_new.x < kRange ? kRange : point_new.x;
  point_new.y = point_new.y < kRange ? kRange : point_new.y;
  point_new.x = point_new.x + RandomDouble(-kRange, kRange),
  point_new.y = point_new.y + RandomDouble(-kRange, kRange);
  return point_new;
}

bool TestMultiTracking() {
  const std::string kWindowName = "test";
  const cv::Scalar kColorCircle = cv::Scalar(0, 255, 0);
  const cv::Scalar kColorEllipse = cv::Scalar(0, 255, 255);
  const cv::Scalar kColorRectangle = cv::Scalar(0, 0, 255);
  const int kCircleSize = 16;
  const cv::Size2f kEllipseSize = cv::Size2f(36, 18);
  const cv::Size2f kRectangleSize = cv::Size2f(30, 15);
  const int kRoiSize = 100;

  cv::Mat src(kSrcSize, CV_8UC3, cv::Scalar(0, 0, 0));

  cv::circle(src, cv::Point(320, 240), kCircleSize, kColorCircle, -1);
  cv::ellipse(src, cv::RotatedRect(cv::Point2f(380, 120), kEllipseSize, 30),
              kColorEllipse, -1);
  cv::rectangle(src,
                cv::Rect(180, 110, kRectangleSize.width, kRectangleSize.height),
                kColorRectangle, -1);

  cv::Ptr<cv::Tracker> track_circle_ = cv::TrackerKCF::create();
  cv::Ptr<cv::Tracker> track_ellipse_ = cv::TrackerKCF::create();
  cv::Ptr<cv::Tracker> track_rectangle_ = cv::TrackerKCF::create();

  cv::Ptr<cv::MultiTracker> multi_tracker_ = cv::MultiTracker::create();
  const auto circle_roi = cv::Rect2d(320 - kRoiSize / 2.0, 240 - kRoiSize / 2.0,
                                     kRoiSize, kRoiSize);
  multi_tracker_->add(track_circle_, src, circle_roi);
  const auto ellipse_roi = cv::Rect2d(380 - kRoiSize / 2.0,
                                      120 - kRoiSize / 2.0, kRoiSize, kRoiSize);
  multi_tracker_->add(track_ellipse_, src, ellipse_roi);
  const auto rectangle_roi = cv::Rect2d(
      180 + kRectangleSize.width / 2.0 - kRoiSize / 2.0,
      110 + kRectangleSize.width / 2.0 - kRoiSize / 2.0, kRoiSize, kRoiSize);
  multi_tracker_->add(track_rectangle_, src, rectangle_roi);

  std::vector<cv::Rect2d> bounding_boxes;
  bounding_boxes.emplace_back(circle_roi);
  bounding_boxes.emplace_back(ellipse_roi);
  bounding_boxes.emplace_back(rectangle_roi);

  while (true) {
    const auto key = cv::waitKey();
    if ('q' == key || 27 == key) {  // quit
      cv::destroyWindow(kWindowName);
      break;
    }

    // random move
    CHECK_EQ(bounding_boxes.size(), 3);
    src = cv::Mat(kSrcSize, CV_8UC3, cv::Scalar(0, 0, 0));
    auto bbx = bounding_boxes[0];
    auto point = cv::Point(bbx.x + bbx.width / 2.0, bbx.y + bbx.height / 2.0);
    cv::circle(src, RandomMove(point), 10, kColorCircle, -1);
    bbx = bounding_boxes[1];
    point = cv::Point(bbx.x + bbx.width / 2.0, bbx.y + bbx.height / 2.0);
    cv::ellipse(src, cv::RotatedRect(RandomMove(point), kEllipseSize, 30),
                kColorEllipse, -1);
    bbx = bounding_boxes[2];
    point = cv::Point(bbx.x + bbx.width / 2.0, bbx.y + bbx.height / 2.0);
    point = RandomMove(point);
    cv::rectangle(src,
                  cv::Rect(point.x - kRectangleSize.width / 2.0,
                           point.y - kRectangleSize.height / 2.0,
                           kRectangleSize.width, kRectangleSize.height),
                  cv::Scalar(0, 0, 255), -1);

    // track
    multi_tracker_->update(src, bounding_boxes);
    bbx = bounding_boxes[0];
    cv::rectangle(src, bbx, cv::Scalar(0, 255, 0));
    bbx = bounding_boxes[1];
    cv::rectangle(src, bbx, cv::Scalar(0, 255, 255));
    bbx = bounding_boxes[2];
    cv::rectangle(src, bbx, cv::Scalar(0, 0, 255));

    cv::imshow(kWindowName, src);
  }

  return true;
}

int main(int argc, char *argv[]) {
  google::InitGoogleLogging(argv[0]);
  google::InstallFailureSignalHandler();
  google::ParseCommandLineFlags(&argc, &argv, false);

  TestMultiTracking();
  return 0;
}

2. 原理剖析

Refitem

2.1 算法与特点总结

特点总结

  • BOOSTING：算法原理类似于Haar cascades (AdaBoost)，是一种很老的算法。这个算法速度慢并且不是很准。
  • MIL：比BOOSTING准一点。
  • KCF：速度比BOOSTING和MIL更快，与BOOSTING和MIL一样不能很好地处理遮挡问题。
  • CSRT：比KCF更准一些，但是速度比KCF稍慢。
  • MedianFlow：对于快速移动的目标和外形变化迅速的目标效果不好。
  • TLD：会产生较多的false-positives。
  • MOSSE：算法速度非常快，但是准确率比不上KCF和CSRT。在一些追求算法速度的场合很适用。
  • GOTURN：OpenCV中自带的唯一一个基于深度学习的算法。运行算法需要提前下载好模型文件。

综合算法速度和准确率考虑，CSRT、KCF、MOSSE这三个目标跟踪算法较好。

所以重点看一下KCF，其他的暂且略过。

2.2 TrackerKCF

the KCF (Kernelized Correlation Filter) tracker. 基于核相关滤波器的追踪器

  • KCF is a novel tracking framework
  • that utilizes properties of circulant matrix to enhance the processing speed. 使用循环矩阵的特性来提高处理速度。
  • This tracking method is an implementation of [110]
  • which is extended to KCF with color-names features ([53]).
  • The original paper of KCF is available at http://www.robots.ox.ac.uk/~joao/publications/henriques_tpami2015.pdf as well as the matlab implementation. 原paper
  • For more information about KCF with color-names features, please refer to http://www.cvl.isy.liu.se/research/objrec/visualtracking/colvistrack/index.html.

具体的论文阅读，详见下篇。

Contact

Feel free to contact me windmillyucong@163.com anytime for anything.

License

CC0

CATALOG
    FEATURED TAGS
    Linux c++ 多线程 OpenCV SLAM thread algorithm pose Math filter Kalman-Filter paper tree #cplusplus Design-Pattern calibrate camera collision-detect git tmux video
    FRIENDS