• 1. OpenCV学习指导 北京大学数字媒体研究所
  • 2. 第一部分OpenCV概述 基础结构与操作 基本OpenCV程序与示例 注:本讲义中部分代码来自OpenCV样例,部分代码来自《OpenCV教程——基础篇》,部分代码则从网上收集。2
  • 3. OpenCV概述OpenCV是Intel开源计算机视觉库。 它由一系列 C 函数和少量 C++ 类构成,实现了图像处理和计算机视觉方面的很多通用算法。 OpenCV 的特点 拥有包括 300 多个C函数的跨平台的中、高层 API 跨平台:Windows, Linux 免费(FREE):无论对非商业应用和商业应用 速度快 使用方便3
  • 4. 4OpenCV structureCXCORE basic structures and algoritms, XML support, drawing functionsCV Image processing and visionHighGUI GUI, Image and Video I/OML Machine Learning algorithmsCVCam video stream processing
  • 5. The Functionality in Pictures
  • 6. OpenCV的下载与安装1、下载地址 http://www.opencv.org.cn/download/ 2、VC6.0下的安装与配置 参见 http://www.opencv.org.cn/index.php/VC6%E4%B8%8B%E5%AE%89%E8%A3%85%E4%B8%8E%E9%85%8D%E7%BD%AE 3、.NET 2003下的安装与配置 参见 http://www.opencv.org.cn/index.php/VC.net_2003%E4%B8%8B%E5%AE%89%E8%A3%85%E4%B8%8E%E9%85%8D%E7%BD%AE 6
  • 7. Configuring MSVS .net 2k3Create a "Win32 Console Project"Make it an "Empty Project" by selecting the box under "Application Settings"A project is initially created by selecting: File -> New -> ProjectCreating the Project
  • 8. Configuring MSVS .net 2k3Right Click the "Source Files" Folder under the project name ("Tutorial" in this case) Add -> Add new ItemCreate the First FileSelect "C++ file" and give it a nameCreating a file makes it possible to set "Additional Include Directives" in the C/C++ pane under the project properties.
  • 9. Configuring MSVS .net 2k3In order to build projects using OpenCV the required libraries and directives must be included in the project’s propertiesRight Click the name of the project and select "Properties" ("Tutorial" in this case)Open the Properties Pane
  • 10. Configuring MSVS .net 2k3Under the C/C++ tab select "General"Set Additional Include DirectivesSelect the "Additional Include Directives"Add the full path to each of the folders which contain ".h" files required to use OpenCVBe sure to include trailing "\"C:\Program Files\OpenCV\cvaux\include\ C:\Program Files\OpenCV\cxcore\include\ C:\Program Files\OpenCV\cv\include\ C:\Program Files\OpenCV\otherlibs\highgui\ C:\Program Files\OpenCV\otherlibs\cvcam\include\ Utilized Directives
  • 11. Configuring MSVS .net 2k3Under the Linker tab select "Input"Set Additional DependenciesSelect the "Additional Dependencies""C:\Program Files\OpenCV\lib\cv.lib" "C:\Program Files\OpenCV\lib\cvaux.lib" "C:\Program Files\OpenCV\lib\cxcore.lib" "C:\Program Files\OpenCV\lib\cvcam.lib" "C:\Program Files\OpenCV\lib\highgui.lib"Utilized DependenciesAdd the full path to each of the ".lib" files required to use OpenCVBe sure to keep the paths in quotes
  • 12. Testing MSVS .net 2k3Now that the environment is configured it would be a good idea to test it to make sure that a program will correctly build and run. #include #include /*This will pop up a small box with "Hello World" as the text.*/ int main( int argc, char** argv ) { //declare for the height and width of the image int height = 320; int width = 240; //specify the point to place the text CvPoint pt = cvPoint( height/4, width/2 ); //Create an 8 bit, 3 plane image IplImage* hw = cvCreateImage(cvSize(height, width), 8, 3); //initialize the font CvFont font; cvInitFont( &font, CV_FONT_HERSHEY_COMPLEX, 1.0, 1.0, 0, 1, CV_AA); //place the text on the image using the font cvPutText(hw, "Hello World", pt, &font, CV_RGB(150, 0, 0) ); //create the window container cvNamedWindow("Hello World", 0); //display the image in the container cvShowImage("Hello World", hw); //hold the output windows cvWaitKey(0); return 0; }The enclosed code can be cut and pasted into the file created in the project space to test OpenCV Testing the First Program
  • 13. Testing MSVS .net 2k3Output of ProgramThe program is built by selecting: Build -> Build Solution Or by pressing "F7"Building the ProgramThe program is run by selecting: Debug -> {Start||Start without Debugging} Or by pressing "F5" or "-F5" Running the Program
  • 14. OpenCV 编码样式指南1、文件命名:有cv和cvaux库文件的命名必须服从于以下规则: 所有的CV库文件名前缀为cv 混合的C/C++接口头文件扩展名为 .h 纯C++接口头文件扩展名为 .hpp 实现文件扩展名为 .cpp 为了与POSIX兼容,文件名都以小写字符组成14
  • 15. OpenCV 编码样式指南2、文件结构 每个文件以BSD兼容的许可声明(模板在Contributors_BSD_Licsense.htm文件中可以找到)开头; 一行最多90个字符,不包括行结束符 不使用制表符 缩进为4个空格符,所以制表符应该用1-4个空格替换(依据开始列确定) 头文件必须使用保护宏,防止文件被重复包含。 混合C/C++接口头文件用extern "C" { } 包含C语言定义。 为了使预编译头机制在Visual C++中工作正常,源文件必须在其它头文件前包含precomp.h头文件。15
  • 16. OpenCV 编码样式指南3、命名约定 OpenCV中使用大小写混合样式来标识外部函数、数据类型和类方法。 宏全部使用大写字符,词间用下划线分隔。 所有的外部或内部名称,若在多个文件中可见,则必须含有前缀: 外部函数使用前缀cv 内部函数使用前缀Icv 数据结构(C结构体、枚举、联合体、类)使用前缀Cv 外部或某些内部宏使用前缀CV_ 内部宏使用前缀ICV_ 16
  • 17. OpenCV 编码样式指南4、函数接口设计:为了保持库的一致性,以如下方式设计接口非常重要。函数接口元素包括: 功能 名称 返回值 参数类型 参数顺序 参数默认值    函数功能必须定义良好并保持精简。函数应该容易镶入到使用其它OpenCV函数的不同处理过程。函数名称应该简单并能体现函数的功能。 大多数函数名形式:cv17
  • 18. 18A Simple OpenCV Program1. #include 2. #include 3. #include 4. int main( int argc, char** argv ) { 5. CvPoint center; 6. double scale=-3; 7. IplImage* image = argc==2 ? cvLoadImage(argv[1]) : 0; 8. if(!image) return -1; 9. center = cvPoint(image->width/2,image->height/2); 10. for(int i=0;iheight;i++) 11. for(int j=0;jwidth;j++) { 12. double dx=(double)(j-center.x)/center.x; 13. double dy=(double)(i-center.y)/center.y; 14. double weight=exp((dx*dx+dy*dy)*scale); 15. uchar* ptr = &CV_IMAGE_ELEM(image,uchar,i,j*3); 16. ptr[0] = cvRound(ptr[0]*weight); 17. ptr[1] = cvRound(ptr[1]*weight); 18. ptr[2] = cvRound(ptr[2]*weight); } 19. cvSaveImage("copy.png", image ); 20. cvNamedWindow( "test", 1 ); 21. cvShowImage( "test", image ); 22. cvWaitKey(); 23. return 0; }
  • 19. 基本数据结构点:CvPoint 、CvPoint2D32f、CvPoint3D32f 矩形框大小:CvSize 、CvSize2D32f 矩形框:CvRect 可以存放1-4个数值的数组:CvScalar 定义迭代算法的终止规则:CvTermCriteria 矩阵:CvMat 、CvMatND 、CvSparseMat IPL图像头部:IplImage 定义不确定的数组:CvArr (仅作函数参数)19
  • 20. 点数据结构CvPoint 二维坐标系下的点,类型为整型 typedef struct CvPoint { int x; /* X坐标, 通常以0为基点 */ int y; /* y坐标, 通常以0为基点 */ }CvPoint; /* 构造函数 */ inline CvPoint cvPoint( int x, int y ); /* 从 CvPoint2D32f类型转换得来 */ inline CvPoint cvPointFrom32f( CvPoint2D32f point ) CvPoint2D32f :二维坐标下的点,类型为浮点 CvPoint3D32f :三维坐标下的点,类型为浮点20
  • 21. 矩形框大小数据结构CvSize 矩形框大小,以像素为精度 typedef struct CvSize{ int width; /* 矩形宽 */ int height; /* 矩形高 */ }CvSize; /* 构造函数 */ inline CvSize cvSize( int width, int height ); CvSize2D32f 21
  • 22. 矩形框数据结构CvRect 矩形框的偏移和大小 typedef struct CvRect{ int x; /* 方形的最左角的x-坐标 */ int y; /* 方形的最上或者最下角的y-坐标 */ int width; /* 宽 */ int height; /* 高 */ }CvRect; /* 构造函数*/ inline CvRect cvRect(int x, int y, int width, int height);22
  • 23. 矩阵数据结构CvMat 二维矩阵 typedef struct CvMat { int type; /* CvMat 标识, 元素类型和标记 */ int step; /* 以字节为单位的行数据长度*/ int rows; /*行数*/ int cols; /*列数*/ int* refcount; /* 数据引用计数 */ union{ uchar* ptr; short* s; int* i; float* fl; double* db; } data; /* data 指针 */ …… 23
  • 24. 矩阵数据结构CvMatND:多维、多通道密集数组 CvSparseMat:多维、多通道稀疏数组 CvArr:不确定数组24
  • 25. 图像头数据IplImage:IPL 图像头 25
  • 26. 图像头数据IplImage:IPL 图像头 typedef struct _IplImage { int nSize; /* IplImage大小 */ int ID; /* 版本 (=0)*/ int nChannels; /* 大多数OPENCV函数支持1,2,3 或4 个通道 */ int alphaChannel; /* 被OpenCV忽略 */ int depth; /* 像素的位深度: IPL_DEPTH_8U,IPL_DEPTH_8S, IPL_DEPTH_16U,IPL_DEPTH_16S, IPL_DEPTH_32S, IPL_DEPTH_32F and IPL_DEPTH_64F 可支持 */ char colorModel[4]; char channelSeq[4]; /* 被OpenCV忽略 */ int dataOrder; /* 0 - 交叉存取颜色通道, 1 - 分开的颜色通道. cvCreateImage只能创建交叉存取图像 */ int origin; /* 0 - 顶—左结构,1 - 底—左结构 (BMP风格) */ int align; /* 图像行排列 (4 or 8). OpenCV 用widthStep 代替 */ 26
  • 27. int width; /* 图像宽像素数 */ int height; /* 图像高像素数*/ struct _IplROI *roi;/* 图像感兴趣区域. 当该值非空只对该区域进行处理 */ struct _IplImage *maskROI; /* 在 OpenCV中必须置NULL */ void *imageId; /* 同上*/ struct _IplTileInfo *tileInfo; /*同上*/ int imageSize; /* 图像数据大小,单位字节*/ char *imageData; /* 指向排列的图像数据 */ int widthStep; /* 排列的图像行大小,以字节为单位 */ int BorderMode[4]; int BorderConst[4]; /* 边际结束模式, 被忽略*/ char *imageDataOrigin; /* 指针指向一个不同的图像数据结构,是 为了纠正图像内存分配准备的 */ }IplImage; 27
  • 28. 图像头数据IplImage结构来自于 Intel Image Processing Library。OpenCV 只支持其中的一个子集: alphaChannel 在OpenCV中被忽略。 colorModel 和channelSeq 被OpenCV忽略。 dataOrder 必须是IPL_DATA_ORDER_PIXEL (颜色通道是交叉存取),然而平面图像的被选择通道可以被处理,就像COI(感兴趣的通道)被设置过一样。 align 是被OpenCV忽略的,而用 widthStep 去访问后继的图像行。 不支持maskROI 。处理MASK的函数把他当作一个分离的参数。MASK在 OpenCV 里是 8-bit,然而在 IPL他是 1-bit。 tileInfo 不支持。 BorderMode和BorderConst是不支持的。 OpenCV处理ROI有不同的要求。要求原图像和目标图像的尺寸或 ROI的尺寸必须精确匹配。28
  • 29. 基本OpenCV操作矩阵的使用与操作 GUI命令 图像的使用与操作 视频的使用与操作
  • 30. 矩阵的使用与操作(1)创建矩阵 CreateMat CvMat* cvCreateMat( int rows, int cols, int type ); rows 矩阵行数。 cols 矩阵列数。 type 矩阵元素类型。 通常以 CV_<比特数>(S|U|F)C<通道数>型式描述, 例如: CV_8UC1 意思是一个8-bit 无符号单通道矩阵, CV_32SC2 意思是一个32-bit 有符号二个通道的矩阵。 函数 cvCreateMat 为新的矩阵分配头和下面的数据,并且返回一个指向新创建的矩阵的指针。矩阵按行存贮。所有的行以4个字节对齐。 删除矩阵 ReleaseMat void cvReleaseMat( CvMat** mat ); 例如:CvMat* M = cvCreateMat( 4, 4, CV_32FC1); cvReleaseMat( &M);30
  • 31. 矩阵的使用与操作(2)复制矩阵CloneMat CvMat* cvCloneMat( const CvMat* mat ); 例如: CvMat* M1 = cvCreateMat( 4, 4, CV_32FC1); CvMat* M2; M2 = cvCloneMat(M1); 初始化矩阵 方法1:用cvMat初始化 double a[] = { 1, 2, 3, 4,5, 6, 7, 8,9, 10, 11, 12 }; CvMat Ma = cvMat( 3, 4, CV_64FC1, a); 方法2:用cvCreateMatHeader初始化 CvMat Ma; cvInitMatHeader( &Ma, 3, 4, CV_64FC1, a); 初始化单位矩阵 CvMat* M = cvCreateMat( 4, 4, CV_32FC1); cvSetIdentity(M);31
  • 32. 矩阵的使用与操作(3)访问矩阵元素 (1)直接访问 cvmSet(M, i, j, 2, 0); //Set M(i,j) t = cvmGet(M, i, j); //Get M(i,j) (2)已知对齐方式的直接访问 CvMat* M = cvCreateMat( 4, 4, CV_32FC1); int n = M->cols; float *data = M->data.fl; data[i*n+j] = 3.0; //假设32位对齐 (3)未知对齐方式的直接访问 CvMat* M = cvCreateMat( 4, 4, CV_32FC1); int step = M->step/sizeof(float); float *data = M->data.fl; (data + i*step)[j] = 3.0; (4)直接访问一个已初始化的矩阵 Ma[i*4 + j] = 2.0; 32
  • 33. 矩阵的使用与操作(4)矩阵间的操作 CvMat *Ma, *Mb, *Mc; cvAdd(Ma, Mb, Mc); //Ma + Mb -> Mc cvSub(Ma, Mb, Mc); //Ma - Mb -> Mc cvMatMul(Ma, Mb, Mc); //Ma * Mb -> Mc 矩阵元素间的操作 cvMul(Ma, Mb, Mc); //Ma. * Mb -> Mc cvDiv(Ma, Mb, Mc); //Ma. / Mb -> Mc cvAddS(Ma, cvScalar(-10.0), Mc); //Ma. -10 -> Mc 单个矩阵的操作 cvTranspose(Ma, Mb); //transpose(Ma) -> Mb CvScalar t=cvTrace(Ma); //trace(Ma) ->t.val[0] double d = cvDet(Ma); //det(Ma) ->d cvInvert(Ma, Mb); //inv(Ma) ->Mb33
  • 34. 矩阵的使用与操作(5)向量乘法 假设Va, Vb, Vc均为n元素向量 double res = cvDotProduct(&Va, &Vb); //Va •Vb -> res cvCrossProduct(&Va, &Vb , &Vc); // Va ×Vb -> Vc 矩阵特征值分解 假设A, E均为n*n方阵,I为n元素向量 cvEigenVV(&A, &E , &I); SVD 假设A, U, D, V均为n*n方阵 cvSVD(A, D, U, V, CV_SVD_U_T|| CV_SVD_V_T); //A=UDV~T标志使U和V以转置方式返回 非齐次线性系统的求解 假设A为n*n方阵,x,b均为n元素向量 cvSolve(&A, &b, &x)34
  • 35. 矩阵的使用与操作:示例(1)#include "cv.h" #include "highgui.h" #include void PrintMat(CvMat *A); // 显示矩阵 void Test_Multiply(); // 测试矩阵乘法 void Test_DCT(); // 计算DCT变换 int main() { Test_Multiply(); Test_DCT(); return 0; } 35
  • 36. 矩阵的使用与操作:示例(2)void PrintMat(CvMat* A) {// 显示矩阵 int i,j; for(i=0;irows;i++) { printf("\n"); switch( CV_MAT_DEPTH(A->type) ) { case CV_32F: case CV_64F: for(j=0; jcols; j++) printf("%9.3f ", (float) cvGetReal2D( A, i, j )); break; case CV_8U: case CV_16U: for(j=0;jcols;j++) printf("%6d",(int)cvGetReal2D( A, i, j )); break; default: break; } } printf("\n"); }36
  • 37. 矩阵的使用与操作:示例(3)void Test_Multiply(){// Test matrix multiply double a[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 }; double b[] = {1, 5, 9, 2, 6, 10, 3, 7, 11, 4, 8, 12 }; double c[9]; CvMat Ma, Mb, Mc; printf("\n=== Test multiply ==="); cvInitMatHeader( &Ma, 3, 4, CV_64FC1, a, CV_AUTOSTEP ); cvInitMatHeader( &Mb, 4, 3, CV_64FC1, b, CV_AUTOSTEP ); cvInitMatHeader( &Mc, 3, 3, CV_64FC1, c, CV_AUTOSTEP ); cvMatMulAdd( &Ma, &Mb, 0, &Mc ); PrintMat(&Ma); PrintMat(&Mb); PrintMat(&Mc); return; }37
  • 38. 矩阵的使用与操作:示例(4)void Test_DCT(){// test 1-d and 2-d dct transform float data[] = { 1, 2, 3, 4, 5, 6, 7, 8 }; CvMat a; a = cvMat(2,4,CV_32FC1,data); printf("\n=== Test DCT ==="); printf("\nOriginal matrix = "); PrintMat(&a); cvDCT(&a, &a, CV_DXT_FORWARD); printf("\n2-D DCT = "); PrintMat(&a); cvDCT(&a, &a, CV_DXT_INVERSE); printf("\n2-D IDCT = "); PrintMat(&a); }38
  • 39. Discrete Cosine Transform(DCT) Transform kernel of two-dimensional DCT is where Clearly, the kernel for the DCT is both separable and symmetric, and hence the DCT may be implemented as a series of one dimensional DCTs. 39
  • 40. Discrete Cosine Transform(DCT)Forward transform and Inverse transform where u, v, x, y = 0, 1, 2, …, N-140
  • 41. Transform Matrix of DCTwhere:41
  • 42. Example of DCT42
  • 43. DCT的几点说明DCT变换避免了复数运算。由于图像矩阵是实数矩阵,那么它的DCT也是实数 DCT是正交变换,其变换矩阵是正交阵,变换核是可分离的。 DCT有快速算法 DCT与IDCT具有相同的变换核,因此具有相同的变换矩阵,即正变换与逆变换公用同一个算法模块 DCT具有更强的信息集中能力,能将最多的信息放到最小的系数上去。 Dr. JI ZHEN43
  • 44. HighGUI操作"Smart" windows Image I/O, rendering Processing keyboard and other events, timeouts Trackbars Mouse callbacks Video I/O44
  • 45. WindowscvNamedWindow(window_name, fixed_size_flag) cvNamedWindow(window_name, x, y) cvDestroyWindow(window_name) creates window accessed by its name. Window handles repaint, resize events. Its position is remembered in registry. cvNamedWindow("ViewA",1); cvMoveWindow("ViewA",300,100); cvDestroyWindow("ViewA"); … cvShowImage(window_name, image); copies the image to window buffer, then repaints it when necessary. {8u|16s|32s|32f}{C1|3|4} are supported. only the whole window contents can be modified. Dynamic updates of parts of the window are done using operations on images, drawing functions etc.45
  • 46. Windows等待按键cvWaitKey int cvWaitKey( int delay=0 ) 如果delay<=0, 则无限等待,则等待delay毫秒则返回 在程序循环中,有时候由于程序一直处于计算中,窗口无法重新恢复(如读出视频中的所有帧并显示),可以加入cvWaitKey,使之等待几毫秒,让窗口完成重新绘制再执行其他操作46
  • 47. 图像的使用与操作(1)创建头并分配数据 CreateImage IplImage* cvCreateImage( CvSize size, int depth, int channels ); size:图像宽、高. depth:图像元素的位深度,IPL_DEPTH_{8U|8S|16U|16S|32S|32F|64F} channels:每个元素(像素)的颜色通道数量.可以是 1, 2, 3 或 4. 释放头和图像数据 ReleaseImage void cvReleaseImage( IplImage** image ); 复制图像 CloneImage IplImage* cvCloneImage( const IplImage* image );47
  • 48. 图像的使用与操作(2)头分配CreateImageHeader IplImage* cvCreateImageHeader( CvSize size, int depth, int channels ) 初始化被用图分配的图像头 InitImageHeader IplImage* cvInitImageHeader( IplImage* image, CvSize size, int depth, int channels, int origin=0, int align=4 ); origin IPL_ORIGIN_TL 或 IPL_ORIGIN_BL. align 图像行排列, 典型的 4 或 8 字节. 函数 cvInitImageHeader 初始化图像头结构, 指向用户指定的图像并且返回这个指针。 释放头 ReleaseImageHeader void cvReleaseImageHeader( IplImage** image );48
  • 49. 程序示例:Create image#include "cv.h" #include "highgui.h " #include int main() { IplImage *cvImg; // image used for visualisation CvSize imgSize; // size of visualisation image int i = 0, j = 0; imgSize.width = 640; imgSize.height = 480; cvImg = cvCreateImage( imgSize, 8, 1 ); // creation of a 8 bits depth gray image for ( i = 0; i < imgSize.width; i++ ) // image is filled with gray values corresponding for ( j = 0; j < imgSize.height; j++ )// to the 256 modulus of their position in the image ((uchar*)(cvImg->imageData + cvImg->widthStep*j))[i] = ( char ) ( ( i * j ) % 256 ); cvNamedWindow( "Testing OpenCV...", 1 ); // creation of a visualisation window cvShowImage( "Testing OpenCV...", cvImg ); // image visualisation cvWaitKey( 0 ); // wait for key cvDestroyWindow( "image" ); // close window cvReleaseImage( &cvImg ); // memory release return( 0 ); // stopping the program }
  • 50. 程序示例:Create image50
  • 51. 图像的使用与操作(3)从文件读图像cvLoadImage IplImage* cvLoadImage(char* fileName, int flag=1) OpenCV支持的图像格式:BMP、DIB、JPG、PNG、PBM、PGM、PPM、SR、RAS和TIFF 写图像到文件cvSaveImage IplImage* cvSaveImage(char* fileName, IplImage* img) 图像转换 cvConvertImage(IplImage* src, IplImage* dst, int flags=0); //灰度图彩色图 cvCvtColor(IplImage* src, IplImage* dst, int code); //彩色图彩色图/灰度图 Code=CV_2:,=RGB,BGR,GRAY,HSV,YCrCb,XYZ,Luv,HLS 51
  • 52. 图像装载与显示示例#include "cv.h" #include "highgui.h" #include // file example.bmp in the images directory char name0[] = "images/example.bmp"; // file example.jpg in the images directory char name1[] = "images/example.jpg";
  • 53. 图像装载与显示示例int main() { IplImage* img0 = NULL; IplImage* img1 = NULL; img0 = cvLoadImage( name0, -1 ); img1 = cvLoadImage( name1, -1 ); cvNamedWindow( "image0", 1 ); cvNamedWindow( "image1", 1 ); cvShowImage( "image0", img0 ); cvShowImage( "image1", img1 ); cvWaitKey(0); // wait for key to close the windows cvReleaseImage( &img0 ); cvReleaseImage( &img1 ); return(0); }
  • 54. 图像处理函数cvSub( img0, img1, res, 0 ) subtract img0 from img1 and store the result in res
  • 55. 休息55
  • 56. 第二部分基本视频操作 OpenCV的动态结构及操作 基于OpenCV的图像/视频处理(6个示例) 基于OpenCV的研究与开发:DEMO 56
  • 57. 视频的使用和操作(1)打开摄像头 CvCapture* cvCaptureFromCAM(camera_id=0); 打开文件 CvCapture* cvCaptureFromFile(videofile_path); CvCapture* cvCaptureFromAVI("inflie.avi"); 捕捉某一帧 cvGrabFrame(capture) //抓住一帧,为快速遍历视频帧 IplImage* img=cvRetrieveImage(capture);//把Grab的帧取出,或 IplImage* cvQueryFrame(capture); 释放捕捉源 cvReleaseCapture(&capture); 57
  • 58. 视频的使用和操作(2)保存视频文件 typedef struct CvVideoWriter; CvVideoWriter* cvCreateVideoWriter( const char* filename, int fourcc, double fps, CvSize frame_size, int is_color=1 ); int cvWriteFrame( CvVideoWriter* writer, const IplImage* image ); void cvReleaseVideoWriter( CvVideoWriter** writer ); 获取/设置视频帧信息 cvGetCaptureProperty(capture, property_id); cvSetCaptureProperty(capture, property_id, value); CV_CAP_PROP_POS_MSEC - video capture timestamp CV_CAP_PROP_POS_FRAMES - 0-based index of the frame CV_CAP_PROP_POS_AVI_RATIO - relative position of video(0-start, 1-end) CV_CAP_PROP_FRAME_WIDTH - width of frames in the video stream CV_CAP_PROP_FRAME_HEIGHT - height of frames in the video stream CV_CAP_PROP_FPS - frame rate CV_CAP_PROP_FOURCC - 4-character code of codec CV_CAP_PROP_FRAME_COUNT - number of frames in video file58
  • 59. 视频操作示例(1)功能:从摄像头或者AVI文件中得到视频流,对视频流进行边缘检测,并输出结果。 主要思想: 获取视频( cvCaptureFromCAM 或cvCaptureFromAVI) 循环处理每帧( cvQueryFrame ) 将多通道数组分割为多个单通道数组(cvCvtPixToPlan) 在每个通道内进行Laplace变换( cvLaplace),计算图像的边缘信息 恢复多通道图像( cvCvtPlaneToPix )并显示59
  • 60. Laplace变换Laplace变换:工程数学中常用的一种积分变换。 普拉斯变换的存在性:关于一个函数f(t)的拉普拉斯变换,只有在拉普拉斯积分是收敛的情况下才存在。也就是说, f(t)必须是在对于t>0的每一个有限区间内都是片断性连续的,且当t趋于无穷大的时候, f(t)是指数阶地变化。 拉普拉斯变换的重大意义在于:将一个信号从时域上,转换为复频域(s域)上来表示60
  • 61. Laplace变换void cvLaplace(const CvArr *src, CvArr * dst, int aperture_size=0) 1、用Sobel算子计算图像二阶x和y的差分 src:输入图像;dst:输出图像;xorder:x方向的差分阶数;yorder:y方向的差分阶数 2、按如下公式求和 对aperture_size=1则给出最快计算结果,相当于对每个图像采用如下内核做卷积 61
  • 62. 视频操作示例(1)#include "cv.h" #include "highgui.h" #include #include int main( int argc, char** argv ){ IplImage* laplace = 0; IplImage* colorlaplace = 0; IplImage* planes[3] = { 0, 0, 0 }; // 多个图像面 CvCapture* capture = 0; If(argc==1||(argc==2&& strlen(argv[1]) == 1&&isdigit(argv[1][0]))) capture = cvCaptureFromCAM(argc == 2 ? argv[1][0] - '0' : 0 ); else if( argc == 2 ) capture = cvCaptureFromAVI( argv[1] ); if( !capture ) { fprintf(stderr,"Could not initialize capturing...\n"); return -1; } 62
  • 63. 视频操作示例(2) cvNamedWindow( "Laplacian", 0 ); for(;;) { // 循环捕捉,直到用户按键跳出循环体 IplImage* frame = 0; int I; frame = cvQueryFrame( capture ); if( !frame ) break; if( !laplace ) { for( i = 0; i < 3; i++ ) planes[i]=cvCreateImage(cvSize(frame->width,frame->height),8,1); laplace = cvCreateImage( cvSize(frame->width,frame->height), IPL_DEPTH_16S, 1 ); //存放单通道Laplace结果 colorlaplace = cvCreateImage( cvSize(frame->width,frame->height), 8, 3 ); //存放多通道Laplace结果 } cvCvtPixToPlane( frame, planes[0], planes[1], planes[2], 0 );//将多通道 //数组分割为多个单通道数组 for( i = 0; i < 3; i++ ) { cvLaplace( planes[i], laplace, 3 ); //计算输入图像的Laplace变换 cvConvertScaleAbs( laplace, planes[i], 1, 0 ); //用线性变换将输入数组 //元素转化为8位无符号整数 } 63
  • 64. 视频操作示例(3) cvCvtPlaneToPix( planes[0], planes[1], planes[2], 0, colorlaplace ); colorlaplace->origin = frame->origin; cvShowImage("Laplacian", colorlaplace ); if( cvWaitKey(10) >= 0 ) break; } cvReleaseCapture( &capture ); cvDestroyWindow("Laplacian"); return 0; }64
  • 65. OpenCV的动态结构及操作动态结构 内存存储 序列 集合与稀疏矩阵 数据保存 配置文件
  • 66. 动态结构:when 2d array is not enoughThe sample task: collect the locations of all non-zero pixels in the image. Where to store the locations? Possible solutions: Allocate array of maximum size (sizeof(CvPoint)*width*height – a huge value) Use two-pass algorithm Use list or similar data structure … (Any other ideas?)66// construct sequence of non-zero pixel locations CvSeq* get_non_zeros( const IplImage* img, CvMemStorage* storage ){ CvSeq* seq = cvCreateSeq( CV_32SC2, sizeof(CvSeq), sizeof(CvPoint), storage ); for( int i = 0; i < img->height; i++ ) for( int j = 0; j < img->width; j++ ) if( CV_IMAGE_ELEM(img, uchar, i, j) ) { CvPoint pt={j,i}; cvSeqPush( seq, &pt ); } return seq; } //CvSeq:定义非固定元素的序列
  • 67. 内存管理Memory storage is a linked list of memory blocks. Functions to remember: cvCreateMemStorage(block_size), cvReleaseMemStorage(storage); cvClearMemStorage(storage); cvMemStorageAlloc(storage,size); All OpenCV "dynamic structures" reside in memory storages. The model is very efficient for repetitive processing, where cvClearMemStorage is called on top level between iterations.67
  • 68. 序列Sequence is a linked list of memory blocks Large sequences may be split into multiple storage blocks, while several small sequences may fit into the single storage block. Sequence is deque: adding/removing elements to/from either end is O(1) operation, inserting/removing elements from the middle is O(N). Accessing arbitrary element is also ~O(1) (when storage block size >> sequence block) Sequences can not be released, use cv{Clear|Release}MemStorage() Basic Functions: cvCreateSeq(flags,header_size,element_size,storage), cvSeqPush[Front](seq,element), cvSeqPop[Front](seq,element), cvClearSeq(seq), cvGetSeqElem(seq,index), cvStartReadSeq(seq,reader), cvStartAppendToSeq(seq,writer).68
  • 69. 集合与稀疏矩阵Set is variant of sequence with "free node list" and it’s active elements do not follow each other – that is, it is convenient way to organize a "heap" of same-size memory blocks. Sparse matrix in OpenCV uses CvSet for storing elements.69// Collecting the image colors CvSparseMat* get_color_map( const IplImage* img ) { int dims[] = { 256, 256, 256 }; CvSparseMat* cmap = cvCreateSparseMat(3, dims, CV_32SC1); for( int i = 0; i < img->height; i++ ) for ( int j = 0; j < img->width; j++ ) { uchar* ptr=&CV_IMAGE_ELEM(img,uchar,i,j*3); int idx[] = {ptr[0],ptr[1],ptr[2]}; ((int*)cvPtrND(cmap,idx))[0]++; } CvSparseMatIterator it; // print the map for(CvSparseNode *node = cvInitSparseMatIterator( mat, &iterator ); node != 0; node = cvGetNextSparseNode( &iterator )) { int* idx = CV_NODE_IDX(cmap,node); int count=*(int*)CV_NODE_VAL(cmap,idx); printf( "(b=%d,g=%d,r=%d): %d\n", idx[0], idx[1], idx[2], count ); } return cmap; }
  • 70. 数据存储数据存储与装载70// somewhere deep in your code… you get 5x5 // matrix that you’d want to save { CvMat A = cvMat( 5, 5, CV_32F, the_matrix_data ); cvSave( "my_matrix.xml", &A ); } // to load it then in some other program use … CvMat* A1 = (CvMat*)cvLoad( "my_matrix.xml" ); 5 5
    f
    1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 1.
  • 71. 配置文件71CvFileStorage* fs = cvOpenFileStorage("cfg.xml", 0, CV_STORAGE_WRITE); cvWriteInt( fs, "frame_count", 10 ); cvWriteStartWriteStruct( fs, "frame_size", CV_NODE_SEQ); cvWriteInt( fs, 0, 320 ); cvWriteint( fs, 0, 200 ); cvEndWriteStruct(fs); cvWrite( fs, "color_cvt_matrix", cmatrix ); cvReleaseFileStorage( &fs );Writing config fileCvFileStorage* fs = cvOpenFileStorage("cfg.xml", 0, CV_STORAGE_READ); int frame_count = cvReadIntByName( fs, 0, "frame_count", 10 /* default value */ ); CvSeq* s = cvGetFileNodeByName(fs,0,"frame_size")->data.seq; int frame_width = cvReadInt( (CvFileNode*)cvGetSeqElem(s,0) ); int frame_height = cvReadInt( (CvFileNode*)cvGetSeqElem(s,1) ); CvMat* color_cvt_matrix = (CvMat*)cvRead(fs,0,"color_cvt_matrix"); cvReleaseFileStorage( &fs );Reading config file 10 320 200 3 3
    f
    cfg.xml
  • 72. 基于OpenCV的图像/视频处理角点检测 Canny边缘检测 轮廓(Contour)检测 视频人脸检测 运动对象检测 运动对象跟踪:利用OpenCV架构大型程序 注:本讲义中部分代码来自OpenCV样例,部分代码来自《OpenCV教程——基础篇》,部分代码从网上收集
  • 73. 程序示例1:角点检测什么是角点? Corners as Interest Points Corner points are formed from two or more edges and edges usually define the boundary between two different objects or parts of the same object 角点检测的应用 stereo matching image registration (of particular importance in medical imaging) stitching of panoramic photographs object detection/recognition motion tracking robot navigation 73
  • 74. 程序示例1:角点检测四种OpenCV角点检测方法 cvGoodFeaturesToTrack,cvCornerMinEigenVal,CornerEigenValsAndVecs:用到了图像差分的相关矩阵。 CornerHarris:利用CornerEigenValsAndVecs得到的图像差分相关矩阵,据其特征值和向量,判断角点,Harris角点探测器R(x, y) =det(M)-k*trace^2(M) FindCornerSubPix:子象素级角点定位,是基于对向量正交性的观测而实现的。 cvPreCornerDetect:计算一个二阶导多项式,角点被认为是函数的局部最大值。其实就是利用了梯度的变化率在边缘走 向的投影能反映尖锐程度。实验表明这是一种比cvGoodFeaturesToTrack粗略的办法。 注意:1,2,4都是检测像素级的角点。 74
  • 75. 程序示例1:角点检测(1)#include #include "cv.h" #include "highgui.h" #define max_corners 100 int main( int argc, char** argv ){ int cornerCount=max_corners; CvPoint2D32f corners[max_corners]; IplImage *srcImage = 0, *grayImage = 0, *corners1 = 0, *corners2 = 0; int i; CvScalar color = CV_RGB(255,0,0); char* filename = argc == 2 ? argv[1] : (char*) "pic3.png"; cvNamedWindow("image", 1 ); srcImage = cvLoadImage(filename, 1); //Load the image to be processed grayImage = cvCreateImage(cvGetSize(srcImage), IPL_DEPTH_8U, 1); //copy the source image to copy image after converting the format cvCvtColor(srcImage, grayImage, CV_BGR2GRAY); //create empty images of same size as the copied images corners1= cvCreateImage(cvGetSize(srcImage), IPL_DEPTH_32F, 1); corners2= cvCreateImage(cvGetSize(srcImage),IPL_DEPTH_32F, 1);75
  • 76. 程序示例1:角点检测(2) cvGoodFeaturesToTrack (grayImage, corners1, corners2, corners, &cornerCount, 0.05, 5, 0, 3, // block size 0, // not use harris 0.4 ); printf("num corners found: %d\n", cornerCount); // draw circles at each corner location in the gray image and //print out a list the corners if(cornerCount>0) { for (i=0; i
  • 77. 程序示例1:角点检测 问题:是否利用OpenCV的角点检测函数就万事大吉?77
  • 78. 程序示例1:角点检测OpenCV的角点检测算法在某些情况下性能不是太好78
  • 79. 程序示例2:Canny边缘检测Edges are the boundaries separating regions with different brightness or color. J.Canny suggested in [Canny86] an efficient method for detecting edges. It takes grayscale image on input and returns bi-level image where non-zero pixels mark detected edges. cvCanny:采用Canny算子寻找并标志输入图像中的边缘 void cvCanny(const CvArr* image, CvArr*edges, double threshold1, double threshold2, int aperture_size=3); Threshold1和threshold2:两个阈值,其中小的阈值用于控制边缘连接,打阈值用于控制强边缘的初始分割 aperture_size:Sobel算子内核大小 Canny算子: 用高斯滤波器平滑图像; 用一阶偏导的有限差分计算梯度的幅值和方向 对梯度幅值进行非极大值抑制 用双阈值算法检测和链接边缘 采用高斯平滑函数79
  • 80. 程序示例2:Canny边缘检测(1)#include "cv.h" #include "highgui.h" char wndname[] = "Edge"; char tbarname[] = "Threshold"; int edge_thresh = 1; IplImage *image = 0, *cedge = 0, *gray = 0, *edge = 0; void on_trackbar(int h){// 定义跟踪条的 callback 函数 cvSmooth( gray, edge, CV_GAUSSIAN, 3, 3, 0 ); //高斯平滑 cvNot( gray, edge ); //数组取反 // 对灰度图像进行边缘检测 cvCanny(gray, edge, (float)edge_thresh, (float)edge_thresh*3, 3); cvZero( cedge ); // copy edge points cvCopy( image, cedge, edge ); cvShowImage(wndname, cedge); }80
  • 81. 程序示例2:Canny边缘检测(2)int main( int argc, char** argv ){ char* filename = argc == 2 ? argv[1] : (char*)"fruits.jpg"; if( (image = cvLoadImage( filename, 1)) == 0 ) return -1; cedge = cvCreateImage(cvSize(image->width,image->height), IPL_DEPTH_8U, 3); // Create the output image // 将彩色图像转换为灰度图像 gray = cvCreateImage(cvSize(image->width,image->height), IPL_DEPTH_8U, 1); edge = cvCreateImage(cvSize(image->width,image->height), IPL_DEPTH_8U, 1); cvCvtColor(image, gray, CV_BGR2GRAY); cvNamedWindow(wndname, 1); // Create a window // create a toolbar, edge_thresh corresponds to the bar value cvCreateTrackbar(tbarname, wndname, &edge_thresh, 100, on_trackbar); on_trackbar(0); // Show the image cvWaitKey(0); // Wait for a key stroke cvReleaseImage(&image); cvReleaseImage(&gray); cvReleaseImage(&edge); cvDestroyWindow(wndname); return 0; }81
  • 82. 程序示例2:Canny边缘检测82阈值=0阈值=30
  • 83. 程序示例3:轮廓(Contour)检测83
  • 84. 程序示例3:轮廓(Contour)检测int cvFindContours( CvArr* image, CvMemStorage* storage, CvSeq** first contour, int header size=sizeof(CvContour), int mode=CV RETR LIST, int method=CV CHAIN APPROX SIMPLE, CvPoint offset=cvPoint(0,0) ); 宏CV_GET_SEQ可以读出轮廓中的任意点 CV_GET_SEQ_ELEM( TYPE, seq, index )  (TYPE*)cvGetSeqElem( (CvSeq*)(seq), (index) ) 84
  • 85. 程序示例3:轮廓(Contour)检测(1)#include "cv.h" #include "cxcore.h" #include "highgui.h" int main( int argc, char** argv ){ IplImage* pImg = NULL; //声明IplImage指针 IplImage* pContourImg = NULL; CvMemStorage * storage = cvCreateMemStorage(0); CvSeq * contour = 0; int mode = CV_RETR_EXTERNAL; if( argc == 3) if(strcmp(argv[2], "all") == 0) mode = CV_RETR_CCOMP; //内外轮廓都检测 cvNamedWindow("src", 1); //创建窗口 cvNamedWindow("contour",1); //载入图像,强制转化为Gray if( argc >= 2 && (pImg = cvLoadImage( argv[1], 0)) != 0 ) { //为轮廓显示图像申请空间。3通道图像,以便用彩色显示 cvShowImage( "src", pImg ); pContourImg = cvCreateImage(cvGetSize(pImg),IPL_DEPTH_8U, 3);85
  • 86. 程序示例3:轮廓(Contour)检测(2) //copy source image and convert it to BGR image cvCvtColor(pImg, pContourImg, CV_GRAY2BGR); cvFindContours( pImg, storage, &contour, sizeof(CvContour), mode, CV_CHAIN_APPROX_SIMPLE); //查找contour } else { cvDestroyWindow( "src" ); //销毁窗口 cvDestroyWindow( "contour" ); cvReleaseMemStorage(&storage); return -1; } cvDrawContours(pContourImg, contour, CV_RGB(0,0,255), CV_RGB(255, 0, 0), 2, 2, 8); //将轮廓画出 cvShowImage( "contour", pContourImg ); //显示图像 cvWaitKey(0); cvDestroyWindow("src" ); cvDestroyWindow("contour"); //销毁窗口 cvReleaseImage( &pImg ); cvReleaseImage( &pContourImg ); //释放图像 cvReleaseMemStorage(&storage); return 0; }86
  • 87. DEMO87
  • 88. 问题输入Lena图片结果会怎么样?88
  • 89. 程序示例4:人脸检测89图像中人脸检测视频中人脸检测
  • 90. 程序示例4:人脸检测90Haar Features + Adaboost:combines the performance of many "weak" classifiers to produce a powerful 'committee’
  • 91. 基于AdaBoost的快速人脸检测基于分级分类器的加速策略 大量候选窗口可以利用非常少量的特征(简单快速的分类器)就可以排除是人脸的可能性! 只有极少数需要大量的特征(更复杂的更慢的分类器来判别是否人脸)91
  • 92. 程序示例4:人脸检测数据结构 CvHaarFeature, CvHaarClassifier, CvHaarStageClassifier, CvHaarClassifierCascade定义Boosted Haar分类器级联结构 Cascade: Stage 1: Classifier 11: Feature 11 Classifier 12: Feature 12 …… Stage 2: Classifier 21: Feature 21 …… 数据操作 HaarDetectObjects:检测图像中的目标 LoadHaarClassifierCascase ReleaseHaarClassifierCascase 92
  • 93. 程序示例4:人脸检测(1)#include "cv.h" #include "highgui.h" int main( int argc, char** argv ) { static CvMemStorage* storage = 0; static CvHaarClassifierCascade* cascade = 0; CvCapture* capture = 0; int optlen = strlen("--cascade="); if( argc != 3 || strncmp( argv[1], "--cascade=", optlen )) return -1; cascade = (CvHaarClassifierCascade*) cvLoad( argv[1] + optlen ); capture = cvCaptureFromAVI( argv[2] ); if( !cascade || !capture ) return -1; storage = cvCreateMemStorage(0); cvNamedWindow( "Video", 1 ); 93
  • 94. 程序示例4:人脸检测(2)for(;;) { IplImage* frame = cvQueryFrame( capture ), *img; CvSeq* faces; if( !frame ) break; img = cvCloneImage(frame); img->origin = 0; if( frame->origin ) cvFlip(img,img); cvClearMemStorage( &storage ); faces = cvHaarDetectObjects( img, cascade, storage, 1.1, 2, CV_HAAR_DO_CANNY_PRUNING, cvSize(20, 20) ); for( int i = 0; i < (faces ? faces->total : 0); i++ ) {//多个人脸情况 CvRect* r = (CvRect*)cvGetSeqElem( faces, i ); cvRectangle( img, cvPoint(r->x,r->y), cvPoint(r->x+r->width,r->y+r->height), CV_RGB(255,0,0), 3 ); } cvShowImage( "Video", img ); cvReleaseImage( &img ); if( cvWaitKey(10) >= 0 ) break; } cvReleaseCapture( &capture ); return 0; }94
  • 95. DEMO 功能更完善的人脸检测demo OpenCV的样例facedetect.c95
  • 96. DEMO中人脸检测的尺度问题96
  • 97. 程序示例5:运动对象检测运动:视频区别于图像的最重要特征97
  • 98. 程序示例5:运动对象检测(1)#include #include #include #include int main( int argc, char** argv ){ IplImage* pFrame = NULL; //声明IplImage指针 IplImage* pFrImg = NULL; IplImage* pBkImg = NULL; CvMat* pFrameMat = NULL; //声明矩阵指针 CvMat* pFrMat = NULL; CvMat* pBkMat = NULL; CvCapture* pCapture = NULL; int nFrmNum = 0; cvNamedWindow("video", 1); //创建窗口 cvNamedWindow("background",1); cvNamedWindow("foreground",1); cvMoveWindow("video", 30, 0); //使窗口有序排列 cvMoveWindow("background", 360, 0); cvMoveWindow("foreground", 690, 0); if( argc != 2 ) { fprintf(stderr, "Usage: bkgrd \n"); return -1; }98
  • 99. 程序示例5:运动对象检测(2)if( !(pCapture = cvCaptureFromFile(argv[1]))) {//打开视频文件 fprintf(stderr, "Can not open video file %s\n", argv[1]); return -2; } while(pFrame = cvQueryFrame( pCapture )) {//逐帧读取视频 nFrmNum++; if(nFrmNum == 1){//如果是第一帧,需要申请内存,并初始化 pBkImg = cvCreateImage(cvSize(pFrame->width, pFrame- >height), IPL_DEPTH_8U,1); pFrImg = cvCreateImage(cvSize(pFrame->width, pFrame- >height), IPL_DEPTH_8U,1); pBkMat = cvCreateMat(pFrame->height, pFrame->width,CV_32FC1); pFrMat = cvCreateMat(pFrame->height, pFrame->width, CV_32FC1); pFrameMat = cvCreateMat(pFrame->height, pFrame->width, CV_32FC1); //转化成单通道图像再处理 cvCvtColor(pFrame, pBkImg, CV_BGR2GRAY); cvCvtColor(pFrame, pFrImg, CV_BGR2GRAY); cvConvert(pFrImg, pFrameMat); cvConvert(pFrImg, pFrMat); cvConvert(pFrImg, pBkMat); }99
  • 100. 程序示例5:运动对象检测(3)else { cvCvtColor(pFrame, pFrImg, CV_BGR2GRAY); cvConvert(pFrImg, pFrameMat); cvAbsDiff(pFrameMat, pBkMat, pFrMat); //当前帧跟背景图相减 cvThreshold(pFrMat, pFrImg, 60, 255.0, CV_THRESH_BINARY);//二值化前景图 //cvErode(pFrImg, pFrImg, 0, 1); //形态学滤波,使用任意结构元素腐蚀图像 //cvDilate(pFrImg, pFrImg, 0, 1); //形态学滤波,使用任意结构元素膨胀图像 cvRunningAvg(pFrameMat, pBkMat, 0.003, 0); //更新背景 cvConvert(pBkMat, pBkImg); //将背景转化为图像格式,用以显示 cvShowImage("video", pFrame); //显示图像 cvShowImage("background", pBkImg); //显示背景图像 cvShowImage("foreground", pFrImg); //显示前景图像 if( cvWaitKey(2) >= 0 ) break; //如果有按键事件,则跳出循环 } } cvDestroyWindow("video"); //销毁窗口 cvDestroyWindow("background"); cvDestroyWindow("foreground"); cvReleaseImage(&pFrImg); //释放图像和矩阵 cvReleaseImage(&pBkImg); cvReleaseMat(&pFrameMat); cvReleaseMat(&pFrMat); cvReleaseMat(&pBkMat); return 0; }100
  • 101. DEMO未采用形态学滤波 采用形态学滤波后,效果会怎么样?101
  • 102. 程序示例6:运动对象跟踪用OpenCV构造大型的视频分析与处理程序 运动物体跟踪 把运动的物体检测出来,对目标编号并获取其运动轨迹。 系统在检测和跟踪到物体后,可以对物体进行特征提取以识别运动物体的种类、大小、运动方向以及其他所必要的特征信息。 102前景检测模块新团块检测模块轨迹生成模块团块跟踪模块轨迹后处理模块视频帧团块 (ID:标识符, Pos:位置, Size:大小)团块位置校正跟踪模块
  • 103. 运动对象跟踪:基本结构(1)团块:CvBlob结构描述团块的位置、大小和标识符 struct CvBlob { float x,y; /* blob position */ float w,h; /* blob sizes */ int ID; /* blbo ID */ };103
  • 104. 运动对象跟踪:基本结构(2)团块列表:CvBlobSeq类用来表示一系列团块,可以是运动轨迹,或当前帧中的所有团块 class CvBlobSeq { public: CvBlobSeq(int BlobSize = sizeof(CvBlob)); virtual ~CvBlobSeq(); virtual CvBlob* GetBlob(int BlobIndex); virtual void DelBlob(int BlobIndex); virtual void AddBlob(CvBlob* pB); virtual int GetBlobNum(); …… protected: CvMemStorage* m_pMem; CvSeq* m_pSeq; char m_pElemFormat[1024]; };104
  • 105. 运动对象跟踪:基本结构(3)团块跟踪参数:用来描述跟踪流程类CvBlobTrackerAuto的参数 struct CvBlobTrackerAutoParam1 { int FGTrainFrames; /* number of frames */ CvFGDetector* pFG; /* FGDetector module*/ CvBlobDetector* pBD; /* blob detector module*/ CvBlobTracker* pBT; /* blob tracking module*/ CvBlobTrackGen* pBTGen; /* blob trajectory generator*/ CvBlobTrackPostProc* pBTPP; /* blob trajectory postprocessing module*/ int UsePPData; CvBlobTrackAnalysis* pBTA; /* blob trajectory analysis module */ }; 105
  • 106. 运动对象跟踪:模块(1)前景检测模块 /* list of FG DETECTION modules */ typedef struct DefModule_FGDetector{ CvFGDetector* (*create)(); char* nickname; char* description; } DefModule_FGDetector; DefModule_FGDetector FGDetector_Modules[] ={ {cvCreateFGDetector0,"FG_0","Foreground Object Detection from Videos Containing Complex Background. ACM MM2003."}, {cvCreateFGDetector0Simple,"FG_0S","Simplyfied version of FG_0"}, {cvCreateFGDetector1,"FG_1","Adaptive background mixture models for real-time tracking. CVPR1999"}, {NULL,NULL,NULL} };106前景检测模块视频帧前景掩码
  • 107. 运动对象跟踪:模块(2)团块检测模块 /* list of BLOB DETECTION modules */ typedef struct DefModule_BlobDetector { CvBlobDetector* (*create)(); char* nickname; char* description; } DefModule_BlobDetector; DefModule_BlobDetector BlobDetector_Modules[] = { {cvCreateBlobDetectorCC,"BD_CC","Detect new blob by tracking CC of FG mask"}, {cvCreateBlobDetectorSimple,"BD_Simple","Detect new blob by uniform moving of connected components of FG mask"}, {NULL,NULL,NULL} };107新团块检测模块新团块 (pos,size)已有团块前景掩码
  • 108. 运动对象跟踪:模块(3)团块跟踪模块 /* list of BLOB TRACKING modules */ typedef struct DefModule_BlobTracker { CvBlobTracker* (*create)(); char* nickname; char* description; } DefModule_BlobTracker; DefModule_BlobTracker BlobTracker_Modules[] = { {cvCreateBlobTrackerCCMSPF,"CCMSPF","connected component tracking and MSPF resolver for collision"}, {cvCreateBlobTrackerCC,"CC","Simple connected component tracking"}, {cvCreateBlobTrackerMS,"MS","Mean shift algorithm "}, {cvCreateBlobTrackerMSFG,"MSFG","Mean shift algorithm with FG mask using"}, {cvCreateBlobTrackerMSPF,"MSPF","Particle filtering based on MS weight"}, {NULL,NULL,NULL} };108团块跟踪模块团块 (ID,pos,size)新团块位置前景掩码视频帧
  • 109. 运动对象跟踪:模块(4)轨迹生成模块 /* list of BLOB TRAJECTORY GENERATION modules */ typedef struct DefModule_BlobTrackGen { CvBlobTrackGen* (*create)(); char* nickname; char* description; } DefModule_BlobTrackGen; DefModule_BlobTrackGen BlobTrackGen_Modules[] = { {cvCreateModuleBlobTrackGenYML,"YML","Generate track record in YML format as synthetic video data"}, {cvCreateModuleBlobTrackGen1,"RawTracks","Generate raw track record (x,y,sx,sy),()... in each line"}, {NULL,NULL,NULL} };109轨迹跟踪模块轨迹列表所有团块(ID,pos,size)前景掩码视频帧
  • 110. 运动对象跟踪:模块(5)轨迹后处理模块 /* list of BLOB TRAJECTORY POST PROCESSING modules */ typedef struct DefModule_BlobTrackPostProc { CvBlobTrackPostProc* (*create)(); char* nickname; char* description; } DefModule_BlobTrackPostProc; DefModule_BlobTrackPostProc BlobTrackPostProc_Modules[] = { {cvCreateModuleBlobTrackPostProcKalman,"Kalman","Kalman filtering of blob position and size"}, {NULL,"None","No post processing filter"}, // {cvCreateModuleBlobTrackPostProcTimeAverRect,"TimeAverRect","Average by time using rectangle window"}, // {cvCreateModuleBlobTrackPostProcTimeAverExp,"TimeAverExp","Average by time using exponential window"}, {NULL,NULL,NULL} }; 110
  • 111. 运动对象跟踪:模块(6)轨迹分析模块 /* list of BLOB TRAJECTORY ANALYSIS modules */ CvBlobTrackAnalysis* cvCreateModuleBlobTrackAnalysisDetector(); typedef struct DefModule_BlobTrackAnalysis{ …… } DefModule_BlobTrackAnalysis; DefModule_BlobTrackAnalysis BlobTrackAnalysis_Modules[] ={ {cvCreateModuleBlobTrackAnalysisHistPVS,"HistPVS","Histogram of 5D feature vector analysis (x,y,vx,vy,state)"}, {NULL,"None","No trajectory analiser"}, {cvCreateModuleBlobTrackAnalysisHistP,"HistP","Histogram of 2D feature vector analysis (x,y)"}, {cvCreateModuleBlobTrackAnalysisHistPV,"HistPV","Histogram of 4D feature vector analysis (x,y,vx,vy)"}, {cvCreateModuleBlobTrackAnalysisHistSS,"HistSS","Histogram of 4D feature vector analysis (startpos,endpos)"}, {cvCreateModuleBlobTrackAnalysisTrackDist,"TrackDist","Compare tracks directly"}, {cvCreateModuleBlobTrackAnalysisIOR,"IOR","Integrator (by OR operation) of several analysers "}, {NULL,NULL,NULL} };111
  • 112. DEMO程序源码参见”OpenCV样例”blobtrack 可以在blobtrack基础上添加自己的算法以实现前景检测、团块检测与跟踪、轨迹生成与后处理等功能 留待课后思考112
  • 113. 基于OpenCV的研究开发:DEMO视频文字/人脸检测、分割与识别 视频指纹提取与比对 113
  • 114. 114视频文字/人脸检测与识别DEMO文字和人脸是图像/视频检索中最重要的两种语义线索 文字:文字检测、分割与识别(OCR) 人脸:人脸检测、标注与识别
  • 115. 视纹提取与应用系统2018/10/24115
  • 116. Thanks!