1394 Camera, DirectShow, and OpenGL

(or 1394 Camera, DirectShow, GLUT, OpenGL, MFC, and Intel’s Image Processing Library)

-- A sample that makes them work all together By Ruigang Yang

 

During the last few years, I have had a number of occasions that I need to capture a frame, do some processing, and finally render it using OpenGL. The tricky part, from a coding standpoint, is the frame capture. This summer, I started to use Microsoft’s DirectShow (abbreviated as DShow sometime) for frame capture. It is by far the most powerful and most flexible architecture I have used. But I didn’t say it is easy to use, like most MS products, the DShow framework is quite convoluted if you know nothing about the Window programming paradigm or the Component Object Model (COM). I put up a few slides for a very brief tutorial of the DirectShow architecture. But even if you read the tutorial, there is still a long way for you to master the concept and then write your own code.

Fortunately, you don’t have to start from scratch. I have put together a simple program that captures the frame (either from a live camera or from a canned sequence), put it in the texture buffer, and render it using OpenGL. The DirectShow stuff is encapsulated in a C++ class. At the beginning, you can just treat it as a black box and use the sample code as a template for your own work.

The code is available as a zip file. The project name is testCap. (It is carved from a large project. So if VC complains about project missing, just ignore it)

To compile it on your own machine, you need to follow a few steps

  1. Install the necessary support software/hardware
n DirectX 8 SDK (install the retail version)

n Intel’s Image Processing Library (IPL)ßI only used its image structure to pass data around. You can easily yank it out. I choose not to because I don’t want to create a brunch here as I am continuing to improve the code.

n A 1394 camera (or any camera) with a WDM compliant driver.

  1. Set the compiler path in your Visual Studio: In Visual Studio, select “Tools”à “Options”. The option dialog box will popup, as shown in the following figure. Put in the path of various include files and libraries. MAKE SURE that Dx8 is the first in the path list., not doing so will result in compiler or link errors, since Visual C++ contains old versions of DirectX (in \VC98\include ).


Figure 1 Options Dialog in Visual Studio
  1. Compile the DirectShow base classes (which comes with the SDK): I don’t know why it is not pre-compiled. Anyway, go to your_mssdk\samples\Multimedia\DirectShow\BaseClasses, compile the libraries (strmbase.lib for release and strmbasd.lib for debug), move them to the mssdk\lib directory. (so you don’t need to add another entry in VC’s search path list)
New for DirectX 8.1 (courtesy of Andraw Nashel)  I will note that for new installations of the Microsoft Platform SDK and DirectX 8.1, the required libraries for DirectShow are actually there, but in the your_drive:\your_mssdk_path\{lib,include}\prerelease\ directories. You must also add the following directories in order to use or compile the base classes in DirectX 8.1 (versus 8.0):
To "Include files" add: your_drive:\your_mssdk_path\include\prerelease\
To "Library files" add: your_drive:\your_mssdk_path\lib\prerelease\
  1. Compile the sample code.

                    New for DirectX 8.1 DirectX 8.1 has a different library structure than 8.0. You need to link the project with strmiids.lib, which is a part of strmbase.lib in Dx 8.0. 
                    If strmiids.lib is not included in the project, you will get some link errors like:
                                Linking...
                                ryCaptureGraph.obj : error LNK2001: unresolved external symbol _IID_ISampleGrabberCB
                                ryCaptureGraph.obj : error LNK2001: unresolved external symbol _CLSID_NullRenderer
                                ryCaptureGraph.obj : error LNK2001: unresolved external symbol _IID_ISampleGrabber
                                ryCaptureGraph.obj : error LNK2001: unresolved external symbol _CLSID_SampleGrabber
                    

If everything works as expected, you should have the executable ready by now. Run the program, you will see two windows, one is the image window, the other is the OpenGL window. Press ESC to quit the program, the space bar to play the video. Check the source code (testCap.cpp) for more details about LIVE capture.

The sample code only uses a fraction of my C++ DirectShow class. I keep it the minimum to make a simple program. You will discover more functionalities if you plow into the details of the implementation. The code inside doesn’t look pretty, I know. But hey, it really works J

Why the code does not work for camera X?

I tested my code primarily using the Sony 1394  camera (DFW_V500) with drivers from Unibrain. Ideally, it should work with all the WDM compliant camera. Here is a quick method to test if your camera is WDM compliant. Start   the graphEdit tool ( Start -->Program --> Microsoft DirectX 8 SDK-->DirectX Utilities --> Graph Edit). Select the Graph--> Insert Filter. A dialog box will pop up (shown below). Expand the "WDM Streaming Capture Devices" list, if your camera is there, it means it is WDM compliant, and my sample code should be able to use it. You can also insert the camera filter and right click on the filter box to test the camera.

 

If your camera passed the WDM test, but my code still doesn't work, often time, it is because the camera doesn't support the video format I hardcoded in my old sample code (if you are not using the Sony camera). I have changed that to provide the option to select a format from a dialog box. Please reload the code from this website.  

Special treatment for VideoPort pin: Joe Slaton reported problems with VideoPort bin in the original code. Here is his fix (from his original email dated 01/17/2002)

"I downloaded your sample code for capturing video from a DV camera. It doesn't quite work for a frame grabber card (at least not with my ATI Radeon VIVO). To get it to work with my hardware (or any hardware with a VP pin), I needed to attach the VP pin on the video capture filter to an overlay mixer and video renderer in order for the graph to capture frames. Placing the code fragment here between the addExtraFilters() and selectVideoFormat() calls in ryCaptureGraph.cpp connected up the required pins and added the needed filters. Unfortunately, it also ended up with an ActiveMovie window."

 

If all these fail, well, the only option remaining is to email me (ryagn@cs.unc.edu)

Multiple-camera capture sample ( source file)

It is an extension to the previous sample, capturing from three cameras. Texture is not copied to the rendering window. It also supports text configure files, one for system config, and the other for camera config. NOTE: you need to install the Fire-I API from Unibrain (not just the Fire-I driver) for this sample code to work properly. The unibrain API is used for automatic camera parameter control, it is possible to remove unibrain related code in videoCap.cpp. In particular, comment out the setCameraProfile function in videoCap.cpp.  

System config format

Trigger Server = foo 5555 <-- synchorinzation server host name and port number; if the server is not avaiable, cameras will be in the free running mode if the trigger mode is off (see the camera config file).
Video Mode = 1    <-- video mode, only support two format 1 and 3, 1 for 320x240 422 format, 3 for 640x480 422
...
bar 0 1 2 camconfig.txt  <-- hostname     relative_camera_index     camera_config_file

    Camera Config format

#Camera initialization
# manual values 
# setting a value will turn the function ON
# defining a value of -1 will turn the function OFF
#Shutter Speed Register Value
# 1/4000 2570
# 1/2000 2566
# 1/1000 2558
# 1/500 2542
# 1/250 2511
# 1/125 2448
# 1/60 2311
# 1/50 2259
# 1/30 2047
# 1/15 2046
# 1/8 2044

#Parameter Value Range
frame_rate 15
auto_exposure 128 // 1 - 255 //0 turns off
brightness 128 // 0 - 255
gain 1 // 0 - 18
gamma 128 // 128-130
hue 128 // 0 - 255
saturation 128 // 0 - 255
sharpness 32 // 0 - 255
shutter 2511 // 1952 - 2575 
white_balance 128 128 // 1 - 255 for U and V (0x80080 = 524416)

# auto modes
white_balance_auto 1
shutter_auto 0 
gain_auto 1

# enable external trigger mode
trigger_mode_on 1

Sample code to read the above text configuration file, code to set the camera (through the FireI interface from unibrain)

Ruigang Yang( last updated 03/05/2002)

;