Computer vision C++ libraries review

I am trying to create an easy to use, minimalistic C++ cross-platform computer vision system, with a non-restrictive license. My biggest challenge was to chose the best libraries and to get them to work together; this took some investigation and experimenting. This posting is a brief description of my findings.

This is what ShapeLogic C++ currently looks like:


Windows


Linux

In order to construct ShapeLogic C++, I had to make choices within the following categories:

• Computer vision and image processing libraries
• GUI libraries
  
• Unit test systems
• Build systems
• Compilers under Windows
  
• C++ IDEs under UNIX
  

ImageJ, the Java open source image processing tool, is the inspiration for the first part of my work: it is very simple to learn, use and program in. This is a follow up to my last posting: Computer Vision C++ vs Java. The result of my work is released as an open source project ShapeLogic C++, under the MIT license.

Computer vision and image processing libraries

The candidates I considered were:

• GIL, Generic Image Library
  
• OpenCV
• VXL

GIL, Generic Image Library

GIL, Generic Image Library by Adobe.

Pros

• Very non intrusive, only based on header files
  
• Puts a wrapper around most image format
• You can write a algorithm once and it will work for most image types
• Part of Boost since 1.35
  

Cons

• Does not come with a lot of image processing algorithms
  

OpenCV

OpenCV, Open Computer Vision by Intel.

Pros

• Very simple
• Works with both C and C++
• Very broad range of algorithms
• Complex algorithms: face detection, convexity defects
• Very popular
  

Cons

• You have to use OpenCV's IplImage
• IplImage byte order is BGR instead of the normal RGB
  

VXL, Vision X Library

VXL a combination of 2 big older vision libraries TargetJR and IUE

Pros

• Well tested technology
  
• Simpler build process using cmake
  
• Uses modern programming techniques: classes, template and STL
  
• It has a lot of functionality
  
• Simple to get started with

Cons

• It is not using normal STL, but in order to work on different compiler it had to make its own version with different names.
• Class structure is somewhat complex.
  

Choice computer vision library for ShapeLogic C++
OpenCV for existing image processing and vision algorithms. GIL for writing new algorithms.

Cross platform GUI

The candidates I considered were:

• GIMP plugin
• GTK+, GIMP toolkit
• FLTK
• HighGui from OpenCV
• PhotoShop plugin
• wxWidget

Run ShapeLogic as a GIMP plugin

Pros

• GIMP is the main cross platform OSS image editing programs.
  
• It is in wide use.
  
• Has a lot of powerful features including scripting functionality in Scheme and Python.
  
• From a user perspective this would be an excellent choice.

Cons

• GIMP is GPL, but you could have a wrapper around plugins in order to access them as GIMP plugins.
• The plugin works with tiles, which gives good performance, but does not fit well with either the way OpenCV or GIL are processing images.

GTK+, GIMP Toolkit

Pros

• GTK+ is a great looking and very powerful framework that works on: Windows, Linux, Mac, a.o.

Cons

• It is written in C and has a homegrown object system, which is not type safe.

GTKMM C++ wrapper around GTK+

Pros

• CTKMM is a great looking and very powerful framework, that works on: Windows, Linux, Mac, a.o.
  
• It feels natural to program in for a C++ programmer.

Cons

• The class hierarchy is somewhat deep since it is built on top of GTK.

FLTK, Fast light toolkit

Pros

• FLTK is very lightweight.
  
• Very clean C++, you actually have a main().
  
• Native C++ build on top of X11 or Windows.
• Fluid, a simple GUI builder
  

Cons

• Not as many widgets.
  
• Dated look.

HighGui from OpenCV

Pros

• Very lightweight.
  
• There is some functionality for displaying images, video and an event handler for mouse events.

Cons

• Does not come with a menu system.

Run ShapeLogic as a PhotoShop plugin

Pros

• PhotoShop is the main image editing programs.
  
• It is in wide use, has a lot of powerful features including macros.
  
• From a user perspective this would be an excellent choice.

Cons

• The PhotoShop SDK is not freely available, you have to apply to get it.
• The plugin does not fit well with either the way OpenCV or GIL are processing images.

wxWidgets

Pros

• wxWidgets is a full featured GUI toolkit, built on top of native toolkits: Win32, Mac OS X, GTK+, X11, Motif, WinCE and more.
• Looks good and modern.
  
• Big community.
• Several GUI builders.
  

Cons

• The programming style is close to Windows MFC programming.
  
• There are many layers.

Choice
This was a hard choice and I went back and forth between FLTK and wxWidgets, but went with FLTK. All the GUI code is separate from the image processing code, so if I wanted to change from FLTK to another toolkit later it should not be too dramatic.

C++ unit test frameworks

There are a lot of different choices and no clear leader. Some of the candidates were:

• Boost.test
• CppUnit
  
• Google C++ Testing Framework
  

Boost.test

Pros

• Boost.test is part of the Boost library.
  
• Powerful with a lot of options.
  

Cons

• You have to manually set up test suites.
  
• It is somewhat heavy.
  
• The documentation is extensive but not easy to read.
  

CppUnit

Pros

• CppUnit is following a standard unit testing convention XUnit.
• Integration with Eclipse CDT.
  

Cons

• You have to manually set up test suites.
  
• It is an extra library to install.

Google C++ Testing Frameworks

Pros

• Google test is following a standard unit testing convention XUnit.
  
• Strong focus on simplicity.
• Documentation is short and easy to read.
  

Cons

• It is an extra library to install.

Choice of C++ unit test framework for ShapeLogic C++

I spent quite a bit of time reading the Boost Test documentation, finally I tried Google C++ Testing Framework and got it working very fast.

Build system

The candidates I considered were:

• Boost build
  
• Make

Boost build

Pros

• Boost build is part of Boost.
  
• Clean design, made as a Make replacement.
• Works on most platforms and with most compilers.
• The scripts are pretty short.
  

Cons

• There is a learning curve.
  

Make

Pros

• Make is the standard for build on C++.
• Widely used.
• Works with Eclipse, MSVC, NetBeans.
• Short scripts.
  

Cons

• It has gotten messy over time.
  
• Shell script dependency.
  
• There is too much magic for my taste.
  

Choice of build system for ShapeLogic C++

I chose to go with Boost Build because it has a cleaner design, but Make looks very competitive when looking over the pros and cons.

Compilers under Windows

In order to compile Boost you need a pretty modern and standard compliant compiler. The candidates that I looked at are:

• Cygwin GCC
  
• MinGW GCC
• MSVC Microsoft Visual C++
  

Cygwin GCC

Pros

• Cygwin GCC is close to GCC under UNIX
  

Cons

• Uses emulation of UNIX system call.
  
• You can only use it to build GPL compatible application.
  

MinGW GCC

Pros

• MingGW integrates well with Eclipse CDT.
• Works more natively with Windows.
• Most libraries build fine with MinGW.
  

Cons

• It was supposed to be able to build FLTK, but I tried a few times and could not get it to work.
• In order to run Make files you also have to install MSYS, which is a minimal shell.
  

MSVC, Microsoft Visual C++

Pros

• MSVC is a high quality compiler.
  
• Most used compiler under Windows.
  
• There is a free Express version.
  

Cons

• There seems to be some restrictions of the Express version that I did not quite understand.
  

Choice of compiler under Windows for ShapeLogic C++

MSVC.

C++ IDEs under UNIX

The candidates I considered were:

• Eclipse 3.4
  
• Emacs / Xemacs
• NetBeans 6.1
  

Eclipse 3.4

Pros

• Eclipse 3.4 CDT has a good debugger.
• Easy to jump from classes to files defining the classes.
  

Cons

• Not nearly as good as Eclipse for Java.
• Unstable under Linux AMD64.

Emacs

Pros

• Emacs is powerful tools that runs in a terminal.
• Takes up less resources.
• Not dependent on Java.
  

Cons

• The Java bases IDE have more features.
• Demands more knowledge to use.
  

NetBeans 6.1

Pros

• NetBeans 6.1 seem a little lighter than Eclipse.

Cons

• It is made to work with Make files, and ShapeLogic C++ is using Boost Build / Bjam.
  

Choice of IDE under UNIX for ShapeLogic C++

Eclipse.

Summary of libraries and tools used

• Boost the C++ library
• Generic Image Library for writing new image processing code
• OpenCV for existing computer vision algorithms
• FLTK, Fast Light Toolkit lightweight cross platform GUI
• Google C++ Testing Framework
• Boost.build v2 for command line based build system

Status of ShapeLogic C++

ShapeLogic C++ 0.4 is the first alpha release. It can do some useful work, but it still mainly an example application.

Currently has

• Comes with some image processing operation
• Comes with 3 brushes
• It is pretty simple to program an image processing algorithm
  

Missing

• Drawing is currently slow and there is only one pen size
• None of the ShapeLogic Java algorithms have been ported yet
• Documentation is poor

Hardest problems

• Learning how FLTK works
• Building a cross platform C++ build script covering several libraries

None of these problems will effect ShapeLogic users.

Porting computer vision code from Java to C++

Before I started porting ShapeLogic from Java, I thought that C++ was moving towards becoming a legacy language. What I have learned from this work is that C++ has advanced substantially since 2002, when I last used it professionally. C++ still seems competitive, at least in computer vision, and according to my old video game colleagues also in games, where I though that C# might have taken a lead by now. Both C++ and Java have substantial advantages.

C++

• OpenCV has a lot of vision algorithms, e.g. face recognition
• C++ is faster than Java
  
• Better for video processing
  
• Programs are shorter
• Generic programming working on primitive types
  
• You can make build script that build under both Windows and UNIX

Java / ImageJ

• ImageJ has more open source algorithms for medical image processing
  
• Better support for medical image files formats under ImageJ
• IDEs are better under Java
• Build process is simpler than C++
  
• Simpler language
• Better support for parallel processing
• A lot easier to dynamically load plugins
  

The next step is to port my framework for declarative programming -- it is based on lazy streams -- and port the Color Particle Analyzer. C++ / Boost have good support for functional programming techniques: Boost.Bind and Boost.Lambda, and the Phoenix library has just been accepted into Boost. When complete, I will do another posting about how it went.

-Sami Badawi
http://www.shapelogic.org

ShapeLogic 1.2 with color particle analyzer released

Here are the release notes for ShapeLogic 1.2

Changes

• Particle analyzer working directly on color and gray scale images without manual user intervention
• Both particle counter and particle analyzer now take parameters and print reports about each particle's color, area, standard deviation to result table
• Color replacer replaces one color within a tolerance with another color. Parameter input dialog with preview check box
• Organize plugins and macros under ShapeLogic? and ShapeLogicOld? menus, until 1.1 they where all placed under shapelogic menu
• ShapeLogic still has beta development status

The particle analyzer in ShapeLogic v 1.2 has gone through limited
testing and seems to work well. There is still a bug in the edge
tracer.

Using particle analyzer as a ImageJ plugin

The particle analyzer was tested on the particle sample images from ImageJ embryos.jpg

To run it from ImageJ select "Color Particle Analyzer" in the ShapeLogic? menu:

First a particle count dialog is displayed:

Here is the result of running the non-customized particle analyzer on it. This is written to a result table that can be exported to Excel:

The categories for the particles are only examples, it is easy to setup different rules for categorizing particles. In ShapeLogic? 1.3 there will be custom rules to recognize specific cells.

ShapeLogic? 1.2 also contains the second version of a color particle counter. It also prints a smaller report of the particle's properties.

Plans for next release

The next release, ShapeLogic v 1.3, will be a more mature particle analyzer which will come with custom rules to recognize specific cells.

Seeking particle images

In order to create these rules, I am looking for images of particles on a relatively uniform background. Please let me know if you have sample images that I could work from, preferably standard images like the embryo sample image that comes with ImageJ.

Possible future plans for particle analyzer

• Create rule for recognizing cells using neural networks or machine learning techniques
  
• Be able to handle a background that is not uniform, and cell organelles
• Incorporate reasoning under uncertainty using the lazy stream library
  
• Find overlapping particles and distinguish them as separate

Download ShapeLogic 1.2

-Sami Badawi
http://www.shapelogic.org

ShapeLogic 1.1 with particle counter released

Here are the release notes for ShapeLogic 1.1

Changes

• Particle counter working directly on color and gray scale images without manual user intervention
• Particle counter finds average color, standard deviation, area and location for each particle
• Framework to build more advanced particle counters and particle analyzers
• Color clustering using K-mean algorithm
• Background color finder
• Extend all the image processing algorithms in ShapeLogic to work in both ImageJ and in plain Java
• Better support for NetBeans
• ShapeLogic still has beta development status
• 29000 lines of Java code
• 440 unit test that all works on local machine

The particle counter in ShapeLogic v 1.1 has gone through limited testing, and seems to work well though tweaking the parameters is still a bit clumsy. Users looking for a mature particle counter should probably wait for ShapeLogic v 1.2.

Test on sample images from ImageJ

The particle counter was tested on the particle images from ImageJ:

embryos.jpg. The un-tweaked particle counter in ShapeLogic 1.1 found

particle count = 9

embryos.jpg contains 6 particles and a few shadows.

After changing the parameter setting it found

particle count = 5

which is the correct value since ShapeLogic 1.1 cannot split overlapping particles.

Direction

This is the first use of ShapeLogic in medical image processing. The next few releases should also be focused on the particle counter, making it more robust and automatic. This is the plan for ShapeLogic 1.2:

• More testing and tweaking of particle counter
• Make it easier to set parameters for particle counter in a macro or a configuration file
• Print report about each particle's color, area, standard deviation to a file
• Different implementations of particle counters
• Vectorize particles by tracing the edge
• Filter particles based geometric properties of the edge using the same techniques as letter matcher
  

Download ShapeLogic 1.1

-Sami Badawi
http://www.shapelogic.org

ShapeLogic 0.9 with lazy stream library released

Here are the release notes, I will soon describe the changes in more details.

This is the first release where ShapeLogic is moving beyond current parameters as a plugin library for ImageJ, currently only used in a letter recognition example. The improved system will be for declarative programming where the user can define rules in either a database or flat file. The focus will still be on image processing and computer vision, but the system will be more broadly applicable. There has been no new work on image processing or letter recognition in this release. ShapeLogic 1.0 will combine these new changes with the current image processing code.

Changes

• Introduce new functional, declarative and query constructs to Java
• Implement lazy streams like Haskell, Scala or Scheme
• These functional constructs are very lightweight and you only need one 200KB jar file to use it in other applications
• Test streams by solving the first 10 mathematical problems from Project Euler
• Enabled Java 6 Scripting for evaluating expressions.
  
• Tested with Groovy, JRuby, JavaScript, but should work with other supported Scripting languages, currently that are 25 of these. This makes it possible for users to add rule, formulas and queries in real time using text format. They can interact with a running Java application, which can be useful in science, finance or web applications.

-Sami Badawi