NLTK under Python 2.7 and SciPy 0.9.0

Python 2.7 has been out for months, but I have been stuck using Python 2.6 since SciPy was not working for Python 2.7. SciPy 0.9 Beta 1 binary distribution has just been released.
Normally I try to stay clear of beta quality software, but I really like some of the new features in Python 2.7 especially the argparse module, so despite my better judgement I installed Python 2.7.1 and SciPy 0.9.0 Beta 1, to run with a big NLTK based library. This is blog post describes the configuration that I use; and my first impression of the stability.

SciPy 0.9 RC1 was released January 2011.
SciPy 0.9 was released February 2011.
I tried both of them and found almost the same result as for SciPy 0.9 Beta 1, which this review was originally written for.

Direct downloads
Here is a list of the programs I installed directly:

• python-2.7.1.msi
• pywin32-214.win32-py2.7.exe
• numpy-1.5.1-win32-superpack-python2.7.exe
• scipy-0.9.0b1-win32-superpack-python2.7.exe
• setuptools-0.6c11.win32-py2.7.exe

Installation of NLTK

The install was very simple just type:

\Python27\lib\site-packages\easy_install.py nltk

Other libraries installed with easy_install.py

• CherryPy
• ipython
• PIL
• pymongo
• pyodbc

YAML Library
On a Windows Vista computer with no MS C++ compiler were I tested this NLTK install I also had to do a manual install of YAML from:
http://pyyaml.org/wiki/PyYAML

Libraries from unofficial binary distributions
There are a few packages that have build problems, but can be loaded from Christoph Gohlke's site with Unofficial Windows Binaries for Python Extension Packages: http://www.lfd.uci.edu/~gohlke/pythonlibs/ I downloaded and installed:

• matplotlib-1.0.0.win32-py2.7.exe
• opencv-python-2.2.0.win32-py2.7.exe

Stability

The installation was simple. Everything installed cleanly. I ran some bigger scripts and they ran fine. Development and debugging also worked fine. Out of 134 NLTK related unit tests only one failed under Python 2.7

Problems with SciPy algorithms

The failing unit test was maximum entropy training using the LBFGSB optimization algorithm. These were my settings:
nltk.MaxentClassifier.train(train, algorithm='LBFGSB', gaussian_prior_sigma=1, trace=2)

First the maximum entropy training would not run because it was calling the method rmatvec() in scipy/sparse/base.py. This method has been deprecated for a while and has been taken out of the SciPy 0.9. I found this method in SciPy 0.8 and added it back. My unit test ran, but instead of finishing in a couple of seconds it took around 10 minutes eating up 1.5GB before it crashed. After this I gave up on LBFGSB.

If you do not want to use LBFGSB, megam is another efficient optimization algorithm. However it is implemented in OCaml and I did not want to install OCaml on a Windows computer.

This problem occurred for both SciPy 0.9 Beta 1 and RC1.

Python 2.6 and 2.7 interpreters active in PyDev

Another problem was that having both Python 2.6 and 2.7 interpreters active in PyDev made it less stable. When I started scripts from PyDev sometime they timed out before starting. PyLint would also show errors in code that was correct. I deleted Python 2.6 interpreter under PyDev Preferences, and PyDev worked fine with just Python 2.7.

I also added a version check the one failing unit test, since it caused problems for my machine.
if (2, 7) < sys.version_info: return

Multiple versions of Python on Windows

If you install Python 2.7 and realize that some code is only running under Python 2.6 or that you have to rollback. Here are a few simple suggestions:

I did a Google search for:
python multiple versions windows
This will show many ways to deal with this problem. One way is calling a little Python script that change the Windows register settings.

Multiple versions of Python have not been a big problem for me. So I favor a very simple approach. The main issue is file extension binding. What program gets called when you double click a py file or type script.py on the command line.

Changing file extension binding for rollback to Python 2.6

Under Windows XP You can change file extensions in Windows Explorer:
Under Tools > Folder Option > File Types
Select the PY Extension and press Advanced then press Change
Select open press Edit
The value is:
"C:\Python27\python.exe" "%1" %*
You can change this to use a different interpreter:
"C:\Python26\python.exe" "%1" %*

Or even simpler when I want to run the older Python interpreter I just type:
\Python26\python.exe script.py
Instead of typing
script.py

Is Python 2.7 and SciPy 0.9.0 Beta 1 stable enough for NLTK use?

The installation of all the needed software was fast and unproblematic. I would certainly not use it in a production environment. If you are doing a lot of numerical algorithms you should probably hold off. If you are impatient and you do not need to do new training it is worth trying it, you can always roll back.

Natural language processing in Clojure, Go and Cython

I work in natural language processing, programming in C# 3.5 and Python. My work includes classification, named entity recognition, sentiment analysis and information extraction. Both C# and Python are great languages, but I do have some unmet needs. I investigated if there are any new languages that would help. I only looked at minimal language that would be simple to learn. The 3 top contenders were: Clojure, Go and Cython. Both Clojure, Go have innovative approaches to non locking concurrency. This is my first impression of working with these languages.

For contrast let me start by listing the features of my current languages.

C# 3.5

C# is an advanced object orientated / functional hybrid language and programming platform:

• It is fast
• Great development environment
• You can do almost any tasks in it
• Great database support with LINQ to SQL
• Advanced web development with ASP.net
• Advanced GUI toolkit with WPF
• Good concurrency with threading library
• Good MongoDB library

Issues

• Works best on Windows
• Not well suited for rapid development

While many features of C# are not directly related to NLP they are very convenient. C# has some NLP libraries: SharpNLP is a port of OpenNLP from Java. Lucene has also been ported. The ports are behind the Java implementation, but still give a good foundation.

Python

Python is an elegant scripting language, with a strong focus on simplicity.

• NLTK is a great NLP library
• Lot of open source math and science libraries
• PyDev is a good development environment
• Good MongoDB library
• Great for rapid development

Issues

• It is interpreted and not very fast
• Problems with GIL based threading model

C# vs. Python and unmet needs

I was not sure what language I would prefer to work with. I suspected that C# would win out with all it advanced features. Due to demand for fast turnaround, I ended up doing more work in Python, and have been very happy with that choice. I have a lot of scripts that can be piped together to create new applications, with the help of the very fast and flexible MongoDB.

I do have some concerns about Python moving forward:

• Will it scale if I get really large amount of text
• Will speed improve on multi core processors
• Will it work with cloud computing
• Part of speech tagging is slow

Java

Java is a modern object oriented language. Like C# it is a programming platform:

• Has most NLP libraries: OpenNLP, Mahout, Lucene, WEKA
• It is fast
• Great development environment: Eclipse and NetBeans
• You can do almost any tasks in it
• Great database support with JDBC and Hibernate
• Many web development frameworks
• Good GUI toolkit: Swing and JavaFX
• Good concurrency with threading library

Issues

• Functional style programming is clumsy
• Working with MongoDB is clumsy
• Java code is verbose

I would not hesitate using Java for NLP, but my company is not a Java shop.

Clojure

Clojure was released in 2007. It is a right sized LISP. Not very big like Common LISP or very small like Scheme.

• Gives easy access to Java libraries: OpenNLP, Mahout, Lucene, WEKA, OpinionFinder
• Innovative non locking concurrency primitives
• Good IDEs in Eclipse and NetBeans
• Easy to work with
• Code and data is unified
• Interactive REPL
• LISP is the classic artificial intelligence language
• If you need speed you can write Java code
• Good MongoDB library

 Issues

• The IDE is not working as well as IDEs for Java or C#

Clojure is minimal in the sense that it is build on around 10 primitive programming constructs. The rest of the language is constructed with macros written in Clojure.

Once I got Clojure installed it was easy to work with and program in. Most of the good features about Python also applies to Clojure: it is minimal and has batteries included. Still I think that Python is a simpler language than Clojure.

Use case
Clojure is a good way to script the extensive Java libraries, for rapid development. It has more natural interaction with MongoDB than Java.

Clojure OpenNLP

The clojure-opennlp project is a thin Clojure wrapper around OpenNLP. It came with all the corpora used as training data for OpenNLP nicely packaged and it works well. You can script OpenNLP approximately as terse as NLTK, from an interactively repl.

I tried it in both Eclipse and NetBeans. They seem somewhat equal in number of features. I had a little better luck with the Eclipse version.

clojure-opennlp is using a Maven built system, but has a nontraditional directory layout, this caused problems for both Eclipse and NetBeans, they both took some configuration.

Eclipse Counterclockwise
The Counterclockwise instruction for labrepl mainly worked for installing clojure-opennlp.
When you were done you had to go in add the example directory the source directories under properties.

NetBeans Enclojure
I imported the project. I had to move the Clojure file from example directory to a different position to get it to work.

Maven plugins for Clojure
The standard Maven directory layout has several advantages, e.g. if you want to mix Java and Clojure code. I created my own Maven pom configuration file up, based on examples of other Clojure Maven projects. They used Clojure plugins for Maven, I could not get this to work. Eventually I ripped these plugins out and was left with very pain POM file that worked.

Go / Golang

Go was announced November 2009. It is created by Google to deal with multicore and networked machines. It feels like a mixture of Python and C. It is a very clean and minimal language.

• It is fast
• Good standard library
• Excellent support for concurrency
• It is trivial to write your own load balancer

Issues

• The Eclipse IDE is in an early stage
• Debugger is not working
• Windows port is not done and has just been released

It was hard to find the right Go Windows port, there are several Go windows port projects with no code.

Use cases
I currently have a problem when downloading a lot HTML pages and parsing them to a tree structure. This does not have the best support in C#. I found a library that translates HTML to XHTML and then I can use LINQ to process it. The library is not documented, not very fast and fails for some HTML files.

Go comes with a HTML library that parses HTML 5, it is simple to write a program with some threads that download and other that parse the files into a DOM tree structure.
I would use Golang for loading large amounts of text in a cloud computing environment.

Cython

Cython was released in July 2007. It is a static compiler to write Python extension modules in a mixture of Python and C.

Process for using Cython

• Start by writing normal Python code
• Find modules that are too slow
• Add static types
• Compile it with Cython using the setup tool
• This produces compiled modules that can be used with normal Python

Issues

• It is still more complex that normal Python code
• You need to know C to use it

I was surprised how simple it was to get it working both under Windows and Linux. I did not have to mess with make files or configure the compiles. Cython integrated well with NumPy and SciPy. This expands the programming tasks you can do with Python substantially.

Use cases
Speed up slow POS tagging.

My previous language experience

Over the years I have experimented with a long list of non mainstream languages: functional, declarative, parallel, array, dynamic and hybrid languages. Many of these were frustrating experiences. I would read about a new language and get very excited. However this would often be the chain of events:

• Download language
• Installed Cygwin
• Find out how the language's build system works
• Try to find a version of the GCC compiler that will compile it
• Get the right version of Emacs installed
• Try to get the debugger working under Emacs
• Start programming from scratch since the libraries were sparse
• Burn out

You only have so much mental capacity, and if you do not use a language you forget it. Only Scala made it into my toolbox.

Do Clojure, Go or Cython belong in your programmer's toolbox

Clojure, Go and Cython are all simple languages. They are easy to install, easy learn, they all have big standard libraries so you can be productive in them right away. This is my first impression:

• Clojure is a good way to script the extensive Java libraries, for rapid application development and for AI work.
• Go is a great language but it is still rough around the edges. There are not any big NLP libraries written for Go yet. I would not try to use it for my main NLP tasks.
• Cython was useful right away for my NLP work. It makes it possible to do fast numerical programming in Python without too much trouble.

-Sami Badawi

Orange, R, RapidMiner, Statistica and WEKA

Review of open source and cheap software packages for Data Mining

This blog posting is comparing the following tools, after working with them for 2 months and using them for solving a real data mining problem:

• Orange
  
• R
• RapidMiner
• Statistica 8 with Data Miner module
• WEKA

Statistica is commercial, all the other are open source. There is also a brief mention of the following Python libraries: mlpy, ffnet, NLTK.

Summary of first impression

This is a follow up on my previous post R, RapidMiner, Statistica, SSAS or WEKA describing my impression of the following software packages after using them for a couple of days each:

• R
• RapidMiner
• SciPy
  
• SQL Server Analysis Services, Business Intelligence Development Studio
• SQL Server Analysis Services, Table Analysis Tool for Excel
• Statistica 8 with Data Miner module
  
• WEKA

Let me summarize what I found:

SciPy did not have what I needed. However I found a few other good Python-based solutions: Orange, mlpy, ffnet and NLTK.

The SSAS-based solutions held promise due to their close integration with Microsoft products, but I found them to be too closely tied to data warehouses so I postponed exploring them.

Statistica and RapidMiner had a lot of functionality and were polished, but the many features were overwhelming.

R was harder to get started with and WEKA was less polished, so I did not spend too much time on them.

Comparison matrix

In order to compress my current findings I am summarizing it in this matrix. This metric is only based on limited work with the different software packages and is not very accurate. The categories are:
Documentation; GUI and graphics; how polished the package is; ease of learning; controlling package from a script or program; how many machine learning algorithms that are available:

	Doc	GUI	Polished	Ease	Scripting	Algorithms
Orange	2	3	2	3	3	2
Python libs	1	1	1	3	3	2
R	3	2	2	1	3	2
RapidMiner	2	3	2	2	2	3
Statistica	3	3	3	2	2	3
WEKA	2	2	2	3	2	3

Criteria for software package comparison

The comparison is based on a real data mining task that is relatively simple:

• Supervised learning for categorization.
• Over 200 attributes mainly numeric but 2 categorical / text.
• One of the categorical attributes is the most important predictor.
  
• Data is clean, so no need to clean outliers and missing data.
  
• Accuracy is a good metric.
• GUI with good graphic to explore the data is a plus.

General observations

The most popular data mining packages in the industry are SAS and SPSS, but they are quite expensive. Orange, R, RapidMiner, Statistica and WEKA all can be used for doing real data mining work. While some of them are unpolished.

There was a similar learning curve for most of the programs. Most programs took me a few days to get working, between the documentation and experimenting.

I had to reformulate my original problem. Neural network models did not work well on my categorical / text attributes. Statistica produced an accuracy of 90%, while RapidMiner produced an accuracy of 82%.
I replaced the 2 categorical attributes with a numeric attribute and accuracy of the best model increased to around 97%, and was much more uniform between the different tools.

Orange

Orange is an open source data mining package build on Python, NumPy, wrapped C, C++ and Qt.

• Works both as a script and with an ETL work flow GUI.
  
• Shortest script for doing training, cross validation, algorithms comparison and prediction.
• I found Orange the easiest tool to learn.
• Cross platform GUI.

Issues:

• Not super polished.
  
• The install is big since you need to install QT.

Python libs: ffnet, NumPy, mlpy, NLTK

A few Python libs deserve to be mentioned here: ffnet, NumPy, mlpy and NLTK.

• If you do not care about the graphic exploration, you can set up an ffnet neural network in few lines of code.
• There are several machine learning algorithms in mlpy.
  
• The machine learning is NLTK is very elegant if you have a text mining or NLP problem.
  
• The libraries are self contained.

Issues:

• Limited list of machine learning algorithms.
  
• Machine learning is not handled uniformly between the different libraries.

R

R is an open source statistical and data mining package and programming language.

• Very extensive statistical library.
  
• It is a powerful elegant array language in the tradition of APL, Mathematica and MATLAB, but also LISP/Scheme.
• I was able to make a working machine learning program in just 40 lines of code.

Issues:

• Less specialized towards data mining.
• There is a steep learning curve, unless you are familiar with array languages.

R vs. Orange written in Python

Python and R have a lot in common: they are both elegant, minimal, interpreted languages with good numeric libraries. Still they have a different feel. So I was interested in seeing how they compared.

Orange / Python advantages

• R is quite different from common programming languages.
  
• Python is easier for most programmers to learn.
• Python has better debugger.
• Scripting data mining categorization problems is simpler in Orange.
• Orange also has an ELT work flow GUI.
  

R advantages

• R is even more minimal than Python.
• Numerical programming is better integrated in R, in Python where you have to use external packages NumPy and SciPy.
• R has better graphics.
• R is more transparent since the Orange are wrapped C++ classes.
• Easier to combine with other statistical calculations.
  

I made small script to solve my data mining problem in both Orange and R. This was my impression:

If all you want to do is to solve a categorization problem I found Orange to be simpler. You have to become very familiar with how Orange read the spreadsheet, the different attribute types, notably the Meta attribute.

Import and export of data from spreadsheet is easier in R, spreadsheet are stored in a data frames that the different machine learning algorithms are operating on. Programming in R really is very different, you are working on a higher abstraction level, but you do lose control over the details.

RapidMiner

RapidMiner is an open source statistical and data mining package written in Java.

• Solid and complete package.
• It easily reads and writes Excel files and different databases.
• You program by piping components together in a graphic ETL work flows.
  
• If you set up an illegal work flows RapidMiner suggest Quick Fixes to make it legal.

Issues:

• I only got it to works under Windows, but others have gotten it to work in other environments, see comment below.
• There are a lot of different ETL modules; it took a while to understand how to use them.
  
• First I had a hard time making a comparison between different models. Eventually I found a way: You chose a cross validation and select different models one by one. When you run the model the will all be stored on the result page and you can do comparison there.

Statistica 8

Statistica is a commercial statistics and data mining software package for Windows.
There is a 90 day trial for Statistica 8 with data miner module in the textbook:
Handbook of Statistical Analysis and Data Mining Applications. There is also a free 30 day trial.

• Generally very polished and good at everything, but it is also the only non open source program.
• High accuracy even when I gave it bad input.
• You can script everything in Statistica in VB.
• Cheap compared to SPSS and SAS.

Issues:

• So many options that it was hard to navigate the program.
  
• The most important video about Data Miner Recipes is the very last out of 36.
• Cost of Statistica is not available on their website.
• It is cheap in a corporate setting, but not for private use.

WEKA

WEKA is an open source statistical and data mining library written in Java.

• A lot of machine learning algorithms.
• Easy to learn and use.
  
• Good GUI.
• Platform independent.

Issues:

• Worse connectivity to Excel spreadsheet and non Java based databases.
• CSV reader not as robust as in RapidMiner.
  
• Not as polished.

RapidMiner vs. WEKA

The most similar data mining packages are RapidMiner and WEKA. There have many similarities:

• Written in in Java.
• Free / open source software with GPL license.
• RapidMiner includes many learning algorithms from WEKA.

My first thought what that RapidMiner has everything that WEKA has, plus a lot of other functionality and is more polished. Therefore I did not spend too much time on WEKA. For the sake of completeness I took a second look at WEKA and I have to say that it was a lot easier to get WEKA to work. Sometimes less is more. Depending on what is more important functionality or ease of use.

Conclusion

There are several good and very different solutions. Let me finish by listing the strongest aspect of each tool:

Orange has elegant and concise scripting and can also be run in an ETL GUI mode.
R has elegant and concise scripting integrated with a vast statistical library.
RapidMiner has a lot of functionality, is polished and has good connectivity.
Statistica is the most polished product, and generally performed well in all categories. It gave good result when I gave it bad input.
WEKA is the easiest GUI to learn and use.

-Sami Badawi

SharpNLP vs NLTK called from C# review

C# and VB.net have fewer open source NLP libraries than languages like C++, Java, LISP and Perl. My last blog post: Open Source NLP in C# 3.5 using NLTK is about calling NLTK, which is written in Python, from IronPython embedded under C# or VB.net.

An alternative is to use SharpNLP, which is the leading open source NLP project written in C# 2.0. SharpNLP is not as big as other Open Source NLP projects. This blog posting is a short comparison of SharpNLP and NLTK embedded in C#.

Documentation

NLTK has excellent documentation, including an introductory online book on NLP and Python programming.

For SharpNLP the source code is the documentation. There is also a short introductory article by SharpNLP's author Richard J. Northedge.

Ease of learning

NLTK is very easy to work with under Python, but integrating it as embedded IronPython under C# took me a few days. It is still a lot simpler to get Python and C# to work together than Python and C++.

SharpNLP's lack of documentation makes it harder to use; but it is very simple to install.

Ease of use

NLTK it is great to work with in the Python interpreter.

SharpNLP simplifies life by not having to deal with the embedding of IronPython under C# and the mismatching between the 2 languages.

Machine learning and statistical models

NLTK comes with a variety of machine learning and statistical models: decision trees, naive Bayesian, and maximum entropy. They are very easy to train and validate, but do not preform well for large data sets.

SharpNLP is focused on maximum entropy modeling.

Tokenizer quality

NLTK has a very simple RegEx based tokenizer that works well in most cases.

SharpNLP has a more advanced maximum entropy based tokenizer that can split "don't" into "do | n't". On the other hand it sometimes makes errors and splits a normal word into 2 words.

Development community

NLTK has an active development community, with an active mailing list.

SharpNLP was last release was in December 2006. It is a port of the Java based OpenNLP, and can read models from OpenNLP. SharpNLP has a low volume mailing list.

Code quality

NLTK lets you write programs that read from web pages, clean HTML out of text and do machine learning in a few lines of code.

SharpNLP is written in C# 2.0 using generics. It is a port from OpenNLP and maintains a Java flavor, but it is still very readable and pleasant to work with.

License

NLTK's license is Apache License, Version 2.0, which should fit most people's need.

SharpNLP's license is LGPL 2.1. This is a versatile license, but maybe a little harder to work with when the project is not active.

Applications

NLTK comes with a theorem prover for reasoning about semantic content of text.

SharpNLP comes with an name, organization, time, date and percentage finder.
It is very simple to add an advanced GUI, using WPF or WinForms.

Conclusion

Both packages comes with a lot of functionality. They both have weaknesses, but they are definitely usable. I have both SharpNLP and embedded NLTK in my NLP toolbox.

-Sami Badawi