Scala vs. Haskell vs. Python

Functional programming is on the upswing, but should you bet your career on it, or is it a short-lived technology fad?

I have long wanted to use functional programming professionally and for the last year I have. Mainly Scala, written in Haskell style, plus some real Haskell programming.

Here is my impression of Scala and Haskell compared to my benchmark language, Python.

Scala

Scala is a functional object oriented hybrid language running on the JVM. It was created by Martin Odersky in 2003. Scala took Java / JVM and organized it nicely according to a few orthogonal principles.
Working in Scala has been a pleasure, there is a lot to like:

• You have easy access to the giant world of Java libraries
• Lot of libraries written for Scala
• Very fast only 2 to 3 time slower than C
• Big ecosystem
• Easy to define a DSL in Scala so you can do everything in Scala
• Very advanced type system
• Adapted in the industry by: Twitter, LinkedIn, Foursquare, ...
• Scala is the most adapted functional language
• Web frameworks: Play, Scalatra, any kind of Java Servlets
• Scalding: a very nice framework for Hadoop programming
• Akka: an Erlang style actor system
• Mixin composition
• Good GUI with Swing and JavaFX
• SBT the best build tool I have used
• Scala is a full stack multi purpose language

Issues

• It is very complex
• It is a kitchen sink language
• Confusing to keep Scala collections and parallel Java collections apart

Eclipse Plugin Scala IDE for Eclipse

The Scala Eclipse plugin is very solid, but not quite as good as the fantastic Java support.

• Syntax highlighting
• Code completion
• Debugger
• Shows compile errors with explanation
• Rudimentary refactoring
• Jump to definition

Monad and Applicative Functor

Two very important concepts in functional programming are monad and applicative functor.

The best reference I found was: Learn You a Haskell for a Great Good!.

A monad gives you simple ways of composing different operations. First it seems like an odd principle. Understanding monad took me several months.

In UNIX and OS X you can create complex programs by piping simple commands together. A monad generalizes this a lot.

Once you understand the monad you will see monads pop up in so many places. The monad is an amazingly powerful construct.

The last place I found monads unexpectedly showed up was in asynchronous programming, e.g. used in AJAX.
You send an external request and you do not block but you have a callback for when the result comes back. This is efficient but messy to program especially if you have a chain of requests to process and you have to have a lot of callbacks floating around. You can do this type of calculations using a future / promise, and luckily a future is a monad so you string a long list of operations after each other in a very simple way.

Scalaz

Scalaz is a Scala library that replicates a lot of Haskell constructs, at the cost of being similarly hard to understand.

You can work with monads in Scala without using Scalaz since the "for-statement" in Scala is syntactic sugar for monadic "for-comprehension".

I have programmed Java in a functional style both professionally and for my open source project. It is possible but it is rather verbose and clunky. Scala is much more powerful, simpler and cleaner than both Java approaches, and Scalaz is a big step up from Scala.

When I started programming in Scala I read a really funny blog post called Truth about Scala that describes how a team starts to use Scala and first they are excited, but it quickly descends into a death spiral of complexity. I was concerned with this and tried to keep my code as simple as possible and avoid Scalaz for a long time. I would advise others to become very comfortable with Scala before starting to work with Scalaz.

Haskell

Haskell is a strongly typed, lazy, pure functional programming language. It is an academic research language created by a committee in 1987.
One reason that I got into Haskell was in order to understand monads and applicative functors, they are important constructs in Haskell and category theory.

There is a steep learning curve for Haskell. Maybe it is more like a hump you have to get over. Just getting to basic proficiency is hard. It took me around one year of low intensity studying, but one day it just made sense.

• Haskell now has a lot of libraries
• Libraries and dependencies are handled by Cabal
• It is fast only 2 - 3 times slower than C
• Great concurrency
• Repa native parallel numerical array processing
• Very small language
• Very pure
• Very terse code
• Very advanced type system
• Hoogle a Haskell search engine
• Great web frameworks Happstack, Snap and Yesod

Issues

• Bad GUI support
• Module system is crude

Hoogle, a Search Engine for Haskell

A colleague told me that when he needed a function he would write out its signature and put it into Hoogle and often it would take him to the function that he needed. First time I tried it, and it actually took me to a function that solved a bigger part of the problem than what I was looking for.

When I searched Hoogle for this function signature:

(a -> Bool) -> [a] -> [Int]

I got these results in EclipseFP:

Eclipse Plugin EclipseFP

EclipseFP with Hoogle

The Haskell Eclipse plugin is quite good:

• Syntax highlighting
• Cabal integration
• Hoogle integration
• Code completion
• Debugger
• GHCi integration with automatic reload

Python

Python is a high-level language built on ideas from functional, imperative and object oriented programming. It was created by Guido van Rossum in 1989.

For many years Python was my favorite language. It is a language for kids and also for scientists and a lot of people in between.

• Python is probably the easiest language to learn
• It took me a day to learn well enough to use
• Very minimal language
• Very terse code
• Excellent wrapper language
• Many implementations: CPython, Jython (JVM), IronPython (CLR), PyPy
• Good bindings to numerical packages: NumPy, SciPy
• Used in computer vision since OpenCV choosing Python to be its scripting language
• Used in natural language processing due to the NLTK
• Great web frameworks: Django, TurboGear, CherryPy

Issues

Python is not quite a full stack language there are a few missing pieces:

• Bad GUI support
• Low-level numerical programming had to be done in external packages
• Concurrency
• Speed around 50 times slower compared to C

Eclipse Plugin PyDev

I like PyDev it has:

• Syntax highlighting
• Code completion
• Debugger

Best Programming Language for Kids

If a kid can understand a technology it is well designed. My daughter is turning 5 and I am thinking about what language I should introduce her to first.

Python

My first inclination was to teach her Python since it is the simplest, but it needs to give immediate visual feedback. Python's lack of a good GUI is a problem.

Haskell

I have also been tempted to show her some Haskell to teach her good habits in a pure and minimal language. But if I tell her that:

"A monad is just a monoid in the category of endofunctors"

she will walk away or scream.

Scala

Kojo is a LOGO like graphical turtle programming environment written in Scala. Scala's type inference makes it simpler for kids who will not have a good concept of types.

My daughter plays with Kojo and she likes it. She comes and asks me if we can do the turtle?

Kojo notice green drawing turtle in the middle

So unexpectedly, Scala, the biggest language, was the most kid friendly language. Based on a very small sample size.

Category Theory

Haskell is using plenty of concepts from category theory. E.g. the monad. In my quest to understand it I started to study category theory.

Category theory has been called: "Abstract nonsense", both by its practitioners and critics. And for very good reasons. It can suck you into a black hole of abstraction.

Category Theory Introductions

You do not need to understand category theory to program in Haskell or Scalaz, but if it helps you here are a few introduction videos.

Dominic Verity presents a gentle introduction to Category Theory:

http://vimeo.com/17207564


Dominic Verity on Category Theory (Part 2)

Error792's category theory class, currently there are 5 parts

Math and Programming

I have often said that there is no connection between math and programming. The only math you need to program is counting, and occasionally, addition. I felt:

Programmers are the grease monkeys of today

We move some data around and throw it on webpages

After working in Scala and Haskell I have changed my tune:

When you program in Scala you feel like an engineer

When you program in Haskell you feel like a mathematician

Adapting Haskell and Scalaz for a Team

Using Haskell and Scalaz takes a special mindset and a lot of dedication. I have been very lucky to work at a place that has attracted physicists, mathematicians and theoretical CS people.

If a big part of your team does not have these qualities you risk wasting time and chasing developers away.

On the other hand if your team is using Haskell or Scalaz you will attract this brand of developers.

Conclusion

I had high expectations when I started using functional programming full time, but I have been disappointed by new technology many times before. Functional programming met my high expectations. It has been challenging and very enjoyable.

I was a C++ programmer for 8 years, and considered C++ the one true way for high speed, high level programming.
Recently I looked at a code sample written in C++ and it hurts my eyes: Filled with boilerplate and state.

Functional programming is addictive and will make you spoiled

Functional programming is here to stay. It has been an important part of C# since v3.0. It is finally getting added to Java in Java 8 coming out soon. The classic functional languages LISP or ML are the basis of: Clojure and F# that have thriving communities and are used in industry. The time has come to invest some time in understanding functional programming.

Python

I enjoy Scala and Haskell more than Python, but Python seem to be the language that I always go back to. It is a power tool that adds very little weight to your programmer's toolbox. You get high return on investment with Python, while with Scala and especially Haskell you have to invest a lot and for a long time before you break even.

Scala

Scala is now popular enough that you can get a job doing it. Moving from Java or C# to Scala is pretty easy. Since you can start programming Scala like Java.  Scala is a big and complex language with a big ecosystem and it takes months to get a deeper understanding. Scala is substantially more powerful than Java 7, but Java 8 has supposedly taken a lot of ideas from Scala.

Haskell

Haskell is definitely the road less traveled, but it is a road, not a trail. It is an academic research language created in 1987. Recently it has started to break into the mainstream. There are a few jobs in Haskell. Gaining basic proficiency in Haskell is quite hard, but afterwards other languages look a little clunky. Writing Haskell feels like doing math.

Scala vs. Haskell

Scala is a safer bet for most programmers, since it is better adapted to more tasks, and you can approximate Haskell pretty well with Scalaz. Scala has a very advanced type system to handle its object oriented features.

Haskell appeals to functional language purists, mathematicians and category theorists. Esthetically I prefer Haskell. It is terser and the type inference is better.

In most cases external factors would dictate whether Scala or Haskell would be a better fit for your project.

Haskell vs. Python

Haskell and Python have a lot in common:

• Minimalistic languages
• White space delimited
• Very terse
• List comprehension
• Important tuple type
• GUI binding to wxWidget, GTK

Haskell is statically typed and optimized towards purity and speed.

Python is dynamically typed and optimized towards pragmatism and simplicity.

Newer follow up article Practical Scala, Haskell and Category Theory

Hive, Pig, Scalding, Scoobi, Scrunch and Spark

Comparison of Hadoop Frameworks

I had to do simple processing of log files in a Hadoop cluster. Writing Hadoop MapReduce classes in Java is the assembly code of Big Data. There are several high level Hadoop frameworks that make Hadoop programming easier. Here is the list of Hadoop frameworks I tried:

• Pig
• Scalding
• Scoobi
• Hive
• Spark
• Scrunch
• Cascalog

The task was to read log files join with other data do some statistics on arrays of doubles. Programming this without Hadoop is simple, but caused me some grief with Hadoop.

This blog post is not a full review, but my first impression of these Hadoop frameworks.

Pig

http://pig.apache.org/

Created by Yahoo!
Language Pig Latin.

Pig is a data flow language / ETL system. It work at a much higher level than direct Hadoop in Java.
You are working with named tuples. It is mildly typed, meaning you can define a type for each field in a tuple or it will default to byte array.

• Pig is well documented
• Pig scripts are concise
• You can use it for both script and interactive sessions
• You can make Pig scripts embedded in Python run as Jython
• Pig is a popular part of the Hadoop ecosystem

Issues

• Pig Latin is a new language to learn
• Pig Latin is not a full programming language, only a data flow language
• You have to write User Defined Function / UDF in Java if you want to do something that the language does not support directly
• Pig Latin does not feel uniform

Scalding

https://github.com/twitter/scalding

Created by Twitter.
Language Scala.

Scalding looks very promising. It had just been open sourced when I looked at it.

What sets Scalding apart from other Scala based frameworks is that you work with tuples with named fields.

There is a blog with code example:
http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/
It contains numerical code for Hadoop that did not look much harder than the equivalent non Hadoop code.

Issues

You call Scalding by running a Ruby build script, scald.rb.

This did not work on my Mac, OS X Lion, but it ran under Ubuntu with no problems.

Scalding has little documentation, but it is built on Cascading that does have good documentation.

Note on aggregate functions in Scalding

Scalding has some predefined aggregate function such as Sum and Count. Unfortunately Sum only works numerical types and I needed it to work on arrays of doubles.

You can build your own aggregator function using Group Builder followed by a scanLeft or foldLeft operation.

Workaround to get Scalding to run on OS X Lion

After spending some time I found a workaround for the problems with running Scalding under OS X Lion:


In the build scripts/scald.rb set:

COMPILE_CMD="scalac"

In build.sbt set scalaVersion to the version of Scala you are using:

scalaVersion := "2.9.1"

After that I was able to run the 5 tutorials, coming with Scalding.

Scoobi

https://github.com/NICTA/scoobi
http://www.opennicta.com/home/scoobi

Created by OpenNICTA.
Language Scala.

Scoobi looked powerful and simple.


Scoobi was easy to build with the Scala build system SBT.

Issues

I was having problem running examples. Turned out that Scoobi had a dependency of the Cloudera Hadoop distributions version 0.20.2. This is a popular Hadoop distribution.

I could not get it to run on my Mac, which has the Apache Hadoop distribution so I gave up. I have not revisited it yet.

Hive

http://hive.apache.org/

Created by Facebook.
Language SQL.

Hive works on tables made of named tuples with types. It does not check the type at write time, you just copy files into the directory that represent a table. Writing to Hive is very fast, but it does check types at read time.

You can run JDBC against Hive.

It was easy to get Hive running and I really liked it.

Issues

A problem was that Hive only understands a few file format:

• Text format
• Tab delimited format
• Hadoop SequenceFile format

Starting from Hive version 0.9, is has support for Avro file format that can be used from different languages.

In order to do Sum by group I would have to create User Defined Aggregation Function. Turns out that UDF and UDAF is badly documented. I did not find any examples about how to write them for arrays of doubles.

Spark

http://www.spark-project.org/

Created by Matei Zaharia from UC Berkeley AMP Lab.
Language Scala.

I was very impressed by Spark. It was easy to build with SBT. It was very simple to write my program. It was trivial to define group by sum for double array, just by defining a vector class with addition.

Issues

I tested my program in local mode. I was very happy that I had a workable solution. Then I investigated moving it to a Hadoop cluster. For this Spark had dependency on Apache Mesos cluster manager. Mesos is a thin virtual layer that Hadoop is running on top of.

It turned out that Spark is not actually running on Hadoop. It is running on HDFS and is an alternative to Hadoop. Spark can run side by side with Hadoop if you have Apache Mesos installed.

Spark is an interesting framework that can outperform Hadoop for certain calculation. It uses the same code from running in memory calculation and code on a big HDSF cluster.

If you have full control over your cluster Spark could be a good option, but if you have to run on an established Hadoop cluster it is very invasive.

Scrunch

https://github.com/cloudera/crunch/tree/master/scrunch

Created by Cloudera.
Language Scala.


Scrunch looked powerful and simple. It is easy to build with SBT.

Issues

Dependent on Cloudera's Hadoop 0.20.2 distribution.

The web site describes it as alpha quality, so I did not do much with Scrunch.

Cascalog

https://github.com/nathanmarz/cascalog

Created by Nathan Marz from Twitter.
Language Clojure, a modern Lisp dialect.

As Scalding Cascalog is built on Cascading.

Easy to build with the Clojure build system Leiningen.

It is used as a replacement for Pig. You can run it from the Clojure REPL or run scripts, and get a full and consistent language.

I tried Cascalog and was impressed. It is a good option if you are working in Clojure

Hadoop vs. Storm

I had to solve the same log file statistics problem in real-time using the Storm framework and this was much simpler.

Why is Hadoop so hard

• Hadoop is solving a hard problem
• Hadoop is a big software stack with a lot of dependencies
• Libraries only work with specific versions of Hadoop
• Serialization is adding complexity, see next section
• The technology is still not mature

Looks like some of these problems are getting addressed. Hadoop should be more stable now that Hadoop 1.0 has been released.

Serialization in Hadoop

Java have a built in serialization format, but it is not memory efficient. Serialization in Hadoop has to be: 

• Memory efficient
• Use compression
• Support self-healing
• Support splitting a file in several parts

Hadoop SequenceFile format has these properties, but unfortunately it does not speak so well with the rest of the world.

Serialization is adding complexity to Hadoop. One reason Storm is simpler is that it just uses Kryo Java serialization to send objects over the wire.

Apache Avro is a newer file format that does everything that is needed by Hadoop but is speaks well with other languages as well. Avro is supported in Pig 0.9 and should be in Hive 0.9.

High level Hadoop frameworks in Java

You do not have to use Scala or Clojure to do high level Hadoop in Java. Cascading and Crunch are two Java based high level Hadoop frameworks. They are both based on the idea is that you set up a Hadoop data flow with pipes.

Functional constructs are clumpy in Java. It is a nuisance but doable. When you deploy code to a Hadoop cluster you have to pack up all your library dependencies into on super jar. When you are using Scala or Clojure you need to also package the whole language into this super jar. This also adds complexity. So using Java is a perfectly reasonable choice.

Here are two high level Java Hadoop libraries:

Cascading

http://www.cascading.org/

Both Scalding and Cascalog is built on top of Cascading.

Cascading is well documented. You can write concise Java code for Hadoop.

Crunch

https://github.com/cloudera/crunch

Scrunch is built on top of Crunch.

Conclusion

I liked all of the Hadoop frameworks I tried, but there is a learning curve and I found problems with all of them.

Extract Transform Load

For ETL Hive and Pig are my top picks. They are easy to use, well supported, and part of the Hadoop ecosystem. It is simple to integrate a prebuilt Map Reduce classes in data flow in both. It is trivial to join data source. This is hard to do in plain Hadoop.

Cascalog is serious contender for ETL if you like Lisp / Clojure.

Hive vs. Pig

I prefer Hive. It is based on SQL. You can use your database intuition and you can access it though JDBC.

Scala based Hadoop frameworks

They all made Hadoop programming look remarkable close to normal Scala programming.

For programming Hadoop Scalding is my top pick since I like the named fields. 

Both Scrunch and Scoobi are simple and powerful Scala based Hadoop frameworks. They require Cloudera's Hadoop distribution, which is a very popular distribution.

Scala, Eclipse and Maven integration tutorial

I have evaluated Scala as a language for cloud computing and Hadoop. One requirement was a robust development environment, with a real build system, a good IDE with code completion and debugging.

The combination of Scala, Eclipse and Maven seemed like a fit for this requirement, but my initial experience was mixed.

Problems with Scala, Eclipse and Maven integration

It was easy to install Scala, Eclipse and Maven, but when I set up a project it had a persistent error in Eclipse:

object Predef does not have a member AnyRef

Other problems:

• There were problems running the unit test.
• I had to restart Eclipse a lot.
• Eclipse had Scala set to version 2.9.0.1 while Maven had 2.8.0. When I tried to change Maven to use 2.9.0.1 the pom.xml file would be marked as having an error.

I searched internet for help but could not find it. After a good deal of experimenting I sorted out the problems and found a good solution.

Software versions

My setup is:

• Scala 2.9.0.1.
• Eclipse 3.7 Indigo
• Scala-ide Eclipse plugin: scala nightly 29 - http://download.scala-ide.org/nightly-update-wip-experiment-2.9.0-1

Scala, Eclipse Maven project setup tutorial

Here are that steps that I took to set up at new Scala, Eclipse and Maven project so it works with unit testing.

Press menu item: File - New - Other...

Select Maven Project

Select the org.scala-tools.archetypes scala-archetype-simple

Add group id and artifact id to project. Click Finish

This will create the project with example program and unit tests, but it will leave Eclipse in an unstable state

In the project's pom.xml file make the changes that I have marked in red:

<properties>
 <maven.compiler.source>1.5</maven.compiler.source>
 <maven.compiler.target>1.5</maven.compiler.target>
 <encoding>UTF-8</encoding>
 <scala.version>2.9.0-1</scala.version>
</properties>


<dependency>
 <groupId>org.scala-tools.testing</groupId>
 <artifactId>specs_${scala.version}</artifactId>
 <version>1.6.8</version>
 <scope>test</scope>
</dependency>

Now both Scala IDE and Maven are both using the same version of Scala. Scala 2.9.0.1


Right click the whole project and select: Configure - Add Scala Nature

Now use the Maven build system to clean, build and run unit tests. Run from either Eclipse or command line.

From Eclipse, right click the whole project and selecting:

Maven clean

Maven install

From command line:

C:\prog\apache-maven-2.2.1\bin\mvn clean

C:\prog\apache-maven-2.2.1\bin\mvn install

Note that you have to use Maven 2.2 and not Maven 3.

Now there should be no more errors.
The unit test: "scalatest.scala" has some problems, delete it.

Run all unit tests from Eclipse. By right clicking the whole project and select Run As JUnit Test

Now you can see the result in the JUnit runner.

Final impression of Scala, Eclipse and Maven integration

Once I had resolved the problems the Scala, Eclipse and Maven combination was a great development environment meeting my requirements.

One thing that is currently missing from the Scala Eclipse plugin is code refactoring. Refactoring works very well in both Eclipse for Java and Visual Studio for C#.

Natural language processing in F# and Scala

I do natural language processing in C# 3.5 and Python. My work includes classification, named entity recognition, sentiment analysis and information extraction. Both C# and Python are great languages, but I have some unmet needs. I am investigating if there are any new languages that would help.

I 2010 I tried out 3 new languages:
Natural language processing in Clojure, Go and Cython

Recently I have investigated F# and Scala. They are both hybrid functional - object oriented languages; inspired by ML / OCaml / Haskell and Java / C#.

Python as the benchmark

Python is widely used in natural language processing. I am most productive in Python for NLP work. Here are a few reasons why:

• NLTK is a great Python NLP library
• Lot of open source math and science libraries e.g. NumPy and SciPy
• PyDev is a good development environment
• Good integration with MongoDB library
• Great for rapid development

Python shortcomings

• Slow compared to compiled language
• GUI support is crude
• Multi-threading is crude
• Compilation does give more robustness

It should be possible to make a super language that has the elegance of Python, but without these shortcomings.

My first Scala experience

In 2006 I thought Scala was this super language. It is very advanced; you can call any Java libraries from Scala, including all the open source libraries. But I ran into a list of problems with Scala:

• The Scala IDE was far behind Eclipse Java
• Scala is a quite complex language
• The Java libraries and the functional programming libraries were badly integrated
• There were no Scala REPL or interpreter like in Python

Scala was stable enough for use, but it did not improve my productivity so after some months I went back to using Python as my scripting language.


Python's weakness

Recently I had to make a small text processing application that end users could use directly. This was not the best fit for Python. Normally my Python programs have no GUI and are controlled by command line parameters.

I had 2 Python options:

Make simple GUI using TkInter
TkInter is a Python wrapper of TK, the cross platform GUI toolkit. It is pretty crude by modern GUI standards, but would have been good enough. However trying to install all the Python libraries that I needed on the end users machine would be setting myself of for a maintains nightmare.

Wrap code in web application
I could wrap a web interface around it. The application is using a lot of memory and I would have to maintain a web application.

I had a 1 week hard deadline for the task and both of these options looked unappealing. I needed something else...

My first F# application

I took a chance on F#, and managed to learn enough F# to finish the program by my 1 week deadline.

There is no GUI builder for F# in Visual Studio, but it was pretty easy to hand code a simple WinForms GUI to wrap around the core code. It was not pretty but you could give it to an end user. The whole application ended up being one 40KB executable file, and it was very fast. F# had actually filled a niche that Python does not do so well.

There were also problems, I wrote the whole application from scratch, while in Python I would have been able to use NLTK, write the code faster and get better results.

All in all this was very good experience. I thought that F# would be a good supplement to my Python library. It would both give me raw speed when I need it and good connectivity with C#, ASP.NET, WPF and Microsoft Office.

Functional programming benefits

Functional programming is a great fit for my NLP work.

I have a lot of different text sources: database, flat file, directory, public RESTful web application services.

I have many word transformations: stop word filters, stemmers, custom filters.

I need many operations building on other operations: Bigram finder, POS tagger, named entity recognizer.

Created different reports: database, csv, Excel.

In functional languages you can just take any combinations of these operations and easily pipe them together while getting good compiler support. This does not fit so well with object oriented programming were you are more concerned with encapsulation.

F# impression

F# is the first compiled language I tried that is comparable to Python in simplicity and elegance. It has a real Pythonic feel:

• F# is fast
• Simple and elegant
• Good development environment in Visual Studio 2010
• Best concurrency support of any language I have seen
• Good database support
• Good MongoDB library
• Simple to combine F# with C# or VB.NET for ASP or WPF
• Good REPL

Issues

• Runs best under Windows
• For an IDE you really need Visual Studio 2008 or 2010, and that cost at least $700
• F# can be compiled and run the shell from SharpDevelop 4.0 and 4.1, but you do not have the same productivity
• The math libraries under .NET are not as good as NumPy and SciPy
• The NLP libraries are better under Python

Scala revisited

After the success with F# I was very curious about why F# has been so much more successful than my first experience with Scala.

I looked at an F# and Scala cheat sheet and thought they look remarkably similar. I watched a few screen casts and found no obvious problems. I bought the book: Programming in Scala, Second Edition, it turned out to be a very interesting computer science book and I read the whole 852 pages. Scala still looked good.

I installed the Scala Eclipse plugin and wrote some code. Both the language and the IDE have come a long way during the last 5 years:

• 15 books about Scala
• 2 great free books
• Tooling is much better
• IDE is much better with code completion
• Native NLP libs: ScalaNLP and Kiama

Of all the issues I had when I first tried Scala. The only remaining one is:
Scala is a pretty complex language

It is incredible how Scala has taken a lot of messy features from Java and turned it into a clean modular system, at the cost of some complex abstractions.

F# vs. Scala

Despite many similarities, the languages have a different feel. F# is simpler to understand, while Scala is the more orthogonal language. I have been very impressed by both.

F# better

• Simpler to understand
• Fantastic concurrency
• Tail recursion optimized
• Works well with Windows Azure

Scala better 

• More orthogonal, reusing the same constructs
• Works with any Java library so more libraries
• Better NLP libraries
• Works well with Hadoop

Cloud computing

Functional programming works well with cloud computing. For me the availability of a good functional language is a substantial factor in selecting a cloud platform.

Google introduced MapReduce to handle massive parallel multi computer applications.

Hadoop is the Java based open source version of MapReduce. To run Hadoop natively it has to run a JVM language like Java or Scala.

Hadoop Streaming extends a limited version of Hadoop to work with programs written in other programming languages as long as they work like a UNIX pipes that read from stdin and write to stdout.

There is a Python wrapper for Hadoop Streaming called Dumbo. Python is around 10 times slower than Java and Dumbo is a limited version of the Hadoop, so if you are trying to do NLP on massive amount of data this might not solve your problems.

Scala is fast and will give you full access to run native Hadoop.

Microsoft's version or MapReduce is called: Dryad or LINQ to HPC. It is not officially released yet, but F# works well with Windows Azure.


NLP and other languages

Let me finish by giving a few short comparisons of F# and Scala with other languages:

Clojure vs. Scala

Clojure is a LISP dialect that it also running on the JVM, and it the other big functional language running there. Clojure has some distinct niches for NLP:

Clojure better

• Language understanding
• Formal semantic: taking text and translating it to first order propositional logic
• Artificial intelligence tasks

Scala better

• It is easy to write fast Scala code
• Smaller learning curve coming from Java

I tried Clojure recently and was very impressed; but more of my work falls in the category that would benefit from Scala.

Java vs. Scala

Java better

• Better IDE tools and support
• Better GUI builders
• Great refactoring support
• Many more programmers that know Java

Scala better

• Terser code
• Closures
• First class function
• More expressive language

C# vs. F#

C# better

• Better IDE tools and support
• Better GUI builders
• There are a lot more programmers that know C#
• Better LINQ to SQL support

F# better

• Terse code
• Better support for concurrency, Synch, continuations
• More productive for NLP

Conclusion

F# and Scala are similar hybrid functional object oriented languages.

For years I have periodically tried functional programming languages to see if they were ready for mainstream corporate computing; and they were not. With the recent spread of functional features into object oriented languages I thought that real functional programming languages would soon be forgotten.

I was pleasantly surprised by how well F# and Scala work now. Functional languages are finally coming of age and becoming useful in mainstream corporate computing. They are stable enough, and they have niches were they are more productive than object oriented languages like C# and Java.

I really enjoy programming in F# and Scala, they are a very good fit for natural language processing and cloud computing. For bigger NLP projects I now prefer to use F# or Scala over C# or Java.

For GUI and web programming the object oriented languages still rules. Stick with C# or Java if the NLP part is small or GUI or web interface is the domineering part.

Java and C# are also improving e.g. by adding more functional features. Many working programmers are well served by just waiting for Java 8 or C# 5. But functional programming is here to stay. Rejoice...


Newer follow up article covering Scala, ScalaNLP and Spark MLib

Cloud Computing For Data Mining Part 1

The first half of this blog post is about selecting a cloud provider for a data mining and natural language processing system. I will compare 3 leading cloud computing providers Amazon Web Services, Windows Azure, OpenStack.
To help me chose a cloud provider I have been looking for users with experience running cloud computing for application similar to data mining. I found them at CloudCamp New York June 2011. It was an unconference, so the attendees were split into user discussion groups. The last half of the post I will mention the highlight from these discussions.


The Hype

"If you are not in the cloud you are not going to be in business!"

This is the message many programmers, software architects and project managers faces today. You do not want to go out of business because you could not keep up with the latest technologies; but looking back many companies have gone out of business because they invested in the latest must have technology, that turned out to be expensive and over engineered.

Reason For Moving To Cloud

I have a good business case from using cloud computing: Namely scale a data mining system to handle a lot of data. To begin with it could be a moderate amount of data, but it could be changed to a Big Data with short notice.


Horror Cloud Scenario

I am trying minimize the risk of this scenario:

1. I port to a cloud solution that is tied closely to one cloud provider
2. Move the applications over
3. After a few months I find that there are unforeseen problems
4. No easy path back
5. Angry customers are calling

Goals

Here are my cloud computing goals in a little more details:

• Port data mining system and ASP.NET web applications to the cloud
• Chose cloud compatible with code base in .NET and Python
• Initially the data volume is moderate but it could possibly scale to Big Data
• Keep cost and complexity under control
• No downtime during transition
• Minimize risk
• Minimize vendor lock in
• Run the same code in house and in the cloud
• Make rollback to in house application possible

Amazon Web Services vs. Windows Azure vs. OpenStack

Choosing the right cloud computing provider has been time consuming, but also very important.

I took a quick stroll through Cloud Expo 2011, and most big computer companies were there presenting their cloud solutions.

Google App Engine is a big cloud service well suited for front end web application, but not good for data mining, so I will not cover that here.

The other 3 providers that have generated most momentum are: EC2, Azure and OpenStack.

Let me start by listing their similarities:

• Virtual computers that can be started with short notice
• Redundant robust storage
• NoSQL structured data
• Message queue for communication
• Mountable hard disk
• Local non persistent hard disk

Now I will write a little more about where they differ, and their good and the bad part:

Amazon Web Services, AWS, EC2, S3

Good:

• This is the oldest cloud provider dating back to 2004
• Very mature provider
• Other providers are catching up with AWS's features
• Well documented
• Work well with open source, LAMP and Java
• Integrated with Hadoop: Electric Map Reduce
• A little cheaper than Windows Azure
• Runs Linux, Open Solaris and Windows servers
• You can run code on your local machine and just save the result into S3 storage

Bad:

• You cannot run the same code in house and in the cloud
• Vendor lock in

Windows Azure

Good:

• Works well with the .NET framework and all Microsoft's tools
• It is very simple to port an ASP.NET application to Azure
• You can run the same code on you development machine and in the cloud
• Very good development and debugging tools
• F# is a great language for data mining in cloud computing
• Great series of video screen casts

Bad:

• Only run Windows
• You need a Windows 7, Windows Server 2008 or Windows Vista to develop
• Preferably you should have Visual Studio 2010
• Vendor lock in

OpenStack

OpenStack is a new open source collaboration that is making a software stack that can be run both in house and it the cloud.

Good:

• Open source
• Generating a lot of buzz
• Main participants NASA and Rackspace
• Backed by 70 companies
• You can run your application either in house or in the cloud

Bad:

• Not yet mature enough for production use
• Windows support is immature

Java, .NET Or Mixed Platform

For data mining selecting the right platform is a hard choice. Both Java and .NET are very attractive options.

Java only
For data mining and NLP there are a lot of great open source project written in Java. E.g. Mahout is a system for  collaborative filtering and clustering of Big Data, with distributed machine learning. It is integrated with Hadoop.
There are many more OSS: OpenNLP, Solr, ManifoldCF,

.NET only
The development tools in .NET are great. It works well with Microsoft Office.
Visual Studio 2010 comes with F#, which is a great language for writing worker roles. It is very well suited for light weight threads or async, for highly parallel reactive programs.

Mix Java and .NET
You can mix Java and .NET. Cloud computing makes is easier than ever to integrate different platforms. You already have abstract language agnostic service for communication with message queue, blob storage, structured data. If you have an ASP.NET front end on top of a collaborative filtering of Big Data this would be a very attractive option.

I still think that combining 2 big platforms like Java and .NET is introducing complexity, compared to staying within one platform. You need an organization with good resources and coordination to do this.

Choice Of Cloud Provider

I still have a lot of unanswered questions at this point.

At the time of writing June 2011 OpenStack is not ready for production use. So that is out for now.

I have run some test on AWS. It was very easy to deploy my Python code to EC2 under Linux. Programming C# that used AWS services was simple.

I am stuck waiting to get a Window 7 machine so I can test Window Azure.

Both EC2 and Azure seem like viable options for what I need. I will get back to this in part 2 of the blog post.

Highlights from Cloud Camp 2011

A lot of people are trying to sell you cloud computing solutions. I have heard plenty of cloud computing hype. I have been seeking advice from people that were not trying to sell me anything and had some real experience, and try to find some of the failures and problems in cloud computing. 

I went to Cloud Camp June 2011 during Cloud Expo 2011 in New York. Cloud computing users shared their experience. It was an unconference, meaning spontaneous user discussion breakout groups were formed. The rest of this post is highlight from these discussions.

Hadoop Is Great But Hard

Hadoop is a Java open source implementation of Google's Map Reduce. You can set up a workflow of operations and Hadoop will distribute them over a multiple computers, aggregate the result and rerun operations that fail. This sounds fantastic, but Hadoop is a pretty complex system, with a lot of new terminology and a steep learning curve.


Security Is Your Responsibility

Security is a big issue. You might assume that the cloud will take care of security, but you should not. E.g. you should clean up the hard disks that you have used it, so the next user cannot see your data.

Cloud Does Not Automatically Scale To Big Data

The assumption is that you put massive amounts of data in the cloud. And the cloud takes care of the scaling problems.
If you have a lot of data that needs little processing. Then cloud computing becomes expensive: you store all data in 3 different locations and it is expensive and slow to take it down to different compute nodes. This was mentioned as the reason why NASA could not using S3, but build its own Nebula platform.

You Accumulate Cost During Development

An entrepreneur building a startup ended up paying $2000 / month for EC2. He used a lot of different servers and they had to be running with multiple instances, even though he was no using a lot of resources. This might be cheap compared to going out and buying your own servers, but it was more expensive than he expected.

Applications Written In .NET Run Fine Under EC2 Windows

An entrepreneur said that he was running his company's .NET code under EC2. He thought that Amazon was more mature than Azure, and Azure was catching up. He preferred to make his own framework.

Simpler To Run .NET Application On Azure Than On EC2

A cloud computing consultant with lots of experience in both Azure and EC2 said: EC2 gives you a raw machine you have to do more to get your application running than if you plop it into Windows Azure.
It is very easy to port an ASP.NET application to Windows Azure.

Cash Flow, Operational Expenses And Capital Expenses

An often cited reason why cloud computing is great is that a company can replace big upfront capital expenses with smaller operational expenses. A few people mentioned that companies live by their cash flow and they do not like to have an unpredictable operational expenses, but are more comfortable with predictable capital expenses.

Practical Probabilistic Topic Models for NLP

Latent Dirichlet Allocation, LDA is a new and very powerful technique for finding the topics in a collection of texts, using unsupervised learning. LDA is a probabilistic topic models. LDA was developed in 2003 and rely on advanced math. This post is a practical guide about how to get started building LDA models and software.

LDA will have a substantial impact on corpus based natural language processing; since it opens up for easy creation of semantic models based on machine learning.

Motivation for topic models

With the Internet we have large amount of text available. Having the text categorized into topics make text search much more precise and makes it possible to find similar documents.

Text categorization is not an easy problem:

• Texts usually deals with more than one topic
• There is no clear standard for categorization
• Doing it by hand is infeasible

Nuanced categorized is a hard problem, with many moving parts, but in 2003 David M. Blei, Andrew Y. Ng and Michael I. Jordan published an article on a new approach called Latent Dirichlet Allocation. LDA can be implemented base on research articles, but if you are not a machine learning academic the math is intimidating and the material is still new.

There is actually good material available, but finding all the pieces takes some work. Most things you need are available online for free. Here is a chronological account for what I did to understanding LDA and start implementing it.

Need for more sophisticated hierarchical topic models

In 2009 I needed a fine grained classification of text, using unsupervised or semi supervised training. I spend a little time thinking about it, and had some idea about making bootstrapped training in a 2 layered hierarchy. It was hackish, complex and I was not sure how numerically stable it was. I never got around to implement it.

David Blei

I went to 4 Annual Machine Learning Symposium in 2009 and asked around for solutions to my problem. Several attendees told me to look at David Blei work. I did but he has written a lot of math heavy articles, so I did not know were to start.

I was lucky to see David Blei give a presentation on LDA first at the 5 Annual Machine Learning Symposium. David Blei works at Princeton and just exudes brilliance. He gave a lucid entertaining description of the LDA with examples. It wa really shocking to see the LDA algorithm find scientific topics on its own with no human intervention.

I saw him give the same talk at the NYC Machine Learning Meetup, and luckily that was videotaped here are part 1 and part 2. I watched these videos a few times. This gave me a good intuition for the algorithm.

I looked through his articles and found a good beginner articles BleiLafferty2009. I read through that several time, but I could not understand it.

I went out and bought the text book that David Blei recommended: Pattern Recognition and Machine Learning by Christopher M. Bishop. After reading the introduction chapter, I read BleiLafferty2009 again and was able to understand it. On page 10 the essence of the algorithm is described in a small text box.

Software implementation of LDA

There are plenty open source implementation of LDA. Here are a few observations:

lda-c in C by David Blei is an implementation in old school C. The code is readable, concise and clean.

lda for R package by Jonathan Chang. Implementing many models with extensive documentation.

Online LDA in Python by Matt Hoffman. Short code, but not too much documentation.

LDA Apache Mahout in Java. Active development community works with Hadoop / MapReduce.

JGibbLDA in Java

GibbsLDA++ C++

Arbylon Projects Java

No matter what language you prefer there should be a good implementation.

Practical software considerations

All the implementations looked good. But if you want to use LDA software then robustness, scalability and extendibility are big issues. First you just want the algorithm to run for simple text input. Next day you want the following options:

• Better word tokenizer
• Bigrams and collocation
• Words stemmer
• LDA on structured text
• Read from database

Programming language choice for LDA

Here is a little common sense advice on choice of programming language for LDA programming.

C
C is an elegant, simple system programming language.
C is not my first choice of a language for text processing.


C++
C++ is a very powerful but also complex language.

NLP lib: The Lemur Project

I would be happy to use C++ for text processing.

C#

C# is a great language.

NLP lib:  SharpNLP. 

You will have to implement LDA yourself or port one of the other implementations. SciPy is getting ported to C# but it does not have the best numeric libraries.

Clojure

Clojure is a moderate sized LISP dialect build on the Java JVM.

NLP lib: OpenNLP through clojure-opennlp.

LISP is classic AI language and you can use one of the Java LDA implementations.

Java

Java is modern object oriented programming language with access to every thinkable library.

NLP lib: OpenNLP.

Python

Python is an elegant language very well suited for NLP.

NLP lib: NLTK, using NumPy and SciPy

R
R is a fantastic language for statistics, but not so great for low level text processing.
NLP lib:
The R implementation of LDA looks great; I think that it is common to do all the preprocessing in another language say Perl. And then do all the rest of the work in R.

Different versions of LDA

There are now a lot of different LDA models geared towards different domains. Let me just mention a couple:

Online LDA
Online means that: you do learning of the models in small batches; instead of on all the documents. This is useful for a continuously running system.

Dynamic LDA
Good for handling text that stretches over a long time interval say 100 years.

Hierarchical LDA
This will handle topics are organized in hierarchies.

Gray box approach to LDA

The math needed for LDA is advanced. If you do not succeed in understand it I still think that you can learn to use the code, if you are willing to take something on faith and get your hands dirty.

Bedtime Science Stories My Science Education Blog

I started a science education blog called: Bedtime Science Stories. Here is a little excerpt from my first post: Can and should a 3 year old girl be into science?

I have a 3 year old daughter that has take a bit of an interest in science. We have been talking about science when I put her to bed at night.

Last Sunday I discovered a new book called Battle Hymn of the Tiger Mother by Amy Chua, who is a law professor at Yale. She is using extreme methods to push her 2 daughters to academic excellence. They had to be the best in their class in everything except drama and physical education. Math was a topic that she really drilled them in. Just reading the back cover sent me into a rage; so much that I decided to start a new blog: Bedtime Science Stories, just to get my anger out.

Science should not be an elite activity. Making it very competitive will make a new generation of kids hate math and science. Understanding our world is worthwhile activity even if you are not the best in your class.

My 3 year old daughter