Big Data: What Worked?

"Big data" created an explosion of new technologies and hype: NoSQL, Hadoop, cloud computing, highly parallel systems and analytics.

I have worked with big data technologies for several years. It has been a steep learning curve, but lately I had more success stories.

This post is about the big data technologies I like and continue to use. Big data is a big topic. These are some highlights from my experience.

I will relate big data technologies to modern web architecture with predictive analytics and raise the question:

What big data technologies should I use for my web startup?

Classic Three Tier Architecture

For a long time software development was dominated by the three tier architecture / client server architecture. It is well described and conceptually simple:

• Client
• Server for business logic
• Database

It is straightforward to figure out what computations should go where.

Modern Web Architecture With Analytics

The modern web architecture is not nearly as well established. It is more like an 8 tiered architecture with the following components:

• Web client
• Caching
• Stateless web server
• Real-time services
• Database
• Hadoop for log file processing
• Text search
• Ad hoc analytics system

I was hoping that I could do something closer to the 3-tier architecture, but the components have very different features. Kicking off a Hadoop job from a web request could adversely affect your time to first byte.

A problem with the modern web architecture is that a given calculation can be done in many of those components.

Architecture for Predictive Analytics

It is not at all clear what component predictive analytics should be done in.

First you need to collect user metrics. In what components can do you do this?

• Web servers, store the metric in Redis / caching
• Web servers, store the metric in the database
• Real-time services aggregates the user metric
• Hadoop run on the log files

User metric is needed by predictive analytics and machine learning. Here are some scenarios for this:

• If you are doing simple popularity based predictive analytics this can be done in the web server or a real-time service.
• If you use a Bayesian bandit algorithm you will need to use a real-time service for that.
• If you recommend based on user similarity or item similarity you will need to use Hadoop.

Hadoop

Hadoop is a very complex piece of software to handle very large amounts of data that cannot be handled by conventional software because it is too big to fit on one computer.

I compared different Hadoop libraries in my post: Hive, Pig, Scalding, Scoobi, Scrunch and Spark.

Most developers that have used Hadoop complain about it. I am no exception. I still have problems with Hadoop jobs failing due to errors that are hard to diagnose. Generally I have been a lot happier about Hadoop lately. I am only using Hadoop for big custom extractions or calculations from log files stored in HDFS. I do my Hadoop work in Scalding or HIVE

The Hadoop library Mahout can calculate user recommendations based on user similarity or item similarity.

Scalding

Hadoop code in Scalding looks a lot like normal Scala code. The scripts I am writing are often just 10 lines of code and look a lot like my other Scala code. The catch is that you need to be able to write idiomatic functional Scala code.

HIVE

HIVE makes it easy to extract and combine data from HDFS. You just write SQL after some setup of a directory with table structure in HDFS.

Real-time Services

Libraries like Akka, Finagle and Storm are good for having long running stateful computations.
It is hard to write correct highly parallel code that scales to multiple machines using normal multithreaded programming. For more details see my blog post: Akka vs. Finagle vs. Storm.

Akka and Spray

Akka is a simple actor model taken from the Erlang language. In Akka you have a lot of very lightweight actors, they can share a thread pool. They do not block on shared state but communicate by sending immutable messages.

One reason that Akka is a good fit for real-time services is that you can do varying degrees of loose coupling and all services can talk with each other.

It is hard to change from traditional multithreaded programming to using the actor model. There are just a lot of new actor idioms and design patterns that you have to learn. At first the actor model seems like working with a sack of fleas. You have much less control over the flow due to the distributed computation.

Spray makes it easy to put a web or RESTful interface to your service. This makes it easy to connect your service with the rest of the world. Spray also has the best Scala serialization system I have found.

Akka is well suited for: E-commerce, high frequency trading in finance, online advertising and simulations.

Akka in Online Advertising

Millions of users are interacting with fast changing ad campaigns. You could have actors for:

• Each available user
• Each ad campaign
• Each ad
• Clustering algorithms
• Each cluster of users
• Each cluster of ads

Each actor is developing in time and can notify and query all other actors.

NoSQL

There are a lot of options, with no query standard:

• Cassandra
• CouchDB
• HBase
• Memcached
• MongoDB
• Redis
• SOLR

I will describe my initial excitement about NoSQL, the comeback of SQL databases and my current view on where to use NoSQL and where to use SQL.

MongoDB

MongoDB was my first NoSQL technology. I used it to store structured medical documents.

Creating a normalized SQL database that represents a structured data format is a sizable task and you easily end up with 20 tables. It is hard to insert a structured document into the database in the right sequence, so foreign key constraints are satisfied. LINQ to SQL helped with this but it was slow.

I was amazed by MongoDB's simplicity:

• It was trivial to install
• It could insert 1 million documents very fast
• I could use the same Python NLP tools for many different types of documents

I felt that SQL databases were so 20th century.

After some use I realized that interacting with MongoDB was not as easy from Scala. I tried different libraries Subset and Casbah.

I also realized that it is a lot harder to query data from MongoDB than a SQL database both in syntax and expressiveness.
Recently SQL databases have added JSON as a data type, taking away some of MongoDB's advantage.

Today I use SQL databases for curated data. But MongoDB for ad hoc structured document data.

Redis

Redis is an advanced key value store that is mainly living in memory but with backup to disk. Redis is a good fit for caching. It has some specialized operations:

• Simple to age out data
• Simulates pub sub
• Atomic update increments
• Atomic list append
• Set operations

Redis also supports sharding well, in the driver you just give a list of Redis servers and it will send the data to the right server. Redistributing data after adding more sharded servers to Redis is cumbersome.

I first thought that Redis had an odd array of features but it fits the niche of real-time caching.

SOLR

SOLR is the most used enterprise text search technology. It is built on top of Lucene.
It can store and search document with many fields using an advanced query language.
It has an ecosystem of plugins doing a lot of the things that you would want. It is also very useful for natural language processing. You can even use SOLR as a presentation system for your NLP algorithms.

To Cloud or not to Cloud

A few years back I thought that I would soon be doing all my work using cloud computing services like Amazon's AWS. This did not happen, but virtualization did. When I request a new server the OPS team usually spins up a virtual machine.
A problem with cloud services is that storage is expensive. Especially Hadoop sized storage.

If I were in a startup I would probably consider the cloud.

Big and Simple

My fist rule for software engineering is: Keep is simple.

This is particularly important in big data since size creates inherent complexity.
I made the mistake of being too ambitious too early and think out too many scenarios.

Startup Software Stack

Back to the question:

What big data technologies should I use for my web startup?

A common startup technology stack is:
Ruby on Rails for your web server and Python for your analytics and hope that a lot of beefy Amazon EC2 servers will scale your application when your product takes off.
It is fast to get started and the cloud will save you. What could possibly go wrong?

The big data approach I am describing here is more stable and scalable, but before you learn all these technologies you might run out of money.

My answer is: It depends on how much data and how much money you have.

Big Data Not Just Hype

"Big data" is misused and hyped. Still there is a real problem, we are generating an astounding amount of data and sometimes you have to work with it. You need new technologies to wrangle this data.

Whenever I see a reference to Hadoop in a library I get very uneasy. These complex big data technologies are often used where much simpler technologies would have sufficed. Make sure your really need them before you start. This could be the difference between your project succeeding or failing.

It has been humbling to learn these technologies but after much despair I now enjoy working with them and find them essential for those truly big problems.

Growing Python projects from small to large scale

You need significantly different principles for developing small, medium and large scale software system.

When my project started to become big I searched the Internet for some guidelines or best practices for how to scale Python, but did not find much. Here are a few of my observations on what technique to use for what project sizes.

General principle

For a small system you can spend most of your time solving the problem, but the bigger the system gets the more time you spend on project plans, coordination and documentation. The complexity and cost does not scale linearly with the size of the project but maybe scales with the square of the size. This holds for different styles of project management both waterfall and agile.

A central problem is minimizing dependencies and avoiding tight coupling. John Lakos has written an excellent book on software scaling called: Large-Scale C++ Software Design here is a summary. It is a very scientific and stringent approach, which is specific for C++. He developed a metric for how much dependencies you have in your system. His technique are not a good fit for smaller projects, you could finish several scripts before you could even implement his methodology.

Small scripts
Keep it simple. Focus on the core functionality. Minimize the time you spend on setting up the project.

Medium applications
Spending some time organizing things, will save you time in the long run.

Large applications
Here you need a lot of structure; otherwise the project will not be stable.

Development environment

Small scripts

I use PyWin Windows IDE.

• It is lightweight
• No need for Java or Eclipse
• Syntax highlighting
• Code completion at run time and some at write time
• Allow primitive debugging
• You do not need to set up a project to use it.

Medium applications
I have used both PyWin and PyDev.

Large applications
I would strongly recommend PyDev Eclipse plugin. It is a modern IDE and runs pylint continuously and has good code completion while writing code. It will find maybe half the error a compiler would find. This improves the stability a lot and was the most important change that I made from my old coding style.

Organization of code

Small scripts
Use one module / file with all the code in. This can have several classes. The advantage is that deployment becomes trivial: you just email the script to the user. This works for modules up to around 3000 lines of code.

Medium applications
Use one directory with all modules in. This gives you fewer issues with PYTHONPATH.

Make a convention for naming field names, database name and parameter name. Put all these names in a module that only contains string constants, and use these in your code instead of raw string.

Use a separate repository for the project. I package the Python and other self written executable together in a repository, even when I have another source control system for the compiled sources.

This works up till around 40 Python modules, then it become hard to find anything.

Large applications
Read and follow the Python style guide. Before I followed a Java style guide since Java is big on coding convention, but the Python style is actually pretty different. A noticeable difference is a Java file contains a main class with a title case name and the file has the same name. In python modules should have short lowercase name while the classes still should have title case names.

Organizing packages as an a-cyclical graph
Refactor the modules into packages. The packages should be organized as an a-cyclical graph. So at the lowest level you would have an util package that is not allowed to reference anything else. You can have other specialized packages that can access the util package. Over that I have the main source directory with code that is central and general. Over that I have a loader package that can access all the other packages.

One problem when you have different directories is that you need the PYTHONPATH include all the code. A good way to do this is to try to add the parent directory to the system path before you import any of the modules.

Documentation

Small scripts
Usually I have:

• Python docstring in the program. 
• Print a usage message

Medium applications
Have a directory for documentation. To keep it simple I prefer to use simple HTML. I find that Mozilla SeaMonkey is simple to use and generates clean HTML you can do a diff on. Often I have:

• User documentation page 
• Programmer documentation page
• Release notes
• Example data

Large applications
At this point using automatically generated documentation and some sort of wiki format for writing documentation is a good idea.

Communication

Input and output account for a sizable part of your code. I prefer to use the most lightweight method I can get away with.

Small scripts and medium applications
Communication is done with flat files, csv files and database.

Large applications
Communication is done with flat files, csv files, database, MongoDB and CherryPy.

MongoDB have dramatically simplified my work, before different types of structured data demanded their own database with several tables. Now I just load the data into a MongoDB collection. MongoDB make very different structured documents look very uniform and trivial to load from Python. After that I can use the same script on very different data.

When you have a script and find out that you need to have other programs call it. It is very simple to create XML, JSON or text based RESTful web service using CherryPy. You just add a 1 line annotation to a method and it is now a web service. You barely have to make any changes to your program. CherryPy feels very Pythonic. This will give you very cheap way to connect to a GUI and a web site written in other languages.

Unit tests

Small scripts
Unit tests give you a small advantage. I still write unit tests unless there is an emergency, and then I usually regret it.

Large applications
The bigger the system the more important it is that the individual pieces works. Large systems are not maintainable if you do not have unit tests.

Source control system

I put any code that I use for production in a source control system. I usually use Subversion or GIT.

Subversion is good for centralized development, and it is nice that each check in has a sequential revision number so that you can see revision number 123 and next 124.

GIT is better for distributed development; it is easy to create a local repository for a project.

Small scripts
One repository for each type of script.

Medium and large applications
One repository for each project.

Use of standard libraries

Small scripts
Use the simplest approach that gets the work done.

Large applications

When my application grew I realized that I recreated functionality from the standard libraries; for instance from these libraries:

• argparse
• csv
• itertools

I refactored my program to use the standard library and found that it were much better than what I had written. For bigger application using standard libraries makes your code less buggy and more maintainable. So spend some time to find what has already been written.

How well does Python scale compared to compiled languages

There are mixed opinions on this topic. Scripts are generally small and large systems are generally written in compiled languages. The extra checks and rigidity you get from a compiled language is more important the bigger you applications get. If you are writing a financial application and have very low tolerance for errors this could be significant.

I am using Python for natural language processing: classification, named entity recognition, sentiment analysis and information extraction. I have to write many complex custom scripts fast.

Based on my earlier experience with writing smaller Python scripts I was concerned about writing a bigger application. I found a good setup with PyDev, unit test and source control. It gives me much of the stability I am used to in a compiled language, while I can still can do rapid development.


-Sami Badawi

Natural language processing in Clojure, Go and Cython

I work in natural language processing, programming in C# 3.5 and Python. My work includes classification, named entity recognition, sentiment analysis and information extraction. Both C# and Python are great languages, but I do have some unmet needs. I investigated if there are any new languages that would help. I only looked at minimal language that would be simple to learn. The 3 top contenders were: Clojure, Go and Cython. Both Clojure, Go have innovative approaches to non locking concurrency. This is my first impression of working with these languages.

For contrast let me start by listing the features of my current languages.

C# 3.5

C# is an advanced object orientated / functional hybrid language and programming platform:

• It is fast
• Great development environment
• You can do almost any tasks in it
• Great database support with LINQ to SQL
• Advanced web development with ASP.net
• Advanced GUI toolkit with WPF
• Good concurrency with threading library
• Good MongoDB library

Issues

• Works best on Windows
• Not well suited for rapid development

While many features of C# are not directly related to NLP they are very convenient. C# has some NLP libraries: SharpNLP is a port of OpenNLP from Java. Lucene has also been ported. The ports are behind the Java implementation, but still give a good foundation.

Python

Python is an elegant scripting language, with a strong focus on simplicity.

• NLTK is a great NLP library
• Lot of open source math and science libraries
• PyDev is a good development environment
• Good MongoDB library
• Great for rapid development

Issues

• It is interpreted and not very fast
• Problems with GIL based threading model

C# vs. Python and unmet needs

I was not sure what language I would prefer to work with. I suspected that C# would win out with all it advanced features. Due to demand for fast turnaround, I ended up doing more work in Python, and have been very happy with that choice. I have a lot of scripts that can be piped together to create new applications, with the help of the very fast and flexible MongoDB.

I do have some concerns about Python moving forward:

• Will it scale if I get really large amount of text
• Will speed improve on multi core processors
• Will it work with cloud computing
• Part of speech tagging is slow

Java

Java is a modern object oriented language. Like C# it is a programming platform:

• Has most NLP libraries: OpenNLP, Mahout, Lucene, WEKA
• It is fast
• Great development environment: Eclipse and NetBeans
• You can do almost any tasks in it
• Great database support with JDBC and Hibernate
• Many web development frameworks
• Good GUI toolkit: Swing and JavaFX
• Good concurrency with threading library

Issues

• Functional style programming is clumsy
• Working with MongoDB is clumsy
• Java code is verbose

I would not hesitate using Java for NLP, but my company is not a Java shop.

Clojure

Clojure was released in 2007. It is a right sized LISP. Not very big like Common LISP or very small like Scheme.

• Gives easy access to Java libraries: OpenNLP, Mahout, Lucene, WEKA, OpinionFinder
• Innovative non locking concurrency primitives
• Good IDEs in Eclipse and NetBeans
• Easy to work with
• Code and data is unified
• Interactive REPL
• LISP is the classic artificial intelligence language
• If you need speed you can write Java code
• Good MongoDB library

 Issues

• The IDE is not working as well as IDEs for Java or C#

Clojure is minimal in the sense that it is build on around 10 primitive programming constructs. The rest of the language is constructed with macros written in Clojure.

Once I got Clojure installed it was easy to work with and program in. Most of the good features about Python also applies to Clojure: it is minimal and has batteries included. Still I think that Python is a simpler language than Clojure.

Use case
Clojure is a good way to script the extensive Java libraries, for rapid development. It has more natural interaction with MongoDB than Java.

Clojure OpenNLP

The clojure-opennlp project is a thin Clojure wrapper around OpenNLP. It came with all the corpora used as training data for OpenNLP nicely packaged and it works well. You can script OpenNLP approximately as terse as NLTK, from an interactively repl.

I tried it in both Eclipse and NetBeans. They seem somewhat equal in number of features. I had a little better luck with the Eclipse version.

clojure-opennlp is using a Maven built system, but has a nontraditional directory layout, this caused problems for both Eclipse and NetBeans, they both took some configuration.

Eclipse Counterclockwise
The Counterclockwise instruction for labrepl mainly worked for installing clojure-opennlp.
When you were done you had to go in add the example directory the source directories under properties.

NetBeans Enclojure
I imported the project. I had to move the Clojure file from example directory to a different position to get it to work.

Maven plugins for Clojure
The standard Maven directory layout has several advantages, e.g. if you want to mix Java and Clojure code. I created my own Maven pom configuration file up, based on examples of other Clojure Maven projects. They used Clojure plugins for Maven, I could not get this to work. Eventually I ripped these plugins out and was left with very pain POM file that worked.

Go / Golang

Go was announced November 2009. It is created by Google to deal with multicore and networked machines. It feels like a mixture of Python and C. It is a very clean and minimal language.

• It is fast
• Good standard library
• Excellent support for concurrency
• It is trivial to write your own load balancer

Issues

• The Eclipse IDE is in an early stage
• Debugger is not working
• Windows port is not done and has just been released

It was hard to find the right Go Windows port, there are several Go windows port projects with no code.

Use cases
I currently have a problem when downloading a lot HTML pages and parsing them to a tree structure. This does not have the best support in C#. I found a library that translates HTML to XHTML and then I can use LINQ to process it. The library is not documented, not very fast and fails for some HTML files.

Go comes with a HTML library that parses HTML 5, it is simple to write a program with some threads that download and other that parse the files into a DOM tree structure.
I would use Golang for loading large amounts of text in a cloud computing environment.

Cython

Cython was released in July 2007. It is a static compiler to write Python extension modules in a mixture of Python and C.

Process for using Cython

• Start by writing normal Python code
• Find modules that are too slow
• Add static types
• Compile it with Cython using the setup tool
• This produces compiled modules that can be used with normal Python

Issues

• It is still more complex that normal Python code
• You need to know C to use it

I was surprised how simple it was to get it working both under Windows and Linux. I did not have to mess with make files or configure the compiles. Cython integrated well with NumPy and SciPy. This expands the programming tasks you can do with Python substantially.

Use cases
Speed up slow POS tagging.

My previous language experience

Over the years I have experimented with a long list of non mainstream languages: functional, declarative, parallel, array, dynamic and hybrid languages. Many of these were frustrating experiences. I would read about a new language and get very excited. However this would often be the chain of events:

• Download language
• Installed Cygwin
• Find out how the language's build system works
• Try to find a version of the GCC compiler that will compile it
• Get the right version of Emacs installed
• Try to get the debugger working under Emacs
• Start programming from scratch since the libraries were sparse
• Burn out

You only have so much mental capacity, and if you do not use a language you forget it. Only Scala made it into my toolbox.

Do Clojure, Go or Cython belong in your programmer's toolbox

Clojure, Go and Cython are all simple languages. They are easy to install, easy learn, they all have big standard libraries so you can be productive in them right away. This is my first impression:

• Clojure is a good way to script the extensive Java libraries, for rapid application development and for AI work.
• Go is a great language but it is still rough around the edges. There are not any big NLP libraries written for Go yet. I would not try to use it for my main NLP tasks.
• Cython was useful right away for my NLP work. It makes it possible to do fast numerical programming in Python without too much trouble.

-Sami Badawi