Haskell and Hadoop the Aftermath

In 2012 Haskell and Hadoop were the hottest technologies. They had a lot of hype and I loved them. Both were based on functional programming and built on towering abstractions.

Elite functional programmers used Haskell. Serious tech startups had to use big data, meaning Hadoop. Three years later I had learned Haskell and Hadoop and my top advice to startups was:

Don't use Haskell or Hadoop!

They won't you give you a competitive advantage they will just slow you down.

That was my personal experience. For years after that I avoided jobs involving Hadoop, but for the last couple of years I have mainly been working in Hadoop with Spark. It's now solid and very productive.

I found the productivity increase quite remarkable. Some of it is a textbook example of technology life-cycle, but a some of it comes down to understanding the power and limitation of functional programming.

Modern Programming Paradigms

There are three main modern programming paradigms:

• Object oriented
• Functional
• Declarative

Object oriented programming gives you fine-grained control. Functional programming uses transformations with less control. In declarative programming you just write queries and you have little control. The higher the abstraction the less control.

Essential Hadoop

The breakthrough that Hadoop / MapReduce made was that by using functional programming transformation you could distribute a computation over thousands of computers in a fault tolerant way. This was a monumental achievement, but what made Hadoop the dominant data platform it is today was that it later combined functional with the declarative programming available in Spark SQL, HIVE or PIG.

Combined functional and declarative programming was once the holy grail in computing, but nobody knew how to do it. Today it is ubiquitous and it is free, until you get the bill from your cloud provider.

Essential Haskell

I expected Haskell to be a mathematical version of Python. It was not. If you are trying to do object oriented programming in Haskell it will cause you a lot of pain. Unlike doing OOP in hybrid languages like F#, OCaml or Scala.

The power of Haskell is that it limits you to a small set of basic operations that compose. This allows you to build a big machine out of simple parts. The lazy evaluation makes it natural to work on infinite streams of data. The powerful type system makes it possible to connect small pieces of code in many different dimensions. My metaphor is:

Haskell is an extra dimensional Lego set

Haskell started as a playground for language researchers experimenting. I wanted to play with all these shiny theoretical toys. That was a big time sink and a part of the reason it took me a long time to learn.

Common Problems

One reason I gave up on both Haskell and Hadoop was that it was hard to get things done. Both were beautiful abstractions built on a tower of unstable software libraries. Everything was evolving quickly. This made it hard to keep the libraries underneath on compatible version. Every time your Hadoop distribution was updated your code would break.

In Haskell this problem was called Cabal Hell after the build system Cabal. There were simple solutions. Haskell now has stable versions of libraries that work with each other. It has a modern build system called Stack. Now tooling in both Haskell and Hadoop is quite good.

The Aftermath

I spent more time and effort learning Haskell and Hadoop than any other technologies. With that much effort I expected them to give me superpowers. Instead they slowed me down. This caused a backlash. I felt naive for jumping on the Haskell and Hadoop bandwagon and wasting so much time.

Now 8 years later the dust has settled and part of my problem was that I was an early adapter of immature technologies. Haskell and Hadoop are now mature but inherently complex technologies. They draw their power from giving up fine control. Instead they let you build machines that you can pipe data through.

Big data in the case of Hadoop. Infinite data in the case of Haskell.

Hadoop is highly successful, and is now a cornerstone of data engineering. Even though it is currently standing in the shadow of Spark that was built to run on top of Hadoop infrastructure.

Haskell is a practical programming language well suited for constructive mathematics and category theory, but it is not a better version of Scala. It is pretty successful at number 19 on RedMonk programming language ranking and is used in industry.

F# vs Scala

F# and Scala are both hybrid functional object-oriented languages created for popular virtual machines.

• F# for CLR / .NET
• Scala for JVM / Java

F# and Scala are now in more direct competition after Microsoft open sourced F# and .NET Core. They have many similarities but a distinctly different feel. It was hard for me to put my finger on the difference. This blog post investigates their design decisions and use cases. Starting with a brief overview of F# and Scala.

F# (F Sharp)

F# is a mature, open source, cross-platform, functional-first programming language. It was created by Don Syme in 2005 as a port of the OCaml language to .NET.

• Core: Strict, strong, inferred, hybrid
• Popularity: Some use in industry and backed by Microsoft
• Complexity: Easy to learn, but part of a big ecosystem
• Maturity: It is 14 years old and part of the .NET, so quite mature
• Tooling: Very good, both .NET based and F# specialized
• Cross platform: with Mono and .NET Core and JavaScript
• IDE: Visual Studio, VS Code

Scala

Scala combines object-oriented and functional programming in one concise, high-level language. It was created by Martin Odersky in 2004.

• Core: Strict and lazy, nominal and structural, hybrid, implicit for IoC
• Popularity: Very popular. No 13 on Red Monk June 2019 list. Spark is written in Scala
• Complexity: It is a quite complex language, but it is easy to get started with
• Maturity: Very stable. Run on JVM, well integrated with JVM ecosystem
• Tooling: Great build tool and package managers
• Cross platform: JVM and JS. Also early work on native / LLVM version
• IDE: IntelliJ, VS Code, Eclipse

Microsoft Open Source Bet

Choosing between F# and Scala used to be pretty easy. If you were doing Windows development you would use F# if you were on an open source stack you would choose Scala.

In 2012 Microsoft open sourced F# and started porting it to Mono a Microsoft supported cross platform version of the CLR. That was cool but not something I would run production code on.

However, in September 2019 Microsoft released .NET Core 3 an open source cross platform version of a big part of their SDK and also a first release of Apache Spark for .NET.

After this .NET and F# are serious contenders for being parts of an open source stack.

Relation to Java and C#

You might think that F# is just the .NET version of Scala and moving from Java to Scala is similar to moving from C# to F#. This is not the case.

Java was a small and simple language with a lot of innovations but some annoying problems. A big part of Scala appeal was that it was a better Java with more features.

C# was also made to be a better Java. It fixed some of the flaws in the original Java, e.g. auto boxing of integers, generics and lambdas. C# is a great but also very big language. F# is more like a leaner version of C#, with less features.

Collection Libraries

Scala has made a big effort to make a full set of immutable and mutable Scala collections and make different Java collections look like they are native Scala collections.

I tried Scala in 2007 it had generic and could use Java generic, but either you were programming in Java or in Scala. This took a long time to get this right and cost was that the standard library code was very complicated. This is not really a problem for the user who won't see this.

Generally, F# is using a few collections: Arrays, lists, seq, set and map.

It is a bit messy to bridge the OCaml and the C# heritage, especially map / dictionaries are clumsy.

Monads

A monad is an important part of functional programming. It is a general principle to express a sequence of operations and work on a lot of different data types:
List, Seq, Future, Option.

Scala monadic for comprehension

Scala's version is syntactic sugar over flatMap(). It is more flexible, it can mix two types of monads say List and Option. Scala's monad will also return the same type as the input type.

F# monadic for comprehension

F#'s version of the monad computation is called computational expression. It has more features than Scala's.

Classes

Classes are considered an anti-pattern by functional programming purist. Some problems with classes are:

• A class maintains state
• A class creates a custom language instead of reuse of operations
• Inheritance is crating tight coupling

Scala has a very sophisticated type and class system and classes are central part of Scala.

F# has support for classes, but it is billing itself as object programming not object-oriented programming. It is made to use classes defined in C#, but will often define objects with methods without a full class definition.

I like that F# is exploring more lightweight alternatives but classes are easy to create and feels natural to use.

Type Classes, IoC, DI, Type Providers

Type class is a powerful abstraction that can make a third party class implement an interface. It plays an important role in Scala and are implemented with helper classes created by implicit.

Inversion of control and dependency injection, are first class in Scala with implicit. This is an advanced but very useful feature of Scala.

Scala has developed these ideas to the point where you can do logic style programming with implicit. A lot of the more sophisticated category theory like programming is based on this.

You can do inversion of control and dependency injection in F# using libraries.

F# has type providers that on the fly generated typed access to a lot of different data sources, e.g. a table on a webpage.

Design Decisions

F# is white space indentation-based language. Scala is a curly bracket language.

F# is a lightweight language with strong compose-ability.

Scala has sophisticated type system, including type classes, this unifies a lot of different classes and facilitating reuse.

F# program feel a little more like a loose collection of definitions while Scala program feels more like a carefully packaged system.

Conclusion

F# and Scala have a lot in common. To a large extent you would still chose F# or Scala based on your platform choice.

Both languages are very well suited for building back end programs that can interact with a universe of libraries written in C# or Java.

Scala has more momentum and a better niche. It is still having status as a better Java. Even after Java added some of the best constructs from Scala. Spark has made Scala a corner stone of data engineering.

F# is more lightweight than Scala. This makes it great for data exploration and great for building small scripts. It still remains to be seen how well supported Spark is going to be for .NET.

From a language evolution perspective, the object functional hybrid has been very successful. F# and Scala's different emphasis has produced different language from similar goals. I am very happy that we now can compare their design decisions on merit not just compare .NET and Java ecosystems.

This article is an elaboration on my last blog post Typed Functional Languages 2019.
Disclaimer I have been a happy Scala user for years, and only occasionally use F#.

ML and Data in AWS, Azure and GCP

Machine learning and data technology are changing fast and the big cloud providers compete with new offerings. This blog is a short introduction to what this looks like in 2019. It is focused on cloud providers Amazon Web Services, Microsoft Azure and Google Compute Platform.

A few things I will discuss -

• Most data in an organization can be put into a data lake to query and combine
• We now have very powerful, user friendly open source ML libraries
• We have NLP and computer vision REST APIs from cloud providers

Let me start with a little history of both ML and data.

History of Machine Learning Libraries

Simplified timeline for languages, libs and APIs

• 1960 Lisp since ML was a small part of A.I.
• 1986 C++ neural network software on a floppy disk in the back of book
• 1997 Open source Java ML like WEKA, good but hard to integrate with you data and code
• 2010 Modern Python open source libs NumPy, Pandas, Scikit-learn easy to use and integrate
• 2015 Spark ML, attempts to make a fast ML pipeline as easy to use as Scikit-learn
• 2017 Deeplearning open source libraries Tensorflow, Keras and PyTorch 
• 2017 Cloud Vision API and Natural Language API

We now have several strong contenders to build or buy production quality ML functionality.

Convergence of Data

Recently I talked with a DBA and was surprised how much the DBA profession has changed. He told me big organizations used to have a big database such as Oracle, SQL Server, Sybase or DB2 and a lot of data stored in different files.

Now maintaining the data lake is one of his main responsibilities. The data lake is a system that allows you to store log files, structured, semi structured and unstructured data files in cheap cloud blob storage and still query and join it with SQL.
He was also in charge of an Oracle database and a few open source databases running, MySQL and Postgres and a MongoDB.

Data Lake Fundamentals

Uniform data that can be joined is very powerful. Here are a few underlying technologies that makes this possible.

In 2004 Google released the famous MapReduce paper, describing how you can do distributed computation using functional programming operations. The idea is that you send your computation to were you data is.

In 2010 Hadoop was released. Hadoop is an open source Java implementation of MapReduce. It turned out of be very hard to program in. Two new technologies made it easier to program MapReduce: Hive and Spark.

Hive

A lot of MapReduce job was just queries on data. Hive is a tool that lets you write these queries as simple SQL. Hive will translate the SQL to a MapReduce job, all you had to do was to add schemas definition describing the files with your data.

Spark

With Spark you can write more complicated MapReduce jobs. Spark is written in Scala which is a natural language to write MapReduce in. Spark is often use to ingest data into the data lake.

All the cloud providers have great support for Spark, AWS has EMR, Azure has HDInsight and GCP has Dataproc.

Combining Data Lake and Normal Database

Combine a data lake with a RDBMS is not easy. There are several approaches.

You can copy over all your relational data to your data lake every day. It takes work to build and operate, but when it is working everything is unified and it is easy to do any kind of analytic queries. Some data lake products have specialized functionality to do this in an easier way, see below.

Data Lake on AWS, Azure and GCP

AWS, Azure and GCP have different data lake solution.

AWS Redshift and Redshift Spectrum

AWS Redshift is a proprietary columnar database build on Postgres 8.
Redshift Spectrum is a query engine that can read files from S3 in these formats: avro, csv, json, parquet, orc and txt and treat them as database tables. First you have to make a Hive table definition in Glue Data Catalog.

Azure Data Lake Store

Microsoft data lake is called Azure Data Lake Storage works with blob storage and is compliant with HDFS the Hadoop distributed file system.

U-SQL is a query tool to combine Azure SQL DB and your data lake.

Google BigQuery

GCP's data lake is called BigQuery works with blob storage and stores native data in proprietary columnar format called Capacitor.
BigQuery is very fast and has a nice web GUI for SQL queries. It is very easy to get started with, since it can do schema auto-detection of your blob data, unlike Hive that needs a table definition before it can process the data.

New Cloud ML APIs

In 2017 Google released their Cloud Vision API and Natural Language API. I heard from several data scientists that instead of building their own computer vision system, named entity or sentiment analysis system, they just use APIs.

It feels like cheating, but ML APIs are here to stay.

When you should build your own ML models and when you you use the APIs?
If you have a hard problem in computer vision or NLP that is not essential to your goal, then using API seems like a good idea. Here are a few reasons why it can be problematic:

• It is not free
• Sometimes it works badly
• There are privacy and compliance issues
• Are you helping train a model that your competitor is going to use next
• Speed e.g. if you are doing live computer vision

Working with ML APIs

If you decide to use the ML API your job will be quite different than if you chose to build and train your own models. Your challenges will be:

• Transparency of data
• Evolution or your data sources
• Transparency of ML models
• ML model evolution
• QA of ML models
• Interaction between ML models

The 2014 book Linked Data is a great source of techniques to use for data transparency and evolution. It describes linked data as transparent data with enough meta data that it can be linked from other data sources. It is advocating using self describing data technologies like RDF and SPARQL.

The response to a Cloud Vision query is nested and complex. I think that schemas or a gradual type system, similar to TypeScript's could give stability when working with semi structured evolving data. Some of the Google's Node API wrappers are already written in TypeScript and so they already have the type definitions.

Cloud ML Developments

There are a few minor cloud ML developments that deserve a mention.

Cloud Jupyter Notebooks

Amazon SageMaker, Microsoft Azure Notebooks and Google Cloud Datalab are Jupyter notebooks directly integrated into the cloud offerings.

I find Jupyter notebooks a natural place to combine code, data and presentation. One problem I have had when programming on cloud is that there are so many places where you can put programming logic.

Model Deployment

Model deployment has traditionally received less attention than other part of the ML pipeline.  Azure and GCP have done a great job of optimizing model deployment into something that can be done in few line of code. It will train a model, save it in a bucket and spin up a serverless function that serves up the model as a REST call.

Auto ML

ML tools that help find best ML models there are now available for GCP, AutoML, Amazon SageMaker and Azure, Automated Machine Learning. These will help you to chose the best model and tune hyper parameters. This seems like a natural expansion of current ML techniques. It does involve using cloud specific libraries.

Transfer Learning

If you have an image categorization task, you could build a classifier from scratch by training a deep convolutional neural network. This can take a long time. With transfer learning you will start with a trained CNN for example Inception or ResNet network. It should be trained on data that is similar to the data that you will be processing.
You train your classifier model by taking the second to last layer in the trained CNN as input. This is much less work than staring to build a 100 layered CNN from scratch. While transfer learning is not specific to the cloud it is easy to do it on the cloud where you have easy access to the per-trained models.

AWS vs Azure vs GCP

The cloud service market is projected to be worth $200 billion in 2019. There is a healthy competition despite AWS head start. Let me end with a very brief general comparison.

AWS was the first cloud service. It started in 2006 and has biggest market share. It is very mature offering both Linux and Windows VMs. They continue to innovate, but the number of services they have are a little overwhelming.

Azure is a very slick experience. Microsoft has embraced open source, offering both Linux and Windows VMs. It has great integration with the Microsoft and Windows ecosystem: SQL Server, .net, C#, F#, Office 365 and SharePoint.

Google Cloud Platform is polished. Is easy to get started with BigQuery and do data exploration in it. GCP has hosted Apache Airflow workflow system. GCP shines with machine learning offering great ML, vision and NLP APIs.

Hive, Pig, Scalding, Scoobi, Scrunch and Spark

Comparison of Hadoop Frameworks

I had to do simple processing of log files in a Hadoop cluster. Writing Hadoop MapReduce classes in Java is the assembly code of Big Data. There are several high level Hadoop frameworks that make Hadoop programming easier. Here is the list of Hadoop frameworks I tried:

• Pig
• Scalding
• Scoobi
• Hive
• Spark
• Scrunch
• Cascalog

The task was to read log files join with other data do some statistics on arrays of doubles. Programming this without Hadoop is simple, but caused me some grief with Hadoop.

This blog post is not a full review, but my first impression of these Hadoop frameworks.

Pig

http://pig.apache.org/

Created by Yahoo!
Language Pig Latin.

Pig is a data flow language / ETL system. It work at a much higher level than direct Hadoop in Java.
You are working with named tuples. It is mildly typed, meaning you can define a type for each field in a tuple or it will default to byte array.

• Pig is well documented
• Pig scripts are concise
• You can use it for both script and interactive sessions
• You can make Pig scripts embedded in Python run as Jython
• Pig is a popular part of the Hadoop ecosystem

Issues

• Pig Latin is a new language to learn
• Pig Latin is not a full programming language, only a data flow language
• You have to write User Defined Function / UDF in Java if you want to do something that the language does not support directly
• Pig Latin does not feel uniform

Scalding

https://github.com/twitter/scalding

Created by Twitter.
Language Scala.

Scalding looks very promising. It had just been open sourced when I looked at it.

What sets Scalding apart from other Scala based frameworks is that you work with tuples with named fields.

There is a blog with code example:
http://blog.echen.me/2012/02/09/movie-recommendations-and-more-via-mapreduce-and-scalding/
It contains numerical code for Hadoop that did not look much harder than the equivalent non Hadoop code.

Issues

You call Scalding by running a Ruby build script, scald.rb.

This did not work on my Mac, OS X Lion, but it ran under Ubuntu with no problems.

Scalding has little documentation, but it is built on Cascading that does have good documentation.

Note on aggregate functions in Scalding

Scalding has some predefined aggregate function such as Sum and Count. Unfortunately Sum only works numerical types and I needed it to work on arrays of doubles.

You can build your own aggregator function using Group Builder followed by a scanLeft or foldLeft operation.

Workaround to get Scalding to run on OS X Lion

After spending some time I found a workaround for the problems with running Scalding under OS X Lion:


In the build scripts/scald.rb set:

COMPILE_CMD="scalac"

In build.sbt set scalaVersion to the version of Scala you are using:

scalaVersion := "2.9.1"

After that I was able to run the 5 tutorials, coming with Scalding.

Scoobi

https://github.com/NICTA/scoobi
http://www.opennicta.com/home/scoobi

Created by OpenNICTA.
Language Scala.

Scoobi looked powerful and simple.


Scoobi was easy to build with the Scala build system SBT.

Issues

I was having problem running examples. Turned out that Scoobi had a dependency of the Cloudera Hadoop distributions version 0.20.2. This is a popular Hadoop distribution.

I could not get it to run on my Mac, which has the Apache Hadoop distribution so I gave up. I have not revisited it yet.

Hive

http://hive.apache.org/

Created by Facebook.
Language SQL.

Hive works on tables made of named tuples with types. It does not check the type at write time, you just copy files into the directory that represent a table. Writing to Hive is very fast, but it does check types at read time.

You can run JDBC against Hive.

It was easy to get Hive running and I really liked it.

Issues

A problem was that Hive only understands a few file format:

• Text format
• Tab delimited format
• Hadoop SequenceFile format

Starting from Hive version 0.9, is has support for Avro file format that can be used from different languages.

In order to do Sum by group I would have to create User Defined Aggregation Function. Turns out that UDF and UDAF is badly documented. I did not find any examples about how to write them for arrays of doubles.

Spark

http://www.spark-project.org/

Created by Matei Zaharia from UC Berkeley AMP Lab.
Language Scala.

I was very impressed by Spark. It was easy to build with SBT. It was very simple to write my program. It was trivial to define group by sum for double array, just by defining a vector class with addition.

Issues

I tested my program in local mode. I was very happy that I had a workable solution. Then I investigated moving it to a Hadoop cluster. For this Spark had dependency on Apache Mesos cluster manager. Mesos is a thin virtual layer that Hadoop is running on top of.

It turned out that Spark is not actually running on Hadoop. It is running on HDFS and is an alternative to Hadoop. Spark can run side by side with Hadoop if you have Apache Mesos installed.

Spark is an interesting framework that can outperform Hadoop for certain calculation. It uses the same code from running in memory calculation and code on a big HDSF cluster.

If you have full control over your cluster Spark could be a good option, but if you have to run on an established Hadoop cluster it is very invasive.

Scrunch

https://github.com/cloudera/crunch/tree/master/scrunch

Created by Cloudera.
Language Scala.


Scrunch looked powerful and simple. It is easy to build with SBT.

Issues

Dependent on Cloudera's Hadoop 0.20.2 distribution.

The web site describes it as alpha quality, so I did not do much with Scrunch.

Cascalog

https://github.com/nathanmarz/cascalog

Created by Nathan Marz from Twitter.
Language Clojure, a modern Lisp dialect.

As Scalding Cascalog is built on Cascading.

Easy to build with the Clojure build system Leiningen.

It is used as a replacement for Pig. You can run it from the Clojure REPL or run scripts, and get a full and consistent language.

I tried Cascalog and was impressed. It is a good option if you are working in Clojure

Hadoop vs. Storm

I had to solve the same log file statistics problem in real-time using the Storm framework and this was much simpler.

Why is Hadoop so hard

• Hadoop is solving a hard problem
• Hadoop is a big software stack with a lot of dependencies
• Libraries only work with specific versions of Hadoop
• Serialization is adding complexity, see next section
• The technology is still not mature

Looks like some of these problems are getting addressed. Hadoop should be more stable now that Hadoop 1.0 has been released.

Serialization in Hadoop

Java have a built in serialization format, but it is not memory efficient. Serialization in Hadoop has to be: 

• Memory efficient
• Use compression
• Support self-healing
• Support splitting a file in several parts

Hadoop SequenceFile format has these properties, but unfortunately it does not speak so well with the rest of the world.

Serialization is adding complexity to Hadoop. One reason Storm is simpler is that it just uses Kryo Java serialization to send objects over the wire.

Apache Avro is a newer file format that does everything that is needed by Hadoop but is speaks well with other languages as well. Avro is supported in Pig 0.9 and should be in Hive 0.9.

High level Hadoop frameworks in Java

You do not have to use Scala or Clojure to do high level Hadoop in Java. Cascading and Crunch are two Java based high level Hadoop frameworks. They are both based on the idea is that you set up a Hadoop data flow with pipes.

Functional constructs are clumpy in Java. It is a nuisance but doable. When you deploy code to a Hadoop cluster you have to pack up all your library dependencies into on super jar. When you are using Scala or Clojure you need to also package the whole language into this super jar. This also adds complexity. So using Java is a perfectly reasonable choice.

Here are two high level Java Hadoop libraries:

Cascading

http://www.cascading.org/

Both Scalding and Cascalog is built on top of Cascading.

Cascading is well documented. You can write concise Java code for Hadoop.

Crunch

https://github.com/cloudera/crunch

Scrunch is built on top of Crunch.

Conclusion

I liked all of the Hadoop frameworks I tried, but there is a learning curve and I found problems with all of them.

Extract Transform Load

For ETL Hive and Pig are my top picks. They are easy to use, well supported, and part of the Hadoop ecosystem. It is simple to integrate a prebuilt Map Reduce classes in data flow in both. It is trivial to join data source. This is hard to do in plain Hadoop.

Cascalog is serious contender for ETL if you like Lisp / Clojure.

Hive vs. Pig

I prefer Hive. It is based on SQL. You can use your database intuition and you can access it though JDBC.

Scala based Hadoop frameworks

They all made Hadoop programming look remarkable close to normal Scala programming.

For programming Hadoop Scalding is my top pick since I like the named fields. 

Both Scrunch and Scoobi are simple and powerful Scala based Hadoop frameworks. They require Cloudera's Hadoop distribution, which is a very popular distribution.