Saturday, April 3, 2010

Data Mining rediscovers Artificial Intelligence

Artificial intelligence started in the 1950s with very high expectations. AI did not deliver on the expectations and fell into decades long discredit. I am seeing signs that Data Mining and Business Intelligence are bringing AI into mainstream computing. This blog posting is a personal account of my long struggle to work in artificial intelligence during different trends in computer science.

In the 1980s I was studying mathematics and physics, which I really enjoyed. I was concerned about my job prospects, there are not many math or science jobs outside of academia. Artificial intelligence seemed equally interesting but more practical, and I thought that it could provide me with a living wage. Little did I know that artificial intelligence was about to become an unmentionable phrase that you should not put on your resume if you wanted a paying job.

Highlights of the history of artificial intelligence

  • In 1956 AI was founded.
  • In 1957 Frank Rosenblatt invented Perceptron, the first generation of neural networks. It was based on the way the human brain works, and provided simple solutions to some simple problems.
  • In 1958 John McCarthy invented LISP, the classic AI language. Mainstream programming languages have borrowed heavily from LISP and are only now catching up with LISP.
  • In the 1960s AI got lots of defense funding. Especially military translation software translating from Russian to English.
AI theory made quick advances and a lot was developed early on. AI techniques worked well on small problems. It was expected that AI could learn, using machine learning, and this soon would lead to human like intelligence.

This did not work out as planned. The machine translation did not work well enough to be usable. The defense funding dried up. The approaches that had worked well for small problems did not scale to bigger domains. Artificial intelligence fell out of favor in the 1970s.

AI advances in the 1980s

When I started studying AI, it was in the middle of a renaissance and I was optimistic about recent advances:
  • The discovery of new types of neural networks, after Perceptron networks had been discredited in an article by Marvin Minsky
  • Commercial expert system were thriving
  • The Japanese Fifth Generation Computer Systems project, written in the new elegant declarative Prolog language had many people in the West worried
  • Advances in probability theory Bayesian Networks / Causal Network
In order to combat this brittleness of intelligence Doug Lenat started a large scale AI project CYC in 1984. His idea was that there is no free lunch, and in order to build an intelligent system, you have to use many different types of fine tuned logical inference; and you have to hand encode it with a lot of common sense knowledge. Cycorp spent hundreds of man years building their huge ontology. Their hope was that CYC would be able to start learning on its own, after training it for some years.

AI in the 1990s

I did not loose my patience but other people did, and AI went from the technology of the future to yesterday's news. It had become a loser that you did not want to be associated with.

During the Internet bubble when venture capital founding was abundant, I was briefly involved with an AI Internet start up company. The company did not take off; its main business was emailing discount coupons out to AOL costumers. This left me disillusioned, thinking that I just have to put on a happy face when I worked on the next web application or trading system.

AI usage today

Even though AI stopped being cool, regular people are using its use it in more and more places:
  • Spam filter
  • Search engines use natural language processing
  • Biometric, face and fingerprint detection
  • OCR, check reading in ATM
  • Image processing in coffee machine detecting misaligned cups
  • Fraud detection
  • Movie and book recommendations
  • Machine translation
  • Speech understanding and generation in phone menu system

Euphemistic words for AI techniques

The rule seem to be that you can use AI techniques as long as you call it something else, e.g.:
  • Business Intelligence
  • Collective Intelligence
  • Data Mining
  • Information Retrieval
  • Machine Learning
  • Natural Language Processing
  • Predictive Analytics
  • Pattern Matching

AI is entering mainstream computing now

Recently I have seen signs that AI techniques are moving into mainstream computing:
  • I went to a presentation for SPSS statistical modeling software, and was shocked how many people now are using data mining and machine learning techniques. I was sitting next to people working in a prison, adoption agency, marketing, disease prevention NGO.
  • I started working on a data warehouse using SQL Server Analytic Services, and found that SSAS has a suite of machine learning tools.
  • Functional and declarative techniques are spreading to mainstream programming languages.

Business Intelligence compared to AI

Business Intelligence is about aggregating a company's data into an understandable format and analyzing it to provide better business decisions. BI is currently the most popular field using artificial intelligence techniques. Here are a few words about how it differs from AI:
  • BI is driven by vendors instead of academia
  • BI is centered around expensive software packages with a lot of marketing
  • The scope is limited, e.g. find good prospective customers for your products
  • Everything is living in databases or data warehouses
  • BI is data driven
  • Reporting is a very important component of BI

Getting a job in AI

I recently made a big effort to steer my career towards AI. I started an open source computer vision project, ShapeLogic and put AI back on my resume. A head hunter contacted me and asked if I had any experience in Predictive Analytics. It took me 15 minutest to convince her that Predictive Analytics and AI was close enough that she could forward my resume. I got the job, my first real AI and NLP job.

The work I am doing is not dramatically different from normal software development work. I spend less time on machine learning than on getting AJAX to work with C# ASP.NET for the web GUI; or upgrade the database ORM from ADO.NET strongly typed datasets to LINQ to SQL. However, it was very gratifying to see my program started to perform a task that had been very time consuming for the company's medical staff.

Is AI regaining respect?

No, not now. There are lots of job postings for BI and data mining but barely any for artificial intelligence. AI is still not a popular word, except in video games where AI means something different. When I worked as a games developer what was called AI was just checking if your character was close to an enemy and then the enemy would start shooting in your character's direction.

After 25 long years of waiting I am very happy to see AI techniques has finally become a commodity, and I enjoy working with it even if I have to disguise this work by whatever the buzzword of the day is.

-Sami Badawi


Utkarsh Sinha said...

Hi! I think AI has started gaining respect recently... its applications in robotics have been immense, for example, the DARPA challenge and RoboCup. These are competitions for students, but they have applications in the real world as well.

NRoy said...

Your post is really informative.
However I would be glad to hear from you more on relation between Business Intelligence and AI.
I have worked SQL server data base and Business Intelligence for almost 5 years and interested to know more about data mining especially on artificial intelligence. Currently I am doing my Masters in Marketing research. I am learning SAS, SPSS, and Lisrel. I am really unsure in which direction I should take my career so that I can explore more in Business Intelligence, AI and other model.

Sami Badawi said...

With your experience I am not sure if I can tell you anything that you do not already know.

Business intelligence tries to make sense of business data with easy drill down, charts, statistic and machine learning. While AI is a broad discipline coverings a lot of different programming topics, e.g. machine learning, which is relatively straightforward, but also much more ambitious topics.

There are many good ways to explore and work with machine learning. You should just try a few methods out and play around.

SSAS and SPSS are great tools for business intelligence. They have nicely integrated charts, statistic and machine learning packages.
If you have a lot of enterprise data in a SQL Server or Oracle databases these are good options. You can set up ETL work flows and everything will live in the same environment.
SSAS is more geared towards DBAs; and SPSS is more geared towards statisticians or business power users.
Both SSAS and SPSS are marketable skills that look good on your resume.

I tried several machine learning tools and compare them in these 2 posts:
Orange, R, RapidMiner, Statistica and WEKA

R, RapidMiner, Statistica, SSAS or WEKA

I come from a programming background and have many custom data mining and machine learning tasks in a heterogeneous environment, with data from many sources. What works best for me is a library of Python scripts that interacts with different machine learning packages: NLTK and Orange.
Other Python ML libraries: PyBrain and OpenCV 2.2.

There are more jobs in Business Intelligence than the kind of data mining I do. It sounds like you have a strong foundation. Marketing research is full of interesting BI problems. Just get your hands dirty, solve some problems and find out what you enjoy working with.

Technology fads changes so fast that it is hard to plan your career direction, I try to point myself in a good direction and roll with the punches. :-)

NRoy said...

Thanks Sami for your reply.
In my previous experience I have explored Business intelligence tool like SSAS, SSRS, and cognos.
This year I thought it will be better to get more idea on data mining. As you rightly said in your post that SSAS data mining is tightly coupled with Cube and data base.
Currently I am studying Masters in marketing analysis. Here I am using SAS 9.2, SPSS, Mat lab, Lisrel for the data analysis. As per my understanding these software (SAS, SPSS) directly interact (using Proc SQL in SAS) with ER model which may present in Oracle or Sql server database. They hardly use Cube as there data source like SSAS. I am not sure about the other usage.
I find a gap in my knowledge in OLAP or cube analysis and statistical model which are used in SAS or SPSS data analysis. Is there really any relation between these 2 types? Like Microsoft SQL server, SAS also has facility to build Cube (OLAP Cube Studio) and report. I think BI is mainly used for reporting and OLAP is for summarized multidimensional data analysis.
Here I am concluding like Business Intelligence and statistical data analysis are complementary to each other.With my previous knowledge of Business Intelligence I thought it will be better to know about data mining. However I am finding a whole new world in this.

kapil phulwani said...

is there any possibility of using artificial intelligence in data mining ? i mean what are the areas that involve this ?