A Pivotal Piece Of Big Data

Big Data can only make a positive impact if it’s transparent to a business process or practical for the end user.  A Big Data Application provides this usability, as it connects that last dot of a Big Data journey – from the back-end data/analytics layer to the front-end application layer.

A Big Data Application can be as simple as a mobile application that queries nearby restaurants with real time table availability or more complex such embedding real time targeted offers within a call center web application. Regardless of the use case, EMC is positioning its recent acquisition of Pivotal Labs to quickly deliver these world-class applications. With the addition of the Pivotal Labs, EMC now delivers a complete Big Data offering –industry leading data platform to scale-out, agile analytics to transform data into insight, and gold standard application development expertise to make Big Data actionable and transform the business.

I decided to break up my morning routine and take a detour to the Pivotal Labs office in San Francisco. I met with Parker Thompson, Director of Business Development at Pivotal Labs, to gain some insight into this well-oiled machine. Parker was kind enough to show me around the office and point out all the exciting customer projects each development team was working on (my future blog posts will feature some of these customers!).

A key aspect of Pivotal’s development methodology is collaboration whereby developers are paired up for code development and use social features within their agile development software called Pivotal Tracker.  But what impressed me most about their methodology was the 9am “stand-up” – a 5 to 10 minute daily meeting where developers are updated with new issues and best practices.  The scene reminded me of Star Trek where everyone attentively stands in front of a large screen displaying a visualization that I couldn’t even begin to decipher.

Here are the highlights of my discussion with Parker Thompson:

Continue reading

Dreaming of Building a 1000 node Hadoop Cluster?

The dream is real for EMC Greenplum as a 1000-node Hadoop Cluster, a.k.a Analytics Workbench, went live May 22, 2012 during EMC World.  When I first heard about this large-scale Analytics Workbench project, I immediately thought how harmful it must be for the environment.  What is the point of creating the world’s largest environment for Hadoop testing and development?  Well the joke is on me because this Big Data platform will facilitate ground-breaking insight to improve the quality of life AND create a greener environment.

I wanted to speak to the person who led the effort in creating this Big Data masterpiece –  Apurva Desai, Sr. Director of Hadoop Engineering at EMC Greenplum.  Apurva worked with our internal systems integration team, led by Gerg Robidoux, to architect the cluster and managed a team to build and test the cluster. It has been rumored that Apurva’s blood is on the cluster since there were many cuts and scrapes suffered while putting the cluster together.  Watch stop motion video clip of the Analytics Workbench being built by Apurva’s team.

Creating a 1000 node Apache Hadoop cluster seems like a dubious task. Why did you take on the project?

Continue reading