EMC CIO Takes On Big Data Problems With Big Data Analytics

Every second of every day, IT generates enormous amounts of data around operational activity – system behavior, application performance, user actions, security activity, and more. Instead of viewing this data explosion as a Big Data problem, IT views it as opportunity to use Big Data solutions such as IT Operations Analytics to improve the quality of their services.

itoa

For example, 75% IT professionals surveyed recently said that they believe that IT Operations Analytics are able to transform data into relevant insights into actionable plans for improvement. I spoke with EMC CIO Vic Bhagat to describe how EMC is embracing Big Data for IT Operations Analytics to solve critical problems affecting EMC IT Operations and customers.

1.  What are the biggest problems faced by IT Operations Management at EMC and how were these problems addressed before the world of Big Data?

IT generates enormous amounts of data when monitoring complex, rapidly growing and changing IT infrastructures and the applications. The challenge for IT Operations Management is to leverage this data to build an adaptive system that is more proactive, and less reactive. The more the system can learn from the data, the better it can identify variances and problems areas in a timely manner to help IT fix issues before it negatively impacts the business such as downtime or poor performance.

In the past, we relied on traditional business intelligence and data warehousing systems to gain intelligence or insight based on historical trends. Now, with analytics, we can uncover important variables and modify them to predict an outcome. And, the more data we collect at a detailed level, the more accurate we can be.

2.  How does Big Data analytics change the game to address these problems more effectively?

It cuts down the time to gain insight. The most heavily used word after ‘selfie’ is now ‘data lake’. Everyone wants to build a data lake since it provides the right architecture and capabilities to cut down the cycle time in deriving newer, predictive insight, and then continuously integrating these results back into our business processes and decision-making. At EMC, we are moving away from data warehouses to a data lake architecture enabling us to not only gain faster insight, but also gain newer insight by bringing together and analyzing both structured and unstructured data.

For example, in a data warehouse you manage structured data such as part numbers, bay numbers, disk numbers, chassis numbers, and more. In a data lake you can manage all of this structured data in addition to unstructured data such as user manuals for each system and component. Let’s now apply this data lake solution to a use case – we continuously monitor the health of a customer’s infrastructure with our call home systems. We can now leverage a data lake with more data sets to not only make more accurate component failure predictions, but we can also provide the relevant information needed from user manuals to fix the problem in a timely manner so the customer experiences no downtime.

3.  What is EMC’s IT Operations Analytics solution leveraging Big Data technologies and techniques?

We are leveraging the entire Pivotal Big Data Suite to ingest and store all of the structured and unstructured data – Pivotal Gemfire XD, Pivotal HD, Pivotal HAWQ, and Pivotal Greenplum Database. Our Data Scientists are then able to apply advanced analytic techniques to the data they need using their choice of tools which are MadLib, R, and Python. This Big Data environment will be part of a wider business data lake strategy, where all enterprise data will be managed, accessed, and used equally by all business applications, not just IT Operations. Only a few legacy or specialized applications will standalone.

4. What benefits has EMC gained from this Big Data solution?

The benefits are enormous and can be extracted from both business and technical benefits. Building predictive models and predicting imminent system failure reduces downtime and the number of alerts and enables us to identify the real issues faster, reducing the cycle for decision making and taking corrective action. This improves our performance, productivity and value we gain from Big Data.

But we are only scratching the surface. The more we can optimize our Big Data environment so that it is elastic and accessible, the faster and more precise Data Scientists will be in solving problems. For example, we can now predict MS Exchange outages two hours in advance.

5. One of the biggest barriers to getting value from Big Data is the skills shortage. How does EMC IT Operations address this issue?

EMC had the foresight to build Centers of Excellence (COE) around the globe, producing the expertise and skills needed to transition into the realm of Data Science. We are fortunate to leverage talent within the company, but also leverage the COE to attract and acquire new Data Science talent outside the company.

6. What books are you currently reading on your Kindle or if you are still paper based like me, what books are stacked on your nightstand?

I’m Kindle based, so I read periodicals such as Techmeme and Engadget. Since we are a company that is data and digital driven, I am reading a book called ‘Leading Digital’. I want help lead this digital revolution at EMC and this book provides great examples of how digital makes significant changes in how a company operates and kills bureaucracy.

Revolution Analytics Boosts the Adoption of R in the Enterprise

The path to competitive advantage is being able to make predictions from Big Data. Therefore, the more you can build predictive analytics into your business processes, the more successful your organization will become. There is no doubt that open-source R is the programming language of choice for predictive analytics, and thanks to Revolution Analytics, R has the enterprise capabilities needed to drive adoption across the organization and for every employee to make data-driven decisions.

Revolution Analytics is to R what the vendor RedHat is to the Linux operating system—a company devoted to enhancing and supporting open-source software for enterprise deployments. For example, Revolution Analytics recently released R Enterprise 7 to meet the performance demands of Big Data whereby R now runs natively within Hadoop and data warehouses. I spoke with David Smith, VP of Marketing at Revolution Analytics to explain how Revolution Analytics has accelerated the adoption of R in the enterprise.

1.  What benefits do Revolution Analytics provide to organizations over just using open-source R?

Continue reading

Alpine Data Labs – Making Predictive Analytics Pervasive and Persuasive

Big Data has exposed the need for deeper data insights through predictive analytic techniques such as data mining, machine learning, and modeling. The interesting thing to note is that predictive analytics has been around for a long time, used by a select few, in select organizations. Its value has always been recognized and applauded, but its true potential never fully realized due to lack of widespread adoption, as well as issues around data accessibility, performance, statistical expertise, business sponsorship, cost, and more. In fact, nearly 90 percent of organizations that do employ predictive analytic software agree that it has given them a competitive advantage, according to a new survey.

The advent of Big Data has driven the uptake of predictive analytics due to the curiosity of very capable Data Scientists, along with new tools and technologies from companies such as Alpine Data Labs.  Alpine Data Labs provides next generation predictive analytics to address legacy issues and meet the new demands of Big Data. But more importantly, Alpine Data Labs is mainstream-oriented whereby business users, not just statisticians and Data Scientists, are compelled to mine data.

collaborate-alpine-data

Backed by $16M in Series B funding, Alpine Data Labs is getting some serious momentum in the Big Data analytics startup space, offering zero coding for creating and deploying complex predictive models on Hadoop. I spoke with Alpine Data Labs CEO Joe Otto to talk about their game changing approach to predictive analytics for Big Data.

1.  Lets first talk about leading predictive analytics incumbents such as SAS, IBM SPSS, and other analytics vendors who got their start years ago with desktop and server software designed for data mining and advanced analytics. How has Alpine Data Labs overcome the issues around these incumbent technologies and address the new needs of Big Data?

Continue reading

Big Data Brings Sales and Marketing Closer Together

Over 50% of the Big Data business opportunity comes from better understanding your customers.  As a result, organizations with Sales and Marketing departments are finally aligning together by simply aligning around high value customers.  EMC is a great example of Sales and Marketing teams ending the turf war, as both departments are working together to create a centralized customer analytics database to better identify customer segments for upsell/cross sell opportunities, target prospects who are more likely to bring in more value, and optimize operations.  This Big Data transformation now enables Sales and Marketing to work off of the same customer account data to harmoniously build the EMC brand and business.

I became very interested in documenting the success of Big Data here at EMC so I captured one of the use cases around optimizing operations for EMC Maintenance and Renewals.  Click here for this newly published Big Data success story that details how EMC gained an incremental $113M above revenue goal from a Big Data strategy that involved the right people, processes, and technology.

blog image