I recently attended a conference at which representatives of a large number of Fortune 500 companies expressed great enthusiasm in launching big data projects. That's not surprising, as big data is the most promising and talked-about wave to sweep IT in some time.
What did surprise me was that many of these companies didn't seem certain how to get started nor, for that matter, why they should get started. There was a sense that because big data is getting big attention everyone should be doing it. That may not be the case.
Wading into a complex Hadoop, or big data, project without a clear understanding of what's at stake and without a clear set of objectives is risky at best. These projects are costly and carry serious implications for both the IT organization and the business as a whole.
I'm reminded of the old saying: Measure twice, cut once. In other words, plan carefully before you execute, because it can be difficult and costly to make corrections afterward.
Developing a big data roadmap before you begin your journey can save you from cutting before you're ready. As you prepare the roadmap, you may find that you don't need Hadoop or big data, that your existing technology base will take you where you want to go.
Hadoop is a big data software framework that enables organizations to store extremely large files. It also allows them to store large numbers of files.
At CapTech, we've helped businesses in many industries interested in Hadoop develop big data roadmaps. We carefully assess the current state, conduct interviews about the target state and existing technology capabilities, evaluate the business case and the use case, and help clients determine how to get from where they are to where they want to go, consistent with the broader business strategy.
The roadmap outlines the tactics that will be used in carrying out the business strategy. For example, it defines specific projects that the IT organization will undertake over a given period of time - say, one or two years - and it helps both the business and IT organization understand the implications for staffing, security, data governance and data management, key considerations that are often overlooked as organizations head into big data projects.
One problem we've been seeing is that organizations implement multiple projects using Hadoop, but don't look at how these projects might affect one another or the organization as a whole nor at whether they serve the same use case.
Another concern is that Hadoop projects are being kicked off without a recognition that Hadoop experts are hard to come by. Given your objectives, how many experts will you need in a year or two? Will they be available?
In some cases, data governance isn't being addressed until after projects are under way. Data governance deals with such issues as determining the origin, or lineage, of data; who is accessing data; and what they are doing with it. That's of vital importance to financial institutions and many other organizations that are interested in leveraging big data without creating security and data management issues. Yet data governance is difficult to establish in Hadoop, although many software organizations are now working toward solutions.
As you develop a roadmap, objectivity is key. Being committed to a particular approach or a particular vendor can blind you to important issues likely to arise on the road ahead. An advantage of using CapTech to develop a big data roadmap is that we not only have extensive experience in this area, but we also are independent. We don't favor particular products or vendors, so we can take an unbiased look at your organization's current state and its use case and business case. That can save you from investing, or continuing to invest, in an approach that isn't consistent with your objectives.
Even if you've already started down the road on a big data project, it's not too late to develop a roadmap to see you through the coming months and years. The key is to be thoughtful about whether and how big data will support your business objectives.
About the Author: Les Morelock is a Managing Director with CapTech in the Charlotte, NC office. He leads business development activity for the Charlotte market with a focus on delivering unique business and technology solutions to key banking and financial institutions and Fortune 500 companies.