Tell me if this story sounds familiar. You have an idea, a hunch, a theory that you'd like to 'flush' out. BUT, in order to do so, you need access to data from last month, maybe it's CDRs, billing data, smart meter data, etc. The emails, conference calls, and requests you'll have to make to get your hands on that data are daunting at best. But it's your data right? Why should you have to jump through so many hoops to get access to it? By the time you can even figure out who "owns" the data, you're onto the next task and your great idea is wasted, yet again.
Over the years, I can't tell you how many times the 'data access' issue has delayed or even completely stopped a project we've been involved with. Not because the idea didn't make sense or the business case didn't prove out, simply because getting the data from 'Point A' to 'Point B' was going to take too much time and effort. This is the definition of a 'data plumbing' issue. Big data tools are getting better, faster, cheaper, and more available every day. But the challenge of extracting and integrating data from a variety of sources has become an issue that organizations simply can't ignore. It's the ugly truth behind data analytics - it often takes more time and energy to extract, clean, and integrate the data than it takes to do the analytics itself.