Industry Leader in Telecom Data Analytics and Revenue Assurance

Tell me if this story sounds familiar. You have an idea, a hunch, a theory that you'd like to 'flush' out. BUT, in order to do so, you need access to data from last month, maybe it's CDRs, billing data, smart meter data, etc. The emails, conference calls, and requests you'll have to make to get your hands on that data are daunting at best. But it's your data right? Why should you have to jump through so many hoops to get access to it? By the time you can even figure out who "owns" the data, you're onto the next task and your great idea is wasted, yet again.

Over the years, I can't tell you how many times the 'data access' issue has delayed or even completely stopped a project we've been involved with. Not because the idea didn't make sense or the business case didn't prove out, simply because getting the data from 'Point A' to 'Point B' was going to take too much time and effort. This is the definition of a 'data plumbing' issue. Big data tools are getting better, faster, cheaper, and more available every day. But the challenge of extracting and integrating data from a variety of sources has become an issue that organizations simply can't ignore. It's the ugly truth behind data analytics - it often takes more time and energy to extract, clean, and integrate the data than it takes to do the analytics itself.

The title is somewhat misleading in that I am sure there are many user tasks that go beyond the scope of Big Data and Cloud Computing to satisfy.  However, being a dinosaur-type with over fifty years of professional experience in the telecom environment, I am continually amazed by the capabilities and associated minimal costs with implementation of systems and processes that were only dreamed about when I started in the telecom business.

If you’re reading this, you’re probably the kind of person who’s attended a Big Data conference, and maybe find yourself sitting in on those “How to scale to your first x million users”.  The problem is that in the real world, the problem isn’t so much scaling up from a prototype into production, but having enough horsepower to wrangle data in the conceptual phase, then scale down into a business application (then scale up again when customers want access to the raw data, but I’m getting ahead of myself.)  As an industry, we’ve relied too heavily on borrowed-terms from architecture because the analogy breaks down when you consider that an architect doesn’t usually have to build first.   Here, we’ll look at how  Google’s BigQuery helped us do just that.