Big Data Infrastructure

Large data volumes have existed for many years, but mainly in transactional systems like ERP’s.  This data was optimized for transaction processing, not reporting.  The SQL necessary to report on the data often required many to dozens of joins, thus performance was sub-optimal.  In come Mr. Kimball and Mr. Inmon who defined methodologies (or religions for some) to stage, normalize, and structure data for optimal analysis and reporting.  Regardless of your allegiances, this required aggregating data and this aggregation removed a portion of the detail, but provided sufficient insight into the trajectory of the business.  Unfortunately, it also meant having to create multiple cubes or rollups to answer the questions necessary to keep the lights on.

However, times are changing, at least technology and the cost of hardware is changing.  As hardware prices, specifically disk, drop and the technology to distribute the load across all the hardware makes leaps and bounds forward, the ability to report, data mine, and draw analytics based conclusions from raw & unstructured data increase exponentially.  The time has come for a trip back to the future.  By that I mean that not too long ago we were using file based databases similar to FoxPro or JD Edwards or others that distributed data across multiple file based “tables”.  Now are coming full circle to something similar, but have gained some extra advantages with the Hadoop Distributed File System (HDFS) and MapReduce functionality that allow us to utilize these “files” or “tables” to their fullest extent with the pooled resources of multiple machines/servers.  No longer are we defined by the size of our single mainframe or blade systems.  From a Business Intelligence perspective, specifically focused on Reporting/Analytics & ETL, these types of advantages allow us to no longer concern ourselves with throughput; rather we can focus on asking the most effective questions.  Identify trends based on individual customer experiences, rather than a one-size-fits-all approach.  Make targeted recommendations or marketing messages to each customer based on purchase history and predictive modeling based on other customers with a similar arc.

Conclusion

Business Intelligence is just a general term used to define the process of driving efficiencies from your everyday activities and learning from your customer’s reactions to those activities.  Similarly, Big Data is a buzz-word meant to incite your interests.  In the end, both are extremely ambiguous and neither is actionable.  However, measuring your business and drawing efficiencies from those measurements is just good business.  Doing so while customizing the experience for each of your customers is even better, so don’t let the buzz-words get in the way of good business.