How Do You Know Your Predictive Model Results Are Valid?

Across our diverse portfolio of clients, we have repeatedly seen the value of getting an early second opinion on the validity of predictive analytics models. Why bring in an outside expert? Even well-intentioned, highly-capable, technically-sophisticated data analytics initiatives can fail if they lack proper context and support from within the business.[1] Having an accurate model is not good enough. Are you confident that you’re on the right track with respect to your analytics initiatives? What changes does the model demand of the organization? Most importantly, how do you know the model results are valid?

An Enthusiastic and Positive Vision

Interest in advanced analytics has never been higher.[2,3,4] As companies transition from data collection for business intelligence into the new reality of Big Data, they are waking up to the value and insights stored away in their data warehouses. Whereas analytics used to be primarily in the purview of high-technology, internet-based companies like Amazon, Google, or Facebook, more traditional companies like GE are now investigating how analytics can be a differentiating factor for their growth in the future.[5] This means organizations are building new infrastructure and hiring new staff (or re-tasking existing employees) hoping to capitalize on this information revolution. Highly capable open source tools are available, and everyone and their brother is becoming a data scientist these days, so achieving success by going it alone should be straightforward, right?

Reality Check

The reality is a much different picture. The proper practice of data science is hard work, and building a useful predictive model that actually has business value is a complex process with a number of moving parts. Can your business overcome these hurdles?

• Infrastructure: You’ve got a new server. Great! Does your IT department support giving your new data scientists free reign to install open source software on it?
• Data Access and Availability: Your company has two decades’ worth of business intelligence stored in different databases. Fantastic! How does IT feel about providing the Sales team data scientists access to data from Engineering?
• People: You’ve hired data scientists. Good move! To whom do they report? How do the analytics solutions they are developing integrate with existing processes? Does middle management know that you’re developing metrics that may change their team alignments?
• Analytics Strategy: You’ve identified a data analytics project that could really move the needle on the bottom line by addressing a key business issue. Excellent! But do the data you have really answer the question being asked? This might be a real issue, but will management agree it has the priority to divert valuable resources away from other strategic initiatives?

Gaining Confidence With a Trusted Partner

For more than 20 years, Elder Research has provided analytics assessments and predictive model validation consulting for industries as varied as insurance, finance, defense, consumer goods, and the federal government. As each new industry has woken up to the value of analytics, we have been there to guide them through the transition. Most recently we validated a predictive model of maritime casualty risk for RightShip. In all of our client engagements, we seek to “teach our clients to fish” — to help them build internal capabilities to ensure their continued analytic success.

Our Model Validation consulting service is an end-to-end assessment of the whole process of building a predictive model. Through meetings and investigations, we learn why this model was built, where it fits in the broader corporate analytics strategy, and how it will be integrated into the production environment. We then attempt to reproduce the observed results through a rigorous, scientific investigation, to determine the robustness of the model that has been built. Upon completion, we provide a concise official letter of judgment and a comprehensive report of our investigation, including any potential threats to model validity and recommendations for improvements. Our goal in a Model Validation engagement is to provide our clients with trusted third-party risk assessment, and we are often able to assure that the model is robust to expected changes in the data and underlying assumptions.

Introducing advanced analytics into an organization is a serious change management task. Predictive models produce results that will require action to be taken in order to realize their full value. Is your organization motivated and ready to take that action? Or will unexpected roadblocks arise? The success of an initial analytics effort is crucial to engender the confidence to drive the analytics strategy throughout the organization.[1] It is only when analytics become pervasive that their full potential will be realized.[4] The cost of “going it alone” could rapidly outstrip the cost of hardware and new hires if the model is not integrated with business needs and actionable results are not accessible.

As valuable as our clients have found the tangible results of our Model Validation service, often the most valuable outcome has been intangible — confidence. Confidence that the model they have built is robust and stable. Confidence that their analytic investment will see a short term return as well as continue to provide value moving forward. Most importantly, confidence that they have the capabilities, people, and infrastructure to take their organizations forward into a data-driven future.

Request a consultation to see how Elder Research can help with model validation or other analytics consulting services.

______________________________________________________________________

1. ­­­­­­­­­­­­­­­­­­­­­­­Helen Mayhew, Tamim Saleh, and Simon Williams, Making data analytics work for you — instead of the other way around (McKinsey & Company, October 2016).
2. Amy Forni, and Rob van der Meulen, Gartner’s 2016 Hype Cycle for Emerging Technologies Identifies Three Key Trends That Organizations Must Track to Gain Competitive Advantage (Gartner, August 16, 2016).
3. Satty Bhens, Ling Lau, and Hugo Sarrazin, The new tech talent you need to succeed in digital (McKinsey & Company, September 2016).
4. Thomas H. Davenport , Analytics 3.0 (Harvard Business Review, December 2013).
5. Steve Lohr, GE., the 124-Year-Old Software Start-Up (The New York Times, August 27, 2016).

Originally published at www.elderresearch.com.