Agile vs. Industrialized?

Industrialization not always the appropriate end

James Taylor (http://jtonedm.com/about/), who is passionate about automated decisionmaking, has a new blog post (http://smartdatacollective.com/jamestaylor/26598/its-time-industrialize-...) in which he argues that analytics is not a “cottage industry” and needs to focus on “industrialization.” He writes: There is a feeling that, because what analysts do is complex and hard for others to understand they should be allowed to swan around picking their own tools while being given lots of autonomy and plenty of freedom to experiment. This is, I believe, a very dangerous idea. It is time for organizations to take a stand and industrialize their advanced analytics efforts.

I’m not sure what “swan around” means—perhaps it’s one of James’ Britishisms—but this might be construed as an argument against the agile analytics approaches I’ve been writing about in the last couple of posts. And I don’t disagree that more enterprise consistency on analytical tools would be desirable. However, while I agree with James that analytics needs more industrialization, I don’t think it’s appropriate everywhere.

You industrialize something when it works perfectly, or almost perfectly, and at scale. That’s not ever going to be true in every case where we’re employing analytics. If it’s purely a discovery exercise—finding out what’s going on with a business problem or dataset—then industrialization may never be appropriate, or appropriate well down the line. If it’s known from the beginning that analytics need to be embedded in a high-volume business process, then it makes sense to prepare for industrialization from the beginning.

Much of analytics—and life, for that matter—are in the middle. When we start out we’re not entirely sure whether the models we’re developing need to scale up to production volumes. Even if we think they may, we’re not sure how long it will take to get to the stage where the model is good enough to industrialize. In this circumstance it makes sense to be agile in your planning and your deliverable structure, because you just don’t know what you’re going to encounter.

The Cisco example I mentioned in my last post is a great example. It eventually went industrial, but only after a period of exploration and agile analysis. I think most analytical models that will eventually be industrialized would benefit from a period of treating it as a cottage industry. However, I agree with James that most cottage industries don’t produce outputs at a high enough level to make a difference.