The metrics we can easily capture from SharePoint, with or without third-party tools, do not typically measure what matters most. Employees care about working efficiently. Managers care about operating metrics. Executives care about financial metrics. Most organizations already have measurement programs in place to track these critical business metrics. But, the system metrics we can capture from SharePoint and third-party tools can provide us with some really great clues that can help get to the real business value of our SharePoint solutions. In this post, I’d like to explore a methodology to use to think about how you can use system metrics to provide clues to help identify and improve both the adoption and business impact of your SharePoint solution. There are five key elements to this methodology:
- Understand the business problem you are trying to solve
- Identify use cases that have meaningful impact
- Determine the metrics that align with each use case
- Understand your baseline and establish a target
- Measure and monitor: be prepared to change
Understand the business problem you are trying to solve
To get credibility in the organization, we need to ensure that the solutions we build with SharePoint demonstrate a positive impact on business metrics--and not just in a soft or indirect way. This means that we need a comprehensive understanding of the business problem we are trying to solve. This is especially true if we are implementing the social computing features of SharePoint. One of the reasons many “social” solutions fail to gain traction is due to the fact that they are disconnected from the key challenges that drive our organization. For example, they are not promoted as a solution or an enabler of a major organizational initiative. So, as a first step, we need to make sure we have a clear (and documented) understanding of the real business problems we are trying to solve.
If you aren’t addressing an important or valuable business problem with your SharePoint solution, you need to go back to the beginning to find that problem, as well as a business sponsor who is committed to solving that problem. Be sure that you are tying your SharePoint solution to a key organizational initiative or goal. If not, you are working on a "side show" project—one whose funding is going to be at risk no matter what your measurement program concludes.
Another reason to have a clear connection to business goals is to help make decisions about which potential SharePoint project(s) to implement. All organizations have limited time, budget, and resources, and most find that there are more possible projects for the SharePoint team than can possibly be accomplished. Therefore, it's important to have a framework for differentiating among opportunities that are competing for scarce resources.
Identify use cases that have meaningful impact
Once you’ve identified the key business problem that you are trying to solve, the next step is to identify specific use cases where using your SharePoint solution can have an impact on those problems. The collaboration solutions you build with SharePoint can have a large impact on your business, but it’s important to be practical when you are choosing use cases – because you are not necessarily going to solve every issue with SharePoint alone. Be practical.
Use case example: Support call deflection
One association project I worked on had a goal of providing a self-service knowledge portal that would allow member organizations to share best practices and learn from others facing similar challenges without always having to depend on engaging with the headquarters team to get answers to simple questions. With limited resources at headquarters, they wanted to be able to focus on more complex issues for their members. One of the impact-based use cases they identified was simple support call deflection. In other words, the use case proposed that reducing the number of simple support calls directed to the headquarters support staff would free the HQ support team up to work on more complex requests because members would find answers to simple questions on their own via the knowledge portal.
This organization was also already maintaining a knowledge base of resources, but at a considerable cost. Another key business goal was to build an online, accessible, and searchable inventory of both curated and crowd-sourced practices. The measurable use case for this business goal was “knowledge resource ownership” to improve the quality of resources available while decreasing the cost of building and maintaining the knowledge base.
Determine the metrics that align with each use case
Each of your use cases should have at least two to three metrics that allow you to create a tangible target objective. For example, in the scenario above, we identified the following metrics for the support call deflection use case:
- Number of routine support calls
- Number of complex support calls
- Number of successful searches for online content by members (searches with at least one result)
- Number of “0 results” searches
The first two metrics were already being captured by the support team. If our solution is going to demonstrate business value, we would want the first number to decrease. We might, however, expect the second number to increase – and measuring the impact of that outcome might ultimately be tied to an improvement in member satisfaction, yet another overall business goal. The second two measures are system metrics that serve as proxies for business value. If users consistently get relevant results for their queries, then the solution should be delivering value. The only way to know for sure, which is why the system metrics only provide clues about value, is to explore search outcomes directly with searchers to determine the specific business value of results found.
Tip: A very easy way to do this, by the way, is to include an opportunity for users to provide search results feedback directly on the search results page. Another way to do this is to send out periodic search impact surveys where you specifically ask users what they were able to find and what they did with the results (i.e., how they measured the impact of what they were able to find).
For the second use case in the example above, to increase the shared ownership of knowledge resources, we identified the following metrics:
- Number of assets contributed by the association: This metric was expected to decrease over time, to demonstrate that the membership was actively engaged in building the repository.
- Number of assets contributed by members: This metric was expected to increase steadily over the first year. An increase in member-contributed assets, especially if the contributions were distributed across many members, is an indicator of engagement on the part of the membership as well as interest in creating sustainable value, an important business goal.
- Number of members contributing at least one asset: The goal for this metric was to ensure that at least 50 percent of the membership contributed something within the first year.
- Percent of registered members (users with a profile) who posted or replied to posts in an online conversation: This metric was particularly important in this example because the solution that SharePoint was replacing, a traditional listserv, was very active but tended to be dominated by a small number of “loud voices.” The goal of this metric was to continuously monitor participation to both ensure that the conversations weren’t dominated by a small group but also to identify potential members who the association might want to pro-actively encourage to engage.
Choose the metrics you want to capture in terms of the use cases that are of highest interest to your stakeholders and their business objectives. The metrics in my examples may be completely irrelevant in your organization – especially if you don’t share the same business goals. It's also important to pick a small number of metrics that are both relevant to the business and have a more direct relationship to business outcomes – and can be collected at a relatively low cost.
Understand your baseline and establish a target
One major mistake people often make in documenting their measures of success with SharePoint is that they try to demonstrate business impact without knowing where they started. This approach is doomed to failure. You need to have a good baseline (starting point) or you will have no ability to measure progress. If you don’t have a great baseline measure—or it’s too late to get one— capture you initial metrics when you launch your solution.
In addition to a baseline measure, you also need a target! Make sure that you get agreement from your stakeholders regarding what the target should be. In other words, you want to try to get agreement on the definition of success – if we hit this target, then the system is successful. I can’t emphasize the importance of this step enough: make sure that your stakeholders agree on the definition of success. In fact, I often start stakeholder interviews with these questions:
- How will you know if the solution we are building is successful?
- What specific measures will you use?
If I ask these questions up front, I have a pretty good clue on what I should be counting!
The image shows some of the targets we used for the support call deflection use case:
Measure and monitor: be prepared to change
Metrics are just a tool. The entire goal of your measurement program is to get better – to make changes to your solution or even your metrics program so that you can make better decisions and have a bigger impact on organizational results. One thing is certain: the complex and dynamic nature of pretty much all organizations means that the SharePoint solutions we build are going to have to be flexible enough to change in order to continue to provide value. The only way to understand what changes you need to make is to define and execute your measurement plan.
Use your measurement plan to assess how users are taking advantage of the solution and let it act as an early warning system to identify both new metrics and new capabilities that can help achieve your business objectives. In other words, use it to provide clues about what you need to change.
When a metric shows an unexpected result, try to find out why the result occurred. Is it due to a one-time event or it is an indicator of a trend? Ask yourself if the result you are seeing tells you that you may be measuring the wrong thing or whether there is something fundamentally wrong with the way the solution has been developed or deployed. At the very least, metrics will help you get ideas for how to improve your solution. Collect and prioritize these new ideas and go back to your original plans and assumptions to see if they need to be changed. Then, build consensus on what needs to be changed, how to make the change, and when and how to introduce the change to your users.
The following table presents a format that I have used to help gain consensus about prospective measurement plans. This format can be used to both create and document the plan and to provide a report card of progress once your solution is implemented.
Key points to take away
Perhaps the most important thing to take away from this discussion is that the only metrics that count are the ones that provide clues, either directly or indirectly, that describe how your solution is delivering meaningful business impact. It’s not always easy to get this information from system metrics, which is why you should not be afraid to do periodic user surveys to try and draw out stories that describe how the solution has delivered business value. Stories are how humans communicate and qualitative stories supplemented with meaningful system metrics are the best way to communicate the value and impact of your SharePoint solution.
For more information and additional examples of both quantitative and qualitative approaches to capturing business value metrics, please check out my white paper, A Practical Framework for SharePoint Metrics.
Note: This article is adapted from a chapter of the same title that I wrote for the book: Prove It!: Using Analytics to Drive SharePoint Adoption and ROI.