While attempting to install, deploy, administer, maintain, improve, and be the general master of your business intelligence deployment with SAP Business Objects Enterprise, it often becomes difficult to determine what metrics you should use for determining the usefulness of your deployment. The goal of this post is to share some insights into those metrics that have been useful in my experience. This is, by no means, a comprehensive list nor is it, necessarily, the correct list for you. The idea here is to provide with you some points to think about as you start planning what you should be monitoring and tracking with your own deployments. As such, I’d love to read your thoughts whether they be via the comments on this post or on twitter.
Notice: All information in this post was generated using data from Sherlock® and all charts and tables were created in Web Intelligence.
So let’s look at a few examples…
Scheduling
Almost every business intelligence deployment has some form of scheduling. While scheduling provides robust means to deliver information to your users when and how they need it, the schedules can quickly build up and get out of control. In addition, there’s always the chance of jobs failing and, as the number of scheduled reports grow, the potential number of failed jobs grows. At times, attempting to triage and correct failing jobs can seem like a full time role.
There are a few metrics that should be monitored when it comes to scheduling.
An initial metric, shown below, is the actual number of scheduled reports that are running during each hour of the day. As you can see from the below chart, this system has a peak of around 80 jobs at midnight and at 4:30 AM. Otherwise, the number of schedules is fairly consistent throughout the day with the exception of a couple of spikes around 7:00 AM and 2:00 PM. There are obviously improvements which can be made here in terms of improving the distribution of the scheduled jobs; however, that’s not what we’re discussing. You could have this report on it’s own schedule so that it is sent to you every week or twice monthly. This would allow you to determine whether the schedules in your deployment are being handled efficiently.
A second metric that is useful to monitoring the schedules in your business objects deployment is the number of failures that you are getting per day or per month. The chart below shows failures for a deployment ranging from September 2011 to February 2012. You can see that there is a spike in the number of scheduled jobs starting in November 2011. You can also easily see that there was accompanying spike in the number of failed jobs. This type of metric can help you to determine how the scheduling system is being utilized and allows you a place to start from when investigating system failures.
You can combine the above with a list of failed schedules from the previous day and it will make for easier control and correction of jobs. The table below shows a potential listing of failed schedules which you could have run every night and emailed to you in the morning.
User and Content
For the users in the SAP Business Objects Enterprise system, there are a few key metrics that are interesting to track. The first of these is the actual activity level of the users with regards to how often they login. If they aren’t logging in frequently, then they obviously aren’t taking advantage of the system and you can remove them therefore saving licenses for valid users. As can bee seen from the screenshot below, 1083 users of this system have never even logged in. Out of the ones that have logged in before, almost 1000 of them haven’t logged in for over 90 days. This leaves about 1700 users who are valid users of the system.
You can dig a bit deeper into this information by looking at a few other metrics. For example, for those users who have never logged in, when were they created? This may give you some insights into why they may not be logging in and whether their logins are required.
Perhaps you want to see, for those users who have not logged in for over 90 days, how many reports they own? There’s obviously a clear winner from below. The user DD91735 owns 3824 reports, but hasn’t logged in for over 90 days.
By adding in an additional column, we can also see the total size of the reports owned by the DD91735 user. Yep, that’s a total of over 9 GB of space being consumed by a user who never logs into the SAP Business Objects system. Now, this doesn’t necessarily mean that the reports are being viewed by someone, but it is an indicator that should make you dig deeper.
Another group of metrics that is useful to track how the system is being used are the growth of content month over month. This allows you track whether users are actively using the deployment and determine if you may need to look into extending the system to handle the growth. The first graphic below shows the growth in the number of reports month over month. As you can see, there is a clear trend of growth even with dips here and there.
Now that we know the number of reports being created, let’s see the size of that content. Again, there is a clear trend in the size of the reports as well with January of 2012 reaching almost 80 GB of content created.
After identifying a clear trend of growth, you can now use another set of metrics to determine from where that growth is occurring. The first important metric is to determine what type of content has been created. We’ll use January 2012 as our analysis point.
So we can see that in January of 2012, the largest content created by size were Web Intelligence documents. In addition, the largest number of reports created were Web Intelligence documents. From there, we should understand whether those documents were created as instances of scheduled reports or not.
As can be seen from above, more than half of the Web Intelligence documents created in January 2012 were instances of reports. In addition to this fact, it could also be useful to determine whether the growth is coming from Inboxes, Favorites, or from Enterprise Folders. As you can see from the screenshot below, out of those WebIntelligence documents that are document instances, the majority of them are being delivered to Enterprise Folders while a significant chunk (i.e., 8 GB) are being delivered to Favorites folders. You could take this further to determine which Enterprise Folders; however, we won’t do that in this post.
While this post has provided some example metrics that you can use for monitoring the effectiveness of your SAP Business Objects Enterprise deployment, it is by no means exhaustive. My goal was to provide some ideas that will get you started with thinking about which metrics would be useful for you and your users. It would be great to read your thoughts on the types of metrics that you would find useful. Feel free to comment here or send me a message on Twitter.
Save the CMS!
Thanks for reading.
Thanks Coy…Good Ideas. I’m really focusing on who looks at what and how often. Then taking that information to try and better allocate my team’s time for converting old Deski reports to Webi or Explorer. Keep up the posts. BI people need to lead by example…so BI on BI is a great project.