In this post, I will answer how I personally handle alerting for my customers. Once you install an enterprise monitoring tool such as vROPS you will see that your environment has hundreds of alerts for Virtual Machines, Hosts, Datastores, etc. I fully understand the frustration when my customers say they get overwhelmed by all these active alerts as they don’t have any clue on where to begin to start fixing them. Management looks at these alerts and says go fix them. Believe it or not, I rarely ever go to the alerting tab at all, after showing my customers my dashboards they stop relying on the alerting tab to fix issues and start using my dashboards instead. Reason being is because my Health-Check Dashboard covers most of all the alerts that matter and more. With that being said, I will answer all the questions that customers have about alerting based on my 9 years experience at Vmware.

First, I need to put a disclaimer. This is based on my own personal experiences and opinions. This is not an official Vmware best practice guide.

In this post I will answer the following:

• Do’s and Don’t for alerting (Experience sharing)
• Too many active alerts, how do I begin to fix all of them? Management expects us to fix all theses. (Resolving alerts)
• How to manage these alerts? (Alert Management)
• How to best define what alerts get ticketed and what alerts get emailed? (Prioritizing Alerts)
• What is my recommended strategy for alerting? (Alerting Strategy)

Do’s (based on personal experience)

• Do create more alerts as needed, and make sure you test all possible scenarios – if you need an alert that is not out of the box, create it yourself. It does take logic and lots of testing.
• Do use dashboards instead as they make the process much easier – see my healthcheck dashboard to understand why (link below)
• Do separate your alerts to 3 separate categories (ticketing, email, and dashboard worthy) – I will explain this more in the last section.
• Do put a suffix in front of any custom alerts you create to help distinguish it from an Out of the box alert
• Do use policies to disable and modify alerts as needed

Don’t (based on personal experience)

• Don’t create a lot of vROPS Policies – this is way too hard to manage even for me. If you have more than 2 policies, you are making it more complicated for yourself in my opinion. 90% of my customers only need one. Only time they need a 2^nd is if they have a vRA, or VDI environment that needs different alerting, metric thresholds, etc. I been doing this for 9 years and I can tell you reverse engineering multiple policies is not easy even for me and possibly the person who created it.
• Do not forward all alerts to ticketing or email as they will get filtered!
  
• Do not delete any Out of the box alerts, disable them from the policies instead – they will come back once you do an upgrade
• Do not modify any of the Out of the box alerts – modify the symptoms from the policies or clone the alert and modify the clone. Then disable the original alert from the policy. Reason is your changes will all get lost once you update vROPS as those alerts will be overwritten back to the default.

Too many active alerts, how do I begin to fix them?

A superior monitoring tool like vROPS will have 1000+ alerts out of the box. Having too many canned alerts is a good thing, in fact this is what you paid for. You paid for the vendor to provide you with preconfigured alerts for all the objects you want to monitor (VMs, Host, vCenter, etc). I rather have 1,000 alerts out of the box than have 200 alerts. Reason is because the less alerts you have the more you must create manually. So, this is not a valid complaint.

Now I fully understand Management looks at these alerts and expect you to use it to make everything better and greater than it was before. Easier said than done as you begin to look at all the active alerts and you are confused on where to begin to fix all of these. To best answer your question on how I deal with this. I rarely go to the active alerts section at all. Yes, I been doing this long enough to know what the alerts mean but even I find it hard to start fixing everything by going to this section. Hence why I skip it totally and create a dashboard.

Now why do I use a dashboard instead? First let me explain the goal of all these alerts is to notify you what is wrong with your environment today. A dashboard allows me the best way to organize the data, modify the thresholds, and view it the best way possible. Download my dashboard and you will understand what I mean. Best thing is, it is free to download on my site and it is also the number 1 most downloaded dashboard on Vmware code. You can download it below, make sure to read the user guide.

http://www.vmignite.com/2021/02/vrops-vsphere-health-checker-dashboard-2-0/

Also, you can use the alert widget to enhance any dashboard

How do I manage these alerts?

Now that I answered the question on how to remediate the alerts by choosing which ones you want to address and add it to a dashboard. The next questions would be how do I best manage the alerts (add/remove/modify)? To best answer this I will break this up to 3 parts below.

Do I ever Remove Alerts?

Answer is no, mainly because the alerts definitions created out of the box is my way of asking vROPS what do you see. So therefore, I don’t remove any alerts personally. However, I do have customers who would like to remove some alerts from alerting for very good reasons. Most of the time, the alert doesn’t apply to their specific environment, so they don’t want to see it at all. Do not delete the alert! Go to your active policy and look for your alert and disable it from the policies. Reason why you don’t want to delete the alert is because it will come back once your update vROPS.

What do I do if I want to modify the alert?

If it something simple like modifying the symptom from a threshold of 80% to 90%. You can change that in the policy. Now if you want to add on to the alert, you will need to clone the alert and add the extra symptoms to the new alert. Then you will need to disable the original alert in the policy. Reason being is because once you update vROPS, the default alert will go back to the original state, therefore all the modifications you made will get reset back.

Do I add more alerts?

Answer is yes, some alerts are specific to customers ask. Make sure any custom alerts you put a naming in front of it such as VMignite.com – VM Alert. This way you will know that this is a custom alert and it also makes things easier to filter as all you need to do is type in VMignite in the search and it will show you all the custom alerts you created.

How do I best define what alerts get ticketed and what alerts get emailed?

Out of the 1000+ alerts vROPs has out of the box, a company must gather all their Engineers, Operators, and management to decide what alerts matter. All alerts should be defined into three categories:

• Dashboard Worthy
  
  • These alerts should only be shown in vROPS. Some example of these would be VMTools outdated or VM Snapshots. These are worthy to be shown on a dashboard but will be a nightmare if they get ticketed or emailed every time this occurs that’s because they happen daily and often. These lower priority alerts often get sent to ticketing systems and email notifications which leads to filtering of all vROPS alerts sent to a folder. So now when something critical such as vCenter is down alert gets sent, they will not see it in their inbox because it got filtered in a folder. This is one of the biggest mistakes that most of my customers do before I got there.
• Ticket Worthy
  
  • These alerts are actionable by Operations and Engineering team. Examples would be Datastore is running out of space, ESXi Host has contention that is affecting all VMs on that host, etc. These tickets are actionable, as someone needs to address these before it becomes a greater problem. However, they are not something you would drop what you are doing to fix it. Which leads us to our next category.
• Email Notification Worthy
  
  • These are alerts that you drop everything you do and go fix it immediately. In other words, these are alerts that you would ask to be excused in the middle of a meeting or drop your uneaten lunch to go fix it. Examples of these would be Email server is down, Datastore is 100% full which caused VMs to fail, ESXi Host crashed which brought down production VMs, E-Commerce site is down so customers can’t buy anything from our site, etc. These are events that Engineering should be the first to know and will need to fix it immediately before things sprawl out of control. Lots of time this gets filtered due to Engineering receiving too many non-critical alerts, which caused them to filter everything from vROPS to an email folder. This leads to the worst case scenario where the paying customer are the first to find out their servers are broken and calls to complain, which leads to upper management being furious and questioning why did IT Operations not catch it and why are we the last to know. The answer is simple, you bought the right tool, but you haven’t got the right strategy in place. Happened to you before? Well-read below on how to put the right strategy in place.

How to implement the right strategy (based on personal experience)

1. Make a list of Alerts that matter for each object (Virtual Machines, vCenter, Host, Datastores, etc). Lost? Just copy and paste my spreadsheet below to an excel. Add and remove as needed. Make sure you tell management you got it from VMignite.com. This at least will get you more credibility as I noticed people don’t like or trust using something one person put together. This list was perfected through many engagements with my Fortune 500 customer which make it more creditable.
2. Setup multiple meetings to get this list done. Noticed I said multiple, this is a timely process but worth it in the end
   1. It is a team effort! Not one person can make all these decisions. You need Operations, Engineering, and management all 100% involved
   2. Go through each line item one by one and ask the question, is this dashboard worthy? Is this also ticket worthy? Should this alert be sent out by Email notification? Make sure you explain them the clear difference between the three or show them this blog post if you must. You may use your Vmware TAM to coordinate this, as I find it always good to have an outsider to manage the meetings.
   3. One line item could fall in all three categories, for example “vCenter server is down” should be on a dashboard, should get a ticket sent, and an email alert as well.
   4. Fill out which team is responsible to resolve the alert. This will prevent conflict later, as everyone has agreed on who will take ownership of that alert.
3. Once everything is all filled out, use the spreadsheet to create the forwarding rules/notifications in vROPS.

Every time I walk into a new engagement with a Fortune 500 company, I always ask the customer the question “what are issues they want to address”. Of course, I always get these cool stories on how performance bottlenecks like high CPU and Memory slowed up their environment, how things are breaking left and right and no one knows about it, and how they would like to find out what is causing all their problems and outages. All companies want that magic 8-ball solution, something that will instantly identify and solve all their problems magically.

In the end, I am always handed a dozen requirements and sure I can just resolve those and be done with the engagement and then wait a few month later when new problems occur. By then, they might possibly ask for help again or even worst not use the tool at all and go back doing things the manual way or buy another tool thinking it will solve all their problems.

Based on my experience: All companies pain points are the same and the typical customer only mention 10% of what today’s top monitoring applications like vROPs can actually do. It is up to the expert to guide them towards to what I call “Monitoring 360 degrees” which basically means maximizing what a monitoring tool should be doing. When I say all companies pain point are the same it is because all their requirements will fall under these 6 categories below. (Disclaimer: this is all my own personal opinion)

All Monitoring Use Cases

1. Being Proactive (Fix problems now to prevent future outages)
2. Root Cause Analysis in minutes (Troubleshooting VM, Host, vCenter, etc)
3. 24/7 Critical Monitoring (Monitoring critical Apps, Websites, Infrastructure)
4. Capacity Management (Realizing Utilization, Growth, Inventory)
5. Optimize Resources (Reclaiming Resources, which resources need more cpu & memory)
6. Compliance (Hardening check, Environment consistency checks)

Now you probably thinking where are the reports, alerts, ticketing, and auto remediation? These are not use cases, these are bells and whistles that apply to all 6 use cases above. For example, anyone of those use cases mentioned above can involve reports and alerting for example if the customer wants it. I can easily create alerts on performance, capacity, compliance, etc.

With this being said, this is what all monitoring tools should consist of at a bare minimum.

VMignite.com Monitoring 360 Degree Checklist

1. Proactive Dashboard
2. Troubleshooting Dashboards
3. 24/7 Critical Monitoring Dashboard
4. Capacity Management Dashboards
5. Optimization Dashboards
6. Critical alerts being sent as email
7. Selected Alerts being generated as tickets

Let me explain how some of this should work.

1. Majority of companies are reactive, but they all want to be proactive – To achieve this, you need the following
   1. Health dashboard – you can’t begin to prevent problems before they happen when you have no idea what problems you have to begin with. That is why you need a dashboard that displays all the problems you have today (Host, vCenter, VMs, Datastores, Clusters, etc). Sounds difficult? Good thing is I created an environment health checker dashboard already if you have vROPs. It is the number one most downloaded dashboard on VMware code. Download it here http://www.vmignite.com/2021/02/vrops-vsphere-health-checker-dashboard-2-0/
   2. Alerts being sent out – Next you need to decide what alerts should be forwarded to your ticketing system and what should be sent as email alerts to engineers to fix immediately. If the alert that caused the outage doesn’t exist yet you will need to create it! I wrote a guide on this here:  http://www.vmignite.com/2021/06/vrops-alerting-dos-and-dont/
2. When outages do happen, you need to figure out what caused it immediately. You will need the following:
   1. Troubleshooting dashboards – An excellent troubleshooting dashboard should be able to find root cause analysis in a minute! For example, to troubleshoot a problem VM in a minute, I will need to be able to eliminate the Host, Network, Physical Server, and Storage from the equation. On top of that I will need to be able to identify all VM bottlenecks such as CPU, Memory, contention, Disk Latency, application, configurations, etc. Sounds impossible to find root cause analysis in a minute? I have proven to be able to do this with all my customers. Sorry can’t share this dashboard, you can download some light versions of my troubleshooting dashboard on my download page
3. All management, engineers, and operators need insight of the entire environment in one pane of glass
   
   1. 24/7 Critical Monitoring Dashboard – let’s say you want to monitor in one pane of glass an environment that consist of 20 vCenters, 1000 Host, 1000 Datastore, 100 Cluster, and 20,000 VMs. On top of that I want to monitor critical websites, vsan, and nsx as well. Also if a site is having problems, show me what objects are causing the problem using the same dashboard. This is a dashboard that I have created for my customers and they have dedicated monitors and even TVs to display throughout the company. Here is a simple version I created, the more advance one is for my customers: http://www.vmignite.com/2021/06/vrops-8-4-executive-dashboard-download/

Once again, all this is my own personal opinion but was based on my 8 year experience working for VMware as a consultant. To learn more granular features on what a powerful monitoring tool can do read the following

http://www.vmignite.com/2020/03/15-features-that-makes-vrops-the-best-monitoring-tool-period/

Updated: 1/09/2022