vROPS – Alerting Do’s and Don’t

In this post, I will answer how I personally handle alerting for my customers. Once you install an enterprise monitoring tool such as vROPS you will see that your environment has hundreds of alerts for Virtual Machines, Hosts, Datastores, etc. I fully understand the frustration when my customers say they get overwhelmed by all these active alerts as they don’t have any clue on where to begin to start fixing them. Management looks at these alerts and says go fix them. Believe it or not, I rarely ever go to the alerting tab at all, after showing my customers my dashboards they stop relying on the alerting tab to fix issues and start using my dashboards instead. Reason being is because my Health-Check Dashboard covers most of all the alerts that matter and more. With that being said, I will answer all the questions that customers have about alerting based on my 9 years experience at Vmware.

First, I need to put a disclaimer. This is based on my own personal experiences and opinions. This is not an official Vmware best practice guide.

In this post I will answer the following:

• Do’s and Don’t for alerting (Experience sharing)
• Too many active alerts, how do I begin to fix all of them? Management expects us to fix all theses. (Resolving alerts)
• How to manage these alerts? (Alert Management)
• How to best define what alerts get ticketed and what alerts get emailed? (Prioritizing Alerts)
• What is my recommended strategy for alerting? (Alerting Strategy)

Do’s (based on personal experience)

• Do create more alerts as needed, and make sure you test all possible scenarios – if you need an alert that is not out of the box, create it yourself. It does take logic and lots of testing.
• Do use dashboards instead as they make the process much easier – see my healthcheck dashboard to understand why (link below)
• Do separate your alerts to 3 separate categories (ticketing, email, and dashboard worthy) – I will explain this more in the last section.
• Do put a suffix in front of any custom alerts you create to help distinguish it from an Out of the box alert
• Do use policies to disable and modify alerts as needed

Don’t (based on personal experience)

• Don’t create a lot of vROPS Policies – this is way too hard to manage even for me. If you have more than 2 policies, you are making it more complicated for yourself in my opinion. 90% of my customers only need one. Only time they need a 2^nd is if they have a vRA, or VDI environment that needs different alerting, metric thresholds, etc. I been doing this for 9 years and I can tell you reverse engineering multiple policies is not easy even for me and possibly the person who created it.
• Do not forward all alerts to ticketing or email as they will get filtered!
  
• Do not delete any Out of the box alerts, disable them from the policies instead – they will come back once you do an upgrade
• Do not modify any of the Out of the box alerts – modify the symptoms from the policies or clone the alert and modify the clone. Then disable the original alert from the policy. Reason is your changes will all get lost once you update vROPS as those alerts will be overwritten back to the default.

Too many active alerts, how do I begin to fix them?

A superior monitoring tool like vROPS will have 1000+ alerts out of the box. Having too many canned alerts is a good thing, in fact this is what you paid for. You paid for the vendor to provide you with preconfigured alerts for all the objects you want to monitor (VMs, Host, vCenter, etc). I rather have 1,000 alerts out of the box than have 200 alerts. Reason is because the less alerts you have the more you must create manually. So, this is not a valid complaint.

Now I fully understand Management looks at these alerts and expect you to use it to make everything better and greater than it was before. Easier said than done as you begin to look at all the active alerts and you are confused on where to begin to fix all of these. To best answer your question on how I deal with this. I rarely go to the active alerts section at all. Yes, I been doing this long enough to know what the alerts mean but even I find it hard to start fixing everything by going to this section. Hence why I skip it totally and create a dashboard.

Now why do I use a dashboard instead? First let me explain the goal of all these alerts is to notify you what is wrong with your environment today. A dashboard allows me the best way to organize the data, modify the thresholds, and view it the best way possible. Download my dashboard and you will understand what I mean. Best thing is, it is free to download on my site and it is also the number 1 most downloaded dashboard on Vmware code. You can download it below, make sure to read the user guide.

http://www.vmignite.com/2021/02/vrops-vsphere-health-checker-dashboard-2-0/

Also, you can use the alert widget to enhance any dashboard

How do I manage these alerts?

Now that I answered the question on how to remediate the alerts by choosing which ones you want to address and add it to a dashboard. The next questions would be how do I best manage the alerts (add/remove/modify)? To best answer this I will break this up to 3 parts below.

Do I ever Remove Alerts?

Answer is no, mainly because the alerts definitions created out of the box is my way of asking vROPS what do you see. So therefore, I don’t remove any alerts personally. However, I do have customers who would like to remove some alerts from alerting for very good reasons. Most of the time, the alert doesn’t apply to their specific environment, so they don’t want to see it at all. Do not delete the alert! Go to your active policy and look for your alert and disable it from the policies. Reason why you don’t want to delete the alert is because it will come back once your update vROPS.

What do I do if I want to modify the alert?

If it something simple like modifying the symptom from a threshold of 80% to 90%. You can change that in the policy. Now if you want to add on to the alert, you will need to clone the alert and add the extra symptoms to the new alert. Then you will need to disable the original alert in the policy. Reason being is because once you update vROPS, the default alert will go back to the original state, therefore all the modifications you made will get reset back.

Do I add more alerts?

Answer is yes, some alerts are specific to customers ask. Make sure any custom alerts you put a naming in front of it such as VMignite.com – VM Alert. This way you will know that this is a custom alert and it also makes things easier to filter as all you need to do is type in VMignite in the search and it will show you all the custom alerts you created.

How do I best define what alerts get ticketed and what alerts get emailed?

Out of the 1000+ alerts vROPs has out of the box, a company must gather all their Engineers, Operators, and management to decide what alerts matter. All alerts should be defined into three categories:

• Dashboard Worthy
  
  • These alerts should only be shown in vROPS. Some example of these would be VMTools outdated or VM Snapshots. These are worthy to be shown on a dashboard but will be a nightmare if they get ticketed or emailed every time this occurs that’s because they happen daily and often. These lower priority alerts often get sent to ticketing systems and email notifications which leads to filtering of all vROPS alerts sent to a folder. So now when something critical such as vCenter is down alert gets sent, they will not see it in their inbox because it got filtered in a folder. This is one of the biggest mistakes that most of my customers do before I got there.
• Ticket Worthy
  
  • These alerts are actionable by Operations and Engineering team. Examples would be Datastore is running out of space, ESXi Host has contention that is affecting all VMs on that host, etc. These tickets are actionable, as someone needs to address these before it becomes a greater problem. However, they are not something you would drop what you are doing to fix it. Which leads us to our next category.
• Email Notification Worthy
  
  • These are alerts that you drop everything you do and go fix it immediately. In other words, these are alerts that you would ask to be excused in the middle of a meeting or drop your uneaten lunch to go fix it. Examples of these would be Email server is down, Datastore is 100% full which caused VMs to fail, ESXi Host crashed which brought down production VMs, E-Commerce site is down so customers can’t buy anything from our site, etc. These are events that Engineering should be the first to know and will need to fix it immediately before things sprawl out of control. Lots of time this gets filtered due to Engineering receiving too many non-critical alerts, which caused them to filter everything from vROPS to an email folder. This leads to the worst case scenario where the paying customer are the first to find out their servers are broken and calls to complain, which leads to upper management being furious and questioning why did IT Operations not catch it and why are we the last to know. The answer is simple, you bought the right tool, but you haven’t got the right strategy in place. Happened to you before? Well-read below on how to put the right strategy in place.

How to implement the right strategy (based on personal experience)

1. Make a list of Alerts that matter for each object (Virtual Machines, vCenter, Host, Datastores, etc). Lost? Just copy and paste my spreadsheet below to an excel. Add and remove as needed. Make sure you tell management you got it from VMignite.com. This at least will get you more credibility as I noticed people don’t like or trust using something one person put together. This list was perfected through many engagements with my Fortune 500 customer which make it more creditable.
2. Setup multiple meetings to get this list done. Noticed I said multiple, this is a timely process but worth it in the end
   1. It is a team effort! Not one person can make all these decisions. You need Operations, Engineering, and management all 100% involved
   2. Go through each line item one by one and ask the question, is this dashboard worthy? Is this also ticket worthy? Should this alert be sent out by Email notification? Make sure you explain them the clear difference between the three or show them this blog post if you must. You may use your Vmware TAM to coordinate this, as I find it always good to have an outsider to manage the meetings.
   3. One line item could fall in all three categories, for example “vCenter server is down” should be on a dashboard, should get a ticket sent, and an email alert as well.
   4. Fill out which team is responsible to resolve the alert. This will prevent conflict later, as everyone has agreed on who will take ownership of that alert.
3. Once everything is all filled out, use the spreadsheet to create the forwarding rules/notifications in vROPS.