Use PowerCLI to See ESXi Host Failed Login Errors

Monitoring for error messages as a VI admin is extremely important part of daily administration, especially when managing large environments with multiple servers and virtual machines. Sometimes we get lost in the noise of notifications and alerts that we can get desensitized to them to a degree. Having a way to see only the errors that exist on a host is a great way to keep an eye on a VI environment. I love making use of PowerCLI and having the ability to query the environment for information needed. What about looking for errors? Is there a way to query events at the host level to see errors that have happened on a particular ESXi host? Yes there is. Let’s take a look at how to use PowerCLI to see ESXi host failed login errors and query information providing useful and relevant information.

Using the Get-VIEvent PowerCLI Cmdlet

Working with PowerCLI provides an easy way to interact with vCenter and the ESXi hosts and query information that is useful for using in scripts, reports, CSV files, etc.

When looking for potential errors and warnings in a VMware vSphere environment, the Get-VIEventcmdlet is especially valuable as it allows you to query the events that have been logged in vSphere. This includes at the vCenter, host, and VM levels.

Below is a description of the Get-VIEvent cmdlet when running the Get-Help cmdlet. There are several parameters you can pass into the cmdlet for specifying the information you are looking for.

1. NAME
2. Get-VIEvent
4. SYNOPSIS
5. This cmdlet retrieves information about the events on a vCenter Server system.
8. SYNTAX
9. Get-VIEvent [[-Entity] <VIObject[]>] [-Finish <DateTime>] [-MaxSamples <Int32>] [-Server <VIServer[]>] [-Start <DateTime>] [-Types <EventCategory[]>] [-Username <String>] [<CommonParameters>]
12. DESCRIPTION
13. This cmdlet retrieves information about the events on a vCenter Server system. An event is any action in the vCenter Server system or ESX/ESXi host. You can filter retrieved events by specifying arguments for the cmdlet parameters.
14. Filters are additive. For example, when you specify the Entity, Start, and Finish parameters, Get-VIEvent filters events both by the entity and the timestamp properties. To specify a server different from the default one, use the
15. Server parameter.
18. RELATED LINKS
19. Online Version: https://code.vmware.com/doc/preview?id=6330#/doc/Get-VIEvent.html
21. REMARKS
22. To see the examples, type: "get-help Get-VIEvent -examples".
23. For more information, type: "get-help Get-VIEvent -detailed".
24. For technical information, type: "get-help Get-VIEvent -full".
25. For online help, type: "get-help Get-VIEvent -online"
28. PS C:\Users\bapple\Documents\GitRepos\VeeamCompareJobs> get-help get-vievent -examples
30. NAME
31. Get-VIEvent
33. SYNOPSIS
34. This cmdlet retrieves information about the events on a vCenter Server system.
37. -------------------------- Example 1 --------------------------
39. Get-VIEvent -Entity MyVM1 -Username admin -Types error -MaxSamples 15
41. Retrieves a list of the last fifteen error events on the MyVM1 virtual machine for the user admin.
42. -------------------------- Example 2 --------------------------
44. Connect-VIServer -Server 10.23.113.41
46. $events = Get-VIEvent -MaxSamples 100
48. foreach ($event in $events) {if ($event.fullFormattedMessage -match "User (.*)@\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b logged in") {Write-Host ("User " + $matches[1] + " logged in at:" + $event.createdTime)} }
50. Gathers information for the users that have logged in.

Use PowerCLI to See ESXi Host Failed Login Errors

Recently, I was involved with troubleshooting a host disconnect believed to be subject to the recently discovered VMSA-2019-0011 security advisory. A denial of service vulnerability in ESXi was reported when multiple failed login attempts to ESXi cause the hostd service to become unresponsive. This results in a DOS of management functionality related to an ESXi host.

If you have a host that is disconnected from vCenter and you want to verify this is the issue, be sure to look closely at the information in the following articles:

• VMSA Security Advisory
• VMware KB

If you look at the VMware KB and examine the vobd.log and see the following, and have not applied the patches you most likely will experience the issue:

1. 2019-04-20T17:11:03.592Z: [UserLevelCorrelator] 459377077473us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after XXX failed login attempts.
2. 2019-04-20T17:11:03.592Z: [GenericCorrelator] 459377077235us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after XXX failed login attempts.

If you are like me, you want to see what is causing the account to be locked out. As much as I like the new HTML 5 vSphere client, it is still painful to use the GUI to look through events quickly or search events.

Let’s see how we can use the Get-VIEvent cmdlet to do the heavy lifting and quickly find events we are interested in. The good thing is login failures are flagged as errors and can be queried as such.

Using the snippet of PowerCLI code below, you can quickly query in your VMware vSphere vCenter Server, all of your hosts and return all errors that have been recorded. In the AddDays section, change this to the number of days you want to look in the past.

1. foreach ($esxhost in Get-VMHost)
3. {
5. $esxhost.Name
7. Get-VIEvent -Entity $esxhost -Types Error -MaxSamples 10000 -Start (Get-Date).AddDays(–7) | select CreatedTime, FullFormattedMessage
9. }

The great thing with this is we can get even more specific and query for the “cannot login” message in the FullFormattedMessage field like so:

1. foreach ($esxhost in Get-VMHost)
3. {
5. $esxhost.Name
7. Get-VIEvent -Entity $esxhost -Types Error -MaxSamples 10000 -Start (Get-Date).AddDays(–7) | where-object {$_.FullFormattedMessage -like "*cannot login*"} | select CreatedTime, FullFormattedMessage
9. }

Use PowerCLI to See ESXi Host Failed Login Errors

Wrapping Up

If you are suffering from VMSA-2019-0011, it is easy to use PowerCLI to see ESXi host failed login errors to track down where failed login attempts may be coming from in addition to patching your ESXi hosts. This allows helping to shore up potential issues going on in the environment or possibly a compromised device on the network as well. With the Get-VIEvent cmdlet, you can easily query the events related to an ESXi host to see specific events like failed logins.