Keeping a “happy” vCenter Server in your vSphere environment is crucial to having a vSphere environment that operates correctly and smoothly. Space issues are definitely an issue that can cause problems. I wanted to shed light on a problem I have seen a couple of times in my home lab environment with my VCSA 7. After logging into the vSphere Client, I have seen the message Log Disk Exhaustion on VCSA. What is the error? I will walk you through triaging and resolving this issue with disk space on your vCenter Server VCSA appliance, so we can take a look at Log Disk Exhaustion on VCSA 7 Fix.
Log Disk Exhaustion on VCSA 7 Update 2 Error
Have you seen this in your environment before with your VCSA appliance? I am running VCSA 7 Update 2. Note the message in the screen grab below:
A bit of Googling led to the following VMware KB: vCenter Server Appliance disk space is full (76563) (vmware.com) The KB article points to some general troubleshooting when you have general disk space issues with your VCSA appliance. There are a few points to note:
- Partitions are not sized correctly
- Large files in the partitions
- Tables in the database consuming space
- Services are failing to clean up files
For my VCSA appliance, I was using the standard, out-of-the-box partition sizes that come with a fresh deployment. I don’t really think I am seeing an issue with the services cleaning up log space as I think the logs are simply accumulating and becoming large enough before being pruned that it is flagging the warning in the VCSA vSphere Client. However, I wanted to clean up the space for the environment.
Log Disk Exhaustion on VCSA 7 Fix
To find where the disk space is being used, you can start by simply using the df -h command to verify which partition is having an issue. As suspected with the error message, the /storage/log partition is the one having issues.
Next, to start looking at where the storage space is being used specifically in the /storage/log partition, you can issue the command:
du -a /storage/log | sort -n -r | head -n 20
By looking at the directory structure listed above by size, I drilled down into the /storage/log/vmware/eam/web partition and found quite a few .log files that were 8 MB a piece for the ESX Agent Manager logging directory. After removing most if not all, I was able to reclaim a couple of gigs worth of space.
Next directory up for scrutiny is the /storage/log/vmware/lookupsvc directory listed in the output earlier. In my environment, below is a screenshot of the log files that I had accumulated under the directory. While not sizeable in and of themselves, combined with the sheer number, these were taking a good chunk of disk space also.
Under the same directory another level deep is the tomcat directory. Under the tomcat directory, I had another set of fairly numerous log files that had accumulated here. Again, I axed the ones that were older and kept just the most recent. This resulted in even more space gained back.
After going through manually pruning the files listed in the directories above, I was able to reclaim quite a bit of space and now had over 4 gigs free compared to the under 1 gig free that I had when I started the exercise.
Concluding Thoughts
If you get the message in the vSphere Client, Log Disk Exhaustion on VCSA, like me you are probably wondering what is going on with the log directory. The Log Disk Exhaustion on VCSA 7 Fix for me was going through the directories listed in the output of the “du -a” command and finding directories there that had many log files contained in the directories. I didn’t really find a silver bullet 1 or two files that were generated and causing the disk space issues. For me, it was just a combination of log files in the storage/log partition that was the culprit. Hopefully, in addition to the VMware KB article referenced in the post, this will help any who want to get past this error and issue on their VCSA appliance.
0 Comments