Possible Admin GUI Memory Leak - v11.1.0
Completed
Recently we had one of our production servers go unresponsive due to memory exhaustion. Taking a look into the event log, the process with the highest usage was the Cerberus GUI with 11 GB. No current individual was utilizing the server and I believe this was due to a process running in a disconnected RDP session.
Trying to reproduce, I opened the admin GUI on the same server and immediately exited from it. I noticed that a secondary process still remained in the background after exiting the application. I expect that this secondary process is part of normal application operation since I was able to reproduce this portion on earlier Cerberus versions in our environment.
After confirming the process remained after exiting, I saw it start off at roughly 130MB which is pretty nominal for a Windows application. Leaving it up for a bit, I saw an increase to 170MB with a bit of time passing and no known user interaction. I did see the memory usage periodically decrease by a few hundred KB that looked like some normal memory deallocation periodically taking place but there was still a marked increase over time.
Given these rates, the disconnected session must have been there for quite a bit which is its own issue in our environment that will need to be remedied, but I wouldn't expect the process to be able to get to 11GB of memory usage. Furthermore, these systems aren't under any sort of heavy load as they are getting stood up for future production use and only have internal access. The only traffic that is hitting these is a periodic health check from a load balancer as well as a couple of monitoring checks from a network monitoring systems, along with an individual or two who may be doing very minor testing (a login with a handful of uploads).
One thing I did notice is a significant amount of memory usage that contained the strings "DRH8D" which may just be garbage data but the other portion of the allocations showed what looked like request tracing from port 58000. I traced this communication to be a communication channel between the service and the admin application that worked on ports 58000 and 61035 on my machine. At that point, I was also able to reproduce a couple hundred MB of background memory utilization.
The following should be some helpful notes in problem reproduction/troubleshooting:
Reproduction:
1. Open Cerberus Admin GUI
2. Exit the admin application
3. Verify a secondary process is still running (I used the following in a CMD prompt: tasklist /FI "IMAGENAME eq CerberusGUI.exe", should be able to use Get-Process in PowerShell)
4. Periodically check memory usage (preferably over a few hours) utilizing the same command in #3, should be able to view a slow rise in memory usage over time in the process.
System:
Windows Server 2016 Standard
Cerberus FTP v11.1.0
Resource Exhaustion Message:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: CerberusGUI.exe (7716) consumed 12238553088 bytes <Further Processes Removed>
Memory Usage Analysis:
-
Thank you for providing us this information, Garrett. I have generated a support ticket from this article,and once we have conducted an investigation, we will update this post.
0 -
Jeff,
I appreciate the ticket creation! How goes the investigation so far?
0 -
Hi Garrett,
We are still investigating. When I opened the ticket, I put it under your name. Can you please check your JUNK/SPAM filter and look for a message from us?
0 -
Hey Garrett,
I wanted to follow up and let you know that we just released 11.2.7 which we think will offer you some better performance. When you get a chance, please update to the latest release and try it out.
0
Please sign in to leave a comment.
Comments
4 comments