I received a phone call from a client last night who said that they could not browse the internet at their organization and that mail was down as well.
Here are the initial observations:
- Their Cisco VoIP phone system was able to make calls and internal machines could ping outside servers by IP, but not by name, indicating that the internet connection was functional.
- I could ping their firewall IP address and I could get into resources on their network other than the server remotely. Remote Desktop (RDP) didn’t respond, I could not telnet to the Exchange mail server (port 25) either.
- The client had already rebooted the server once, and was logged into themselves and reported nothing strange. They confirmed that on the server they could also ping outside IP addresses but the DNS server was not responding.
Okay, so this sounded like maybe the DNS server was the issue, but it was running okay, and the server itself could resolve hostnames. I was starting to lean towards firewall (on server or desktops) or switch issues at this point, but not being there and not having the ability to see the devices on the network with my own two eyes was hindering my progress. They opted to have us come out first thing in the morning and check it out.
When our tech arrived at start of business, he immediately realized that the server was unreachable from machines on their network and began looked over system logs, firewall & system settings and did the typical diagnostic in an attempt to restore connectivity.
Then he found it…
There were suspicious changes to the registry and upon running malware scanning tools, he quickly discovered the culprit. Malicious software of some sort had had its way with the machine.
The only thing that had happened leading up to this issue was that the client had been in touch with Intuit for a Quickbooks upgrade. The Intuit technician had performed a remote session and updated the QB install and the in turn the database files.
There are other potential ways that malware was put on the server, but the timing is suspicious as nobody else had accessed this machine remotely in many weeks, so we are going to assume that it came from the technician remotely accessing it, as we have no exact logging information that would indicate the time the changes were made.
Why am I blogging about this? In the event anyone else has an issue, they may find this post and confirm that they had a similar incident leading up to the event.
This could potentially happen if the installer of the software had become infected / exploited themselves, and it could be entirely accidental. Then again, some might want to speculate that people providing support to US based companies from foreign countries might intentionally wish to gain access to a server.
What can you do to protect yourself? Instead of using a software vendors foreign based technicians or engineers to update your software, you might fare better by using the services of a local based IT company. Also, having backups of your system files and system images will insure that you can “go back in time” and restore a system to a functional state should something of this nature happen.
Want to discuss this further? Feel free to add your two cents to our blog!
Should you desire computer or server support in the Greenville, Spartanburg or Anderson South Carolina area for your business, please call us at 864-990-4748 or use the contact form on our website. We would love to assist you!