The purpose of this document is to ensure your VSA server(s) is prepared to receive the VSA release patch, which contains critical security fixes.
Before you restore full connectivity between Kaseya VSA server(s) and deployed agents, there are certain steps that we are recommending:
- Ensure your VSA server is isolated
- Check System for Indicators of Compromise (IOC)
- Patch the Operating Systems of the VSA Servers
- Using URL Rewrite to control access to VSA through IIS
- Install FireEye Agent
- Remove Pending Scripts/Jobs
Each of these steps is described in more detail below.
Step 1 – Ensure your VSA server is isolated
Depending on where and how you host your VSA server, this process will vary between platforms. Before powering on the VSA server, please ensure that you isolate it from inbound and outbound traffic and segregate it from your main network. There are several ways to do this depending on your specific case. Booting it up in Safe Mode could be one way to achieve that.
If you host your server in AWS we recommend that you check the NIC network security group and the public inbound ports assigned to the application server. You should deselect/disable ports 80/443/5721 (or any other non-default Kaseya port you might have used). Leave only the ports enabled, which provide you access to the device. For example, port 3389 is the standard port for RDP. Keeping this enabled will allow you to RDP into the virtual machine (VM) and perform the next steps without exposing the VM to the internet.
In order to disable Kaseya network communication, please stop the Kaseya Edge Services service and set it to Disabled.
Step 2 – Check your System for Indicators of Compromise (IOC)
Please make sure you have run the “Compromise Detection Tool” on your VSA server. If you have not already run this tool, it can be downloaded from here. Please ensure you follow the directions available via the link.
- If, at this point, you are unsure about the results received, please contact our support team at helpdesk.kaseya.com, or send an email to firstname.lastname@example.org. We will be happy to assist you.
- For information about the detection tool, see the appendix below.
Step 3 – Patching the Operating Systems of the VSA Servers
For the following steps we require internet access, so if your machine is completely isolated from the internet, please restore “Outbound” internet connectivity now. Before doing so please double check that the Kaseya Edge Services service is still disabled.
- Apply the latest Microsoft patches to both the Windows operating system and SQL server. Please verify that your SQL server is on the latest available patch. This page tells you about the latest available patch for the product you use: https://docs.microsoft.com/en-us/sql/database-engine/install-windows/latest-updates-for-microsoft-sql-server?view=sql-server-ver15
This process might require a few reboots, depending on how many outstanding patches there are. Please ensure no Windows or SQL patches remain outstanding.
Step 4 – Using URL Rewrite to control access to VSA through IIS and Firewall Rules
*Updated July 11th 2021*
Based on customer feedback, we have made changes to the IIS rewrite tool in order to give customers more control of their environments using their firewalls. Please rerun the tool referenced below – it will automatically update the IIS configuration even if you ran the tool previously.
The purpose of this step is to control access to your VSA server and only allow the necessary access to the system User Interface (UI). This MUST be done on the VSA server.
Part A - There is a tool available for download, which helps automate this configuration process. You can obtain the tool at: https://app.box.com/s/1yel8q8nw4sxpujbhtudaqapk5yd84qk
Once this is completed, you should expect the following:
Access to your VSA UI will block certain communication from the outside world (only what is necessary) on inbound port 5721.
Part B – Configure your firewall to only allow inbound traffic to the VSA on port 5721 (the agent port).
This will require your VSA users to be on the local network or VPN’d into the local network reducing the attack surface.
Given your unique environmental needs, you can create rules in your firewall to also allow port 443 inbound traffic to your VSA where required. For example, certain IP addresses for integrations or locations where you must allow access to the web GUI.
This provides customers the flexibility to use their existing firewall to control access to the VSA web GUI to meet their unique requirements while securing the agent port (5721) IIS access which is required for agents and some communications.
Please review a list of popular integrations that you may wish to whitelist at https://helpdesk.kaseya.com/hc/en-gb/articles/4403869952657
Step 5 – Install FireEye Agent
Kaseya is providing complimentary licenses of FireEye Endpoint Security agents for each customer’s VSA Server(s).
To initiate this process, please send an email to email@example.com and be sure to include the following information:
- Your Company Name
- Your Mobile Number (only to contact you in case the information is not complete)
- VSA Device Role (Web Server or Database Server)
- If you have a split server configuration, you will need to install FireEye on both the Web Server and the SQL server. Please provide this information for both servers.
- The hostname of the Server
- Your Domain Name (for example, kaseya.com)
- Internal IP Address
- Egress (External) IP Address (Please use this URL on the server itself to confirm your gateway IP - https://www.whatismypublicip.com/)
- VSA Agent Port (By default it is 5721, please provide your port number if you changed this)
Once Kaseya receives your email, our teams will contact FireEye to have the agents provisioned, and will then reach out to you with instructions, and a link to the installer.
Please ensure you send us all the information requested above as it is necessary for the FireEye agent to function properly.
6 – Remove Pending Scripts/Jobs
7 - VSA SQL Database Assessment
In response to the July VSA Security Incident, Kaseya has created a PowerShell script that connects to your local SQL Instance and generates an HTML report of important data points that Kaseya has deemed valuable to review. The inclusion of these data points is not directly correlated to known Indicators of Compromise (IOC) but rather an audit of the user, procedure, and agent connection information to provide assurance of the state of the VSA prior to the restarting of VSA Servers.
Please note that we strongly recommend that you do NOT give your VSA Server internet access while running these scripts. Ensure your VSA remains OFFLINE and associated services are stopped until you receive an official update from Kaseya.
Please make sure you have run the "VSA SQL Audit Report" tool on your VSA Server. If you have not already run this tool, it can be downloaded from here. The link contains instructions and a sample report.
If you need assistance or have run the report and suspect your VSA application has been compromised, please contact our support team at helpdesk.kaseya.com, or send an email to firstname.lastname@example.org. We will be happy to assist you.
Do not start up your VSA Application until this VSA patch has been applied!
Detection Tool Information
This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present.
The latest version searches for the indicators of compromise, data encryption, and the REvil ransom note. We recommend that you re-run this procedure to better determine if the system was compromised by REvil.
This continues to be enhanced, so please always refer to the download site for the latest version.
From the review of data provided by clients, we have identified IOCs. We are providing the following IOC information to aid our customers and security researchers in their investigations. Kaseya’s investigation is ongoing and, as such, this information is subject to change.
The following IP addresses were seen accessing VSA Servers remotely to perform the attack sequence:
The following files were used as part of the deployment of the encryptor:
N/A – Legitimate File with random string appended
Legit certutil.exe Utility
Encoded malicious content
Decoded contents of agent.crt
Web Log IOCs
The following are excerpts from the IIS access logs of a compromised VSA server. They depict a sequential series of HTTP requests that the threat actor made to perform their attack. If this sequence of requests is present in the IIS logs of a VSA server, it suggests the threat actor either attempted to or successfully used it to perform their attack.
POST /dl.asp curl/7.69.1
GET /done.asp curl/7.69.1
POST /cgi-bin/KUpload.dll curl/7.69.1
GET /done.asp curl/7.69.1
POST /cgi-bin/KUpload.dll curl/7.69.1
POST /userFilterTableRpt.asp curl/7.69.1
Based on what most customers tend to use, here are some areas within the VSA you can consider looking into to make sure that everything is as you would expect it to be:
- Agent count
- Policy Management
- Agent procedures
- Users (with their Roles and Scopes)
- While the above-linked tool checks all the below, if you wish to perform a manual verification, please check the following. Search the Kaseya installation directory (e.g. C:\Kaseya\Webpages for the following files: UserFilterTableRpt.asp and userFilterTableV6KES.asp.
Note: Please be sure to purge any copies if found.
- Search IIS logs for references to UserFilterTableRpt.asp . You can find the IIS logs typically under C:\inetpub\logs\LogFiles\W3SVC1 . Order the directory content by ‘Date modified’ and open the log files modified between the 2nd and the 7th of July. Search the content for the above filenames or any items listed under Web Log Indicators.
- Search for Kaseya\webpages\managedfiles\vsaticketfiles\agent.crt. If found, please archive this file into a password-protected zip file and submit it to us via the support desk including the password applied. Once submitted, please purge the artifact from the hard drive.
- Search for Kaseya\webpages\managedfiles\vsaticketfiles\agent.exe . If found, please archive this file into a password-protected zip file and submit it to us via the support desk including the password applied. Once submitted, please purge the artifact from the hard drive.
IMPORTANT: DO NOT EXECUTE this file as it can be an exploit.