On Premises VSA Startup Readiness Guide - July 7th, 2021

INTRODUCTION 

The purpose of this document is to ensure your VSA server(s) is prepared to receive the VSA release patch, which contains critical security fixes.  

Before you restore full connectivity between Kaseya VSA server(s) and deployed agents, there are certain steps that we are recommending:  

• Ensure your VSA server is isolated 
• Check System for Indicators of Compromise (IOC)  
• Patch the Operating Systems of the VSA Servers 
• Using URL Rewrite to control access to VSA through IIS 
• Install FireEye Agent 
• Remove Pending Scripts/Jobs

Each of these steps is described in more detail below. 

Step 1 – Ensure your VSA server is isolated

Depending on where and how you host your VSA server, this process will vary between platforms. Before powering on the VSA server, please ensure that you isolate it from inbound and outbound traffic and segregate it from your main network. There are several ways to do this depending on your specific case.  Booting it up in Safe Mode could be one way to achieve that.  

If you host your server in AWS we recommend that you check the NIC network security group and the public inbound ports assigned to the application server. You should deselect/disable ports 80/443/5721 (or any other non-default Kaseya port you might have used). Leave only the ports enabled, which provide you access to the device. For example, port 3389 is the standard port for RDP. Keeping this enabled will allow you to RDP into the virtual machine (VM) and perform the next steps without exposing the VM to the internet. 

In order to disable Kaseya network communication, please stop the Kaseya Edge Services service and set it to Disabled.  

Step 2 – Check your System for Indicators of Compromise (IOC) 

Please make sure you have run the “Compromise Detection Tool” on your VSA server. If you have not already run this tool, it can be downloaded from here. Please ensure you follow the directions available via the link. 

• If, at this point, you are unsure about the results received, please contact our support team at helpdesk.kaseya.com, or send an email to support@kaseya.com. We will be happy to assist you. 
• For information about the detection tool, see the appendix below. 

Step 3 – Patching the Operating Systems of the VSA Servers 

For the following steps we require internet access, so if your machine is completely isolated from the internet, please restore “Outbound” internet connectivity now. Before doing so please double check that the Kaseya Edge Services service is still disabled. 

• Apply the latest Microsoft patches to both the Windows operating system and SQL server. Please verify that your SQL server is on the latest available patch. This page tells you about the latest available patch for the product you use: https://docs.microsoft.com/en-us/sql/database-engine/install-windows/latest-updates-for-microsoft-sql-server?view=sql-server-ver15 

This process might require a few reboots, depending on how many outstanding patches there are. Please ensure no Windows or SQL patches remain outstanding. 

Step 4 – Using URL Rewrite to control access to VSA through IIS and Firewall Rules

*Updated July 11th 2021*
Based on customer feedback, we have made changes to the IIS rewrite tool in order to give customers more control of their environments using their firewalls. Please rerun the tool referenced below – it will automatically update the IIS configuration even if you ran the tool previously.

The purpose of this step is to control access to your VSA server and only allow the necessary access to the system User Interface (UI). This MUST be done on the VSA server.

Part A - There is a tool available for download, which helps automate this configuration process. You can obtain the tool at: https://app.box.com/s/1yel8q8nw4sxpujbhtudaqapk5yd84qk 

Once this is completed, you should expect the following:

Access to your VSA UI will block certain communication from the outside world (only what is necessary) on inbound port 5721.

Part B – Configure your firewall to only allow inbound traffic to the VSA on port 5721 (the agent port).
This will require your VSA users to be on the local network or VPN’d into the local network reducing the attack surface.

Given your unique environmental needs, you can create rules in your firewall to also allow port 443 inbound traffic to your VSA where required. For example, certain IP addresses for integrations or locations where you must allow access to the web GUI.

This provides customers the flexibility to use their existing firewall to control access to the VSA web GUI to meet their unique requirements while securing the agent port (5721) IIS access which is required for agents and some communications.

Please review a list of popular integrations that you may wish to whitelist at https://helpdesk.kaseya.com/hc/en-gb/articles/4403869952657 

Step 5 – Install FireEye Agent  

Kaseya is providing complimentary licenses of FireEye Endpoint Security agents for each customer’s VSA Server(s). 

To initiate this process, please send an email to fireeye@kaseya.com and be sure to include the following information: 

• Your Company Name 
• Your Mobile Number (only to contact you in case the information is not complete) 
• VSA Device Role (Web Server or Database Server) 
• If you have a split server configuration, you will need to install FireEye on both the Web Server and the SQL server. Please provide this information for both servers. 
• The hostname of the Server  
• Your Domain Name (for example, kaseya.com) 
• Internal IP Address 
• Egress (External) IP Address (Please use this URL on the server itself to confirm your gateway IP - https://www.whatismypublicip.com/) 
• VSA Agent Port (By default it is 5721, please provide your port number if you changed this) 

Once Kaseya receives your email, our teams will contact FireEye to have the agents provisioned, and will then reach out to you with instructions, and a link to the installer. 

Please ensure you send us all the information requested above as it is necessary for the FireEye agent to function properly.   

Do not start up your VSA Application until this VSA patch has been applied! 

APPENDIX 

Detection Tool Information 

This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present. 

The latest version searches for the indicators of compromise, data encryption, and the REvil ransom note. We recommend that you re-run this procedure to better determine if the system was compromised by REvil. 

This continues to be enhanced, so please always refer to the download site for the latest version. 

From the review of data provided by clients, we have identified IOCs. We are providing the following IOC information to aid our customers and security researchers in their investigations.  Kaseya’s investigation is ongoing and, as such, this information is subject to change.  

Network IOCs 

The following IP addresses were seen accessing VSA Servers remotely to perform the attack sequence: 

35.226.94[.]113 
161.35.239[.]148 
162.253.124[.]162 

Endpoint IOCs 

The following files were used as part of the deployment of the encryptor: 

Web Log IOCs 

The following are excerpts from the IIS access logs of a compromised VSA server. They depict a sequential series of HTTP requests that the threat actor made to perform their attack. If this sequence of requests is present in the IIS logs of a VSA server, it suggests the threat actor either attempted to or successfully used it to perform their attack. 

POST /dl.asp curl/7.69.1 
GET /done.asp curl/7.69.1 
POST /cgi-bin/KUpload.dll curl/7.69.1 
GET /done.asp curl/7.69.1 
POST /cgi-bin/KUpload.dll curl/7.69.1 
POST  /userFilterTableRpt.asp           curl/7.69.1 

Based on what most customers tend to use, here are some areas within the VSA you can consider looking into to make sure that everything is as you would expect it to be: 

• Agent count 
• Policy Management 
• Agent procedures 
• Reports 
• Users (with their Roles and Scopes) 
• Views 
• While the above-linked tool checks all the below, if you wish to perform a manual verification, please check the following. Search the Kaseya installation directory (e.g. C:\Kaseya\Webpages for the following files: UserFilterTableRpt.asp and userFilterTableV6KES.asp.  
  Note:  Please be sure to purge any copies if found. 
• Search IIS logs for references to UserFilterTableRpt.asp . You can find the IIS logs typically under C:\inetpub\logs\LogFiles\W3SVC1 . Order the directory content by ‘Date modified’ and open the log files modified between the 2nd and the 7th of July. Search the content for the above filenames or any items listed under Web Log Indicators.
• Search for Kaseya\webpages\managedfiles\vsaticketfiles\agent.crt. If found, please archive this file into a password-protected zip file and submit it to us via the support desk including the password applied. Once submitted, please purge the artifact from the hard drive. 
• Search for Kaseya\webpages\managedfiles\vsaticketfiles\agent.exe . If found, please archive this file into a password-protected zip file and submit it to us via the support desk including the password applied. Once submitted, please purge the artifact from the hard drive.  
  IMPORTANT: DO NOT EXECUTE this file as it can be an exploit.