On Premises VSA Startup Readiness Guide - July 7th, 2021

INTRODUCTION 

The purpose of this document is to ensure your VSA server(s) is prepared to receive the VSA release patch, which contains critical security fixes.  

Before you restore full connectivity between Kaseya VSA server(s) and deployed agents, there are certain steps that we are recommending:  

  • Ensure your VSA server is isolated 
  • Check System for Indicators of Compromise (IOC)  
  • Patch the Operating Systems of the VSA Servers 
  • Using URL Rewrite to control access to VSA through IIS 
  • Install FireEye Agent 
  • Remove Pending Scripts/Jobs

Each of these steps is described in more detail below. 

Step 1 – Ensure your VSA server is isolated

Depending on where and how you host your VSA server, this process will vary between platforms. Before powering on the VSA server, please ensure that you isolate it from inbound and outbound traffic and segregate it from your main network. There are several ways to do this depending on your specific case.  Booting it up in Safe Mode could be one way to achieve that.  

If you host your server in AWS we recommend that you check the NIC network security group and the public inbound ports assigned to the application server. You should deselect/disable ports 80/443/5721 (or any other non-default Kaseya port you might have used). Leave only the ports enabled, which provide you access to the device. For example, port 3389 is the standard port for RDP. Keeping this enabled will allow you to RDP into the virtual machine (VM) and perform the next steps without exposing the VM to the internet. 

In order to disable Kaseya network communication, please stop the Kaseya Edge Services service and set it to Disabled 

Step 2 – Check your System for Indicators of Compromise (IOC) 

Please make sure you have run the “Compromise Detection Tool” on your VSA server. If you have not already run this tool, it can be downloaded from here. Please ensure you follow the directions available via the link. 

  • If, at this point, you are unsure about the results received, please contact our support team at helpdesk.kaseya.com, or send an email to support@kaseya.com. We will be happy to assist you. 
  • For information about the detection tool, see the appendix below. 

Step 3 – Patching the Operating Systems of the VSA Servers 

For the following steps we require internet access, so if your machine is completely isolated from the internet, please restore Outbound internet connectivity now. Before doing so please double check that the Kaseya Edge Services service is still disabled. 

This process might require a few reboots, depending on how many outstanding patches there are. Please ensure no Windows or SQL patches remain outstanding. 

Step 4 – Using URL Rewrite to control access to VSA through IIS and Firewall Rules

*Updated July 11th 2021*
Based on customer feedback, we have made changes to the IIS rewrite tool in order to give customers more control of their environments using their firewalls. Please rerun the tool referenced below – it will automatically update the IIS configuration even if you ran the tool previously.

The purpose of this step is to control access to your VSA server and only allow the necessary access to the system User Interface (UI). This MUST be done on the VSA server.

Part A - There is a tool available for download, which helps automate this configuration process. You can obtain the tool at: https://app.box.com/s/1yel8q8nw4sxpujbhtudaqapk5yd84qk 

Once this is completed, you should expect the following:

Access to your VSA UI will block certain communication from the outside world (only what is necessary) on inbound port 5721.

Part B – Configure your firewall to only allow inbound traffic to the VSA on port 5721 (the agent port).
This will require your VSA users to be on the local network or VPN’d into the local network reducing the attack surface.

Given your unique environmental needs, you can create rules in your firewall to also allow port 443 inbound traffic to your VSA where required. For example, certain IP addresses for integrations or locations where you must allow access to the web GUI.

This provides customers the flexibility to use their existing firewall to control access to the VSA web GUI to meet their unique requirements while securing the agent port (5721) IIS access which is required for agents and some communications.

Please review a list of popular integrations that you may wish to whitelist at https://helpdesk.kaseya.com/hc/en-gb/articles/4403869952657 

Step 5 – Install FireEye Agent  

Kaseya is providing complimentary licenses of FireEye Endpoint Security agents for each customer’s VSA Server(s). 

To initiate this process, please send an email to fireeye@kaseya.com and be sure to include the following information: 

  • Your Company Name 
  • Your Mobile Number (only to contact you in case the information is not complete) 
  • VSA Device Role (Web Server or Database Server) 
  • If you have a split server configuration, you will need to install FireEye on both the Web Server and the SQL server. Please provide this information for both servers. 
  • The hostname of the Server  
  • Your Domain Name (for example, kaseya.com) 
  • Internal IP Address 
  • Egress (External) IP Address (Please use this URL on the server itself to confirm your gateway IP - https://www.whatismypublicip.com/) 
  • VSA Agent Port (By default it is 5721, please provide your port number if you changed this) 

Once Kaseya receives your email, our teams will contact FireEye to have the agents provisioned, and will then reach out to you with instructions, and a link to the installer. 

Please ensure you send us all the information requested above as it is necessary for the FireEye agent to function properly.   

6 – Remove Pending Scripts/Jobs

Prior to startup, we recommend you clear out any pending VSA procedures/scripts/jobs that accumulated since the shutdown. Download the script that we have provided with the instructions on the following link: https://helpdesk.kaseya.com/hc/en-gb/articles/4403843938321

 

7 - VSA SQL Database Assessment

In response to the July VSA Security Incident, Kaseya has created a PowerShell script that connects to your local SQL Instance and generates an HTML report of important data points that Kaseya has deemed valuable to review. The inclusion of these data points is not directly correlated to known Indicators of Compromise (IOC) but rather an audit of the user, procedure, and agent connection information to provide assurance of the state of the VSA prior to the restarting of VSA Servers.

Please note that we strongly recommend that you do NOT give your VSA Server internet access while running these scripts. Ensure your VSA remains OFFLINE and associated services are stopped until you receive an official update from Kaseya.

Please make sure you have run the "VSA SQL Audit Report" tool on your VSA Server. If you have not already run this tool, it can be downloaded from here. The link contains instructions and a sample report.

If you need assistance or have run the report and suspect your VSA application has been compromised, please contact our support team at helpdesk.kaseya.com, or send an email to support@kaseya.com. We will be happy to assist you.

Now that you have completed these steps, you are ready to install the patch when released. We will provide you details for obtaining the patch prior to the release.

Do not start up your VSA Application until this VSA patch has been applied! 

 

APPENDIX 

 

Detection Tool Information 

This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present. 

The latest version searches for the indicators of compromise, data encryption, and the REvil ransom note. We recommend that you re-run this procedure to better determine if the system was compromised by REvil. 

This continues to be enhanced, so please always refer to the download site for the latest version. 

From the review of data provided by clients, we have identified IOCs. We are providing the following IOC information to aid our customers and security researchers in their investigations.  Kaseya’s investigation is ongoing and, as such, this information is subject to change.  

Network IOCs 

The following IP addresses were seen accessing VSA Servers remotely to perform the attack sequence: 

35.226.94[.]113 
161.35.239[.]148 
162.253.124[.]162 

 

Endpoint IOCs 

The following files were used as part of the deployment of the encryptor: 

Filename 

MD5 Hash 

Function 

cert.exe 

N/A – Legitimate File with random string appended 

Legit certutil.exe Utility 

agent.crt 

939aae3cc456de8964cb182c75a5f8cc 

Encoded malicious content 

agent.exe 

561cffbaba71a6e8cc1cdceda990ead4 

Decoded contents of agent.crt 

mpsvc.dll 

a47cf00aedf769d60d58bfe00c0b5421 

Ransomware Payload  


Web Log IOCs
 

The following are excerpts from the IIS access logs of a compromised VSA server. They depict a sequential series of HTTP requests that the threat actor made to perform their attack. If this sequence of requests is present in the IIS logs of a VSA server, it suggests the threat actor either attempted to or successfully used it to perform their attack. 

POST /dl.asp curl/7.69.1 
GET /done.asp curl/7.69.1 
POST /cgi-bin/KUpload.dll curl/7.69.1 
GET /done.asp curl/7.69.1 
POST /cgi-bin/KUpload.dll curl/7.69.1 
POST  /userFilterTableRpt.asp           curl/7.69.1 

Based on what most customers tend to use, here are some areas within the VSA you can consider looking into to make sure that everything is as you would expect it to be: 

  • Agent count 
  • Policy Management 
  • Agent procedures 
  • Reports 
  • Users (with their Roles and Scopes) 
  • Views 
  • While the above-linked tool checks all the below, if you wish to perform a manual verification, please check the following. Search the Kaseya installation directory (e.g. C:\Kaseya\Webpages for the following files: UserFilterTableRpt.asp and userFilterTableV6KES.asp 
    Note:  Please be sure to purge any copies if found. 
  • Search IIS logs for references to UserFilterTableRpt.asp . You can find the IIS logs typically under C:\inetpub\logs\LogFiles\W3SVC1 . Order the directory content by ‘Date modified’ and open the log files modified between the 2nd and the 7th of July. Search the content for the above filenames or any items listed under Web Log Indicators.
  • Search for Kaseya\webpages\managedfiles\vsaticketfiles\agent.crt. If found, please archive this file into a password-protected zip file and submit it to us via the support desk including the password applied. Once submitted, please purge the artifact from the hard drive. 
  • Search for Kaseya\webpages\managedfiles\vsaticketfiles\agent.exe . If found, please archive this file into a password-protected zip file and submit it to us via the support desk including the password applied. Once submitted, please purge the artifact from the hard drive 
    IMPORTANT: DO NOT EXECUTE this file as it can be an exploit. 
Was this article helpful?
3 out of 3 found this helpful
Have more questions? Contact us