Why does patching take so long on some of my endpoints?
The length of time for any patch window can vary significantly. Primary factors include:
· The number and size of the patches
· The file source configuration and the resources available on the file source
· Frequency of patching
· Reboot Action definition
Number of patches and File Source config
File source plays a big part. If the file source is “from internet” the patches download via the Windows Update Agent (WUA). The time for Kaseya to patch the machine is pretty similar to the amount of time it would take if you launched Control Panel > Windows Update > Install Updates on the local machine. This is because the methods to install are almost identical (there are a few additional steps from Kaseya’s side, including the reboot action and rescan, but these don’t add much time – maybe a couple of minutes). WUA downloads the patch components that the individual machine needs. This may be MUCH less data than the full distributable version of a patch. Let’s use an example of a service pack, which is a bundle of updates, some of which may been installed on the machine already, others which may not be required for a particular machine. WUA may need to download just a few KB or MB or a multi-GB patch for a machine – it doesn’t need to download any components that are already present on the workstation and does not need to download anything the machine does not qualify for (like multiple languages). HOWEVER, when the file source is anything other than “from internet”, the FULL distributable version of the patch will be downloaded to the machine. This will include the FULL patch, which will include ALL of the rolled-together updates, all languages, etc. This can be several GB of data.
In addition, when the file source is other than “from internet”, the file must be downloaded to the defined file share server, then transferred from the server to the endpoint. That means that a large file needs to be transferred twice, which can theoretically double the amount of time it takes to get the patch onto the machine. If the file source is defined as a UNC path via the system server (option available only on premise), the file has to transfer THREE times (from the internet to the KServer, from the KServer to the file share server, then finally from the file share server to the endpoint). That’s going to take time. If the file source machine is busy with other tasks, each request from an endpoint to the server must wait for those other tasks to complete before the next one will execute. If 10 machines are requesting patches from a server at the same time, the first machine may get the patches very quickly, but the 10th machine has to wait for the prior nine before #10’s request is processed.
The frequency of patching plays a role. Cycles that run weekly will generally have fewer patches to install, though the cycle after “Patch Tuesday” may have a larger percentage of patches than other weekly cycles since this is the day MS releases the bulk of their patches. (Patch Tuesday is the second Tues of each month.) However, if you’re running a patch cycle only once per month, there’s a larger bulk of patches to install and therefore will increase overall run time of the cycle.
Reboot Action is the last major factor. IF the file source is not “from internet”, then it is possible that a patch cycle may require multiple install scripts. This occurs when:
· One or more patches are flagged by MS as requiring a reboot OR
· At least one patch is the batch is flagged as “Internet Based Install Only”
Kaseya will bundle together multiple patches into a single script. However, any patch flagged as requiring a reboot will get its own install script. Each install script will follow a reboot action after the script finishes. The next install script will fire after the reboot, continue with installations, and reboot at the end of that install script.
Let’s say you have three patches, each one has been flagged by MS as requiring a reboot. The patch cycle will include three install scripts. The preliminary patch scripts (verify credentials, etc.) will run, the first patch will be downloaded, the patch will be executed, then the reboot action will fire. After the reboot, a re-scan will run, the second patch will be downloaded and executed, then rebooted according to the reboot action. That reboot is followed by a re-scan, download and install of the third patch, then a final reboot action and re-scan. This final re-scan completes the cycle. However, if none of the three patches are flagged as requiring a reboot, Kaseya will bundle all three patches into a single install script. The patches will download, execute, reboot action will fire, then the post-reboot re-scan will close out the cycle.
Internet-only patches MUST be installed by WUA – the cannot be distributed through a file server. When the file source is other than “From internet” and some (but NOT all) patches for the cycle are internet only, the internet-only patches will be bundled into one install script and the distributable patches will be bundled into a separate install script. The distributable patches will be downloaded via the defined file source, executed, then a reboot action will fire. Post reboot, the rescan will run, then the next install script will fire. That script will install the internet-only patches via WUA, reboot action fires, and the post-reboot re-scan completes the cycle.
Of course, as with all processes, if the endpoint is doing other work (running other agent procedures or running a resource intensive process like a backup at the time of patching), this will certainly delay the patch cycle since Kaseya procedures run in serial, not parallel. Each procedure must wait for the previous procedure to complete before the next can fire. If a backup is running at the same time patch is scheduled to run, the backup must complete before patch can even begin.
The “fastest” way to install patches on an individual machine is to use the file source as “from internet.” This will download only the necessary components and install just like using the Windows Update function on the local machine. However, this option will force each individual machine to download directly from the internet. This might end up being more time consuming if there are a lot of machines running updates at the same time or if the network does not have sufficient bandwidth to support and manage downloads.
It is always possible that there is something artificially causing a delay in the cycle. These are almost always environmental or based on a less than desirable configuration. It’s quite possible the ‘delays’ are just due to the workload, and when that’s the case, it’s often possible to find alternatives to help the process be more efficient (whether that’s increasing the schedule distribution, allocating additional server resources, increasing available bandwidth, etc.).