vProtect error: The server refused connection

SUMMARY

This article discusses how to correct the error “The server refused connection” when trying to backup VMware guest using vProtect.

ISSUE

After the vProtect job starts, it cannot route to the ESX Host server that current has the VMware guest.

Purpose

To correct error “The server refused connection” when trying to backup VMware guest using vProtect.

Symptoms / Description

----- VProtect Messages ----

VixDiskLib_Open failed for disk [DS4] Formfast/Formfast.vmdk: The server refused connection.

----- End VProtect Messages ----

NOTE: Required ports are 443 & 902 however these are typically only opened on API requests. If VMWare is receiving the backup request then above is not the issue. Dig further

RESOLUTION

vCenter and managed ESXi servers

If ESXi servers belong to a vCenter and both are accessible on the network, Unitrends recommends that you add the vCenter and its ESXi servers to the appliance. This enables the appliance to contact the vCenter for management operations (including vMotion support) and to directly contact the ESXi servers for backup and recovery, potentially improving performance by reducing network traffic around the vCenter server

HOST TABLE IS NO LONGER REQUIRING TO HAVE INDIVIDUAL ESX HOSTS ADDED. HOWEVER ALL HOSTS MUST BE ABLE TO RESOLVE VIA DNS/NSLOOKUP.

Verify all the associated ESX host(s) are added in the UI & confirmed listed in bp.vmware_connections database table.

bpdb=# select * from bp.vmware_connections;
connection_id | esx_uuid | vcenter_uuid | login | password | address
---------------+--------------------------------------+--------------------------------------+-----------------------------+------------------+--------------
1 | | 16B499FB-5EAF-4B2E-800C-A7094F6C9945 | administrator@vsphere.local | *************** | 172.52.7.20
17 | a0fef3ea-7000-e487-0000-00000000000f | | root | *************** | 192.5.4.231
18 | a0fef3ea-7d5b-e311-0000-00000000001f | | root | *************** | 192.168.4.232
19 | a0fef3ea-7000-e487-0000-00000000000e | | root | *************** | 192.168.4.233

Additional Troubleshooting

grep -i "902" /usr/bp/logs.dir/vprotect_backup.log

--SAMPLE OUTPUT to review to see if the port is refusing connection--
2018-10-26 10:47:43.191 +0100 : D vix_debug: NBD_ClientOpen: attempting to create connection to vpxa-nfc://[VNAS] vmware/vm.vmdk@<ip-address>:902
2018-10-26 10:47:45.200 +0100 : D vix_debug: NBD_ClientOpen: Couldn't connect to <ip-address>:902 Connection terminated by server
2018-10-26 10:47:45.200 +0100 : D vix_debug: DISKLIB-DSCPTR: : "vpxa-nfc://[VNAS] vmware/vm.vmdk@<ip-address>:902" : Failed to open NBD extent.
2018-10-26 10:47:45.201 +0100 : D vix_debug: DISKLIB-LINK : "vpxa-nfc://[VNAS] vmware/vm.vmdk@<ip-address>:902" : failed to open (NBD_ERR_NETWORK_CONNECT).
2018-10-26 10:47:45.201 +0100 : D vix_debug: DISKLIB-CHAIN : "vpxa-nfc://[VNAS] vmware/vm.vmdk@<ip-address>:902" : failed to open (NBD_ERR_NETWORK_CONNECT).
2018-10-26 10:47:45.201 +0100 : D vix_debug: DISKLIB-LIB : Failed to open 'vpxa-nfc://[VNAS] vmware/vm.vmdk@<ip-address>:902:902' with flags 0x1e NBD_ERR_NETWORK_CONNECT (2338).
2018-10-26 10:47:45.201 +0100 : D vix_debug: VixDiskLib: Failed to open disk vpxa-nfc://[VNAS] vmware/vm.vmdk@<ip-address>:902!52 7e f4 94 48 37 9a a3-19 03 e6 ca 83 d1 6d ab. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 4799.

Now Verify the port is closed using the same IP to the vCenter.

nmap -p 902 <ip-address>

--SAMPLE OUTPUT to verify the VMWare port is closed; this port is indicating this is a VMware issue; see notes below.--
Starting Nmap 5.51 ( http://nmap.org ) at 2018-10-26 11:17 BST
Nmap scan report for vcenter-app.local (<ip-address>)
Host is up (0.00026s latency).
PORT STATE SERVICE
902/tcp closed iss-realsecure
MAC Address: <MAC Address> (VMware)

--NOW suggest a scheduled reboot of the vCenter as it has been reported to have resolved the issue in some cases and if that does not resolve the issue they will need to reach out to VMWare for a resolution on the failing port.--
Additionally you may verify the version of the VMWare; this one is on version 6.0 & additionally use the same steps to review port 443.

psql

[root@PCL-HAT-UEB1 ~]# psql
psql (9.2.5)
Type "help" for help.

select * from bp.vmware_vcenter_servers ;

CAUSE

Critical VMware port 902 and/or 443 is failing to remain open resulting in a connection refusal.

NOTES

--VMware vSphere 4 - ESX and vCenter Server external Link--
https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-171B99EA-15B3-4CC5-8B9A-577D8336FAA0.html

VMware vSphere 4 - ESX and vCenter Server - Required Ports
-Port 902-
The default port that the vCenter Server system uses to send data to managed hosts. Managed hosts also send a regular heartbeat over UDP port 902 to the vCenter Server system. This port must not be blocked by firewalls between the server and the hosts or between hosts.

-Port 443-

The default port that the vCenter Server system uses to listen for connections from the vSphere Client. To enable the vCenter Server system to receive data from the vSphere Client, open port 443 in the firewall.

The vCenter Server system also uses port 443 to listen for data transfer from the vSphere Web Access Client and other SDK clients.

If you use another port number for HTTPS, you must use <ip-address>:<port> when you log in to the vCenter Server system.