This post describes an error while configuring HA with nested ESXi 6.5 due to «Secure boot».
I hope it will help others lab users.

I got the following error while configuring vSphere HA for a cluster of nested ESXi hosts in a fresh new lab.
Cannot install the vCenter Server agent service. “Unknown installer error”

There are many VMware KBs and others blog post regarding this error.
However none of them was matching this case.

Update 20/10/2017: More details in this KB ESXi 6.5 host with secure boot enabled triggers “vSphere HA host status” alarm

Lab details

One physical server upgraded to “VMware ESXi 6.5.0a”
Nested ESXi VMs “VMware ESXi 6.5.0a”
vCenter VMs “vCenter Server Appliance 6.5.0a”

Nested ESXi VM settings:
Created from the GUI
Compatible with: ESXi 6.5 and later
Guest OS Family: Other
Guest OS Version: VMware ESXi 6.5
Virtual Hardware:
CPU and Memory increased to 2 and 8GB
CPU>Hardware Virtualization: Tick the box “Expose hardware assisted virtualization to the guest OS”

With the previous settings you will get automatically by default
VM Options > Boot Options
Firmware: EFI (recommended)
Secure boot: Ticked box for “Secure Boot” (EFI boot only)

Error troubleshooting

Create a cluster (No settings)
Add a nested ESXi host to this cluster
Enable HA for the cluster
Error for the host during the “Configuring vSphere HA” task:
Cannot install the vCenter Server agent service. “Unknown installer error”

From the nested ESXi logs i have extracted these lines at the time of the error:
2017-03-04T22:37:07Z esxupdate: 68200: Transaction: INFO: Final list of VIBs being installed: VMware_bootbank_vmware-fdm_6.5.0-4944578
2017-03-04T22:37:07Z esxupdate: 68200: imageprofile: INFO: Adding VIB VMware_bootbank_vmware-fdm_6.5.0-4944578 to ImageProfile ESXi-6.5.0-20170104001-standard
2017-03-04T22:37:07Z esxupdate: 68200: Transaction: ERROR: Secure Boot enabled: Cannot skip signature checks. Installing unsigned VIBs will prevent the system from booting

The “What’s New in VMware vSphere® 6.5” contains the following information:
Virtual Machine Secure Boot
Virtual machines must be booted from the EFI firmware to enable Secure Boot. EFI firmware supports Windows,Linux, and nested ESXi.

So it seems to be linked to secure boot with nested ESXi hosts and only if the physical server is running ESXi 6.5.

Workaround

Shutdown the nested ESXi server.
Edit the VM settings and untick the box in VM Options > Boot Options > Secure boot
Restart the VM/nested ESXi
Put ESXi host in maintenance mode
Move out of the cluster
Move back in the cluster
Exit maintenance mode
HA is finally configured 😉

However the following warnings and error were displayed in the “host events” before getting HA configured:
Attempting to install an image profile bypassing signing and acceptance level verification. This may pose a large security r
SECURITY ALERT: Installing image profile ‘(Updated) ESXi-6.5.0-20170104001-standard’ with acceptance level checking disabled.
Alarm ‘Host error’ on 10.0.11.14 triggered by event 3074 ‘Issue detected on 10.0.11.14 in Lab1: Attempting to install an image profile bypassing signing and acceptance level verification. This may pose a large security r

Conclusion

Disabling secure boot for the nested ESXi VM is just a workaround.
I do not understand why, while configuring HA, the installation of agents is bypassing signing and acceptance level verification.
Actually, maybe a physical ESXi server using secure boot will have the same issue.