In order to remove specific hosts from VM Backup that are added with the vCenter installation, you can simply click on [Edit] under one of the hosts:



Then you will be taken to a vCenter details screen, showing you the hosts with the hosts within the environment too. At this point you can click on the [Remove Host] button: