Installing the X Window System on your server nodes will consume resources you might rather give to a virtual machine, as well as increase the security risk to your servers. Instead, consider creating a server to control your nodes. That system can be a desktop system with a full X Window installation. Install the libvirt tools and virt-manager to that system and use a URI connection string to connect to your client. Using SSH will keep the connection secure. Use the VIRSH_DEFAULT_CONNECT_URI environment variable to hold the connection string. virsh will look there before trying the default connection URIs.
You may have problems connecting with virt-manager over SSH. If you launch virt-manager as an unprivileged user, it will usually try to switch you to the root user. This will clear environment settings like your ssh-agent settings. If prompted for the root password, select the “run unprivileged” option. This will preserve your current SSH settings. You don’t need to run as root because you aren’t controlling a hypervisor on the local system. For privileged access you will connect to a privileged user on the remote node.
At this time, virt-manager does not support creating new guests on remote systems. But you can manage your system just fine.