Server manager is a great tool to administer and monitor servers remotely, and it works well with containers based on Windows Server Core. You need to take a similar approach to the IIS management console, configuring a user in the container with administrator access and then connecting from the host.
Just as with IIS, you can add a script to the image, which enables access so you can run it when you need it. This is safer than always enabling remote access in the image. The script just needs to add a user, configure the server to allow remote access from administrator accounts, and ensure the Windows Remote Management (WinRM) service is running:
net user serveradmin "s3rv3radmin*" /addnet localgroup "Administrators" "serveradmin" ...
