I deploy .net web app via azure pipelines onto IIS on an azure vm via the web deploy task.

To do this I had to initially configure the deployment group by running a script on the azure vm as explained here (see step 5):

You can uninstall the deployment group agent to remove the access:

1. Launch powershell,go to agent folder.
2. Run .\config.cmd remove to uninstall.

After it's done, it will remove the agent service and also VSTS group user below.

If you specify user not default, you can even directly delete the user from the machine to remove the permission.