Fixing MSDeploy Error: Web deployment task failed. (Root element is missing.)

In one of my MRCollective open source projects AzureWebFarm, we configure a set of Windows Azure Web Roles such that a change deployed to any of them using MSDeploy will automatically sync to the other servers. Up to and including MSDeploy v2 this worked fine, but in a Windows Server 2012 Azure server environment MSDeploy v3 returns the following exception (if deploying to a role with more than one instance):

Web deployment task failed. (Root element is missing.)

The more verbose version of this exception:

System.Xml.XmlException: Root element is missing.
     at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo(String res)
     at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
     at System.Xml.XmlReader.MoveToContent()
     at Microsoft.Web.Deployment.TraceEventSerializer.Deserialize(Stream responseStream, DeploymentBaseContext baseContext, DeploymentSyncContext syncContext)
     at Microsoft.Web.Deployment.AgentClientProvider.RemoteDestSync(DeploymentObject sourceObject, DeploymentSyncContext syncContext, Nullable`1 syncPass) at Microsoft.Web.Deployment.DeploymentObject.SyncToInternal(DeploymentObjectdestObject, DeploymentSyncOptions syncOptions, PayloadTable payloadTable, ContentRootTable contentRootTable, Nullable`1 syncPassId) at Microsoft.Web.Deployment.DeploymentObject.SyncTo(DeploymentProviderOptions providerOptions, DeploymentBaseOptions baseOptions, DeploymentSyncOptions syncOptions) at MSDeploy.MSDeploy.ExecuteWorker()

You might also observe the following exception on the server:

wmsvc.exe Error: 0 : ERROR_SERIALIZER_ALREADY_DISPOSED - The object 'Microsoft.Web.Deployment.TraceEventStreamSerializer' has already been disposed. 

Tracing this further showed that MSDeploy v3 appears to use multiple connections, and those connections were distributed to multiple servers in our test web farms for a single deployment, causing msdeploy to fail pretty miserably with these XML errors.

The solution was to ensure that all MSDeploy connections from the client end up at a single server in the web farm, rather than being distributed to multiple servers. To achieve this in Windows Azure, you can create a custom load balancer endpoint in ServiceDefinition.csdef - take a look at the loadBalancerProbe="WebDeploy" and the LoadBalancerProbe config setting in our example ServiceDefinition.csdef for reference. You can read more about custom load balancer endpoints in this post.

That custom probe hits a file we created called Probe.aspx. That file contains a really simple piece of code which attempts to get an Azure Blob Storage lease on a blob we specify. Only one instance will ever be able to get that lease at a time, and if it ever fails, another instance will pick it up, ensuring this solution still gives us redundancy for deployments.

Should an instance successfully get the lease, it will return a HTTP OK status on that load balancer endpoint, and the Azure Fabric will know that it's fine to route web deploy requests to that instance. All the other instances will fail to get the lease, and return a HTTP error status back to the endpoint, meaning the Azure Fabric will not send any web deploy requests to all other instances (ensuring it all works happily with MSDeploy v3).