Troubleshooting CRM actions and SPN
Recently we have started using custom actions. Basically they allow to fire workflows or plugins by simply sending specific organization request. More can be read here:
- Create your own actions - https://msdn.microsoft.com/en-us/library/dn481600.aspx
- Actions - https://technet.microsoft.com/en-us/library/dn531060.aspx
- Four Important Concepts Helpful in Understanding Actions in Dynamics CRM - http://www.powerobjects.com/2014/07/23/4-important-concepts-helpful-understand-actions-dynamics-crm/
Mostly when you face some problem with firing them errors are pretty straightforward and self-explaining. Such as
[Microsoft.Xrm.Sdk.OrganizationServiceFault]: This workflow cannot run because one or more child workflows it uses have not been published or have been deleted. Please check the child workflows and try running this workflow again.
or
[Microsoft.Xrm.Sdk.OrganizationServiceFault]: Workflow associated with this custom operation is not activated
But recently we have faced very weird error:
[Microsoft.Xrm.Sdk.OrganizationServiceFault]: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
After hours of troubleshooting without any real result, we found specific application server behind our load balancer that was giving that error. Related trace threw some light on the root of the problem:
>TrackingId: 8AFB7646, Depth: 3, Message: SandboxPlugin.Execute - calling Execute
...
>System.ServiceModel.Security.SecurityNegotiationException: Authentication failed on the remote side (the stream might still be available for additional authentication attempts). ---> System.Security.Authentication.AuthenticationException: Authentication failed on the remote side (the stream might still be available for additional authentication attempts). ---> System.ComponentModel.Win32Exception: The target principal name is incorrect
and on sandbox in event log we found well know error:
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server <service-account-name>. The target name used was HTTP/<app-server-name>. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Ensure that the target SPN is only registered on the account used by the server. This error can also happen if the target service account password is different than what is configured on the Kerberos Key Distribution Center for that target service. Ensure that the service on the server and the KDC are both configured to use the same password. If the server name is not fully qualified, and the target domain (<domain-name>) is different from the client domain (<domain-name>), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.
Solution is to add into SPNs list of service account missing application server.
That's the story of how one noteless misconfiguration during initial Dynamics CRM environment setup after half year wasted about 3 man-days.
Things to take away from that story:
- SPN configuration is very sensitive. Its misconfiguration may have a long story afterwards
- Workflows fired by actions run in sandbox
- CRM organization service communicates with sandbox via WCF