tl;dr: If you break TLS negotiation between the SMP and agent remove the RSA key negoatiation cipher suites!
It all started with what was supposed to be a simple change and turned out to be a serious issue (for the few affected computers).
On Tuesday whilst working on TaskServer communication issue with a customer we reviewed the Symantec Agent Communication profile in their environment and agreed to change it from TLS 1.0 to TLS 1.1 and TLS 1.2.
This is a change that I'd done before and it never caused any issue in test or product environments.
However that days things quickly turned from bad to worse. The change caused the agent on the SMP and on the task server we were troubleshooting to fail to communicate with the SMP.
We quickly reverted the changes on the SMP and got busy resolving the issue, as it possibly impacted a few more computers. The error from the agent logs was clear:
"The client and server cannot communicate, because they do not possess a common algorithm"
We first tried to deleted the Agent Storage (the agent secure store under c:\ProgramData\Symantec\Symantec Agent\Ldb" as the communication profile data received from the SMP is stored in that location rather than the policy file.
This didn't really work and generated a few issues on its own, so we switched to an alternate option, that we found via Microsoft MSDN, after searching for the above string [1]:
I recently encountered the following issue. My customer was trying to use one of the following ciphers between a client & server:
- TLS_RSA_WITH_AES_128_CBC_SHA256
- TLS_RSA_WITH_AES_256_CBC_SHA256
- TLS_RSA_WITH_AES_128_CBC_SHA
- TLS_RSA_WITH_AES_256_CBC_SHA
When attempting to use any of the above ciphers, the server was failing. If they used any of the following algorithms, it worked fine:
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
It turns out you can encounter this error (SEC_E_ALGORITHM_MISMATCH) if the server’s certificate KeySpec poperty is set to 2 (AT_SIGNATURE). A certificate with this KeySpec is only good for signing. This is why the (EC)DHE Key Exchange Algorithms worked.
The cipher suites of choice using the RSA key exchange is for the KeySpec to be set to 1 (AT_KEYEXCHANGE).
Once this change was made, the server was able to use the first 4 specified algorithms.
The MSDN entry also had a pointer to some software [2], so we tested the tool and it helped us solved the issue by disabling the cipher suites that use RSA as key exchange. The alternate option of having the server X509 certificate regenrated not being valid in this case.
Here is the configuration we kept for the server:
[1] http://blogs.msdn.com/b/winsdk/archive/2014/05/01/encountering-quot-the-...