In March of this year, Cylance kicked-off their “Unbelievable Tour” to provide a head-to-head comparison of their endpoint protection product against industry leaders, including Symantec. Cylance scheduled demonstrations in 35 cities between March and April. At RSA in late April, they announced the results of their tests claiming that their product outperformed Symantec.
Certainly, we were surprised and curious about the test results so our Labs conducted their own tests to compare Symantec Endpoint Management to CylancePROTECT.
While different types of tests can result in different results, the Symantec Lab tests are structured to exhibit real operating environments where threats emerge from many different vectors. Singular static tests comparing detection rates from isolated environments may produce interesting results but have little bearing in reality. In Real World Tests, Cylance performed well but not as well as Symantec. However, in Prevalence Tests and FP Testing, the gap was significantly wider. Cylance has a high number of false positive indications which can send administrators chasing red herrings. This was to be expected given that Symantec uses more than one method (in fact up to 5 different techniques) to identify malware. High detection accuracy is something we take pride in. More importantly the rate of false negatives also higher than Symantec. A false negative is a scenario when a known malicious file is not detected by an antimalware software. No actions were taken by CylancePROTECT in response to known exploits. False negatives are a serious matter.
| Test Type | Total Samples Tested | Symantec Detections | Cylance Detections |
|---|---|---|---|
| Virus Total - PEEXE | 100 | 100 | 99 |
| Virus Total - MAC samples | 100 | 100 | 0 |
| Virsus total - Document Samples (Doc, Pdf, Xls, etc) | 100 | 84 | 0 |
| Virus Total - HTML files | 100 | 100 | 0 |
| Virus Total - Image files | 100 | 100 | 0 |
| Virus Total - Audio/Video files | 10 | 8 | 1 |
| False Positive Test | 20 | 0 | 3 |
| Exploit Test | 10 | 10 | 0 |
Another interesting factoid was that Cylance only scans PEEXE (program executable) file types. Standard document files such Doc and PDF files are not scanned. In some cases malware detected by Cylance remains running and active in memory. Quarantined malware files remain accessible to the end user. Cylance malware remediation is limited in functionality requiring additional remediation-capable anti-malware software, like SEP.
Our tests were run using latest versions of both products along with the standard (default) configurations. We would welcome independent testing by AV Test or similar 3rd party test organization. To date, no Cylance test results have been made available to the public from any standard test organizations.