Episode 194 - Evaluating Security Product Vendors

In light of recent news about “Vendors Behaving Badly” we want to talk about how a security professional should evaluate vendors and their products.

Recent News:
Tanium exposed hospital’s IT while using its network in sales demos: https://arstechnica.com/security/2017/04/security-vendor-uses-hospitals-network-for-unauthorized-sales-demos/

Lawyers, malware, and money: The antivirus market’s nasty fight over Cylance: https://arstechnica.com/information-technology/2017/04/the-mystery-of-the-malware-that-wasnt/

1. There are so many different sources of information about vendors and their products. You owe it to yourself to evaluate not just the vendor but also each source of information.
   1. Analyst Firms: Gartner/Forrester/etc
      1. Always remember they take a very generic view using a notional enterprise as the standard.
      2. Current customer interviews are important but, remember, those customer contacts likely came from the vendor.
      3. The perception of “Pay for Play” is there no matter how much the firms want to squelch that.
      4. These tests presume a lot so make sure you understand what the conditions of the test were.
      5. The “Pay for Play” perception exists here too….
      6. The results of the testing aren’t specific but can help show outliers in a group
   2. 3rd Party Testing: NSS Labs, etc.
      1. Obviously your best and most relevant source of information. :-)
   3. Podcasts
      1. If you have developed a reliable network of peers you can reach out and ask folks. But, remember, buy them a beer for their troubles…
      2. Always remember perspective is everything. Some people just don’t like Company_Z and will always hate their products.
   4. Networking
2. Information Sources
   1. Start with 3rd party data and demos. This will determine if your requirements (you did write out your requirements, right?) are met by the product
      1. Do not allow the vendor to drive the definition of “success” in a PoC
      2. Try to break it. I mean REALLY try to break it.
      3. Remember during the PoC is going to be the best support and interaction you will ever get. If that sucks you might want to move along.
      4. Test all of your use cases. (you do have documented use cases, right?)
   2. Do a PoC (Proof of Concept).
3. Product Evaluation Rules
   1. Service providers such as penetration testers and MSSPs
4. Edge Cases