We developed some best practices for tool evaluation and selection over the years. Ideas for tools come from everywhere - within Security, within Development, and from lots of cold calling from prospective vendors. It turns out it does not matter where the idea for a product or tool originated, as long as the evaluation process and selection process are consistent for all the other similar tools in that genre (SAST, SCA, DAST, etc.).

What we did was evaluate five SAST vendors using the Web Application Security Scanner Evaluation Criteria from WASC (there are newer ones out there too). The first pass eliminated the poorest choices and the stronger ones were then asked to do a 'bake off' using the same representative application so we could measure actual scan times, outputs, etc. This was all done with the development teams who were expected to use the tool on a regular basis, so they had a voice in the selection process. We would up with the best choice (for us) based on real data that proved it worth to us.

You can do a similar activity with any other tool (security tool or not) and gain the same advantages. It works REALLY well with developer training and by including dev teams in the selection of that training, they took to it quickly since they were part of the effort from the beginning. For training, what we did was prepare a questionnaire of sorts for each vendor to complete, asking for things we need and want and gain an understanding of how the training or platform will integrate with internal employee training tracking systems (like SumTotal or SABA). After we got back their submissions, the strongest ones were asked to do a 1 hour WebEx to show us what they had and we were able to focus the time on seeing the product in action, rather than ask questions we already had them answer. Again, the best ones bubbled to the top and the final decision was left to the would-be learners by vote.

These processes are reusable for virtually any development-related technology, and as the same people participate in these evals and selection activities, they begin to get REALLY good and efficient at it. It turns out it's also a pathway into Enterprise Architecture if one wants to do that for a career.

What it all boils down to is involving development team leads and members from the start to reduce resistance to change or added processing so they're really the ones 'owning' the implementation which greases the skids to quick adoption and use.

I think identifying the right tools and process comes with experience, listening to other vendors, attending security conferences, for me attending the SANS Web Application Pen Testing class and RSA this year really helped start to develop a program by opening my eyes to the exposures out there and how important it is to have a tool bag full of stuff. We have settled into Veracode (SAST), Burpsuite Pro and Zap (Dynamic), along with several open source tools to do ocasional internal pen testing. memory scanning and open source scanning are two areas we have not explored yet, but are important to consider as well in your appsec program.