The expectation that our customers have is that when scanning the same application multiple times, Veracode will return similar results. This is true for the majority of our scans. However, in rare instances, there can be inconsistencies in the results we return, which can manifest in the form of changing numbers of flaws, or changing locations of a given flaw. This can lead to customers wondering about the causes of this behavior, and if it is possible to predict the results of performing static analysis.

While this seems like a straightforward question, it is actually has quite a complicated answer, and relates to some of the most fundamental questions in mathematics and computer science. In summary, there is a class of problem that has not been proven to be efficiently solvable (https://en.wikipedia.org/wiki/P_versus_NP_problem). The closest that math has been able to come is approximating the answer, and static program analysis - which is essentially an implementation of several core problems from graph theory (https://en.wikipedia.org/wiki/Graph_theory), is one of these types of problems.

Static analysis as performed by Veracode, however, is not done for the purity of mathematics; it has a specific goal, which is to perform analysis of an application for security flaws in a timely manner. In this practical implementation of complex mathematical concepts, we are always looking for ways to balance the coverage of our scans with the demands of our customers such as speed and flexibility with preparing their applications.

When customer scans exhibit inconsistencies between scans, the reason can usually be attributed to two things: the trade-offs between speed & coverage that we make in order to deliver results in a reasonable time frame, and the sensitivity of static analysis to even small changes to its input. This sensitivity can be difficult to predict in advance, but a summary explanation of it is that there is a non-linear relationship between the complexity of the model that our static analysis builds of an application, and way that the application processes input.

When encountering this situation, we encourage customers to work with Veracode to determine if their scenario matches one of the circumstances above. Our support team can initiate an investigation into scan behavior, during which our engineering team will research the issue to determine if this situation could be used to improve our scan performance in the future. And our Application Security Consulting team can help you review the scan results, which could reveal how the flaws identified in the scan could be used to improve the scan results the next time it's submitted.

Veracode is also introducing several enhancements to our static analysis process aimed at both improving scan consistency and providing more insight into flaws and scan behavior. We are constantly refining our scanning algorithms, and will be rolling out several changes that improve scan consistency throughout the course of 2017. We are also introducing platform features aimed at highlighting the differences between two scans of an application, providing users insight how their choices may affect static scan results.

Veracode remains committed to helping our customers secure their software through use of our industry leading static analysis service. Our monthly cycle of continuous improvements helps us be responsive to customer feedback, and our services team helps provide guidance to our customers and ensure that their feedback is passed onto our product team for action.

Two submission of exactly the same input should produce exactly the same results. The most common way this problem occurs is that there is in fact some change, but the change is small or seemingly insubstantial from the customer's point of view.

As @Bill T (Veracode)​  mentioned, our analysis is designed to give results in a human usable time frame and therefore some sacrifices have to be made. One of those sacrifices is that small changes can have non-linear effects on the results. This is a constant source of development and improvement for us.