Georges (Community Member) asked a question.

Issue with finding counts in reporting

 

​

I set up a dashboard to track the time-to-fix per application per severity, filtering out irrelevant data using a custom filter. Please see screenshot attached with application names obfuscated for obvious reasons (for your clarity: same color for rows belonging to one app).

​

Issue: For any given application, the data in this dashboard doesn't match the numbers of flaws when I go to "Triage Flaws" module for that application and apply the same filters. The orange example from the screenshot shows 1 Very High, 17 Highs and 163 Mediums in the dashboard, while the Triage Flaws module shows 0 Very Highs, 1 High and 111 Mediums (see other screenshot attached)

​

Can anyone shed a light as to what am I missing here?

 

Best regards,

Georges

​


  • Georges (Community Member)

    Second screenshot mentioned in the original post

    • Screenshot 2022-04-29 at 18.19.34
  • Georges (Community Member)

    P.S.: I'm inclined to believe the numbers from "Triage Flaws" are correct as they match those on the "Results : Latest" page

    • Screenshot 2022-04-29 at 18.31.12
  • KP7766 (Community Member)

    Hi Georges,

     

    It seems to be because the searches in Analytics and Triage Flaws are for different logical buckets of flaws:

     

    1. Your Triage Flaws search is looking at Open flaws only, so it makes sense that it matches what's being returned in your current Latest Results of open flaw counts. Are you trying to find Time to Resolve (TTR) of open flaws? If so, know that the Analytics measure you're using (Findings "Time to Resolve") won't work - that measure only registers and stores a TTR value after a flaw is closed either by approved mitigation or a scanner auto-closure. You can verify the TTR measure behavior by adding a filter to your Analytics search on the "Finding Status = Open" and you'll see that TTR field go null across the board. If instead you're trying to see how long your currently open flaws have been open as a sort of alternative TTR measurement, the Findings dimension "Finding Age" should be able to give you that.
    2. In terms of what your Analytics search is showing, without any filters on "Finding Status = Open", scan/finding open date restrictions, etc., those counts by severity are showing you the full historical flaw counts in each severity bucket for each application since its very first scan, inclusive of the those currently open and those already closed (except some of the closed flaws, per your filter - you've excluded those with approved mitigations, just note that this Mitigation filter isn't applying to any flaws auto-closed by the scanner). The TTR measure, though, is calculating on only a subset of those severity bucket flaw counts: those already closed (recall that it's null for any flaws currently open). For a true TTR historical average, I'd recommend adding a filter of "Finding Status = Closed", which will show all closed flaws in each Severity bucket and the accurate TTR across them.
    3. Final tip is that depending on what you're looking for, you may also want the following filters:
    • "Policy or Sandbox Scan = Policy": Without this filter, your results will include not just your regular Policy scan flaws but also those within your developers' Sandboxes. Unless you're using Sandboxes in a way that you want those results included in this analysis, you should explicitly exclude them with this filter.
      • "Policy Rule Passed = No": If your company uses Policies to disallow only a subset of all found flaws (i.e. those above X severity), it can be helpful to apply this filter so you're only looking at the flaws you care about - those that don't pass policy.

     

    If it's helpful, I'll also share how my team tracks TTR:

    1. Add Findings explore dimension "Resolved Month", sorted with the most recent month at the top
    2. Add Findings explore measure "Time to Resolve"
    3. Add this custom Calculation we named "Rolling 6 Mo. Average", which returns the rolling trailing average of each month and the 5 months preceding it): mean(offset_list(${fact_findings.mean_time_to_resolve}, 0, 6))
    4. Then visualize a line, with these visualization settings applied:
    • Limit to the first 12 rows (to show the last 12 months), but make sure restriction is applied in visualization settings, not in the actual table, or it'll mess with your oldest trailing 6-month averages.
      • Under "Series", apply column to the monthly TTR counts and a line to the rolling 6 month average:image
      • Under "Series Positioning", choose Overlay to put the Rolling Avg line overtop the individual months' bar graph counts.

    From here, you can apply additional filters for only certain applications, only certain severities, etc.

     

    We find displaying the data this way tells a better narrative to our stakeholders than all-time averages - it shows how our teams are doing over time and whether their TTRs for flaws closed that month are getting shorter or longer.

     

    Hope all this helps!

     

    Cheers,

    Kate

    Expand Post
  • Georges (Community Member)

    Hello Kate and thank you so much for this detailed answer! This indeed helps a lot πŸ™‚

    ​

    > Are you trying to find Time to Resolve (TTR) of open flaws?

    ​

    Indeed, I'm looking for the Veracode estimates and use that number to roughly see which teams are drowning in their backlog the most so we can allocate resources appropriately. I was not aware that there was a boolean field tracking the open-ness of a finding, so I added that now to replace the "Mitigation Status" filter. Indeed after applying that, the report shows all "βŒ€" in the TTR column (which I've now removed because it wasn't the richt metric in the first place πŸ™‚ )

    ​

    ​

    > If so, know that the Analytics measure you're using (Findings "Time to Resolve") won't work - that measure only registers and stores a TTR value after a flaw is closed either by approved mitigation or a scanner auto-closure

    ​

    Which metric would help me acheve the above? Have been scrolling through the available fields, but can't quite see which one is pointing to the dimension that puts the orange circles to the "more time needed"side of the "Fix First" graph.

    ​

    ​

     

    Thanks in advance for any feedback.

     

    Best regards,

    Georges

    Expand Post
    • KP7766 (Community Member)

      Hi Georges,

       

      Glad this was helpful. I'm not sure what this "Fix First" graph you refer to is, but from what you described, there are two other ways you could could consider which teams need more help or extra resources:

       

      1) If you do have SLAs set in your policies, take a look at the Findings explore dimension "Grace Period Expiration Date." It offers the date that the flaw's SLA expires, after which point the flaw is overdue and the application will move from Conditional Pass policy state to a Did Not Pass failing policy state. You could analyze, by team, how many open flaws a team has that are very close to their due dates (however you want to define "close" - 1 week, 1 month, 1 quarter?) to get an idea of which teams are keeping up with their flaw loads and remediating comfortably before SLA and who which still have many open flaws approaching their due dates -- or even past them.

      • One thing to note - the Grace Period Expiration Date dimension only works for open flaws. It clears out to null once a flaw has been mitigated or closed, which means you can't use this dimension for analyzing already-closed flaws (I've been requesting for years that Veracode make this a permanent record to support historical analyses, either snapshotting whatever the Grace Period Expiration Date was at the time of close and keeping that dimension populated or with a separate Boolean dimension. Analytics Product Owners, if you're reading this, please refer to this idea; it's been sitting unreviewed in "New" status for 2 years. https://community.veracode.com/s/idea/0873n000000XbKlAAK/detail).

       

      2) If you don't have SLAs in your policies, you could instead use the Findings explore dimension of "Finding Age" - it's a count of the days since the finding was first opened. You could measure the average age of open flaws for each team and see whose flaws have been open longest, with breakdowns by severity if you want to get more granular from a risk prioritization perspective.

       

      Cheers,

      Kate

      Expand Post
      • Georges (Community Member)

        Hey Kate, thank you again for the swift and detailed response! Very much appreciated.

        ​

        We have SLA's set in the policies, but IMHO this will highlight the wrong metric as some teams are fixing the Veracode findings and others are not doing so at all. Measuring the overshoot on expiration date will put the teams on top that have made a conscious decision not to look at the Veracode findings at all anymore until more resources are provided, so that's a catch-22. I need to figure out who needs it the most for efficient resource allocation purposes.

        ​

        That being said: the fix-first-graph I was talking about:

        When going into an application and selecting "Triage Flaws" from the left menu, the top half of the screen defaults to "Source Code Viewer" mode, but you can also switch it to "Fix First Analyzer". This produces the graph I'm speaking of. I'd like a measure of summed circle area per finding criticality and prioritize budget weighted by the application's criticalities.

        ​

        ​

        After some digging, I've figured out that this metric is present in the filters of the "triage flaws" module and is called "Effort" there, however this dimension's source is ambiguous.

        ​

        In Veracode Documentation (https://docs.veracode.com/r/review_severity_exploitability) the "Effort to fix" metric is described as a property of a finding:

        ​

        > Each finding instance receives an effort-to-fix rating based on the classification of the finding. The effort-to-fix rating is a scale from 1 to 5, as explained in this table.

        ​

        It is the numbers in the leftmost column of that table (or at least 1-4 as they actually have an indication of effort) that I'd like to map to the max effort described in the rightmost column, which I'll have to come up with a table calculation for (if(effort==1){score += 2}; if(effort==2){score+=8}; etc.). I just found out that these calculations even exist in the analytics dashboards, so I haven't looked into how to do that yet and am also unsure if this is even possible. Worst case, I'll have to query the Veracode API and do the dashboarding in PowerBI.

        ​

        For a score 5, I'm fine with a configurable number or leaving it out entirely; I'm comparing teams, so omitting this for all and creating a separate dashboard on the total number of 5's will give me this global idea.

        ​

        ​However in the analytics dashboard, effort is is a property of the underlying CWE (i.e., ${dim_cwe.remediation_effort}) and I was under the impression that this fix effort was weighted towards the complexity of the implementation. I.e., if two findings are similar but the number of Paths and number of steps is bigger, I'd assume that the latter one's effort metric would be bumped up. Maybe someone from VC can validate this assumption?

        ​

        ​

        I'm aware this is a long shot, but any guidance on measuring this would be highly appregiated.

        ​

        ​

        Warm regards, Georges

        ​

        Expand Post

Topics (2)

No articles found
Loading

Ask the Community

Get answers, share a use case, discuss your favorite features, or get input from the community.