Michael Edenzon
February 27, 2024
•
15
min read
Proof and affirmative evidence is the foundation of Automated Governance. In part one of this series, we establish the need for two types of proof, proof of measurement and proof of adherence, and the need for affirmative evidence to support such claims. The inconvenient truth is that not all proof is created equal; there is good and bad proof. In part two of our series on Automated Governance we will explore the information that makes for good proof and how it can be achieved.
To do this, we must first understand the definition of evidence and proof and how it varies in practice across the domains of law and science. In his oft-cited 1992 paper Standards of Proof in Science and Law, Lee Loevinger, jurist and former Commissioner of the FCC, states that evidence is the intellectual support for some conclusion or proposition, and proof is the sufficiency of evidence to support a conclusion or proposition. Loevinger writes that the difference between scientific and legal proof is the inherent reproducibility of the claim; if I can recreate the same conditions, will I get the same result? Of course, in science, the answer is yes. Scientific proof is predictive, which means that the validity of a scientific claim is determined by the ability of others to replicate the experiment. Whereas legal proof is postdictive, meaning that in litigation it may be impossible to recreate the conditions of a past event, therefore the validity of a claim must be proven on the evidence itself. It is for this reason that certain evidence may be deemed “inadmissible” by the rules of evidence if it does not make a claim more or less probable without it.
Automated Governance is a methodology of evidence collection and policy enforcement. It claims that a software asset is compliant or non-compliant based on proof that all required controls have been measured and all applicable policies have been enforced. The domain of compliance is inherently legal, therefore proof is postdictive. Therefore, the validity of a claim’s proof can be determined only by the sufficiency of the evidence that supports it.
As a DevOps platform engineer, there are enough challenges to capturing and evaluating evidence from the pipeline. Controls, policies, and procedures are added in piecemeal and are rarely devised with automation in mind, and there is ongoing debate about the extent to which DevOps (or DevSecOps) should own software compliance. The CI/CD pipeline, in theory, contains all of the necessary information to support this responsibility and and its place in the developer workflow make it the optimal solution for reducing toil. As a result, many organizations attempt to “bake in compliance” to their build and release pipelines without first understanding what it means to prove compliance.
When automated, the process of capturing evidence is often limited to only what’s known at the point of capture, and depending on the control, first hand information may suffice. However, evidence requirements vary with each control or system of record and may demand cross-referencing with external sources. In some cases, the control’s required evidence format necessitates manual interpretation to properly measure and evaluate its contents.
Efforts to harmonize across such a wide gap in requirements inevitably produces insufficient evidence that runs afoul of legal principles that, implicitly or explicitly, govern the vast majority of compliance frameworks. This is an insidious truth of integrated software compliance automation. Most Automated Governance implementations fail because they fail to prove compliance based on the evidence alone.
This raises the obvious question; If the same evidence deemed sufficient for manual governance is deemed insufficient for automated governance, does this imply inherent flaws in manual processes? Not exactly.
The transition away from manual governance to partial or complete automation entails obstacles unrelated to technical implementation. Humans are considerably less trusting of automation, especially in its infancy. This phenomenon is amplified when the automation in question serves a business process of high criticality such as software governance. The reasons for this are twofold; first, trust is an inherently human characteristic defined by one’s belief in the reliability of another’s word as well as an expectation of technically competent role performance.
By virtue of this fact, automation must supply evidence that supports a higher burden of proof. While this may sound abstract and somewhat academic, we must remember that compliance is viewed as a legal process, wherein the standards of evidence are structured to demonstrate proof beyond a reasonable doubt. In simpler terms, the proof must be so convincing that one has no choice but to trust it. Since humans are less trusting of automation than they are of other humans, automation must significantly outperform when it comes to standards of evidence.
While this may sound abstract and somewhat academic, the following analysis of federal evidence rules combined with tangible comparisons to everyday examples in your DevOps environment will help illustrate these concepts.
The following are a handful of rules from the Federal Rules of Evidence, a set of rules that governs the introduction of evidence in United States federal trial courts, and a corresponding example of how misalignment with evidence rules within your DevOps environment could cause insufficient evidence to prove the compliance of software assets.
FRE Rule 1002. Requirement of the Original: Reproduction of the original of a document is required to prove its contents when its contents are sought to be proved.
When the contents of a document must be proven, the original document is the only sufficient evidence.
Your delivery pipeline executes a suite of functional and regression tests against a group of software assets. As a result, a report is generated detailing the entire suite of test cases executed against all of the assets in question. The pipeline outputs the following log statements containing test report file path and the count of the passed and failed tests.
A common transgression of the rule of best evidence is to implement automation that evaluates the log output statement as evidence of the number of tests passed. It is well understood that log statements can be manipulated. Moreover, the log statement is not the original document so it is not considered best evidence.
Observation of this rule would be the automated capture and evaluation of the test report that was created as a result of the test execution.
FRE Rule 901. Authenticating or Identifying Evidence: The proponent must produce evidence that proves that the evidence in question is what the proponent claims it is.
All forms of evidence require evidence to prove its authenticity with the exclusion of certain sealed or publicly circulated print documents.
On each execution of your continuous integration pipeline a software artifact is built and published. Embedded in the pipeline build script is a line of code that generates an attestation detailing the provenance of the artifact built. It contains a litany of information, including commit information, artifact hash, and a snapshot of environment variables at the time of build. The attestation produced by the build script is published to a database so it can be pulled by the delivery pipeline prior to deployment with the objective of preventing artifacts of unknown origins from reaching a production environment.
On the surface, this seems like a worthwhile attempt at authenticating an artifact. After all, it does supply affirmative evidence in support of the authenticity of the software artifact. However, this process is actually a transgression of the rule of authentication and identification because there is no evidence to support the claim that the provenance attestation was not falsified retroactively after the conclusion of the build pipeline. In other words, the attestation alone does not meet the standards for authenticity because any individual or system with access to the attestation database could upload a fabricated attestation attributed to the software artifact in question. Yes, controls placed around database access are important, but treating the existence of access controls as evidence is a form of argument from ignorance and offers very little by way of proof.
A common observance of this rule is to employ digital signing for attestations generated in a build pipeline. Tools like Sigstore provide a simple way to do this and employ powerful features like ephemeral keys and transparency logs that ensure only authorized build servers can produce valid attestations.
FRE Rule 602. Need for Personal Knowledge: A witness may testify to a matter only if evidence is introduced sufficient to support a finding that the witness has personal knowledge of the matter.
To testify about an event a witness must prove that they witnessed the event firsthand. Second hand knowledge of an event does not count.
A few years before your company completed their DevOps transformation, the AppSec team hired a team of consultants to build automation around their Static Application Security Testing (SAST) program with the aim of better informing developers about vulnerabilities detected in their codebase. The automation script captures vulnerability reports at the end of every SAST execution and populates critical and high CVEs in the company’s central GRC database whereby SLAs for remediation are tracked. Before each release, the manual governance process treats the GRC database as the source of truth for known vulnerabilities in a given software asset and enforces policy based on the evidence it contains.
The most common transgression of the rule of personal knowledge is to direct your governance automation to pull the list of critical and high CVEs from the GRC database and measure them against company policy. This runs afoul of three rules: personal knowledge, best evidence, and authentication. The GRC database received the results from an intermediary (the custom script) and stores a summary of evidence without any evidence to support its authenticity. This methodology incurs the risk of false, tampered, or wrongly identified evidence without any mode of reconciliation.
Observance rule of personal knowledge would require your governance automation to be first hand witness of the test execution, capture and digitally sign the test execution report, attribute it to the software asset and version in question, and store as evidence before evaluating the contents to determine the asset’s compliance status.
Evidence rules for trial courts translate, in principle, to the principles of Automated Governance. However, you should not need a law degree to employ them. To simplify and make these principles accessible in your DevOps Automated Governance implementation, we propose seven rules to observe across all forms evidence collection.
We know, it’s frustrating. But automation is subject to a higher burden of proof and its validity can only be proven by the evidence itself. In the next part of our multi-part series on Automated Governance we will discuss our methodology for evidence collection in accordance with the rules and frameworks listed above.
Schedule a demo today!
