Iron Bank's Weak Link: The Myth of Hardened Container Security

Container security has become a critical issue in the software industry, especially with the widespread adoption of containerised applications. In the recently published "State of Hardened Container Images Report" by John Speed Meyers and Paul Gilbert from Chainguard Labs, the report identifies the challenges and solutions in the pursuit of secure container images. In this article I delve into the report's key findings, underscoring the importance of hardened container images and the current landscape of container security.

The State of Hardened Container Images Report paints a concerning picture of the current state of container security. Most popular container images harbour hundreds of known vulnerabilities CVEs, (Common Vulnerabilities and Exposures), resulting from software bloat and slow update cycles. Efforts to mitigate these issues have been incremental at best, often leaving organisations struggling to keep up with vulnerability management. 

Most concerning is Iron Banks container vulnerabilities. Iron Bank, officially known as the "Iron Bank Container Repository," is a curated repository of container images maintained by the U.S. Air Force. It is part of the Platform One initiative, which aims to provide DevSecOps capabilities across the Department of Defense.



The repository is designed to support military-grade security requirements and is utilised by various branches of the U.S. military and related defence entities. Iron Bank emphasises secure software development practices and aims to deliver container images that are thoroughly vetted and maintained to reduce the presence of known vulnerabilities. However, the report by Chainguard Labs reveals that even these supposedly hardened images still harbour a significant number of vulnerabilities.

Key Findings:

• The 50 most downloaded images from Iron Bank average 110 CVEs.
• There are over 1000 CVEs in Iron Banks ‘Big Bang’ container images, 31 of which are critical CVEs.
• Popular Debian-based images average nearly 300 CVEs, largely due to the inclusion of numerous components.
• Updating packages in Docker official images only yields a modest 6% reduction in CVEs.
• Debloating technology reduces CVEs by approximately 65%.
• Red Hat-provided container images average nearly 200 CVEs, excluding those marked as "will not fix."
• Canonical's Chiselled images have few or no CVEs but are limited in availability and require advanced user expertise.

The report emphasises that container security is intrinsically linked to open-source software security. Containers bundle software applications and their dependencies, often containing numerous open-source components. Consequently, the vulnerabilities within these components become the responsibility of the organisations using them. This reality underscores the critical need for secure-by-default container images. The report demonstrates that simply updating the packages only results in a 6% reduction in CVEs.

Analysis of Debian-based official images from Docker Hub, revealed an average of 280 CVEs per image. Even when focusing only on high and critical severity CVEs, the average remains over 30. The substantial number of components (averaging 273) in these images contributes significantly to the high CVE count.

Efforts to reduce CVEs through frequent package updates prove insufficient. Analysis by Chain Guard found updating Debian packages in a subset of popular images reduced CVEs by less than 6%, as even the latest packages still contain vulnerabilities. Debloating, or removing unnecessary components, offered better results, reducing CVEs by 64%. 

Red Hat's Universal Base Images (UBI) offer another approach to secure container images. Analysis from Chain Guard compared various UBI variants to their Chainguard equivalent containers, finding that UBI images contain more CVEs regardless of whether "will not fix" CVEs are included.

Iron Bank, officially known as the Department of Defense Centralised Artefacts Repository, was established by the U.S. Air Force to provide a secure, vetted source of container images for use in critical systems. The initiative is part of the Department of Defense’s Platform One project, which aims to streamline and secure software development across military and defence environments. By offering hardened images, Iron Bank seeks to minimise vulnerabilities and ensure that software deployed within defence infrastructures meets stringent security standards.

However, despite the rigorous vetting process, an analysis of the 100 most downloaded images from Iron Bank revealed an average of 110 CVEs per image. This finding highlights the inherent complexity and potential risks associated with maintaining a repository of container images, even within a highly controlled and secure environment. CVEs, or Common Vulnerabilities and Exposures, are publicly disclosed security flaws that can be exploited by attackers to compromise systems.

The "Big Bang" images, a subset of Iron Bank, represent a collection of high-security containers specifically developed and vetted by military teams. These images are designed to offer enhanced security features and are intended for use in the most sensitive and critical applications. Despite their intended high security, a deeper dive into the "Big Bang" images revealed over 1,000 CVEs. This significant number of vulnerabilities underscores the challenges in achieving absolute security, even with the most stringent development and vetting processes.

While many of these CVEs have not been exploited in the wild, their mere presence is concerning. As more developers adopt containerized images for their applications, the potential attack surface expands, making it imperative to address these vulnerabilities proactively. The widespread use of containerized images means that any unaddressed vulnerabilities could have far-reaching implications, especially in critical sectors like defence and critical infrastructure. 



The discovery of these vulnerabilities in Iron Bank’s images emphasises the need for continuous monitoring, timely patching, and robust security practices in the development and maintenance of container images. It also highlights the importance of collaboration between security experts, developers, and organisations to mitigate risks and ensure the integrity of containerised applications.

Recent Example of a Critical CVE

A recent discovery by Snyk, identified a critical vulnerability affecting Docker and other container engines. Collectively called "Leaky Vessels", the vulnerability poses a significant threat to the isolation that containers inherently provide from their host operating systems. 

• CVE-2024-21626 involves a file descriptor leak in runc, potentially enabling attackers to access the host system and escalate privileges. 
• CVE-2024-23651 involves a race condition in Docker and Buildkit that could lead to container breakouts and host access.
• CVE-2024-23652 affects Buildkit and allows attackers to potentially delete arbitrary files on the host during image building.
• CVE-2024-23653 exists in Buildkit and could enable attackers to break out of containers during the image-building process.

Deep diving into CVE-2024-21626, this vulnerability resides in runc, a critical tool responsible for spawning containers. Due to an internal file descriptor leak in versions up to and including 1.1.11, attackers can manipulate the working directory (process.cwd) of a newly spawned container process. Like an unlocked door, the leak leaves the file descriptor open, providing access to unauthorised users with potential container breakout. 

Common Vulnerabilities and Exposures (CVE). (2024). Retrieved from CVE.

Chiselled images are a new concept introduced by Canonical, the company behind Ubuntu, to address security concerns in containers. Traditional container images often come with a variety of software packages and libraries, many of which may have vulnerabilities in CVEs. Chiselled images aim to mitigate this risk by including only the essential components needed to run a specific application, thereby reducing the attack surface.

This approach aligns with the principle of least privilege, a fundamental concept in cybersecurity that advocates for limiting access rights for users to the bare minimum necessary to perform their jobs and in the case of containers, the concept of the least possible number of packages and libraries necessary for the container to work.

However, this minimalism comes with trade-offs. Chiselled images are not as versatile as traditional container images, which include a wide range of tools and libraries to support various use cases. As a result, Chiselled images may not be suitable for all applications, particularly those requiring a broader set of functionalities. This limitation necessitates a higher level of expertise from developers and users, as they need to understand precisely which components are required for their applications and how to incorporate them securely.

Moreover, the current availability and usability of Chiselled images pose challenges for widespread adoption. Being a relatively new offering (as at the publication of this report), these images are still in the early stages of development and may not be as polished or user-friendly as more established container images.

The "State of Hardened Container Images Report" underscores the urgent need for secure-by-default container images. The high number of CVEs in popular container images presents a significant risk, necessitating the need for organisations and developers to regularly security test their containers. As container adoption continues to grow, prioritising security through hardened images and regular security scanning will be essential for safeguarding software deployments.