By Jim Reavis
January 17, 2000 - It is an article of faith among Open Source software advocates that the freely available source code of Linux makes it easier to identify and patch bugs than Closed Source software and hence provide greater overall systems security. But is there factual evidence behind this, or is it just a theory? After all, according to theory, a bumblebee shouldn't be able to fly, but I have been stung several times! We decided to go look for empirical evidence of the impact of Open Source software upon the speed at which vulnerabilities can be patched.
Despite the loftiness of our goal, there are far too many difficult-to-quantify factors and we cannot claim this to be a scientific pursuit. Any veteran of this industry will tell you that you really can't prove security. However, by narrowing the scope of our research to common data elements in the bug fix process, it is possible to find some meaningful answers to the question of bug fix speed.
What we decided to do was to look at the security advisories issued by Microsoft and Red Hat in 1999 and gauge the time lag between the point of a "general community awareness" of a security problem and the point at which a patch was released. We also threw Sun Microsystems into the mix for comparison's sake. Purists of the Open Source philosophy might cringe at our decision to select Red Hat to represent all of Linux, when they are merely one of many distributors. Indeed, there is often a significant time lag between the time a developer patches a Linux bug and the time Red Hat issues an advisory. We wanted this exercise to be of assistance to IT professionals who are evaluating Linux and Microsoft products for use within corporate environments. For the most part, these people are going to rely upon vendor support channels for patching their operating systems. Except in extreme cases, systems administrators will wait for a package from their chosen distributor to implement on their own Linux servers. This is not only a reflection of prudent conservatism, but sysadmins really want signed packages that they can verify the authenticity of before implementing. Red Hat being the largest distributor, we felt they were a logical choice. Here is how we did it:
We wanted to set some arbitrary criteria for defining when a security issue reached a point of critical mass of publicity. We were not interested in tracing every security vulnerability back down to ground zero. Instead, we wanted to define a starting point when the vendor absolutely should have been aware of the problem. We used the following events as evidence of critical mass:
Advisory issued by well-known security consultants and hacker groups (e.g. ISS, L0pht, eEye, etc.)
Postings to relevant mailing lists (e.g. BugTraq, Win32 dev lists, etc.)
CERT Advisory or similar
Sometimes, the vendor's own admission of a problem without having a patch
We consider the time between the critical mass point and the vendor producing a patch to be "recess" for the malicious hacker and "script kiddie". This is the scariest time for a systems administrator - they have a vulnerability that is well known, but have no technical solution. We totaled the days between these points to calculate the total amount of recess granted by each vendor, and the average amount of recess per advisory. While in the future measuring this time in hours and will be warranted, this is sufficient for now.
Although some advisories are obviously more critical than others, we didn't use any sort of weighting factor that would be open to subjective interpretation. A bug is a bug is a bug in this survey. We made some minor date adjustments when a vendor's first attempt at a patch was broken, but it was a statistical nit as far as the overall picture.
An assessment of how well you are administering your systems. As many of the CERT activity reports show, most of the problems associated with unwanted hacking involve known vulnerabilities that have already been solved by your vendor.
Information on how bug-free your systems are. This is not intended to be an analysis of software development methodologies themselves or an estimate of defects still within the software that are unacknowledged.
Number of advisories issued by a company equates to relative security or insecurity. We have not studied the reasons behind the wide range in frequency of advisories issued by the companies in question and its impact on overall security. Whether a company issues one or one hundred advisories, their response to fixing vulnerabilities should be uniformly quick.
How did our contestants fair? Red Hat had the best score, with 348 recess days on 31 advisories, for an average of 11.23 days from bug to patch. Microsoft had 982 recess days on 61 advisories, averaging 16.10 days from bug to patch. Sun proved itself to be very slow, although having only 8 advisories it accumulated 716 recess days, a whopping three months to fix each bug on average. We can assume Sun and Microsoft should have and can do better, but we cannot prove that due to the fact that they use a closed system for developing software and fixing bugs. However we can clearly see that Red Hat could have done much better and cut their turn around time in half, had they only been more attentive to their own community. There were instances when software had already been patched by the author, but Red Hat was slow in creating RPM distribution files and issuing advisories.
|
Obviously, all vendors benefited on our scorecard from security vulnerabilities being disclosed only to them, accounting for many of the patches being available the same day as the advisory. We call these "friendly" advisories as opposed to "full disclosure" advisories. This will certainly stoke the argument between the full disclosure crowd and those arguing for releasing bug details only when the vendor has patched them. No matter which side of the argument you are on philosophically, we believe that full disclosure is the dominant trend, and vendors will need to better cope with being blindsided. Red Hat did a better job of handling the "full disclosure" bug releases, usually solving these problems in under 2 weeks, with 67 days being the extreme case. Microsoft usually took over 3 weeks to patch "full disclosure" bug releases, with a worst case of 146 days. Sun took in excess of 100 days 3 times, including a mind-numbing 385 days for a CDE problem related to a CERT advisory.
A frustrating part of "friendly" advisories is the fact that they are not accompanied by any date history. If Bindview, for example, warns Microsoft of a problem with NT, they do not provide the public any information about how long the bug fix has been a "work in progress", and when they first contacted Microsoft. This prevents us from gauging how efficiently the vendor is solving problems stemming from "friendly" advisories.
If you assumed that most of the initial advisories came from industry leading security firms with a staff of programmers and a fancy lab, you would be wrong. In fact, a wide majority of the bugs are discovered by individuals working independently. We think that this fact in itself is an indirect nod to the power of Open Source - you never know where a bug or a fix is going to come from and Open Source lets everyone participate in the process.
What have we proven in compiling these numbers? We think an entire year of data, while not conclusive, provides a fairly good indication that Open Source software can have its security vulnerabilities identified and repaired in a more timely manner than traditional closed source software. A bug fix from Microsoft takes almost 50 percent longer to reach the market as a fix from Red Hat - this despite the fact that Microsoft has huge advantages in funding and employees. An attentive Linux administrator who did not want to wait for Red Hat RPM files could get their patches even more quickly. The resources provided via Open Source seem to be providing Linux with the advantage needed to turn around bug fixes so rapidly. Is it possible that the slowness on the part of closed source vendors like Microsoft is due in part to a different and more rigorous quality assurance testing process? The process certainly is different, however Microsoft had to re-release five of its security patches in 1999 due to regression errors in the code, so it cannot likely be argued that the more lengthy release cycles by closed source vendors is translating into higher quality code.
Information about security vulnerabilities is moving across the Internet with greater and greater speed and is disseminated to a wider range of people. In the future, the gap between general awareness of a security vulnerability and its repair will become a more critical issue, leading to increased downtime, financial setbacks and loss of privacy. Open Source software seems to be at least part of the solution for closing that gap.