RHEL 6 vs. Clones: Security advisories in the 1st half of 2012



Introduction

There are three popular Red Hat Enterprise Linux clones: CentOS, Scientific Linux and Oracle Linux. All of these clone projects download source RPM packages from Red Hat and re-compile them to produce their own distributions.

These Linux distributions are often installed on servers which are connected to the internet. In that task it is essential to take care of security bugs quickly to avoid a system compromise. All of these three clones issue their own security advisories and updates.

I decided to compare the contents of these advisories as well as the delay publishing the advisories. This comparison is for version 6 of these distributions.

I was partially inspired to do this comparison after I saw a pathetic graph on Oracle’s web site which tries to get people to convert their CentOS installations to Oracle Linux. The graph that Oracle uses there includes only kernel security bugs even though it can be equally important or more important to quickly fix security bugs in other system software, such as internet facing service daemons. (The page has been archived by WebCite at http://www.webcitation.org/6A44JQ7rI and the graph at http://www.webcitation.org/6A705qdDS.)

Advisory contents: Red Hat Enterprise Linux

This is “the upstream” that everyone else is copying from. RHEL advisories are very well written and contain all the essential information about the fixed security problems. They are issued through some automated system which makes sure that all information is where it should be. The advisory contents are very consistent: they can be easily parsed with a program to extract the essential contents for further processing.

My observation here is that you get what you pay for (even though the advisories themselves are freely available): Red Hat’s security advisories are of very high quality.

Advisory contents: CentOS

CentOS advisories are very terse. They only contain the advisory identifier, the severity and the name of the updated software on the subject line. The contents of the advisories consist of a link to the upstream advisory and a list of the updated RPM packages with SHA256 sums. In the past there has been inconsistencies and it is difficult to machine parse old advisories, but this has improved a lot over the years.

Last year (2011) there was a period of several months when the CentOS project did not issue any security advisories or updates for CentOS 6. Many CentOS users got frustrated and worried about their system security for obvious reasons, as evidenced by looking at various CentOS related mailing lists, forums and blogs at that time. Hopefully the project has solved their problems and this will not happen again.

Advisory contents: Oracle Linux

  • Archives at: https://oss.oracle.com/pipermail/el-errata/
  • CVE identifiers: yes (but not in the header)
  • Unique advisory identifier: yes
  • Link to the upstream advisory: yes
  • Description of the issues: very brief
  • PGP signature: no

Oracle Linux security advisories are very similar to CentOS advisories, except there is RPM changelog excerpt at the end. In the past there has been several inconsistencies which make machine parsing old advisories difficult. The CVE identifiers are in the changelog excerpt at the end of the advisories and therefore difficult to find.

Oracle Linux differentiates itself from the other two clones by providing an alternative kernel called “Oracle Unbreakable Enterprise Kernel”. Updates to this alternative kernel are not included in this comparison because it does not make sense to compare apples to oranges and the author fails to see any added value from using this special kernel.

Advisory contents: Scientific Linux

Scientific Linux security advisories are more verbose than advisories of the other clones. They contain a good description of the fixed security issues. The advisories do not have any unique identifiers which makes it difficult to refer to any specific advisory. The advisories are mostly consistent (this has improved over the years), but it is difficult to match them to the corresponding upstream advisories.

It appears that Scientific Linux missed two RHEL advisories during the first half of 2012. At least they are not in the mailing list archives. The corresponding updated RPM packages did appear in the repositories though.

Update: One of these two missing advisories (RHSA-2012:1009) was about new software (java-1.7.0-openjdk) which was added in 6.3 release. Scientific Linux had not yet released version 6.3 at that time, thus no advisory was needed.

Security advisory publishing delays

The following graph displays the time (in days) it took for the different clones to publish a security advisory after the upstream issued an advisory (this includes all RHEL 6 advisories issued in the first half of 2012):

RHEL 6 vs. clones: Security advisory publishing delays in 1st half of 2012 (SVG graph)

There are a couple of gaps for Scientific Linux. These security updates were made available in the repositories, but there were no advisories published about these updates. The reason for this is unknown to the author.

Update: One of these two missing advisories (RHSA-2012:1009) was about new software (java-1.7.0-openjdk) which was added in 6.3 release. Scientific Linux had not yet released version 6.3 at that time, thus no advisory was needed.

The following graph shows the average publishing delays:

RHEL 6 vs. clones: All security advisories, average publishing delays in 1st half of 2012 (SVG graph)

The following graph shows how frequently each of the distributions was the fastest one to publish an advisory:

RHEL 6 vs. clones: All security advisories, first distribution to publish in 1st half of 2012 (SVG graph)

The following graph shows how frequently each of the distributions was the slowest one to publish an advisory:

RHEL 6 vs. clones: All security advisories, last distribution to publish in 1st half of 2012 (SVG graph)

Important and Critical security advisory publishing delays

All security bugs are not created equal. Some are more important than the others. Updates with “low” or “moderate” security impact can often wait until a later time (depending on the particular environment). Therefore it makes sense to have a second look and focus only on security issues that are marked “important” or “critical” by the upstream vendor.

The following graph displays the time (in days) it took for the different clones to issue a security advisory after the upstream issued an advisory with important or critical severity level:

RHEL 6 vs. clones: Important and critical security advisory publishing delays in 1st half of 2012 (SVG graph)

The following graph shows the average publishing delays for important and critical advisories:

RHEL 6 vs. clones: Important and critical security advisories, average publishing delays in 1st half of 2012 (SVG graph)

The following graph shows how frequently each of the distributions was the fastest one to publish an important or critical advisory:

RHEL 6 vs. clones: Important and critical security advisories, first distribution to publish in 1st half of 2012 (SVG graph)

The following graph shows how frequently each of the distributions was the slowest one to publish an important or critical advisory:

RHEL 6 vs. clones: Important and critical security advisories, last distribution to publish in 1st half of 2012 (SVG graph)

Critical security advisory publishing delays

One more look, this time focusing only on critical advisories.

The following graph displays the time (in days) it took for the different clones to issue a security advisory after the upstream issued an advisory with critical severity level:

RHEL 6 vs. clones: Critical security advisory publishing delays in 1st half of 2012 (SVG graph)

The following graph shows the average publishing delays for critical advisories:

RHEL 6 vs. clones: Critical security advisories, average publishing delays in 1st half of 2012 (SVG graph)

The following graph shows how frequently each of the distributions was the fastest one to publish a critical advisory:

RHEL 6 vs. clones: Critical security advisories, first distribution to publish in 1st half of 2012 (SVG graph)

The following graph shows how frequently each of the distributions was the slowest one to publish a critical advisory:

RHEL 6 vs. clones: Critical security advisories, last distribution to publish in 1st half of 2012 (SVG graph)

Conclusions

All three clones have some delays in publishing security updates and advisories because it takes time to re-package the source RPMs published by Red Hat.

The CentOS project managed to publish one security advisory before the upstream advisory was sent on the Red Hat mailing list (RHSA-2012:0451 vs. CESA-2012:0451). This is quite an accomplishment.

If fast security fixes are important to you, you should purchase a subscription for the original upstream product (Red Hat Enterprise Linux).

There are no huge differences in the delays between the different clones. Currently all of these distributions produce updates for important and critical issues quickly enough for most purposes.

Scientific Linux was generally slightly slower than CentOS to publish advisories, but this could be simply a result of the fact that their advisories actually have a lot of content which needs to be edited. CentOS advisories are quick to produce as they do not contain any problem descriptions and only have a link to the corresponding Red Hat advisory.

Update: One reader pointed out: “Delays in issuing bug fixes are good for you – no delay means they did not test the stuff before pushing it out.” I completely agree with this. Publishing security updates is a balancing act between the need to get things out quickly and on the other hand making sure that the fixed version actually works. However the real testing of the backported patches has been already done by Red Hat at the time when they publish their advisory and source RPM. The clones only need to perform a limited amount of testing when packaging the patched version to ensure that there is no compatibility issues with their releases.

Oracle Linux: truth in advertising?

Oracle Linux gets a special mention here. It does not have any special advantage over CentOS or Scientific Linux even though their marketing department wants to make you believe otherwise, quoting from their web site (archived by WebCite at http://www.webcitation.org/6A44EPTRC):

Oracle invests significantly in testing Oracle Linux and releasing critical bug fixes faster, enabling enterprises to deploy with confidence.

As can be seen from the above graphs Oracle was never the fastest clone to publish a critical security advisory within the evaluated time period. In fact it was the slowest distribution in 82% of critical cases. Truth in advertising? Does not seem that way. However the marketing materials do not have an answer to the question “faster than who?”…

Comments welcome

Feedback and other comments are welcome. You can add your comments below. Please stay on topic and do not start a flame war as everyone seems to have their favorite RHEL clone and some people seem to be very emotional about it. Feel free to contact the author privately if your comment is not suitable for public display.

  • zkaspar82


    It would be interresting to use another (and IMO much better, but more time consuming..) metric to graph BUILDTIME from specified packages..

    • Janne Snabb


      I was actually considering this as an option while making this comparison. At this time I decided to go with the advisory publishing times (this data comes from the “Date” headers of the advisory e-mail messages in case it is not obvious — and yes, I have taken the different time zones in account).

      From my point of view as a sysadmin the advisory publish times are more interesting than the RPM build times. I can not know that I need to quickly “yum update” my servers unless I see a relevant advisory first.

      There is also one problem if comparing RPM package times: a single advisory often updates several RPMs and they may be built at different times. Which RPM packages to compare against each other and how to present that data in a meaningful way?

      Also, there is a delay after the package is built before it is actually available for download on the distribution servers. A built package does not help if I can not get my hands on it.

      Thanks for your feedback!

  • Mark Cox


    Hi; http://www.redhat.com/security/data/metrics/ contains release_dates.txt which gives the date that an advisory was available that customers could consume on Red Hat Network (and therefore the date the distro will notify them if they don’t read the mailing list). This could be several minutes to an hour earlier than the mailing list post as we tend to hold the mailing list posts until we complete a batch of pushes. You can also get XML representations of our advisories at http://www.redhat.com/security/data/cvrf/

    • Janne Snabb


      Thank You for these helpful pointers! Looks very interesting. I would have used the data if I was aware of it. Luckily Red Hat advisories are easy to parse even without well formed XML data, so I did not do too much extra work. 🙂

      I was thinking that the real Red Hat release time must be a bit earlier than the corresponding mailing list post. Otherwise it would have not been possible to see CentOS mailing out a security advisory before Red Hat does (as was the case with RHSA-2012:0451).

      This comparison was mainly about comparing the different clones against each other and for this purpose it does not really matter if the Red Hat baseline time is from the mailing list post or from the RHN publication time.

      Obviously Red Hat will always be faster than any of the clones. Red Hat is doing all the major work here and everyone else is just copying the results of their work. It would not be possible for the clones to publish patches earlier unless they fork the code base and start maintaining their own set of patches (unlikely to happen: it would be a major undertaking).

      Thanks again for your comments!

    • Janne Snabb


      Some additional observations:

      To investigate Red Hat Network services further I signed up for RHEL 30 day evaluation subscription to see how the Red Hat Network notifies subscribers about relevant advisories.

      Since starting the evaluation there has been two security advisories which have been relevant to my evaluation system.

      Both of the RHN Errata Alerts were sent and received at a later time than the corresponding advisories on the public “rhsa-announce” mailing list. Thus the public mailing list seems to be the fastest notification method. Even though the advisories are available earlier within the RHN service, it does not send out notifications immediately.

      I have not yet tried polling the various XML feeds every minute. 🙂

  • Dan


    Is there a way I can get JPEGS or GIFs of your graphs?

  • Dan


    Can you share your graphs in a JPEG format?

    • Janne Snabb


      Eh? I already shared them in PNG format with a permissive license (see my reply to your other comment). Feel free to convert them to any format you wish. PNG is supported by every possible program made or updated within last 10 years.

      JPEG format is not very suitable for computer generated graphs (it is more appropriate for photos and such). GIF is obsolete these days. Therefore I assumed PNG would be appropriate raster format (although very sub-optimal compared to SVG which can be zoomed in or printed/displayed on a large medium without losing precision as it is vector based).

      Have a look at the Wikipedia article for more detailed information: https://en.wikipedia.org/wiki/Portable_Network_Graphics

      Please do not repeat your request further in these comments because it has nothing to do with the topic of the article. I am sure your employer has some skillful staff who can do any conversion for you to use in your marketing activities. Not my job, sorry.


  • I would also like to point out that CentOS also provides yum-presto and delta RPMs for updates as explained here:

    http://docs.fedoraproject.org/en-US/Fedora/14/html/Software_Management_Guide/ch06s26.html

    No one else, including Red Hat, is providing this feature for Enterprise Linux.

  • arl


    Nice article.

  • Suresh


    Well written article. I mostly used unpaid distros (starting from Redhat 5.2), but this is a very interesting perspective of RedHat’s position and commitment.
    I recently started using CentOS (in addition to regular fedora/ubuntu) and get a clear picture. CentOS during development and RHEL for deployment.
    Thanks and keep writing 🙂