mp3/vorbis bitrate comparison

What Ogg Vorbis quality setting produces the same file sizes as LAME --alt-preset standard?

introduction

While there are many factors to consider when comparing one lossy audio codec to another, it helps to remember that the main reason we compress music is to reduce file sizes. Therefore, I believe, a comparison in the quality of two codecs should be done with files of the same size. But what settings should we use?

Before we do any comparison, we need to establish a standard. With compressed music the "standard" is mp3, and extensive testing suggests transparency (a term meaning the quality of the compressed audio is not distinguishable from the original) is most efficiently reached with the LAME codec, version 3.90.3, using the setting "--alt-preset standard".

Using audio files encoded with "lame --alt-preset standard" as a reference, we can encode those same audio files using Vorbis (version 1.0.1, the latest official release) using various quality settings until an Ogg Vorbis file is produced that most closely matches the size of the mp3 produced with lame. Because LAME and Vorbis approach encoding audio in different ways, this file size comparison will need to be repeated with each audio file, and a large and various set of audio files need to be encoded to produce a meaningful result.

procedure

These are the steps I followed in this test:

1. Choose a diverse selection of 60 audio tracks to use in the test. The music should cover many generes and eras without being too heavily weighted towards one type of music. This selection process is limited by my access to original sources and own good judgement. (I apologize for my noticable lack of classical music.)
2. Rip each track with Exact Audio Copy (EAC), using its most secure settings, and losslessly encode to FLAC, including tags. These files represent the original sources as closely as possible.
3. Transcode each FLAC file to mp3 using LAME version 3.90.3 with the "--alt-preset standard" setting, including tags in id3v1 format.
4. Measure the size of the resultant mp3.
5. Transcode the same FLAC file to Ogg Vorbis using oggenc version 1.0.1 with the "-q 6" setting, including tags in the regular Vorbis Comment format.
6. Measure the size of the resultant Ogg Vorbis file.
7. If the Ogg Vorbis file is smaller than the mp3, re-transcode the Ogg Vorbis file using increasingly higher quality settings until it is larger than the mp3.
8. If/when the Ogg Vorbis file is larger than the mp3, re-transcode the Ogg Vorbis file with a lower quality setting (in increments of 0.1) until the Ogg Vorbis file is smaller than the mp3.
9. Record data:
• FLAC Average Bitrate (measured with the XMMS FLAC plugin)
• FLAC Compression Ratio (measured with the XMMS FLAC plugin)
• MP3 File Size (measured with the GNU/Linux command line utility "ls")
• MP3 Average Bitrate (measured with the XMMS mp3 plugin)
• Vorbis File Size (measured with the GNU/Linux command line utility "ls")
• Vorbis Quality Setting (recorded with each encoding)
• Vorbis Nominal Bitrate (measured with the XMMS Vorbis plugin)
• Vorbis Average Bitrate (measured with the "ogginfo" utility)

results

The table of results follows below. Data is sorted by Vorbis quality then by Vorbis average bitrate.

*Slightly smaller than mp3 average bitrate because Vorbis files were prepared slightly smaller than the mp3s.

conclusion

This is not a comparison of audible quality! Only personal listening tests could be used to determine which of these mp3s sound better than these Vorbis files, or vice versa, and the results of those kinds of tests are highly influenced by the quality of your equipment and ears. What this experiment does find is that, on average, Ogg Vorbis files encoded with a quality setting of 6.5 are approximately the same size as files encoded with "lame --alt-preset standard", although the range of Vorbis quality settings is rather wide - from "-q 8.3" to "-q 5.4".

If there is any judgement to be made on the quality of these two encoders, it is this: files at the top of the list are more efficiently encoded with Vorbis, and files at the bottom are more efficiently encoded with LAME. That is, Vorbis can use higher nominal bitrates with the files at the top of the list while still producing a file size equal to that produced by LAME. Again, this does not mean these files sound better with Vorbis, although they should. The reverse could be said about the files at the bottom of the list, with LAME appearing to be the more efficient encoder.

It is important to remember that this test was designed using two codecs with settings that most people will find transparent nearly all of the time. This means that if proper listening tests were performed on these files it is not likely that most people would hear any differences.

Most listening tests compare files produced with a particular target (nominal) bitrate. This experiment shows how greatly the average bitrate can vary from the target bitrate. Given that the primary (if not the sole) motivation for audio compression is about file size, it would probably be more fair if listening tests compared files of equal sizes, not equivalent target bitrates. Unfortunately, there is no simple way to produce a file of a given size using today's codecs - the only way is through a test similar to this, encoding and re-encoding a file until the desired file size is produced.

While I do not expect the nature of listening tests to change, I do hope that the issue of file size is not overlooked in our comparisons. Transparency and/or quality with the lowest possible file size, not necessarily bitrate, should be our clearly stated goal.

No CPU cycles were harmed during the production of this experiment. But many, many were used.