H.264 vs H.265: The Evolution Of Video Codecs

by Paul Arnote

Ever since computers became commonplace and mainstream, users have sought ways to incorporate video into their computing lives. In 1984, the CCITT (now called the ITU-T) created the first main video compression standard, then called H.120. Its quality was quite poor, especially when compared to what computer users today are accustomed to. Despite its poor quality, it allowed developers to refine their approach to video encoding, and to achieve the high quality video we all can access today.

Video files, especially high quality ones, have always been behemoths when it came to file size. While that is still true today, the video codecs used today to encode and compress video result in infinitely smaller file sizes than raw video, and significantly smaller file sizes than those codecs used just 20 years ago.

I'm sure that many of you reading this can remember the infancy of home video, with the relatively low resolution VHS and Beta tape systems (compared to DVD and Blu Ray, VHS has resolution of 240 x 320, vs. 480 x 720 for DVD, and 1080 x 1920 for Blu Ray). Boy, oh boy, have we ever come a long, long way since those days! Advances in video compression standards gave users DVDs, which offered much higher resolution than the tape-based home video systems, and much clearer pictures as a result. DVDs are currently in the process of being supplanted by Blu Ray discs.

The progression and advances in the past 40 years has been mind boggling. As CPU speeds have steadily increased, coupled with drastic advances in the video processing capabilities of today's video cards, advances were inevitable. Add in the (over)abundance of mobile platforms recording and streaming unprecedented amounts of video, those mobile networks crave a video compression standard that results in smaller file sizes with higher video quality, all so they can save precious (and limited) bandwidth on their networks while maintaining the quality that users expect and demand. So, the service providers to all those mobile devices running around out there have an intense interest in supporting/encouraging the development of more efficient video compression standards. A quick search of ITU-T members shows all three of the "big" U.S. mobile networks (Verizon, T Mobile, and AT&T) as members.

I'll leave the "history" of the video compression standards to Wikipedia, which covers them quite nicely. We'll focus our discussion more on the two current video compression standards that are available and used today, and the two that dominate today's video landscape. We'll also take a brief peek into the future of video compression.

H.264

H.264, also referred to as AVC, or Advanced Video Coding, was released in 2003. It gave rise to the very common *.mp4 video files we all are familiar with. In fact, H.264 is the most common video compression standard used to encode videos on Blu Ray discs.

Having come out 18 years ago, the H.264 standard was released before 4K, and now 8K, video was available. (Forget for a moment that you cannot even see in 4K, much less 8K, but that is a discussion for another time.) You will often see the open source implementation of H.264 referred to as x264.

As such, H.264 is starting to show its age. It is **just barely** able to handle 4K video. As for 8K video, fuggedaboutit. It has to do with how H.264 is set up to handle CTUs, or coding tree units. H.264 can handle between 4x4 and 16x16 pixel block size CTUs. As such, H.264 peaks out just at the level of 4K video.

H.265

As standards started to be drawn up for 4K video and beyond, it became apparent that the H.264 standard would need to evolve in order for video compression requirements to keep pace.

Released in 2013, H.265 expands CTU capabilities from a 16x16 block of pixels, up to a CTU of 64x64, and an assortment of other enhancements that affect performance and file size. H.265 is also known as High Efficiency Video Coding. The open source version of the H.265 codec is also known as x265.

Here's the REAL upside to H.265: smaller file sizes. When a video is encoded with H.265 versus H.264, the H.265 video, when encoded at the same resolution as the H.264 video, will be 40 to 50 percent smaller in file size. So, users can expect clearer images compressed into a smaller space than ever before. In my own experiences, H.265 encoded video files were around 40 percent smaller than their H.264 encoded counterparts. Encoding was done on the same machine, under similar loads, at the same resolution.

So, if there's an upside, there's always a downside. That downside is, due to the complexity of the video compression algorithm, encoding is much slower with H.265 than H.264. In my own tests, H.265 video took at least two to three times more time to encode than the same video encoded with H.264. Using Handbrake (available in the PCLinuxOS repository), converting a 93 minute *.webm encoded video to H.264 encoded *.mp4 video took roughly two hours to complete its encoding. Just changing the encoder to using H.265 (and leaving all the other settings the same as the H.264 encoding), that same 93 minute video took over five hours to encode in a H.265 encoded *.mp4 file.

As far as the decoding part (which is what happens when you watch the playback of a compressed video), that appeared to be seamless and fully transparent to the end user … which is what you want. The non-technical consumer of video really doesn't care about the encoding side of things. They are pleased to just be able to watch high quality video playback.

H.265 appears to be the encoder d'jour, now represented by up to 40 percent of Blu Ray discs using the encoder to compress video. It offers disc makers and content creators a choice of either having smaller file sizes at the same resolution, enabling them to offer even more "extra" content on discs, or offering even higher resolution video than previously available in the same amount of space used by H.264 encoded video.

The newer video compression standard supports up to 8K UHD video, and is the second most widely used video coding format, right after H.264/AVC.

On a side note, this month's Tip Top Tips column, featuring a tip from Archie, centers around converting video to H.265 using VLC. As with all tips, your mileage may vary, and mine did when attempting to do the conversion with VLC. My copy of VLC would crash whenever attempting to use it to convert any video to the H.265 encoder. Nearly all of my "experience" using the H.265 encoder has been with using Handbrake as the program that handles the re-encoding/transcoding. Using Handbrake, I've never experienced any problems, aside from my own impatience, brought on by the much slower encoding of H.265 video. In the end, I'm always pleased by the high quality video resolution of H.265 encoded videos and the significantly smaller file sizes.

What's The Future?

Don't think that the ITU-T is resting on its hind quarters, happy with where H.265 has taken them. Nope. They are already working on the next version of video compression. It seems that the "future" may be just around the corner.

In July 2020, the H.266 video compression standard was finalized, with the aim of making 4K broadcast and streaming commercially viable. It is also being dubbed the "Versatile Video Coding" video compression standard. (Is it just me, or should they be running out of superlative adjectives to name these new video compression standards by now?)

The early claims surrounding VVC/H.266 is that it will be capable of yet another 50 percent reduction in file sizes over H.265. When compared to H.264 (by far the most prevalent video compression encoder in use at the time of the writing of this article), that will represent high quality video, stored in a file that is roughly only one-quarter of the H.264 file size. VVC/H.266 will also support up to 16K video, as well as 360 degree videos.

All of this comes at a cost. Remember me telling you that H.265 video encoding takes two to three times as long to encode as it does to encode a video using H.264? Well, it seems that VVC/H.266 will take up to 10 times as long as using H.265! There is a significant cost in computing and the time it took to encode H.266 video, according to an article about the new video compression standard on Bitmovin.com. Decoding complexity of H.266 is expected to be roughly twice that of H.265, too.

There is currently no x266 (open source) encoder for H.266 available at this time, either. However, the same group, MulticoreWare, who oversaw the creation of the x265 codec, is also overseeing the creation of the forthcoming x266 open source implementation of the H.266 video compressor.

I suspect that the resource-heavy compression and encoding of H.266 will be somewhat mitigated by newer, more powerful CPUs and GPUs, as they come out on the market for the general consumer.

Also, don't discount some company or consortium that just dumps everything that ITU-T has done, and tries to take the whole process in a whole new direction. Even though it seems like it might be a colossal undertaking, there are companies out there doing just that. One such company is Cinegy LLC, with their Daniel2 codec. They have literally thrown out the bath water, the tub and the baby, and left all legacy dependencies and old codec architectures behind to create a fully-GPU based encoder/decoder. In the process, they claim to have created the world's fastest decoder. It's no secret that GPUs use incredibly fast processors (typically, far faster than CPUs), so it makes sense to offload some of the CPU-intensive work to the GPU. And, it's going to be the GPU anyways that is responsible for displaying the video image, thereby eliminating the bottleneck between the slower CPU and the faster GPU.

Here's the summary statement by Cinegy about the Daniel2 codec, from their website:

"Traditionally video codecs have been defined by chip makers and not by software developers. Daniel2 is different. It was designed by software developers that for decades had suffered implementing "industry standard" codecs such as MPEG2 or H.264 in software running on ordinary PC hardware, but these had not been created with PC hardware in mind. Daniel2 was designed to be a massively multithreaded parallel processing software codec that runs extremely fast on CPUs and even faster on GPUs with their thousands of parallel cores and ultra fast RAM."

It will be interesting to see if the Daniel2 codec, or any other alternative codec, receives any traction in the market, which could prove to be an insurmountable obstacle. Users aren't going to be any too happy to lose legacy support, which is part of what Daniel2 does. Still, since "the Daniel2 codec is acquisition and production codec meant to be used for recording from camera sources, editing and post-production as well as playout," users may still see some benefit from its use earlier on in the production cycle.

Summary

Things could get quite interesting for video enthusiasts in the next few coming years. Higher resolution video stored in smaller files sounds like a win, win, win for everyone concerned. Sure, the newer video compression codecs are slower to encode and require a LOT of computing power and time, but that is a one time expenditure. The end results will be worth it, allowing users to store more high quality content in the finite spaces they have available.