lurkertech.com → Lurker's Guide → Square and non-square pixels

Square and Non-Square Pixels

By Chris Pirazzi. Thanks to Bob Williams for much of the information. Thanks to Charles Poynton for pointer to RP 187 and other info.

Table of Contents

• What Are They?
• Why Do I Care?
• What Is the Pixel Aspect Ratio of Non-Square Pixels?
  • Why Isn't It 4:3 or 16:9?
  • Why Isn't It What My Software App or Library Says?
  • Why Isn't It What SMPTE RP 187 Says?
  • Why Isn't It What MPEG Says?
  • Why Isn't It What HDMI / CEA-861 Says?
  • Why Isn't It 720/640 or 720/768?
    • Wait, How Can That Work?
• How do I Convert Between Square and Non-Square?
• How Do I Tell Which Pixel Aspect Ratio I Have?
• Picture Aspect Ratio or Frame Aspect Ratio

What Are They?

Pixels in the graphics world are square. A 100 pixel vertical line is the same length as a 100 pixel horizontal line on a graphics monitor.

Some pixels in the standard-def video world (specifically, the digital electrical signals universally used in studios for 480i and 576i production, as defined by the infamous ITU-R BT.601-4 or "Rec. 601") are non-square. A 100 pixel vertical line may be longer or shorter than a 100 pixel horizontal line on a video monitor, depending on the video system.

Pixels in the HD (e.g. 1080i and 720p) video world are, fortunately, square.

The term which describes this squareness or non-squareness is pixel aspect ratio, expressed as a fraction of horizontal (x) pixel size divided by vertical (y) pixel size. The pixel aspect ratio for square pixels is 1/1.

Why Do I Care?

When you as software programmer manipulate standard-definition video images in your application's memory, those images will often have the very same non-squareness as the Rec. 601 studio signals. This can be true whether the pixels came to you uncompressed, or packaged inside compressed chunks like MPEG frames, M-JPEG fields, or DV fields. Even if you download a video stream off the internet, it still might contain non-square pixels (because the original studio cameras that captured it produced non-square pixels, or it had to be converted to non-square for encoding).

Any rendering of geometry-based graphics to an image, including text, must take the pixel aspect ratio into account, otherwise it will come out "stretched" or "squished."

Many image processing operations assume a certain pixel aspect ratio. For example, a blur operation with a radially symmetric kernel may look bad on non-square pixels.

What Is the Pixel Aspect Ratio of Non-Square Pixels?

The pixel aspect ratio of non-square pixels is:

System	hSpacing	vSpacing
480i	10	11
576i	59	54

You can think of these values as the ratio of the width of a pixel to the height of a pixel. For example, say you want to draw a circle that appears round on the display device and whose diameter is n horizontal pixels (luma sampling instants). Draw an ellipse which is n pixels wide and n*hSpacing/vSpacing pixels (picture lines) high. Notice that vSpacing is in the denominator: the greater the vertical spacing of pixels (picture lines), the fewer vertical pixels you need in order to match a given number of horizontal pixels (luma sampling instants) on the display device.

For a full explanation of where these values come from, please read:

What Pixels are Square? How Non-Square is Non-Square?

Why Isn't It 4:3 or 16:9?

You may be wondering why the pixel aspect ratio of non-square pixels is not 4:3, or 16:9. In this case you are confusing pixel aspect ratio with picture aspect ratio, a separate concept which we define below.

Why Isn't It What My Software App or Library Says?

The pixel aspect ratio of non-square pixels was completely and unchageably determined at the moment that video engineers chose luma sampling frequencies for their square and non-square devices.

This happened before almost any video software got written.

The problem was, software folks like us didn't realize that. What ensued was a melee of confusion and religious wars about what the pixel aspect ratio was, that took place at the library (QuickTime, Video for Windows, DirectShow, ...), application (Premiere, Final Cut, After Effects, ...) and documentation (Video Demystified, ...) level.

There's some details on the confusion in this section of my document on video systems.

If you've got a standard-def video project in After Effects or Final Cut or Premiere or any other app and the project settings claim to use pixel aspect ratios other than the ones above, 99% of the time the apps are simply flat-out wrong, and are mis-interpreting (or mis-labeling) the pixel aspect ratio of the video footage you have loaded into them. Sometimes, it all works out ok: even though the app is delusional about the pixel aspect ratio of its own data, it assumes all input and output video footage has the same, incorrect pixel aspect ratio, so it doesn't "damage" any footage on the way through the program. But problems can arise when:

• The software itself is creating synthetic elements on top of your input video footage, or the video consists entirely of synthetic elements created by the software itself. For example, imagine you overlay (what is supposed to be) a square over the video, or you produce a video that consists entirely of (what is supposed to be) a square. If the app has an incorrect idea of pixel aspect ratio, then when you play the final rendered output on a video monitor, the original video footage will look ok but the "square" won't actually be square: it will be slightly stretched or squished.

• The software is doing its own conversion between non-square pixels (say, for example, coming from a non-square uncompressed video file on input) and square pixels (say, for example, to output the movie to a format suitable for a web-based movie player), or vice versa. In this case, the software will produce video that looks slightly stretched or squished when played on the final device.

If you have one of these problems, sucks to be you. In some cases, the erroneous pixel aspect ratios are close enough to the real pixel aspect ratios that the difference is small (the image might be stretched or squished by only a few pixels) and can be ignored. But in some cases, the difference is critical: for example, if you need the output of your app to line up perfectly with the original footage so that it can be edited back into the original footage.

Why Isn't It What SMPTE RP 187 Says?

SMPTE RP 187-1995 was designed to solve this problem once and for all, and it had many good effects for high-def video. But sadly, for standard-definition video, it was too late, and came up with some unworkable proposals, both in the area of pixel aspect ratio and production aperture, so it was effectively ignored.

Why Isn't It What MPEG Says?

The MPEG-1 and MPEG-2 standards have some rather entertaining things to say about pixel aspect ratio. MPEG-1 labels were unable to represent the actual ratios and MPEG-2 labels made it very difficult, so most MPEG streams are actually mislabeled, as explained here.

Why Isn't It What HDMI / CEA-861 Says?

Unbelievably, the specs on which the modern consumer digital electrical video standard HDMI is based also give incorrect pixel aspect ratio numbers, which are causing mass confusion problems in modern equipment. For more on the sordid details, see here.

Why Isn't It 720/640 or 720/768?

You may have noticed that non-square pixel data tends to come in buffers that are 720 pixels wide (720 comes from ITU-R BT.601-4, which started this whole mess).

Furthermore, you may have noticed that square data tends to come in buffers that are 640 pixels wide (480i video system) or 768 pixels wide (576i video system).

So why, you might ask, isn't the pixel aspect ratio of 480i video equal to 720/640?

And why, you might ask, isn't the pixel aspect ratio of 576i video equal to 720/768?

The answer is that that is not the criterion the video engineers used to choose the pixel aspect ratio. For all the sordid details on that, please read this section of my video systems introduction.

Wait, How Can That Work?

"Ok smarty," you may now be saying, "if the pixel aspect ratio is not 720/640 for 480i, or 768/720 for 576i, then how can I resolve this with the fact that my images are 720, 640, or 768 pixels wide?"

Put another way, "How do I deal with this math:"

• 480i square: 640
• 480i non-square: 720 * (10/11) = 654 6/11, which is not equal to 640!
• 576i square: 768
• 576i non-square: 720 * (59/54) = 786 2/3, which is not equal to 768!

The answer to this excellent question is that a 720-pixel-wide non-square pixel image and a 640- or 768-pixel wide square pixel image do not represent the same horizontal range of the underlying signal!

480i 720 non-square pixels 640 square pixels

576i 720 non-square pixels 768 square pixels

When converting, you need to scale by 10/11 or 59/54 and then pad or crop. We'll give you the details below.

To understand exactly what we mean by "underlying signal" (you're probably thinking "What do you mean? The edge of the image is the edge of the image!"), I recommmend that you pop over to learn some of the shocking secrets about video signals on this page:

Programmer's Guide to Video Systems

How do I Convert Between Square and Non-Square?

Here is how to convert a 720-wide non-square image into a 640- or 768-wide square image:

480i Non-Square Sampling 720 x 486		576i Non-Square Sampling 720 x 576
↓ scale horizontally by 10/11 ↓ 654 6/11 x 486 ↓ crop to 640 ↓		↓ scale horizontally by 59/54 ↓ 786 2/3 x 576 ↓ crop to 768 ↓
640 x 486 480i Square Sampling		768 x 576 576i Square Sampling

When you crop, you should keep the image centered.

Here is how to convert a 640- or 768-wide square image into a 720-wide non-square image:

480i Square Sampling 640 x 486		576i Square Sampling 768 x 576
↓ scale horizontally by 11/10 ↓ 704 x 486 ↓ pad to 720 ↓		↓ scale horizontally by 54/59 ↓ 702 54/59 x 576 ↓ pad to 720 ↓
720 x 486 480i Non-Square Sampling		720 x 576 576i Non-Square Sampling

"Pad" means need to pad the image out to 720 non-square pixels, keeping the image centered. A better way to do it, if you can, is to start with a greater number of square pixels than 640 or 768 (again with the same horizontal center), and scale those directly to 720.

A video input or output device which is encoding or decoding an analog signal will never need to pad or crop: it can measure or synthesize the continuous analog signal for each video line any way it wants. Each analog video line is long enough to accomodate either horizontal range.

Important Note: In this section, we gave some conversions for 480i and 576i images using one particular set of image dimensions as an illustrative example. The particular image dimensions we chose are just examples; you should not read any magic significance into them. 480i and 576i video can come in a huge variety of dimensions (e.g. for 480i you'll find 640x480, 640x486, 704x486, 720x486 and others), but in every case the process of converting between square and non-square is the same:

• First you scale the input pixels horizontally by:
  • 480i non-square to square: 10/11
  • 480i square to non-square: 11/10
  • 576i non-square to square: 59/54
  • 576i square to non-square: 54/59
  Notice how these ratios do not depend on either the horizontal or vertical image dimensions.

• Then you crop or pad horizontally as necessary to reach your desired horizontal output dimension in pixels (keeping the image center in the same place).

It is critical to understand that the scaling step does not depend on either the horizontal or vertical image dimensions, and the crop/pad step depends only on the horizontal input and output image dimensions. The vertical image dimensions play no role whatsoever in either step. The MPEG committee and so many others have misunderstood this point and as a result given birth to many incorrect scaling ratios, as explained here. If you find yourself repeatedly coming back to the image dimensions as a core parameters of the conversion, then go back and read What Pixels are Square? How Non-Square is Non-Square? again.

How Do I Tell Which Pixel Aspect Ratio I Have?

It depends on the situation, but here are some hints:

• If you are presented with some data, a major hint that you have non-square pixels is that your images will be 720 pixels wide (or 704, in the case of MPEG). 640 and 768 are major hints of square data.

• Certain types of compressed data will always be one or the other. For example, DVC (DV/DVCPro/1394/FireWire...) data will always be non-square because that is what is called for in the DVC spec. M-JPEG data could be either, unfortunately. MPEG data could also be either but is typically non-square (and also has the size 704 instead of 720 for annoying MPEG reasons).

• Any drawing to a graphics monitor using OpenGL/GDI/DirectDraw/etc. is in square pixels (unless you are drawing to a TV monitor using some funky non-square mode of your graphics card).

• As we graphically explained in our document about video systems, an analog video signal consists of discrete video lines, each of which is a continuous analog signal that represents the luma and chroma for that line of the image. So an analog video signal has no "pixels." When a video board samples a video line, it can sample whatever part of the line it pleases, and it can sample the line at whatever interval it pleases. Practically speaking, it will choose either square or non-square sampling, but you'll likely have to consult your API or manual to find out which.

• A digital electrical video signal (e.g. SDI) itself contains non-square pixels, so if you are capturing or playing SDI, it is most likely that your in-memory buffers will be non-square as well (unless your video device has a scaler). Again consult your API or manual.

Picture Aspect Ratio or Frame Aspect Ratio

Pixel aspect ratio is different than picture aspect ratio (sometimes also called frame aspect ratio), which is a fraction of total hoizontal (x) picture size over total vertical (y) picture size, for a stated definition of "picture."