By Werner Ruotsalainen on Tue, 04/24/2012
I've recently received an e-mail with a question from one of my readers, Eric Vinicius, asking for my opinion on the effects of upscaling Full HD video (1920*1080) to the slightly wider (2048 pixels) screen of the iPad 3 to fill it entirely and not to leave a 64-pixel black bar on the two sides of it.
Today, now that I've created a highly reliable, high-quality test video (it's available HERE), I've finally run some tests to find out what the situation really is.
First and foremost, let me present you two (zoomed-in) crops of two still images. (Note that this is the standardized ISO 12233 resolution chart I use in almost all my camera benchmarks, starting with THIS.) Which of the two do you prefer?
Surely the first (the original), and not the second (the resized). It's just better-defined with less smoothing artifacts. Check out, for example, the contours of the letters. If you want to have something easily comparable, check out the aliasing (blurring) at the very top of the number “6”. There isn't a single pixel aliasing on the non-resized version, while there is with the resized one.
The first shot has been cropped from the above-mentioned movie played back at original (nominal) Full HD resolution; the second from the first after applying a 1920 -> 2048 (with locked aspect ratio conversion). The full-res image (the one I've cropped from) of the latter is HERE and that of the former HERE.
This is a well-known effect: if you apply any kind of magnification to an image, it'll become (somewhat) blurry – unless the magnification factor is an integer (2, 3 etc.). No wonder Apple have also only used an integer magnification (of two) when introducing Retina screens for both the iPhone and the iPad. Anything else will result in a blurred image – as can be clearly seen in the above comparative shots.
How this all applies to the iPad 3?
The situation is similar on the (new) iPad (3), where, if you play back content on the built-in, 2048*1538 screen, an upsizing will always take place and you just can't get rid of it as there's no “no full screen” icon in the stock, built-in Videos app or the vast majority of third-party apps. The third-party apps that support non-fullscreen mode, TopPlayerHD, eXPlayer HD and It's Playing (these are all listed in the "Video playback in non-fullscreen window?" row of the work-in-progress main chart of my forthcoming iOS Multimedia Bible), use a much smaller window and not the native size of the video).
The following screenshots and crops show a situation very similar to the first two crops presented above.
First, HERE is a screenshot of the iPad 3 playing back the video (using hardware decoding) in full screen.
If you compare it to the unaltered (Full HD-sized), already-introduced MBP shot, you'll see the upsized version exhibits exactly the same upsizing artifacts as the ones I've done on the desktop. As with that upsizing, there without doubt IS some difference and yes, you've guessed: not in the favor of the iPad. To what extent? Let's take a closer look on the fine detail, as we've done in the introduction section. I give you three, oops, two guesses: which of the following two heavily zoomed-in crops have been taken on the iPad 3 and which on my natively Full HD-wide Macbook Pro?
You may have guessed: the first is that of the iPad 3 and the second is that of the MPB.
Incidentally, I've also scrutinized the (comparative) resolution difference vertically. They're as follows (iPad 3 first, non-upsized MBP second):
As you can see, the effective vertical resolution also (as with the horizontal resolution) definitely suffers because of the slight upsizing. The contours of the numbers are much less defined than on the (non-upsized) MBP shot, as with the previous pair of shots.
Unfortunately, currently, none of the third-party video players offer a windowed playback mode either, not even when using software decoding, let alone the hardware one. (Not that anyone would want to use software decoders in ANY of these apps at Full HD – the CPU power is plain insufficient to correctly decode hi-res videos even in the latest iDevices.)
All in all, the situation is indeed worse than it should be. Apple should provide some kind of a setting in Videos like “Black bars around Full HD videos (to avoid oversampling)” for people that would happily trade in some screen estate for as high resolution as possible.
Side remark: what about external monitors? Is there any quality degradation on them?
Note that this all (quality degradation because of upsizing) doesn't apply when playing video using the VGA or the HDMI adapter on all new A5(X)-based devices - that is, ones (currently, the iPad 2, 3 and the iPhone 4S) capable of driving external monitors or TV sets in Full HD resolution over VGA/HDMI, as opposed to A4-based ones (namely, the iPad 1, the iPhone 4 and the iPod touch 4) that are only capable of reduced, XGA (when using the VGA adapter) / 720p (using the HDMI adapter) resolution or even older devices (the iPhone / iPod touch models before the iPhone 4 and the iPod touch 4) that could only make use of the composite / component cables and output Standard Definition, low-quality video.
If the connected monitor is capable of displaying Full HD resolution (its native resolution is at least 1920*1080), the content will be played in that resolution and no upsizing – and subsequent image quality degradation – will take place when driven from an iPad 2, 3 or the iPhone 4S. This only applies to native, dedicated playback and not to the traditional screen mirroring, which is, by default, enabled on the iPad 2/3 and the iPhone 4S, and uses a far lower resolution and a 4:3 screen aspect ratio. The built-in Videos supports native, dedicated playback over VGA/HDMI; so do the following media players (as of now): 8player, BUZZ Player HD, AVPlayerHD, GoodPlayer, CineXPlayer, EC Player (I haven't listed ones incapable of using TV output in hardware-decoding and only in software-decoding mode).
As usual, the latter information can also be found in my all-in-one chart of the forthcoming iOS Multimedia bible. See the "HDMI" row (the information also applies to other cable types (at least VGA), not only HDMI, of course). Note that the row contains additional, player-specific info on enabling TV output if it's not auto-enabled and the possible problems of the TV output-capable but, in some way, in this regard, buggy players - consult the chart if you need this info.
What can be done?
Fortunately, the solution is far easier than many would think. Just show the video in a non-fullscreen screen area. This means you'll want to put it in the following frame (instead of just making it fullscreen by executing [mp setFullscreen:YES animated:YES];):
[mp.view setFrame:CGRectMake(32/2, 200, 1920/2, 1080/2)];
HERE's a full-screen screenshot of the same test video file played back in this mode. If you do zoom in, you can easily notice that it has indeed the same clarity as the video played back using native resolution (without upsizing). A zoomed-in shot showing the vertical bars you may want to compare to the previous horizontal crops:
As you can see, there's absolutely no quality degradation.
Benchmarks, speed, CPU usage
Of course, I've made some rigorous benchmarks (using my standard benchmarking files – see my previous, dedicated article) to see whether there's any performance hit introduced by my algorithm. As was easy to predict, there isn't. The hardware decoder plays back the H.264 video as effectively as in fullscreen mode – but with a definitely higher clarity because of the lack of upsizing.
Accessing my player
(NOTE: this is the first time any article on this problem is posted. In addition, a working(!), non-theoretical solution with full source code(!) is presented – hence the “EXCLUSIVE” in the title of the article.)
As usual, I've made available the full source code (it's HERE), with the resolution tester video bundled. Feel free to change it to any other (H.264) video file (just make sure you also rewrite the code to refer to the new filename.)
I've put the new code in my 1.5-year-long source code framework I've been using for teaching video playback, layering and (SRT) subtitle file processing. (As many of you surely know, I also teach iOS programming. This stuff is also part of my material.) Unfortunately, when you use setFrame instead of just calling setFullscreen, you have to rotate the video player itself; hence the loooong deviceRotated with tons of manual rotating, transiting and scaling.
A message to all media player developers
(I'll also mail this to all known media player devs so that they can introduce this feature to their players as soon as possible.)
Now that I've shown the advantages of using non-fullscreen playback and even provided a demo code of implementing it, please do consider implementing it in your own player. The iPad3 has always been a VERY good media player but, with my discovery, it has become even better than before. If you (or your customers) are like me (love watching Full HD video in bed with the iPad close to my eyes), the slight decrease in the size of the playback area won't be an issue, while the quality increase will surely be visible.
(You may send me some “greetings” in the Credits and/or Thanks section in Help or the app description ;-) ).
Feel free to ask any questions on the algorithm.
If you don't have a developer account to deploy the source...
… but would like to use the code for the additional clarity: then you may want to wait for third-party apps implementing this feature. Or mine. I'll clean up the code, add File Sharing access to the Documents directory via iTunes and a file selector dialog and will publish the player in the AppStore – for free and as an open source player. I'll keep you all posted on news on the media player I'll release.
UPDATE (some hours later): I've released a cleaned up version, now with iTunes File Sharing support. I've migrated all the AppDelegate code to the detail view controller, which means handling rotation can be done much more easily. (No custom code is needed to rotate / reposition anything.) The sources are HERE. Note that it's really a barebones application - actually, the number of custom instructions it contains is 12 (root view controller) + 8 (detail view) + one additional entry in the info.plist to allow for File Sharing + one additional framework added. As you can see, one can code a simple, but, image quality-wise, still great (much better than anything else out there at the moment) video player with directory listing support using no more than 20(!) instructions!
Just deploy it on your iPad 3; after this (not that it'll list it under Apps), you'll already be able to drop H.264 (M4V/MOV) files into it via iTunes File Sharin. Fire up the app, select a flick and enjoy it ;-)