Resizing Images - Computerphile
HTML-код
- Опубликовано: 21 сен 2024
- Nearest Neighbour and BiLinear resize explained by Dr Mike Pound
Fire Pong: • Fire Pong (Rule Zero) ...
Google Deep Dream: • Deep Dream (Google) - ...
FPS & Digital Video: • FPS & Digital Video - ...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscom...
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
"So there's a 3x3 image. It's- y'know, 's- high quality."
I couldn't stop laughing. It was the way he said it.
Bilinear sampling/scaling is used for almost everything[1] in modern real-time 3D graphics (3D games, etc.) because there are dedicated hardware parts for that operation in every GPU, and as such it's dirt cheap to use. Some games in some situations use nearest neighbour, usually for style, like Minecraft (blocky textures stay blocky however close you are).
Back in the early-to-mid '90s it was usually nearest neighbour everywhere, because there was no GPU to help out, and CPU couldn't afford to waste cycles on interpolation math for every single screen pixel (one read, one write VS one read, three pairs of subtraction, addition and multiplication, and then write). Early 3D home consoles, like Playstation 1. did the same.
The first Unreal game had an interesting approach (in its software rendering mode) - to use dithering instead of interpolation. In that way, no intermediate (interpolated) colours needed to be calculated, but the exact same source colours (like for nearest neighbour) were "shuffled" around to create an ilusion of gradient.
So yeah, those were my 5 cents, maybe someone will find it interesting.
(1 - trilinear and anisotropic just build on top of the same basic idea)
And yet it's soooo slow to do in a software rasterizer. Just as slow or slower than anisotropic texture lookups :(
Never seen a youtube comment this relevant and informative. Thanks :)
soylentgreenb Quake used 16 pixel intervals for perspective division to essentially "hide" that cost, because you had some parallelism between integer and floating point operations (done on FPU), so it dispatches perspective calculations, does inbetween pixels using integer math, and then the floating point results arrive just-in-time for next batch of pixels. Very clever :)
Thank you, man! I should confess most of these things I couldn't fully understand, but I clearly see a lot of valuable information here.
Oh wow. Most people only have 2 cents to give but you have 5! very generous of you
"If you're a pixel-artist, doing... uh, you know... pixel-art..."
Ah yes, finally somebody understanding what's really important at being a pixel artist!
Sarcasm at its finest 🫠🤌
I just love this guy :). Brilliant at explaining these things. He makes a lot of stuff sound real easy, and in this case it really is, but on myself or by the usual professor it would definitely take longer to understand. Or maybe I just have a different kind of attention in class than while watching RUclips. And the graphics always are on point as well. Not too much info to distract, but right enough to consume the information easier.
+Xaab Xaa thanks, Mike will appreciate that & a lot of thought goes into those graphics! >Sean
Totally agree with you! He makes it seem so easy and explains it pretty damn good
Very interesting as always! :)
+Xaab Xaa I wanted to write the same but now I simply put my autograph under your comment!
Yeah i love Mike as well. He is great at explaining stuff.
Damn I get so excited when Dr. Mike Pound is in the thumbnail. XD
***** the password video blew my mind. Even though I've known for a while the importance of a strong password, he just delivered the message so well. :D
RobustEnigma agreed! watched his password video this morning and then spent the next two hours changing all my passwords
Manu Leach hahaha, LastPass is my friend! 9+ for basics, 13+ for important stuff. Mike is awesome!
Sounds like you like to get taken to Pound town...
@@Fingerblasterstudios or Poundland
Explaining bilinear and bicubic with the side view was very illustrative and conceptually helpful, thank you!
Its 'simple' things like these we take for granted but when looked at closer it makes you realize how absurdly fast an average computer is these days. Being able to do this at a massive scale many times per second.
Underrated comment.
You are right is my computer slow? or am I asking it to do too much?
p.s. I remember when computers really were slow - programming games on a Commadore Pet
A 9x9 parker square
+ To da top you go!
+ get this man higher up
+
+ I tip my hat to you.
hahahaha
I think of this kind of stuff every time I see a cop-drama where they have some low resolution image and then magically "blow it up" to get a license plate number or something else that was clearly not visible before. That's not how this works, television!
I know right, it's so annoying. I think only once have I seen a TV show where they blow up the image and it remains blurry.
Jay, I wanna restore your comment, but RUclips won't let me.
He posted a great video of tons of examples from pop culture. Search "Let's Enhance" on RUclips.
Enhance!
Gotta love mystery diners!
If you have a low resolution video you might be able to do something.
Gee wonder why this has shown up in my feed today 😂
Really like this guy, he explains it so well and it's a joy to watch him talk about this stuff!
Mike is just phenomenal at explaining how things work. What a brilliant guy.
Why couldn't I have had teachers with such clarity when I was going through school? On some of these videos I gain more perspective in 10 minutes than I did with entire lectures in class. Keep up the great work, this channel is awesome!
Mike is my favorite Computerphile member, I just love his videos
more with Mike please! As a student of computer science I can tell that he is way more interesting to listen to than many of my professors!
More Mike videos, please. This guy has so good explanations and the way of talking, just wow. Good.
"3x3 image. It's you know, high quality" lmfao xD
The way he explained was outstanding!
I love this channel. I know about computers, and it confirms things I thought or gives me the small and fun details that you may not learn in uni.
Dr. Mike Pound. I really enjoy when i listen to this guy. It is like he can explain anything to you. He was born to teach!.
"But that's for another video"
Gah! Again you tease us with a cliffhanger!
Literally watch the videos for Dr. Mike
I understand the "basics" in the videos Dr Mike Pound is starring but the math itself I have no clue whatsoever. I wish my math teacher in highschool was as talented in the art of teaching. I find "math" intriguing but I got lost somewhere along the road some 20 years ago. As usual A+ Dr Mike Pound video :D
Unfortunately a lot of the math that goes into the topics he covers aren't high-school-level math. You should go check out some books and lectures online. Don't let crappy high school teachers stop your education :D
Since you mentioned zooming out, can you do a video on anti-aliasing ?
Back in the day I did interpolation of wind fields on a map using a sparse set of weather station data. One of my algorithms was inverse-square weighted. I could use a selected set of weather stations to interpolate a particular grid point - even ones far away from the point. The inverse square effect made far ones negligible and heavily favoured near ones.
The animation at 7:49 was a brilliant clarification of what he was trying to get at.
It actually helps A LOT to think of pixels not as points in the picture but as areas.
deep learning ai: "Hold my beer"
This man needs more books on his shelves.
This is about the only guy I en joy watching in this series.
New to computerphile but I must say that this presenter is fantastic!
Thank you computerphile. I know this is an old video but this video helped me to understand GIS interpolations.
Do add this to the Computer Vision series. :)
I see a lot more videos on CV but haven't been added to the playlist.
Thankyou for making them. ^ ^
This explanation is sooo much better than the one in the book I studied from! thanks!
The elements of statistical learning in the background, nice
Nearest Neighbor is interesting since, in a sense, it allows for lossless scaling. If your scale factor is an integer, if you upscale, you can always downscale later on and return to your original image without any loss.
I’ve found this to be a useful method for reducing analog noise in SNES captures since the noise gets averaged out (i.e. capturing at a 2x scale and transcoding losslessly to a 1x scale, allowing for arbitrary Nearest Neighbor scaling later on during post-processing). Nearest Neighbor gets most of its usefulness when paired with an integer scale factor.
Deep learing algorithms like waifu2x have a lot of potential for upscaling images. They are also a way to add real "new" information to images when upscaling. And for video there is Superresolution of course.
Superresolution is just a bigger image displayed in a frame which is smaller in size, right? That does involve resizing, but it has nothing to do with upscaling. The source image isn't made small then scaled up, it starts huge and scales down. Similar, but very different concept. In upscaling you're (re)creating information that was not there, while in downscaling you're selectively throwing out information and fitting the remainder to a smaller grid. I mean, it uses the same algorithms, but the intent and usage behind it are not the same as what is being described in the video.
Waifu2x does seem interesting, but one must be careful how one trains deep learning algorithms. The name Waifu makes me think they trained it on manga and anime pictures, so it would probably be more suited to rescaling cartoons, and might have strange artifacts when working on photographs.
DEWG Ah yes. That's why I asked if you meant the same kind of superresolution that I knew of, since it seemed odd you would be bringing it up in a discussion of "upscaling", when it's quite clearly the inverse of that and by no means a novel application of scaling in general.
On the other hand, this superresolution that you just showed me is pretty interesting. It's not so much a novel application of upscaling as it is a different idea, entirely. It requires multiple slightly different images tho, so would be more applicable to video than images. It might be useful for professional photographers, but if they truly care about the deep resolution of their images they would likely be using film.
_"The name Waifu makes me think they trained it on manga and anime pictures [...]"_
_"If you're training entirely on road signs, then it may be helpful in up scaling road signs, but it'll fair horribly at reconstructing a face."_
They never said otherwise. That's exactly how it works.
Vulcapyro Not badmouthing it, by any means. I made that comment before I even looked it up... and I was right.
You can indeed add new "real" information while upscaling. But it will be domain specific or wrong. Google "google deep dream"; you can go looking for cats in a picture of grass, and you can find them and put cats in there. If you're upscaling and allow a neural network not specifically trained on grass to fill in the missing pixels while upscaling, you'll just end up with a lawn full of cronenbergs.
Awww. I hope bicubic is coming soon! That looks really interesting!
This whole video was really fascinating, to be honest. I had always just assumed it worked in that way Sean Riley described it at the beginning: that way that's similar to but not quite the same as nearest neighbour.
but what about, for example, the pixels with coordinate 8 in your example? to my understanding, it gets the color of pixel 2 in nearest neighbor, so then column 2 gets 4 new columns and column 0 gets 2 new columns, and that doesn't make any sense. the man behind the camera suggested much more reasonable idea, because in his idea every old column gets 3 new columns. also with bilinear there is a problem with these pixels because they aren't among any pixels, so what is done with them in this method?
I'd love to learn more about image scaling done by tools like Waifu2x that use things like "Deep Convolutional Neural Networks". Really, the images that you can get out of tools like these are AMAZING, especially when it comes to lineart and/or digitally produced art. I believe this particular tool was designed to upscale comic book pages.
The most important part he didn't explain, how to scale down images without creating blockiness or moiré.
Bilinear filtering doesn't work for downscaling images, especially not for factors
Actually, if you're a pixel artist its a bad thing to use that form of nearest neighbor because you want to keep pixel aspect ratio. Good pixel artists will use what Sean says, where 1 pixel becomes 4, 9, 16, etc. I think the more practical form of nearest neighbor (and better illustration) is where 1 pixel becomes the center of a larger grid of pixels depending on your scaling ratio.
A short TL;DR:
PointResize() -> i.e nearest neighbor, crappy
BilinearResize() -> Better, but blurry
BicubicResize() -> Sharper, but with ringing
LanczosResize() -> Slightly less sharper than bicubic, but with way less ringing
BlackmanResize() -> Same as Lanczos but with less ringing
Spline64Resize() -> Sharp and with little ringing (slightly prone to aliasing)
Nowadays what it's used is something like NEEDI3 (Neural Network Edge Directed Interpolation 3) which uses a Neural Network to upscale images rather than applying normal resizing kernels.
Of course there are other resizing kernels like GaussResize(), SincResize() etc but they're for different use.
Nearest Neighbor isn’t always crappy. There’s a video demonstrating that on my channel. That footage would look like garbage were it not for Nearest Neighbor.
Of course, there are plenty more methods for resampling an image: "Pixel Mixing" and "Box Filters" are a few of those. My favorite is Pixel Mixing though, but it seems that very few programs use it. Also, keep in mind that when you resize something, the program *should* (but too often will not) have converted the image to a linear RGB colorspace first, to do the actual math of the resizing, and then convert the image back to it's original colorspace. For most images tossed around on the Internet, the standard is to store them in the sRGB colorspace, which is not a linear colorspace.
what about the upscaling done in emulators, like Super Eagle 2, SAI...
Usually when i want to downsize a photograph to release online the way i get the sharpest results with the least amount of artifacts (aliasing and moiré) is to first determine what my final resolution should be, say it's 1600 pixels wide, and the image is currently 5120px wide. What i do is i first upscale it to 6400px wide (which is 1600x4) using bicubic, then i usually blur the image just a tad using gaussian blur (with a strength of maybe 2px) then i usually sharpen the image a slight bit using unsharp mask and a factor of maybe 1.4. What this does is it reduces the amount of moiré in the final image, the blur and sharpening needs to be adjusted for each image depending on the content. Then finally i do a bilinear resize from 6400px to 1600px and it is the sharpest results i've ever gotten with the least amount of artifacts. I'm not really sure exactly why it produces a lot better results than from resizing directly from the original size but it does.
Really nice explanation, but I would love to see the graphical result of these re-sizing methods
A video about perlin noise would make my day!
That is awesome, I was googling image resizing algorithms a few hours ago.
Great video, as most of them.
You know what would be funny to do? The "paper change" moment that happens in some of the Numberphile videos. =D
After seeing this, I feel like I could actually implement it. Great explanation.
I hope Bicubic also brings up the other methods of which bicubic is just a special case of. And then there's the spline method. And then the lineart methods like hqx and such. They try to only use colors from the original.
I'm excited to learn how resizing smaller works
I would love this revisited in light of the Rittenhouse trial, and types of interpolation, and whether or not they are adding information that didn't exist in the original or if they are true the original source.
I don't understand what you say about scaling down... if I scale down there's no more "pixel values that sit in between the original pixels". I have to throw away some of my original pixels, and map them differently. Maybe if it's a x2 down sizing, take an average of every 4 pixels in the original image to represent 1 pixel in the down-sized image.
Could you leave a link showing what the scaling looks like for the differing techniques?
6:15 It seems that you could simply take a weighted average of all 4 points. You can calculate the distances with the Pythagorean theorem and scale them down proportionally so that they add up to 100%, and use the resulting numbers as weights.
That cliffhanger. Love these videos.
Would be cool if you could do a video on neural net upscaling algorithms like NNEDI3 and Waifu2x also.
I was kind of disappointed that you didn't show an example of how a scaled image with bilinear (and bicubic) interpolation would look, preferably the same image that was used as an example of nearest-neighbor. Other than that, great video as always!
Wouldn't the default assumption be that pixels are "in the middle", the pixel at 0,0 is really rooted at 0.5,0.5, so scaling it 3 times makes it 1.5,1.5. Which for demonstration purposes makes nearest neighbour much more intuitive and correct to what actually happens. As the example @ 2:58 is properly wrong, somehow scaling by a factor of 3 gets you a 2*2 scaled pixel in the topleft?
I assume it's an oversight, but it stuck out to me.
For linear interpolation between two points, the actual formula is:
pixel1 + (pixel2 - pixel1) * weight
That's just another way of saying the same thing. The way Mike phrased was pixel1 * (1-weight) + pixel2 * weight. They are equivalent expressions
Another great video! It would be helpful to see some animations to show how each sampling type changes the look of a 3x3 array into the output array, just for intuition
Here are your court summons.
Explanation is very concise and simple.
Programmers who have implemented Perlin Noise are VERY familiar with pixel interpolation. :D
"smoothstep" interpolation is preferred over linear most times for Perlin.
I really like his watch
What about how image blurring works? Like when one uses Gaussian Blur in something like Photoshop or Paint.NET
British Toby McGuire sure knows a lot about computers.
Love ow you explain IT. Great stuff!
I'd love to see an explanation of the fractal scaling process that was used in the 90s. Lizard Tech owned the technology at one point. It was used to take early digicam images to post sizes or larger.
I'm curious as to why you used 50 fps instead of 60. Is that still the standard in the UK?
+bob32qwerty yep, see the linked 'fps' video for reference :) >Sean
Its the standard in europe, since we use 50hz electricity instead of 60hz. With digital stuff it doest matter as much as analog stuff from the past, but it somehow stuck around.
Computer monitors too?
That seems like it'd be detrimental to any lower-wage competitive gamers when everyone else has monitors that provide ~17% more information.
Though I guess you guys would just buy them from Amazon anyways generally
ty
I'd also guess it doesn't matter like at all. Ten extra hertz wont tell you much more informations
I was hoping for a follow-up that explains that in common computer images pixel values (0-255) are NOT intensity values, and should not be interpolated directly. They are gamma-encoded, and should first be translated to linear intensity values, interpolated, then converted back. For example, middle point between 0 and 255 is 187, not 128 (or 127.5)
I think he knows that but he isn't talking about it because it doesn't help explain the topic of this video.
Did not know that. Thanks for that info.
how do you scale down? I didn't understood how is it the dual to the problem
Can't wait to see he video on the other interpolation methods
this is a bit too simple for me since i already knew this but i am excited for the bicubic interpolation video. i'll definitely watch that one
Ha - finally someone could explain that to me so that I understand it. Thank you!
Have you done a video on Photoshop bending (screen, multiply, etc) - can you think of doing one?
I like the way you explain it.
The first book at the back is Elements of Statistical learning. I wish someone makes lectures for that.
I'm honestly a fan of 50fps 60fps sometimes sickens me if i watch it too long but 50 is a nice balance between smooth and calm good job england
Blind test or it didn't happen ;)
Crazy Luigi Are you saying the bandwidth on RUclips doesn't scale accordingly for 60fps?
Penny Lane Only when it has that option altogether.
Most computer monitors refresh at 60 Hz. On those, 50 fps causes a lot of judder, especially relative to the smoothness it has otherwise. As far as I'm concerned 24/25/50 are basically a relic of traditional filmmaking and PAL television. It's a little sad that many professionals take this practice to a context where it doesn't belong.
Penny Lane The power grid hasn't been used as a reference in forever. It might help if you want to synchronize exposures to those types of lights. In that case you'd better make sure there the rate is actually synchronized and you have a global shutter.
A variable refresh rate gets slightly hairy. Monitors need predictable timing to function optimally. Things like overdrive can only work properly when you know the timing of the next refresh. If the refresh comes sooner than expected, you could have overdriven harder. If it comes later, you end up overshooting. Tech like G-Sync isn't perfect because it has to deal with this.
It's also more difficult to get consistency with multiple screen updates. Suppose there are two updates within a millisecond of each other. If the first one triggers a refresh, the next one suffers an unpredictable and inconsistent delay. You pretty much need exclusive/dominant access to reap the benefits.
I'm sure it will eventually become commonplace. Just know that it isn't as simple as it may seem.
thumbnail is just legendary
This guy is the fastest thinking person I have ever heard.
Fun fact: the thumbnail for this video was actually a rejected cover design for I Have No Mouth And I Must Scream.
How about an algorithm that uses linear interpolation if the 2 original pixels have a similar color, and nearest neighbour if the colors are very different? So gradients would stay nice, while edges would be preserved.
He's so good at explaining, thanks!
6:12 Why just getting the interpolation in 1 direction, will it be better if you look sideways too?
This helped me understand a Computer vision concept. Thanks!
Bro can u explain this concept? I can't understand
I'd like to see a video on DCCI (Directional Cubic Convolution Interpolation). It's a great but very expensive algorithm, which is "easy in principle but hard to actually build efficiently".
ahh, so close to covering bilinear vs trilinear filtering that is the choice almost every video game offers yet I never understood why.
My english is not very well but for me it is easier to understand you than my Professor Thanks :D
im from germany
That cliffhanger ...
Very nicely explained
I have often rotated digital photo's of LP covers to get nicely aligned pix. Now I think it would be better to leave them sloppily aligned, but more digitally accurate.
How do we calculate the weights. example: 1/3 and 2/3
Why didn't you show the results of the methods for comparison?
This guy is my favourite
Explain this formula of bilinear
For f(x,y)=ax+by+cxy+d
I wonder how would it in nearest neighbor if the new pixel is exactly in the middle. Basically happens when scaling ratio is an even number.
the way he explained the process, would it matter if i scaled a picture vs turning it by 90°, then scaling then turning it back? would a merging of the two follow a better result?
edit: yes paint.net has some difference between the turned bilinear scaled images
How about fractal based image compression? I think it'd be quite useful in the future.
how so?
+Integral Cube It means that you can enlarge images more easily with less apparent loss in quality. Only thing is you have loss quality as it's more of an approximation than anything of the detail that was there.
Do you mean fractal based scaling, rather than (file) compression? Since most real images vary smoothly at small scales as far as the human eye can tell, I think smooth curves make more sense.
Why did that one image show 133.32 rather than 133.33?
When Mike is in the thumbnail, there's no scrolling further.