Another option would be to resize the images using css and get a background job to resize just those images so next time they will be the right size. That way you do it in chunks.
Great video. I like how you walked us through every step as you're learning all this for yourself. Very entertaining, as well! The color of your slides in a few cases could have been better, though, as was previously mentioned.
A better solution would have been to dynamically resize the image as they are requested. That way you only resize the images that users are actually requesting. Once the image has been resized, store that on S3 and serve that going forward. Still an informative presentation, thank you.
there is one problem with that - it takes 2s to do that per image, right? So, even if you did that concurently, images still would all appear 2s after the page loaded
@@Qrzychu92 Even if that were the case, that would happen once. To avoid this, you could show a thumbnail in its place until the resized image is ready. That is if you have a thumbnail(which you should) to show to begin with, if not, we are back to your point which is valid.
@@ThePhan7em yes, once for every image :) if you were unlucky, you could be the user that triggers it on every image you see. It all depends whether this is acceptable in the app context. In this case, where users rely havily on images in decision making, I think parsing all images upfront wa s good decision.
@@Qrzychu92 I dont agree tbh. You may end up paying for storing images that may only be seen once, then storing it at cost. Have a look at imgix.com. You can roll your own AWS Lambda or the likes relative ease.
@@ThePhan7em then why even bother with scaling the images on upload when you can scale them on demand? It's possible that storage is cheaper than running AWS lambda. Like I said, it all depends. Imagine instagram scaling images when poeple first see them. Would everyone notice? Probably not.
Probably, but you'd have to wrap each ImageMagick invocation in calls to something like s3cmd get/put, and like he said in the talk, the overhead of starting and setting up s3cmd processes each time makes it slow.
he could have put the S3 put calls from threads, that would dramatically parallelize uploading of the resized images. Resizing can be done by multiprocessing to resolve the GIL issue. I.e. ruby (or python or any other language x) would do the job just as fast.
Fun talk, but the story about time pressure is obviously BS. If you have to solve a trivially data parallel task asap, you don't spend 12 days learning a new language because it's designed for massive parallelism and run it on a single node for 4 days. You split the data to N batches, where N is the number of cores, and run N instances of whatever global interpreter locked language you're comfortable with and be done 12 days faster. Or split the data to 100 patches, make a shell script to kick up 100 instances in EC2, run your program on each and be done before lunch...
Mogrify is just a wrapper of command line call of ImageMagic, so the same issues here like with the first attempt to upload to S3. Bindings to magick++ library or Vix should be used instead.
I wonder if using AWS lamda would have saved, money developer time, download and upload time of images. it would probably be cheaper and much faster. Nevertheless great talk and information on Elixir :)
I guess it's not possible to use lambda on Digital Ocean(they're not on EC2 from what I heard on the video) but I agree with you, they would have saved time and money. Great video though!
Well the images are stored already in AWS S3, so there is no problem to spin of lambda to do the batch work. So there would be no need to extra download and upload time waste needed.
Before you upload your videos it might help if you view it from beginning to end to see if it's good. My point is this. You displayed some white-colored text against a bright yellow background, which makes for difficult reading.
Why were you limited to 20 cores? Couldn't you have run the job on multiple machines? 4 * 20 for example (splitting up the images into 4 groups) and had it all done in a day? (I didn't watch the whole thing. Sorry if I missed the answer)
You are right, Elixir can be deployed to different machines and be interconnected as well. But I guess as per the video the guy didn't have much time. But still a wonderful video.
Really cool, I love Elixir, but using it for a one shot to resize images it really overkill, specially if as told this was something needed ASAP. 3M records is manageable, extract it in a file, split it in N*k files, write a batch script that download the images, process them and upload them, send it to N machines, start k processes that run concurrently on each machine and wait. If you have 5 machines with 20 cores, even at 1.6s/image (probably can be improved with batch processing), it will only take 13 hours, the next day your images would be ready. Enjoy your free time to learn more about Elixir.
Arguably he might've done the same batch execution w/ C++, using a Ruby supervisor task to spawn as many small processes to accomplish the same task. Speed differential .... maybe gain another 33% - 50% in performance??
As for resizing images ... well, they'll load faster, but he could've just as well changed the CSS to scale the images instead. I think I could've figured a way to do it in a half hour but tell the boss it took 2 days ... :)
Yeah I was wondering why the images needed to be resized at all... maybe the quality of dynamically scaling them down in the browser is slightly less than you can get with ImageMagick.
This is exactly how I handled it with Ruby 1.9. In my case I was creating millions of animated 2 frame banner ads. I used mini-magik to create the frame, then a cmd shell all out to `convert' utility to animate the frames together. Each batch gets fed to a process which is Fork 'd and processes the batch sequentially. Get a big AWS instance with lots of cores handle all the processes and start spawning forks until the machine falls over, then dial back a couple. could generate millions of images in a matter of hours.
Niko L Not really, In every conference there will people from different levels of expertise. Especially in new programming language like elixer,there will many people who came to explore the basics of Elixir
good talk, please consider not using white on yellow slides. Most of those slides were very unreadable.
Whoever told you bright yellow slides with white text was acceptable is acting against your best interests.
Agreed. It was like putting hot lava stones on my eyes..
Another option would be to resize the images using css and get a background job to resize just those images so next time they will be the right size. That way you do it in chunks.
Great video. I like how you walked us through every step as you're learning all this for yourself. Very entertaining, as well! The color of your slides in a few cases could have been better, though, as was previously mentioned.
Communicated very well - thank you!
great talk. i have watched all the way. i liked the last piece especially erlang server handling other servers.
Why wasn't CSS used for resizing images?
That's what I thought from the beginning. I thought someone there would stop his talk or in comments. Only 2 people mentioned this
1) Decrease loading time (browser)
2) Save storage (some kB make a difference on 2.7m images)
CSS is the correct 1 day solution.
A better solution would have been to dynamically resize the image as they are requested. That way you only resize the images that users are actually requesting. Once the image has been resized, store that on S3 and serve that going forward. Still an informative presentation, thank you.
there is one problem with that - it takes 2s to do that per image, right? So, even if you did that concurently, images still would all appear 2s after the page loaded
@@Qrzychu92 Even if that were the case, that would happen once. To avoid this, you could show a thumbnail in its place until the resized image is ready. That is if you have a thumbnail(which you should) to show to begin with, if not, we are back to your point which is valid.
@@ThePhan7em yes, once for every image :) if you were unlucky, you could be the user that triggers it on every image you see.
It all depends whether this is acceptable in the app context. In this case, where users rely havily on images in decision making, I think parsing all images upfront wa s good decision.
@@Qrzychu92 I dont agree tbh. You may end up paying for storing images that may only be seen once, then storing it at cost. Have a look at imgix.com. You can roll your own AWS Lambda or the likes relative ease.
@@ThePhan7em then why even bother with scaling the images on upload when you can scale them on demand?
It's possible that storage is cheaper than running AWS lambda. Like I said, it all depends. Imagine instagram scaling images when poeple first see them. Would everyone notice? Probably not.
Wouldn't it be easier to simply have resorted to GNU parallel and ImageMagick?
Probably, but you'd have to wrap each ImageMagick invocation in calls to something like s3cmd get/put, and like he said in the talk, the overhead of starting and setting up s3cmd processes each time makes it slow.
he could have put the S3 put calls from threads, that would dramatically parallelize uploading of the resized images. Resizing can be done by multiprocessing to resolve the GIL issue. I.e. ruby (or python or any other language x) would do the job just as fast.
He said he wanted to try Elixir.
Fun talk, but the story about time pressure is obviously BS. If you have to solve a trivially data parallel task asap, you don't spend 12 days learning a new language because it's designed for massive parallelism and run it on a single node for 4 days.
You split the data to N batches, where N is the number of cores, and run N instances of whatever global interpreter locked language you're comfortable with and be done 12 days faster. Or split the data to 100 patches, make a shell script to kick up 100 instances in EC2, run your program on each and be done before lunch...
Gracias! Muy buena presentación.
Nice talk on Elixir! thanks
Perfect speech. Thank you
Great talk
Great, Buena presentación, sigo aprendiendo Elixir y tratando de cambiar de Ruby a Elixir :)
Mogrify is just a wrapper of command line call of ImageMagic, so the same issues here like with the first attempt to upload to S3.
Bindings to magick++ library or Vix should be used instead.
I wonder if using AWS lamda would have saved, money developer time, download and upload time of images. it would probably be cheaper and much faster. Nevertheless great talk and information on Elixir :)
I guess it's not possible to use lambda on Digital Ocean(they're not on EC2 from what I heard on the video) but I agree with you, they would have saved time and money. Great video though!
probs
Well the images are stored already in AWS S3, so there is no problem to spin of lambda to do the batch work. So there would be no need to extra download and upload time waste needed.
Awesome talk!, thanks
It's very helpful for me who new to the elixir.
Same XD
Before you upload your videos it might help if you view it from beginning to end to see if it's good.
My point is this. You displayed some white-colored text against a bright yellow background, which makes for difficult reading.
I'm sintax guy too! Nice talk, thanks for sharing.
Why were you limited to 20 cores? Couldn't you have run the job on multiple machines? 4 * 20 for example (splitting up the images into 4 groups) and had it all done in a day?
(I didn't watch the whole thing. Sorry if I missed the answer)
You are right, Elixir can be deployed to different machines and be interconnected as well. But I guess as per the video the guy didn't have much time. But still a wonderful video.
I want this color syntax :(
Great talk!
Really cool, I love Elixir, but using it for a one shot to resize images it really overkill, specially if as told this was something needed ASAP. 3M records is manageable, extract it in a file, split it in N*k files, write a batch script that download the images, process them and upload them, send it to N machines, start k processes that run concurrently on each machine and wait. If you have 5 machines with 20 cores, even at 1.6s/image (probably can be improved with batch processing), it will only take 13 hours, the next day your images would be ready. Enjoy your free time to learn more about Elixir.
Like for that Ruby-Threading troll
Flaw in the logic of using digital ocean - you don't pay for transfer costs between s3 and ec2. Do the whole thing in aws!
Did you open source your work on github?
github.com/dabit/elixir_images
Good talk.
Nice talk!
*Why doing it the easy way if we can make it complicated?*
Nixe talk, congrats!
or maybe use responsive CSS images?
Good and fun talk. :)
It is HTTPoison not HTTPotion!
maybe if he typed how to upload to s3 with erlang instead of elixir he would find something, if you are already on the otp maybe was worth a try
ah ok i c i was too fast :D
Ever heard of Sidekiq?
Christophe Verbinnen AFAIK sidekiq still uses single cpu core unless you purchased their enterprise version. Or am I missing something
Unless you use jruby or you start on sidekiq process per core. Pretty straight forward to setup.
Arguably he might've done the same batch execution w/ C++, using a Ruby supervisor task to spawn as many small processes to accomplish the same task. Speed differential .... maybe gain another 33% - 50% in performance??
As for resizing images ... well, they'll load faster, but he could've just as well changed the CSS to scale the images instead. I think I could've figured a way to do it in a half hour but tell the boss it took 2 days ... :)
Yeah I was wondering why the images needed to be resized at all... maybe the quality of dynamically scaling them down in the browser is slightly less than you can get with ImageMagick.
This is exactly how I handled it with Ruby 1.9. In my case I was creating millions of animated 2 frame banner ads. I used mini-magik to create the frame, then a cmd shell all out to `convert' utility to animate the frames together. Each batch gets fed to a process which is Fork 'd and processes the batch sequentially. Get a big AWS instance with lots of cores handle all the processes and start spawning forks until the machine falls over, then dial back a couple. could generate millions of images in a matter of hours.
Now this 15:20 it is so different
I don't get why he explains elixir features at an elixir conference... you'd assume people at an elixir conference already know how to use elixir
Niko L Not really, In every conference there will people from different levels of expertise. Especially in new programming language like elixer,there will many people who came to explore the basics of Elixir
Latin America #1
Wtf
Next time you just use Cloudinary XD, nice talk by the way.
So you wanna pay 140 a month for a slow service?