Видео

Would've been interesting to compare the software engineering approaches or alternative solutions to the problems presented. Even this being a sponsored video and the focus being presenting how the tool can help you, I think it's biggest selling points are A) the centralization of many helpful features and B) sensible and very useful default behaviors. Here is my very incomplete list of how one may tackle these issues outside of prefect: - Scheduling: cron jobs or Windows services - Error Handling: now that's just software engineering 🫠; slap a @retry from tenacity or write one yourself; or alternatively, decouple what you want to retry behind an interface and use a Decorator design pattern to add the behavior dynamically. - Monitoring: collecting good telemetry is certainly a monumental task, but it can be done. Again, decoupling core behaviors, logging, and other debugging information is crucial for good software maintainability. - Parallelization: threading, multiprocessing, or asynchronous programming can help. About scaling and taking advantage of computer hardware, I'm out of my depth. - Data Persistence: simplest possible solution: add a timestamp to the generated file, create a job to occasionally delete (or archive) old results. Now here are the advantages I see for using prefect (besides convenience): - Scheduling: platform independence, configurability, all done in-Python; a hand-rolled solution can also be hard to verify/test, so having a reliable library really helps bring confidence in your solution up. - Error Handling: configurability on how to act depending on errors. I also assume the behavior integrates seamlessly with the web UI, so you can monitor, for instance, how many retries it took to get something from the web. Props to them if they also have rate-limiting behavior built-in, that's a hassle to code on your own. - Monitoring: very sensible defaults that log the start, end, and execution time of the tasks. The live monitoring is also mind-blowing, I have no idea how I would even begin to write that on my own. That being said, I'm not sure of its utility in day-to-day life. Some scripts tend to be in the nature of "write, get to work, schedule, then forget about them"; it seems particularly useful for debugging, but I'm a sucker for TDD and the need to debug rarely comes up. The greatest utility I can think of is monitoring in case your service depends on unreliable third-party endpoints, or if you're dealing with multiple services that need to coordinate among themselves. - Parallelization: the parallelization part doesn't seem to have anything mind-blowing. But using the hardware optimally is certainly a challenge, and prefect gets my full respect for doing that for you. It also adds delayed rewards in the form of future optimizations that your code will benefit from, merely by using prefect's service. Two thumbs up. Even three, perhaps. - Data Persistence: Not much praise from me here. I'm sure the service is fantastic, but I'd need to see some more use cases to determine if it's worthwhile or not. I suppose it's an added bonus if you're using prefect for other benefits. In conclusion, prefect seems like an excellent tool. It's an aggregation of a very cohesive set of features that every script-er desires for their day-to-day work. I stand by the generic software engineering advice of weighing the pros and cons of writing the facilities you actually need yourself. If you adopt the tool, and want some liberty in case the project evolves in a direction you couldn't predict, my advice is to do some good design: separate your implementations from prefect-specific stuff, wrap its services around interfaces you can control, and keep the library at an arm's length. Exceptional video, very well-structured. Please don't take anything I'm saying as adversarial, I just love talking about software engineering. Thank you for doing such a great job of highlighting the tool's strengths. I believe a lot of people will benefit from incorporating it into their workflows.

Bro said “importing a lib is better than writing java code” then proceeded to write most of his logic in a SQL string 🤦‍♂️

I really like this place. Everyone in the comment area is talented and speaks nicely.

The whole trick about performant python code is calling as little native python code as possible.

This is amazing. Do open source this soon!

Congratulations on overcoming your disability!

Have you guys seen "the One Billion Row Challange" ? they end up with only a few seconds!, using Java-GraalVM and some optimization.

good video

Great channel

why use python?

First time watching a video from you and really loved the explanation and animation! Keep going 🔥

why would you not just use jupyter notebooks at this point though

thanks, I enjoyed watching. Unfortunately, Python is the only programming language that I have mastered. The question that is always in my mind is that all these nice tools and features in OOP provided by Python seem to also be available in many other languages, so what quality or features in Python enables one to write a so-called 'pythonic' style code that has superiority in terms of readability???

Love the production... johnny Harris typa themes

The only problem the challenge rules forbid to use third party libraries 😢

cool! *_now do it in cuda_*

Meanwhile nodejs is like what billion rows? The one I done yesterday?

I cry for the decade I lost becoming a master at pandas only for people to bandwagon on a new technology that's basically a knockoff and not even better in my opinion and forget about pandas completely. Guess it's time to learn polars. Beyond frustrating

Thank you Doug, very cool :)

nice tutor in a story telling way. I enjoyed it thank you

the worst crime here was making them be seperated by semicolons

Such a cool video, why did I only find out about this now?

I haven't watched the video but i can say as fast as c++ is.

the fastest way to loop in python is to not loop in python

Mojo could be interesting to be included

I'm calling it by the end you'll just be using python as a thin wrapper for a C implementation

I enjoyed every single bit of this video and seeing how it uses techniques used by others in different languages. Although Java is still way ahead, this makes me super happy. Thanks Doug! Subscribe!!!

14:10 🤢🤮👎

This was super entertaining man, well done

Programming is just applied mathematics. So another version manim.

you removing the global variables and making it less readable, is that a way to improve performance or you really just don't like more readable code? Weird anyways wether against readability or even worse, it improving performance

So interested. Would totally use it. Would totally want to contribute. Cant wait to see it open sourced!

A lot of python libraries, especially GPU libraries, are actually executing linked C/C++ code. A good example most people should know is "torch," which you access via Python but which is actually calling C++/CUDA code. Obviously, a SIMD operation is going to beat an interpreter performing a bunch of string conversions, even if it's Java with JIT. The benchmark has to be clear about language features and the constraints of which operations or features are being tested, as well as the testing methodology and how to achieve the same (stated) performance of the bytecode assembler (JVM) using Assembler. Rather than making it a pissing contest, it would be more laudable to demonstrate circumstances where normal people can unleash experienced performance.

Great video! If I could make a suggestion though, the background music is a little too loud. It was hard to follow some of what you were saying because of it (like for the cython stuff for example). Overall awesome video though and learned a lot!

I am trying to do something similar with C++ for astronomy animations, but I am really struggling with exactly the aspects you cover here. You have given me some additional things to consider, thank you very much! On a side note, this video is very well structured and presented; a real pleasure to watch. I've subscribed for more :)

TIL that [some in] the Python community call[s] "__" "dunder"! Like, I get it-double underscore-but I'd never heard it despite 22 years of casual Python programing. Anyway, you should open source your library, and you shouldn't wait to clean it up first. That's what sub-1.0 semvers are for!

(12@3)))

you make high quality video.

Wondering about the efficiency of using the sim(), min(), and max() functions over chunks of the array/file rather than with only two operands.

just tell me why.

I personally wouldn't recommend polars just yet. its great and very powerful but its python bindings are still highly under construction.

I'm learning Zig and decided to practice this benchmark and see how fast it could go. If we use the same optimization as the last taichi variant (pre-allocating all necessary memory), it takes 1150ms. If we leave allocation inside (creating and zeroing the matrix, cleaning the memory after we got the result) its about 1800ms (i7-14700k, Ubuntu).

Instead of manim, have you ever seen "motion canvas"?

Shocker

BRO SUMMOMING SALT

in this video; ruclips.net/video/VioxsWYzoJk/видео.html python just counts to 1 billion 1 minute 52 seconds. but how can you parse that 13 gb's of data in less than just counting?

14:11 instead of raising a TypeError, you should return NotImplemented

Ekspecially

btw the lofree flow feels so much nicer than the nuphy

I really love these kind of tools, so happy to see more being created. This really reminds me of Aarthficial's MotionCanvas typescript library