![Doug Mercer](/img/default-banner.jpg)
- Видео 14
- Просмотров 746 099
Doug Mercer
США
Добавлен 15 фев 2015
I create overly edited educational computer science content.
I made my own animation library
To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/DougMercer .
You’ll also get 20% off an annual premium subscription.
-------------------------------
Sign up for 1-on-1 coaching at dougmercer.dev
-------------------------------
I made my own animation library (in Python). It's awesome.
In this video, I show five of the cool features I'm really excited to use, and five of the tough challenges that I had to solve to implement them.
This library's primary goal is to make animating code easier. It's not nearly as flexible or robust as manim, but it's already helpful (for me) in its current form.
Most animations in the video, including section screens and the thumb...
You’ll also get 20% off an annual premium subscription.
-------------------------------
Sign up for 1-on-1 coaching at dougmercer.dev
-------------------------------
I made my own animation library (in Python). It's awesome.
In this video, I show five of the cool features I'm really excited to use, and five of the tough challenges that I had to solve to implement them.
This library's primary goal is to make animating code easier. It's not nearly as flexible or robust as manim, but it's already helpful (for me) in its current form.
Most animations in the video, including section screens and the thumb...
Просмотров: 26 969
Видео
How Fast can Python Parse 1 Billion Rows of Data?
Просмотров 218 тыс.4 месяца назад
To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/DougMercer . You’ll also get 20% off an annual premium subscription. Sign up for 1-on-1 coaching at dougmercer.dev The 1 billion row challenge is a fun challenge exploring how quickly we can parse a large text file and compute some summary statistics. The coding community created some amazingly clever solution...
Stop using plain Python scripts (do this instead)
Просмотров 15 тыс.8 месяцев назад
Sign up for the totally free tier of Prefect Cloud here: prefec.tv/doug-mercer Sign up for 1-on-1 coaching at dougmercer.dev One of the most frustrating parts of the workday is doing something that you know could be automated, but just… isn’t yet. In this video, we use Prefect to schedule a Python script to run every week. After, we find that scheduling was really only the first of five problem...
Write Python code people actually want to use
Просмотров 13 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev Writing code that "works" shouldn't be your end goal. It's really just the start... In this video, we adapt a clumsy, non-Pythonic API into an easy to use, easy to understand Pythonic one. We use magic methods such as getitem , len , enter , and exit to make our objects a context manager and support the len() function and square bracket indexing. An...
Compiled Python is FAST
Просмотров 109 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev Python has a bit of a reputation fast to write, but slow to run. In this video, we focus on a simple to understand dynamic programming problem that would be terribly slow in native Python or numpy. We show that Python can achieve (and actually exceed) C level performance with the help of just-in-time and ahead-of-time compilers such as mypyc, Cython...
ChatGPT made an iPhone app for me
Просмотров 25 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev ChatGPT and I create an IOS SwiftUI app, but I'm just the idea guy. Since I have no idea how to write Swift, ChatGPT helps me write an basic fitness app with pretty interesting animations without even knowing the language. #chatgpt #ios #swift Chapters 0:00 Intro 0:17 Hello World 1:04 Buttons and Counters 2:25 Title and Success Indicator 3:26 Fixing...
Never use PowerPoint again
Просмотров 290 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev In this video, we use Marp to create a presentation from our text editor using Markdown syntax. Marp is an excellent way of quickly creating beautiful presentations with syntax highlighted code blocks, TeX typeset math, and stylized images without having to worry about a finicky WYSIWYG editor like PowerPoint. #markdown #marp #marpit I use Marp a lo...
What's new in Python 3.11?
Просмотров 1,5 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev Python 3.11 is out! In this video, I show you some of the cool features and improvements made in this release. Release notes - docs.python.org/3.11/whatsnew/3.11.html Other notes: - At the time of this video's release, MyPy doesn't support most of the new typing features. If you're eager to use them right away, consider using pyright. - pyperformanc...
Scary good voice cloning | Python Text to Speech
Просмотров 6 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev In this video, Doug is lazy and lets TorToiSe text-to-speech do his voice-over for him. Specifically, we create 11 custom voices and illustrate how they sound while teaching you how to do the same. We also discuss some of the ethical considerations of such a powerful model, and highlight the precautions taken by the TorToiSe author. Thanks (and sorr...
Best Python speech to text library (OpenAI Whisper)
Просмотров 10 тыс.Год назад
Sign up for 1-on-1 coaching at dougmercer.dev Whisper, OpenAI's new automatic speech recognition model, is *awesome*. In this video, I show you how to use it and present a few interesting examples of transcribing English and translating spoken German and Japanese to English. github.com/openai/whisper #machinelearning #ai #openai #nlp #python Chapters 00:00 Introduction 00:22 Installing Whisper ...
The Magic that Makes Python Tick
Просмотров 2,8 тыс.2 года назад
Sign up for 1-on-1 coaching at dougmercer.dev In this video, we implement Python's range object in pure Python. We use this as a case study to explore Python's data model and learn how to create rich objects that interact naturally with Python's built-in operators and functions. #python #objectorientedprogramming Chapters 0:00 Introduction 1:02 What does range() do? 2:17 How will we test our im...
Your Python code is almost entirely untested.
Просмотров 2 тыс.2 года назад
Sign up for 1-on-1 coaching at dougmercer.dev Property-based testing is an alternative to example-based testing where tests verify that properties of the output are satisfied given randomly generated input. This randomized approach allows us to more thoroughly test against a broad range of input and can detect hard-to-find bugs better than example-based testing. In this video, we discuss the mo...
Would've been interesting to compare the software engineering approaches or alternative solutions to the problems presented. Even this being a sponsored video and the focus being presenting how the tool can help you, I think it's biggest selling points are A) the centralization of many helpful features and B) sensible and very useful default behaviors. Here is my very incomplete list of how one may tackle these issues outside of prefect: - Scheduling: cron jobs or Windows services - Error Handling: now that's just software engineering 🫠; slap a @retry from tenacity or write one yourself; or alternatively, decouple what you want to retry behind an interface and use a Decorator design pattern to add the behavior dynamically. - Monitoring: collecting good telemetry is certainly a monumental task, but it can be done. Again, decoupling core behaviors, logging, and other debugging information is crucial for good software maintainability. - Parallelization: threading, multiprocessing, or asynchronous programming can help. About scaling and taking advantage of computer hardware, I'm out of my depth. - Data Persistence: simplest possible solution: add a timestamp to the generated file, create a job to occasionally delete (or archive) old results. Now here are the advantages I see for using prefect (besides convenience): - Scheduling: platform independence, configurability, all done in-Python; a hand-rolled solution can also be hard to verify/test, so having a reliable library really helps bring confidence in your solution up. - Error Handling: configurability on how to act depending on errors. I also assume the behavior integrates seamlessly with the web UI, so you can monitor, for instance, how many retries it took to get something from the web. Props to them if they also have rate-limiting behavior built-in, that's a hassle to code on your own. - Monitoring: very sensible defaults that log the start, end, and execution time of the tasks. The live monitoring is also mind-blowing, I have no idea how I would even begin to write that on my own. That being said, I'm not sure of its utility in day-to-day life. Some scripts tend to be in the nature of "write, get to work, schedule, then forget about them"; it seems particularly useful for debugging, but I'm a sucker for TDD and the need to debug rarely comes up. The greatest utility I can think of is monitoring in case your service depends on unreliable third-party endpoints, or if you're dealing with multiple services that need to coordinate among themselves. - Parallelization: the parallelization part doesn't seem to have anything mind-blowing. But using the hardware optimally is certainly a challenge, and prefect gets my full respect for doing that for you. It also adds delayed rewards in the form of future optimizations that your code will benefit from, merely by using prefect's service. Two thumbs up. Even three, perhaps. - Data Persistence: Not much praise from me here. I'm sure the service is fantastic, but I'd need to see some more use cases to determine if it's worthwhile or not. I suppose it's an added bonus if you're using prefect for other benefits. In conclusion, prefect seems like an excellent tool. It's an aggregation of a very cohesive set of features that every script-er desires for their day-to-day work. I stand by the generic software engineering advice of weighing the pros and cons of writing the facilities you actually need yourself. If you adopt the tool, and want some liberty in case the project evolves in a direction you couldn't predict, my advice is to do some good design: separate your implementations from prefect-specific stuff, wrap its services around interfaces you can control, and keep the library at an arm's length. Exceptional video, very well-structured. Please don't take anything I'm saying as adversarial, I just love talking about software engineering. Thank you for doing such a great job of highlighting the tool's strengths. I believe a lot of people will benefit from incorporating it into their workflows.
Bro said “importing a lib is better than writing java code” then proceeded to write most of his logic in a SQL string 🤦♂️
... did you look at the Java solution? I would absolutely prefer to write the couple line SQL statement than the 400 line Java solution, even if it's 6x slower
I really like this place. Everyone in the comment area is talented and speaks nicely.
The whole trick about performant python code is calling as little native python code as possible.
This is amazing. Do open source this soon!
Working on it! But got side tracked on a crazy refactor that makes the library even more useful
Congratulations on overcoming your disability!
Have you guys seen "the One Billion Row Challange" ? they end up with only a few seconds!, using Java-GraalVM and some optimization.
good video
Thanks!
Great channel
Thanks=]
why use python?
Generally? I enjoy writing it. It has a great ecosystem, and is a high level, multi-paradigm language that is usually very fast to write something good enough. This video is definitely showing something that it's not ideal at. But it still did OK with the help of PyPy
First time watching a video from you and really loved the explanation and animation! Keep going 🔥
Thanks! Will do =] recently I've been working on making my own animation library so it's easier for me to keep making these videos. More to come soon!
why would you not just use jupyter notebooks at this point though
thanks, I enjoyed watching. Unfortunately, Python is the only programming language that I have mastered. The question that is always in my mind is that all these nice tools and features in OOP provided by Python seem to also be available in many other languages, so what quality or features in Python enables one to write a so-called 'pythonic' style code that has superiority in terms of readability???
That is a good question. I think answering it would be pretty wishy-washy and philosophical, though... Generally speaking "I like enough of the high level language decisions of Python that I generally feel productive and enjoy writing idiomatic Python". I can't say the same for Java or C++. More practical answer-- why not try some other, totally different languages and get a vibe of what they are like? Every time I've learned a new language or new programming paradigm, it kind of gave me a new way of thinking that benefitted me generally Consider Go, Zig, Rust, or if you want functional play around with Ocaml or Haskell. Bit of a cop out answer, but ¯\_(ツ)_/¯
Love the production... johnny Harris typa themes
Thanks so much! ❤️ Johnny Harris, so that compliment means a lot =]
The only problem the challenge rules forbid to use third party libraries 😢
Yeah =[ PyPy is my closest to valid submission (just built in Python language features)
cool! *_now do it in cuda_*
This person did Dask + cuDF github.com/gunnarmorling/1brc/discussions/487
Meanwhile nodejs is like what billion rows? The one I done yesterday?
github.com/gunnarmorling/1brc/discussions/256 Hmm?
I cry for the decade I lost becoming a master at pandas only for people to bandwagon on a new technology that's basically a knockoff and not even better in my opinion and forget about pandas completely. Guess it's time to learn polars. Beyond frustrating
I still use pandas! It's just not the right tool for the job here
Thank you Doug, very cool :)
No prob! Thanks for watching =]
nice tutor in a story telling way. I enjoyed it thank you
the worst crime here was making them be seperated by semicolons
Agreed. (Of course it's from the twisted mind of a Java dev...)
Such a cool video, why did I only find out about this now?
¯\_(ツ)_/¯ welcome!
I haven't watched the video but i can say as fast as c++ is.
I wish... (Kind of close, though)
the fastest way to loop in python is to not loop in python
Mcoding mentioned
Mojo could be interesting to be included
I plan on doing a video on mojo eventually. Stay tuned!
oooo yea
I'm calling it by the end you'll just be using python as a thin wrapper for a C implementation
Hah! I did consider it...
@@dougmercer The moment you get numpy involved thats basically whats happening anyway :p
I enjoyed every single bit of this video and seeing how it uses techniques used by others in different languages. Although Java is still way ahead, this makes me super happy. Thanks Doug! Subscribe!!!
Thanks =]
14:10 🤢🤮👎
😂
This was super entertaining man, well done
Thanks =]. I'm glad people are still enjoying this video (even when generative AI has moved so fast since this video came out)
Programming is just applied mathematics. So another version manim.
you removing the global variables and making it less readable, is that a way to improve performance or you really just don't like more readable code? Weird anyways wether against readability or even worse, it improving performance
I thought it improved performance cause I was changing several things at once. In reality, it doesn't change performance, and I regret changing them
So interested. Would totally use it. Would totally want to contribute. Cant wait to see it open sourced!
A lot of python libraries, especially GPU libraries, are actually executing linked C/C++ code. A good example most people should know is "torch," which you access via Python but which is actually calling C++/CUDA code. Obviously, a SIMD operation is going to beat an interpreter performing a bunch of string conversions, even if it's Java with JIT. The benchmark has to be clear about language features and the constraints of which operations or features are being tested, as well as the testing methodology and how to achieve the same (stated) performance of the bytecode assembler (JVM) using Assembler. Rather than making it a pissing contest, it would be more laudable to demonstrate circumstances where normal people can unleash experienced performance.
Yup, understood. I've done another video on Cython, Numba, mypyc, and Taichi. Feel free to try implementing this in Torch... would love to see it. Also, this is just for fun... not a "pissing contest"
Great video! If I could make a suggestion though, the background music is a little too loud. It was hard to follow some of what you were saying because of it (like for the cython stuff for example). Overall awesome video though and learned a lot!
Thanks! And def agree- I've reduced music levels in later videos. Thanks again for the feedback and comment =]
I am trying to do something similar with C++ for astronomy animations, but I am really struggling with exactly the aspects you cover here. You have given me some additional things to consider, thank you very much! On a side note, this video is very well structured and presented; a real pleasure to watch. I've subscribed for more :)
That sounds awesome. good luck on the project! Let me know if you eventually open source it-- would love to check it out. There's definitely a lot that can go wrong with animation 😂. I'm in the middle of refactoring a core part of my library. Unit tests are a must, and do your best to clearly define any assumptions (e.g., I apply transformations in the order that their applied, not the order of the starting frame).
TIL that [some in] the Python community call[s] "__" "dunder"! Like, I get it-double underscore-but I'd never heard it despite 22 years of casual Python programing. Anyway, you should open source your library, and you shouldn't wait to clean it up first. That's what sub-1.0 semvers are for!
Hah, yeah, I remember when I first heard dunder-- I was like, huh? 😂 I think before that I heard them referred to as "magic methods". Not sure which is better ¯\_(ツ)_/¯
(12@3)))
you make high quality video.
Thanks =]
Wondering about the efficiency of using the sim(), min(), and max() functions over chunks of the array/file rather than with only two operands.
Give it a shot! You can clone the repo in the description
just tell me why.
Cause it's fun 🙃
Ain't nothin' but a heartache
I personally wouldn't recommend polars just yet. its great and very powerful but its python bindings are still highly under construction.
Fair point
I'm learning Zig and decided to practice this benchmark and see how fast it could go. If we use the same optimization as the last taichi variant (pre-allocating all necessary memory), it takes 1150ms. If we leave allocation inside (creating and zeroing the matrix, cleaning the memory after we got the result) its about 1800ms (i7-14700k, Ubuntu).
I wanna learn Zig this year...
Instead of manim, have you ever seen "motion canvas"?
Shocker
BRO SUMMOMING SALT
in this video; ruclips.net/video/VioxsWYzoJk/видео.html python just counts to 1 billion 1 minute 52 seconds. but how can you parse that 13 gb's of data in less than just counting?
PyPy interpreter (compiles to C) + multiprocessing (so multiple cores doing work instead of one)
14:11 instead of raising a TypeError, you should return NotImplemented
Doh, yeah -- that's the preferred way of doing it. Not sure what I was thinking (this feature was added in a bit of a rush)
Ekspecially
btw the lofree flow feels so much nicer than the nuphy
Ah interesting-- it does seem nice. I recently went off the keyboard deep end and got a ZSA Voyager. I really like it (but miss wireless operation =[)
I really love these kind of tools, so happy to see more being created. This really reminds me of Aarthficial's MotionCanvas typescript library