How SQLite and Python are getting better
HTML-код
- Опубликовано: 26 дек 2022
- Comparing performance of Python and SQLite and what 10 years can do.
*Correction: thanks to the awesome commenters for pointing this out. In the video at 5:42 there are zero rows selected. This was because I was selecting the value 'YES' instead of 'Yes'. When all rows are selected, the time is 3.4 seconds.
▶️ Are you using the SLOWER Python? - • Are you using the SLOW...
▶️ M1 Max vs Intel Core i9 Python Race | XPS 15 2022 - • M1 Max vs Intel Core i...
▶️ M1 Ultra vs Intel Core i9 Python Test DESKTOPs - • M1 Ultra vs Intel Core...
▶️ M1 MacBook vs Intel i9 MacBook Python Test - • M1 MacBook vs Intel i9...
▶️ C++ Sorting 1000000 Items - • M1 MacBook Air vs Inte...
▶️ Open CV - • OpenCV Build SLOW? Mak...
Sebastian Raschka's blog post: sebastianraschka.com/Articles...
#python #sqlite #programming
💻NativeScript training courses - nativescripting.com
(Take 15% off any premium NativeScript course by using the coupon code YT2020)
👕👚iScriptNative Gear - nuvio.us/isn
- - - - - - - - -
❤️ SUBSCRIBE TO MY RUclips CHANNEL 📺
Click here to subscribe: / alexanderziskind
- - - - - - - - -
🏫 FREE COURSES
NativeScript Core Getting Started Guide (Free Course) - nativescripting.com/course/na...
NativeScript with Angular Getting Started Guide (Free Course) - nativescripting.com/course/na...
Upgrading Cordova Applications to NativeScript (Free Course) - nativescripting.com/course/up...
- - - - - - - - -
📱LET'S CONNECT ON SOCIAL MEDIA
ALEX ON TWITTER: / digitalix
NATIVESCRIPTING ON TWITTER: / nativescripting 
Наука
I think mostly it's because of powerful m1's SSD. 10y ago, he was probably using hdd.
was thinking the same, the hardware considerations matter too
SSD's do matter
*Correction: thanks to the awesome commenters for pointing this out. In the video at 5:42 there are zero rows selected. This was because I was selecting the value 'YES' instead of 'Yes'. When all rows are selected, the time is 3.4 seconds, which means SQLite got a bit slower. But to get the real deal numbers, I would have to use my M1 Max MacBook to run the 2013 versions of Python and SQLite.
Sqlite will keep the data in memory whenever possible, which in this case is probably always since the entire dataset is 75MB. In this case, the query will measure the RAM advancements where as, just reading the file from disk or populating the DB is mainly I/O bound. The conclusion is that disk has become considerably faster. Hard to say how much python and SQLite improved since it is not running on comparable hardware.
I was curious that the first test was fast because it was the same line 6 million times. Would the speed change if the text was different? Is there any optimization due to the content of the file being the same?
Power consumption should also be a consideration for Python in large application.
I guess most of the time it's neglectable compared to engineering time and thus cost
As someone commented, it's mostly because of nvme SSDs and hardware upgrades too.
5:42 -> 0 lines read 😅 I think that explains the speed bump.
Could be interesting to take those functions and compare python 3.3 (most recent stable version of python in 2013) on M1 max with completely rewritten code to take advantage of python 3.11 code on M1 max. The hardware gap here makes the comparison kind of not relevant in your tests.
Thanks! I've pinned a comment addressing this.
Not good at math but also not good at realizing that the difference is all in the SSD vs. a mechanical HDD. Hardly a surprising result.
04:16 Why it doesn't matter what the line is? Especially when it comes to writing, wouldn't the length of the strings be crucial?
length of strings is crucial very obviously, and there are a 100 other factors, this video is shit-on-steroids.
Really appreciate you starting the video with "Hey Google" That certainly didn't cause any issues in my listening environment :P
SQLite is a really good database, I've been using it in production for the last 10 years as my website's database for my company. It flies on an Optane drive if you need to squeak out some extra performance.
Im new to all of this. How do set up sqlite to be used in production when deploying your code to be hosted online? Im having trouble understanding this.
that's obviously ssd vs hdd.... tried something similar size for reading before and it read it in 10s of seconds on a hdd in a dual socket xeon server from 2019... got upgraded to ssd and the time went way down....
Your read of the SQLite file read 0 lines of the 6 million lines. That is why it was so fast. It didn't read any of the lines.
Thanks! I've pinned a comment addressing this.
Why don't you recreate the software stack (as of the test was conducted, didn't look into it, but there could be some old docker images :) ), test both on the same hardware, which eliminates the hardware difference :)
Hi Alex, what's your vscode theme?
its just one of the default dark ones
What I see is a huge improvement in text file reading.
After years of .NET coding, I must have said that python read text file (and CSV) very fast.
Monkeypatch is used in the Python world. At least, it's used when talking about unit tests.
if anyone is more intrested in speed ups in python can see this video > Why Python 3.11 is so fast | Guido van Rossum and Lex Fridman <
(*didnt provide link as youtube might censor my comment)
Thanks! Here is the link (I'm immune to my own channel's filter 😀) ruclips.net/video/TLhRuZ9cJWc/видео.html
Hey this might sound weird and out of place but I just noticed that you have really beautiful eyes!!!
Thanks!
Check pypy3 instead of python3. I tested Mandelbrot on ThinkPad X13s, Debian WSL 2: python3 62sec, pypy3 9sec
Nice! Might need to start adding that to my Python related tests in the future.
We also call it monkey patching in Python. 🐒🐍
oh its older than java
Eheheheh, my apple beats your old pc, ehehehehe
👆 If Alex triggered your google home with this and if you think he knew exactly what he was doing there…