- Видео 67
- Просмотров 125 574
Data Slinger
США
Добавлен 15 окт 2021
Pulling Data from the Chicago Data Portal with Python
The Chicago Data Portal has lots of great data, BUT their API is HORRIBLE. This walkthrough shows how to pull data from the Chicago Data Portal with Python so hopefully you don't have to grind through all the odd "ticks" and "hiccups" of using it.
NOTE: I was able to resolve the error pulling the 311 data around timestamp 23:00 by changing the limit to 4000000.
package versons:
pyarrow==14.0.2
pandas==1.3.5
00:00 Intro
01:00 Signing Up
03:00 Finding API and Docs
04:20 Python API Code
07:00 Tirade against Sodapy/setup
09:00 Setting up an App Token
12:45 Using a Where Clause
14:40 Dotenv Usage
16:00 Watch me faceplant on an F-String
19:20 Using a different dataset
22:00 Changing Timeout Parameter
24:30 S...
NOTE: I was able to resolve the error pulling the 311 data around timestamp 23:00 by changing the limit to 4000000.
package versons:
pyarrow==14.0.2
pandas==1.3.5
00:00 Intro
01:00 Signing Up
03:00 Finding API and Docs
04:20 Python API Code
07:00 Tirade against Sodapy/setup
09:00 Setting up an App Token
12:45 Using a Where Clause
14:40 Dotenv Usage
16:00 Watch me faceplant on an F-String
19:20 Using a different dataset
22:00 Changing Timeout Parameter
24:30 S...
Просмотров: 74
Видео
Removing a .env File from Github with BFG Repo-Cleaner
Просмотров 99Месяц назад
BFG Repo Cleaner Site: rtyley.github.io/bfg-repo-cleaner/ Commands used: wget repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar bfg delete-files .env git reflog expire expire=now all git gc prune=now aggressive git push force Alias entry for .bashrc: alias bfg='java -jar /usr/local/bin/bfg.jar' 00:00 Intro 00:30 What to do if you accidentally commit secrets 01:00 Demo Setup 04:35 Push...
How to fix error: externally-managed-environment in Python 3.12 the Wrong Way
Просмотров 1,4 тыс.Месяц назад
On Linux: Add the following to /etc/pip.config [global] break-system-packages = true On Windows: C:\ProgramData\pip\pip.ini [global] break-system-packages = true Reference Posts: stackoverflow.com/questions/38869231/python-cant-find-the-file-pip-conf stackoverflow.com/questions/75608323/how-do-i-solve-error-externally-managed-environment-every-time-i-use-pip-3 00:00 Introduction 01:40 Check Pyt...
Upgrading Linux Mint from Virginia to Wilma
Просмотров 183Месяц назад
This process may take 1:30 - 2 Hours so plan accordingly. I did this demo on a VirtualBox VM. I believe the "hiccups" at the end were do to this. 00:00 Intro 2:00 Install Mint Upgrade 2:45 Check Current Version 3:15 Start Upgrade 3:50 Take a snapshot 5:45 Agreeing to Change Repos (20 minute wait after this) 6:15 Agree to Changes (About 1 hour wait after this) 7:25 Agree to Downgrade (4-5 minute...
PlaylistGuide-LLM Walkthrough
Просмотров 1212 месяца назад
This is a quick walkthrough and overview of my final project for LLM-Zoomcamp 2024
Two Ways to Add SSH Keys to your VM on GCP
Просмотров 1254 месяца назад
00:00 Generating SSH Key 05:00 Adding SSH Key to VM Metadata 06:44 Creating a config File 09:53 Adding SSH Key to Project Metadata 11:47 Using SSH with VSCode 20:00 Cleanup
Setting up a Managed PostgresSQL Instance on GCP
Просмотров 564 месяца назад
00:00 Setting up Postgres 03:34 Configuration 06:22 VPC Peering 11:40 Setting up VM 17:20 Inserting Data 25:36 Cleanup
Use user-data to deploy Docker and Docker Compose in a VM on GCP with Terraform
Просмотров 1375 месяцев назад
I made a previous video showing how to deploy a Virtual Machine on GCP and install Docker and Docker Compose with Terraform using a startup script. While this will work it is not the cleanest way to accomplish it. This video will show how using user-data will execute the code on creation and will not run the code on subsequent reboots. Thank you to Dmitro Aleksandrov and Joseph Machado - www.st...
Creating a VM on GCP with Docker and Docker Compose Installed with Terraform
Просмотров 2327 месяцев назад
In this walkthrough we will be setting up docker and docker compose on a VM in GCP with Terraform. The Terraform install script from the Github repo will work if you are on Ubuntu or Mint. You can change where the flag file is written to, but MAKE sure that it is a directory that is present at startup. Github Repo: github.com/MichaelShoemaker/DockerInstallWithTerraform
Creating a Data Flow Diagram and an Animated GIF Data Flow Diagram
Просмотров 2,2 тыс.8 месяцев назад
Yes, I realize I forgot to connect GCS to Big Query. :-) Careful when looking for images. The extension I was trying to think of was .webp. Those don't play nice. :-/ 00:00 Introduction 02:50 Draw.io 03:20 Finding Images 10:10 Adding Images to Chart 16:40 Adding Connections 25:20 Adding Text Boxes 27:27 Exporting Static Diagram 28:06 Animating Connections 29:08 Installing Peek 30:06 Recording t...
Connecting to Minio Running in Docker on Pi3 - Duct Tape and Bubble Gum
Просмотров 2918 месяцев назад
This video we walkthrough how to find :-) and run Minio Arm 32 bit version. It's a bit buried. Then I stumble through how you can "roll your own" Docker Image from the binary and use that to run a Minio Docker Container
GCP VM Instance Low Space: Expand Disk Quick Fix
Просмотров 248 месяцев назад
GCP VM Instance Low Space: Expand Disk Quick Fix
DE Zoomcamp 2024 - Homework #5 Solution by Michael Shoemaker
Просмотров 1838 месяцев назад
DE Zoomcamp 2024 - Homework #5 Solution by Michael Shoemaker
Connecting to Minio Running on Raspberry Pi 4
Просмотров 4018 месяцев назад
In the two previous videos we went through: How to setup docker and docker compose on a Raspberry Pi 4 ruclips.net/video/xla4w9Qq1hQ/видео.html How to mount an External SSD Drive to the Pi ruclips.net/video/thXjZvKdmu8/видео.html In this video I walkthrough how run Minio on the Pi, connect to it from your local PC and write to it using Python. Cert Generators Mentioned github.com/FiloSottile/mk...
Setting up a Raspberry Pi 3 and Installing Docker
Просмотров 1,3 тыс.9 месяцев назад
Walkthrough of setting up a Raspberry Pi 3 with Pi Imager and installing docker and docker compose. Ways to find your Pi on the network: Login to your router Use Nmap Use Fing - www.fing.com/products/fing-app The SD to USB converter I show: Samsung - Pro Ultimate and Reader 256GB microSDXC Memory Card
Setting up a Raspberry Pi 4 and Installing Docker
Просмотров 2,8 тыс.9 месяцев назад
Setting up a Raspberry Pi 4 and Installing Docker
DE Zoomcamp 2024 - Homework #3 Solution by Michael Shoemaker
Просмотров 7639 месяцев назад
DE Zoomcamp 2024 - Homework #3 Solution by Michael Shoemaker
Mounting an External Hard Drive in a Raspberry Pi
Просмотров 4,1 тыс.9 месяцев назад
Mounting an External Hard Drive in a Raspberry Pi
Enabling SVM in BIOS for AORUS Motherboard
Просмотров 2,9 тыс.10 месяцев назад
Enabling SVM in BIOS for AORUS Motherboard
Installing Docker and Docker Compose on an EC2 Instance
Просмотров 1,1 тыс.10 месяцев назад
Installing Docker and Docker Compose on an EC2 Instance
Upgrading Linux Mint to 21.3 Virginia from 21.2 Victoria
Просмотров 40910 месяцев назад
Upgrading Linux Mint to 21.3 Virginia from 21.2 Victoria
Fix Zoom Shortcut Not Working Linux Mint Mate
Просмотров 32811 месяцев назад
Fix Zoom Shortcut Not Working Linux Mint Mate
DE Zoomcamp 1.3.3 - Terraform Variables
Просмотров 8 тыс.11 месяцев назад
DE Zoomcamp 1.3.3 - Terraform Variables
How to Install Tesseract OCR on Windows and use it with Python
Просмотров 21 тыс.Год назад
How to Install Tesseract OCR on Windows and use it with Python
How to see your EC2 Instances Actual Memory and Hard Drive Size
Просмотров 319Год назад
How to see your EC2 Instances Actual Memory and Hard Drive Size
Connecting to AWS with AWS Toolkit from Visual Studio Code
Просмотров 5 тыс.Год назад
Connecting to AWS with AWS Toolkit from Visual Studio Code
Connecting to SQL Server with SQLAlchey
Просмотров 909Год назад
Connecting to SQL Server with SQLAlchey
thank u sir
Thanks. Great job.
ok verygood! perfect
Thanks for this video i have been searching for solution sor days hopefully it will resolve my error. will update final result here !
@@ankitpatel3668 Please update if there are any "hiccups" for you and I'll add them to the description. Hopefully save somebody some time if they experience the same. Thanks for watching!
works well nice ty
mkdir local_dev cd local_dev 5:23 5:41 5:51 6:09 7:52 9:17 10:34 10:52 12:00 12:12 12:26 12:40 13:00 14:00 14:16 Video improvement tips: Script it before. Side by side Terminal and Browser doc. Copy paste / no highlight copy. thanks
Thanks for the timestamps and feedback. Can you clarify what you mean by "Copy paste / no highlight copy"? Do you mean just ctrl + c - ctrl +v?
@@dataslinger6379 Thanks for the tutorial! That was about the partial command copied by click and drag prone to error and taking 2 minutes to figuring out the error :) Better just using the documentation copying button! Cheers!
thank you
thank you very much its help me a lot
Thank you very much sir, very helpful
Oh man, can't tell you how grateful I am. Found this gem from your slack message from the Data Talks Club zoomcamp, after I couldn't pip pgcli and not be able to do it the right way, with the virtual environment recommendation - and it was actually the only result I got when searching for the error on slack! I can't thank you enough :)
veri good work tekyou
Wow super usefull
is it needed to install docker and postgreSQL on the linux?
Just docker. You could probably run Mage on your PC without docker, but I haven't tried it that way before.
You are a God. You are literally a God.
You are FAR to kind. 🙂Just trying to save others time and hassle when I can. Thank you for watching and your kind words! Keeps me motivated.
Thanks for your video! I used to use Linux Mint, doing a fresh install with newer minor/major versions, because the upgrade would be buggy. Now I use Solus, it's rolling release and very stable. It has all the software I need, but not as large a selection as Linux Mint, since it's based on Ubuntu. Solus is independent and was made from scratch. Solus 4.6 was released yesterday 14 Oct 2024. Solus XFCE is light, fast and stable, which I would recommend for older laptops or anyone who wants a stable desktop environment with a moderate level of customisation possible. Solus also has Budgie, Gnome and KDE Plasma spins 👍 Solus has an update sync every Friday, announced and discussed weekly on their forum website. Their packages website shows Shannon which is the stable release version. Hope that helps! (Posted on Tue 15 Oct 2024 at 12:52 UTC)
Interesting. I'm always up to experiment and try new things out. Hopefully have time in the coming weeks to spin up a VM and give it a test drive. Thank you for sharing!
I forgot to explain that since Solus is rolling release, it only has to be installed once, then keep installing all updates and restarting, to run the newest version of Solus, the kernel, Firefox, LibreOffice etc Right now Solus 4.6 XFCE is a Beta but it's still very stable and usable. It has XFCE 4.18 included. XFCE 4.20 should be released on 15 Dec 2024. Once Solus developers test that version and release the update, there will be no need to follow any special upgrade instructions. Just install all updates and restart and you'll be running the newer XFCE 4.20. Should be done in about 5-10 mins max, depending on your internet speed. Solus is considered to be the most stable rolling release distro 🙂👍 (Posted on Tue 15 Oct 2024 at 13:01 UTC)
Thank you. I couldnt find it also on internet. With your video, i activated it 😎
Very NIce! Simple, straightforward, AND worked! Thank you!😎
Awesome and quick tutorial, thanks for the video!
Super helpful, thanks!
Thanks a lot dude, switched to linux not too long ago because of windows 10 crowdstrike stuff previously .and I like linux and thx so much dude I was worried I couldn't upgrade my linux kernal I mean that sheild in my system tray was really red so yeh stumbled across this video and thanks a lot dude I appreciate it man.❤
You're very welcome. Thanks for watching and glad it helped!
thank you so much for taking your time to explain each options i am grateful.
Thank you for the video. I appreciate the step by step and not editing out any slips. I learned a lot from your video, great content!
Nice, clean explanation on how to mount the external drive!
Theres another option called "SVMEnable" should I enable that too?
Thanks!
Oh wow. Thank you so much! This is actually the first money I've made from my videos. Thanks for the support.
It worked, thank you very much ☺
Thanks!
Nice introduction. Thank you!
hey Mich, can this be implemented on windows 10? Thanks
thanks
This is perfect. I just had this idea myself, and lo and behold this video shows up. I plan to set up a Spark cluster as well.
Thanks for watching and glad it helped! A while back I did a single node Spark on my Pi. Took about 20 seconds to do a count on a small dataset IIRC. 😅 Sounds like a fun project though!
Hi I had a doubt regarding the 2019 data that are in parquet form
Thanks for the video - still having a little trouble - my drives contains media i want to play on kodi, but kodi requires the filepath to be in the home directory. Can i mount the drive directly in the home directory?
That I'm not sure of. I believe you can mount it to your home directory, but I think the "more correct" way to do it for organization would be to do the mount as you normally would. Actually I think in the video I probably should have mounted it to media instead of mnt. 😕Then create a symbolic link between a directory in your home directory and the mounted drive on /media/your_drive.
@dataslinger6379 I actually did mount to media - also solved my issue, there's an option in kodi for the root directory to be a video source. Thanks again for the video, very helpful for me as someone who's new to raspberry pi os!
thank you so much very appricated
how is the price calculated? Do you pay one month for a vm or pay-as-you go after every started minute ?
Just a question - is it able to read the hard drive without any external power source required?
Not sure I understand. Are you asking if it can read the hard drive without any power to the Pi or the hard drive? The Pi needs power, but the hard drive does not have an additional power cable and draws it's power from the USB Port. I believe most External Hard Drives are power over USB these days. At least those for personal use.
Yes yes. Like for Older Pis it required an external power source to work with Hard drives. I am trying your method right now hopefully it will work!
@@GhtsGameplay Ah got it. It worked fine for me on a Pi3 and a Pi4. Not sure about other models. Let me know if you try it and it doesn't work and what model your Pi is. Then I can add that disclaimer to the description.
@@dataslinger6379 Hello, coming back here after months. Bought an external SSD cause the Hard drive kind of failed due to low power. The HDD worked fine on my laptop but doesnot show up or the light blinks when I connect to Pi.
Can you show us how to install terraform?
The install is pretty straight forward. developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli Also, plenty of other people have made videos for installing. ruclips.net/video/oURFbTZLMl0/видео.html Why re-invent the wheel? 🙂
@@dataslinger6379 I was looking for an option to use terraform instead of installing it locally. Is it possible?
@@anzhemeng8833 You could make a VM on any cloud provider and run it from there (GCP, AWS, Digitalocean etc) Think Hashicorp may have a hosting service, but not positive about that.
This is so useful! I like all the videos you have posted on RUclips. Keep up the good work man.
It would be very nice if you can post some video about bash scripts and let's say 'walk-through' some very useful docker commands. Thanks!
Great stuff mate, thank you
Thank you for watching!
Thank you ! very helpful.
thank you for sharing this, was very useful for me. ur guide is slightly outdated with regards to the upated github folder, nevertheless i managed to pull it off. Thank you !
Hello, Michael! Thanks for the amazing content! I have a few questions: 1. The Mage Ai docs recommend deploying the application using their official terraform scripts and cloud run when working with GCP. Any particular reason you used a VM? Which approach should be more realiable / cost effective (really hard to be cheaper than in your video though). 2. Does the mage built in data exporter to bigquery automatically enforces schema, according to the pandas dataframe dtypes? Or is there a way to create a schema dictionary like one can do when using the regular BigQuery Python API? Thanks a lot!!
When I made this video there wasn't really much content around Mage. That why I made it. I think that the way they use cloud run with the setup now (not super familiar with it) would come in at about the same price as just using the VM. Probably a little higher. Using just a VM is a great way to get familiar with a tool and for personal projects. Using their Terraform Setup is taking a big step towards "Production Ready", but not quite there. Also, it's awesome to read through their setup to learn about Terraform and Mage and how the different components interact. As for the Big Query question I honestly don't know. I'd jump into their Slack Channel and ask there. Tommy and team were always ninja quick with responses when I ran into issues. Hope that helps and Thanks for watching. 🙂
@@dataslinger6379 Thanks a lot for the quick response! Keep up the amazing work.
@@dataslinger6379 Thanks a lot for the response, Michael! Keep up the great work
Hello, how can i contact with you?
thank you!
Thanks. I've been trying to set up a service user via "IAM Identity Center" and I wasn't even aware of (or forgotten about) IAM as a separate thing. So "IAM Identity Center" isn't just a modern replacement of IAM then?
Honestly not too familiar with IAM Identity Center, but my understanding and if I recall correctly, you would use IAM Identity Center to connect to an existing Identity Source like Active Directory where as IAM is for kind of "one off"/personal project service accounts and users.
Amazing video, helped me a lot thanks!
Thank you for the kind words!
Hello, I was looking at your video channel. We may be helping a company that uses secure images to increase supply chain security and help cloud native development. Would you be willing to help try their software, make a video, and help show devs how to use their tools? This is not an offer, but just to start a conversation about your willingness to take on sponsorship. Please provide me with your email if you are interested. You'd have a chance to look at their technology and decide if it's the type of software that you'd be interested in covering in your channel.
thanks for the perfect explanation! Consider making a video on docker networks!
Thanks for watching! On a bit of a break now, but have something in the works about that....kind of. ;-)
I get a return code 100 when I attempt to install new mint version "Vanessa". Any idea how to fix this? thank you!
Might have to do with hardware requirements. Any clues in /var/log? Might show there why it's failing.
@@dataslinger6379 No nothing there, I'll figure it out thank you. maybe cleanup apt, and going to look into dpkg issues but not certain.
Hello Michael, Thank you for the video! Help me please with a couple of questions. Regarding the last point (bonus) you mentioned in case we run a query like `SELECT COUNT(*) FROM external_table`. In this scenario, it seems that the query is processed and billed 0 bytes, but there are no number of rows in the external object's metadata. Is this due to the cache being enabled in the SQL settings? Additionally, when we talk about the BigQuery best practice of not clustering tables that are smaller than 1 GB and using partitions when you expect to have more than 1GB per partition. Does this refer to the physical or logical size of the data?