A homing pigeon is faster than my fiber Internet
HTML-код
- Опубликовано: 7 июн 2024
- Bird beats Internet. Or does it? It hasn't been tested since 2009!
Thanks to 45Drives for paying for my flight to Canada. You can check them out here: www.45drives.com
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: redshirtjeff.com
2nd Channel: / geerlingengineering
Some of the items used in this video (affiliate links):
- Sandisk 1TB Extreme PRO USB Flash Drive: amzn.to/3qG8X8h
- OWC 5 port Thunderbolt 3 Hub: amzn.to/3qNW4sI
- Sabrent Rocket Q 8TB NVMe SSD: amzn.to/3OWkOqL
- RIITOP NVMe USB 3.1 Gen 2 Dock: amzn.to/44nrfJ6
Thanks especially to the following for helping film this video:
- @TechnoTim
- @Level1Techs
- @CraftComputing
- @45Drives
Contents:
00:00 - Never underestimate bandwidth
01:22 - South Africa, 2009
01:45 - Homing Pigeon 101
03:09 - Time To Transfer
03:59 - Flight!
04:58 - Pigeon transfer speed
05:34 - I am become pigeon, transferer of data
07:17 - Flight to Canada
08:14 - Results
08:46 - Pigeon Data Transfer Equation
09:29 - Risks and problems - IPoAC 
Наука
Who'd have thought a GoPro would make such a good ips/port blocker! Not only was it monitoring the port, but it also blocked incoming traffic!
Haha totally missed opportunity for a joke there.
@@JeffGeerling haha, I always enjoy the hard work you put into your videos, Jeff. Glad to see that you're doing well and still trucking along!
gopro firewall
made my day
Unfortunately, it's not a very effective traffic filter. It blocks even the expected and desired traffic
Never underestimate the bandwidth of a truck full of hard-drives driving down the highway. That original quote about station wagon is by Andrew Tanenbaum, 1981
Now that I think of it, I think Tanenbaum's text was the source of the exercise to compute Bernie the St. Bernard's bandwidth
@@stevepolingit was from his book about computer networks, great memory
Euhm hard drives in 1981 :/ "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." tapes mate tapes ;) even now it's better to send a truck full of data tapes then a truck full of hard drives.
Doing some calculations:
The interior of a cargo truck (first one I could find) is 26'5" x 8'2" x 8'3"
The dimensions of a 3.5" HDD is 5.75" x 4" x 1"
The largest capacity for a 3.5" HDD is 26 TB.
Assuming 10% (sorry, don't have a realistic estimate, this is completely random) packing overhead in each dimension, the number of hard drives that would fit in such a truck in one 2D layer perpendicular to the road would be 24 x 24 drives, which is 576 drives, which is 14.625 PiB / "slice".
The legal speed limit for trucks is ~100 km/hr, and if there are on average 3 truck interior sized gaps between trucks (again, completely random, sorry) then there is on average 3.65 PiB of information continuously flowing per 4" slice. The bandwidth is then 39 in/m x 1000 m/km x 100km/hr / (60 s/min x 60 min/hr x 4"/HDD) x 3.65PiB ≈ 1 EiB/s / highway lane
Maybe my math is hopelessly wrong, but if not 1 EiB is to 1 GiB what 1 GiB is to 1 Byte.
Okay, now I think my math is hopelessly wrong due to hopelessly optimistic scenarios and assumptions (obscenely huge number of trucks on the road, with very tight packing). The actual bandwidth is likely ~2-3 orders of magnitude lower under more realistic conditions, especially if a single truck is considered (as opposed to one every
@@JK-mo2ovI also watched the video
Put the pigeon in a box and strap that box to another pigeon. That way they can take turns flying each other to their respective 'homes'. Voilà! Two way data transfer!
Hahaha
I feel like there's a Monty Python joke in there somewhere, but that was about migrating coconuts and not storage drives. But I suppose two doves could carry one if they grip it by the case.
Pigeon duplex
That's basically a VPN
@@leftcoastbeardnot about where he grips it. It's a simple matter of data ratio. A 16 Oz bird can't carry a 16tb hard drive! Even if he weighs the same as a 1lb coconut.
I've met Pijeff in person. He's even more frightening than Red Shirt.
Hi Jeff! I love that Pigeon-Jeff was using the updated RFC2549 "IP over Avian Carriers with Quality of Service" which proposes the use of QoS tiers "Concorde, First, Business, and Coach". I love those April 1st RFCs
I saw the thumbnail and _immediately_ clicked to hear the latest progress on the IPoAC standard.
Came here to make or like this comment ♥
packet loss was always a huge problem, often caused by hawks and similar big birds...
@@autohmaeReally focuses the mind on the appropriate error correction strategy.
This RFC still doesn't support NAT, the birds keep eating them!
I wonder how many truck drivers have unknowingly set bandwidth records when they get tasked to drive a shipping container from WD or Seagate
Good question, but most don't care what they're hauling as long as they get paid for their miles.
As long as they're licensed to haul it, they don't care
And what kinds of records can be achieved with kilometers-long freight trains full of these... Ping time, of course, will be garbage, but bandwidth... A whole another league.
BUT the drives coming from the Factory are most likely totally empty RE Data, so useful Bandwidth would be close to Zero
So would it be a PDOS instead of a DDOS attack if we all sent pigeons your way Jeff?
Whole new meaning to "Your network 💩ed itself."
This video took a lot of dedication. Jeff in pigeon head walking around the airport is something I won't ever forget.
I wonder what kind of looks he got from the other passengers.
@@davidgreen8512 And airport security...
Time for a data transfer startup based on pre-programmed drones with relay terminals to recharge or change carriers.
Lol as a drone enthusiast this sounds *painful*
@@darkmann12 Its based on AI/ML and managed on blockchain.
I already see investors lining up for my vaporware tech 👍
This actually sounds viable, if planned properly.
But I'll stick with bicycle couriers for now, at least for
@@ask_carbonbut what will you name the coin? That's all I need to know - how fun is the coin's name.
Has anyone tried mass driver-glider drones yet? You launch with a mass driver (catapult, basically), and then glide to the next one. No need for recharging!
Gives a whole new meaning to "your data is in the cloud"!
Just a fun addition: If you fill up a cargoship that can carry 20000-containers with SSDs (assuming 200 gram/SSD, and 1 TB / ssd and 25 tons / container), you will have a container ship that carries ~2 billion terabytes of data. Just for the sake of it, this container ship will be the datacenter we want to transfer to the other place, so no transfer from or to the ship is necessary (it is so big anyways, this would just make sense)!If it takes 8 weeks to ship 2 billion terabytes to its destination, e will have achieved a total transfer speed of ~400 terabytes/second. Quite good, imo!
Google is jealous because their Dunant subsea cable can only do 250 terabits per second.
@@C4rb0neum They can be constantly upgraded as DWDM is always changing
Jeff as a South African, as I saw the headline, I immediately thought of the Durban experiment, and surprise me when you referenced the story. I was working for an ISP at the time and let me just say, the story, was sensational at best.
It's mostly for fun :)
having experienced Telkoms kak internet I think a pigeon was certainly quicker.
Afrihost es ook kak. Internet shaping blockheads
A lot of film studios hand-carry footage on SSDs back from location shoots exactly for this reason.
Can we try an A380 full of SSD's as a bandwidth test next? 😉
and an Airbus A380 full of... Arc A380s
#mentourpilot are you listening?
Back in the 90s, it took over 24 hours to transfer a file from the US to New Zealand for a specific use case. And that assumed the connection did not fail in that time. Our solution was to cut 5 copies of the file onto 9-track tape, put that in a case, and hand the case and a plane ticket to a courier. Travel time was less than 24 hours, and at least one of the tapes would work. (Rarely wsa that not the first one. We only cut 5 as that was how many fit in the case.)
I recall when the Event Horizon Telescope had finished observing, they had petabytes of interferometry data at each location. So they took their super expensive helium hard drives and hopped on a plane to bring them back to their main office
I remember talking to one of the scientists who worked on it after a colloquium, and apparently a lot of the data was also transferred by truck. When we asked him what would happen if the truck crashed and they lost all the data, he just shrugged lol.
Now backblaze needs a pigeon recovery system
What a great video idea and execution. Greetings Jeff, I enjoyed this. :)
Ha, high praise indeed! Thank you :)
Concidering that i get about 5Mbps upload, everything is faster than my internet
Even a hummingbird would probably be faster!
@@JeffGeerling Be careful with that extra induced noise, could create EMI/EMC problems down the line.
I’d love to be a fly on the wall when this was pitched. Great video!
You'd have to be careful, a pigeon might've eaten you!
Sorry to knock you off your perch Jeff, but surely pigeons prefer GRUBs?@@JeffGeerling
I remember the wikipedia page for IP over Avian Carriers protocol had a photo of a dead pigeon as an example of data loss.
It's still there.
Years ago when I was working at a service provider, I needed to copy 3 PB of data from New York City to Dallas over the 10GbE backbone. I started on a Friday and as of the following Monday not even 1% done. This was going to take too long. I had the time to buy a storage array, set it up copy everything over and then have FedEx deliver to Dallas. All that in less than 1/4 the time it would have taken a direct copy. That sure changed my view of data coping over the internet.
Those little samsung bar usb drives are significantly cheaper than the sandisk options while being somewhat comparable in performance (slower writes but faster reads). The caveat being the metal enclosure is... well, not easily removed. My father and I used to raise homing pigeons(and doves). We actually got most of our birds from the soulard market. Unfortunately we moved to Imperial, a more rural area, and the foxes got to them.
RIP to those poor birds! Foxes almost got the chickens we used to have too... they are cunning little creatures.
What is it with foxes attacking means of information transfer? Where I live in Northern Virginia, they used to go after newspapers.
(It's actually a really cute story-a daddy fox was taking them back to his den to use to teach his kits how to pounce)
I used to be a professional Pijeff! Back when most of our sites had T1 or T3 access at best, if we wanted to migrate a datastore from a manufacturing facility in, say, upstate Wisconsin back to our primary datacentre in Toronto, the only way we'd actually get everything moved between end of day Friday and Monday morning would for me to fly down there with a few external hard drives in my backpack, clone the files overnight, then fly back the next day and start dumping everything back on the servers at HQ.
The OG human pigeon!
I remember the line about the station wagon while working on mainframes in the 80s. We had a daily delivery of tapes between two locations that were about 30-40 minutes apart, or more in bad weather. We were working on telecom programs which management hoped would replace the station wagon (it may have been a van in our case.) The comm line bandwidth available to us at that time was so meager it didn't come within several orders of magnitude. We kept the program around for small stuff and named it STAWGN.
Pigeon backup is peak "security through obscurity"
We acctually do know how pigeons know their way. They have magnetoreception so they can sense earth magnetic fields and navigate using that on long distances and when they are getting closer to their destination they can recognize landmarks on the ground to get exaktly where they need to go.
That is the leading theory I've read, though details are still not quite nailed down. Some tests in the field have proven it is not just magnetic.
They also see ultraviolet, so a whole other spectrum of colours to landmark on.
That'd be fun when the magnetic pole shifts.
I think they use natural GPS and store all the grid data in their feather cells. We just haven't figured out how yet
Sorry Jeff, your analysis is floored, that higher bandwidth bird is a heron, not a crane!
This has always been the case. It was quicker to copy LSL, get on your bike and ride across town and give it to your friend than it was to transfer it over modem.
Terrabyte per mile, I love it
with my upload speed, it's faster to format and fill AOL diskettes and drive them to their destination.
Heh "floppies over snail mail is faster than the Internet"
@@JeffGeerlingI miss my 1tb synchronous fiber optic, cable sucks. my upload is similar to "really good dial-up"! I shouldn't have moved!
Strap a floppydisk to a snail, snailnet.
@@kreynolds1123I actually know someone who worked on a system where the last (or first) leg of data transfer was "DbD" or Disk by Donkey :)
What a wonderfully whimsical and interesting idea for a video. Really loved this one. Also hoping we get to see more of your time with Wendell and with 45Drives!
When I took Data communication in school on the final exam we had to describe the transmission characteristics of a 747 cargo plane (large packet size, high latency). In the '80s I had to transmit data on a couple of floppies from NY to LA and we were better off overnighting them than to try to transmit over a modem. Back then data communication was often harder than the programming. First thing I thought of with the title was RFC-1149, glad you mentioned it.
So, Amazon, Microsoft and Google should all have a Greg on site at each datacenter. That way, there's a pigeon service for everyone!
They actually do it. By trucks, not pigeons though.
As the joke says, who’s gonna be affected if Microsoft’s data centers go down? 😅
Back in the '90s, as a teen during the waning days of dial-up, I was associated with a local greyhat group. We used a lot of sneakernet for large scale sharing/pooling of data for both throughput and security reasons. I fully expect that a lot of modern groups in both blackhat and greyhat circles still do this for similar security reasons, if not transfer speed and transfer convenience ones, since a random guy with half a dozen flash drives hanging off a carabiner on his beltloop or backpack (or in the day boxes of floppies and Iomega Zip disks in his backpack) is still way harder to packet sniff or man in the middle than online data transfer packets.
This was AMAZING! I love birds almost as much as I love computers, and seeing how he takes care of these birds, what he does to condition them and keep track of them and their health is really important. Great video man!
I heard about this concept about 12 years ago. On a whim I went to look up sneaker net thinking it would be funny, and realized I went down a rabbit hole, I gained a whole new respect for sneaker net.
Quality content dude. I tell my customers our latency is better than dial up, dsl, cable, smoke signals and carrier pigeons. But the carrier pigeons have more bandwidth....
at 5:49 I take it all back LOL still just a nerd doing nerd stuff.
Talk about unique content, you win the prize. I love the amount of thought that went into the testing protocol. Thanks for sharing.
The quote is from the book "Computer Networks" by Andrew Tanenbaum: "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway" (I could swear it was a ship, but I checked on the book): page 57 of the second edition, page 83 of the third edition and page 91 of the fourth edition. This book is a classic used in many university courses. I had one and when I married my wife had another one. It is the network equivalent of the Dragon book for compilers and the Art of Computer Programming for algorithms. This is why is so frequently quoted along with the often misquoted "apocalypse of the two elephants". Some say he was paraphrasing Warren Jackson, Director of University of Toronto Computing Services but my second edition says nothing about that, so I can't confirm it. In fact at the end of the Chapter 1 there is a problem about a St. Bernard dog carrying floppy disks and over what range of distances the dog is faster than a 300bps phone line.
This is a great video concept! Love it!
Doves/pigeons look kind of blobbish in terms of body shape but they are actually remarkably good fliers, even beyond those with homing abilities. That big, curved chest holds a huge pair of pectoral muscles. They are fast and extremely agile - if they manage to escape a raptor's first (usually surprise) attack they usually get away entirely.
I used to do data center migrations. We used to put servers and tape libraries on planes for large moves and sometimes when it was bank data there was a bank employee that would travel with them. A lot faster way to move Pb /ExoB
One day I want to be Greg, just raising and caring for birds.
I had this problem back in university when a project required lots of photos or video.
I actually ended up doing the math on it once and given the crappy internet I had, it was sometimes better to not even bother trying to upload overnight... and instead drive 45 minutes to the university, upload over LAN, and drive 45 minutes back. That informed my project priorities more than the actual due dates.
LOL. I see this now. I did my graduation paper in 2010 with TCP/IP over Pidgeon :).
Huh, I always thought that all doves are pigeons, but it turns out that all pigeons are doves!
And all these comments are amazing!
I had a wayward homing pigeon hang out in my yard one morning a couple weeks ago. This was just the most appropriate video to mention that.
Really off-the-wall interesting video Jeff, good to see the old methods are still the best, and broadband providers have still some way to go to beat even the humble homing pigeon.
Back when I was in grad school I calculated the bandwidth of Bernie the St. Bernard carrying a little whiskey cask filled with 8" floppies across the Alps. I'm not suggesting a follow-up, but I wonder...
I first heard about pigeon vs internet just a few weeks ago and immediately did the math based on updated internet speeds and storage density. Same conclusion, pigeon is faster up to a certain distance.
An advantage of internet is that you know immediately if your transfer has failed and can restart it while with a pigeon you wouldn't know until you've waited for a significant time past expected arrival. Pigeon could still win out over shorter distances, but over longer distances it boils down to when the interruption in internet transfer takes place.
Gives a new meaning to "uptime"
I understand adding the final "upload" time, but I don't feel the initial "copy" step should count towards the "pigeon time" since you could theoretically have had the data directly on the physical medium which is intended to be transferred via pigeon to begin with without needing an intermediary step and when you transfer it over network you are uploading it directly from that original source as well.
More people would've complained of biasness and cheating. It's better that he did it like that. After all, you must edit the video, THEN upload it. So it came from your computer, not from the SD card. Of course, there are scenarios and scenarios. But I think that there's enough of them that imply that the source is the computer, so this was the best, safest approach.
It was fun to found this vid right after i rode a bike for 10km to my colleague with an external ssd to copy some container images and models, because doing it over the internet was painfully slow for reasons)
When you face a huge volume of data - you start to appreciate portable drives
Me: *sees Jeff in pigeon mask*
"Has he finally lost it?"
This was fun yet serious, no, just fun. I’m glad you remembered to include the station wagon quote. Glad to see you are back in good humor. Cheers!
Imagine: disconnecting from the constant data stream of the internet to roughly three times a day data dumps with Alphabet and the alphabet agencies watching the birds overhead and asking themselves "How are we supposed to hack their network and do 'Man-in-the-middle' attacks on them NOW?" Now imagine said bird getting a gleam in its eye and performing an airdrop on an agent while still flying.
This was interesting as well as entertaining. Great Video. Thank You.
I love that you mentioned RFC1149. From the start, I was wondering if you would. Most people think I am joking when I mention that an RFC for IPoAC exists.
Positively insane premise and I absolutely loved it.
0:55 I love how Red Shirt Jeff takes one for the team to explain bandwidth vs speed
This is the type of high concept, high tech content i crave.
this video is amazing. thanks so much for doing this! and I hope you enjoyed my home province :)
Nova Scotia is beautiful! And outside the weather delaying my return home, the weather has also been refreshing compared to the 110°F heat index in St. Louis!
This was an interesting experiment. Thanks for sharing another great video.
1:00 thus dispelling the myth that “the internet is not a big truck that you just dump things on”
Is everything alright Jeff? Coo once for yes.
Kaw
Good old sneakernet. Loved the video Jeff!
Wow - cool video, featuring the pigeon test from my Country! 🤣😎
This technique is known in Argentina as "telesovaco" (from "Tele Armpit") due to the gesture of the messenger taking an envelope and going out to his destination.
I immediately thought of the station wagon full of tapes model when I saw the thumbnail! Brilliant video, Jeff.
Really sweet vid. Nice, nice job.
Every test of the Pigeon Packet Protocol is a boon for humanity.
Also, given that an increase in storage speed always seems to outpace network transfer speed, it's likely pigeons will always be faster than wires for transferring data.
Moderate/high tangent here: But if youse haven't seen it, I highly recommend the Jim Jarmusch film Ghost Dog - Way of the Samurai.
The protagonist lives on a (New York?) rooftop and works as an enforcer for a Mafia guy - but all his communication with his lord is done via carrier pigeon.
It's a quirky and genuinely fun film, starring Forrest Whittaker and with a soundtrack by The RZA.
I work at “the big G”, and I actually used to work on a product similar to the snow mobile. We had one customer that would have taken 8 months to upload multiple petabytes into GCP…but we sent them 2 appliances, and the entire process took about 6 weeks.
Now this is the content I subscribed for!
There was a software company in Germany in the late 80s or early 90s that made computer games. Because of the lack of high speed internet or high costs of the internet at that time, they deposit discs in trains. They did not even buy a ticket. On person entered the train at station X and put the disc somewhere in the train. At another station another person just got the disc. However, sometimes the second person missed the train or something else went wrong. Not a perfect solution.
And there is RFC 1149 about TCP/IP over carrier pigeons / avian carriers. And RFC 2549: IP over Avian Carrier with Quality of Service.
Now I want to see the station wagon full of tapes race to see just how much bandwidth that could have.
I hope you enjoyed your stay at my home province! 😂 I flew in and out of the Halifax International Airport this month and am a little disappointed I didn't see you or a human pigeon on my travels. 😢😅
I am so happy you mentioned the April Fools' Day joke, IPoAC. It has always been one of my favorites.
It's worth noting that you don't _necessarily_ need to account for the transfer time from the medium to the recipient during a physical data transfer depending on the nature of the storage media and the recipient system, it could (and often is, in real life scenarios) simply be incorporated into the recipient system directly, potentially as quickly as plugging a transfer drive directly into a free hot-swap bay on the front of a server rack or other similar system
Years ago I did a calculation of the bandwidth of the factory where I was working. We made DVDs and Blu-rays. Assuming maximum production, and double checking to make sure the loading dock could handle that many trucks, the bandwidth was thousands of times faster than the Internet of the time over any disance that we could reach with a road. I suspect that a modern race between today's faster Internet and an NVME drive factory or an enterprise rotating disk factory would be even more unbalanced.
Amazing Jeff, love this! :)
My brother in law and his farther both raced pigeons. They raced in The Up North Combine LTD. The birds would be sent to France from the North east of England. that could well over 300 miles
Thanks, Pi-Jeff --- great video!
Was a great April fools RFC back in the day (April 1, 1990).
I've toyed around with this idea since I first heard of IPoAC; but that transfer speed onto and off of the storage device is a real dream killer.
The single point return problem for pigeons is similar to the problem of needing infrastructure for any network. This isn't a big deal for most of the developed world anymore, but where I live, in rural northern Michigan, it's very much a concern for a lot of people. In our case the comparison is often pigeon vs no internet. I'm lucky enough to live off of a road where a cable line passes between towns, so I can get cable internet, but if I were 1/2 mile north or south it would be modem (or Musk, if you can get it).
I am glad you didn't actually let any bird carry the drives over an extended distance on it's leg.
Yeah I am hoping if we go any further I'll have a better solution for more data, and less physical space.
This was super cool, I would have never even thought about this to be honest. I feel like their are two really important things to bring up though when it comes to physical transfer of data.
1. Cost- it’s so much cheaper to just use the internet in most circumstances. If your internet is too slow you would find more utility in just getting a larger pipe. I.e instead of the garden hose go with the firehose.
2. Convenience- it’s really impractical to transport physical storage over long distances. Even in the situations where it might make sense there is so much logistical hurdles that you just don’t have to worry about when you keep everything digital.
This is the kind of content I'm here for 🐦
Literally the most inspirational video I've watched in a while!
Network engineer here: First off, I love this video. This is clearly a labor of love and I'm digging the all the little nerdy intricacies.
Secondly, I want to give my normal PSA that I give to folks when they are talking about speed. You mentioned that one doesn't always get gigabit speed. With a 1Gbps physical connection (line speed), you will never get 1Gbps data transfer (because of overhead). Now if the service provider gave you a 10Gbps physical handoff (line speed) they would truly be able to give you 1Gbps of actual throughput.
Just adding a note that my home connection is actually 1.3 Gbps but I'm still stuck on my old 1 Gbps router (which does up to 930 Mbps as measured by iperf3-which means less than that for real transfer speed).
@@JeffGeerling oh nice! I seem to remember seeing optics with 1.(some decimal)Gbps. That's awesome that providers are using those. Means you can potentially get 1Gbps throughput at Layer 3. I am curious if the copper handoff to you is 2.5Gbps or 10/100/1000. Or are they giving you a fiber handoff?
Really great video! I know it's a bit silly... But it was great!
I sense a new Monty Python movie coming.
What is the air speed velocity of a homing pidgeon?
- Loaded with 1 or 2 TB of data?
Ehh… I don’t knoww….. aaahhhhhhhhh….
This is probably my favourite video this year
And this is my favorite comment 😄
It have already been tested multiple times, in England specially.
But it's always a god thing to test it from time to time, as both internet speed and storage density increase each year.
Thank you for the video.
I'm just waiting to see this error on my screen: "forgot to shut the pigeon cage, and the remaining two pigeons escaped without an IP packet"
Now this is quality RUclips! Thanks, Jeff!
Fascinating, and Insane, love it.
0:51 FINALLY! The Revenge on Red Shirt Jeff!