Hi Hussein, I am a SW Engineer and I love your videos. I am a Virgin Media customer and the outage has actually lasted a lot longer than the first day and continued for another 2-3 days afterwards. Today has been the first day without an interruption. Yesterday we were seeing it go up and down every 30 minutes. If Virgin Media releases a statement going into more detail about what caused the outage from their side, it would be great if you could do a follow up video which goes into more detail (i.e. why they sent the BGP withdrawals in the first place). Many thanks again!
Nice video, I have Virgin Media and during the outage some sites worked as normal, RUclips being one of them. It would be great to have a deeper explanation for why this was, your videos are great for gaining a better understanding!
thank you for sharing, my guess is your ip prefix was not among the ones that got withdrawn and as a result ip packets can find their way to your router. I can’t explain why some websites wouldn’t work, but I can guess, one theory it could be that virgin couldn’t accept BGP updates some ASes where those websites belong to, the other is RUclips ISP (google) still had the old BGP entry for virgin media and held on to it longer and didn’t accept the withdrawal compared to other BGP routers. the other bgp routers lost those entries causing the failure to route the ip packets back to virgin. best thing to understand what happened is traceroute and tcpdump during the outage
Love how you always clearly paraphrase tech blogs! I often watch your channel and when there's a notification of you uploading a new content I'll surely immediately jump in and watch!! I was wondering lately, how do you get these informative blogs? Are you subscribed on their newsletter of sort? or do you dig them on the internet? Sorry for such a silly question.
Omg no wonder I was troubleshooting WiFi during the night.. I thought it was due to the apparent cloudflare maintenance in manchester or something to do with pihole DHCP server... But changing DNS didn't do anything too :)
We don’t know the actual cause though. So far all explanations are around the symptoms of what’s been seen. And explaining this is good for people who don’t know how BGP works etc but all the BGP withdrawals are the symptoms. No one from VM has come out and said who fat fingered a change or what config stuff went wrong to actually cause the issue. 🤔
business as usual for vm customers, sadly. so glad i'm not one anymore. and it's even worse for their business customers. i rented 5 static ip's from them, and regularly experienced quite bad packet fragmentation, mismanagement of their gre tunnelling, and generally poor qos. and a cursory search just now shows their current business customers still experience these issues.
You know I think it's time to start to disintermediate ISPs at the layer 3 level. Just use them for layer two. In the commercial space I've set up autonomous systems for high net worth individuals or folks who just want to have complete independence. BGP in 2023 is not a heavy protocol to run. There are even implementations on raspberry pi. Yes you have to peer and you also need an ASN. But that's all able to be arranged, especially in cities with colocation hotels that peer with just about everybody. Hurricane offers later 2 and peering. Independent ISPs are also many times layer two providers. They're easy to work with. And in New York City you're looking at under $800 a month for fully symmetrical 1 gig independent wavelength stand with a 4-hour service contract. It's not as labyrinth or black boxe as most people would think. No it's not going to be $39 a month. But it offers so much security privacy and you control your own reliability. DIY layer 3
Hi Hussein, I am a SW Engineer and I love your videos. I am a Virgin Media customer and the outage has actually lasted a lot longer than the first day and continued for another 2-3 days afterwards. Today has been the first day without an interruption. Yesterday we were seeing it go up and down every 30 minutes. If Virgin Media releases a statement going into more detail about what caused the outage from their side, it would be great if you could do a follow up video which goes into more detail (i.e. why they sent the BGP withdrawals in the first place). Many thanks again!
This was fun! Thank you for making this video, Hussein!
Sending love to the Engineers who worked hard to bring Virgin Media back online!
Nice video, I have Virgin Media and during the outage some sites worked as normal, RUclips being one of them. It would be great to have a deeper explanation for why this was, your videos are great for gaining a better understanding!
thank you for sharing, my guess is your ip prefix was not among the ones that got withdrawn and as a result ip packets can find their way to your router.
I can’t explain why some websites wouldn’t work, but I can guess, one theory it could be that virgin couldn’t accept BGP updates some ASes where those websites belong to,
the other is RUclips ISP (google) still had the old BGP entry for virgin media and held on to it longer and didn’t accept the withdrawal compared to other BGP routers. the other bgp routers lost those entries causing the failure to route the ip packets back to virgin.
best thing to understand what happened is traceroute and tcpdump during the outage
This is fascinating, definitely my fav video on this channel to date!
Love how you always clearly paraphrase tech blogs! I often watch your channel and when there's a notification of you uploading a new content I'll surely immediately jump in and watch!!
I was wondering lately, how do you get these informative blogs? Are you subscribed on their newsletter of sort? or do you dig them on the internet? Sorry for such a silly question.
Please make a dedicated video on BGP! Ty :)
nicely explained.
@hnasr I remember facebook running into problems in 2021 and you made video about it.
best youtuber
Curious to know why outage occurred like what happened behind the scene which lead ISP to withdraw connections ?
Omg no wonder I was troubleshooting WiFi during the night.. I thought it was due to the apparent cloudflare maintenance in manchester or something to do with pihole DHCP server... But changing DNS didn't do anything too :)
We don’t know the actual cause though. So far all explanations are around the symptoms of what’s been seen. And explaining this is good for people who don’t know how BGP works etc but all the BGP withdrawals are the symptoms. No one from VM has come out and said who fat fingered a change or what config stuff went wrong to actually cause the issue. 🤔
Apparently it was an internal ddos on their core routers which caused issues with there fpc cards
This looks similar to that Roger comms outage in Canada last year
business as usual for vm customers, sadly. so glad i'm not one anymore. and it's even worse for their business customers. i rented 5 static ip's from them, and regularly experienced quite bad packet fragmentation, mismanagement of their gre tunnelling, and generally poor qos. and a cursory search just now shows their current business customers still experience these issues.
You know I think it's time to start to disintermediate ISPs at the layer 3 level. Just use them for layer two. In the commercial space I've set up autonomous systems for high net worth individuals or folks who just want to have complete independence. BGP in 2023 is not a heavy protocol to run. There are even implementations on raspberry pi. Yes you have to peer and you also need an ASN. But that's all able to be arranged, especially in cities with colocation hotels that peer with just about everybody. Hurricane offers later 2 and peering. Independent ISPs are also many times layer two providers. They're easy to work with. And in New York City you're looking at under $800 a month for fully symmetrical 1 gig independent wavelength stand with a 4-hour service contract. It's not as labyrinth or black boxe as most people would think. No it's not going to be $39 a month. But it offers so much security privacy and you control your own reliability. DIY layer 3
Although i managed to understand this, some visuals if possible would be easier to go through
"Virgin media got f***ked" sorry I couldn't control myself lol
Virgin Media turned on torrenting