Renumbering my PROXMOX Cluster because of an IP Oops!
HTML-код
- Опубликовано: 19 апр 2023
- Sometimes you need to change the IP addresses on your Proxmox cluster, and it's never a particularly fun time. But, come along for the tale of my late-night frustration as I realized I'd have to renumber my cluster, and learn from my mistakes so you don't make them too.
In this case it was caused by Comcast changing my IPv6 prefix (something they haven't done in 2 years), but it can happen any time you need to move IP ranges for any reason. So, make sure you have backup ring networks for Corosync, and be aware of how to bring the cluster back from any accidents.
Feel free to chat with me more on my Discord server:
/ discord
If you'd like to support me, feel free to here: ko-fi.com/apalrd 
Наука
![MURDER DRONES Series Finale [TRAILER]](http://i.ytimg.com/vi/p6mhC5Iatns/mqdefault.jpg)
Maybe the only tech tuber who doesn't pretend like IPv6 doesn't exist
It really makes things easier to deal with when you use it properly
@@apalrdsadventures Clearly not easier if your ISP can mess up your network randomly
@@rexsceleratorum1632 It is perfectly normal, expected behaviour for your IPv6 ISP to change your prefix, just like it is perfectly normal, expected behaviour for your IPv4 ISP to dynamicall change your /32 assignment. What's mostly annoying here is that Proxmox doesn't gracefully handle interfaces having multiple IPv6 addresses.
Yeah back in 2017 I setup a ProxMox cluster for a LAN party, had to re-IP right before the event... Glad to see it's still broken.
It seems to be related to Proxmox doing its own internal name resolution within the cluster (which makes sense, but is tied to the hosts file and Corosync working) plus using static IPs in Corosync. So it doesn't really affect single nodes.
@@apalrdsadventures It would be nice if there is a WebGUI tool for situations like this. Maybe ansible in the future?
1:00 you can also get a IPv6 PI /48 prefix as a private person, but you need to apply for your own AS number, and get your own IPv6 prefix from a LIR of your choice.
Then you can find a hosting provider which supports BGP and PI address space (like Vultr) and peer over a IPsec+GRE tunnel. I plan to do this but still have to learn a lot to do this. :)
Definitely sounds cheaper than getting an ISP that does BGP, but a bit more work than I'm ready to do just yet
Man, I really enjoy your content. I also really appreciate the manner in which you reply to others in the comment section. You have lots of patience. Thank you for what you're doing.
I get a new prefix every 24h :) I think it's a good habit to also use ULAs (RFC 4193) in a "managed" network - they are under your control and never need to change. Combined with another address family and/or OSPF is a great way to stay connected..
It's a combination of a lot of things. Ideally I'd have both a ULA and GUA and use the ULA for corosync, but Proxmox's network configuration isn't a fan of that (it manages /etc/network/interfaces and will remove extra addresses if you add them manually and then make a network change from the GUI). But other than that, using ULAs in a globally-routable network isn't good either because the hosts need a GUA address to communicate to with the internet anyway. So ULAs are not for the 'main' interface, just the internal-only ones. This particular cluster only had one interface, which was the main one.
With a provider-independent prefix it wouldn't be an issue, and using Corosync across the isolated storage / migration network (which does use ULAs) it wouldn't be an issue either.
@@apalrdsadventures I know some of these words.
A+ Sound Check
Thanks man, quite worked for me
Been there, done that. In my case investigation where those addresses were hardcoded took really long time. Plus some of those files you cannot edit until you stop corosync and pve-cluster and start local pmxcf.
It have FULL CIDR so even expansion of network from /24 to sth bigger will brake some things. I had strike twice with this. First static IP address made by infra was made by somebody lazy and he make arp static entry in dynamic pool. And some day different new device negotiate same ipv4. Second time network CIDR change from 24 to 21.
It will be good to post those instruction you made as some of those things you mention are not obvious until they strike you
Changing the subnets are always fun!! Especially if you have large number of servers with static IPs.
Hello, yes another question, when I try to edit the files you suggest I get permission denied, and this is actually on the node itself. Node 1 Actually. The files still have the old 192.168 address. I couldn’t set up another back net work because I don’t have the interfaces in two of Lenovo nodes that I have. I guess there’s a way you can add a interface with USB. Also I believe there is a port they can be installed to main board you can swap out the current module installed which in this is a serial port.
I reIPed Proxmox once. Angry seems like an understatement.
0:05 Bro you got me 😂😂😂
Very good content and I am glad you could recover your setup... But why would your external IP interfere with your LAN? What's the reason for having the cluster on public V6 space?
I'm sorry if that's a silly question, btw.
In IPv6 all addresses are in the public space, normally
@@apalrdsadventures Got it, thanks. Looks like there's so much I don't know about IPv6, then.
Not sure if I understand the subject fully, but why chorosync net was in public IPv6 space anyway?
If it is a management network, couldn't it just be setup in local address space?
One of my clusters has a public network (public v6 space) and private network (private v4 space purely for storage, not routable anywhere else). That has Corosync on both.
The other cluster has just the public v6 network, so all of the addresses are public as is normal in v6.
Why don't you add the new ips to the corosync config 1st? You can have more than one. Then the configuration would replicate throughout. Then do the other changes. Then remove the old corosync ips once working on the new net work?
In this case, the network wasn't routing the old prefix, so the nodes wouldn't have an IP on both networks at the same time, even if it was in the corosync file.
But adding an additional address temporarily could have solved it, yes.
I suggest adding ULA (unique local addresses) to your network. All local configuration will never change (unless you make the change) and the address range is non routable on the Internet. (fd00::/8). If you ever have a prefix change again, the only thing that will break are those services pointing to those ISP assigned prefix address ranges.
I actually used ULAs and removed them from the general LAN networks since there are less than ideal issues with address precedence (ULAs are less preferred than v4 addresses, which are less preferred than global v6 addresses) that result in v6 never being used.
I do use ULAs on isolated networks like the storage and cluster backend networks, which can't route anywhere.
@@apalrdsadventures I use them only for any local service that will only be advertised via ipv6 or for services like a dns server. I have my RA ND send out my ipv6 dns servers only via the ula addresses. I would think your cluster configs (ceph) would work well with the ula addresses though. I have not done any work on Linux clustering in a decade though.
@@apalrdsadventures
> (ULAs are less preferred than v4 addresses, which are less preferred than global v6 addresses)
Thankfully there is an IETF draft in review that would change this default policy table behaviour; search for "draft-buraglio-6man-rfc6724-update-02".
Ya, I don't use IPv6 internally when it's tied to the WAN unless been given my own static IP from the ISP. Not fun to wake up one morning to find things broken and figure out what the hell happened. Changing the IPs in ProxMox cluster is tricky but can be done. So for my servers I rather stick to IPv4 internally where I have full control of it. IPv6 can be attached as secondary so it would not mess up the critical services.
It's been a few years since my ISP has changed my prefix, so I wasn't expecting it. My IPv4 does change occasionally (~6mos or so), but the v6 hasn't ... for at least as long as I've been logging it.
Using private IPv6 ULAs would also work just fine.
I changed the ip on proxmox and it still replies to pings on the old ip address as well as the new one.... ipv4 though but I burned over an hour on this today trying to figure it out
Did you change the etc/network/interfaces file or through the web UI?
yeah it was prob because you didnt restart your net interfaces, so it just added another ip on your interface
when this happens also doing cd /etc && grep -r "your-op" can help
Hello, thanks for the video. You have one of the only videos on this subject. My issue is similar but self-inflicted I changed my home net work from flat 192.168 to subnets 10.10, 10.20 ect. My main reason I ran out of IP addresses. So I did this without really thinking it through with the multi node proxmox cluster ceph shared storage etc.. so I’m like now what. So I went online for info on changing IP address. Your video came up. I was hoping for more of a live tutorial on the commands, etc. been I’m very new at this. I still have the other network running sort of it can’t access the Internet, so I can access the proxmox etc.. none of the VM’s can access the Internet and the nodes can’t talk to each other. I’m just wondering what would be my best method should I create the back network as you suggested I’m not sure what would be my best method. Thanks I appreciate your time and knowledge.
If you can configure a second network (corosync calls them “rings”) for corosync you’ll have a much better time. If that’s ipv4 and ipv6 then you can renumber them one at a time. Otherwise setup a temp network on ring1 (and add a second ip in etc network interfaces), then renumber the first one one node at a time with the second ring syncing changes, then delete the second ring.
The gui won’t let you set more than one ip per interface but Linux will. Just add a whole other “iface vmbr0 inet static” with an address and no gateway.
@@apalrdsadventures hello and thanks for the reply. OK as it turns out after checking one of the nodes that I have a supermicro has multiple interfaces and using one for 10 gig storage I couldn’t figure another one for a back network the other two tho are Lenovos P 93 I believe similar one has USB C thunder bolt the other one possibly not. But I do believe there is another way to add an additional port. These are equipped with serial port, I believe in the rear. I watched your IP 6 video and was under the impression that you didn’t need an interface for the IP 6 address as you were using the loop back interface so I was a bit confused on that. Also, I am running a UniFi net work. So my ultimate goal would be to have separate networks for storage and back net work for Ceph right now it’s running off the main network.
It doesn’t have to be separate physical nics. You’re just using it temporarily. A second ip on the same host works.
@@apalrdsadventures OK I’ll give it a try I’m with Comcast, and when you select to enable IP6 it asked what do you want for a prefix and you can select from 48 to 64 I believe with DHCP there is also SLAAC which I have no idea about, what should I select and why? Some say there is a way to possibly mimic IP4 addresses is the way of keeping things in order kind of like IP for when you do add a subnet number to match the VLAN.
Slaac is stateless auto configuration. Usually that’s the right choice. But for Proxmox corosync networks you must use static.
lol, "I POops"
What is your advantage to running ipv6 at home? DNS sounds like it would be way way easier to fix something like this with.
In general DNS should deal with this, but Proxmox resolves nodes within the cluster on its own without involving anything outside of the cluster, using Corosync.
As to v6, in general v6 is simpler to administer (automatic link-local, no NAT, simpler subnetting, no overlapping RFC1918 regions). In general, a v6-only host can get to the entire internet (via NAT64) while getting a v4-only host to the entire internet is harder.
@@apalrdsadventures for your last point - do you know of any major websites that are v6-only?
What if you don’t use IP V6?
Same as changing a v4 subnet, just with longer addresses
sounds like an upside of NAT!
Gross, no. What's the upside?
It took me way too long to figure out that every time you said "etsy," it was supposed to be "et cetera." 😒
I googled... He's right in pronunciation according to the creator of the system layout but I'll never ever say that :D
Thanks for sharing your experience with IP configuration issues in Proxmox. It's helpful for other administrators facing similar challenges.