I didn't move the cable tonight. The plan was to stage everything and then flip it in one sitting, but by the time I'd finished pre-flight checks I'd found four things that would each have broken something after the move — and none of them would have looked like a networking problem when they did. So I stopped on purpose, before the one step that can't be undone from a keyboard. No cable moved, nothing rebooted, no VM shut down. Everything below is staged on disk and inert until the next boot.
The job itself is small: move the Proxmox host and its two VMs off the flat LAN they've been on and onto the routed segment built in earlier sessions. The reason it's worth a whole evening of prep is that the stakes are higher than they used to be. The cloud migration finished since I last touched this plan, and now the production sites run off one of these two VMs instead of a cloud box — so this isn't a lab-only change anymore, it's a few minutes of the real sites being down. The thing that makes it survivable at all is that the tunnel fronting those sites is outbound-only: it dials out, nothing dials in, so the VM's local address genuinely doesn't matter to it. What matters is whether the VM has a working route and working DNS. Reframe the risk that way and it stops being about the IP change and starts being about any gap where neither of those is true — which is exactly what the DNS check turned up.
The DNS layers were lying to me
Both VMs looked like they had DNS configured. Neither configuration was actually in control.
One VM's staged DNS setting pointed at the existing LAN router's resolver — which only answers hosts on its own directly-attached subnet. Once this VM is on a routed segment, that query goes nowhere. The other VM had no DNS line staged at all, because DHCP had always been the one writing it, and switching to a static address quietly removes that.
Underneath both of those, the real answer was a layer I hadn't been looking at: /etc/resolv.conf on both VMs is actually generated and owned by a mesh-VPN tool's local DNS proxy, and it was forwarding everything upstream to an address that's only reachable from the old network. I only found this because I happened to run a privileged diagnostic command — the unprivileged version of the same check reported nothing wrong at all, which is a little unsettling in retrospect.
To make it worse, the two VMs sit on two separate accounts of that mesh-VPN tool, with different logins, so a fix applied in one account's console does nothing for the other. Both had to be changed independently. Both accounts also still had a leftover entry for a cloud node that's been decommissioned for a while — not dangerous, just clutter I should delete.
The actual fix was simple once I found the right layer: add a global nameserver in each account's console, and explicitly turn on the toggle that overrides locally-advertised DNS. That second part is the gotcha — I skipped it the first time through, and without it the global nameserver is purely advisory. The client keeps preferring whatever local value it already captured, silently. I didn't trust the config after that; I verified by actually doing an external name lookup from both VMs and confirming it resolved, not by reading a settings page back.
Three more things that would have failed quietly
- The hypervisor's own hosts file still mapped its node name to the old address. Left alone, that would have greyed out the web UI or hung the management interface right after the move — looking like a hypervisor problem instead of a stale hosts entry. This was the one edit I made that's actually live right now instead of staged, which is fine only because a shutdown is imminent anyway.
- A single-character typo — a letter where a matching digit belonged — in an interface name inside one of three NIC offload-disable commands. It's been failing silently at every single boot for a long stretch, so one of the three offload settings was never actually applied, even though the other two were. It's corrected in the staged config now, but if disabling that offload was originally meant to fix a corruption or stalling problem, that fix has effectively been running at two-thirds strength the whole time without anyone noticing.
- This project's own blog automation had an SSH target pointing at one VM's old address. That's updated now. There's a dated backup of the old config sitting around with the previous value still in it, but it isn't loaded by anything, so it's just a thing to remember exists.
There are also a couple of stale pre-migration DNS resolver backup files on both VMs, still referencing the old gateway. Inert where they sit — the only way they bite is if some future me restores one of them without thinking about why it was backed up in the first place.
Access notes worth remembering
One VM has root SSH disabled and uses password auth; the other is reachable by a normal SSH login. The standard hypervisor console-attach path doesn't work on the first VM at all — there's no serial device defined for it. The real fallback, and the one I actually used to write tonight's staged files, is running commands inside the guest through the hypervisor's guest-agent exec path. It needs no network and no password, which makes it the actual recovery method if a VM ever boots with broken networking — better than the mesh-VPN tool for that purpose precisely because it doesn't depend on the network existing at all.
Pre-flight, for the record
One VM is carrying a dashboard, a handful of personal and project APIs, a captioning/translation tool, and a web server — about ten services, all confirmed enabled and active before I touched anything. One of those runs as a per-user service rather than a system one, with lingering enabled so it keeps running without an interactive login. A trading-related process has no systemd presence whatsoever — not system, not user, not even a cron entry — which is consistent with "not running"; apparently it lives inside another application's own internal scheduler instead of at the OS level. The firewall is active but has no rule scoped to the old subnet, so nothing will start auto-blocking traffic just because the move happens. Neither VM has any network-mounted filesystems to worry about.
That VM's own self-check reported something that looked contradictory — the live network tool said the interface was still in automatic mode, which seemed to mean the static config hadn't actually been staged. It had been; the live tool was reporting the in-memory profile the interface booted with, not the on-disk file I'd just written. Disk is what gets read at boot, so the mismatch was expected, not a bug — but it's a good reminder that a status command and a config file can disagree without either one being wrong.
The other VM runs a container runtime with exactly one container present — an exited demo container with no restart policy, nothing actually running. I checked whether that runtime's own internal subnet overlapped the new VLAN segment, since that was the real collision risk here, and it doesn't. Two background services set to always restart and a surviving user crontab are also present and accounted for. This VM's own automated self-report had recommended tagging the new network at the hypervisor level, and that recommendation is wrong: the switch port it's plugged into is already untagged, access-mode. Adding a tag on top of that would double-tag the traffic and break connectivity outright. The report was guessing at a piece of network topology it has no way to actually see from inside a VM, which is worth remembering the next time one of these self-checks sounds confident.
What's staged and what's still soft
New static network configs are already written to disk — not active — on the hypervisor and on both VMs, each with its own on-disk backup of the config it's replacing. One VM's connection-profile file needs specific restrictive permissions set on it or the network manager just refuses to load it; worth remembering if a future edit to that file quietly stops taking effect. IPv6 is deliberately left on automatic/no-address on one of the profiles, since the new segment doesn't offer it — that's expected, not a gap to chase. Two small, non-blocking things are also sitting in that staged config: a low-priority connection flag carried over from the old DHCP-based profile, harmless as long as it's the only profile present, and one interface set to come up only on a hotplug event rather than unconditionally — which is the first thing to check if that VM ever comes up with no network at all.
The cutover itself — not yet run
The plan is ten steps: confirm the target switch port is actually free, shut down both VMs, shut down the hypervisor, physically move the uplink cable, boot the hypervisor back up, verify connectivity from the physical console — not SSH — in a specific order while noting exactly which check fails rather than just that something did, boot and verify both VMs the same way, confirm the production sites and a messaging gateway respond, and only then free the old uplink for the WAN connection. It has to run from the physical console because an early step kills any SSH session mid-command, and if the new network config turns out to be wrong, SSH won't reach the box at either the old or the new address.
One thing I made sure to write down in advance so I don't panic over it during the actual cutover: the switch won't show a route for the new segment until something is physically live on that port. Right now, with nothing plugged in, that route correctly shows nothing. The time to worry is if it's still missing after the hypervisor boots on the new port — before that, its absence is just expected, not a warning sign.
The claim I haven't actually tested
The new segment has only ever been validated from a laptop that kept a fallback path the whole time — a temporary address, host-specific routes, Wi-Fi still up underneath it. What I haven't tested is a host whose only route is through the new gateway, with nothing to fall back to — which is exactly the position the hypervisor will be in the moment it boots after the move. The switch's return routes for the new segment are confirmed present in its own routing table, so there's no specific reason to expect trouble. But "the path works" and "the path works as the only path" are different claims, and only the first one has actually been demonstrated. This is the same category of mistake I've made before in this project: testing a change while a fallback connection is quietly still active, and mistaking that for the change actually working.
Left for after the move
A few things to come back to once the cutover actually happens: update the saved remote-access shortcuts and the management UI bookmark that still point at the old addresses, and write an access-control list for the new segment — which isn't just busywork, because the path between the old and new segments is currently wide open, and that's the only thing keeping laptop-based management working right now. Writing that ACL means actually deciding whether to keep that access or close it. The switch's time-sync setting is still unconfigured from an earlier session. Two dead decommissioned-cloud-instance entries need deleting from the mesh-VPN accounts. And at some point it's worth consolidating those two separate accounts into one — not tonight, that's its own change with its own risk, not something to bolt onto a network cutover.
The actual lesson
Every one of tonight's four defects was the same shape: a config line that gets accepted without error, which is not the same thing as a config line that takes effect. A DNS setting with valid syntax that nothing reads. A global nameserver that's set but merely advisory until a toggle says otherwise. A typo that fails the same way at every single boot without ever raising an alarm. None of these would have been visible right after the actual cutover — they'd have shown up as "containers can't pull images" or "the sites are down," with plain ping working the whole time and pointing nowhere near the real cause. The rule that keeps earning its place in this project: verify by function, not by configuration.