How the
lab works

Proxmox Backup Server takes daily deduplicated snapshots of every VM and container, stored on a separate NAS, with tiered retention.

What made me start

I had been adding services and VMs to Proxmox for a while before I started to think that if this thing got corrupted or went down, all of that work was gone and I would be starting from the beginning again. That is when I knew I had to figure out a backup solution.

The first thing I worked out was not which software to use, it was where the data would go. Do I even have the space for this. The NAS was the obvious answer since that is what it is built for, so the real question became whether Proxmox could back up to a network share at all. It can, and Proxmox ships its own backup server that works hand in hand with it, so not having to rely on a third party for it was a nice relief.

The setup

Proxmox Backup Server runs as its own VM, with its datastore pointed at a share on the NAS. I put it on hardware I already had rather than waiting until I found a separate box for it. A daily job snapshots every VM and container. Retention keeps the last three, plus a daily for a week and a weekly for a month. Because the backup server deduplicates at the chunk level, a daily full backup costs very little extra space.

The week of red tasks

The Proxmox task log started showing entries in red. I ignored them for a couple of days figuring I would come back to it, which is the part I would do differently. When I finally sat down and read them, the pattern was obvious. They were all landing at the time I had set the nightly backup to run.

My first instinct was that I had configured the job wrong, so I retuned the backup settings a couple of times. That turned out to not be my issue. Once I stopped guessing and actually read the output from the failed runs, it was clear the backup data was never reaching the NAS at all. The backup service runs as a specific low-numbered user ID, and the NAS was remapping that ID at the share, so the backup server could not write its own chunk files. It was a permissions problem on the storage side, not a Proxmox problem.

The fix was a chain of small things: create a matching user and group with that exact ID on the NAS, add an access control entry on the share, set the share to map all users to the admin identity, set the service's file-creation mask, and repair the permissions on the files that had already been written wrong. I also forced the older NFS version, because the newer one made the remapping worse.

The whole thing took about a week, and most of that was not troubleshooting. It was waiting. I would change one thing, then wait until the next night's run to find out whether it helped. If you are setting this up yourself, that is the part to get right before anything else: make sure the backup server has the access it needs on the storage device to write files unattended.

Making it survive updates

That worked, but it was sitting on hand-edited system files on the NAS, and a firmware update does not treat those as protected configuration. An update can put the old settings back, which makes sense, that is always a possibility any time a patch gets rolled out. I could have written myself a note to redo it after every update, but then I have to remember, and I would only find out I forgot once the backups had already been failing for a while.

So I looked into automating it and found I could run a script at boot through the NAS's own task scheduler. It checks whether the user, the group, and the access control entry are there, and recreates them if they are not. It is idempotent, so running it when nothing is wrong does nothing.

# Runs at boot via the NAS task scheduler. Idempotent by design:
# it checks first and only acts when something is missing.
TARGET_UID=34          # the ID the backup service runs as
TARGET_USER=pbsbackup
SHARE_PATH=/volume1/<backup-share>

# Recreate the group only if the ID is not already mapped to it
if ! getent group "$TARGET_USER" | awk -F: -v g="$TARGET_UID" '{exit !($3==g)}'; then
  groupadd -g "$TARGET_UID" -o "$TARGET_USER"
  logger -t pbs-uid-fix "Recreated group"
fi

# Recreate the user only if the ID is not already mapped to it
if ! getent passwd "$TARGET_USER" | awk -F: -v u="$TARGET_UID" '{exit !($3==u)}'; then
  useradd -u "$TARGET_UID" -g "$TARGET_UID" -o -M -s /sbin/nologin "$TARGET_USER"
  logger -t pbs-uid-fix "Recreated user"
fi

# Reapplying the ACL is harmless if nothing changed
synoacltool -add "$SHARE_PATH" "user:${TARGET_USER}:allow:rwxpdDaARWc--:fd--"

It has since been through a real firmware update and reboot. I did not take the scheduler's word for it either. I forced a backup afterward and watched it finish clean. A green status only proves the share mounts. Writing a chunk is what proves the identity mapping is correct.

A single sign-on for every service in the lab. Authentik is the identity provider, and each app authenticates against it instead of keeping its own separate username and password.

Too many logins

By the time I had a dozen services running, every one of them had its own separate login. That means reused or weak passwords, no central way to revoke access when something needs shutting off, and no consistent multi-factor. Putting everything behind one identity provider means one strong login, one place to turn access off, and far fewer app-level passwords sitting around waiting to leak.

The two patterns

Services connect using one of two patterns, depending on what they support. Apps that speak OIDC redirect to Authentik, receive a token, and provision the user automatically. Apps that do not support OIDC sit behind forward authentication at the reverse proxy: every request is checked against an Authentik outpost before the proxy will pass it through, and the app's own built-in login is disabled entirely so it cannot be reached around the SSO.

The two hostnames

Forward auth turns on a distinction that is easy to miss. The reverse proxy reaches the Authentik outpost by its internal service name, while the user's browser has to be sent to the public hostname. Two different addresses for the same system, and both look correct in isolation. Getting those two host settings right is what makes proxied login work.

A four-node Proxmox VE cluster with Ceph distributed storage. Every virtual machine and container lives on shared storage, so any node can run any workload, and a node can go down without taking services with it.

Why more than one box

I started on a single machine, which meant everything I had built on it was one hardware failure away from gone. I also wanted to learn high availability and shared storage the way they work in production, not as a diagram. Ceph pools the disks from every node into one self-healing storage layer, so there is no single NAS sitting in the middle as both the bottleneck and the thing that takes everything down if it dies.

How the storage works

Four nodes join one cluster and hold quorum together. Ceph runs a monitor and manager plus an object storage daemon across the nodes, pooling their disks into a single resilient pool of just over a terabyte. VMs and containers are provisioned on that pool, which means they can migrate between nodes instead of being trapped on one box. The per-node operating system disks only hold ISO images and container templates, nothing that matters if a node is lost.

The migration that failed

When I first set up Proxmox I did not know how storage was going to work, and I did not have four nodes yet, so I was using the local SSDs in each machine just to have somewhere to put things. It ran fine.

Then I tried to move a VM to another node while it was running, and it failed. That is when it clicked that the storage layout was the problem. Anything sitting on a local disk is stuck on the box it is sitting in, and nothing is protecting it either.

So I waited until I had at least three nodes, which is what a cluster needs to hold quorum, and pooled the disks into Ceph across all of them. Migration became fluid after that, and the storage is available to every node instead of to one.

A security information and event management platform watching the entire lab. Agents run on every node and container, and the edge firewall ships its logs in too, so security events across the environment land in one place.

What uptime monitoring misses

I already had monitoring that tells me a host is up. It tells me nothing about who tried to log into it. I wanted a single pane for the things a security team actually watches: authentication failures, file-integrity changes, and firewall events, and somewhere to practice detection and threat hunting rather than just reading about them.

What it watches

Wazuh runs as an all-in-one install, manager and indexer and dashboard together, and is never exposed to the internet. Agents run on all four cluster nodes and every container. The edge firewall forwards its logs over syslog, and Wazuh decodes the firewall log format automatically, so firewall events are searchable alongside everything else by source.

The listener the installer does not create

The installer does not create the agent-communication listener. It has to be added as a second listener, separate from the syslog one, and until it exists the agents fail quietly: nothing errors, they just never appear. Everything looks deployed and nothing is reporting, which is the worst way for something to be broken.

Full logging stays off. It writes everything rather than the events you are actually looking for, and the disk goes fast. Getting the dashboard to surface the alerts I wanted took some tuning, and the firewall block alerts still fire with full logging off, which is the part that mattered.

Services are reachable from the internet without opening a single inbound port on the home network. A tunnel reaches out from inside the lab, a reverse proxy routes each request, and admin-only tools stay on a private mesh that is never public at all.

Why nothing is port-forwarded

The normal way to reach something at home from outside is to forward a port on the router, and I did not want to do that. Every forwarded port is another opening that has to stay secured indefinitely. A tunnel works the other way around: the connection is made outbound from inside the lab, so there is no exposed port and no public address to attack, with a major provider sitting in front for TLS and filtering. And some tools should simply never be public, so they are not.

How traffic gets in

A Cloudflare Tunnel runs inside the lab and makes an outbound connection to Cloudflare. Public subdomains are DNS records pointing at that tunnel. Every public request lands on a reverse proxy that terminates TLS with Let's Encrypt certificates and routes to the correct internal service. Anything that does not need to be public, the metrics server, the SIEM, the backup server, and the proxy's own admin panel, is not tunneled at all. Those are reachable only over a Tailscale mesh. A network-wide DNS filter handles name resolution for the whole lab.

Keeping the surface small

The discipline here is keeping the public surface as small as it can possibly be. Only the handful of services that genuinely need to be public get a tunnel route. Everything else is mesh-only. That has a nice second effect: someone scanning my home address finds nothing to hit, because there is nothing listening.

Prometheus scrapes metrics from every node and Grafana turns them into dashboards. Real-time and historical visibility into cluster health, resource use, and trends.

Why I wanted history

A host sitting at 90% memory right now is a different problem depending on whether it has been climbing all week or spiked ten minutes ago, and without history there is no way to tell which one you are looking at. I wanted to be able to catch a trend before it turns into an outage, plan capacity with data instead of guesses, and troubleshoot by looking at what actually happened.

The two pieces

Prometheus scrapes node-exporter metrics from the lab hosts and is kept internal, never public. Grafana reads Prometheus as its data source and renders the dashboards, and it logs in through the same single sign-on as everything else, so the one public pane is still behind the central login.

The split that matters

The split is the decision worth explaining. Prometheus holds the raw metrics and every scrape target in the lab, which makes it exactly the kind of thing that should not be internet-facing, so it stays mesh-only. Grafana is the single public-facing dashboard and is gated behind the same SSO as everything else. That keeps the sensitive half private without giving up the convenience of checking the dashboards from anywhere.