five years. that’s how long i’ve been running services on my own hardware. mail, git, media, chat, a handful of sites, all of it sitting behind one wan ip and a caddy config i’ve tweaked more times than i can count.
last week someone else was running code on one of those servers.
not a disk failure. not a bad update. not a certificate that expired at 2am. a person, or more likely a bot with a person’s payload, had a shell on my gitea container for about five weeks before i noticed. this is the writeup i wish i’d had before it happened.
the symptom was a memory leak
it started the way boring problems start. gitea kept dying.
the container had 1gb of memory, which had been fine for years. then it wasn’t. it would get oom-killed, systemd would restart it, and an hour or two later it would die again. fifteen times across three days. i bumped memory, restarted, and went back to my evening.
when i finally sat down and looked, the container wasn’t full of gitea. it was full of logs.
gitea logs every http request at info level by default. something was hitting it at roughly 78 requests per second, enumerating issue and pr filters with random label combinations. that produced 4,671 log lines per minute. journald was using 438mb of ram and had grown to 3.1gb of journals on disk. the cgroup filled with page cache, the kernel picked a victim, and the victim was gitea.
so the memory limit wasn’t the problem. it was the smoke alarm i kept silencing.
what was actually running
once i got a shell and started looking at processes, the picture changed fast.
there was a miner. xmrig, pointed at two nicehash pools, happily converting my electricity into someone else’s monero. there was also a botnet loader. both were disguised as kernel threads, named things like ksoftirqd, .kworker523, .scsi_eh106, which is enough to survive a casual ps from someone who isn’t looking closely.
persistence was everywhere:
- the git user’s crontab
- shell rc files
- systemd user units
- global and per-repo
post-receive.dhooks .gitconfig
it had also injected a script tag into gitea’s custom header template, loading javascript from a lookalike domain. anyone who loaded the login page got it.
the entry point was cve-2026-60004, a pre-auth rce in gitea before 1.27.1. the exploit chained the diffpatch/markup endpoint with the internal manager endpoint to write a packObjectsHook into .gitconfig, which turns a git operation into code execution as the service account. no login required. open registration was enabled at the time, which is its own separate gift to the internet.
the timeline was the part that made my stomach drop. code execution as the git user since about october 1.
five weeks.
what five weeks of git access means
the container was unprivileged. no root, no sudo. that constraint is the only reason this stayed a bad week instead of a very bad month, and i will be singing the praises of unprivileged lxcs for a while.
but the git user could read app.ini and the gitea sqlite database. that means smtp credentials, internal and jwt secrets, user password hashes, access tokens, and every repository the service could see. no root, but plenty of keys.
i can’t prove data left the box. i’m not going to pretend i can. the working assumption from here is that everything the git user could read was read, and anything that touched those secrets gets rotated.
there was also the registration angle. when i went through the user table, 199 of the accounts were junk: 144 @example.com scanner signups, a pile of @test.local and @poc.local, throwaway domains, the usual. all inactive, zero repos, all created by bots that found an open signup form. in hindsight those were the free sample. i just didn’t read the label.
cleanup, in the order i did it
first i stopped the bleeding, then i made sure nothing came back.
- removed every persistence mechanism i could find, including the hooks and the
.gitconfigentry - replaced the gitea binary with the official 1.27.1 release and verified the checksum
- rotated
internal_token,secret_key, lfs and oauth jwt secrets - deleted every access token, runner token, repo secret, and session
- changed the admin passwords
- disabled registration, and required sign-in to view anything
- raised container memory, which was no longer the actual fix but bought headroom
- blocked the c2 addresses on both the host and the container with persistent systemd units
- sinkholed the payload domains in my local dns
then i deleted 199 accounts. all of them returned 204. i kept a database backup from before the purge, because deleting users is the kind of thing you want a rollback for even when you’re sure.
then the second fire
with the box patched and clean, the flood was still running. it had never been about gitea’s memory at all.
i pulled the real client addresses out of nginx’s error log, because every layer in front of gitea was reporting the proxy instead of the source. 37,214 distinct public ips, 98.9% of them outside the us, hammering the issue pages at 60 to 80 requests per second.
the top sources by distinct ip:
- spain: 2,138
- canada: 1,870
- brazil: 1,811
- iraq: 1,520
- france: 1,401
- uk: 1,342
- turkey: 1,164
- italy: 965
- saudi arabia: 947
- ecuador: 934
- us: 401 (1.1%)
while i was in there i found the other half of the problem. nginx had 113,879 worker_connections are not enough errors in its log, the default worker_connections of 768, a nofile limit of 1024, and 2,504 sockets sitting in time_wait. the box wasn’t just being scanned. it was dropping legitimate connections during every burst and i never saw it because access logging was off.
the fix i should have had on day one
i wanted this to stay on my own hardware, so no cloudflare, no waf subscription. i put a country allow-list in front of the web ports with nftables on the edge container.
- us and canada only, tcp 80 and 443 only
- 36,183 cidrs from ipdeny’s aggregated zones, 29,721 us plus 6,462 ca
- loaded in 1,500-element chunks, because netlink rejects a 36k-element message with
message too long - a systemd timer refreshes the list daily, and 45 seconds after boot
- the drop rule is installed only after the full set loads, so a refresh never leaves an empty allow list
- the script refuses to apply a zone file with fewer than 500 cidrs, which catches a truncated download or an html error page
- loopback, rfc1918 and my wan ip are always accepted, so hairpin nat still works
- ports 25, 465, 587, imap, ssh and turn are untouched, because mail and ssh need to work from anywhere
canada is the number two source of junk traffic, which is awkward, but it stays on the list. i’d rather keep the collaborators and the travel.
i also raised nginx to worker_connections 4096 with worker_rlimit_nofile 65535.
the measurements after:
- 8,329 packets dropped in the first 30 seconds, counter climbing
- time_wait sockets down from 2,504 to 302
- nginx error log growth: zero bytes in 30 seconds
- every public endpoint still returning 200
- gitea sitting at 159mb, quiet
the noise that had been killing the container for three days was gone in about four seconds of packet filtering.
what i got wrong
the technical fixes are the easy part to write down. the mistakes are the useful part.
- i treated an oom kill as a capacity problem. it was a symptom, and it took me three days to read it properly.
- i left open registration enabled on a public gitea with no captcha. 199 bot accounts was a warning shot and i scrolled past it.
- i ran default nginx connection limits on a box directly exposed to the internet.
- i logged every request at info level and never capped journald.
- i had no cdn or waf in front, so port 443 was the open door.
- i had no alert on repeated oom restarts. i found this because services felt slow, not because anything told me.
what saved me was segmentation: an unprivileged container, a dedicated service account, no root, and a proxmox host that scanned clean for the same indicators. defense in depth sounds like a compliance slide until the day it’s the reason you still have a network.
what’s still open
i’m not going to write “and everything is fixed now,” because that’s not true yet.
- rebuild the lxc from a clean image and restore from a known-good backup. deleting files is not proof of integrity after five weeks of someone else’s shell.
- rotate the smtp provider password, the google oauth client secret, and every credential that lived in a repo.
- reissue access tokens and the actions runner token.
- add the c2 addresses to the router for a lab-wide block, since the host isn’t forwarding.
- decide whether the git host should be public at all, or behind tailscale or cloudflare access.
- the allow-list is ipv4 only. the git host has no aaaa record today, and if i ever add one it walks right around the whole filter.
five years
i’ve been lucky, and i know it. five years of running services, learning in public, breaking my own dns at 11pm, and the first real incident is the one that teaches you what the last five years of posts were missing.
the uncomfortable part isn’t the miner. it’s how ordinary the symptoms were. a container restarting. a log file growing. a service that felt slow. none of it looked like a breach, and all of it was one.
if you’re running your own services, go check three things tonight: whether registration is open on anything public, what your web server’s connection limit is set to, and whether you’d notice if a container restarted fifteen times in three days. the answer to the last one is the one that matters.
thanks for reading. and if you’ve been through one of these, i’d genuinely like to hear what you found that i haven’t. the kettle’s still on.