IP bans, dev/shm crashes, and teaching Quivya to throw punches
"Two fronts this cycle: getting a card-scraping pipeline to survive contact with a real hosting provider, and giving Quivya's players a way to challenge each other directly."
Two unrelated projects decided to have their crises in the same week, so here's a joint dispatch: the Yu-Gi-Oh price/pop scraping pipeline (the guts behind the TCG Data Platform and PSA-to-eBay projects) finally left my laptop for a real server, and Quivya got its first taste of actual PvP.
The scrapers meet a real server, and the server meets a ban hammer
The whole pipeline — card_db, recent_sales,
ars_scrap, pop_reports, psa_vault — moved
onto a Hetzner box, orchestrated with plain systemd units, Postgres running as a
podman quadlet, and Caddy in front of the FastAPI site. I wrote a
deploy/bootstrap.sh that does the entire setup end to end: system
deps, matched Chrome/chromedriver, both repos cloned side by side, migrations,
a one-shot referential seed, units enabled. Idempotent and fail-fast, so a botched
run leaves nothing to clean up by hand.
Then, within days, eBay and 130point.com both started serving quiet blocks that
never happened from home. Same script, same code, different fate depending on
whose IP asked. Confirmed it wasn't a fluke by running the identical scrape from
both places back to back. The fix ended up two-pronged: a
run_scrapers.py orchestrator to scrape locally and sync results up to
the server, and per-site proxy routing so the ones that must run server-side don't
share a fingerprint.
# recent_sales/.env
EBAY_PROXY_URL=http://residential-proxy:8080
PTS130_PROXY_URL=http://residential-proxy:8081
# separate proxies per site so a burned cookie jar
# doesn't take both sessions down with it
Underneath that, headless Chrome inside the container was crashing on
/dev/shm, fixed by disabling image loading and giving every retry a
fresh incognito driver instead of a shared profile — I tried the shared-profile
"optimization" first, made things worse, and reverted it the same day. On the
product side: manager/editor/viewer roles now gate the quarantine and ingest
queues, the pop-report diff view got unified across sources, sales pages are
paginated, and results can go out via share links instead of handing out server
credentials.
Quivya learns to throw down
Quivya shipped to the App Store, then Android's EAS build promptly refused to
cooperate until I bumped the target SDK to 35 and patched
expo-modules-core to match. Two full security/correctness audit
passes followed — 26 issues, then 22 more on the second sweep — before either
store build felt trustworthy enough to call 1.0.2, then 1.0.3.
The actual feature people asked for: 1v1 friend challenges — a direct invite with a push notification to join, a global banner so a challenge doesn't get buried under a Friends tab nobody checks, and online/in-game status so you know who's actually reachable. New quiz categories landed too, mostly because testers kept asking for them: Naruto, Hunter x Hunter, Jujutsu Kaisen, meme-origin trivia, and — very on brand for a French side project — Préfectures de France.
"The server doesn't get IP-banned for running your code. It gets IP-banned for running your code at scale, predictably, forever — which, in hindsight, is a pretty accurate description of what a server is for."
Both are live and getting used, which is the only metric that matters. Check the Projects page for the write-ups on the TCG Data Platform, the PSA-to-eBay sync, and Quivya itself, or hit the Contact page if any of this sounds like a problem you're currently having.