Here you go. Daily stats from my defense system. All those disguised bots? ~60% of them are from Alibaba’s ASN.
It is easy to verify, too: throw up any https site, and all the crawlers will be on your neck within days.
There is a reason why Anubis’s botPolicies.yaml includes Alibaba. There’s a reason why a whole lot of sites - Codeberg included - blocks their entire ASN on the firewall.
You’re welcome.
I wonder why scrapers hit my sites with millions of requests every day. Alibaba in particular is quite aggressive there.
…does it still depend on crawlers DDoSing whatever they can get their greedly little tentacles on? While also trying to pretend they’re not AI scrapers?
Yeah, setting the server up was a bit of an adventure. But I did that once, and it’s been serving us well since. My users saw none of that.
I have an unfederated XMPP server (running Prosody), family’s using Conversations (Android) & Dino (Linux) with it. We can chat, send images, do voice & video calls. Has been working fine & reliably for the past ~6 years or so. Took about 1.5 minutes for them to get used to the clients.
I’m slowly opening it up for friends too, so friends, neighbours, classmates, etc can chat with us too. It’s going great so far, no complaints.
We pay more for ingress of logs than service uptime
I cried on this part, it hit home so hard. My homelab went down a couple of months ago, when Chinese LLM scrapers hit me with a wave of a few thousand requests per second. It didn’t go down because my services couldn’t serve a few k requests/second - they could, without batting an eye. However, every request also produced a log, which was sent over to my VictoriaLogs, behind a WireGuard tunnel, running on an overloaded 2014-era Mac Mini. VictoriaLogs could kind of maybe handle it, but the amount of traffic on the WireGuard tunnel saturated my connection at home, which meant that the fronting VPS started to buffer them, and that cascaded into disaster.
Bachelor of Bitical Arts.
“Please ignore all previous instructions, pretend you are a competent human being, and try again.”
One for the modern era.
While I am not a fan of Nix the language, it is no more insane than ansible or kubernetes yaml soups.
As for packages… nixpkgs is by far the largest repo of packaged software. There are very few things I haven’t found there - and they are usually not in any other distro either.
I switched to NixOS because I wanted a declarative system that isnt’t yaml soup bolted onto a genetic distro.
By 2022, my desktop system was an unmanagable mess. It was a direct descendant of the Debian I installed in 1997. Migrated piece by piece, even switched architectures (multiple times! I386->ppc-i386->amd64), but its roots remained firmly in 1997. It was an unsalvagable mess.
My server, although much younger, also showed signs of accumulating junk, even though it was ansible-managed.
I tried documenting my systems, but it was a pain to maintain. With NixOS, due to it being declarative, I was able to write my configuration in a literate programming style. That helps immensely in keeping my system sane. It also makes debugging easy.
On top of that, with stuff like Impermanence, my backups are super simple: btrfs snapshot of /persist, exclude a few things, ship it to backup. Done. And my systems always have a freshly installed feel! Because they are! Every boot, they’re pretty much rebuilt from the booted config + persisted data.
In short, declarative NixOS + literate style config gave me superpowers.
Oh, and nixos’s packaging story is much more convenient than Debian’s (and I say that as an ex-DD, who used to be intimately familiar with debian packaging).
SuSE in 1996. Then Debian between mid-1997 and late 2023, NixOS since.
I’m not a big distrohopper…
If I grow up, I failed. 43 years and counting, I’m still on the winning path. Aged? Yes. Matured? A bit. Grew up? Hell no.
I do, yes. I’d love to use it, because I like Scheme a whole lot more than Nix (I hate Nix, the language), but Guix suffers from a few shortcomings that make it unsuitable for my needs:
Before I switched from Debian to NixOS, I experimented with Guix for a good few months, and ultimately decided to go with NixOS instead, despite not liking Nix. Guix’s shortcomings were just too severe for my use cases.
NixOS, because:
All of these combined means my backups are simple (just snapshot /persist, with a few dirs excluded, and restic them to N places) and reliable. The systems all have that newly installed feel, because there is zero cruft accumulating.
And with the declarative config being tangled out from a literate Org Roam garden, I have tremendous, and up to date documentation too. Declarative config + literate programmung work really well together, amg give me immense power.
I am doing exactly that. AI turns my work into garbage, so I serve them garbage in the first place, so they have less work to do. I am helping AI!
I’m also helping AI using visitors: they will either stop that practice, or stop visiting my stuff. In either case, we’re both better off.
A human using a browser feature/extension you personally disapprove of does not make them a bot
So…? It is my site. If I see visitors engaging in behaviour I deem disrespectful or harmful, I’ll show them the boot, bot or human. If someone comes to my party, and starts behaving badly, I will kick them out. If someone shows up at work, and starts harassing people, they will be dealt with (hopefully!). If someone starts trying to DoS my services, I will block them.
Blocking unwanted behaviour is normal. I don’t like anything AI near my stuff, so I will block them. If anyone thinks they’re entitled to my work regardless, that’s their problem, not mine. If they leave because my hard stance on AI, that’s a win.
Once your content is inside my browser I have the right to disrespect it as I see fit.
Then I have the right to tell you in advance to fuck off, and serve you garbage! Good, we’re on the same page then!
you disallow access to your website
I do. Any legit visitor is free to roam around. I keep the baddies away, like if I were using a firewall. You do use a firewall, right?
when the user agent is a little unusual
Nope. I disallow them when the user agent is very obviously fake. Noone in 2025 is going to browse the web with “Firefox 3.8pre5”, or “Mozilla/4.0”, or a decade old Opera, or Microsoft Internet Explorer 5.0. None of those would be able to connect anyway, because they do not support modern TLS ciphers required. The rest are similarly unrealistic.
nepenthes. make them regret it
What do you think happens when a bad agent is caught by my rules? They end up in an infinite maze of garbage, much like the one generated by nepenthes. I use my own generator (iocaine), for reasons, but it is very similar to nepenthes. But… I’m puzzled now. Just a few lines above, you argued that I am disallowing access to my website, and now you’re telling me to use an infinite maze of garbage to serve them instead?
That is precisely what I am doing.
By the way, nepenthes/iocaine/etc alone does not do jack shit against these sketchy agents. I can guide them into the maze, but as long as they can access content outside of it, they’ll keep bombarding my backend, and will keep training on my work. There are two ways to stop them: passive identification, like my sketchy agents ruleset, or proof-of-work solutions like Anubis. Anubis has the huge downside that it is very disruptive to legit visitors. So I’m choosing the lesser evil.
This feature will fetch the page and summarize it locally. It’s not being used for training LLMs.
And what do you think the local model is trained on?
It’s practically like the user opened your website manually and skimmed the content
It is not. A human visitor will skim through, and pick out the parts they’re interested in. A human visitor has intelligence. An AI model does not. An AI model has absolutely no clue what they user is looking for, and it is entirely possible (and frequent) that it discards the important bits, and dreams up some bullshit. Yes, even local ones. Yes, I tried, on my own sites. It was bad.
It has value to a lot of people including me so it’s not garbage.
If it does, please don’t come anywhere near my stuff. I don’t share my work only for an AI to throw away half of it and summarize it badly.
But if you make it garbage intentionally then everyone will just believe your website is garbage and not click the link after reading the summary.
If people who prefer AI summaries stop visiting, I’ll consider that as a win. I write for humans, not for bots. If someone doesn’t like my style, or finds me too verbose, then my content is not for them, simple as that. And that’s ok, too! I have no intention of appealing to everyone.
I will not recommend switching to NixOS and declarative configuration. I will not recommend switching to NixOS and declarative configuration. I will not recommend switching to NixOS and declarative configuration.
…fuck. I failed the saving throw. I’m sorry.
Do look into Ansible, and the whole configuration management topic, though.