Simon is a Computer Science BSc graduate who has been writing about technology since 2014, and using Windows machines since 3.1. After working for an indie game studio and acting as the family's go-to technician for all computer issues, he found his passion for writing and decided to use his skill set to write about all things tech.
Since beginning his writing career, he has written for many different publications such as WorldStart, Listverse, and MakeTechEasier. However, after finding his home at MakeUseOf in February 2019, he would eventually move on to its sister site, XDA, to bring the latest and greatest in Windows, Linux, and DIY electronics.
When I first started using Fedora, I wanted to grab all the third-party software for my OS from RPM Fusion. When I visited the website, I was greeted not with the RPM Fusion main page, but with what looked like an anime girl in a detective getup looking pensively at me. Before I could comprehend what I was looking at, the anime girl pointed me forward, and the website loaded.
I thought it was just a one-off thing, but the more Linux websites I visited, the more I saw the anime girl appear before I could access the page. As it turns out, my computer was being given math tests, and it's a huge reason why some volunteer projects can stay online.
The anime girl has a name: Anubis
Yes, like that Anubis you're thinking of
Before we explore what exactly was going on with those requests, let's explore how this anime girl came to be in the first place. It started when developer Xe Iaso was getting pretty annoyed with Amazon's web crawlers. Iaso had set up a 'robots.txt' file to tell crawlers not to peek into her Git server, but Amazon's ones were supposedly not reading it, or not caring.
The bots were causing the server to have some nasty performance issues, and they managed to skirt around anything she deployed to stop them. As such, she decided to invent and deploy a system similar to those "Are you a bot?" messages you see from Cloudflare, but this one would be open-source for the community to use. And thus, Anubis was born.
Anubis didn't have an anime mascot to begin with; that would come later, when artist Celphase designed her and rendered three poses for her to use. There's the "thinking" face when Anubis is presenting a proof-of-work challenge, a "success" pose that shows her pointing forward, and a "block" pose with a stop sign. Because Anubis is open-source, you'll often see this character pop up when browsing FOSS-focused websites to give your PC a quick challenge.
So, why is she called Anubis? Well, she's named after the Egyptian god of the dead who would weigh souls and deem those who are worthy of entry to the afterlife. Similarly, Anubis "Weigh[s] the soul of incoming HTTP requests," according to its website.
Anubis' job is to provide proof-of-work challenges to incoming connections
It's easy for your PC, but requires an insane amount of effort for scrapers
So now we know what (or rather, who) Anubis is, and why she was made, but what is she even doing? Why does she pop up when you access a site, and what's the loading bar for?
Well, when you see Anubis pop up on your PC, she's giving your PC a simple mathematics problem to solve. Your PC takes this challenge, solves the problem, and hands it back to her. The math problem isn't anything huge; it should take a few seconds, if that. Once your PC turns in its math homework, Anubis grants your computer access to the website she's protecting.
This is meant to deter web scrapers from accessing the website. See, web scrapers aren't designed to be huge on processing power; they're meant to zip across the internet, harvest data, and send it back as quickly and cheaply as possible. This strategy allows companies that deploy scrapers to invest in quantity over quality; instead of one bot that slowly trawls the internet, it has, say, a million bots that do very light work to keep the cost of operations down.
When you encounter Anubis and your computer performs her math problem, it's pretty negligible in terms of computing power, time, and cost. However, when one million crawlers get stopped by Anubis, that added 'weight' on what should be an extremely lightweight mission begins to add up.
Think of it like this: you're driving down the road, and you come across a bridge. There's a barrier before you cross the bridge, and you have to pay $1 to cross. For you, it barely affects your finances; you pay the dollar and drive over. However, if a company tried to send a million cars over the bridge over the span of a day as part of its business, they would have to fork out $1,000,000 every single day, which isn't ideal.
What would happen in this situation? Well, individuals will continue to use the bridge because, to them, it's only $1 to cross. Meanwhile, companies would steer clear of the bridge entirely. Same with Anubis; its checks allow individuals to 'cross' while scrapers for companies and LLMs avoid it to prevent racking up huge costs.
Not everyone loves Anubis' mascot, though
Personally, I think everyone should use it
While Anubis is very useful, not everyone wants to have an anime girl appear when people visit their website. For example, Duke University was very interested in using Anubis to prevent scrapers from accessing DukeSpace, where it keeps its publications. However, they didn't want academics to see an anime mascot every time they accessed the digital archive.
To solve this, Duke University made a deal with Xe Iaso that it would use Anubis, but Iaso would apply custom branding to better suit the institute. Now, when you visit DukeSpace, you see the familiar Anubis UI, but the University's own mascot replaces the anime girl. It's more professional-looking, for sure, but I'd be lying if I said I didn't miss the original Anubis.