If I Do My Job Well, You Can't Tell I'm Here
A Tale of an Erowid Sysadmin
v1.1 Nov 2015 (v1.0 May 2015)
Originally published in Erowid Extracts #27
Citation: Erowid. "If I Do My Job Well, You Can't Tell I'm Here: A Tale of an Erowid Sysadmin". Erowid Extracts. May 2015;27:4-5. (v1.1) Online edition: Erowid.org/general/about/about_article16.shtml
Working for Erowid is a unique experience. There is no office and there are no core hours — we coordinate via email and chat when working hours overlap. This means I can do what a lot of people dream about: make the whole world my office. I am slowly moving around the globe with just a backpack full of technology and a single bag of clothes.
My original home base is in Germany. As I write this I am in the Dominican Republic. When these lines went to print, I was on Curaçao. When subscribers received the May 2015 issue of Erowid Extracts, I logged in from Colombia. During November 2015, as this story went live on the site, I'm self-stationed in Colonia del Sacramento, Uruguay. But for most of 2014, I lived in a small fishing village in the south of Morocco. Every few weeks I rented a car to explore the country. I've had tea with Sahrawi fishermen; attended couscous parties at friends' houses (the big family lunch that is taken after prayer on Fridays in Islam); given hitchhiking cops a lift to their checkpoint in Western Sahara; visited the "forbidden city" of Smara; and spent countless hours learning traditional healing methods from a Berber herbalist in Fez. All without ever leaving my office.
Public-facing site? Check. Backend machines 1 through 3? Check. The 12 virtual machines? Check.
My typical day starts with a quadruple espresso, a hello to headquarters via IRC, a quick check of the servers and multiple monitoring services to make sure there are no catastrophes, and then an evaluation of the systems' health. Public-facing site? Check. Backend machines 1 through 3? Check. The 12 virtual machines? Check. After reading 40+ emails of detailed system status information, I'll have likely cursed at least once, usually about some new security update that needs to be done "right now".
A constant stream of robots accesses Erowid, from the friendly Googlebot (checking for updated content), to the out-of-control scraper spiders (run by the student who wants to grab a "copy" of Erowid without realizing their bot can't handle it), to the hostile attackers (distributed denial of service attacks, or DDoS). It's important to keep a tab on heavy-hitting traffic to make sure it doesn't cause Erowid.org to become slow or unresponsive. A single minute of downtime means seventy people can't access the information they were looking for. Existing at the same domain name for twenty years means that our servers are subject to constant attack from around the world. Few appear to be targeted at Erowid specifically, and instead are just attacks against any popular web service.
By the time this daily routine is complete, somewhere between thirty minutes and three hours have passed, and then I have an actual breakfast.
Next, it's on to the more complex projects, all the while keeping an eye on the monitoring systems that dutifully inform us of problems in real time.
Some of these bigger sysadmin tasks of the last twelve months have included:
- Migrating all our backend installations (such as our email management and site statistics systems) into "jails", which is FreeBSD-lingua for "virtual machines". One can think of them as private cloud setups — private so that we don't hand data over to third parties.
- Moving all WordPress instances (the software used for the Erowid Review and columns like Teatime) into a dedicated virtual machine so that the inevitable next WordPress exploit will not endanger data on Erowid.org itself. WordPress is notorious for security breaches that allow malicious software to be run on the exploited machine.
- Updating Erowid.org to the current version of Apache, the webserver we use. The webserver is the main software that delivers pages to readers. This might not sound like much of a task, but Erowid's complex structure made it a big deal.
- Establishing better monitoring for Erowid's systems, which has served us well when dealing with performance degradations or a recent (small) DDoS attack.
One longer-term Very Important Project I've been working on when I have time is developing a separate website to empower volunteers to work on Erowid's code and allow visitors to prioritize improvements to the site by vote. It's 90% done and I am positive we can launch it publicly later this year. Keeping Erowid.org accessible and secure is my number one priority, so this project, while important, has repeatedly been put on the back burner. Over the past year, we've had to deal with:
- about 200 software security updates
- two machines that were unresponsive because one of our hosting providers screwed up
- one hard disk failure
- lots of script kiddies trying their automated exploit tools on us (here's a hint: stop looking for win.ini — this file does not exist on a FreeBSD server)
- innumerable misconfigured scrapers that hit Erowid.org with way too many requests in way too short intervals, thereby overloading the server.