← home

Building a metal search bar

Metal Archives is like a wiki for heavy metal. I use it for exactly one thing: looking up bands, exclusively on my phone at record shops and shows. The site isn't responsive, so I'd spend years trying to tap a three-pixel search box in the top right corner of the screen.

I finally tried to fix it for myself in the smallest way possible: one giant search bar, usable on my phone, from anywhere, returning a band's subgenre and location. That's it.

It half-failed at first. Metal Archives has no public API and bot detection sophisticated enough to block all my searches. I got a proof of concept working on my home WiFi, then hit a wall. Along the way I learned about proxies, anti-bot systems, serverless functions, and headless browsers.

Then I got lucky! I found an unofficial API and emailed the Metal Archives webmaster to ask if scraping was okay. They said yes, just don't kill the servers. An engineer friend suggested I scrape the data, store it in a local database, and query that directly. I ran a scraper overnight, and woke up to 195,000+ bands in an SQLite database.

I built an API to query it, deployed it on Fly.io, and now it exists!

→ Use it here: metal-search.fly.dev

Read the slightly longer version here.