Ben Boyter, an Australian developer behind SearchCode.com, has announced the migration of the database from MySQL and Redis to SQLite on the large-scale website that serves a significant number of users. The SearchCode database, boasting a massive size of 6TB, is perhaps the largest SQLite database for a public website in the world.
Having been in operation for over a decade, SearchCode initially utilized a stack comprised of PHP, CodeIgniter, MySQL, Memcached, Apache, and Sphinx before undergoing multiple transformations. Transitioning through various iterations, the platform eventually shifted to Python and during the Covid era embraced the Go language, along with revamping the self-authored search engine. Consequently, the current tech stack stands at Go, MySQL, Redis, and Caddy.
Boyter has expressed his intention to streamline the application to a single binary without any external dependencies. The switch to SQLite aligns perfectly with this objective, despite prior considerations of using the bbolt key-value database in the past. Although the SearchCode database appears straightforward, Boyter remains insistent on retaining SQL functionality.
One of the key challenges encountered by Boyter with SQLite is the dreaded “database is locked” issue, which can be resolved by segregating connections between read and write operations. At present, with the recent migration completed just a few days ago, no major issues have been reported.
TLDR: Ben Boyter, creator of the web platform SearchCode.com, successfully migrated the database to SQLite from MySQL and Redis, overcoming challenges such as locking issues and maintaining SQL functionality. The transition marks a significant shift in the website’s tech stack and operational efficiency.
Leave a Comment