Mistakes and lessons from building Sqids (formerly Hashids)

1. What is it
2. Brief history
3. Upgrade & rebrand
4. Usage & statistics
5. Mistakes
6. Lessons
7. What’s next

What is it

Sqids is a small open-source library that generates unique IDs from numbers.

Usually, people use them to convert their database primary keys to IDs. For example:

/video?id=123      # instead of this
/video?id=86Rf07   # you can have this

There’s plenty more to explore about the project , but this article is about the behind-the-scenes of Hashids/Sqids (along with all the mistakes and bloops that have happened).

Brief history

I came to San Francisco in 2009, and a few years and startups later, I was still a young engineer figuring out my way around the fast-growing Internet industry.

During those days of working for different companies, I noticed several times when I needed to create more or less the same custom code to generate IDs, whether it was for invite codes or page navigation.

I got tired of writing it over and over again, and what’s worse was there was no open-source solution that did the same thing for the backend and frontend (usually just a single library that did approximately what I needed, and written in one language only).

So, I started spending my nights and weekends in my tiny studio apartment, hacking away at some PHP code, wondering if I could come up with something that could encode one or several numbers into a single (& ideally tiny) ID.

If I remember correctly, I wanted a few basic things:

• Support for custom alphabet (easy)
• Support for unique IDs, so that not everybody would generate the same output (hard)
• Support for padded IDs, so they appear longer if needed (who knows?)

It took me a few weeks and in April 2012, I had a basic PHP version. In August, I added a JavaScript port.

And that was pretty much it. It was out on GitHub, and being a proper developer - I did absolutely no marketing 💪

I just went on my way, working at my day job / contracting as usual.

But then later, a funny thing happened. A developer pinged me that they’ve converted it to Go (back then a new and exciting language).

I thought that was kinda cool.

A few months later, another developer pinged me that they’d ported it to another language. And then another. And before you know it, we had a few versions scattered across GitHub.

That’s when I made a website.

Yes, it was very green.

No, I had zero reasons for making it this green (I guess I just wanted something different).

Of course, throughout the years, the website has changed its look a few times and grown its support for more languages.

Some projects and companies started using it, and it became a thing.

Then a few years went by… 🕷️🕸️

The project was sitting somewhat idle.

There was not much to do from the development perspective.

A library algorithm like this is not exactly meant to be changed, because then the generated IDs would change. So, the project sat there mostly in maintenance mode.

I even offloaded my PHP and JavaScript versions to 2 other developers (who did an amazing job taking care of them and growing them further).

I moved on with my life and worked on other things.

Upgrade & rebrand

But there was something that kept bothering me about it throughout the years…

Unfinished work. Unaddressed suggestions by others.

And I badly wanted to change a few things myself, like:

• Renaming some stuff to lose association with encryption
• Unifying the code under one roof
• Updating the clunky and outdated website
• Having a proper playground to show examples
• And maybe changing the name 👀

After a long time, I realized that there were quite a few people and companies that were actively using it. I finally decided to do it.

After discussing the proposed changes with the community, I started working on them.

The refresh was received well, and the project was renamed to Sqids.

There’s not a big reason why this name was chosen. I think of it as:

• Short Quick IDs, or
• Short Quality IDs

Sqids.org was also available, so that helped too.

Usage & statistics

How can I write this post without doing some numbers? 😄

I’ve been fortunate enough to hear kind words from different projects and companies that use the library (whether it’s pre or post-upgrade). Here are some cool ones:

• Codepen used them for page URLs (not sure if they still do)
• The latest versions of ClickHouse are shipped with Sqids built-in
• Apache Superset uses them for permalinks

I’ve heard of big companies that use the library in some capacity, but I’d rather not single them out (in case they don’t want the publicity).

If you’d like a shoutout to your company or project, ping me and I’ll see if I can link it here or on the Sqids website.

Downloads

The following statistics are for both versions (Hashids and Sqids). From what I can see:

Python: 949k downloads per month

JavaScript: 888k downloads per month

PHP: 698k downloads per month

Ruby: 20M downloads total

.NET: 8.7M downloads total

Rust: 16k downloads total

Of course, these numbers have been growing since ~2013. They are taken from publicly available stats, and again for both versions: Hashids & Sqids.

Traffic

Besides the description of the project, the website contains mainly links to the repositories, the FAQ page, and a playground. Monthly visits go up and down, but generally, they’re slightly under 10k.

Here’s what a traffic bump from Hacker News looks like 👋

Here’s a list of the countries that visit the most:

I’m not sure how much this order is affected by the site’s i18n. Google does not seem to index all the pages of the 21 languages the site has been translated into. I can see that it can index the same page in one language, but not the other 🤷‍♂️

Ports

Today, there are 45 implementations of the library (in Hashids or Sqids form). There are more than 100 contributors that have added code and specification suggestions, along with continued support and maintenance of the project.

The popularity of implementations can be seen here when organized by GitHub stars. As of right now, the top 5 spots go to:

1. .NET
2. JavaScript
3. Go
4. Python
5. PHP

Here are some lesser-known ones that I find interesting:

• GDScript , for Godot game engine
• Zig , seems to be the favorite among many
• R , programming language for statistics
• PL/pgSQL , for PostgreSQL
• Starlark , configuration language for Bazel

Let me know if you can think of more statistics I could show, and I’ll see if I can dig something up.

Mistakes

From the beginning, I had no grand vision of any kind. I just wanted to create a simple library for myself, and I was curious if I could do it.

But since the project became useful to other people and companies, it has evolved to address different use-cases.

Here are some of the mistakes I’ve made over its lifetime:

🤔 Not naming things properly from the start

I am surprised by how solidified the name Hashids has become. To this day I see people talking about it and recommending it by using its older name. I guess I’m trying to say: thanks for all the love 😅

Initially, when it wasn’t a thing yet, some pointed out that since IDs are not hashes (because hashes are irreversible), it wasn’t appropriate to have the word “hash” in Hashids.

I agree.

Initially, I was focused on the IDs that YouTube was using in the URL, and to me, they looked like short hashes, so that’s why I named it like that.

But, lesson learned. Naming things is hard. Changing the name is also hard (but not impossible). If you can - try to do it right from the start.

🔐 Association with encryption

When I was first messing around with the code (pre-release), I named the functions “encrypt” and “decrypt”, as opposed to “encode” and “decode”.

And then forgot to rename them. And even more so, I added a configuration parameter that I called “salt” only because I couldn’t come up with a better name for it.

All of this caught the eyes of encryption and security folks, who raised a fuss about it. Some even did a cryptographic analysis on it.

To me, it was always a bit silly, since this tiny library is basically a decimal-to-hex converter (with a few bells and whistles). You wouldn’t try to “break” a hex converter, would you? (there’s nothing to break)

Nevertheless, I get it. I should’ve used different terms and named things correctly from the start.

* Now, the website even makes an extra point to show that this is not an encryption library, and there’s nothing to hack.

🧭 Not setting direction

It was a single library under one repo.

Then more people were making adjustments. Adding or removing stuff. Sometimes a minor code change in one language would produce different IDs. They’d all be scattered across different GitHub accounts.

All this decentralization was useful for growth but difficult for uniformity.

Now, of course, there’s a spec. And dedicated tests. And specific functions to follow. And more or less identical READMEs. And a guide on how to port to other languages.

Everything is more organized and unified. And if somebody’s confused, maintainers seem to be addressing issues promptly.

I wish I would’ve organized this better from the start.

Lessons

🥷 Software engineers are amazing

I was not surprised that there are great engineers out there. I was surprised how quickly help came when I needed it 😅

The project was built by the work of amazing open-source developers. From bug fixes, to code cleanup, to major restructuring, to whole new implementations. Everything was done by volunteers, and I’m forever inspired by people coming together to build something.

I’ve learned to welcome people by providing more resources and wasting as little of their time as possible.

Currently, this project:

• Mentions how to contribute on the org level and the website.
• When somebody implements a new library, I will give them maintainer access to the repo so they’re not blocked by me.
• When somebody does pull requests, I try to respond promptly to show them appreciation.

People’s time and work = Access to the project.

⏳ Give your project/product plenty of time to grow

It’s been more than 10 years and this project is still growing.

You know the drill: it’s a slow progress, occasionally followed by a successful uptick from some random source, and then back to slow progress.

Don’t underestimate the benefits of time on some of the products you love and use these days.

😓 Listen to the pain points

Often, users will bring up something that doesn’t work well.

They might even offer solutions to fix it.

I feel like the pain points are important to pay attention to, but the solution might not always be the best idea mainly because you know the product and direction better than the users’ limited points of view.

I’ve had that happen here too.

An engineer brought up a valid quirk of the library, along with a proposed solution to fix only that particular issue. However, the fix had major downsides, and it was hard for the engineer to see them (because it got rid of their original problem). Long story short: the quirk has been addressed but in another way.

👾 Don’t forget to have fun

Sometimes, it’s not all about the sales, website traffic, or app visitors. Sometimes, you just gotta make a library cause it’s cool.

🔕 Ignore the noise

If I would’ve listened to the haters, I would’ve deleted the project when:

• A jealous co-worker accused me of building this during work time (even tho they had no idea when I worked on this)
• Security experts were calling me names because I used some security terms
• Random people would say why use this when you can use X, Y, and Z

And the list goes on. Just move on; it’s all good.

What’s next

If you’ve gotten that far, thanks for reading 😂

What’s next for the project is the same thing as usual:

1. Support for existing libraries via GitHub
2. Welcoming new implementations in different programming languages
3. Suggestions on what to improve and how to improve it

The specification is solidified, and the project is growing.

Thanks for all the love.

Onwards and upwards! 🦑