[CASE STUDY]
A lesson in scaling: educating millions of students during a pandemic
The education industry is slow when it comes to tech adoption despite the fact that they spend around $700B a year on software purchases. In fact, 60-70% of software goes unused in schools. In order to help close this gap, Kami’s platform allows teachers to digitally display documents on the student’s laptops or Chromebooks instead of printing individual handouts. Teachers and students can also take notes directly on the platform.
Even before the pandemic, Kami’s business was doing well and there was clearly a need for this type of digital alternative to printing in schools. But in 2020 when the pandemic hit, schools closed and students were forced to learn from home. The Kami team was glad to see their platform was able to keep students and teachers connected… until they saw what it meant to scale their Postgres database in the face of exponential growth.
At the peak of the pandemic, Kami was growing at a rate of over 1M users per week, and their Postgres database simply couldn’t keep up. They doubled their database size only to find that it bought them two weeks of runway. They spent sleepless nights removing performance bottlenecks that earned them mere days of uptime.
USERS
30 M
GROWTH
20 x
COUNTRIES
180 +
MONTH MIGRATION
4
Forecasting growth & a storm ahead
Kami has a lean engineering team of seven members, and they originally built their platform on Postgres. That served them well for many years, and they were able to scale from zero users to several million. However, when businesses like Kami start to see tremendous growth, Postgres begins to hit limitations.
The team made a few attempts to scale and optimize the database manually. They started with upsizing their PostgreSQL primary to 160vcpu, 4TB RAM, which was the biggest server they could get. However, the write-heavy data from user-driven content was becoming too much to handle. They scaled up write capacity, but they couldn’t scale horizontally anymore. There were also operational issues with implementing transaction ID wraparounds. All the written data needed to be vacuumed after 2 billion transactions. But they were pushing close to a billion transactions every day!
Ability to scale horizontally and add capacity over time as needed
Accommodate write-heavy workloads
Compatible with Postgres so the transition would be fast
The team considered a number of different databases, but they didn’t have time to run performance tests. They looked at Yugabyte but felt there were a few aspects where they lacked in maturity (especially schema changes). They also looked at Citus which is a Postgres extension designed to help you scale out. However, Citus had some limitations around the write traffic (using a single node to coordinate writes) and could not scale horizontally for writes.
Luckily, someone on the team remembered reading about a distributed SQL database named CockroachDB on Hacker News a few years ago, and decided to investigate further.
"We knew how complicated it would have been to stay with Postgres and set up sharding. There would have been a constant drag of managing multiple shards of data. No one on our team wanted to go through that manual labor. And even if we did set up sharding, we weren’t going to be able to grow our business 10x."
-Jordan Thomsco-founder, CTO
Trusting a famously resilient creature: CockroachDB
Usage and sales were doubling month over month and the database was facing immense pressure already. Co-founder and Chief Technology Officer Jordan Thoms was ready to start evaluating CockroachDB when fellow Co-founder and Chief Revenue Officer Bob Drummond reminded him that he needed a solution fast. He had early insight into sales, and knew that Kami was expecting 10x load growth within the next two months.
Jordan reached out to Cockroach Labs in May 2020 and kicked off a brief evaluation. The Kami team liked that they could continue to use SQL and that CockroachDB offers the ability to partition data by location so they can easily cater to a global audience and scale their environment across regions. Additionally, they were impressed with CockroachDB’s performance when it came to managing a dataset that is continuously being updated.
"After switching to CockroachDB, our team is much more productive and we didn’t have to recruit a bunch of Postgres DBAs. We are really pleased with its performance and our ability to massively scale. It’s the winning technology for the future."
-Jordan Thomsco-founder, CTO
Executing a live migration
A “traditional” database migration strategy would include setting up a production system to write to the new database and the old database, and read only from the old one. In the background, you migrate previous data from the old database to the new one, and ultimately switch reads over to the new database and turn writes off to the old database. If you have the time, it’s smart to limit the effects of a migration to a small set of users at first before gradually rolling out to a larger audience.
But this was not a traditional situation, and Kami simply did not have the time.
Instead they needed to do a live migration which meant setting up a load and correctness test. Initially all documents were written to Postgres, but they progressively enabled ‘New DB’ (i.e. CockroachDB) for more users over time. Then they started to mirror all writes to the new database (CockroachDB) and old database (Postgres), and they would read from both databases and check that the result matched. Mismatches were collected into a new table and investigated. This approach allowed them to gradually gain confidence in the performance and correctness of CockroachDB with real-world usage.
The migration started in early June and concluded in mid-September, right in time for the beginning of the 2020 school year. The team was under an incredible amount of pressure, but they we
What’s next for Kami
Because of the pandemic, 5+ years of digital education trends were condensed into about seven months (March - September 2020). Fortunately, Kami was able to complete a fast migration to a powerful new database that made their platform available to over 30 million users across the globe.
Kami’s business grew 20x in the six months after the 2020-2021 school year began. Right now, usage is heaviest in the U.S. (which accounts for over half of the schools using the platform), but Kami is seeing substantial growth in Canada, UK, other European countries, and South Asia.
The Kami team is glad to see the education industry adopting new technologies and hopes it will positively impact teachers and students' lives well into the future.
You can learn more about Kami at their website: www.kamiapp.com.
Ready to get started?
Go hands-on with 100% free CockroachDB Serverless. Spin up your first cluster in just a few clicks.
