Gyroscope Maintenance
Behind the scenes from last week’s heart rate graphs
When we first started 4 years ago, very few people had access to their heart rates and the occasional measurements were manually taken or triggered during a workout. Now, hundreds of millions of people have continuous heart rate monitors (from a variety of devices like Apple Watch, Garmin, Fitbit or Oura) that are running day and night—and also sending that data to Gyroscope.
Our users have also grown immensely in the last few years, from just a few early testers to hundreds of thousands of members relying on it to understand their health. In that time, the data most people generate has also increased by orders of magnitude — the first Apple Watch barely checked heart rates every 5 minutes, while now we get continuous sampling, with millions of data points synced when someone first connects, plus all of our own trackers like Places, Mood, Sleep, and more.
Fortunately it has been over a year since we’ve had any major outages or downtime, so any gap in reliability of Gyroscope is quickly noticed by thousands of people. We got a lot of concerned messages during the week, so I want to assure everyone that our infrastructure is more stable than ever and syncing will continue working smoothly (and did for the past week, so there are no data gaps).
Fun fact: we are now writing about 22 million new heart rate measurements each day. So over time, this adds up to a lot and our storage space has been filling up quickly!
Data like steps, heart rate and sleep constantly update throughout the day, so that people can see their latest graph and data in realtime, use it to make healthier decisions, and store it forever for easy retrieval in the weekly, monthly and yearly reviews. Having all of this history and context coming in lets us make really powerful features, like the new Health Score and continuous monitoring to warn you if you’re about to get sick.
Our current infrastructure to store and analyze all of this data now costs hundreds of thousands of dollars to provide a lightning fast syncing experience for everyone, and we are continuing to invest more to track more powerful types of data and ensure maximum reliability as we grow.
Free vs Pro
Until now, we’ve just stored everything for everyone, and retained all synced data forever. With a small number of users, this was never an issue but now it is starting to become more expensive when a single person can add hundreds of millions of new data points.
In the future, we will start limiting the storage space of these massive datasets (heart rate, steps, computer usage, etc.) for free members to ensure that Pro members can store more history and always access their old history for the rest of their life. Since free members see reports from the last few weeks in the app, this generally shouldn’t have much of an impact or be noticeable.
We will also start cleaning up inactive accounts which haven’t been used in over a year, so those resources can be used by active users who actually check the app. If keeping your old history on Gyroscope and being able to view it for many years later is important, we suggest upgrading to Pro now!
What happened last week?
You may have noticed a server maintenance message in the app last week and that the heart rate graph was not showing in the app or website. This is nothing to worry about, is now finished, and all the heart rate info should be displaying instantly again in the app.
What happened?
This originally started when we were doing maintenance on the heart rates, which were starting to take up too much space and impact overall performance. We fixed this issue fairly quickly, but after major changes something called a “vacuum” needed to be done to clean things up and resume normal operation.
Why did it take so long to finish?
Usually a vacuum is very fast and happens automatically, so on any other of our data points it would’ve been finished in a few minutes. Due to fact that there were billions of rows to process, the vacuum ended up taking multiple days to process.
To make things more complicated, we continued to do all the syncing and write hundreds of millions of new heart rates each day during this time (all of which are viewable now). This is kind of like doing an oil change on a car while going down the freeway at 100mph.
During that time, and to make sure that syncing new data continued uninterrupted, we decided to disable reading from the heart rates. This let us speed things up and prevent other features from failing under extreme load (which is what happened briefly on Monday night). The major features like Health Score, continuous monitoring, sleep tracking, background syncing, and workout sharing continued to be fully available during the entire week and were unaffected by the maintenance.
Still, we realize this missing any of the features or metrics is extremely inconvenient and that many people rely on the graphs in the app to live their lives, so we are very sorry for the unavailability! We’re now making some major changes and improvements to reduce the need for this type of downtime in the future, and so any future maintenance will be unnoticeable.
The new version of the app is now available! See what’s new.
