![]() ![]() A person who wears suits to work might refer to this as a "big data" problem. In theory, this is a straightforward aggregate to run, but because the raw dataset Backblaze provides is many gigabytes in size, it can't be run in Pandas because it can't be fit into memory. For our purposes, we need only one row per drive with two pieces of information: how long it operated for and whether it failed or not. Kaplan-Meier regression enables us to use this partial information to build the survival curve instead of throwing it away.īackblaze provides a dataset that has one row per drive, per day that it operated, with a full snapshot of the drive's self-reported SMART stats on that day. This means we don't know exactly how long it would have taken for it to fail, but we know that it worked without failing for at least one year. For example, if a hard drive is retired from the data center after only one year to make space for one with a larger capacity, then this drive didn't fail – instead, it is "right-censored" in the data. This is important because not every hard drive will be in the datacenter long enough to fail. This estimator builds survival curves from data in a way that handles missing data in a useful way. To construct the survival curve, we will use a Kaplan-Meier estimator. I also refine his technique somewhat by using Apache Spark to improve performance and expressiveness of the aggregation. In this post I'll repeat Ross Lazarus' analysis using data that has been updated through Q3 2019. Reading this series was the first I had heard of a survival curve, and it seems like such a great visualization for understanding this data that I'm surprised Backblaze doesn't report it themselves. Ross Lazarus, an Australian computational biologist, used this dataset to build survival curves for hard drives in a series of blog posts in 2016. This is a commonly used technique in studies of medical patient survival rates after receiving some treatment, but equipment failure is another good application.īackblaze commendably makes fully granular source data available in addition to the summary statistics in their blog posts. This is a plot of the fraction of a population that hasn't had some terminal event happen to it yet ("death" or "failure") as a function of time elapsed since some starting point. For example, a failure rate may hold steady at a reasonable value while the drive is under warranty, only to fail at a higher rate after it hits a certain age.Ĭapturing patterns like this is the domain of survival analysis, and in particular a visualization called the " survival curve". However, not all of Backblaze's drives are the same age, an individual drive's chance of failure may vary over its lifetime in a way that is not captured by a simple summary statistic. The key metric presented in Backblaze's blog posts, the annualized drive failure rate, is a reasonably good starting point for understanding which hard drives are more reliable than others. Because they use large numbers of consumer hard drives, which are the same ones I would consider buying to use in my desktop computer, I like to consult their blog whenever I am shopping for a new one. ![]() Therefore, this report only includes 108,461 hard drives.Survival Analysis: Backblaze Hard Drives ¶īackblaze, a cloud backup service, provides one of the best public services on the internet by periodically posting hard drive failure rates for the drives in their datacenter. However, 199 drives (108,660 minus 108,461) were not included in the list above because they were used as testing drives or Backblaze did not have at least 60 of a given drive model. Of that number, there were 1,980 boot drives and 108,660 data drives. The table of lifetime failure rates below is over the period beginning in April 2013 and ending June 30, 2019.Īs of June 30, 2019, Backblaze had 110,640 spinning hard drives in its ever-expanding cloud storage ecosystem. It also released the lifetime failure rates for the hard drive models that are in service as of June 30, 2019. The percentage failure rates were revealed in a Backblaze blog and presented in a table: ![]() On This Page :Ĭloud storage service supplier Backblaze revealed the Q2 2019 failure rates for its installed hard drives. In addition, with MiniTool Partition Wizard, you can back up data and recover data to avoid data loss. Read this post to know the detailed information. Do you know the specific hard drive failure rates? Recently, Backblaze revealed its Q2 2019 and the lifetime hard drive failure rates. ![]()
0 Comments
Leave a Reply. |