Re: Raid question

I think you're missing a couple of basic statistics, the kind that could help you
loose less at the casino ;)

First, the important thing to understand is the literal translation of MTBF,
*Mean* time before failure. It's not an exact time, it's the average time to
fail based on some number of units tested (probably by the manufacturer). The
same kind of thing applies to light bulbs, coffee makers and televisions.

What the testing resulted in is usually a bell curve where the MTBF is the peak
of the curve. Think of it like taking a class, there is an average grade but
not everyone (possibly not anyone) scored the average. Most were a little bit
higher or lower than the mean (this is were the term standard deviation comes in).
Some may be way higher or lower.

Back to hard drives, there is some probability that a drive (or any component)
will fail at any time. In a simplified view, we don't usually consider the time
variable since the vendor doesn't always provide enough information for that
analysis. If we did have that information, it would look something like this:

Let's say the vendor tested 100 drives. All failed at a different times. The
vendor computes the average and that is the MTBF of the sample. To keep the
math simple, let's say that's 5 years. They can also compute the standard
deviation or variance from the mean, let's say that's 1 year (again to keep the
math simple). Now, a typical distribution might look like this:
        Time(years) # failed probability
            0-1 3 3% (3/100)
            1-2 7 10% (10/100)
            2-3 8 18%
            3-4 14 32%
            4-5 20 52%
            5-6 24 76%
            6-7 15 91%
            7+ 9 100%

In real stats, the vendor might derive the equation for the curve that fits the
above distribution but this example gets the idea across.

Note the probabilities accumulate to 100%. That's saying the chance that the
part will fail in 0-1 year is 3% up to certainty (100%) it will fail eventually.

Now that we have the probabilities, we can compute the chance of losing data in
various scenarios and assess our risk for each year where P(t) is the
the probability of failure of the system (lose data) and p(t) is the probability
of a single drive failing in that year.

Scenario 1 - single drive fails
        P(t) = p(t)

Scenario 2 - RAID-0 (data loss if *either* drive fails)
        Either drive 0 *or* drive 1 fails
        P(t) = p0(t) + p1(t)
        P(t) = 2*p(t) (if drive 0 and drive 1 are identical)
        in general, P(t) = n*p(t) where n is the number of drives

Scenario 3 - RAID-1 (data loss if both drives fail)
        Both drive 0 *and* drive 1 fails
        P(t) = p0(t) * p1(t)
        P(t) = p(t)^2

Scenario 4 - RAID-1+0 (data loss if either mirror set fails)
        4 drives, drives 0/1 in first RAID-1, 2/3 in second, 2 RAID-1's striped
        P(t) = (p0(t) * p1(t)) + (p2(t) * p3(t))
             = (p(t)^2) + (p(t)^2) (identical drives)
             = 2*P(t)^2
         
The general idea is when multiple things must fail, the probability of that
combination is multiplied and since probabilites are fractions, they get
smaller.

Also note that once a drive fails in RAID-1, the probability of failure is the
same as a single drive until the failed disc is replaced and the mirror rebuilt.
This is where hot-spares come in. The window of time of high risk is reduced
because the array is restored without waitng for an operator to replace the
failed drive.

Now that we're armed with some stats, lets do some risk assessment.

The risk exposure is the probability of data loss multiplied by the value of the
data. Since we're time dependent, we need to sum each year for the life of the
system (say 3 years). For simplicity, I'm ignoring the time value of money.

e(t) = C(t) * P(t) exposure in year t where C is cost and P is probability
E(t) = e(0-1) + e(1-2) + e(3-2) = C(t) * (P(0-1)+P(1-2)+P(2-3))

Let's try a simple scenario:
  - we just have video on our mythbox from our DVD collection that we've ripped.
  - the cost to replace our data is just the time/effort required to re-rip
    our video collection, say 10 hours at $10/hour (we're hiring a poor student
    to do it). The cost of losing our data is 10*$10 = $100.
  - The probability of a failure where data is lost is what we computed above.
    Let's just consider the 3 year lifetime. For a single drive, it's 31%,
    for a RAID-0, it's 62%, for a RAID-1, it's 4.33%, computed with the info above.
  - E(t)(single drive) = $100 * 31% = $31
  - E(t)(stripe) = $100 * 62% = $62
  - E(t)(mirror) = $100 * 4.33% = $4.33
  - E(t)(RAID1+0) = $100 * 2 * 4.33% = $8.66
  - you can see what a big difference the mirror makes.
  - upgrading from a single drive to a RAID-1 reduces our exposure by $26.33
  - in this case, it doesn't make much difference which option we choose, the
    loss is less than the cost of the drive to provide redundancy so do nothing
    is the best choice unless we can do something for less than $26.33 over the
    life of the system.
  - if our data was worth $1000 instead, our exposure goes up by a factor of 10,
    RAID-1 would reduce our exposure but $263.30. If a second drive costs $100,
    we are /saving/ $163.30.

As you can see, the value of data affects our choice of RAID. In a more complex
corporate scenario, we also factor in cost of unscheduled downtime which can be
big. Similarly, if performance were a big deal, that will affect our descision
for striping. Let's say we were running a VOD service in a hotel and a single
drive was too slow to stream to meet the demand or our guests. Not working for
a day to restore data might be a better choice than unhappy guests all the time
so RAID-0 would be appealing.

Von, I think you were on the right track but were over-complicating your
thinking. You were also missing the all important risk exposurei, the dollars
and cents component that helps decide if you're better off with a RAID 1+0 of
4 500GB discs versus just a RAID-1 or two 1000GB discs or just a single disc.

All this is good stuff, especially when you need to justify the costs of RAID
and backups to the bean counters who don't understand why the storage gear in
the data centre costs 10 times that of an OEM hard drive from OTV.

The tough part is usually putting a value on your data and downtime.
Received on Mon Jun 30 10:33:51 2008