CExamEmpModels

{$r_i =\sum_{i=1}^{i-1} (d_{i} - (x_{i}+u_{i})) + (d_i-u_i)/2$}, ...,

{$$1-\alpha \approx Prob (-z_{\frac{\alpha}{2}} \leq \frac{\hat{\theta}-\theta}{\sqrt{Var(\hat\theta)}} \leq z_{\frac{\alpha}{2}} ).$$}

{$$1-\alpha \approx Prob (-z_{\frac{\alpha}{2}} \leq \frac{p_n-p}{\sqrt{p(1-p)/n}} \leq z_{\frac{\alpha}{2}} );$$}

{$$1-\alpha \approx Prob \left(-z_{\frac{\alpha}{2}} \leq \frac{\hat{\theta}-\theta}{\sqrt{Var(\hat\theta)}} \leq z_{\frac{\alpha}{2}} \right)$$}

{$N = \text{# of times } x \text{ was observed in the data.}$}

{$$1-\alpha \approx Prob \left(-z_{\frac{\alpha}{2}} \leq \frac{\hat{\theta}-\theta}{\sqrt{Var(\hat\theta)}}{} \right)$$}

{$$\displaystyle{Var[S_n(x)] \approx \frac{ (\text{# of observations } > x)(\text{(# of observations } <= x)}{n^3}}. $$}

{$$Var(S_n(x)) = \frac{1}{n^2} \left[\cdot Var(Y) + t(x)^2Var(Z) + 2t(x) Cov(Y,Z)\right],$$}

(See discussion of the Covariance of 2 binomial RVs here).

Then {$N$} is binomial with parameters {$n$} and {$p(x)$}, and {$$E[p_n(x)] = \frac{1}{n} E[N] = \frac{1}{n} (n p(x)) = p(x); $$}

{$$Var[p_n(x)] = \frac{1}{n^2} Var[N] = \frac{1}{n^2} (np(x)(1-p(x)) = \frac{p(x)(1-p(x))}{n} \approx \frac{p_n(x)(1-p_n(x))}{n}; $$}

{$Cov(Y,Z) =E(YZ)-E(Y)E(Z) = - n (1-S(c_{j-1}) (S(c_{j-1})-S(c_j)).$}

{$$Var(S_n(x)) = \frac{1}{n^2} \cdot Var(Y) + t(x)^2Var(Z) + 2t(x) Cov(Y,Z),$$}

{$$E(S_n(x))= 1- \frac{1}{n} (E(Y) + t(x) E(Z)) = 1- \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) \right] = (1-t(x))S(c_{j-1})+t(x)S(c_j);$$}

{$$= 1- \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) \right]$$} {$$ = (1-t(x))S(c_{j-1})+t(x)S(c_j)$$;}

{$$= 1- \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) $$} {$$ + n t(x) (S(c_{j-1})-S(c_j)) \right] = (1-t(x))S(c_{j-1})+t(x)S(c_j)$$;}

{$$E(S_n(x))= 1- \frac{1}{n} (E(Y) + t(x) E(Z)) $$} {$$= 1- \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) \right] = (1-t(x))S(c_{j-1})+t(x)S(c_j)$$;}

{$$what??$$}

{$$E(S_n(x)) = 1- \frac{1}{n} (E(Y) + t(x) E(Z)) = 1- \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) \right] = (1-t(x))S(c_{j-1})+t(x)S(c_j)$$;}

{$E[S_n(x)] = S(x) \approx S_n(x);j $}

{$$E(S_n(x)) = 1- \frac{1}{n} (E(Y) + t(x) E(Z)) = 1- \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) \right] = (1-t(x))S(c_{j-1}+t(x)S(c_j)$$;}

{$$E(S_n(x) = 1- ]frac{1}{n} (E(Y) + t(x) E(Z)) = 1 - \frac{1}{n} \cdot \left[n (1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) \right] = (1-t(x))S(c_{j-1}+t(x)S(c_j)$$;}

{$$\hat{q_2}=\frac{\text{# of deaths between 2 and 3}}{k}. $$}

showing that the estimate is unbiased.

{$$E(S_n(x) = 1- E(Y) + t(x) E(Z) = 1- \frac{1}{n} \cdot n(1-S(c_{j-1}) + n t(x) (S(c_{j-1})-S(c_j)) (1-t(x))S(c_{j-1}+t(x)S(c_j) = (1-t(x))S(c_{j-1}+t(x)S(c_j)$$;}

{$$S_n(x) = 1 - \frac{1}{n} (Y + t(x)Z ),$$}

{$$E(S_n(x) = 1- E(Y) + t(x) E(Z) = 1- \frac{1}{n} \cdot n(1-S(c_(j-1)) + n t(x) (S(c_{j-1})-S(c_j)) (1-t(x))S(c_{j-1}+t(x)S(c_j) = (1-t(x))S(c_{j-1}+t(x)S(c_j)$$;}

{$$E[S_n(x)] = \frac{1}{n} E[Y] = \frac{1}{n} (n S(x)) = S(x); $$}

{$$E[\hat{q_2}| \text{# of people alive at }2 = k] = E[Y/k] = \frac{1}{k}\frac{k(S(2)-S(3))}{S(2)} = \frac{S(2) - S(3)}{S(2)};$$} showing that the estimate is unbiased. Note that {$$\hat{q_2}=\frac{\text{# of deaths between 2 and 3}}{k}. $$}

{$$S_n(x) = 1 - \frac{1}{n} Y + t(x)Z,$$} where

{$Y = \text{# of observations } <= c_{j-1},$}

{$Z = \text{# of observations } > c_{j-1} \text{ and } <= c_{j},$}

{$t(x) = \frac{x-c_{j-1}}{c_j-c_{c-1}}.$}

{$$E(S_n(x) = 1- E(Y) + t(x) E(Z) = (1-t(x))S(c_{j-1}+t(x)S(c_j) (1-t(x))S(c_{j-1}+t(x)S(c_j)$$;}

{$$Var(S_n(x)) = $$}

{$E[S_n(x)] = S(x) \approx S_n(x); $}

{$$E[\hat{q_2}| \text{# of people alive at }2 = k] = E[Y/k] = \frac{1}{k}\frac{k(S(2)-S(3))}{S(2)}\approx \frac{S_n(2) - S_n(3)}{S_n(2)}=\frac{\text{# of deaths between 2 and 3}}{k}; $$} {$$Var[\hat{q_2} | \text{# of people alive at }2 = k] = Var[Y/k] \approx \frac{\text{(# of deaths between 2 and 3)(}k -\text{# of deaths between 2 and 3)}}{k^3}. $$}

{$$E[\hat{q_2}| \text{# of people alive at }2 = k] = E[Y/k] = \frac{1}{k}\frac{k(S(2)-S(3))}{S(2)}\approx = \frac{S_n(2) - S_n(3)}{S_n(2)}=\frac{\text{# of deaths between 2 and 3}}{\text{# of people alive at 2}}; $$} {$$Var[\hat{q_2} | \text{# of people alive at }2 = k] = Var[Y/k] \approx \frac {S_k(x)(1-S_k(x)}{k}=Var[\hat{q_2} | \text{# of people alive at }2 = k] \approx \frac{\text{(# of deaths between 2 and 3)(}k -\text{# of deaths between 2 and 3)}}{k^3}. $$}

{$$E[S_n(x)] = \frac{1}{n} E[Y] = \frac{1}{n} (n S(x)) = S(x) \approx S_n(x); $$} {$$Var[S_n(x)] = \frac{1}{n^2} Var[Y] = \frac{1}{n^2} (nS(x)(1-S(x)) = \frac{S(x)(1-S(x))}{n} \approx \frac{S_n(x)(1-S_n(x))}{n}; $$}

{$$Var[\hat{q_2} | \text{# of people alive at }2 = k] = Var[Y/k] \approx \frac {S_k(x)(1-S_k(x)}{k};$$}

{$$\displaystyle{Var[\hat{q_2} | \text{# of people alive at }2 = k] \approx \frac{\text{(# of deaths between 2 and 3)(}k -\text{# of deaths between 2 and 3)}}{k^3}}. $$}

{$\displaystyle{Var[S_n(x)] \approx \frac{ (\text{# of observations } > x)(\text{(# of observations } <= x)}{n^3}} $}.

{$E[\hat{q_2}| \text{# of people alive at }2 = k] = E[Y/k]\approx \frac{S_k(x)}{k} = \frac{\text{# of deaths between 2 and 3}}{\text{# of people alive at 2}}; $}

{$\displaystyle{{$Var[S_n(x)] \approx \frac{\text{(# of observations } > x)(\text{# of observations } <= x)}{n^3}} $}.

{$Var[\hat{q_2} | \text{# of people alive at }2 = k] = Var[Y/k] \approx \frac {S_k(x)(1-S_k(x)}{k}$};

This is {$\displaystyle{\frac{\text{(# of deaths between 2 and 3)(}k -\text{# of deaths between 2 and 3)}}{k^3}} $}

{$Var[\hat{q_2} | \text{# of people alive at }2 = k] = Var[Y/k] \approx \frac {S_k(x)(1-S_k(x)}{k} = \frac{\text{(# of deaths between 2 and 3)(}k -\text{# of deaths between 2 and 3)}}{k^3} $}

{$Var[S_n(x)] = \frac{1}{n^2} Var[Y] = S(x)(1-S(x))/n \approx S_n(x)(1-S_n(x))/n; $}

Empirical estimate of (say) {$q_2 = \frac{S(3) - S(2)}{S(2)}$} is {$\hat{q_2}=\frac{S_n(3) - S_n(2)}{S_n(2)} = \frac{Y}{X}$}, where

{$X$} = # of people alive at duration 2,

{$Y$} = # of deaths between duration 2 and 3.

It's not possible to calculate {$Var[\hat{q_2}]$}, because {$X = 0$} would make the denominator zero; an alternative is to calculate the conditional variance given, say, {$X=k$}. Note that we can estimate the resulting RV, {$Y/k$}, with its own empirical estimate {$S_k(x) =Y/k$}, so that as the case above,

{$Var[\hat{q_2 | \text{# of people alive at }2 = k} | ] = Var[Y/k] \approx \frac {S_k(x)(1-S_k(x)}{k} = \frac{\text{(# of deaths between durations 2 and 3)(}k -\text{# of deaths between durations 2 and 3}}{k^3} $}

It's not possible to calculate {$Var[\hat{q_2}]$}, because {$X = 30$} would make the denominator zero; an alternative is to calculate the conditional variance

Empirical estimate of (say) {$q_2 = \frac{S(3) - S(2)}{S(2)}$} is {$\hat{q_2}=\frac{S_n(3) - S_n(2)}{S_n(2)} = \frac{Y}{n-X}$}, where

{$E[\hat{q_2}] = ?? \approx S_n(x); $}

It's not possible to calculate {$Var[\hat{q_2}]$}, because {$X = 30$} would make the denominator zero; an alternative is to calculate

{$Var[\hat{q_2} | X= 29 ]$ = Var[] S(x)(1-S(x))/n \approx S_n(x)(1-S_n(x))/n; $}

Given a complete data set of {$n$} points, the empirical estimate of {$S(x)$} is {$S_n(x)= Y/n$},

Empirical estimate of (say) {$q_2)$} is {$\hat{q_2}=\frac{S_n(3) - S_n(2)}{S_n(2)} = \frac{Y}{n-X}$}, where

{$X$} = # of deaths between duration 0 and 2,

{$Y$} = # of deaths between duration 2 and 3.

{$E[\hat{q_2}] = S(x) \approx S_n(x); $}

single-decrement mortality probabilities {$q_{j}^{'(d)}$}, by treating the death counts as uncensored observations and withdrawls as censored observations;

single-decrement withdrawl probabilities {$q_{j}^{'(w)}$}, by treating the withdrawl counts as uncensored observations and deaths as censored observations.

{$\displaystyle{q_j = \frac{S(c_j)-S(c_{j+1})}{S(c_j)} = \frac{x_j}{r_j} }$}.

Large Data Sets: Sometimes the data is so large that it's not reasonable to keep track of all the {$y_j$}'s. So instead we have intervals of interest: Let {$c_o < c_1 < \ldots < c_k$} be boundaries of such intervals.

{$d_i$} = # of truncated observations where the truncation point is in {$ [c_{i-1}, c_{i} ) $};

I. All truncation occurs at the beginning of the interval, and all censoring occurs at the end of the interval.

(For example, all lives enter and leave study using their insuring age.)

Define the risk set to be

{$r_1 = d_1$},

{$r_2 = r_1 - (x_1+u_1) + d_2$}, ...,

{$r_i = r_{i-1} - (x_{i-1}+u_{i-1}) + d_i$}, ...,

II. Truncation and censoring occur uniformly throughout the interval, and the uncensored observations occur in the middle of the interval.

(For example, more detailed data gathering would invariably give something closer to this scenario than the former.)

Define the risk set to be

{$r_1 = (d_1-u_1)/2$},

{$r_2 = d_1 - (x_1+u_1) + (d_2-u_2)/2$}, ...,

{$r_i =\sum_{i=1}^{i-1} (r_{i} - (x_{i}+u_{i})) + (d_i-u_i)/2$}, ...,

II. Truncation and censoring occur uniformly throughout the interval, and the uncensored observations occur in the middle of the interval.

(For example, more detailed data gathering would invariably give something closer to this scenario than the former.)

Define the risk set to be

{$r_1 = (d_1-u_1)/2$},

{$r_2 = d_1 - (x_1+u_1) + (d_2-u_2)/2$}, ...,

{$r_i =\sum_{i=1}^{i-1} (r_{i} - (x_{i}+u_{i})) + (d_i-u_i)/2$}, ...,

It follows that the probability that someone is alive at {$c_j$}, but does not survive past {$c_{j+1}$}, is

{$y_j = \frac{S(c_j)-S(c_{j+1}}{S(c_j)} = \frac{x_j}{r_j} $}.

Moreover, in the situation where we can make more refined categories about the uncensored observations {$x_j$}'s, we can calculate single-decrement probabilities. For example, in a mortality study we can have someone either withdrawl or die in a time interval. We can then calculate

single-decrement mortality probabilities {$q_{j}^{'(d)}$}, by treating the death counts as uncensored observations and withdrawls as cencored observations;

single-decrement withdrawl probabilities {$q_{j}^{'(w)}$}, by treating the withdrawl counts as uncensored observations and deaths as cencored observations.

Large Data Sets: Sometimes the data is so large that it's not reasonable to keep track of all the {$y_j$}'s. So instead we have intervals of interest: Let {$c_o < c_1 < \ldot < c_k$} be boundaries of such intervals.

Example: Mortality study on a large population; {$c_j$}'s can be integer ages.

For {$i = 1, \ldots, n,$} let

(Note indexing here is a little different from the book, in order to match the rest of the chapter.)

Two approaches:

I. All truncation occurs at the beginning of the interval, and all censoring occurs at the end of the interval.

(For example, all lives enter and leave study using their insuring age.)

Define the risk set to be

Define a Kaplan-Meier type survival function on the boundary points by

{$S(c_1)= 1-\frac{x_1}{r_1}$},

{$S(c_2)= S(c_1)\left(1-\frac{x_1}{r_1} \right)$},

{$S(c_j)= \prod_{i=1}^j \left(1-\frac{x_i}{r_i}\right)$}.

{$u_i$} = censored value for that observation.

Large Data Sets: Sometimes the data is so large that it's not reasonable to keep track of all the {$y_j$}'s. So instead we have intervals of interest: Let {$c_o < c_1 < \ldot < c_k$} be boundaries of such intervals. For {$i = 1, \ldots, n,$} let

{$d_i$} = # of truncated observations where the truncation point is in {c_{i-1}, c_{i})$};

{$x_i$} = # of uncensored observations in {$(c_{i-1}, c_{i}$};

{$u_i$} = # of censored observations with values in {$(c_{i-1}, c_{i}]$};

Then the risk set is

{$r_1 = d_1$},

{$r_2 = r_1 - (x_1+u_1) + d_2$}, ...,

{$r_i = r_{i-1} - (x_{i-1}+u_{i-1}) + d_i$}, ...,

Example: Mortality study on a large population; {$c_j$}'s can be integer ages.

Construction of Empirical Models (20-25%)

1. Estimate failure time and loss distributions using:

2. Estimate the variance of estimators and confidence intervals for failure time and loss distributions.

3. Apply the following concepts in estimating failure time and loss distribution:

a) Unbiasedness

b) Consistency

c) Mean squared error