Test for outliers. In samples from a normal population, gaps equal to or larger than the following values of G1, G2, and G3 occur with a probability P = 0.02 where outliers can occur only at one end, or with P = 0.04 where they may occur at either end. | |||||||||||
N | 3 | 4 | 5 | 6 | 7 | ||||||
G1 | .976 | .846 | .729 | .644 | .586 | ||||||
N | 8 | 9 | 10 | 11 | 12 | 13 | |||||
G2 | .780 | .725 | .678 | .638 | .605 | .578 | |||||
N | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 |
G3 | .602 | .579 | .559 | .542 | .527 | .514 | .502 | .491 | .481 | .472 | .464 |
Test for groups containing outliers. Compute the range from the f observations in each of k groups, where all groups in the series are equal in size. The observed ratio R* of the largest range to the sum of the k ranges will equal or exceed the following critical values at a probability of P = 0.05. |
|||||||||
No. of Ranges k | Critical R* for Ranges Each from f Observations | ||||||||
2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
2 | 0.962 | 0.862 | 0.803 | 0.764 | 0.736 | 0.717 | 0.702 | 0.691 | 0.682 |
3 | .813 | .667 | .601 | .563 | .539 | .521 | .507 | .498 | .489 |
4 | .681 | .538 | .479 | .446 | .425 | .410 | .398 | .389 | .382 |
5 | .581 | .451 | .398 | .369 | .351 | .338 | .328 | .320 | .314 |
6 | 0.508 | 0.389 | 0.342 | 0.316 | 0.300 | 0.288 | 0.280 | 0.273 | 0.267 |
7 | .451 | .342 | .300 | .278 | .263 | .253 | .245 | .239 | .234 |
8 | .407 | .305 | .267 | .248 | .234 | .225 | .218 | .213 | .208 |
9 | .369 | .276 | .241 | .224 | .211 | .203 | .197 | .192 | .188 |
10 | .339 | .253 | .220 | .204 | .193 | .185 | .179 | .174 | .172 |
No. of Ranges k | Critical (k + 2)R* for Ranges Each from f Observations | ||||||||
2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
10 | 4.06 | 3.04 | 2.65 | 2.44 | 2.30 | 2.21 | 2.14 | 2.09 | 2.05 |
12 | 4.06 | 3.03 | 2.63 | 2.42 | 2.29 | 2.20 | 2.13 | 2.07 | 2.04 |
15 | 4.06 | 3.02 | 2.62 | 2.41 | 2.28 | 2.18 | 2.12 | 2.06 | 2.02 |
20 | 4.13 | 3.03 | 2.62 | 2.41 | 2.28 | 2.18 | 2.11 | 2.05 | 2.01 |
50 | 4.26 | 3.11 | 2.67 | 2.44 | 2.29 | 2.19 | 2.11 | 2.06 | 2.01 |
Probits (normal deviates + 5) corresponding to percentages in the margins. | ||||||||||
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
0 | | 2.67 | 2.95 | 3.12 | 3.25 | 3.36 | 3.45 | 3.52 | 3.59 | 3.66 |
10 | 3.72 | 3.77 | 3.82 | 3.87 | 3.92 | 3.96 | 4.01 | 4.05 | 4.08 | 4.12 |
20 | 4.16 | 4.19 | 4.23 | 4.26 | 4.29 | 4.33 | 4.36 | 4.39 | 4.42 | 4.45 |
30 | 4.48 | 4.50 | 4.53 | 4.56 | 4.59 | 4.61 | 4.64 | 4.67 | 4.69 | 4.72 |
40 | 4.75 | 4.77 | 4.80 | 4.82 | 4.85 | 4.87 | 4.90 | 4.92 | 4.95 | 4.97 |
50 | 5.00 | 5.03 | 5.05 | 5.08 | 5.10 | 5.13 | 5.15 | 5.18 | 5.20 | 5.23 |
60 | 5.25 | 5.28 | 5.31 | 5.33 | 5.36 | 5.39 | 5.41 | 5.44 | 5.47 | 5.50 |
70 | 5.52 | 5.55 | 5.58 | 5.61 | 5.64 | 5.67 | 5.71 | 5.74 | 5.77 | 5.81 |
80 | 5.84 | 5.88 | 5.92 | 5.95 | 5.99 | 6.04 | 6.08 | 6.13 | 6.18 | 6.23 |
90 | 6.28 | 6.34 | 6.41 | 6.48 | 6.55 | 6.64 | 6.75 | 6.88 | 7.05 | 7.33 |
0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | |
99 | 7.33 | 7.37 | 7.41 | 7.46 | 7.51 | 7.58 | 7.65 | 7.75 | 7.88 | 8.09 |
No. of Deaths | 0 or 10 | 1 or 9 | 2 or 8 | 3 or 7 | 4 to 6 |
Weight, w | 0.3 | 0.7 | 1.0 | 1.2 | 1.3 |
Coefficients x* for computing the responses YL and YH predicted by least squares at the lowest and highest of k log-doses when these are spaced at equal intervals. | ||||||||
No. of Doses | Predicted End Y | Coefficient x* for Mean Response bar(y)t at Log-Dose | Divisor | |||||
1 | 2 | 3 | 4 | 5 | 6 | |||
3 | YL | 5 | 2 | 1 | 6 | |||
YH | 1 | 2 | 5 | 6 | ||||
4 | YL | 7 | 4 | 1 | 2 | 10 | ||
YH | 2 | 1 | 4 | 7 | 10 | |||
5 | YL | 3 | 2 | 1 | 0 | 1 | 5 | |
YH | 1 | 0 | 1 | 2 | 3 | 5 | ||
6 | YL | 11 | 8 | 5 | 2 | 1 | 4 | 21 |
YH | 4 | 1 | 2 | 5 | 8 | 11 | 21 |
Coefficients x1 for computing the slope b of a log-dose response curve when the doses are spaced on an arithmetic scale as shown. | ||||||||
Coefficients x1 for Computing b from the Responses y at Doses, in mL, of | ||||||||
No. of Doses | 1 | 1.5 | 2 | 3 | 4 | 5 | Divisor eb¢i | Mean Log-Dose bar(x) |
4 | | 29 | 12 | 12 | 29 | | 14.4663 | 0.38908 |
5 | 34 | | 9 | 5 | 15 | 23 | 24.7827 | 0.41584 |
5 | | 20 | 11 | 2 | 11 | 18 | 13.3249 | 0.45105 |
6 | 15 | 8 | 3 | 4 | 9 | 13 | 14.1017 | 0.37588 |
Factorial coefficients x1 for analyzing a balanced bioassay, in which successive log-doses of Standard (Si) and of Unknown (Ui) are spaced equally, each with the same number (f) of responses totaling Tt. | |||||||||||
Factorial Coefficients x1 for Each Dose | |||||||||||
Design | Row | S1 | S2 | S3 | S4 | U1 | U2 | U3 | U4 | ei | Ti |
2,2 | a | 1 | 1 | 1 | 1 | 4 | Ta | ||||
b | 1 | 1 | 1 | 1 | 4 | Tb | |||||
ab | 1 | 1 | 1 | 1 | 4 | Tab | |||||
3,3 | a | 1 | 1 | 1 | 1 | 1 | 1 | 6 | Ta | ||
b | 1 | 0 | 1 | 1 | 0 | 1 | 4 | Tb | |||
ab | 1 | 0 | 1 | 1 | 0 | 1 | 4 | Tab | |||
q | 1 | 2 | 1 | 1 | 2 | 1 | 12 | Tq | |||
aq | 1 | 2 | 1 | 1 | 2 | 1 | 12 | Taq | |||
4,4 | a | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Ta |
b | 3 | 1 | 1 | 3 | 3 | 1 | 1 | 3 | 40 | Tb | |
ab | 3 | 1 | 1 | 3 | 3 | 1 | 1 | 3 | 40 | Tab | |
q | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Tq | |
aq | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Taq |
Value of Constant for Design | |||||
For Computing | Equation No. | Constant | 2,2 | 3,3 | 4,4 |
M¢ | 8, 10 | c | 1 | 4/3 | 5 |
L | 26, 29 | c¢ | 1 | 8/3 | 5 |
Factorial coefficients x1 for analyzing a partially balanced assay, in which successive log-doses of Standard (Si) and of Unknown (Ui) are spaced equally, each with the same number ( f ) of responses totaling Tt. If the number of successive doses of the Unknown exceeds by one the number on the Standard, interchange Si and Ui in the heading and reverse all signs in rows a, ab, and aq. | ||||||||||
Factorial Coefficients x1 for Each Dose | ||||||||||
Design | Row | S1 | S2 | S3 | S4 | U1 | U2 | U3 | ei | Ti |
2,1 | a | 1 | 1 | 2 | 6 | Ta | ||||
b | 1 | 1 | 0 | 2 | Tb | |||||
3,2 | a | 2 | 2 | 2 | 3 | 3 | 30 | Ta | ||
b | 2 | 0 | 2 | 1 | 1 | 10 | Tb | |||
ab | 1 | 0 | 1 | 2 | 2 | 10 | Tab | |||
q | 1 | 2 | 1 | 0 | 0 | 6 | Tq | |||
4,3 | a | 3 | 3 | 3 | 3 | 4 | 4 | 4 | 84 | Ta |
b | 3 | 1 | 1 | 3 | 2 | 0 | 2 | 28 | Tb | |
ab | 3 | 1 | 1 | 3 | 5 | 0 | 5 | 70 | Tab | |
q | 3 | 3 | 3 | 3 | 2 | 4 | 2 | 60 | Tq | |
aq | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 10 | Taq |
Value of Constant for Design | |||||
For Computing | Equation No. | Constant | 2,1 | 3,2 | 4,3 |
M¢ | 8, 10 | c | 1/2 | 5/6 | 7/6 |
L | 26, 29 | c¢ | 3/4 | 25/12 | 49/12 |
Factorial coefficients x1 for analyzing assays with a 3- or 4-dose sequence of 1.5, 2.0, 3.0, and 4.0, each dose having the same number ( f ) of responses. | |||||||||||
Dose of Standard | Dose of Unknown | ||||||||||
Design | Row | 1.5 | 2.0 | 3.0 | 4.0 | 1.5 | 2.0 | 3.0 | 4.0 | ei | Ti |
4,4 | a | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Ta |
b | 29 | 12 | 12 | 29 | 29 | 12 | 12 | 29 | 3940 | Tb | |
ab | 29 | 12 | 12 | 29 | 29 | 12 | 12 | 29 | 3940 | Tab | |
q | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Tq | |
aq | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 8 | Taq | |
3,3 | a | 1 | 1 | 1 | 1 | 1 | 1 | 6 | Ta | ||
b | 25 | 3 | 28 | 25 | 3 | 28 | 2836 | Tb | |||
ab | 25 | 3 | 28 | 25 | 3 | 28 | 2836 | Tab | |||
q | 31 | 53 | 22 | 31 | 53 | 22 | 8508 | Tq | |||
aq | 31 | 53 | 22 | 31 | 53 | 22 | 8508 | Taq | |||
3,3 | a | 1 | 1 | 1 | 1 | 1 | 1 | 6 | Ta | ||
b | 28 | 3 | 25 | 28 | 3 | 25 | 2836 | Tb | |||
ab | 28 | 3 | 25 | 28 | 3 | 25 | 2836 | Tab | |||
q | 22 | 53 | 31 | 22 | 53 | 31 | 8508 | Tq | |||
aq | 22 | 53 | 31 | 22 | 53 | 31 | 8508 | Taq |
Value of Constant for Design | ||||
For Computing | Equation No. | Constant | 4,4 | 3,3 |
M¢ | 8, 10 | ci | 7.2332 | 5.3695 |
L | 26, 29 | c¢i2 | 0.10623 | 0.06100 |
Values of t, t2, F i and 2 for different degrees of freedom n that will be exceeded with a probability P = 0.05 (or 0.95 for confidence intervals). | |||||||||||
n | t | t2 = F1 | F2 | F3 | 2 | n | t | t2 = F1 | F2 | F3 | 2 |
1 | 12.706 | 161.45 | | | 3.84 | 19 | 2.093 | 4.381 | 3.52 | 3.13 | 30.1 |
2 | 4.303 | 18.51 | 19.00 | 19.16 | 5.99 | 20 | 2.086 | 4.351 | 3.49 | 3.10 | 31.4 |
3 | 3.182 | 10.128 | 9.55 | 9.28 | 7.82 | 21 | 2.080 | 4.325 | 3.47 | 3.07 | 32.7 |
4 | 2.776 | 7.709 | 6.94 | 6.59 | 9.49 | 22 | 2.074 | 4.301 | 3.44 | 3.05 | 33.9 |
5 | 2.571 | 6.608 | 5.79 | 5.41 | 11.07 | 23 | 2.069 | 4.279 | 3.42 | 3.03 | 35.2 |
6 | 2.447 | 5.987 | 5.14 | 4.76 | 12.59 | 24 | 2.064 | 4.260 | 3.40 | 3.01 | 36.4 |
7 | 2.365 | 5.591 | 4.74 | 4.35 | 14.07 | 25 | 2.060 | 4.242 | 3.38 | 2.99 | 37.7 |
8 | 2.306 | 5.318 | 4.46 | 4.07 | 15.51 | 26 | 2.056 | 4.225 | 3.37 | 2.98 | 38.9 |
9 | 2.262 | 5.117 | 4.26 | 3.86 | 16.92 | 27 | 2.052 | 4.210 | 3.35 | 2.96 | 40.1 |
10 | 2.228 | 4.965 | 4.10 | 3.71 | 18.31 | 28 | 2.048 | 4.196 | 3.34 | 2.95 | 41.3 |
11 | 2.201 | 4.844 | 3.98 | 3.59 | 19.68 | 29 | 2.045 | 4.183 | 3.33 | 2.93 | 42.6 |
12 | 2.179 | 4.747 | 3.89 | 3.49 | 21.03 | 30 | 2.042 | 4.171 | 3.32 | 2.92 | 43.8 |
13 | 2.160 | 4.667 | 3.81 | 3.41 | 22.36 | 40 | 2.021 | 4.085 | 3.23 | 2.84 | 55.8 |
14 | 2.145 | 4.600 | 3.74 | 3.34 | 23.68 | 60 | 2.000 | 4.001 | 3.15 | 2.76 | 79.1 |
15 | 2.131 | 4.543 | 3.68 | 3.29 | 25.00 | 120 | 1.980 | 3.920 | 3.07 | 2.68 | 146.6 |
16 | 2.120 | 4.494 | 3.63 | 3.24 | 26.30 | 1.960 | 3.841 | 3.00 | 2.60 | ||
17 | 2.110 | 4.451 | 3.59 | 3.20 | 27.59 | ||||||
18 | 2.101 | 4.414 | 3.55 | 3.16 | 28.87 | ||||||
Adapted from portions of Tables III to V of Statistical Tables for Biological, Agricultural and Medical Research, by R. A. Fisher and F. Yates, published by Oliver and Boyd, Ltd., Edinburgh.
|
A | absorbance for computing % reduction in bacte- rial growth from turbidimetric readings. |
b | slope of the straight line relating response (y) to log-dose (x) [Equations 2b, 4, 5, 6]. |
c | constant for computing M¢ with Equations 8 and 10. |
c¢ | constant for computing L with Equations 26 and 29. |
ci | constant for computing M¢ when doses are spaced as in Table 8. |
c¢i2 | constant for computing L when doses are spaced as in Table 8. |
C | term measuring precision of the slope in a confidence interval [Equations 27, 28, 35, 36]. |
2 | statistical constant for testing significance of a discrepancy [Table 9]. |
M2 | 2 testing the disagreement between different estimates of log-potency [Equations 39, 40]. |
eb | ei from row b in Tables 6 to 8. |
eb¢i | multiple of S(x bar(x))2 [Table 5; Equation 6]. |
ei | sum of squares of the factorial coefficients in each row of Tables 6 to 8. |
eq | ei from row q in Tables 6 to 8. |
f | number of responses at each dosage level of a preparation; number of replicates or sets. |
fS | number of observations on the Standard. |
fU | number of observations on the Unknown. |
F1 to F3 | observed variance ratio with 1 to 3 degrees of freedom in numerator [Table 9]. |
G1, G2, and G3 | relative gap in test for outlier [Table 1]. |
h | number of Unknowns in a multiple assay. |
h¢ | number of preparations in a multiple assay, including the Standard and h Unknowns; i.e., h¢ = h + 1. |
i | interval in logarithms between successive log-doses, the same for both Standard and Unknown. |
k | number of estimated log-potencies in an average [Equation 24]; number of treatments or doses [Table 4; Equations 1, 13, 15, 16]; number of ranges or groups in a series [Table 2]; number of rows, columns, and doses in a single Latin square [Equations 1a, 16a]. |
L | length of the confidence interval in logarithms [Equations 24, 26, 29, 38], or in terms of a proportion of the relative potency of the dilutions compared [Equations 31, 33]. |
Lc | length of a combined confidence interval [Equations 42, 43]. |
Lc¢ | length of confidence interval for a semi-weighted mean bar(M) [Equation 48]. |
LD50 | lethal dose killing an expected 50% of the animals under test [Equation 2c]. |
M | log-potency [Equation 2]. |
M¢ | log-potency of an Unknown, relative to its assumed potency. |
bar(M) | mean log-potency. |
n | degrees of freedom in an estimated variance s2 or in the statistic t or 2. |
n¢ | number of Latin squares with rows in common [Equations 1a, 16a]. |
N | number; e.g., of observations in a gap test [Table 1], or of responses y in an assay [Equation 16]. |
P | probability of observing a given result, or of the tabular value of a statistic, usually P = 0.05 or 0.95 for confidence intervals [ Tables 1, 2, 9]. |
P* | potency, P* = antilog M or computed directly. |
R | ratio of a given dose of the Standard to the corresponding dose of the Unknown, or assumed potency of the Unknown [Equations 2, 30, 33]. |
R* | ratio of largest of k ranges in a series to their sum [Table 2]. |
s = bar(s)2 | standard deviation of a response unit, also of a single estimated log-potency in a direct assay [Equation 24]. |
s2 | error variance of a response unit. |
Si | a log-dose of Standard [ Tables 6, 7]. |
S | the sum of. |
t | Student's t for n degrees of freedom and probability P = 0.05 [Table 9]. |
T | total of the responses y in an assay [Equation 16]. |
T ¢ | incomplete total for an assay in randomized sets with one missing observation [Equation 1]. |
T1 | S(y) for the animals injected with the Standard on the first day [Equations 18, 36]. |
T2 | S(y) for the animals injected with the Standard on the second day [Equations 18, 36]. |
Ta | Ti for the difference in the responses to the Standard and to the Unknown [Tables 6 to 8]. |
Tab | Ti for testing the difference in slope between Standard and Unknown [Tables 6 to 8]. |
Taq | Ti for testing opposed curvature in the curves for Standard and Unknown [Tables 6 to 8]. |
Tb | Ti for the combined slope of the dosage-response curves for Standard and Unknown [Tables 6 to 8]. |
Tb¢ | S(x1Tt) or S(x1y) for computing the slope of the log-dose response curve [Equations 10, 23, 28]. |
Ti | sum of products of Tt multiplied by the corresponding factorial coefficients in each row of Tables 6 to 8. |
Tq | Ti for testing similar curvature in the curves for Standard and Unknown [Tables 6 to 8]. |
Tr | row or set total in an assay in randomized sets [Equation 16]. |
Tr¢ | incomplete total for the randomized set with a missing observation in Equation 1. |
Tt | total of f responses y for a given dose of a preparation [Tables 6 to 8; Equations 6, 13, 14, 16]. |
Tt¢ | incomplete total for the treatment with a missing observation in Equation 1. |
Ui | a log-dose of Unknown [Tables 6 to 8]. |
v | variance for heterogeneity between assays [Equation 45]. |
V = 1/w | variance of an individual M [Equations 44 to 47]. |
w | weight assigned to the M for an individual assay [Equation 38], or to a probit for computing an LD50 [Equations 2a, 2b]. |
w¢ | semi-weight of each M in a series of assays [Equations 47, 48]. |
x | a log-dose of drug in a bioassay [Equation 5]; also the difference between two log-threshold doses in the same animal [Equation 12]. |
x* | coefficients for computing the lowest and highest expected responses YL and YH in a log-dose response curve [Table 4; Equation 3]. |
x1 | a factorial coefficient that is a multiple of (x bar(x)) for computing the slope of a straight line [Table 5; Equation 6]. |
bar(x) | mean log-dose [Equation 5]. |
bar(x)S | mean log-dose for Standard [Equation 9]. |
bar(x)U | mean log-dose for Unknown [Equation 9]. |
X | log-potency from a unit response, as interpolated from a standard curve [Equations 7a, 7b, 19]. |
XM | confidence limits for an estimated log-potency M
[Equations 25, 30]. |
XP* | confidence limits for a directly estimated potency P* (see Digitalis assay) [Equation 33]. |
y | an observed individual response to a dose of drug in the units used in computing potency and the error variance [Equations 13 to 16]; a unit difference between paired responses in 2-dose assays [Equations 17, 18]. |
y1 . . . yN | observed responses listed in order of magnitude, for computing G1, G2, or G3 in Table 1. |
y¢ | replacement for a missing value [Equation 1]. |
bar(y) | mean response in a set or assay [Equation 5]. |
bar(y)t | mean response to a given treatment [Equations 3, 6]. |
Y | a response predicted from a dosage-response relationship,often with qualifying subscripts [Equations 3 to 5]. |
z | threshold dose determined directly by titration (see Digitalis assay) [Equation 11]. |
bar(z) | mean threshold dose in a set (see Digitalis assay) [Equations 31, 32, 33]. |