James P. Scanlan, Attorney at Law

Home Page

Curriculum Vitae

Publications

Published Articles

Conference Presentations

Working Papers

page1

Journal Comments

Truth in Justice Articles

Measurement Letters

Measuring Health Disp

Outline and Guide to MHD

Summary to MHD

Solutions

page3

Solutions Database

Irreducible Minimums

Pay for Performance

Between Group Variance

Concentration Index

Gini Coefficient

Reporting Heterogeneity

Cohort Considerations

Relative v Absolute Diff

Whitehall Studies

AHRQ's Vanderbilt Report

NHDR Measurement

NHDR Technical Issues

MHD A Articles

MHD B Conf Presentations

MHD D Journal Comments

Consensus/Non-Consensus

Spurious Contradictions

Institutional Corresp

page2

Scanlan's Rule

Outline and Guide to SR

Summary to SR

Bibliography

Semantic Issues

Employment Tests

Case Study

Case Study Answers

Case Study II

Subgroup Effects

Subgroup Effects NC

Illogical Premises

Illogical Premises II

Inevitable Interaction

Interactions by Age

Literacy Illustration

RERI

Feminization of Poverty S

Explanatory Theories

Mortality and Survival

Truncation Issues

Collected Illustrations

Income Illustrations

Framingham Illustrations

Life Table Illustrations

NHANES Illustrations

Mort/Surv Illustration

Credit Score Illustration

Intermediate Outcomes

Representational Disp

Statistical Signif SR

Comparing Averages

Meta-Analysis

Case Control Studies

Criminal Record Effects

Sears Case Illustration

Numeracy Illustration

Obesity Illusration

LIHTC Approval Disparitie

Recidivism Illustration

Consensus

Algorithm Fairness

Mortality and Survival 2

Mort/Survival Update

Measures of Association

Immunization Disparities

Race Health Initiative

Educational Disparities

Disparities by Subject

CUNY ISLG Eq Indicators

Harvard CRP NCLB Study

New York Proficiency Disp

Education Trust GC Study

Education Trust HA Study

AE Casey Profic Study

McKinsey Achiev Gap Study

California RICA

Nuclear Deterrence

Employment Discrimination

Job Segregation

Measuring Hiring Discr

Disparate Impact

Four-Fifths Rule

Less Discr Alt - Proc

Less Discr Altl - Subs

Fisher v. Transco Serv

Jones v. City of Boston

Bottom Line Issue

Lending Disparities

Inc & Cred Score Example

Disparities - High Income

Underadjustment Issues

Absolute Differences - L

Lathern v. NationsBank

US v. Countrywide

US v. Wells Fargo

Partial Picture Issues

Foreclosure Disparities

File Comparison Issues

FHA/VA Steering Study

CAP TARP Study

Disparities by Sector

Holder/Perez Letter

Federal Reserve Letter

Discipline Disparities

COPAA v. DeVos

Kerri K. V. California

Truancy Illustration

Disparate Treatment

Relative Absolute Diff

Offense Type Issues

Los Angeles SWPBS

Oakland Disparities

Richmond Disparities

Nashville Disparities

California Disparities

Denver Disparities

Colorado Disparities

Nor Carolina Disparitie

Aurora Disparities

Allegheny County Disp

Evansville Disparities

Maryland Disparities

St. Paul Disparities

Seattle Disparities

Minneapolis Disparities

Oregon Disparities

Beaverton Disparities

Montgomery County Disp

Henrico County Disparitie

Florida Disparities

Connecticut Disparities

Portland Disparities

Minnesota Disparities

Massachusetts Disparities

Rhode Island Disparities

South Bend Disparities

Utah Disparities

Loudoun Cty Disparities

Kern County Disparities

Milwaukee Disparities

Urbana Disparities

Illinois Disparities

Virginia Disparities

Behavior

Suburban Disparities

Preschool Disparities

Restraint Disparities

Disabilities - PL 108-446

Keep Kids in School Act

Gender Disparities

Ferguson Arrest Disp

NEPC Colorado Study

NEPC National Study

California Prison Pop

APA Zero Tolerance Study

Flawed Inferences - Disc

Oakland Agreement

DOE Equity Report

IDEA Data Center Guide

Duncan/Ali Letter

Crim Justice Disparities

U.S. Customs Search Disp

Deescalation Training

Career Criminal Study

Implicit Bias Training

Drawing Inferences

Diversion Programs

Minneapolis PD Investig

Offense Type Issues CJD

Innumerate Decree Monitor

Massachusetts CJ Disparit

Feminization of Poverty

Affirmative Action

Affirm Action for Women

Other Affirm Action

Justice John Paul Stevens

Statistical Reasoning

The Sears Case

Sears Case Documents

The AT&T Consent Decree

Cross v. ASPI

Vignettes

Times Higher Issues

Gender Diff in DADT Term

Adjustment Issues

Percentage Points

Odds Ratios

Statistical Signif Vig

Journalists & Statistics

Multiplication Definition

Prosecutorial Misconduct

Outline and Guide

Misconduct Summary

B1 Agent Cain Testimony

B1a Bev Wilsh Diversion

B2 Bk Entry re Cain Call

B3 John Mitchell Count

B3a Obscuring Msg Slips

B3b Missing Barksdale Int

B4 Park Towers

B5 Dean 1997 Motion

B6 Demery Testimony

B7 Sankin Receipts

B7a Sankin HBS App

B8 DOJ Complicity

B9 Doc Manager Complaints

B9a Fabricated Gov Exh 25

B11a DC Bar Complaint

Letters (Misconduct)

Links Page

Misconduct Profiles

Arlin M. Adams

Jo Ann Harris

Bruce C. Swartz

Swartz Addendum 2

Swartz Addendum 3

Swartz Addendum 4

Swartz Addendum 7

Robert E. O'Neill

O'Neill Addendum 7

Paula A. Sweeney

Robert J. Meyer

Lantos Hearings

Password Protected

OIC Doc Manager Material

DC Bar Materials

Temp Confidential

DV Issues

Indexes

Document Storage

Pre 1989

1989 - present

Presentations

Prosec Misc Docs

Prosec Misc Docs II

Profile PDFs

Misc Letters July 2008 on

Large Prosec Misc Docs

HUD Documents

Transcripts

Miscellaneous Documents

Unpublished Papers

Letters re MHD

Tables

MHD Comments

Figures

ASPI Documents

Web Page PDFs

Sears Documents

Pages Transfer


Flaws of Meta-Analysis

(Sept. 2, 2010)

 

Note: This page discusses the problems of standard meta analysis arising from the patterns described on the Measuring Health Disparities (MHD) and Scanlan’s Rule (SR) pages.  The discussion here is somewhat perfunctory mainly because the same points have been made in so many places with regard to other issue, such as on the Subgroup Effects sub-page of SR.    

 

Meta-Analysis is an increasingly used procedure for appraising the effect of a factor across a range of studies.  The effect size in each study, however measured, is weighted by the inverse of the variance.  Other things being equal that means the results are weighted by sample size.

 

When the effect on a dichotomous outcome is issue – for example, in the study of the extent to which a factor increases mortality  – common measures of effect size are the relative risk and the odds ratio.   But for reasons discussed generally on the Scanlan’s Rule page, such studies are problematic because of the way relative risks and odds ratios are affected by the overall prevalence of an outcome.

 

The point can be illustrated with data from Table 1 of the 2006 British Society for Population Studies.  A better illustration would involve base rates that are in ranges that one actually observes in different studies of the same issue.  But for present purposes, the data in Table 1 suffice.

 

Table A reflects the situations where that a treatment moves the risk distribution of each set of controls half a standard deviation to the right.  Thus, the effect is the same in each of the 15 studies.

 

But assuming that the risk ratio for mortality is the measure employed in the analysis, one will get different results depending on which studies are included.  Even if all studies are included, the analysis would be fundamentally flawed.  The same of course holds is the measure employed is the risk ratio for survival or the odds ratio. 

 

As discussed in many places with regard to other issues, the only useful measure is that set out on the Solutions  subpage of the MHD, which is to say the probit.

 

Users of meta analysis commonly attempt to determine whether there exists heterogeneity among the studies and if so to take that into account in interpreting the results (as discussed here, for example).  But as I explain the Subgroup Effects sub-page of SR, risk ratios tend to differ solely because of the different base rates, creating a systematic tendency for perceived heterogeneity regardless of whether there is any real heterogeneity (which would be identified by the Solutions/probit approach).  See also the Reporting Heterogeneity sub-page of MHD.

 

Table A – Data from BSPS Table 1 for Illustration of Flaws of Meta Analysis (b0902)

Cut Point

Study

Control

Treated

RRMort

RRSurv

OR

A 99

1

99.76%

99.00%

99.22%

4.24

4.27

B 97

2

99.13%

97.00%

97.84%

3.47

3.55

C 95

3

98.38%

95.00%

96.51%

3.12

3.23

D 90

4

96.25%

90.00%

93.48%

2.67

2.86

E 80

5

90.99%

80.00%

87.87%

2.22

2.53

F 70

6

84.61%

70.00%

82.55%

1.96

2.37

G 60

7

77.34%

60.00%

77.42%

1.77

2.29

H 50

8

69.15%

50.00%

72.31%

1.62

2.24

I 40

9

59.48%

40.00%

67.46%

1.48

2.19

J 30

10

49.20%

30.00%

61.28%

1.37

2.24

K 20

11

36.69%

20.00%

54.63%

1.26

2.31

L 10

12

21.77%

10.00%

46.06%

1.15

2.50

M 5

13

12.71%

5.00%

39.72%

1.09

2.74

N 3

14

8.38%

3.00%

35.86%

1.06

2.95

O 1

15

3.44%

1.00%

29.58%

1.03

3.47