Neighbourhood Effects Measured with a Counterfactual Design: A Journal Critique

In 2017 I did a postgrad Advanced Statistical Methods course, as it had been 10 years since I had been formally evaluated. As part of my professional development, I like to upgrade my skills in areas that are relevant to the students I work with. As a lot of my work with you pertains to research design, proposals, thesis support as well as research reports, I got a lot out of this course. You can download my journal critique on a Counterfactual Design. It is design to help determine causal inference when using an Observational Method of research outside of the lab. The control group in this design provides 'counterfactual' information; the comparison group where the intervention did not take place can be compared with the observed results with the group which was exposed.

A weakness of this design is that one cannot ever estimate with high accuracy what has occurred in the counterfactual group, as the absence of the intervention in itself, has unpredictable consequences. So, we can never say exactly what might or might not have happened if the intervention were to have taken place.

The paper I review used matched pairs of participants, so that a participant in the intervention group was matched with a participant in the control group, on a bundle of covariates. There is always 'error' or as I prefer to call it 'mystery' (the unknowable) which is what makes life~ Life. The counterfactual design is, I think, an efficient design for data collection and analysis of results, providing one keeps one's feet on the ground when interpreting results. Which should be the case regardless of the design used.

For the particular study of the article critiqued here, I suggested a mixed-method approach~ adding some qualitative elements (e.g., structured interviews) and collecting a broader variety of quantitative covariates (such as level community capacity building, level of spirituality and cultural specific practices).

And I featured the neighbourhood I grew up in for a time, and was again living in whilst writing this paper~ an 'M' area in Cairns where there is high unemployment, youth crime and low SES. I wanted to highlight the 3 M Pride initiative of some of the local youth, as an example of variables that can contribute positively to 'negihbourhood effects'. Download my critique here

Or you can read it now, it's posted after the pic~

3MPride neighbourhood effects counterfactual design

Neighbourhood Effects Measured with a Counterfactual Design: A Journal Critique Charmayne I. Paul, Advanced Research Methods, Wollongong University, 2017 Observational measurement of the long-term negative outcome causal effects of a childhood spent in a low socio-economic neighbourhood is clouded by selection bias (Harding, 2003; Hedman & van Ham, 2012). Selection bias, the unmeasured factors affecting both neighbourhood of residence and the outcomes for each individual child, are variables that could also explain an association between level of poverty and negative outcomes.

Regression models do not adequately account for unobserved covariates. For example, the FNQ regional city Cairns has 3 ‘M’ areas which are the lowest-poverty urban neighbourhoods in the State outside of Brisbane, with a high rate of high school drop-outs and teen pregnancy; and an average of third generation unemployment; a lack of citizenship role models; high youth crime and inhalant abuse; and poor collaborations across social service providers (Cairns Regional Council, 2017; Karam, Sinclair, & Rackstraw, 2014; Smallbone, Rayment McHugh, & Smith, 2013; Silva, 2014).

In the peer-reviewed study by Harding (2003), the neighbourhood-effects hypothesis was tested using a longitudinal counterfactual model with controlled sensitivity analysis. Although selection bias was not removed or ignored, it was evaluated with regards to explaining how the differing levels of bias emerging from unobserved covariates modified conclusions about the total treatment effect.

Harding’s casual model could explain the high rate of high school drop-out and pregnancy for youth who spent their adolescence residing in high poverty areas the USA from during the 80s-late 90s. This finding was sound in that hypothetical covariates could not explain the associations found between neighbourhood and the two main outcomes. Further, the neighbourhood effects found were larger in magnitude than those in previous literature.

However, a mixed method design would increase robustness of the study as more unobservable and unknown covariates could emerge to explain a causal pathway. Harding’s observational study used a matched (case-control) comparative design which did not randomly assign participants to intervention or control groups. As such ‘identical’ matched pairs of 10-year-olds residing in low or high poverty neighbourhoods were measured on covariates (e.g., annual family income, level of education for parent/s, welfare provisions, and family structure).

Exceptions were made for covariates that the treatment could not affect, for example year of birth, gender and ethnicity. The matching aided comparison of the treatment and control groups and added to the body of knowledge about how those in the treatment group could be matched effectively with untreated participants. The treatment group was defined as those participants aged between 11 and 20 years living in neighbourhoods of poverty.

The main IV was neighbourhood poverty rate during adolescence (See Appendix 1, Results: Flow of participants). Participants living in high poverty areas were defined as being different to those living in low, on observed, unobserved and unobservable outcomes. Matching each treated participant with one or more of the controls enabled the treated participants to be, on average, ‘identical’ to control participants on observable variables prior to the treatment phase. The control participants in this context are the ‘counterfactual’.

The covariates were considered balanced as the participant characteristics able to affect the outcome were well matched. A counterfactual causal framework necessitates deliberation in defining the treatment and the most appropriate covariates for matching participants. Covariate measures did not have reliability and validity values mentioned (CONSORT, n.d.).

The estimand was the true propensity score, which was defined as the unknown probability that a participant would receive the treatment. These scores are not known and are estimated from the observed covariates using a logit or probit model. The score enables the design for analysis of non-randomised observational data that ‘mimics’ a randomised controlled trial (Austin, 2011).

The distribution of observed baseline characteristics in treatment and control groups will be similar, conditional on the balance provided by the propensity score. Previous literature testing matched treatment and control participants on true propensity scores resulted in a balance across observed and unobserved covariates (Austin, 2011; Rosenbaum and Rubin, 1983).

The difference between the two groups following treatment is an unbiased estimator of the average treatment effect (ATE). A limitation of the true propensity score is that the treatment variables must be binary (e.g., low and high poverty neighbourhoods). Binomial probability is the potential for a score to be in one of two states. There is a loss of information in reducing the dependent variable to a nominal level of measurement, which in turn negates statistical power.

Harding also reports that a dichotic DV clouds nonlinear effects, which are not uncommon in the literature on neighbourhood effects. Further, propensity score matching does not control for selection bias with regards to unobserved covariates that do not correlate highly with observed covariates and are different for participants in the treatment group as compared to the control group.

As Harding notes, this biases the treatment effect estimates. However, the causal model is robust for selection bias in non-experimental settings (Rosenbaum, & Rubin, 2003). The teenage pregnancy measurement did not include the time following the birth of the child, so as not to overlap youth who moved to a high poverty neighbourhood after the birth of a child with the neighbourhood effect.

Miscarriages and abortions were not included in the measurement of teen pregnancy, nor mortality of the child which could further inform about potential covariates contributing to tragic outcomes for youth in low-income residencies (c.f., Mason, 2017). Whilst there was no mention in Harding’s study of how sample size was determined, the population of interest (teens residing in high poverty areas in the USA), population of the study (youth living in high poverty in the USA between 1968-1997 whose parents completed the PSID), and the population of causal inference (teens living in high poverty areas of the USA) were clearly delineated.

The Panel Study of Income Dynamics (PSID; 2017) is a longitudinal survey which collects economic and demographic data annually from families (and individuals) in the USA. The survey is ideal to measure neighbourhood effects in that the Geocode Match Files links geographic census data and rates of poverty and other neighbourhood variables. As low-income families are oversampled, the PSID database enables a sample size suitable for analysis of neighbourhood effects.

And, as selective attrition is low, the survey is suited to the longitudinal design and counterfactual model as used by Harding. Respondents born between 1958-1977 that had information pertaining to the outcome variables and who could be matched on census tracts for at least one of the years between the ages of 11 and 20. With regards to the assumptions Harding paid attention to, matching estimators required the same assumptions to be met as for those of regression estimators.

Although the true propensity score matching is a simple method to determine the sensitivity of findings to any unobserved covariates. The causal effect was determined by the difference in outcome betwe