Recognizing limitations: modeling considerations for the future

One aptly named recent paper, Causation between health and income: a need to panic, demonstrates that better health improves income, while income in turn affects health.¹ On this topic, Dean Jamison notes, “Mostly around medical interventions, directionality is crystal clear. It’s when you move to the work around whether health expenditures make a difference on aggregate health outcomes that the causality question becomes more complicated. Even with the models, it is sort of judgment based. When you do statistics that show that a healthy economy is going to produce more than sickness, there is a degree of plausibility.”

Other studies across a number of countries have also highlighted the bi-directionality of the link between health and wealth, with varying degrees of strength.^2,3 Yet a recent study conducted in high- and upper-middle-income countries, where the US falls, demonstrates a noticeably stronger link between health capital and economic growth.⁴

Alongside these considerations, there are several limitations that stakeholders should consider in using this model.

Model inputs and data collection

While the model is robust and complex, inputs are based solely on the data that were available, and outcomes were interpreted at the national level. The model is not causal as by the nature of our data collection, we could not randomly assign inputs to the population or control for extemporaneous influences to the observed outcomes.

• Data quality - The model is only as good as the data we have: limited consistent time-series data means that we aren’t as able to include some of the levers and variables from our framework in our model (such as variables relating to Health as an Industry, or the impact of innovation), or detect longer term trends or their impacts.
• Outcome measure selection - We sought to take a holistic approach to health outcome and economic measures, but disagreement and variation exists among experts and settings as to which measures are best to use in a model of this nature. The complex and bi-directional relationship between health and wealth also means that the model outcomes can themselves feed back into the model’s inputs, but it was not feasible to capture these feedback loops in this first iteration of the model.

Analysis in context

The model approach and use of time-dependent data produces a level of uncertainty in the analysis:

• Assume uncertainty - A lasso regression approach, which assumes we do not already know what the important factors are, was used in our model. While this provides a useful, unbiased outcome in terms of understanding what the data say are the most impactful indicators, we compromise some precision that we would otherwise achieve if we did more granular analysis on indicators we already know to be significant. While our approach is consistent with the broader objectives of the model, more granular analysis in the future would help to clarify some of the questions raised by the model.
• Time horizon and unit of analysis - Our dataset utilizes 15 years of data (2004-2019) at the state level. In order to develop a macroeconomic model with greater certainty, we would leverage data over a longer time horizon with observations at a more granular level (e.g., individual). This would allow for the model to detect greater variations and patterns between observations, and yield fewer null results. However, it is notable that even with the timeframe used, the model is able to pick up a number of trends and observations.
• Time effects – It is reasonable to assume that a certain time-lag exists between investments and resulting health outcomes. There are also a number of possible confounders that would either accelerate or slow down these effects. Using best available evidence, assumptions were made to enable conservative estimates of these effects within the analysis (see Technical Report for more information).

Interpretation and future analyses

This model is a starting point. The US is unique amongst other nations in its governance, health policies and structure. A comparison across varying health systems may yield further outcomes worth exploring.

Additional considerations include:

• Regulation and accountability - Measures of governance such as regulation and accountability are critical to effective investment, but reliable measures are not consistent and comparable at the state level. FRASE constant and current were used as proxy variables, but more health-specific variables would be favorable.
• Generalizability - This dataset was constructed at the state level in the US, which, compared to the rest of the world, is relatively data saturated. While this model can be replicated to derive insights in other places, further targeted data collection and modeling should be conducted to establish stronger generalizability.
• Correlation is not causality – The relationships between health investment, outcomes and economic growth are complex and bi-directional. This makes determining causality exceedingly difficult. Results should be interpreted carefully, as they are suggestive at most and primarily intended to increase theoretical understanding. Future work should leverage natural experiments in observational data (e.g. change in policy or investments over time, and at different times across states or demographic shocks) or develop general equilibrium or other modeling approaches that take into account the bi-directionality and feedback from the economy and its relationship with health.
• Linear modeling to assess the impacts by state - Impacts, particularly in our scenario analysis, yield a linear result (eg, if you double the input, the output will also double). This limits our ability to assess diminishing returns or model potential moderators of the relationships between health investments and outcomes. In reality, more extreme impacts may warrant a different approach for analysis (e.g. interactions, quadratics). We purposely did not take this approach in this model, as the objective was to seek an unbiased prioritization of factors that requires a holistic dataset. More specific modeling would require a series of sub-analyses.

Conclusions

The results, limitations and observations yielded by this study give rise to a number of opportunities to improve our model and conduct further analysis to build upon it.

There are some ways in which further analysis using the existing model may be useful. For example, there are eleven additional related socioeconomic outcome measures for which data are available and meet our criteria in our source dataset (see Technical Report for more details). While too numerous for the scope of this analysis, these measures, such as productivity, health workforce, and state-level debt, may be useful in evaluating the model further. There would also be an opportunity to fine-tune the three scenarios included, including through more expansive analysis on each to explore different impacts and thresholds. Significance testing, while out of scope at this phase, may give us further insight as to the results that should be prioritized, versus those that might be more prone to error.

The expansiveness of this model opens the door for a potential lifetime of additional analysis. The most obvious place to start would be to take each of the modeled outcome measures and conduct sub-analysis for each of the relationships that show impact. This may include state-level analysis or the inclusion of further input variables to gain a more precise understanding of the relevant components of each relationship. A more granular analysis (e.g., at the municipal or individual level), would yield more variation to explore.

Another approach would be to conduct sub-analysis starting with the socioeconomic outcomes (Step 3). Selecting one and exploring the most important factors in steps 1 and 2, while also going deeper into the literature, would begin to allow for a more granular understanding of policy action and the precise results that could be expected from taking action.

Adapting this model to explore other geographies outside of the US would also be a valuable step forward. Finally, it may be useful to consider feedback loops in future modeling, including the impact that improved health and economic outcomes have on investments in health.