key: cord-0203959-egk67kcj
authors: Griffin, Beth Ann; Schuler, Megan S.; Pane, Joseph; Patrick, Stephen W.; Smart, Rosanna; Stein, Bradley D.; Grimm, Geoffrey; Stuart, Elizabeth A.
title: Methodological considerations for estimating policy effects in the context of co-occurring policies
date: 2021-06-08
journal: nan
DOI: nan
sha: 930999f170fc833adbbd5b3efa60134737c67ac5
doc_id: 203959
cord_uid: egk67kcj

Objective. Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about optimal methods for disentangling the effects of concurrently implemented policies. In this paper, we examined the impact of co-occurring policies on the performance of commonly used models in state policy evaluations. Data Sources. Outcome of interest (annual state-specific opioid mortality rate per 100,000) was obtained from 1999-2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files. Study Design. We utilized Monte Carlo simulations to assess the effect of concurrent policy enactment on the evaluation of state-level policies. Simulation conditions varied effect sizes of the co-occurring policies as well as the length of time between enactment dates of the co-occurring policies, among other factors. Data Collection. Longitudinal annual state-level data over 18 years from 50 states. Principal Findings. Our results demonstrated high relative bias (>85%) will arise when confounding co-occurring policies are omitted from the analytic model and the co-occuring policies are enacted in rapid succession. Moreover, our findings indicated that controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise, with larger variance estimates when co-occurring policies were enacted in near succession of each other. We also found that the required length of time between co-occurring policies necessary to obtain robust policy estimates varied across model specifications, being generally shorter for autoregressive (AR) models compared to difference-in-differences (DID) models.

Empirical studies that evaluate the effectiveness of a given policy on health outcomes are a staple of health policy research. 1 Generally, these studies are observational and capitalize on geographic and temporal variation in policy adoption to identify policy effects. 2 Under ideal conditions for causal inference, the policy of interest would be adopted by a sufficient number of jurisdictions (e.g., communities, states, countries) with no other potentially confounding policies adopted right before or right after the policy -under these circumstances, one could rigorously isolate the true policy effect. Yet in practice, jurisdictions often adopt a cluster of policies within a brief span of time. 3 Concurrently enacted policies lack sufficient periods of time between enactment dates, posing challenges to isolating the effect of the primary policy independent of co-occurring secondary policies. Recent work has begun to elucidate the methodological challenges faced when trying to disentangle the individual policy effects of concurrent policies; 4 however, unanswered questions remain regarding the impact of co-occurring policies on the performance of commonly used models for estimating policy effects.

Methodologic challenges arising from concurrent policies apply broadly to policy research and are particularly relevant for analyses of opioid policies as states have generally enacted multiple opioid-related policies as the opioid crisis has evolved. For example, many states implemented some combination of naloxone laws, Good Samaritan laws, and medical marijuana laws during 2015-2017. In addition to pre-existing prescription drug monitoring program (PDMP) laws, by 2017 the majority of states had implemented at least 3 of these 4 categories of policies. Thus, analytically, it is very unlikely that it will be possible to identify a sizeable number of states that have implemented the policy of interest in "isolation." Furthermore, these policies and regulations may be enacted in rapid succession. For example, Florida nearly simultaneously enacted a PDMP and stricter pain clinic regulations. 5, 6 Similarly, as opioid-related overdoses climbed, many states adopted naloxone access laws and Good Samaritan laws simultaneously or in short succession to reduce opioid overdose mortality. [7] [8] [9] In a prior scoping review of evaluation studies examining the impact of U.S. state-and federallevel policies on opioid-related outcomes published in 2005-2018, it was found that of the 145 studies reviewed, 94 (65%) did not control for any co-occurring policies. 10 Studies published more recently were more likely to adjust for co-occurring policies, likely reflecting both the increasing number of state policies enacted, as well as increased attention to this methodological concern, which was highlighted in the review. When co-occurring policies were accounted for, the set of polic(ies) adjusted for varied notably across studies. For example, comparison of two published PDMP studies found that both control for a fully disjointed set of policies: pill mill laws, ACA Medicaid expansion, OxyContin reformulation, and passage of Medicare Part D 11 compared to naloxone laws, Good Samaritan laws, requirement for physical examination for prescribing and identification upon dispensing, and medical marijuana laws. 12 However, few studies explicitly discussed the rationale underlying the co-occurring policies they adjusted for. More broadly, Matthay, Gottlieb, et al. 13 found that only a third of social policy studies explicitly considered the presence of concurrent policies.

In practice, policy evaluation studies are more likely to account for potential confounding due to differences in state-level characteristics (e.g., sociodemographic characteristics) than arXiv.com Version 5 posted June 8, 2021

confounding due to differences in state policy environments. 10 However, co-occurring policies could represent a more significant confounding threat, as these policies may have a stronger relationship with the outcomes compared to other state-level confounding factors. As such, addressing the potential confounding role of co-occurring policies in evaluation studies could enhance the rigor of policy analyses and policymaking. Obtaining unbiased estimates of policy effects is imperative to identifying and implementing effective policies that can improve public health and well-being across a range of areas, including opioid misuse as well as gun violence and COVID-19 mitigation.

In this paper, we conducted a simulation study to examine the impact of co-occurring policies on the performance of commonly used models in opioid policy evaluations. We had two key objectives -(1) in the context of correctly specified models that control for co-occurring policies, we sought to determine the minimum length of time between enactment of the primary and co-occurring policies needed to obtain unbiased effect estimates, and (2) to determine the impact of model misspecification that omits the co-occurring policy on estimation of the effect of the primary policy. Simulation conditions vary effect sizes of the primary and co-occurring policies as well as the length of time between enactment dates of the primary and co-occurring policies, among other factors. The results provide insights and best practice guidelines for applied policy researchers.

We utilized Monte Carlo simulations to assess the effect of concurrent policy enactment on the evaluation of state-level policies. We considered the performance of two different statistical models -a correctly specified model that includes regression terms for both policies simultaneously and a misspecified model that omits the co-occurring policy -over a range of scenarios where a group of "treated" states enacted two policies (i.e., the primary policy and a co-occurring, secondary policy). In the context of the correctly specified model, we examined simulation conditions varying the length of time and relative timing of the policy enactment dates for the two policies in order to identify the minimum length of time needed between policy enactment dates to obtain robust estimates of the primary policy. Additionally, we explored the impact of misspecifying the statistical model by omitting the co-occurring policy on estimation of the primary policy effect. The corresponding author's Institutional Review Board approved this study.

The simulation was based on longitudinal annual state-level data. The outcome of interest was the annual state-specific opioid mortality rate per 100,000 state residents, obtained from the 1999-2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files. Eighteen years of annual observations across 50 states provided a total of 900 observations. Consistent with other studies, 9, 14, 15 the U.S. Department of Labor, Bureau of Labor Statistics. 16 This covariate was selected because of the frequency of its use in opioid policy studies. 10 In our simulation, the original opioid mortality data represented outcomes under the null treatment condition (i.e., no policies in effect for a given state and given year); outcomes reflecting the effect of simulated primary and secondary policies were generated for treated states as described below.

The simulation design builds directly from similar prior work that compared statistical methods for evaluating the impact of state laws on firearms deaths 17 and opioid-related mortality. 18 In each simulation iteration, we first selected a random subset of k states to be the "treated" group that enacted both the primary and co-occurring policy, with the remaining states serving as the comparison group that never enacted either policy. Two time-varying policy indicators were generated denoting whether the primary policy ( 1it) and co-occurring policy ( 2it) were in effect for each state and each year. For comparison states, 1it = 2it = 0 for the entire study period. For treated states, we explored four different conditions in which the period between primary and cooccurring policy enactment dates varied, ranging from approximately 0-1 years (Condition 1), 3-4 years (Condition 2), 6-7 years (Condition 3), to 9-10 years (Condition 4). In the first year of enactment, 1it and 2it were coded as fractional values between 0 and 1, indicating the percentage of the year each policy was in effect. Once the primary or secondary policy was implemented, they remained in effect throughout the study period; thus, kit = 1 (for k=1,2) for all remaining years.

We generated the outcome values ( * ) using the following formulas: * = + 1 * 1 + 2 * 2 where denotes the observed outcome value for state i in year t (obtained as described in the Data section) and 1 and 2 denote the policy effects for the primary and co-occuring policies.

Simulation conditions also varied the following factors:

(1) Effect size. We considered the performance of the candidate statistical models over four different treatment effect scenarios. Namely, we let ( 1 , 2 ) = (0%, 0%) (e.g., both policies had null effects), (0%, -15%), (-15%, 0%) (-10%, -10%), (-15%, -5%), (-5%, -15%), (-10%, -20%), and (-20%, -10%).

(2) Number of treated units. We also investigated the role of sample size in the scenarios, considering two cases where there were 5 and 30 states, respectively, enacting both policies.

(3) Timing of policy effect. State policies often do not become 100% effective immediately after enactment, making it important to consider variation in effect onset. We considered two conditions: an instantaneous effect and a 3-year linear phase-in effect. In both the data generating and analytic models, an instantaneous effect was specified as a simple step-function with a value of zero when the policy was not in effect and a value of one when the policy was in effect (as described above). The gradual policy effect allowed the policy effect to grow linearly in the first 3 years after enactment with values starting at 0 and reaching 1 after 3 years. (4) Ordering of policy enactment dates. Finally, we considered cases where the order of enactment was random (e.g., ~50% of the time the primary policy would occur first) and cases where the order of enactment was fixed so the primary policy always preceded the secondary policy. For each permutation, we assessed performance across 5000 randomly generated datasets. Simulations were conducted in R; code is available in the Appendix. Extensive results for all models and settings considered in the simulation are available upon request.

We focused on the performance of a linear autoregressive (AR) model as this model was recently identified as the optimal model for estimating the effects of a single state-level opioid policy on opioid-related mortality. 18 Additional results for the classic two-way fixed effects model (e.g., difference-in-differences) are provided in the Appendix. Within this model class, we examined two model specifications. First, we fit the correctly specified version of the outcome model that includes regression terms for both the primary and the co-occurring policy; second, we fit a misspecified version of the outcome model that omits the co-occurring policy.

The correctly specified AR model was:

denotes the time-varying state-level covariate (unemployment rate) and denotes the error term. Notably, Model (1) includes time fixed effects, , to quantify temporal trends across time, but adjusts for state-specific variability with the AR term ( • −1 ) rather than state fixed effects. Inclusion of the AR term created a "change" model, as the policy effect was defined as the expected difference in the outcome, given the prior year's outcome. As such, we coded both policy variables using change coding (i.e., ( 1 − 1 , −1 ), based on work demonstrating that effect size estimates from AR models can be substantially biased when using standard effect coding ( ). 19 The coefficient estimates ( � 1 , � 2 ) on the change coding terms represented the AR estimator of the policy effect of ( 1 , 2 ). For this work � 1 was of primary interest. However, results additionally included model estimates for the effect of the co-occurring policy ( � 2 ). The misspecified AR model was:

where the co-occurring policy variable was left out of the model. Interest was in understanding the resulting bias in the estimate of the primary policy under the misspecified model and how this varied with the relative timing of the primary and co-occurring policies. Model (2) is often fit in practice when researchers focus on a single policy of interest without controlling for any additional co-occurring policies as potential confounders. [20] [21] [22] [23] For the misspecified model, only estimates of the primary policy effect ( � 1 ) are reported.

Both AR models used state population as an analytic weight, an approach commonly used in opioid policy evaluations. [24] [25] [26] [27] Using population weights in state-level analyses of opioidmortality rates resulted in models for which each opioid-related death was treated as equally important regardless of which state it occurred in. The full set of simulations considers additional models from Griffin, Schuler, et al. 18 

We report several statistical metrics commonly used to judge model performance with respect to both the estimated effect of the primary policy (� � ) (from both Models (1) and (2)) and the estimated effect of the co-occurring policy (� ) (from Model (1)). Performance metrics include bias, variance, and root mean squared error as well as Type I and S errors and coverage arXiv.com Version 8 posted June 8, 2021 rates, given the high prevalence of frequentist null hypothesis significance testing in the statelevel policy evaluation literature.

(1) Bias. Bias assesses the average difference between the estimated effect and true effect across all simulations. It indicates the tendency of the estimated effects of a given model to fall closer or further from the true effect on average. For the null effect settings, bias is shown on an effect size scale to denote small (<0.2), moderate (0.2-0.4) and large (>0.4) bias. For the non-null settings, the percent relative bias is shown, with <5% denoting virtually no bias, 5-10% small bias, 10-20% moderate bias and >20% large bias. 28 Since all of our assumed effects were negative, positive relative bias implied that the estimated effect was overestimated in the negative direction, while negative relative bias implied that the estimated effect was underestimated and closer to the null, relative to the true effect.

(2) Variance. Variance is calculated as the average variance of the estimated effect across all simulations and indicates the precision of the estimated effects of a given model. ) where represents the true policy effect and � represents the estimated policy effect from a given simulation and model. It gives a sense of how much error occurs for a given model specification, taking into account both bias and variance.

(4) Type I error rate. Type I error rate is the frequency of incorrectly rejecting the null hypothesis (i.e., there is no policy effect). When data are generated such that there is no true policy effect (i.e., the null hypothesis is true), the model should identify a statistically significant effect (i.e., reject the null hypothesis) no more than 5% of the time if tested with an 0.05 level of significance.

(5) Type S Error. This is the rate by which a model estimates a statistically significant finding in the wrong direction. Thus, in the case where the assumed policy effects were all negative, we counted the rate at which the model estimated the policy as having a significant positive effect (i.e., lower and upper limits of 95% confidence interval were both positive).

(6) Coverage. The coverage rate is the percentage of simulation runs for which the 95% confidence interval covers the true effect of a given policy for both the primary and co-occurring policies. Figure 1 shows model performance with a correctly specified model for the simulation condition with 30 treated states, an effect size of -10% for both policies (with an additive effect across policies), and random ordering of the primary and co-occurring policies. As shown, there was minimal impact on relative bias as the mean length of time between enactment dates increases for both the primary and co-occuring policy effect. For all parameters, the relative bias was virtually non-existent (always < 2%). In contrast, increasing the mean length of time between enactment dates had the most notable impact on the variance of the primary and co-occurring policy effect estimates, with variance getting cut by 2/3 rds as the mean length of time between enactment dates increased to more than 3-4 years. The variance remained relatively stable beyond the 3-4 year scenario. RMSE was relatively stable across settings with varying lengths of time between enactment dates. Although the Type S error rates were low (< 5%) across simulation conditions with varying lengths of time between enactment dates, Type S error rates arXiv.com Version 9 posted June 8, 2021

for the primary and co-occurring policy effects were highest for the condition in which states enacted policies in rapid succession (~0-1 year apart), suggesting the possibility for ~5% of the models to estimate statistically significant policy effects that were in the wrong direction for either the primary or secondary policy. Of note, coverage rates were relatively stable across settings with varying lengths of time between enactment dates. Figure 2 shows the impact on model performance for the misspecified model that leaves out the co-occurring policy for the simulation condition with 30 treated states, an effect size of -10% for both policies (with an additive effect across policies), and random ordering of the primary and co-occurring policies. As expected, the relative bias for the estimated primary policy effect was larger in magnitude compared to the relative bias under the correctly specified models (Figure 1 ). Most notably, the relative bias was substantially worse in the case with ~0-1 years between enactment dates, at 82.3%. However, relative bias was much smaller (approximately -7%) and similar across conditions in which the length of time between enactment dates was 3-4-years or greater. Variance was relatively unaffected by the length of time between enactment dates in these models. Relatedly, coverage was lower (74%) when states enacted policies in rapid succession (~0-1 years apart), but Type S error rates were actually slightly lower (< 1%). Figure 3 showcases the role of both policy ordering and effect size on the performance of correctly specified models for the simulation condition in which the primary and co-occurring policies were enacted in rapid succession (within 0-1 year of each other). Notably, relative bias was consistently higher in the simulation setting in which the primary policy always was enacted first ("ordered" panel in Figure 3 ) compared to the setting in which the ordering of the two policies was randomly determined ("unordered" panel in Figure 3 ). The relative bias in the ordered setting ranged from 11 -23% for the primary policy and from -6 to -24% for the cooccurring policy, whereas estimates from the unordered setting consistently had minimal relative bias. Additionally, in the ordered setting, the models consistently overestimated the effect of the primary policy and underestimated the effect of the co-occurring policy which was enacted second. Finally, in the ordered setting, relative bias of the both primary and co-occurring policies was related to effect size, with greater bias observed when the true policy effect was smaller.

The results presented in Figures 1-3 show results for the simulation condition with 30 treated states; performance was generally worse when there were only 5 treated states (results available via companion Shiny app). For example, for the 0-1 year interval, the relative bias for the primary policy estimate under Model (1) was approximately -7% with 5 treated states, compared to approximately 1% with 30 treated states. Compared to conditions in which the policy had an immediate effect, performance was also lower for conditions in which the policy effects phase-in across a 3-year period, with relative bias around 10% for the same scenarios considered in Figure  1 . Appendix A summarizes the results for the classic two-way fixed effects difference-indifferences (DID) model. The AR models were generally more robust to the impacts of decreasing the length of time between enactment dates than the classic DID model. For example, in the simulation conditions considered in our study, the misspecified DID models yielded relatively unbiased effect estimates for the primary policy when the length of time between the posted June 8, 2021 enactment dates for the primary and co-occurring policy was at least 6-7 years, compared to 3-4 years for the AR model. Also, the variability of the policy effects in the AR models were often as much as three times lower than the estimated variances from the classic DID models.

Overall, we conducted a novel simulation study to assess the minimum length of time between enactment of the primary and co-occurring policies needed to obtain unbiased effect estimates as well as to determine the impact of model misspecification that omited the co-occurring policy on estimation of the effect of the primary policy. As we discuss below, we found that the required length of time between primary and co-occurring policies necessary to obtain robust policy estimates varied across model specifications but was generally shorter for AR models compared to DID models. Additionally, our results demonstrated the magnitude of the bias that may arise when confounding co-occurring policies were omitted from the analytic model. As such, researchers should think carefully about co-occurring policies that are likely to influence the outcome and ensure inclusion of them in analytic models. Our findings highlight several key issues regarding co-occurring policies in the context of opioid-policy research, yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19.

This study aimed to identify the minimum length of time needed between the enactment dates of the two policies to obtain unbiased estimates of the primary policy while controlling for the cooccurring policy. Our findings indicated that AR models that adjusted for co-occurring policies were able to obtain accurate policy effect estimates even when the policies were enacted in rapid succession (0-1 years) but yielded more precise estimates when policies were at least 3-4 years apart. Notably, AR models outperformed classic DID models on this front, as DID models yielded estimates with much greater variance under all scenarios. Thus, our findings indicate that controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise, with larger variance estimates reflecting the uncertainty inherent when trying to establish the true policy effect when co-occurring policies were enacted in near succession with the primary policy. The relative imprecision of policy estimates in this setting may result in spurious null findings, in which policies are deemed to be ineffective when they truly are effective.

These findings inform important guidance for applied researchers conducting policy evaluation studies in the context of concurrently implemented policies. Specifically, we strongly encourage researchers to carefully examine the distribution of times between enactment dates for policies being examined -e.g., by computing the length of time between enactment dates for each state. Examining the distribution of length of time between enactment dates across states will be informative regarding the potential for bias in the estimated effects of the primary and cooccurring policies. If there is sufficient time between the enactment dates of the policies (e.g., at least 3-4 years for AR model, at least 6-7 for DID model), a regression model that controls for all co-occurring policies will have minimal to no confounding. In cases when the policies were enacted very closely together, results from our study and simulation tool can be used to gauge the potential size of the bias for a particular analysis. posted June 8, 2021 Our findings also demonstrate that, in cases where the co-occurring policy is a true confounder, omitting it from the regression model will introduce confounding bias into the estimate of the primary policy. In this scenario, the effect of the co-occurring policy was essentially attributed to the primary policy. Under simulation conditions that mimic realistic settings in opioid policy research, our results indicated that the magnitude the bias for estimate of the primary policy may be as large as 82% when co-occurring policies are enacted within rapid succession of each other (0-1 years of each other). We note that in our simulation study, we generated a single cooccurring policy; in reality, it is likely that there may be multiple co-occurring policies. Conceptually, our co-occurring policy can be thought of as the cumulative effect of all cooccurring policies; as such, our findings could be interpreted as an upper-bound. Specifically, in practice, controlling for some, but not all, co-occurring policies could be expected to attenuate bias relative to our results from the misspecified model, assuming that all of these policies had effects in the same direction. We note that the strength of confounding likely varies across cooccurring policies; as we discuss below, researchers should prioritize adjustment for policies anticipated to have the strongest confounding effect.

We underscore that due to the interconnected nature of the opioid ecosystem, co-occurring policies do not need to directly target the outcome being examined to still have a confounding effect on the estimated effect of the primary policy. For example, while mandatory PDMPs are an important co-occurring policy that might confound studies of prescribing guidelines, as both directly impact prescribing, [29] [30] [31] [32] such PDMPs may also be a confounder for naloxone policies examining overdose since mandatory access PDMPs have also been shown to indirectly have a protective effect on overdose rates. [32] [33] [34] However, not all co-occurring policies are likely confounders. For example, policies likely to increase the number of buprenorphine waivered prescribers, such as hub-and-spoke policies, 35 would likely have little effect on opioid analgesic prescribing.

Additionally, our findings indicate that the correctly specified model was notably robust for all scenarios considered when fitting a linear AR model to opioid-related mortality when there is variation in the ordering of enactment of the primary and co-occurring policies across states. However, performance deteriorates meaningfully in the cases where the primary policy always comes first and/or when the length of time between the two policies is short (0-1 years; relative bias >20% for the primary policy). Intuitively this makes sense, as fixed ordering between policies essentially provides less information because the policy enacted second is never observed in the absence of the first policy. In practice, it is likely that there will be variation in policy ordering across states; however, in some contexts, adoption of one policy may consistently precede adoption of another policy (especially when the number of treated states is small).

The study has several important limitations that should be considered alongside the findings. Our simulation was relatively simplistic, representing a reasonable starting point for studying model performance with concurrently implemented policies. Future work should consider additional nuances that might impact our findings and provide additional guidance to the field. For example, it would be worthwhile to extend the simulation to consider cases where the comparison states could sometimes have only one of the two co-occurring policies. Additionally, posted June 8, 2021 it would be meaningful to explore the ability of the models to identify potential synergistic effects between the co-occuring policies and to consider additional outcomes. posted June 8, 2021 

Mostly Harmless Econometrics: An Empiricist's Companion

Designing difference in difference studies: Best practices for public health policy research

The revolution will be hard to evaluate: How cooccurring policy changes affect research on the health effects of social policies. medRxiv

What to do when everything happens at once: Analytic approaches to estimate the health effects of co-occurring social policies

Effect of Florida's prescription drug monitoring program and pill mill laws on opioid prescribing and use

Opioid overdose deaths and Florida's crackdown on pill mills

State variation in opioid treatment policies and opioid-related hospital readmissions. BMC health services research

Opioid-overdose laws association with opioid use and overdose mortality

Association between state laws facilitating pharmacy distribution of naloxone and risk of fatal overdose

The state of the science in opioid policy research

The effect of prescription drug monitoring programs on opioid prescriptions and heroin crime rates

Prescription Drug Monitoring Programs, Opioid Abuse, and Crime

What to do when everything happens at once: Analytic approaches to estimate the health effects of cooccurring social policies

The effects of recreational marijuana legalization and dispensing on opioid mortality

Opioids for the Masses: Welfare Tradeoffs in the Regulation of Narcotic Pain Medications. Cambridge: Massachusetts Institute of Technology

Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths: A Simulation Study. RR-2685-RC

Moving beyond the classic difference-indifferences model: A simulation study comparing statistical methods for estimating effectiveness of state-level policies

Application of least squares regression to relationships containing auto-correlated error terms

Prescription drug monitoring programs produce a limited impact on painkiller prescribing in Medicare Part D

Implications of prescription drug monitoring and medical cannabis legislation on opioid overdose mortality

Features of prescription drug monitoring programs associated with reduced rates of prescription opioid-related poisonings

Trends in opioid prescriptions among Part D Medicare recipients from 2007 to 2012

Prescription drug monitoring programs, nonmedical use of prescription drugs, and heroin use: Evidence from the National Survey of Drug Use and Health

The effect of prescription drug monitoring programs on opioid utilization in Medicare

The Affordable Care Act, Public Insurance Expansion and Opioid Overdose Mortality

Prescription drug monitoring programs and death rates from drug overdose

Propensity score analysis methods with balancing constraints: A Monte Carlo study

Changes in opioid prescribing after implementation of mandatory registration and proactive reports within California's prescription drug monitoring program

Opioid and non-opioid analgesic prescribing before and after the CDC's 2016 opioid guideline

The association of state opioid misuse prevention policies with patient-and provider-related outcomes: A scoping review

Prescription drug monitoring programs and prescription opioid-related outcomes in the United States

Association between state policies on improving opioid prescribing in 2 states and opioid overdose rates among reproductive-aged women

Systematic evaluation of state policy interventions targeting the US opioid epidemic

System-level factors shaping the implementation of "hub and spoke" systems to expand MOUD in rural areas