Perils of Personalized Treatment Effects
Episode #456: Hidden Subgroups
I greatly prefer health researchers who are obsessed with problems over those who are obsessed with solutions. The former will work away on a problem until it’s solved. They will try old things and new things. They’ll go back to the drawing board, over and over. And all along the way they will learn more and more about that particular problem. That’s a good thing. I suspect that’s how actual solutions tend to be found.
The latter, however, will often try out their solution on one problem, find that it doesn’t seem to work, and then move to the next problem…and the next…and the next…desperately searching for the problem where their solution finally happens to be a solution. They will learn little along the way, and solve less.
Of course sometimes these solution-obsessed investigators will simply maintain that their solution really does work, no matter how much evidence piles up to the contrary. One increasingly popular way of doing this is to appeal to the existence of otherwise hidden personalized treatment effects, as Benjamin highlights for us here:
So what is happening here? In a nutshell, when we run randomized controlled trials, we typically wind up with an estimate of an average treatment effect at the end. Why we are often limited to estimating average treatment effects is beyond the scope of this post (I touch on it here if you want to know more), but what it means in practice is that you can’t really tell if the average benefit you observed means everyone allocated to the treatment under test benefited similarly, or whether some people benefited a lot while others didn’t (i.e. the average of 0 and 4 is 2, but so is the average of 3 and 1).
Of course many new treatments don’t actually seem to work when we evaluate them in an RCT (sad to say, but important to understand). But the point about average treatment effects still technically stands. So what some people have learned to say in the face of a disappointing null result is, “Aha! Sure, the average treatment effect we observed in this trial was ‘null’, but we can’t rule out possibility that some patients benefited. So what we need to do now is give me even more money to go and find these personalized treatment effects that I’m super certain are really there trust me give me more money.”
So let’s entertain this idea and accept that there was a trial with a “null” finding of no average effect of treatment, but that at an appreciable number of patients allocated to the treatment under test really did benefit from it, and they are kind of hidden in the average. What does this imply? It implies that the net benefit in those patients is offset by roughly the same amount of harm in the other patients. Just as if I told you that the average of two numbers is 0 and one of them is 5…what’s the other number?
So now you are faced with two competing hypotheses. The first is that the treatment simply doesn’t do anything (with respect to the outcome at hand, and relative to the other treatment you are comparing it to) - bearing in mind that we live in a universe where most new treatments don’t seem to work. Remember, drug discovery is hard, not easy. Or you have to accept that your treatment affected patients in a manner that resulted in roughly equal amounts of both benefit and harm in your sample. Assuming you are not conflicted, this shouldn’t be a hard choice.
Finally, just to further justify my incredulous view of such “logic”, imagine that someone repeated the trial and found the same null result. What does that tell us? If you accept the possibility in the first trial that the observed null effect was due to a balance of harm and benefit among the patients, then you are accepting that there was something (or more than one something) about the patients that interacted with the treatment in a manner that produced these lurking, heterogeneous treatment effects. This implies that, to see the same null effect in the second trial, those same treatment-effect modifying factors have to be similarly present and distributed across the second sample of patients as they were in the first. Remember, trials, at least in medicine, tend to be made up of patients enrolled as they present for care. They are never* random samples of anything. This means that the prospect of winding up with two samples that reproduce the same balance of benefits and harmful treatment effects (already a stretch) is, or should be, beyond any reasonable belief.
Note: This post is not about aspirin.