ResearchKit is flawed [just like most research studies]

(Apologies for this week’s hyper-geeky posts. I’m about to double-down now. In fact, let’s call this part of an ongoing series on the promises and perils of ResearchKit for digital health science. How’s that?)

It seems like Research Kit is a winner out of the gates. Nevertheless, in the reaction to last Monday’s announcement, a number of reports have identified a common concern: limited generalizability.

The idea here is that people who own iPhones differ from the general population (particularly from those who pocket the Android). That’s true. TL; DR: iPhone users are more educated, higher income, and less likely to be males and racial/ethnic minorities, compared to Android users.

(Note: I’ve seen lots of blogs calling this selection bias. Selection bias is a potential issue with ResearchKit, and there are lots of potential selection biases depending on what kind of study you’re conducting. However, what seems to concern people most is limited generalizability).

This is a problem. From a research perspective, it means that what we learn from ResearchKit studies will only apply to iPhone users. But, here’s the thing: most studies have this problem.

Psychological research is based on studies of undergraduates — clear generalizability issues. Our most important research fundings about health risk factors come from big studies that are rife with generalizability issues. Nurse’s Health Study recruited nurses, in part because they are knowledgeable about health, and because they are extraordinarily conscientious about research participation. Framingham Heart Study recruited patients from a single city in the Northeast (an area that is far healthier than the rest of the country). We could go on all day like this.

That said (with widespread use), I think we can mitigate some of these concerns. Here’s how:

  • Recruit large sample sizes. This creates more variability (difference) in your study sample. With recruitment potential at a global scale, this is an area in which ResarchKit can really excel.
  • Expand ResearchKit to Android (and Windows). This would expand the pool of potential study participants to non-iPhone users. If Apple handles the open-sourcing appropriately (and there’s little reason to suspect otherwise), this shouldn’t be a major problem.This one is critical for another reason. Some historically disconnected populations are not only more likely to own smartphones, but they use their phone’s advanced data-related features (e.g., text messaging, watching videos, playing games, taking pictures) more than other groups. I think this might also extend to ResearchKit apps, when they’re properly designed.
  • Targeting specific populations. There is certainly merit in ResearchKit’s ability to recruit huge samples, but there’s also potential for micro-targeting specific groups. We’ll know more in a few weeks, but I suspect that many ResearchKit apps will be distributed using Apple’s enterprise features which allow one to [largely] bypass the App Store. This might allow us to identify folks who meet particular criteria (via social media, in clinic, Mturk) and screen in/out potential participants. I’m particularly bullish on the potential of ResearchKit to help us reach people with rare diseases — this has been a persistent challenge for the research world.

Look — I’m not minimizing the challenge here and the presence of biases in existing research studies is no excuse for introducing them into new studies. Indeed, we will have to be very careful about interpreting data from ResearchKit studies — at least in the short term.