ResearchKit is flawed [just like most research studies]

(Apologies for this week’s hyper-geeky posts. I’m about to double-down now. In fact, let’s call this part of an ongoing series on the promises and perils of ResearchKit for digital health science. How’s that?)

It seems like Research Kit is a winner out of the gates. Nevertheless, in the reaction to last Monday’s announcement, a number of reports have identified a common concern: limited generalizability.

The idea here is that people who own iPhones differ from the general population (particularly from those who pocket the Android). That’s true. TL; DR: iPhone users are more educated, higher income, and less likely to be males and racial/ethnic minorities, compared to Android users.

(Note: I’ve seen lots of blogs calling this selection bias. Selection bias is a potential issue with ResearchKit, and there are lots of potential selection biases depending on what kind of study you’re conducting. However, what seems to concern people most is limited generalizability).

This is a problem. From a research perspective, it means that what we learn from ResearchKit studies will only apply to iPhone users. But, here’s the thing: most studies have this problem.

Psychological research is based on studies of undergraduates — clear generalizability issues. Our most important research fundings about health risk factors come from big studies that are rife with generalizability issues. Nurse’s Health Study recruited nurses, in part because they are knowledgeable about health, and because they are extraordinarily conscientious about research participation. Framingham Heart Study recruited patients from a single city in the Northeast (an area that is far healthier than the rest of the country). We could go on all day like this.

That said (with widespread use), I think we can mitigate some of these concerns. Here’s how:

  • Recruit large sample sizes. This creates more variability (difference) in your study sample. With recruitment potential at a global scale, this is an area in which ResarchKit can really excel.
  • Expand ResearchKit to Android (and Windows). This would expand the pool of potential study participants to non-iPhone users. If Apple handles the open-sourcing appropriately (and there’s little reason to suspect otherwise), this shouldn’t be a major problem.This one is critical for another reason. Some historically disconnected populations are not only more likely to own smartphones, but they use their phone’s advanced data-related features (e.g., text messaging, watching videos, playing games, taking pictures) more than other groups. I think this might also extend to ResearchKit apps, when they’re properly designed.
  • Targeting specific populations. There is certainly merit in ResearchKit’s ability to recruit huge samples, but there’s also potential for micro-targeting specific groups. We’ll know more in a few weeks, but I suspect that many ResearchKit apps will be distributed using Apple’s enterprise features which allow one to [largely] bypass the App Store. This might allow us to identify folks who meet particular criteria (via social media, in clinic, Mturk) and screen in/out potential participants. I’m particularly bullish on the potential of ResearchKit to help us reach people with rare diseases — this has been a persistent challenge for the research world.

Look — I’m not minimizing the challenge here and the presence of biases in existing research studies is no excuse for introducing them into new studies. Indeed, we will have to be very careful about interpreting data from ResearchKit studies — at least in the short term.

ResearchKit looks like a winner [right now]

Here’s a researchers dream: Wake up one morning and find that 11,000 people have signed up for your latest study.

“To get 10,000 people enrolled in a medical study normally, it would take a year and 50 medical centers around the country,” said Alan Yeung, medical director of Stanford Cardiovascular Health. “That’s the power of the phone.”

Here’s a researchers nightmare: Losing 80% of those 11,000.

Out of the gate, ResearchKit appears to be a smashing success. However, there’s a problem — with most mobile apps (particularly those that are commercialized by the download or rewarded for large user bases), the crucial question is “if you build it will they come?

The problem is that with research, particularly longitudinal research studies, there’s another [much more important] question: “if you build it, will they stay?

Hyperbolic live blogging Apple’s ResearchKit

It’s always dangerous to post live comments during an Apple live event, particularly if you’re a rabid early adopter, fan-person admirer of Apple products, but oh well…

ResearchKit is an absolute gamechanger for health/medical research. It has potential to be the best thing to happen to behavioral research in a generation.

My real-time almost certain to be amended thoughts (in no order whatsoever):

  • ResearchKit will be open source. That’s great for all of the usual reasons. But it’s a savvy business move. This ensures less friction for integrating ResearchKit applications in National Institutes of Health grants. Counterintuitively [for those who don’t attend to these things], it will also help ease concerns about privacy.
  • We all struggle with patient recruitment, particularly when we don’t see them in clinic. Some of the biggest problems: finding people, recruiting, consenting, paying, and retaining them. Problem solved greatly mitigated.
  • ResearchKit might open a new market for study discovery and participant recruitment.
  • It its promotional materials, Apple is positioning ResearchKit for observational data collection. For this to work with intervention science, we’ll have to build ResearchKit hooks into health/medicine apps. It will be interesting to see what APIs Apple makes available. If Apple history is a guide, don’t expect this to happen right away.
  • If Apple allows ResearchKit to hook into non-resarch apps, watch out. Aside from cool new data, the commercial market for data aggregation will explode.
  • There is potential for changing the way that we run big cohort studies (e.g., Nurse’s Health Study, Jackson Heart Study, CARDIA, Framingham). Will it be cheaper to send every study participant an iPhone, versus the usual approach of creating, sending, scanning, and collating data from paper surveys? Probably. Incidentally, the National Institutes of Health has been funding fewer of these cohort studies, likely given resource constraints. Time to beef up on those epidemiology skills.
  • The ability to collect contextual data is going to be “great.” Beep. “We see that you’re inside Big Jo’s Burger Barn. How many minutes do you think it will take to burn that burger off?” Get ready for new science on in-vivo data collection.
  • We don’t yet know how ResearchKit will integrate with Apple Watch, but there is great potential for integrating new health metrics [particularly as Apple enhances Watch’s sensors].
  • Yes, some people will freak about the idea of a researcher collecting data from their Snapchatting device. There will be at least 1200 blog posts on the topic this week alone. I think that’ll be a short term problem though (how many cameras are pointed at you right now?).
  • That fancy new research data collection platform we’re creating [as I write]? History.