The FDA took a more formal step today towards considering a new framework for app and other wearable device evaluation.​ The framework, actually created outside the FDA by the International Medical Device Regulators Forum (IMDRF) offer a system to evaluate what they title "software as a medical device" - which includes apps and many wearables like smartwatches. However, the framework can be applied more broadly for almost any medical software.

The report offers a realistic view of the challenges of rating these apps stating "is unique in that it operates in a complex highly connected-interactive socio-technical environment in which frequent changes and modifications can be implemented more quickly and efficiently. Development of software as a medical device is also heavily influenced by new entrants 66 unfamiliar with medical device regulations and terminology developing a broad spectrum of applications."

The framework helps define what is and is not software as a medical device as shown in the first figure below. Based on the potential risk of the software or app, the framework outlines certain steps for review or evaluation - tailored to the risk.

The entire document is long at 46 pages but worth a read for anyone interested in this space. See the second link below.

fwww.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM524904.pdf

The transition from a paradigm in crisis to a new one from which a new tradition of normal science can emerge is far from a cumulative process, one achieved by an articulation or extension of the old paradigm. Rather it is a reconstruction of the field from new fundamentals, a reconstruction that changes some of the field's most elementary theoretical generalizations as well as many of its paradigm methods and applications.
― Thomas S. Kuhn, The Structure of Scientific Revolutions
 
 
You likely know someone who owns a wearable device like a fitness tracker or smartwatch. Offering the ability to collect data outside of the lab or hospital and now in real life and real time, wearables have become increasingly popular. One hundred and ten million of these devices are expected to be shipped in 2016. Reasons for use are varied from simple curiosity about personal health all the way to population level medical investigations. Yet recently the health implications of wearables have been placed into question by several biomedical research studies.  
 
The first study, published in the medical journal JAMA on September 20th 2016, suggests that using a wearable device was not more effective than standard behavioral weight loss approaches. The other study, recently published online in the medical journal Lancet Diabetes and Endocrinology, also suggests wearables may offer no benefit over control conditions in increasing levels of physical activity. Both studies are notable for the fact that they feature a large sample sizes observed over long periods of time, were conducted by prominent researchers, and published in prestigious journals. These articles created a media wave with stories even in the New York Times questioning the value of wearables for health. However, many have raised concerns about the methodology of these studies, arguing that that study design among other factors may have unduly influenced results. The goal of this article here is not to argue the merits or faults of these studies but rather to examine why they attracted so much attention and how the field can move forward.
 
At its core, the wearable movement values data. These devices enable the collection of every heartbeat and every step. Believing that you are active is replaced with data that you took 10,000 steps yesterday. Believing your heart medicine is working is replaced with data that your pulse has remained below a certain threshold. There is a culture shift away from belief and towards data. By quantifying our daily experiences, we have a new lens to approach health.
 
But what happens when the situation is reversed, when the data does not support wearables. Many have personal experience or strong beliefs that these devices are helpful for weight loss and increasing physical activity. Yet now data is appearing that contradicts their beliefs. The culture of data and quantification that grew with these wearable devices is suddenly turned against itself.  Proponents of wearables cite that the data collected in these studies is flawed, largely because of methodology, while those less enthused with wearables cite validation for lack of impact all along.
 
Asking each side how they know they are correct is perhaps a more useful question than whether they are correct. We already know we can easily collect large amounts of data with these devices, but how do we measure its real world impact? What measurements and feedbacks loops lead to behavioral change? How accurate are these measurements and what assumptions do we make in analyzing them? What does missingness in the data represent and how do we model it?   How does the frequency of measurement and feedback influence behavior? Are population level patterns and insights meaningful for individuals? Who can benefit the most from wearables? Many of the questions are unanswered because the pace of technology has advanced quicker than scientific methodology to measure it. Topics such as longitudinal data analysis, causal inference, and experimental design may be the new limiting factors. New methods like Agile Science for mobile health and new partnerships with data science fields are needed to make full sense of these new streams of mobile and wearable real time data.
 
Thus focusing the discussion away from what we can collect to how we collect and measure is subtle but critical paradigm shift. It involves opening up the black box of sensors magically producing results and studying the raw data itself, as outlined by professor Jukka-Pekka Onnela at the Harvard T.H. Chan School of Public Health. His focus on transparency and data reproducibility suggests a new path forward. Before debating the health merits of this new data, lets actually understand exactly what it is and how it works. How reproducible and valid are the results in the two recent wearable studies mentioned above?
 
For the proponents of wearables, it may be time to measure more than just what their sensors record. Creating the tools to collect all this new data was an impressive first step, but it is now time for the next step. Now it is time to create the experimental and analytical methodology.  As Thomas Kuhn stated in his famous book, there must be a focus on “new fundamentals” for successful scientific paradigm shifts. In the case of mobile health and wearable sensors, that may involve an embedded paradigm shift into how we measure and study wearables beyond simply what they can measure.