Measuring Quality in Small Practice Settings

Health care quality pioneer Don Berwick has shared with audiences for years Garrison Keillor’s famous observation that "everyone thinks they are above-average.” The insight draws a laugh every time. Everyone thinks they have above-average driving skills. And their looks and intellect are above average, and of course, so are their kids.

People laugh because they know it can’t be true, even at Lake Wobegon. The moment one decides to assess any characteristic of members in a population, one can be assured of finding a distribution of results and depending on the nature of the distribution, many members—sadly—turn out to be below average.

Such is the case in health care. In fact, Jack Wennberg’s seminal observation of marked variations health care quality and spending have spawned an entire academic industry in which hundreds of investigators reveal new examples of care variations every year. To this day the research draws immense interest from consumers, payers and policy wonks.

To draw meaningful conclusions in such studies, investigators must accumulate sufficient sample sizes. Otherwise, they cannot rule out the possibility that apparent differences in performance are due to chance alone.

This turns out to be easy in hospital settings, where volumes are high. But a study in this week’s JAMA suggests it’s not so easy for small group practices, where the majority of decisions actually get made in the US health care system.

In the study, Nyweide and colleagues set out to determine the practice caseload size that is necessary to detect significant differences in performance with respect to several common measures of health care quality and cost.

The measures included mammography screening rates for women aged 66 to 69 years, hemoglobin A1c testing rates for 66-75 year-olds with diabetes, preventable hospitalization rates, and 30-day readmission rates after discharge for congestive heart failure.

Using a 1-year caseload, Nyweide and colleagues found that essentially no practices having less than 6 PCPs saw enough patients to reliably detect a 10% relative difference in costs or in any quality measure.

Only 9% of practices having 6-10 PCPs could reliably detect a 10% relative difference in costs, and less than 3% of these practices could do so for mammography or hemoglobin A1c testing, for example.

What to Do?
Nyweide’s study suggests that a complete overhaul may be required in the way Medicare and other payers think about performance assessment in ambulatory care settings.

In considering how to design a better approach, I suggest that three seemingly reasonable methods need to be avoided, but that two other methods might actually work. The three to avoid are:

Pooling results from multiple, currently used measures of ambulatory care practice. This won’t work because a physician's performance on one quality measure correlates poorly with her performance on others. Pooling data in this instance risks masking deficient performance in particular areas.

Assessing performance over a period of time longer than one year. This would be prohibitively expensive and fail to pick-up recent changes in physician performance (although 1-year moving averages, as are sometimes used in reporting stock market performance might mitigate this problem).

Combining data from loosely-affiliated physician groups (like IPAs or PHAs) into one statistic. With rare exceptions, these “virtual groups” contain individuals that do not work together to improve quality or control costs, nor do they share responsibility or risk for care outcomes or cost controls. As with the first item mentioned above, such aggregations will mask deficient performance on the part of certain physicians.

Here are two methods that might work:
Pool patients from all payers, not just Medicare. This achieves increased sample size with no downside. This approach will become easier to do since private insurers are increasingly adopting the PQRI metrics originally promulgated by Medicare to drive their pay for performance schemes. Health reform legislation currently winding its way to President Obama’s desk should include provisions requiring private insurers to adopt PQRI.

Develop diagnosis-independent measures of clinical process performance. Diagnosis-specific measures like mammography screening are always going to be plagued the sample size problem. Thankfully, PCPs execute many clinical processes that are independent of patient diagnosis and highly reflective of the quality of care. These include updating medication lists, inquiring about cigarette smoking and referring smokers to cessation programs, asking sexually active people about birth control and STD prevention, etc.

Because such activities are diagnosis-independent, the sparse-data problem discussed by Nyweide is avoided.

Glenn Laffel MD, PhD
Sr. Vice President, Clinical Affairs, Practice Fusion