DSS’s new patient survey has little to say

DSS’s consultants reported on the results from their new patient experience/satisfaction survey for PCMH Plus members at this month’s MAPOC Care Management Committee meeting. The survey is important as the experimental PCMH Plus payment model risks inappropriately denying needed care and cherry-picking more lucrative patients. The main result, revealed in answer to a question, is that the new survey’s response rate is just ten percent.

Ten percent is a very low response rate. It means the researchers missed input from 90 percent of the HUSKY members they contacted. There is ample evidence that surveys with response rates below 70 percent can skew results and are unreliable. The US Census’s response rate for their ACS survey has been over 90 percent. Some journals require 80 percent or higher response rates for publication. The average commercial survey response rate is 33 percent.

DSS’s switch to a new, physician-centered survey of patients has been controversial. Last September, DSS announced that they had abandoned the gold-standard CAHPS patient survey used by 46 other state Medicaid programs and the federal agency that oversees Medicaid, for a new one focused on patients’ happiness with their primary care physician. Ten of the eleven questions in DSS’s new survey ask patients for feedback on their “doctor” or “practice”; there are no questions about health outcomes. But CAHPS asks the right questions – if patients got the care they needed and if they are healthier. The CAHPS survey was developed in 1995 by independent researchers to evaluate healthcare programs and care delivery. Over 400 studies have verified its value in assessing whether healthcare programs and providers are helping patients heal and stay well.

Ironically, the reason DSS gave for switching from the CAHPS survey that has served Connecticut and other states well for so long, was the expectation that the new shorter survey will improve response rates giving better information. CAHPS response rates are between two and four times better than DSS’s new survey. So that clearly didn’t work out.

Parsing the results of DSS’s new survey, there isn’t much information to act on. Not surprisingly, the survey found that both PCMH Plus and control group HUSKY patients generally like their doctors. Differences between PCMH Plus and control group members were small and likely not significant given the very low response rate. Patients who’ve been with their doctor over ten years are more satisfied than those with shorter tenure. This is to be expected – if you’re happy with your doctor, you’re more likely to stay; if you aren’t happy, you’re more likely to switch. There were differences in satisfaction between health systems, but without those response rates, it is unclear if they are meaningful.

It’s unfortunate that a critical source to evaluate whether PCMH Plus patients are getting good care and whether taxpayers are getting good value has been lost. Advocates called on DSS to return to using CAHPS as soon as possible – for better understanding patients’ needs, to identify gaps, and to compare performance to other states, allowing Connecticut to improve care and maximize scarce resources.