Junk Science From the Health Quality Council of Alberta

Feb. 16, 2011 - PRLog -- The Health Quality Council of Alberta (HQCA) released Satisfaction and Experience with Health Care Services: A Survey of Albertans in December 2010. As the HQCA notes on its website:

"Listening and responding to Albertans to continuously improve the quality and safety of Alberta’s health system is the mission of the Health Quality Council of Alberta (HQCA). This mission is realized through the HQCA’s legislated mandate to survey Albertans on their experience with the health system and the services it provides."

The HQCA first conducted a population-based survey in 2003 and followed this up with surveys in 2004, 2006, 2008 and now in 2010. These surveys examine overall quality, access and satisfaction with specific health services  through the eyes of Albertans who actually experienced the system identifying what is and is not working.

Unfortunately, this survey suffers from the same methodological flaws as its predecessors. These are examined in detail with regards to, The Urban and Regional Emergency Department Patient Experience Report 2009 (see: Latest Health Quality Council of Alberta Report Is Just Bad Science). Chief among these, pretending that a 38% response rate on a descretionary sample qualifies as a random sample. It doesn’t. No inferences can or should be made to the Alberta population from the results of this survey. As the QCA stated in our criticism of the earlier report:

“A lack of response violates the requirements of statistical theory and means that in principle, no inferences can be made . The problem cannot be made to go away by increasing sample size. In writing What is a Survey?, Robert Ferber, Chair and other members of the Section on Survey Research Methods of the American Statistical Association put it plainly when they wrote:

'A low response rate does more damage in rendering a survey’s results questionable than a small sample, since there is no valid way of scientifically inferring the characteristics of the population represented by the nonrespondents.' "

Thus, when the report states  that: "The 2010 response rate was 38% and the margin of error was comparable to previous years at approximately ±1.5% 19 times out of 20.", the report is guilty of flagrant overreaching.

What this means in practice can be illustrated using the first conclusion of the report. Specifically; "Based on perceptions of the quality of services, 65% of Albertans believe the quality of health care services in the province is excellent or good." Such a conclusion is pure fantasy.

There is no scientific or statistical basis for inferring that 65% of Albertans believe the quality of health care services is excellent or good. 65% of those responding to the survey believe this, but that is not the same thing as saying 65% of Albertans believe it.

Addreessing issues in health care is difficult enough without having to deal with research that  overreaches. Policy and practice decisions need to understand the limitations of information if sound decisions are to be based on that information.

# # #

The Quality Council of Alberta is a membership-based organization dedicated to the application of Quality, Deming's System's Thinking and Evidence & Knowledge-Based Management.