'Snapshot of health' is really a warning about statistics

The county-by-county health "snapshot" of Idaho, issued with great fanfare about three weeks ago, is -- to put it bluntly -- bogus.

State health officials said they were hoping the statistical data in the snapshot, and the county rankings for various health issues, would serve as a starting point for a discussion on health care and health needs in each county.

That's a laudable goal, something that is genuinely worthwhile.

My problem with this "snapshot," which listed Elmore County's health status as 24th among 44 counties in the state, is that the numbers are likely to be set in stone, despite the fact they are often inaccurate. If you're going to have a discussion and start using statistical data as points to discuss, it would be nice if the numbers were accurate.

But they're not, and when I challenged a number of health-care officials at the state level and in Elmore County, it didn't take long for all of them -- literally all of the ones with whom I talked -- to quickly admit that the numbers, generated by midwestern researchers, probably aren't accurate.

To be honest, newspapers and politicians use statistics like this all the time, and rarely do we question them or look at the methodology behind how they were generated.

But I decided I needed to do that after I sat down to write a story based on the numbers released. I quickly spotted a couple of numbers claiming to represent Elmore County that I knew were wrong because I'd done stories on those issues in the recent past and the numbers I used then, generated locally, didn't match the numbers in the "snapshot."

One of the numbers that leaped out at me was the county graduation rate. The actual numbers, reported to the state, for students entering ninth grade and completing graduation are much, much higher than the 56 percent listed in the report.

Elmore County is a relatively poor county, as are many inner city areas, and that number was developed based on national averages, which include some inner-city school districts in major metropolitan areas where the drop-out rate is 80 percent or better. So adjusted demographically, that number might have been about right -- but it isn't even close to the actual number for Elmore County.

Last year, according to reports on file with the state, the Mountain Home School district had a graduation rate of 94.74 percent. Nor did Glenns Ferry pull the number down. In a district where one dropout can sometimes represent a couple percentage points all by themselves, the Glenns Ferry School District has consistently, over recent years, had graduation rates above 90 percent.

That rate was listed in the health "snapshot" because people with better educations tend to be healthier than those who don't have a good education.

But then, of course, since Idaho functionally no longer has compulsory education, those numbers apply only to kids who were in school in the ninth grade (if you don't require anybody to report to anyone that they're home schooling their kids, don't require them to submit a curriculum and don't require them to take any standardized tests to see what they've learned, you no longer have compulsory education).

So I started looking in detail at the methodology used in developing the numbers in this well-publicized county-by-county health "snapshot" of Idaho.

In almost every case, the numbers are based on national averages, adjusted for the demographics of Elmore County. To be fair to the researchers, using national statistics and surveys, and then adjusting for local demographics, was about all they could realistically hope to do. Trying to get the hard data from every county in the nation would have been a task of Herculean proportions.

But as far as the actual accuracy of their numbers goes, they were little more than wild-assed guesses (WAGs) pulled out of thin air -- and that might be giving a bad name to thin air.

For example, one of the numbers listed on the report summary showed 15 percent of the county's population engages in binge drinking.

But when you get into the methodology, you realize it's based on a national average and the margin of error ranged between 11 percent and 21 percent. That's almost a 100 percent difference between the high and low numbers! In my book, that constitutes a WAG.

Furthermore, it was based on a telephone survey by a federal agency, which called far fewer households than local polling companies say are needed to obtain a demographically accurate picture (and telephone surveys in recent years have had increasing problems because they generally use only land-line telephones and more and more people, in particular in the younger demographic groups, have opted out of land lines in favor of cell phones).

Applying national statistics to Elmore County doesn't work real well because Elmore County's demographics are a little odd, for at least two major reasons -- the presence of the airbase, which gives us an abnormally large number of "residents" between the ages of 18-21, and our nearby proximity to the state capital.

One of the statistics involved the teenage pregnancy rate. This represents the number of live births per 1,000 females in the county. Once again, it's based on national averages, adjusted demographically. Based on the number listed in the report, that should be just over 150 births in Elmore County to teenage mothers, out of just an estimated under 600 total, or roughly a quarter of the total number of the births listed in Elmore County.

Who knows? That number might be about right. Or not. Officials at both the school district and the hospital say they believe the teen pregnancy rate is a serious problem, with the hospital saying their stats from the state indicate Elmore County is about third in the state in teen pregnancy rate.

But here's why you can't trust any of the "hard" numbers used in the "snapshot," or for that matter even some of the local hospital and state numbers.

The number listed in the "snapshot" is based on a national average of girls ages 14-19. But there's an awful lot of 18- and 19-year-old married couples out on base.

I remember when I first moved here, I thought this was the most pregnant town I'd ever seen, until somebody pointed out to me that the young couples on base tend to start their families there early while they're covered by military health care. In fact, pregnancies are THE boom business for the base hospital, which has considerably more births than those recorded at Elmore Medical Center, the hospital in town.

The actual number of births to teenage mothers at Elmore Medical Center is 24 out of 102 births in 2007, 41 out of 178 in 2008, and 40 out of 160 in 2009. So the estimate in the report, in terms of percentages, is pretty close -- about one-fourth of all births, assuming that the births at the local hospital probably didn't involve military personnel (which seems to be a reasonable assumption) -- although the actual number the report lists is a little high. The most recent data available from the Bureau of Vital Statistics was for 2007, and that showed a total of 539 births for all ages of mothers in Elmore County. That number includes births at the base hospital.

But even when trying to use the state's "hard" statistics, rather than the estimate in the report, the numbers for Elmore County are still skewed because the state stats are based on where the birth certificate was issued, that is, where the child actually was born. A lot of people, especially those with a little better means and therefore a little older, opt to have their children born at hospitals in Boise (or Gooding, Twin Falls or Jerome counties if you're on the Glenns Ferry side of the county). For example, the families of my grandchildren all live in Mountain Home, but almost all of the grandkids were technically born in Boise.

So any statistic like that rapidly loses meaning because it's impossible to figure out how many people in Elmore County, of any age, gave birth in a given year. It's absolutely worthless to be "used as a starting point for a discussion" on teen pregnancies (which we generally think of as kids in high school) or even health care in general.

The various death rate numbers used in the "snapshot" also are horribly skewed.

One of the numbers used in the "snapshot" is the number of man-years lost for "premature deaths" -- people who died younger than age 75. I'm not sure why the study's authors (the Robert Wood Johnson Foundation and the University of Wisconsin Population Health Institute) used that break point, since the average life expectancy in the United States is a little over 78 years of age.

They said Elmore County lost nearly 6,700 man-years due to "premature deaths." But when I went and got the Idaho Bureau of Vital Statistics report for Elmore County, and looked at the actual breakdown of the most recent numbers available, then added them up, it only came to a little over 1,350 man-years lost, or only about 20 percent of what the report said. Nor did the report take into account the man-years "gained" by people who live over the age of 75 (and we've got an unusually high number of centenarians in this county), which it seems is a countervailing "good" number that should have been included to balance out the "bad" number.

In fact, according to the county coroner's office, the number of death certificates issued in Elmore County has actually been declining over the last few years, with 206 issued in 2007, 196 in 2008 and 184 in 2009, despite the fact the county population has been going up.

Yet even those "hard" numbers can't be trusted, and the report's motor vehicle accident death rate listed for the county is a good example of why.

The number listed in the "snapshot" -- 23 -- is based on national statistics per 100,000 population. We have just under 30,000 people in Elmore County, so adjusting the number used on the summary cover of the "snapshot" report, we should get about 7 fatalities a year due to auto crashes.

In fact, based on a story we did in 2008, we have one of the highest fatality rates on the interstate (alone) of any county in the state, more than double and nearly three times the state average by mile of interstate. But, oddly enough, the vast majority of those deaths in the last five years have involved out-of-county or out-of-state residents, so it doesn't really reflect the "health" of county residents at all.

The actual numbers of traffic fatalities, over the last five years, have ranged widely between a low of 5 and a high of 21. That's people officially listed as having died in traffic accidents here.

But even that number, which is based on the actual recorded deaths due to that cause in Elmore County, is still wrong.

Those numbers reflect only those people who were pronounced dead at the scene or before they could be transported to a hospital in Boise. An official at the sheriff's office told me he estimated the actual deaths were probably double what was listed, because so many of those people "officially" died in Ada County, which is where the death certificate would then be issued.

Elmore County, like a lot of rural counties surrounding Ada County, doesn't have a trauma center, or for that matter a cardiac center for heart attacks, so people with major traumas or heart attacks get sent to Boise where they "officially" die. If anything, this has got to skew Ada County's death rate numbers horribly. But what it means is, even when you have "hard" numbers, it's tough to pin down just what the actual death rate of Elmore County citizens is.

My ol' pappy used to tell me: "Figures don't lie, but liars figure."

You can play all kinds of games with statistics. I remember, in college, my statistics instructor offering a warning about the use of statistics by showing that he could "prove," statistically, a direct correlation between the number of stork sightings in Denmark and the number of births there.

I have no problems with a discussion on health care. It's clearly needed, both at the local and national level. But if we're going to do it, we've got to get away from using numbers that are pure WAGs to support our arguments, which is essentially what this famous "snapshot" really is, and what a lot of the national health care debate is about.

I recently read an article from one of the nation's top newspapers where one of the men involved in the Congressional Budget Office analysis of the proposed health care legislation was interviewed. He readily admitted they were guessing at the numbers. Nobody really knows what the bill will actually cost, he said, but everybody is using their numbers.

He pointed out that several years ago, when Congress changed some of the Medicaid benefits programs, his office was widely criticized for the low number it had projected on the costs impact. But that number turned out to not only have been the best guess anybody had floated around at the time, but was still much higher than the real number once the benefit changes took place and the actual numbers were analyzed. Seems more seniors started using generic prescription drugs than anybody had anticipated, so the costs were lower than projected.

So be careful of any "statistics" you see. We all need to check out the methodology first (which usually means you have to have at least taken a course in statistics at some time in your life to understand that), consider all the mitigating factors, and then you still need to take the numbers with a grain of salt.

As I've shown with the "snapshot" of health care in Elmore County, taking national statistics and adjusting them, demographically, for Elmore County, doesn't take into account the unique characteristics of this county and even the "local" numbers can be suspect. I'm sure other counties in the state have similar unique situations that make their snapshots equally invalid.

So when it comes to that report -- buyer beware.