We recently posted about the confusion about the diagnosis of the seven year old girl in Indonesia for whom local tests indicated H5N1 infection but where the WHO laboratory was not able to confirm it. We contrasted the news reports from AP and Bloomberg, the first of which quoted Indonesian Ministry of Health sources as alleging WHO had determined the girl didn't have the virus while the second said the tests were inconclusive, quoting WHO itself. This is a matter of bad sourcing by the AP. The Indonesian authorities are not reliable and this is a typical example.
New specimens were sent to the WHO reference lab, this time consisting of lung tissue instead of nose and throat washings. The diagnosis of H5N1 was confirmed (Reuters).
This got us wondering what it means when it is stated that "local tests" in Indonesia have indicated a diagnosis of H5N1. These tests have been accurate in the past and proved to be so again. But who does them? Using a variety of sources we have pieced together a picture and this is what we have come up with.
For many years a US Naval Research Laboratory, NAMRU2 has been in Jakarta. It is technically expert and been a valuable technical resource to the Indonesians for a long time. In 2000 NAMRU2 got caught in a diplomatic crossfire over a US arms embargo related to Indonesia's repression of the East Timor rebellion, and in December of last year the Indonesian government sent a notice to its scientists that they were no longer to work with the laboratory. Some collaborations were restored, however, and NAMRU2 has recently been important in training the Indonesians in molecular biology techniques needed to identify H5N1 with modern tools like PCR. One of the fruits of this was setting up avian flu diagnostic capabilities at the Indonesia National Institute of Health Research and Development Laboratory (Badan Litbangkes). This effort began as recently as July 2005. The Indonesians also used expertise from many other places, including Japan and Singapore.
According to what we have heard, "local tests" mean splitting a sample between the Indon lab and NAMRU2. The results are compared and with either showing a positive test a specimen is sent to a WHO reference lab for confirmation. The Indon lab sends its specimens to Malik Peiris in Hong Kong and NAMRU2 sends its specimens to CDC in Atlanta. This system seems to have worked well until a few weeks ago when the Indon lab started showing positives when the other labs didn't. We have heard that outside experts were asked to look into the problem and they determined the Indonesians weren't running positive and negative controls after each run, claiming lack of funds. Whether this is the cause of the problem we have not been able to determine. We also don't know if the problem has been solved. The Indonesians have not admitted there is a problem. We can only report what we have heard from a variety of sources.
Laboratory quality control aside, it has become abundantly clear that information from the Indonesian Ministry of Health is unreliable and should be judged with appropriate scepticism. This is a tragedy for Indonesia and a problem for the rest of us.
Comments
We have heard that outside experts were asked to look into the problem and they determined the Indonesians weren't running positive and negative controls after each run, claiming lack of funds.... Laboratory quality control aside, it has become abundantly clear that information from the Indonesian Ministry of Health is unreliable and should be judged with appropriate scepticism.
But, it was the Indonesian lab that correctly identified 7 year-old Yohanna as having been infected with H5N1. What went wrong at the WHO lab?
Posted by: Jun | June 16, 2006 7:42 AM
"This is a tragedy for Indonesia and a problem for the rest of us." I would have thought it was the other way round? Whatever.
Or maybe it's a "problem" because we can solve it? Can we?
Posted by: lugon | June 16, 2006 8:37 AM
I'm not an expert on H5N1 PCR but have been involved with the PCR of other pathogens. There are many different versions of PCR according to the sensitivity and quantification accuracy. Genetic material and other contaminates can substantially interfere with the assay, which is why one must be scrupulous with controls. Even the best of labs have been known to foul up from time to time. Does anyone know if the statistics on H5N1 PCR variants have been published, such as the PPV--positive predictive value?
Posted by: Marissa | June 16, 2006 8:44 AM
Revere, I am confused. Several weeks ago the same thing happened. The local test (which in the past has refered to the navy lab I believe) was positive and sent to Hong Kong. Hong Kong said it was negative, which was not credible, since the 18 year and a sibling had both tested positive locally and the sibliing was also positive in Hong Kong. Thus, the liklihood that a sibling of a twice confirmed H5N1 positive would have symptoms leading to death and be a false positive locally was EXTREMELY unlikely (and Hong Kong reversed its negative and called both siblings positives)
http://www.recombinomics.com/News/06160601/H5N1_Indonesia_Surveillance_Concerns.html
The latest case is another example. This time the sibling died with bird flu symptoms and the second member of the cluster was positive locally. Hong Kong said no, and then said yes, indicating that the initial samples was nose and/or throat, followed by lung which was positive.
Thus, Hong Kong is batting zero (0/2) and the local test is 1000 (2/2) in the descrepancy category.
There seems to be a problem in Hong Kong, coupled with WHO's desire to call negatives in clusters, even when the liklihood of such pronouncements being correct is EXTREMELY low.
This also happened earlier in a Lampung cluster. Two relatives were positive last fall and Hong Kong said the third relative was negative, even though two family members were confirmed in Hong Kong. Four months later, Hong Kong confimred the third member (who was initially sent home with a diagnosis of typhus). Now all three are "official" confirmed WHO cases and form a cluster with the characteristic time gap indicating H2H.
Thus, the WHO/Hong Kong batting average on the clusters is really 0/3. The earlier example may have fallen into the "news management" category. When the first results came out, the size of the cluster was 2 since the 3rd member was negative. The when the third was confirmed, the cluster was still 2 because in the February WHO update confirming the 3rd case only mentioned confirmation of one family member. Thus, no WHO update acknowledges that there were three confirmed cases that had all of the hallmarks of H2H.
Now after WHO has issued countless denials of H2H, including Lampung, guess what they are doing with this earlier cluster of three in Lampung (originally described last fall)
http://www.recombinomics.com/News/10240503/H5N1_Lampung_Cluster_Confirmed.html
Posted by: Henry Niman | June 16, 2006 8:44 AM
Jun: I don't know if it was the Indon lab or NAMRU2 (or both) that correctly identified the little girl's diagnosis. My impression is that the sample sent to the WHO lab (nose and throat washings) was not sufficient for confirmation which only came when actual lung tissue was sent, presumably from an autopsy.
lugon: Good question although a parsing of my diction that my diction probably can't sustain. One of the purposes of posting on this is that it might help solve the problem to some extent. It is also possible it will make it worse. We think acknowledging it is better than not acknowledging it, however.
Posted by: revere | June 16, 2006 8:47 AM
Marissa: Minor technical point. PPV depends on sensitivity, specificity AND prevalence, so it is not a measure of a test's accuracy but it's conditional accuracy. The accuracy of PCR itself is given by the sensitivity and specificity. The PPV may be strongly influenced by the actual proportion of true positives in the sampling frame the PCR is applied to.
Posted by: revere | June 16, 2006 9:16 AM
Forgive me for this off-topic thought:
Although this new site may have some advantages, the graphic design is far inferior to your previous site. It is painful to look at and uncomfortable to read.
Posted by: Helena | June 16, 2006 11:05 AM
Helena: Sigh. I realize there are some difficult adjustments in this treansition period (for me, too). One thing that might help is to switch to Firefox as a browser if you are using IE. It makes a huge difference. The publishers also say they will be changing the appearance of individual sites. I haven't had time to get a nice banner yet and the grey borders will be changed.
Changing the appearance of a site that you visit a lot is very disorienting I know. With any luck this will evolve. H5N1 can do it, why can't we?
Thanks for the feedback, though (this is not meant sarcastically!).
Posted by: revere | June 16, 2006 11:36 AM
Henry: According to my info, if NAMRU2 does it, it goes to CDC, not Hong Kong. So if Hong Kong got it that would indicate it came from the Indon lab. However as far as I know the "local tests" and the confirmations have been tracking quite well until this latest case. You may know different.
Peiris's (Hong Kong) reputation is that he is very meticulous and he doesn't fudge data, although he is very conservative and may not confirm unless he is very sure. It is not my impression that his judgment is influenced by what WHO or the Indonesian gov't wants, but that's not based on hard evidence, only what I've heard. That's all I know.
Posted by: revere | June 16, 2006 12:11 PM
Revere: Any thoughts on York Chow's suspicions that H5N1 may have become more virulent and more widespread?
Posted by: Aileen | June 16, 2006 2:19 PM
Aileen: Saw it. Good example of misuse of the word virulent (for transmissible), but that's a common error. The folks in HK have the SARS experience as backdrop which was really quite terrifying for the medical people involved. Not clear if this is meant to punch up their border people or is a genuine concern that the exposure in this case is more casual than many others. As I recall it was also a truckdriver from Shenzhen that brought SARS.
So far I don't see anything special about this case unless they know something they are not revealing. My hunch is not.
Posted by: revere | June 16, 2006 2:56 PM
Revere, yes, you're right about the PPV, which in this case makes it harder. It would be nice if someone pulled all the PCR data together and reviewed it; we might better understand why we have these constant yes/nos on H5 confirmation. As Henry indicated, something's not right with someone's lab here.
Posted by: Marissa | June 17, 2006 8:54 AM
In the UK we managed to shoot our selves in the foot by proudly announcing that we were pretty much AI free, in fact so free that the background levels of LP AI were so low as to be unbelievable probably due to poor sample collection and storage. Could the test discrepancy not just be a case of sample degradation in transit to HK?
Posted by: JJackson | June 17, 2006 11:37 AM
JJ: That's my guess, too. The nose and throat washings may not have had much virus and it didn't make it. Another aspect of lab quality assurance is how they keep, store and send their samples.
Posted by: revere | June 17, 2006 11:54 AM
Would this have any effect on the Cambodian seroprevalence studies (and others) that found little to nothing?
Anon_22 cites in the comments 'Serological tests on 446 close family contacts of the first 12 human H5N1 patients in Indonesia showed only 2 positive' and (for Cambodia) '...villagers in households near the H5N1 case's household were asked about exposures to poultry and poultry products in the past year and tested for H5N1 antibodies by microneutralization'. What is microneutralization? Does this imply anything about the sampling method or is this just the test method?
Posted by: Dizzy | June 17, 2006 7:19 PM
Dizzy: These are different labs, so it doesn't pertain specifically. A neutralization test is a test to see if there are antibodies in the serum of the subject. They test to see if the patient's serum can neutralize the infective ability of the virus to infect. Thus it is a test method for antibodies, not a sampling method.
Posted by: revere | June 17, 2006 7:30 PM
Thanks revere. I asked because it occurred to me that even if the labs were different they could still make the same mistake with the same sampling method.
Posted by: Dizzy | June 17, 2006 7:51 PM