2. “Defending a Knowledge Hierarchy in Forensic Science” by Simon Cole

View PDF

May 6th, 2012

I am grateful to Professor Roger Koppl for taking the time to respond so thoughtfully and in such great depth[1] to my contribution to a special issue of the Fordham Urban Law Journal[2] occasioned by the publication of the National Academy of Science (NAS) report Strengthening Forensic Science in the United States: A Path Forward.[3]  I will try, as best I can, to respond to his points. At the same time, however, I will endeavor to be restrained in my response.  Readers should keep in mind that the occasion for this exchange is the “moment of opportunity” for the reform of forensic science created by the publication of the NAS report.  There is so much about which Professor Koppl and I agree that I would not want rather nuanced disagreements to overly distract readers from the overall need for reform about which Professor Koppl and I have no disagreement whatsoever.

My Article was centered around the NAS report’s claims that there was something amiss in forensic science, that the failure of something called “scientific culture” was at the root of the problem, and that the adoption of something called “scientific culture” was the key to the solution.  I argued that the NAS’s discussion of scientific culture lacked nuance and equated all scientific activity with what I called “discovery science”[4]——an equation that, incidentally, is resurrected in the penultimate paragraph of Professor Koppl’s Response.  I proposed to improve upon the NAS’s discussion of forensic science by positing a more nuanced view of forensic scientific work divided into five general task categories.  I argued that a failure to distinguish these categories had played some role in generating the confusions and controversies that have beset forensic science over the past couple of decades.  For example, the dispatching of practitioners to answer queries about the validity of their own practice yielded absurd responses, such as the “fingerprint examiner’s fallacy,”[5] the “individualization fallacy,”[6] the “casework fallacy,”[7] the “zero error rate,”[8] and the like.

Professor Koppl does not seem to disagree greatly with the above analysis.  As is typical of these sorts of things, it was my proposals for reform that conclude the paper that generated the greatest disagreement.  At the most general level, I merely proposed that efforts to reform forensic science consider some task differentiation, rather than treating “forensic science” as a monolithic entity.[9]  Again, there is little disagreement over that.  But I went further.  I suggested an unabashedly “hierarchical” model in which individuals assigned certain high level tasks would exercise control over the claims made by individuals assigned lower level tasks.[10]  What I had in mind, in concrete terms, was that line practitioners would no longer be expected to, or even permitted to, validate their own practices.  Validation work would be expected of and performed by people with training primarily in conducting scientific research, rather than of people with training primarily in conducting forensic assays.  I then offered medicine as a loose analogy for the sort of organizational structure I had in mind, an analogy that has been used by others as well.  Here is where Professor Koppl disagrees.  I will try to respond to three key issues below.

I.  The Medical Analogy

Because I analogized my proposed organizational structure with the current structure of medicine, Professor Koppl seeks to counter my proposal by undermining what he portrays as my implicit assumption of the virtue of modern medicine’s organizational structure.[11]  Thus, he challenges the notion that medicine’s organizational structure was chosen rationally or democratically, is in fact hierarchical, yields “good” research, or yields good outcomes.[12]

My main response is that Professor Koppl has read too much into my use of medicine as an analogy for thinking about the organizational structure of forensic science.  My use of medicine as an analogy was by no means intended as a claim that medicine has a “correct” organizational structure, that medical knowledge is always correct, or that health care is delivered efficiently.  Though I am not an expert on medicine or on health economics, I am aware enough of the deficiencies of modern medicine, especially in the United States, on all these fronts.  I am neither equipped nor inclined to defend the knowledge production or economics of modern medicine.  My assertions about medicine were intended as casual observations, rather than empirical claims about the rationality of either medical knowledge or delivery.  I believe that my words conveyed this casualness, “society is reasonably content with the hierarchical model in medicine.”[13]

What I meant to convey was merely that the medical model made a certain amount of intuitive sense compared with the current forensic model.  The organizational structure of medicine does not ask radiological technicians to defend the validity of knowledge claims deriving from radiology.  When I wrote that, “society is reasonably content with the hierarchical model in medicine,” I meant to convey that society is reasonably content with this situation, not content with all of medical knowledge and delivery.  In other words, I do not perceive a groundswell of public support for the notion that clinical trials on radiological knowledge claims should no longer be conducted by biomedical researchers, but should instead be delegated to radiological technicians.  The current organizational structure of forensic science does precisely that, and that makes no sense at all, as I believe Professor Koppl would agree.  I was not intending to make a strong empirical claim about the rationality of medical knowledge or delivery.[14]

In any case, the medical analogy was intended to be merely that: an analogy.  I do not think it is necessary for me to defend the validity of all medical knowledge or the efficiency of all health care delivery to support my use of medicine as a model for rethinking the organizational structure of forensic science.[15]

II.  Capture of NIFS

Readers should recall that the first recommendation of the NAS report was the creation of the National Institute of Forensic Science (NIFS), independent of law enforcement.[16]  Professor Koppl charges that my proposal would create a regulatory institution (NIFS), “captured” by law enforcement with control over forensic science.[17]  I have two responses to this.  First, this charge slightly confuses a knowledge hierarchy with a regulatory hierarchy.  The essence of my argument was that individuals with training in conducting scientific research (whom I called “Basic Researchers”) should be the only individuals empowered to——or expected to——validate or evaluate forensic knowledge claims.[18]  In this sense, they are superior in a knowledge hierarchy to individuals whom I called “Analysts,” who should not be permitted to——or expected to——validate or evaluate their own practices.[19]  But a knowledge hierarchy does not necessarily imply a regulatory hierarchy.  Basic Researchers need not necessarily be employees of NIFS, or even funded by it.  “While some of them might have ‘official’ positions in forensic science (e.g., NIFS scientists), others might be independent of forensic science and employed by universities, industrial corporations, or non-government organizations.”[20]  My argument was about education, training, and research orientation, not about regulatory oversight.  To draw on the medical analogy again: any credentialed medical researcher can evaluate a medical knowledge claim, whether employed by the National Institutes of Health, a university, or a pharmaceutical firm, but radiological technicians and practicing pharmacists are not generally trusted——or expected——to evaluate medical knowledge claims.

My second response is perhaps simpler.  An NIFS captured by law enforcement is not an NIFS.  The NAS was admirably clear on this point: the NIFS must be independent of law enforcement.  The creation of any institution to regulate forensic science that is not independent of forensic science is not an implementation of NAS recommendation #1,[21] but rather a deception, co-option, or, to use Professor Koppl’s term, “capture.”  I am second to none in my abhorrence of such a development, and I accept no responsibility for the pernicious outcomes that would result.  In short, I entirely agree with Professor Koppl that a law enforcement-controlled “NIFS” would have pernicious effects, but these effects have nothing to do with my proposal, which concerns a knowledge hierarchy based on education, training, and orientation and presupposes an independent NIFS.

Professor Koppl is, of course, right to be concerned about capture, a point I tried to make in my Article.  Recent activity in Congress suggests strong interest in creating precisely the kind of institution both Professor Koppl and I are warning against: a law enforcement-controlled forensic science oversight institution.[22]  How to prevent this perverted outcome is a political problem for which I do not feel I have any special expertise in proposing a solution.  However, I do feel that the entity with the greatest power to forestall such an occurrence is the NAS itself.  As a mainstream scientific institution, the NAS has unique authority to speak about the regulation of forensic science.  A strong statement from the NAS that a law enforcement-controlled oversight institution is fundamentally incompatible with the NAS’s recommendation would be perhaps the most persuasive argument against the creation of such an institution.

III.  Koppl’s Free-Market Adversarial Proposal

Professor Koppl closes by proposing his own solution to the ills of contemporary forensic science, which he has articulated at greater length elsewhere in an elegant series of papers.  To greatly oversimplify, Professor Koppl, like a true economist, seeks to harness the power of competition to improve forensic outputs.[23]  His argument links together strands from three schools of thought: free-market economics, legal adversarialism, and Popperian philosophy of science.  All of these lead to a similar familiar conclusion (hence Popper’s own linking of them in The Open Society and Its Enemies[24]): competition between ideas is the best way of ensuring their general excellence.  At the practical level for forensic science, Koppl is proposing competition between adversarially inclined experts retained by the two parties, the prosecution and defense, as the best way to come closest to “the truth.”

Again, I am neither equipped nor inclined to engage in a sustained critique of Professor Koppl’s proposal, which I think has much to recommend it.  On a smaller scale, some of his proposals, such as replacing open review within laboratories with masked review between laboratories,[25] are useful proposals for forensic science. Instead, I will make two points.  First, his proposal would seem to leave out some important points.  For example, his proposal leaves out the notion of admissibility law altogether.  Thus, it would seem any evidence should be admissible; it should all simply be subject to the adversarial process.  But what if a form of evidence is invalid or of unknown validity?  Is it still acceptable to use, so long as an adversary has had the chance to advance its own interpretation?  In Koppl’s free-market competition, will juries be inclined to believe the expert confidently asserting that her own technique is valid or the expert challenging the technique’s validity?  What if a technique is so novel, so invalid, or simply so preposterous that the adversary cannot find an expert to rebut the proponent’s claim? Professor Koppl’s proposal is also vulnerable to the common criticism that juries may be inclined to credit the expert witness who is most persuasive, rather than the one who is most correct or truthful.[26]

My second point is that, with the above caveats properly addressed, I would not be unhappy with Professor Koppl’s proposal, just as I would not be unhappy with a variety of proposals to reform forensic science other than mine.  For example, in my Article, I noted that I would not be unhappy with the proposal that all forensic science tasks be upgraded so that doctoral level scientists must perform them.[27]  The question is less which proposal is best, than which proposal is most likely to actually be implemented, and which one is least likely to be perverted in its implementation.  By this measure, I must say that I find Professor Koppl’s proposal at least as naive as mine.

Free-market competition requires a level playing field.  The imbalance of resources between defense and prosecution is notorious to legal scholars,[28] and there are indications of pro-prosecution bias among judges and jurors as well.[29]  Forensic science reformers have long observed that defense experts are almost exclusively drawn from the ranks of retired prosecution experts and that the cadre of professional defense experts is virtually non-existent.[30]  Moreover, there are few, if any, models for a forensic laboratory that exclusively serves defendants.  All of this suggests to me that Professor Koppl’s vision of a level playing field in which defense and prosecution experts compete in a free-market is at least as naive as my proposal that forensic scientific knowledge claims be evaluated by those qualified to evaluate them.

At the risk of sounding overly conciliatory, I think that the two general visions espoused by Professor Koppl and myself are not greatly incompatible.  For the most part, Professor Koppl’s and my proposals are taking aim at different targets, and some of the reforms we propose should be compatible.  For example, Professor Koppl’s vision of vigorous competition between experts in applying analytical tools in specific cases is in no way incompatible with my argument that the general validation of those tools should be performed by research scientists, rather than line practitioners.  The challenge lies in harnessing the current public and political interest in forensic science to bring these visions to fruition.


         *    Simon Cole is Associate Professor of Criminology, Law & Society, University of California, Irvine; Ph.D. (Science & Technology Studies), Cornell University; A.B., Princeton University.

**   Suggested citation: Simon A. Cole, Reply, Defending a Knowledge Hierarchy in Forensic Science, 39 Fordham Urb. L.J. City Square 97 (2012).

[1].  Roger Koppl, Leveraging Bias in Forensic Science, 39 Fordham Urb. L.J. City Square 37_ (2012), http://urbanlawjournal.com/?p=424.

       [2].  Simon A. Cole, Acculturating Forensic Science: What Is ‘Scientific Culture’, and How Can Forensic Science Adopt It?, 38 Fordham Urb. L.J. 435 (2010), http://urbanlawjournal.com/?p=598.

       [3].  Nat’l Research Council, Nat’l Acad. of Scis., Strengthening Forensic Science in the United States: A Path Forward (2009), available at http://www.nap.edu/openbook.php?record_id=12589&page=R1.

        [4]Cole, supra note 2, at 447.

        [5]Simon A. Cole, Grandfathering Evidence: Fingerprint Admissibility Ruling from Jennings to Llera Plaza and Back Again, 41 Am. Crim. L. Rev. 1189, 1197-2000 (2004).

        [6]Michael J. Saks & Jonathan J. Koehler, The Individualization Fallacy in Forensic Science Evidence, 61 Vand. L. Rev. 199, 203-08 (2008).

        [7]Simon A. Cole, “Implicit Testing”: Can Casework Validate Forensic Techniques?, 46 Jurimetrics 117, 123-26 (2006).

        [8]Simon A. Cole, More Than Zero: Accounting for Error in Latent Fingerprint Identification, 95 J. Crim. L. & Criminology 985, 1034-38 (2005).

        [9].  See Cole, supra note 2, at 459-60 (discussing the monolithic nature of present-day forensic science).

     [10].  See id. at 468-72.

     [11].  See Koppl, supra note 1, at 42-47.

     [12].  See id.

     [13]Cole, supra note 2, at 468.

     [14].  However, the fact that Professor Koppl was able to muster so much evidence against this empirical claim, which I did not intend to make, in a sense illustrates my point.  We know something about the validity of medical knowledge and the efficiency of health care delivery.  Thus, Professor Koppl is able to muster evidence in support of his argument that neither is optimal. See supra note 11 and accompanying text.  But we know almost nothing about the validity of forensic scientific knowledge or the efficiency of its delivery.  Instead, we have a coterie of external “critics,” such as the NAS, Professor Koppl, and myself saying “we have no studies; we have no data.”

       [15].     For another use of a medical analogy for criminal justice reform, see James M. Doyle, Learning from Error in American Criminal Justice, 100 J. Crim. L. & Criminology 109, 118 (2010).

     [16].  See Nat’l Research Council, supra note 3, at 19 (suggesting the creation of an “independent federal entity”).

     [17].  See Koppl, supra note 1, at 48-51.

     [18].  See Cole, supra note 2, at 457-59.

     [19].  See id.

     [20].  Id. at 469.

     [21].  See supra note 16 and accompanying text.

     [22].  Criminal Justice and Forensic Science Reform Act of 2011, S. 132, 112th Cong. (2011).

     [23].  See Koppl, supra note 1, at 53-56.

     [24].  See generally Karl Popper, The Open Society and Its Enemies (1945).

     [25].  See supra note 23.

     [26].  See, e.g., André A. Moenssens, Admissibility of Scientific Evidence——An Alternative to the Frye Rule, 25 Wm. & Mary L. Rev. 545, 546 (1984) (“[J]uries may be overly impressed by experts with seemingly impressive credentials.  Additionally, juries may give greater weight to expert opinions than the opinions deserve on the basis of scientific validity.”).

     [27].  See Cole, supra note 2, at 471-72.

     [28].  See, e.g., Christophe Champod & Joëlle Vuille, Scientific Evidence in Europe ——Admissibility, Evaluation and Equality of Arms, 9 International Commentary on Evidence, no. 1, 2011, available at http://www.degruyter.com/dg/viewarticle.fullcontentlink:pdfeventlink/content?Uriformat=INT&t:ac=j$002fice.2011.9.issue-1$002f1554-4567.1123$002f1554-4567.1123.xml; William J. Stuntz, The Political Constitution of Criminal Justice, 119 Harv. L. Rev. 780, 808-09 (2006) (discussing the allocation of resources for prosecution); William J. Stuntz, The Uneasy Relationship Between Criminal Procedure and Criminal Justice, 107 Yale L. J. 1, 28, 35 (1997) (discussing the effects of criminal defendant’s wealth on the course of criminal litigation).

     [29].  See D. Michael Risinger, Navigating Expert Reliability: Are Criminal Standards of Certainty Being Left on the Dock?, 64 Alb. L. Rev. 99, 131-35 (2000).

     [30].  See, e.g., Gary Edmond & Kent Roach, A Contextual Approach to the Admissibility of the State’s Forensic Science and Medical Evidence, 61 U. Toronto L.J. 343, 363 (2011).