Discussion Guide and Transcript
Episode Four
Research Ethics Reimagined Episode 4 “Researching Justice with Jim Greiner, PhD”
August 28, 2024
- In this episode of PRIM&R's podcast, "Research Ethics Reimagined," we explore the intersection of legal empiricism, justice, and research ethics. Our guest is Jim Greiner, PhD, Professor at Harvard Law School and Director of the Access to Justice Lab. Listen on Spotify | Listen on Apple| Listen on Amazon Discussion Questions
- 1.) Legal Empiricism and Justice
- Greiner discusses how the Access to Justice Lab aims to transform the practice of law and the U.S. justice system into an evidence-based field. Why do you think there has been resistance to evidence-based approaches in the legal profession compared to other fields like medicine?
- The interview touches on the concept of "irreducible complexity" in legal cases. How might this belief impact the adoption of empirical research methods in law, and what challenges does it present for researchers?
2.) Research Ethics in Legal Studies
- Greiner discusses the challenges of applying current research regulations, particularly those related to prisoners, to legal and justice system studies. How might these regulations be revised to better accommodate this type of research while still protecting participants?
- The idea of treating IRBs more like courts, with precedent and transparency in decision-making, is discussed. What potential benefits and challenges do you see in implementing such a system for research ethics review?
3.) Bringing Research and Practice in Law
- The Access to Justice Lab conducts randomized controlled trials in the legal profession. How might the results of these studies impact legal practice and policymaking? What barriers might exist in translating research findings into practice?
- Greiner mentions surprising findings from their research, such as the effectiveness of pro bono matching for simple divorces. How can unexpected research outcomes be effectively communicated to stakeholders and incorporated into practice?
Key Terms and Acronyms
Access to Justice Lab: An entity at Harvard Law School focused on conducting randomized controlled trials in the legal profession.
IRB: Institutional Review Board, a committee that reviews and monitors research involving human participants.
Irreducible Complexity: In the legal context, the belief that each case is unique and cannot be meaningfully compared to others, making it difficult to apply statistical or scientific methods to legal practice.
Legal Empiricism: The application of scientific research methods to study legal systems and practices.
Randomized Controlled Trial (RCT): A type of scientific experiment that randomly assigns participants to different conditions to test the effectiveness of an intervention.
Additional Resources
- Access to Justice Lab - Official website of the Access to Justice Lab at Harvard Law School.
- National Center for Access to Justice - Organization working to make the civil and criminal justice systems fairer and more accessible.
- PRIM&R's Research Ethics Timeline - A resource for exploring the milestones of research ethics with humans.
- American Bar Association - Access to Justice Initiatives - Information on various initiatives to improve access to justice in the United States.
Transcript
Transcript, Ep.4, “Researching Justice with Jim Greiner, PhD”
Host: Ivy R. Tillman, EdD, CCRC, CIP, Executive Director of PRIM&R
Guests: Jim Greiner, PhD
A transcript generator was used to help create written show transcript. Written transcript of podcast is approximate and not meant for attribution.
Tillman: Today, I am very pleased to have Dr. Jim Greiner with us. Dr. Greiner is a professor at Harvard Law School and director of the Access to Justice Lab. The Access to Justice Lab is the only entity in the United States that focuses on randomized controlled trials in the legal profession. The Access to Justice Lab is conducting innovative work, implementing randomized field experiments to learn what works for individuals and families who cannot afford to hire lawyers.
We were honored to have Dr. Greiner deliver a keynote address during our annual conference, which was held most recently in Washington, DC. During those remarks, he spoke about his work advancing equity and justice in research. He also touched upon how to make IRB determinations more transparent to build trust in science and research.
At Harvard Law School, he teaches courses on civil procedure, expert witnesses, and and access to justice. Before joining Harvard Law School in 2007, Dr. Greiner completed his PhD in statistics at Harvard University and previously practiced law for six years, including three years with the Department of Justice.
Thank you again, Dr. Greiner for speaking with us all in December at our conference and for sharing your thoughts with us today. So before we dive into the work that you're doing and the access to justice lab, which is fascinating, I want to back up and really have you discuss how you define and envision justice, and particularly justice practices in this space and legal empiricism.
Greiner: Sure, the idea of justice typically comes from people who are in the system, either the participants or the people who are running it. What we try to do in the Access to Justice Lab is take those definitions, which are surprisingly common across time and space, and try to figure out what practices best meet those goals.
There are a lot of differences across, you know, jurisdictions, geographic jurisdictions about emphasis on different things. So, some people are more willing to suffer certain types of costs than others, more incarceration versus more damage to family structures or to socioeconomic circumstances like jobs and housing and those sorts of things versus less of those sorts of things and less cost to administer the system.
There are differences like that, but most people have the same goals. And what they mean by the justice system is a combination of the formal court system, administrative agencies that act a lot like courts. So administrative agencies that adjudicate eligibility for Social Security benefits, for unemployment benefits and those sorts of things, and then the laws that are generally governing behavior required to get people where they want to be. So, for example, wills and estate planning mostly is done outside the courtroom. And a lot of it, if it ends up in court is typically just a administrative proceeding. You're just filing the paperwork and that's it.
But you still have to adhere to those laws. So that's the justice system. What we hope in the justice system is it approximates some idea of justice, you know, goodness in the society of doing what the society wants. Many would argue, including me, that our current system does a poor job of that.
Tillman: And so that's the work that you're doing; and the Access to Justice Lab are also known as the A2J lab.
We did some research, of course, from the website and the Access to Justice Lab is designed to use empirical research to make the U.S. justice system work better for everyone. So can you explain, I think you've kind of started to explain by what you mean by that, but really share where this idea for the access to justice lab came from.
Greiner: Yes. So the, the idea is to try to transform the practice of law and the United States court system and the justice system generally into an evidence based field. Many people think, including me, actually, before I entered it, that because law is a profession, a traditional profession, like medicine, that there would be the same dedication to evidence based thinking and defining evidence based thinking in the same way as say medicine and in medicine for decades.
Now we have insisted, for example, that new drugs and new medical devices cannot be marketed to the public. They cannot be recommended unless they undergo credible testing. And typically that means a randomized experiment of the kind, again, that people are familiar with. And that's something that we understand and appreciate and live with and want in medicine.
It has costs to it, severe costs, in terms of people potentially, for example, dying, while effective medication is unavailable. while it's being tested. The problem is we don't know that it's effective until we test it. And there are many examples of medicines that were quite promising and that everyone thought were certain to work and ended up not being work or have backfired and caused harm.
And I thought, you know, entering the profession that the legal profession and the justice and the court system would be the same way that we would have the same dedication to evidence based thinking. Instead, which what we have in the legal profession is disdain, contempt or amusement for this type of evidence-based thinking and the adoption of lots of practices that are justified primarily because we've been doing them for a long time or they approximate things that we've been doing them for a long time or because of their adaptations of longstanding practices that are adapted because of resource constraints.
And so, for example, it may surprise people that the idea of bail, which almost everyone has heard of, is not something that is hundreds of years old. It is a relatively, comparatively speaking, a relatively recent invention in United States history; and it was adopted and put into place without an investigation into seeing whether it is effective at what it aims to do, or whether, the practices that were in place before bail were effective at what they intended to do.
In other words, all of the practices are adopted that way.
And so the Access to Justice Lab's purpose is to try to transform that way of thinking and that way of doing things into something that is evidence based, where we have evidence about what works and what doesn't, so that policymakers can make much better choices about the policies that they institute.
Tillman: Yes, it's fascinating to consider evaluating these practices in a similar fashion that medical practices are evaluated. We're going to get into really some of the reception that you've received from this work in a little bit. But how did you get interested in this particular type of research? Were you exposed to research? Sounds like I know the answer, but can you kind of walk us through your journey to this type of research.
Greiner: Sure. It was remarkably accidental. I practiced law for a while, but not in trial courts that dealt with high volume cases. I was doing other type where, where a lot of these statistical techniques can be of most use.
So you don't need a randomized trial to figure out, for example, whether a law is unconstitutional. That's not a scientific judgment. And that's the sort of law work that I was doing as a practitioner. When I started to be an academic after going for a stats degree, I thought that it would be interesting to have some studies that were analogous to things that I had heard about in medicine, never studied before, except in stat school, you know, studied some in stat school about how to do this.
I thought it would be interesting to do some in law, just like in medicine. But I thought that. There would be hundreds or thousands of these sorts of studies in law, the way there are in medicine. And instead to give you an idea of what happened shortly afterwards, a law student and I spent about two and a half years looking for examples of these types of studies in law from, you know, throughout the United States history.
And we found a total, depending on how you count them, about 50. And there are probably, you know, 10 to 20 or 30, 000 going on at any one time in medicine. You know, at any particular moment in medicine, so the comparison, whereas, you know, what we're looking at is all of the United States history. We could find about 50.
So there were very, very few of them. And I, and so I said, well, you know, maybe that's something that I can contribute to because I believe in evidence based thinking. And when I tried it myself, I can't encounter the hostility and the disdain of it. And the rejection of this type of evidence based thinking, and that's a brick wall that I think is standing in the way of adoption of good policies.
And when I see a brick wall, I like to punch through it.
Tillman: And for those who don't have the privilege of actually seeing us on camera, Jim has a brick wall actually behind him.
Greiner: I actually have one behind me. It's correct.
Tillman: I love that analogy. So why do you think there was, or there is so much resistance and disdain?
Greiner: So, I struggle to find an explanation that distinguishes law and medicine. Because in the 1930s, if you had talked about the science of medicine, people would have looked at you like you were crazy. Medicine was not a science. Medicine made a conscious decision to try to transform itself into a science.
And part of that was incorporating lessons from chemistry and biology and, and those sorts of things. But part of it was also a decision among the doctors. To give up control about what works and what doesn't, and they gave that control up to statisticians. And that was a wrenching decision that many in the medical profession fought.
But those who were dedicated evidence-based thinking, won that battle, or at least partially won that battle. There was no such attempt in law. Even though in, again, in 1930, there were occasional examples of using, using randomization in law and drawing inferences about, for example, whether assignment to Judge A versus Judge B makes a big difference in criminal justice outcomes.
That was a study that was done in the 1930s, and it took advantage of the fact that that cases were assigned randomly to judge a versus judge B. And so if you'd stood in 1930, you would not have known whether law or medicine was going to transform itself into an evidence based field. One did and one didn't.
And to go back to your question now, I struggle to distinguish between the two because all of the things that are quintessential about a profession, the idea of professional judgment being something very important, and the idea of having a statistician come in and tell you what works and what doesn't is threatening to professional development.
The idea of professional judgment, right? And what is good for each individual person as opposed to what works on average, which is what statisticians tell you all of that's true in both fields. And so honestly, I am the only thing that I can think of that distinguishes law and medicine is that rich people get sick in approximately sort of the same way as poor people.
And therefore, if rich people demand something that actually works, they will have institutions that protect their interests, poor people do not encounter the legal system in any way. Resembling the way rich people do, there is no similarity between them whatsoever, and that's not a particularly lefty statement.
That's just a statement of fact, right? And I don't think anyone. I think anyone of any political persuasion would agree with that statement. And so that is the only hypothesis I've come up with is far from something that I call proven.
Tillman: Wow, that gives you a lot to think about. But I can definitely and you're right, it's not based in any political ideology, it's really a fact.
And so in the systems that have been created and how individuals based upon our social constructs and socioeconomic factors, I'm sure play a part into some of these decisions. So, you know, when you speak of this type of research and the website for individuals listening when you get a chance to go look at the Access to Justice Lab website, because it does a beautiful job of it kind of walking through the process.
Jim, can you do that for us? Can you kind of explain what types of studies are you currently conducting? What do they look like? How are they designed? Who's involved? How are the results disseminated? Are communities involved?
Greiner: So basically the idea is that that either we in the lab or that someone working in the justice system or someone working representing or speaking for people who encounter the justice system comes up with some idea that they think will improve matters or whether it could be that they want to test something that the justice system is doing right now.
And we work with them to identify an alternative, both the thing that they would like to test and the alternative. So what does that look like in practice? Well, it might look like, for example, providing self help materials to people who are seeking guardianship, an order of guardianship over an incapacitated adult.
So grandma can no longer take care of her assets. She can't make good financial decisions and we need to establish guardianship. That requires a court order. Meaning a lawsuit. You have to file a lawsuit in the criminal context. It might mean, for example, providing risk assessment instruments to judges who are making bail decisions.
A risk assessment instrument is a quantitative tool that is designed to um, classify whether an arrestee is risky if released or is low risk if released. Um, and there's more complicated than that. That gets you the idea. And what we want to find out is, does that risk assessment instrument improve judges decision making?
And so once we've identified the idea we want to test and the alternative, typically the alternative is business as usual, whatever it is that we're doing right now. Then we work with local communities with local stakeholders, and typically that's a lot of them. So, for example, in the criminal justice studies that I mentioned in the risk assessment example, that requires working with the sheriff's office, law enforcement, with prosecutors, with public defenders.
With court administrators at the local level and at the state level, it requires work with I guess what you'd call county elders, whatever the governing body is inside the county or the city where you are working, and then county staff. And what we've learned to our chagrin in some instances is it's not enough to convince the top of any organization, you have to convince the people who are doing the actual work.
And so that's why I separate county elders and county staff, for example. And we have to get everybody on board. And the process takes months, sometimes years. And then we need funding because medical studies, these studies are slow and they are, they do require money for staff in order to produce the study.
And we set up a system whereby we try to have cases or people coming into a particular point. In the bail example, it would be when they're arrested and facing a judge for the first time to see what bail will be set. In the self-help materials example for guardianship it would be when people present at a legal services provider's office.
And we randomize them to one treatment or the other treatment usually to the new, the new idea or business as usual. And then we follow everyone according to defi outcomes that are defined up front, again in conversation with stakeholders and with the community. For example, in the bail study, what does the community and the stakeholders care about?
They care about failure to appear, whether people come back to court. They care about re arrest or re charge or re conviction for new, potential new offenses that occur during, if people are released during the predisposition period, especially if those are violent. They care about the number of days of incarceration.
How many, how many days are people incarcerated? And the reason for that is both expensive to the county, but potentially equally as perhaps more important incarceration, we think. is bad for the individual incarcerated, bad for the family, bad for the community. It prevents people from doing things like getting jobs, having relationships with their children, with their families.
For some people, temporary incarceration might, I emphasize might, in my view this isn't proven yet, but it might turn out to be true, might be a life changing positive experience, right? Maybe they get access to treatment, or to counseling, or whatever it is that they didn't have before. But, again, it's something that we can try to assess.
And so, we follow outcomes for everyone, and that takes a long time too, which is one of the reasons why these studies are slow and expensive. And when we produce the results, we report back to stakeholders, and we try to publish in academic journals, and we try to attend conferences where people of the type of folks who do this sort of work attend.
And let them know what our evidence shows, and hopefully they take it into account. The last part, in particular, has been a real challenge for the lab. We have, we have had real challenge getting by in. Even for people who have participated in the study. And that's something that is just that a human beings when you make a decision and start doing something day to day, it's very hard for one to accept the idea that what one is doing may be ineffective or may be counterproductive.
It may be producing the opposite of the desired aim. That is a very difficult thing for people to accept, particularly professionals who say, I have special training. I have specialized experience, the world thinks I'm cool because I have degrees and that sort of thing, and, and now you're telling me that my professional judgment is dead wrong.
Tillman: Yeah, I can imagine so. I'll get into when we start talking about like the IRB process that you went into. If we have time, I want to ask about like the informed consent process associated with this type of work. Because I think that's fascinating and it may look a little bit different than what we're accustomed to, even in medical or social science research.
Yeah. But what surprised you about some of your own findings from some of your research?
Greiner: Sure, so I have given up on being able to think that I can predict what the research is going to show, because I've been wrong so many times that it's just silly. To give you an example, we recently did, as I mentioned, a bail, a study about provision of risk assessment instruments at bail.
I thought they would improve criminal justice outcomes, providing the risk assessment instrument versus not providing it. I thought at worst, how could it possibly be harmful? Of course it has to improve. It's better, it's better information. So far it doesn't. So far it has no effect whatsoever. And it's unfortunate because producing the information, getting it into a type of format that a judge can use, and providing it to the judge, educating judges on what the information looks like and how to interpret it, that's not free.
That can cost counties hundreds of thousands of dollars, millions over time. Even if you try to automate it, it requires a lot of people to do a lot of computer programming. And so far, again, this isn't final, but so far we're not seeing any effect, good or bad. Many people had thought those risk assessment instruments would cause bad racial effects.
That they would increase the racial tinge, call it, that exists within our criminal justice system. Didn't do anything. To give you another from something that I did not predict, we recently published a recent about three or four years ago, a study of the effectiveness of a pro bono matching program for folks who want to seek simple divorces.
And what that meant was folks who had a very little in the way of income and assets, virtually nothing typically, and are wanting to get divorces would call this service and say, I have no, basically no assets, very little in the way of incomes. I can't hire a lawyer. Can you match me with a lawyer who's willing to volunteer to help me with my divorce?
And the matching service that had a stable of attorneys that it could call on typically had enough attorneys to provide lawyers to about 15 percent of those who called. It was completely overwhelmed. And so we randomized to whom they provided. And I got to tell you, I thought that matching service would not be particularly effective.
It often took the matching service months to come up with a match. Even when we randomized people to receive a match, it took them sometimes months. And the process was not supposed to be complicated because there were no assets and there were no children, there's no child custody issue. For the overwhelming majority of people, it's just, and this is almost a direct quote, “I don't want to be married to that bum anymore.”
In other words, I just don't want to be married. Right? Or I want to remarry, and I can't because I'm still married. Right? Or something like that. That's all they were seeking is just the piece of paper saying my marriage is over. In some instances, the spouses had not seen each other for a decade and they needed just the piece of paper.
And that pro bono mashing service turned out to be wildly effective. And it was quite surprising to me because, again, it was not supposed to be a complicated thing to go get this piece of paper, and, and yet it apparently was. And so, those are a couple of surprises.
Tillman: Interesting. Interesting. That's akin to the process. When you go in and you hypothesize that some system may work and, and that it doesn't, or some, or one would be a little bit more simple and it's more complicated, but there are positive results. That is definitely akin to the scientific process that many of our researchers experience. So, let's turn back to the Access to Justice Lab and some of the history behind it.
So, I'm going to read this and then we'll ask a question on the back end. So, the history of the Access to Justice Lab notes that for more than a century U. S. legal and adjudicatory practices have reflected an ethos of professionalism in which individual lawyers and judges purport to make irreducibly complex judgments about each client's or litigant's legal needs and the merits of their claims.
The belief is that only a lawyer knows what's best for a client. And the only person a lawyer should listen to about their client's legal needs is another lawyer. As a result, law currently recognizes only two sources of truth about what works and what doesn't. The pronouncements of legal elites and each individual lawyer's or judge's own personal experience.
With that in mind, when you present your findings, what happens when the data tells a story that the community does not like?
Greiner: That's quite a mouthful. Even just reading it, it's quite a mouthful. So just a quick explanation. The idea that each case is irreducibly complex. What that means, and I hear this constantly from lawyers, is that each individual is their own universe.
And there's no point in comparing one case to another case in any way, to another, one person to another person in any way. You can't learn anything about person A from looking at person B's experiences. Or you can only do that if you're a professional. There's no way for a statistician to come in and measure anything.
Only a professional can do it. And so, if that were true, for example, if each human body were irreducibly complex, there would be no point in studying whether penicillin, on average, helps combat, is an effective antibiotic, right? Because each person's body is so different. That there's no point in making any inferences.
Of course, we know that's not true. And we know it's not true in the case of, that I mentioned, of a lot of simple divorces. That they are, people have similar aims, and in some way they have similar circumstances. Of course, each one is a little bit different, just like everybody is a little different. So anyway, we asked what the reaction is.
And, as I mentioned, we've really struggled in getting people to buy into our research findings. Even people who have participated in the studies themselves and we attribute that to the fact that this type of thinking is quite alien to the legal profession. It's quite different from the way we, even I teach in law school because people come to law school in the first year especially and expect to see a certain type of education.
And I do provide that education because I, because they need it for their eventual practice if they're going to practice. On the other hand, I try to show them also there are different ways of thinking about this. But we have, for example, struggled to, to suggest to folks that they may want to consider, reconsider practices that they are currently implementing that we have found to be.
In some instances, mildly harmful and reconsider practices that are expensive but have no benefit. Try to find out why it is that a particular expensive practice works and take away the obstacle. For example, in the divorce study that I mentioned, where the idea is the lawyer is having a big effect on whether folks get a divorce.
The lawyer shouldn't have a big effect on whether people get a divorce. That divorce proceeding with no assets, no income stream, no child custody, no domestic violence in at issue. There may be those things in the marriage, but they are handled in separate legal proceedings. So, they're not at issue in whether someone gets a divorce.
You can just get the divorce and be done. That should not be, a lawyer should not make a big difference there. That is a indication of a severe access to justice problem. And the idea should be, take a look at your system, where are the hurdles? Actually the hurdles in this case are quite high and they are unnecessary.
And so, the example that we cite in that particular paper that everyone can understand is, for example, there was a particular moment in the divorce proceedings. When a pro se litigant, meaning someone without a lawyer, had to find and use an old-fashioned typewriter. I couldn't find one. I've actually sat down and thought if I needed a typewriter, where would I go?
I don't have a clue. Now, I might start with a local library, but I don't think they have one either unless it's locked up in a glass case as a museum exhibit. But that was an example of something that local practice. Even if it didn't formally by law required, effectively required, because there was a high, not certain, but high degree of probability that the legal piece of paper that you needed to send to the court would be rejected if you didn't have access to a typewriter.
And so that's the kind of examination that we hope our studies will engender. And again, this is not, this is something that we still struggle with at the lab.
Tillman: Absolutely. So, when we talk about your presentation at our annual conference in December, you spoke about conducting research with prisoners, particularly.
So, can you share about the work you've done, the research with prisoners, but also how you found the process, the regulatory IRB process for getting to that approval, your experience with that.
Greiner: Absolutely. So, just to be clear, the regulations, it's actually section 303C, I have it up in front of me, defines prisoners as anyone who is incarcerated at any point in the criminal justice system.
So, a prisoner isn't someone in a prison. Because at least in the legal system, we think of a prison as something where people, some place where people go after they are convicted. And a jail is somewhere people go before they're convicted or if they're convicted of only something mild. In other words, jails typically are temporary local institutions.
Prisoners are for longer term residence, but the regulations, I think probably properly don't make that distinction if you're incarcerated or even if your freedom is limited in a severe way and this is part of IRB training, if you are in a hospital and a guard is standing outside your door and won't let you out, that's an incarcerated individual, right?
And so there's, it's a very broad range and currently the regulations you can tell are Uh, written with the assumption that only medical research is being done with quote unquote prisoners, with prisoners under this definition. And you can tell that because, and again I have the provision up in front of me, section 305A3, so 45 CFR, 46, 305A3, requires an IRB to decide whether the risks involved in the research are commensurate with the risks that would be accepted by non-prisoner volunteers.
Now that makes total sense if you're talking about medical research. Right? Because there's nothing, frequently at least, there is nothing that is relevant to the research about the fact that people are incarcerated. You're just, you just need human bodies, and you need human bodies to, to undergo different forms of treatment and see which one is more effective.
And your assumption is that human bodies in prison, among incarcerated individuals are similar enough to those outside that we can still learn something. I don't know how to apply that subsection if what I'm trying to do is see if there are safe alternatives to incarceration. The only people who are interested in alternatives to incarceration are those who are incarcerated.
If, for example, what I'm trying to do is test an intervention that might allow a municipality to reduce its reliance on incarceration, what I am apparently instructed to do under this regulation is see whether people would want, for example, to be bailed out of jail if they are not incarcerated or if they are not in jail or if they have never been arrested.
And I find that difficult, a difficult thought exercise. More broadly, the assumption that only medical research occurs is built into other aspects of the regulations. Not just, it's most stark and obvious in the prisoner regulations, but it's built in elsewhere.
Tillman: And so I think you answered my next question, which is really, if you could inform some revisions, particularly to the provision around prisoners or even the definition, because as you were speaking and recalling your experience, I was thinking about, past experiences that I've had with researchers who were non clinicians who wanted to do research with incarcerated individuals and even just the definition and where the individual was located at the time really mattered and IRBs grapple with how to how to review this type of research and facilitate it while maintaining compliance with the regulations.
So, you know, what kind of suggestions would you have as far as revisions to our current regulations?
Greiner: Yeah, first of all, let me just say to some, I think history has taught us, forcefully and in a gruesome way that we need regulations. One doesn't, there are just too many horrible examples of inhumane treatment by researchers of study participants of acting, as you mentioned earlier, without consent and the idea of consent without acting with adequate information and regulation is necessary.
Oversight is necessary. It is no good to say, these are educated, highly educated people. They're professionals. We can trust them. That was true of the Nazis, that was true of the folks who perpetrated Tuskegee, that was true of folks who, you know, perpetrated Willowbrook, all of the horrible stories.
What are the, what should regulations around incarcerated individuals look like? I don't have complete answers, but I think one thing to do is if you are going to write a set of regulations that is specific to incarcerated individuals in the United States, which apparently, again, sections 303 and 305 are, are designed to do.
You need to incorporate a more thorough understanding of how incarceration works in the United States and what different stages of the criminal justice process are doing and designed to do. You need to make hard judgments, for example, about whether an individual on parole, whose activities are strictly regulated, but are not, but the individual is not physically incarcerated at all times.
Then there are some people who are halfway, thus the term halfway houses. You need to make judgments about them, and that's just definitional. That's just to figure out what you're going to require of them. You also need, I think, to be more receptive to the idea that research that is designed to, I guess this is the academic term, interrogate, to investigate current practices is high risk in the sense that the outcomes are important, but it is therefore all the more important to do.
So, for example, if we have, what we're trying to figure out is whether incarceration is necessary or whether we can decarcerate. That will mean doing a study where the, the intervention, some intervention like a risk assessment instrument or, or something like that, will change who is incarcerated and who isn't.
If it doesn't, there's no study and that is a big consequence. That is an important consequence. There is no way in my view to characterize that kind of study as low risk. On the other hand, death is a big consequence and we routinely implement studies in which the outcome variable is death within five years.
That's a typical cancer study or a major heart attack or something like that. A major cardiac event within high is within five years. Within five years, that is a typical hard study. And so the fact that the consequences are big, are salient, are important, is a reason to pursue the credible research, the rigorous research, not a reason to avoid it.
And the fact that the circumstances are unfamiliar, in other words, we say, my goodness, we're randomizing incarceration. That's a terrible idea, because we've never done it before, because we've only done it very infrequently. Whereas we randomize medicine that can kill people or save their lives all the time.
That's fine. That's not convincing. The fact that what we're going to need to do is to say, wait a minute, these are big circumstances. These are big outcomes. These are important outcomes in people's lives. These are real people's lives, but that is all the more reason to do the research because we don't want to be incarcerating people unnecessarily.
And on the other hand, for some people, and I'm not taking a position on this personally, for some people, we don't want to let people out of jail who we think are, have a very high probability of committing some horrible crime. There are some people who think that way too, right? And what we want to know is to provide them with strong information to allow policy makers to make democratic judgments.
Incarcerated individuals are different. And if the idea is to try to see whether we can, whether we need to dispense with or whether we need to increase or right size incarceration, we need credible evidence.
Tillman: So, my last question relates to a concept that you discussed at the conference, which was fascinating to many of us, and you mentioned the concept of thinking of IRBs as like a court, but also having this level of transparency to the determinations and precedent that's been set. Can you describe that, that concept more and what that could look like?
Greiner: Sure. So I think the problem that this is, that this idea, line of thinking is designed to address is inconsistency in decision making within an IRB and then inconsistency in decision making across different IRBs.
The latter problem across IRBs didn't used to be, I'm guessing, put it this way, didn't used to be as big a deal until scholars started to make routine across institution collaborations. So those are, those are just run of the mill these days. It is the exception as opposed to the rule that one scholar will pursue one IRB, one operation that requires IRB approval.
It's much more likely that there will be multiple investigators and they will be at different institutions. And therefore, coordination is something that has to happen. And what I have experienced and what I think others have, including folks, you know, who work in IRBs, is that there are differences in how folks approach the same study.
Or there are differences in how folks approach two studies that are essentially the same within an IRB. And I've had both experiences. I've had experiences where we, where it was essentially the same study for all intents and purposes for IRB. We just changed the language in some letters that we were sending out to folks who had been sued in debt collection from one study to the other.
And two IRB officers reacted quite differently and it had nothing to do with the language in the letter. The one that, the second one had, it was not citing the language in the letter at all, which is simply saying this study shouldn't have been approved to begin with. And in the cross institutional context, one IRB said that this is not human subjects research and therefore doesn't even need to go in, undergo approval.
The other one said this needs to go to the full board. We can't approve it at the staff level because we think it's high risk. I happen to, by the way, to think in that instance, the second IRB was correct and I wanted them to approve the research. It would have been much easier if. Yeah, for me, if they had said it's not human subjects research, but it was definitely, it was high risk and it needed to go through full review.
So what does that mean? There are things that the legal system has partially at least figured out to try to address this kind of inconsistency. And you mentioned one is the idea of precedent where there can be, where there is some sort of summary of a IRB application and then the decision and the reasons for the decision are written down and the underlying documents are accessible to the community so that we can see what it was that affected the decision makers process.
And say they did this before and this is similar to that. Of course it won't always be similar, but that's the point is to make a judgment. And that other institutions can look at the same thing. So they can also be available to other institutions. In other words, if University A deals with one case, one study in a particular way, then maybe that's a reason for University B to think that they should do the same.
Other ways of doing it are to have what amounts to, in the court system, an appeals process so that there is some higher-level IRB, I don't know what you'd call it, appeal, appellate IRB, appeals IRB. And what they would do is, again, issue reasons for their decisions that other IRBs could then look at and make and draw analogies to.
And say, if that decision is this way, then we should probably make the same sort of decision or make a different decision. The fact that those ideas might be useful to increase uniformity within and across IRBs. That's what courts do. They look at individualized circumstances and they make, they take complicated and vague regulations, sometimes regulations like the current ones that are not particularly well adapted to a new fact pattern, and they make decisions.
That's what courts do. Administrative agencies do the same when they're acting like courts, which is why I'm just calling them courts. And on the basis of those sort of published decisions or decisions that are accessible to the public, the people can have greater uniformity and predictability in what IRBs will approve and that also, by the way, that can then allow researchers to structure studies up front in order to anticipate what potential objections will be.
And that I think reduces workload. I think if we can solve, speaking for the, we here as the research community, if I can, if we can solve problems up front, we want to. Nobody likes for the IRB process to take months, and if we can anticipate and solve problems in the constructing the field operation based on knowledge of how people have ruled before and therefore how they're likely to rule in the future, we can potentially save everybody, including us, an awful lot of work.
Tillman: So how has that, I'm sure you received a lot of feedback at our conference, when you are talking about this concept, how has that been received?
Greiner: So, I think a lot of people have said, sounds great, but how would we do it? And I think there are ways to do it, to begin doing it, put it this way, even at low cost.
I mentioned that there are information aggregators who make it their business, being folks like Lexis and Westlaw and Bloomberg. What they like to do is market the fact that they have access to types of decisions or to things that people want. And therefore, if you pay for their service, if you subscribe to their service, they'll, you'll get access to their information.
And so, even simply having IRB officers write a one or two paragraph summary of the study, and then a one paragraph decision about it, that would be a first step. And again, I think if those were made in some way accessible to an information aggregator, the information aggregator would do all the rest of the work.
They would then accumulate the information and go to universities and basically extort us for subscription fees, which is exactly what they do. And we would pay it, right? And say, if you want access to this IRB, you know, these IRB, this IRB library, you got to pay the fee, right? And I would go to the university and I'd say, pay the fee. Right?
And so that I think is a one potential sort of free step that you that are not entirely because you'd still have to write the paragraph. But again, I think this is something you would tell me where I think this is something it would not be hard to do. Having finished a file, you probably got an idea of what the file looks like and a couple sentences about why you're approving it at the level you're approving it or why you're sending it on to the full board or something like that.
Again, I think this is something you just sit down and tap out. So that would be a first step. I think there are other things like an appeals IRB would require resources. And, but I think the publication part and the accumulation information part, you just take advantage of market forces.
Tillman: Fascinating. Fascinating concept. I would love to continue that conversation. So, thank you. Thank you, Jim, for your time and really opening our eyes to a discipline that can definitely benefit from collaboration with our ethics professionals. Thank you.
Greiner: Ivy, again, thank you so much for the opportunity.
##
We were honored to have Dr. Greiner deliver a keynote address during our annual conference, which was held most recently in Washington, DC. During those remarks, he spoke about his work advancing equity and justice in research. He also touched upon how to make IRB determinations more transparent to build trust in science and research.
At Harvard Law School, he teaches courses on civil procedure, expert witnesses, and and access to justice. Before joining Harvard Law School in 2007, Dr. Greiner completed his PhD in statistics at Harvard University and previously practiced law for six years, including three years with the Department of Justice.
Thank you again, Dr. Greiner for speaking with us all in December at our conference and for sharing your thoughts with us today. So before we dive into the work that you're doing and the access to justice lab, which is fascinating, I want to back up and really have you discuss how you define and envision justice, and particularly justice practices in this space and legal empiricism.
Greiner: Sure, the idea of justice typically comes from people who are in the system, either the participants or the people who are running it. What we try to do in the Access to Justice Lab is take those definitions, which are surprisingly common across time and space, and try to figure out what practices best meet those goals.
There are a lot of differences across, you know, jurisdictions, geographic jurisdictions about emphasis on different things. So, some people are more willing to suffer certain types of costs than others, more incarceration versus more damage to family structures or to socioeconomic circumstances like jobs and housing and those sorts of things versus less of those sorts of things and less cost to administer the system.
There are differences like that, but most people have the same goals. And what they mean by the justice system is a combination of the formal court system, administrative agencies that act a lot like courts. So administrative agencies that adjudicate eligibility for Social Security benefits, for unemployment benefits and those sorts of things, and then the laws that are generally governing behavior required to get people where they want to be. So, for example, wills and estate planning mostly is done outside the courtroom. And a lot of it, if it ends up in court is typically just a administrative proceeding. You're just filing the paperwork and that's it.
But you still have to adhere to those laws. So that's the justice system. What we hope in the justice system is it approximates some idea of justice, you know, goodness in the society of doing what the society wants. Many would argue, including me, that our current system does a poor job of that.
Tillman: And so that's the work that you're doing; and the Access to Justice Lab are also known as the A2J lab.
We did some research, of course, from the website and the Access to Justice Lab is designed to use empirical research to make the U.S. justice system work better for everyone. So can you explain, I think you've kind of started to explain by what you mean by that, but really share where this idea for the access to justice lab came from.
Greiner: Yes. So the, the idea is to try to transform the practice of law and the United States court system and the justice system generally into an evidence based field. Many people think, including me, actually, before I entered it, that because law is a profession, a traditional profession, like medicine, that there would be the same dedication to evidence based thinking and defining evidence based thinking in the same way as say medicine and in medicine for decades.
Now we have insisted, for example, that new drugs and new medical devices cannot be marketed to the public. They cannot be recommended unless they undergo credible testing. And typically that means a randomized experiment of the kind, again, that people are familiar with. And that's something that we understand and appreciate and live with and want in medicine.
It has costs to it, severe costs, in terms of people potentially, for example, dying, while effective medication is unavailable. while it's being tested. The problem is we don't know that it's effective until we test it. And there are many examples of medicines that were quite promising and that everyone thought were certain to work and ended up not being work or have backfired and caused harm.
And I thought, you know, entering the profession that the legal profession and the justice and the court system would be the same way that we would have the same dedication to evidence based thinking. Instead, which what we have in the legal profession is disdain, contempt or amusement for this type of evidence-based thinking and the adoption of lots of practices that are justified primarily because we've been doing them for a long time or they approximate things that we've been doing them for a long time or because of their adaptations of longstanding practices that are adapted because of resource constraints.
And so, for example, it may surprise people that the idea of bail, which almost everyone has heard of, is not something that is hundreds of years old. It is a relatively, comparatively speaking, a relatively recent invention in United States history; and it was adopted and put into place without an investigation into seeing whether it is effective at what it aims to do, or whether, the practices that were in place before bail were effective at what they intended to do.
In other words, all of the practices are adopted that way.
And so the Access to Justice Lab's purpose is to try to transform that way of thinking and that way of doing things into something that is evidence based, where we have evidence about what works and what doesn't, so that policymakers can make much better choices about the policies that they institute.
Tillman: Yes, it's fascinating to consider evaluating these practices in a similar fashion that medical practices are evaluated. We're going to get into really some of the reception that you've received from this work in a little bit. But how did you get interested in this particular type of research? Were you exposed to research? Sounds like I know the answer, but can you kind of walk us through your journey to this type of research.
Greiner: Sure. It was remarkably accidental. I practiced law for a while, but not in trial courts that dealt with high volume cases. I was doing other type where, where a lot of these statistical techniques can be of most use.
So you don't need a randomized trial to figure out, for example, whether a law is unconstitutional. That's not a scientific judgment. And that's the sort of law work that I was doing as a practitioner. When I started to be an academic after going for a stats degree, I thought that it would be interesting to have some studies that were analogous to things that I had heard about in medicine, never studied before, except in stat school, you know, studied some in stat school about how to do this.
I thought it would be interesting to do some in law, just like in medicine. But I thought that. There would be hundreds or thousands of these sorts of studies in law, the way there are in medicine. And instead to give you an idea of what happened shortly afterwards, a law student and I spent about two and a half years looking for examples of these types of studies in law from, you know, throughout the United States history.
And we found a total, depending on how you count them, about 50. And there are probably, you know, 10 to 20 or 30, 000 going on at any one time in medicine. You know, at any particular moment in medicine, so the comparison, whereas, you know, what we're looking at is all of the United States history. We could find about 50.
So there were very, very few of them. And I, and so I said, well, you know, maybe that's something that I can contribute to because I believe in evidence based thinking. And when I tried it myself, I can't encounter the hostility and the disdain of it. And the rejection of this type of evidence based thinking, and that's a brick wall that I think is standing in the way of adoption of good policies.
And when I see a brick wall, I like to punch through it.
Tillman: And for those who don't have the privilege of actually seeing us on camera, Jim has a brick wall actually behind him.
Greiner: I actually have one behind me. It's correct.
Tillman: I love that analogy. So why do you think there was, or there is so much resistance and disdain?
Greiner: So, I struggle to find an explanation that distinguishes law and medicine. Because in the 1930s, if you had talked about the science of medicine, people would have looked at you like you were crazy. Medicine was not a science. Medicine made a conscious decision to try to transform itself into a science.
And part of that was incorporating lessons from chemistry and biology and, and those sorts of things. But part of it was also a decision among the doctors. To give up control about what works and what doesn't, and they gave that control up to statisticians. And that was a wrenching decision that many in the medical profession fought.
But those who were dedicated evidence-based thinking, won that battle, or at least partially won that battle. There was no such attempt in law. Even though in, again, in 1930, there were occasional examples of using, using randomization in law and drawing inferences about, for example, whether assignment to Judge A versus Judge B makes a big difference in criminal justice outcomes.
That was a study that was done in the 1930s, and it took advantage of the fact that that cases were assigned randomly to judge a versus judge B. And so if you'd stood in 1930, you would not have known whether law or medicine was going to transform itself into an evidence based field. One did and one didn't.
And to go back to your question now, I struggle to distinguish between the two because all of the things that are quintessential about a profession, the idea of professional judgment being something very important, and the idea of having a statistician come in and tell you what works and what doesn't is threatening to professional development.
The idea of professional judgment, right? And what is good for each individual person as opposed to what works on average, which is what statisticians tell you all of that's true in both fields. And so honestly, I am the only thing that I can think of that distinguishes law and medicine is that rich people get sick in approximately sort of the same way as poor people.
And therefore, if rich people demand something that actually works, they will have institutions that protect their interests, poor people do not encounter the legal system in any way. Resembling the way rich people do, there is no similarity between them whatsoever, and that's not a particularly lefty statement.
That's just a statement of fact, right? And I don't think anyone. I think anyone of any political persuasion would agree with that statement. And so that is the only hypothesis I've come up with is far from something that I call proven.
Tillman: Wow, that gives you a lot to think about. But I can definitely and you're right, it's not based in any political ideology, it's really a fact.
And so in the systems that have been created and how individuals based upon our social constructs and socioeconomic factors, I'm sure play a part into some of these decisions. So, you know, when you speak of this type of research and the website for individuals listening when you get a chance to go look at the Access to Justice Lab website, because it does a beautiful job of it kind of walking through the process.
Jim, can you do that for us? Can you kind of explain what types of studies are you currently conducting? What do they look like? How are they designed? Who's involved? How are the results disseminated? Are communities involved?
Greiner: So basically the idea is that that either we in the lab or that someone working in the justice system or someone working representing or speaking for people who encounter the justice system comes up with some idea that they think will improve matters or whether it could be that they want to test something that the justice system is doing right now.
And we work with them to identify an alternative, both the thing that they would like to test and the alternative. So what does that look like in practice? Well, it might look like, for example, providing self help materials to people who are seeking guardianship, an order of guardianship over an incapacitated adult.
So grandma can no longer take care of her assets. She can't make good financial decisions and we need to establish guardianship. That requires a court order. Meaning a lawsuit. You have to file a lawsuit in the criminal context. It might mean, for example, providing risk assessment instruments to judges who are making bail decisions.
A risk assessment instrument is a quantitative tool that is designed to um, classify whether an arrestee is risky if released or is low risk if released. Um, and there's more complicated than that. That gets you the idea. And what we want to find out is, does that risk assessment instrument improve judges decision making?
And so once we've identified the idea we want to test and the alternative, typically the alternative is business as usual, whatever it is that we're doing right now. Then we work with local communities with local stakeholders, and typically that's a lot of them. So, for example, in the criminal justice studies that I mentioned in the risk assessment example, that requires working with the sheriff's office, law enforcement, with prosecutors, with public defenders.
With court administrators at the local level and at the state level, it requires work with I guess what you'd call county elders, whatever the governing body is inside the county or the city where you are working, and then county staff. And what we've learned to our chagrin in some instances is it's not enough to convince the top of any organization, you have to convince the people who are doing the actual work.
And so that's why I separate county elders and county staff, for example. And we have to get everybody on board. And the process takes months, sometimes years. And then we need funding because medical studies, these studies are slow and they are, they do require money for staff in order to produce the study.
And we set up a system whereby we try to have cases or people coming into a particular point. In the bail example, it would be when they're arrested and facing a judge for the first time to see what bail will be set. In the self-help materials example for guardianship it would be when people present at a legal services provider's office.
And we randomize them to one treatment or the other treatment usually to the new, the new idea or business as usual. And then we follow everyone according to defi outcomes that are defined up front, again in conversation with stakeholders and with the community. For example, in the bail study, what does the community and the stakeholders care about?
They care about failure to appear, whether people come back to court. They care about re arrest or re charge or re conviction for new, potential new offenses that occur during, if people are released during the predisposition period, especially if those are violent. They care about the number of days of incarceration.
How many, how many days are people incarcerated? And the reason for that is both expensive to the county, but potentially equally as perhaps more important incarceration, we think. is bad for the individual incarcerated, bad for the family, bad for the community. It prevents people from doing things like getting jobs, having relationships with their children, with their families.
For some people, temporary incarceration might, I emphasize might, in my view this isn't proven yet, but it might turn out to be true, might be a life changing positive experience, right? Maybe they get access to treatment, or to counseling, or whatever it is that they didn't have before. But, again, it's something that we can try to assess.
And so, we follow outcomes for everyone, and that takes a long time too, which is one of the reasons why these studies are slow and expensive. And when we produce the results, we report back to stakeholders, and we try to publish in academic journals, and we try to attend conferences where people of the type of folks who do this sort of work attend.
And let them know what our evidence shows, and hopefully they take it into account. The last part, in particular, has been a real challenge for the lab. We have, we have had real challenge getting by in. Even for people who have participated in the study. And that's something that is just that a human beings when you make a decision and start doing something day to day, it's very hard for one to accept the idea that what one is doing may be ineffective or may be counterproductive.
It may be producing the opposite of the desired aim. That is a very difficult thing for people to accept, particularly professionals who say, I have special training. I have specialized experience, the world thinks I'm cool because I have degrees and that sort of thing, and, and now you're telling me that my professional judgment is dead wrong.
Tillman: Yeah, I can imagine so. I'll get into when we start talking about like the IRB process that you went into. If we have time, I want to ask about like the informed consent process associated with this type of work. Because I think that's fascinating and it may look a little bit different than what we're accustomed to, even in medical or social science research.
Yeah. But what surprised you about some of your own findings from some of your research?
Greiner: Sure, so I have given up on being able to think that I can predict what the research is going to show, because I've been wrong so many times that it's just silly. To give you an example, we recently did, as I mentioned, a bail, a study about provision of risk assessment instruments at bail.
I thought they would improve criminal justice outcomes, providing the risk assessment instrument versus not providing it. I thought at worst, how could it possibly be harmful? Of course it has to improve. It's better, it's better information. So far it doesn't. So far it has no effect whatsoever. And it's unfortunate because producing the information, getting it into a type of format that a judge can use, and providing it to the judge, educating judges on what the information looks like and how to interpret it, that's not free.
That can cost counties hundreds of thousands of dollars, millions over time. Even if you try to automate it, it requires a lot of people to do a lot of computer programming. And so far, again, this isn't final, but so far we're not seeing any effect, good or bad. Many people had thought those risk assessment instruments would cause bad racial effects.
That they would increase the racial tinge, call it, that exists within our criminal justice system. Didn't do anything. To give you another from something that I did not predict, we recently published a recent about three or four years ago, a study of the effectiveness of a pro bono matching program for folks who want to seek simple divorces.
And what that meant was folks who had a very little in the way of income and assets, virtually nothing typically, and are wanting to get divorces would call this service and say, I have no, basically no assets, very little in the way of incomes. I can't hire a lawyer. Can you match me with a lawyer who's willing to volunteer to help me with my divorce?
And the matching service that had a stable of attorneys that it could call on typically had enough attorneys to provide lawyers to about 15 percent of those who called. It was completely overwhelmed. And so we randomized to whom they provided. And I got to tell you, I thought that matching service would not be particularly effective.
It often took the matching service months to come up with a match. Even when we randomized people to receive a match, it took them sometimes months. And the process was not supposed to be complicated because there were no assets and there were no children, there's no child custody issue. For the overwhelming majority of people, it's just, and this is almost a direct quote, “I don't want to be married to that bum anymore.”
In other words, I just don't want to be married. Right? Or I want to remarry, and I can't because I'm still married. Right? Or something like that. That's all they were seeking is just the piece of paper saying my marriage is over. In some instances, the spouses had not seen each other for a decade and they needed just the piece of paper.
And that pro bono mashing service turned out to be wildly effective. And it was quite surprising to me because, again, it was not supposed to be a complicated thing to go get this piece of paper, and, and yet it apparently was. And so, those are a couple of surprises.
Tillman: Interesting. Interesting. That's akin to the process. When you go in and you hypothesize that some system may work and, and that it doesn't, or some, or one would be a little bit more simple and it's more complicated, but there are positive results. That is definitely akin to the scientific process that many of our researchers experience. So, let's turn back to the Access to Justice Lab and some of the history behind it.
So, I'm going to read this and then we'll ask a question on the back end. So, the history of the Access to Justice Lab notes that for more than a century U. S. legal and adjudicatory practices have reflected an ethos of professionalism in which individual lawyers and judges purport to make irreducibly complex judgments about each client's or litigant's legal needs and the merits of their claims.
The belief is that only a lawyer knows what's best for a client. And the only person a lawyer should listen to about their client's legal needs is another lawyer. As a result, law currently recognizes only two sources of truth about what works and what doesn't. The pronouncements of legal elites and each individual lawyer's or judge's own personal experience.
With that in mind, when you present your findings, what happens when the data tells a story that the community does not like?
Greiner: That's quite a mouthful. Even just reading it, it's quite a mouthful. So just a quick explanation. The idea that each case is irreducibly complex. What that means, and I hear this constantly from lawyers, is that each individual is their own universe.
And there's no point in comparing one case to another case in any way, to another, one person to another person in any way. You can't learn anything about person A from looking at person B's experiences. Or you can only do that if you're a professional. There's no way for a statistician to come in and measure anything.
Only a professional can do it. And so, if that were true, for example, if each human body were irreducibly complex, there would be no point in studying whether penicillin, on average, helps combat, is an effective antibiotic, right? Because each person's body is so different. That there's no point in making any inferences.
Of course, we know that's not true. And we know it's not true in the case of, that I mentioned, of a lot of simple divorces. That they are, people have similar aims, and in some way they have similar circumstances. Of course, each one is a little bit different, just like everybody is a little different. So anyway, we asked what the reaction is.
And, as I mentioned, we've really struggled in getting people to buy into our research findings. Even people who have participated in the studies themselves and we attribute that to the fact that this type of thinking is quite alien to the legal profession. It's quite different from the way we, even I teach in law school because people come to law school in the first year especially and expect to see a certain type of education.
And I do provide that education because I, because they need it for their eventual practice if they're going to practice. On the other hand, I try to show them also there are different ways of thinking about this. But we have, for example, struggled to, to suggest to folks that they may want to consider, reconsider practices that they are currently implementing that we have found to be.
In some instances, mildly harmful and reconsider practices that are expensive but have no benefit. Try to find out why it is that a particular expensive practice works and take away the obstacle. For example, in the divorce study that I mentioned, where the idea is the lawyer is having a big effect on whether folks get a divorce.
The lawyer shouldn't have a big effect on whether people get a divorce. That divorce proceeding with no assets, no income stream, no child custody, no domestic violence in at issue. There may be those things in the marriage, but they are handled in separate legal proceedings. So, they're not at issue in whether someone gets a divorce.
You can just get the divorce and be done. That should not be, a lawyer should not make a big difference there. That is a indication of a severe access to justice problem. And the idea should be, take a look at your system, where are the hurdles? Actually the hurdles in this case are quite high and they are unnecessary.
And so, the example that we cite in that particular paper that everyone can understand is, for example, there was a particular moment in the divorce proceedings. When a pro se litigant, meaning someone without a lawyer, had to find and use an old-fashioned typewriter. I couldn't find one. I've actually sat down and thought if I needed a typewriter, where would I go?
I don't have a clue. Now, I might start with a local library, but I don't think they have one either unless it's locked up in a glass case as a museum exhibit. But that was an example of something that local practice. Even if it didn't formally by law required, effectively required, because there was a high, not certain, but high degree of probability that the legal piece of paper that you needed to send to the court would be rejected if you didn't have access to a typewriter.
And so that's the kind of examination that we hope our studies will engender. And again, this is not, this is something that we still struggle with at the lab.
Tillman: Absolutely. So, when we talk about your presentation at our annual conference in December, you spoke about conducting research with prisoners, particularly.
So, can you share about the work you've done, the research with prisoners, but also how you found the process, the regulatory IRB process for getting to that approval, your experience with that.
Greiner: Absolutely. So, just to be clear, the regulations, it's actually section 303C, I have it up in front of me, defines prisoners as anyone who is incarcerated at any point in the criminal justice system.
So, a prisoner isn't someone in a prison. Because at least in the legal system, we think of a prison as something where people, some place where people go after they are convicted. And a jail is somewhere people go before they're convicted or if they're convicted of only something mild. In other words, jails typically are temporary local institutions.
Prisoners are for longer term residence, but the regulations, I think probably properly don't make that distinction if you're incarcerated or even if your freedom is limited in a severe way and this is part of IRB training, if you are in a hospital and a guard is standing outside your door and won't let you out, that's an incarcerated individual, right?
And so there's, it's a very broad range and currently the regulations you can tell are Uh, written with the assumption that only medical research is being done with quote unquote prisoners, with prisoners under this definition. And you can tell that because, and again I have the provision up in front of me, section 305A3, so 45 CFR, 46, 305A3, requires an IRB to decide whether the risks involved in the research are commensurate with the risks that would be accepted by non-prisoner volunteers.
Now that makes total sense if you're talking about medical research. Right? Because there's nothing, frequently at least, there is nothing that is relevant to the research about the fact that people are incarcerated. You're just, you just need human bodies, and you need human bodies to, to undergo different forms of treatment and see which one is more effective.
And your assumption is that human bodies in prison, among incarcerated individuals are similar enough to those outside that we can still learn something. I don't know how to apply that subsection if what I'm trying to do is see if there are safe alternatives to incarceration. The only people who are interested in alternatives to incarceration are those who are incarcerated.
If, for example, what I'm trying to do is test an intervention that might allow a municipality to reduce its reliance on incarceration, what I am apparently instructed to do under this regulation is see whether people would want, for example, to be bailed out of jail if they are not incarcerated or if they are not in jail or if they have never been arrested.
And I find that difficult, a difficult thought exercise. More broadly, the assumption that only medical research occurs is built into other aspects of the regulations. Not just, it's most stark and obvious in the prisoner regulations, but it's built in elsewhere.
Tillman: And so I think you answered my next question, which is really, if you could inform some revisions, particularly to the provision around prisoners or even the definition, because as you were speaking and recalling your experience, I was thinking about, past experiences that I've had with researchers who were non clinicians who wanted to do research with incarcerated individuals and even just the definition and where the individual was located at the time really mattered and IRBs grapple with how to how to review this type of research and facilitate it while maintaining compliance with the regulations.
So, you know, what kind of suggestions would you have as far as revisions to our current regulations?
Greiner: Yeah, first of all, let me just say to some, I think history has taught us, forcefully and in a gruesome way that we need regulations. One doesn't, there are just too many horrible examples of inhumane treatment by researchers of study participants of acting, as you mentioned earlier, without consent and the idea of consent without acting with adequate information and regulation is necessary.
Oversight is necessary. It is no good to say, these are educated, highly educated people. They're professionals. We can trust them. That was true of the Nazis, that was true of the folks who perpetrated Tuskegee, that was true of folks who, you know, perpetrated Willowbrook, all of the horrible stories.
What are the, what should regulations around incarcerated individuals look like? I don't have complete answers, but I think one thing to do is if you are going to write a set of regulations that is specific to incarcerated individuals in the United States, which apparently, again, sections 303 and 305 are, are designed to do.
You need to incorporate a more thorough understanding of how incarceration works in the United States and what different stages of the criminal justice process are doing and designed to do. You need to make hard judgments, for example, about whether an individual on parole, whose activities are strictly regulated, but are not, but the individual is not physically incarcerated at all times.
Then there are some people who are halfway, thus the term halfway houses. You need to make judgments about them, and that's just definitional. That's just to figure out what you're going to require of them. You also need, I think, to be more receptive to the idea that research that is designed to, I guess this is the academic term, interrogate, to investigate current practices is high risk in the sense that the outcomes are important, but it is therefore all the more important to do.
So, for example, if we have, what we're trying to figure out is whether incarceration is necessary or whether we can decarcerate. That will mean doing a study where the, the intervention, some intervention like a risk assessment instrument or, or something like that, will change who is incarcerated and who isn't.
If it doesn't, there's no study and that is a big consequence. That is an important consequence. There is no way in my view to characterize that kind of study as low risk. On the other hand, death is a big consequence and we routinely implement studies in which the outcome variable is death within five years.
That's a typical cancer study or a major heart attack or something like that. A major cardiac event within high is within five years. Within five years, that is a typical hard study. And so the fact that the consequences are big, are salient, are important, is a reason to pursue the credible research, the rigorous research, not a reason to avoid it.
And the fact that the circumstances are unfamiliar, in other words, we say, my goodness, we're randomizing incarceration. That's a terrible idea, because we've never done it before, because we've only done it very infrequently. Whereas we randomize medicine that can kill people or save their lives all the time.
That's fine. That's not convincing. The fact that what we're going to need to do is to say, wait a minute, these are big circumstances. These are big outcomes. These are important outcomes in people's lives. These are real people's lives, but that is all the more reason to do the research because we don't want to be incarcerating people unnecessarily.
And on the other hand, for some people, and I'm not taking a position on this personally, for some people, we don't want to let people out of jail who we think are, have a very high probability of committing some horrible crime. There are some people who think that way too, right? And what we want to know is to provide them with strong information to allow policy makers to make democratic judgments.
Incarcerated individuals are different. And if the idea is to try to see whether we can, whether we need to dispense with or whether we need to increase or right size incarceration, we need credible evidence.
Tillman: So, my last question relates to a concept that you discussed at the conference, which was fascinating to many of us, and you mentioned the concept of thinking of IRBs as like a court, but also having this level of transparency to the determinations and precedent that's been set. Can you describe that, that concept more and what that could look like?
Greiner: Sure. So I think the problem that this is, that this idea, line of thinking is designed to address is inconsistency in decision making within an IRB and then inconsistency in decision making across different IRBs.
The latter problem across IRBs didn't used to be, I'm guessing, put it this way, didn't used to be as big a deal until scholars started to make routine across institution collaborations. So those are, those are just run of the mill these days. It is the exception as opposed to the rule that one scholar will pursue one IRB, one operation that requires IRB approval.
It's much more likely that there will be multiple investigators and they will be at different institutions. And therefore, coordination is something that has to happen. And what I have experienced and what I think others have, including folks, you know, who work in IRBs, is that there are differences in how folks approach the same study.
Or there are differences in how folks approach two studies that are essentially the same within an IRB. And I've had both experiences. I've had experiences where we, where it was essentially the same study for all intents and purposes for IRB. We just changed the language in some letters that we were sending out to folks who had been sued in debt collection from one study to the other.
And two IRB officers reacted quite differently and it had nothing to do with the language in the letter. The one that, the second one had, it was not citing the language in the letter at all, which is simply saying this study shouldn't have been approved to begin with. And in the cross institutional context, one IRB said that this is not human subjects research and therefore doesn't even need to go in, undergo approval.
The other one said this needs to go to the full board. We can't approve it at the staff level because we think it's high risk. I happen to, by the way, to think in that instance, the second IRB was correct and I wanted them to approve the research. It would have been much easier if. Yeah, for me, if they had said it's not human subjects research, but it was definitely, it was high risk and it needed to go through full review.
So what does that mean? There are things that the legal system has partially at least figured out to try to address this kind of inconsistency. And you mentioned one is the idea of precedent where there can be, where there is some sort of summary of a IRB application and then the decision and the reasons for the decision are written down and the underlying documents are accessible to the community so that we can see what it was that affected the decision makers process.
And say they did this before and this is similar to that. Of course it won't always be similar, but that's the point is to make a judgment. And that other institutions can look at the same thing. So they can also be available to other institutions. In other words, if University A deals with one case, one study in a particular way, then maybe that's a reason for University B to think that they should do the same.
Other ways of doing it are to have what amounts to, in the court system, an appeals process so that there is some higher-level IRB, I don't know what you'd call it, appeal, appellate IRB, appeals IRB. And what they would do is, again, issue reasons for their decisions that other IRBs could then look at and make and draw analogies to.
And say, if that decision is this way, then we should probably make the same sort of decision or make a different decision. The fact that those ideas might be useful to increase uniformity within and across IRBs. That's what courts do. They look at individualized circumstances and they make, they take complicated and vague regulations, sometimes regulations like the current ones that are not particularly well adapted to a new fact pattern, and they make decisions.
That's what courts do. Administrative agencies do the same when they're acting like courts, which is why I'm just calling them courts. And on the basis of those sort of published decisions or decisions that are accessible to the public, the people can have greater uniformity and predictability in what IRBs will approve and that also, by the way, that can then allow researchers to structure studies up front in order to anticipate what potential objections will be.
And that I think reduces workload. I think if we can solve, speaking for the, we here as the research community, if I can, if we can solve problems up front, we want to. Nobody likes for the IRB process to take months, and if we can anticipate and solve problems in the constructing the field operation based on knowledge of how people have ruled before and therefore how they're likely to rule in the future, we can potentially save everybody, including us, an awful lot of work.
Tillman: So how has that, I'm sure you received a lot of feedback at our conference, when you are talking about this concept, how has that been received?
Greiner: So, I think a lot of people have said, sounds great, but how would we do it? And I think there are ways to do it, to begin doing it, put it this way, even at low cost.
I mentioned that there are information aggregators who make it their business, being folks like Lexis and Westlaw and Bloomberg. What they like to do is market the fact that they have access to types of decisions or to things that people want. And therefore, if you pay for their service, if you subscribe to their service, they'll, you'll get access to their information.
And so, even simply having IRB officers write a one or two paragraph summary of the study, and then a one paragraph decision about it, that would be a first step. And again, I think if those were made in some way accessible to an information aggregator, the information aggregator would do all the rest of the work.
They would then accumulate the information and go to universities and basically extort us for subscription fees, which is exactly what they do. And we would pay it, right? And say, if you want access to this IRB, you know, these IRB, this IRB library, you got to pay the fee, right? And I would go to the university and I'd say, pay the fee. Right?
And so that I think is a one potential sort of free step that you that are not entirely because you'd still have to write the paragraph. But again, I think this is something you would tell me where I think this is something it would not be hard to do. Having finished a file, you probably got an idea of what the file looks like and a couple sentences about why you're approving it at the level you're approving it or why you're sending it on to the full board or something like that.
Again, I think this is something you just sit down and tap out. So that would be a first step. I think there are other things like an appeals IRB would require resources. And, but I think the publication part and the accumulation information part, you just take advantage of market forces.
Tillman: Fascinating. Fascinating concept. I would love to continue that conversation. So, thank you. Thank you, Jim, for your time and really opening our eyes to a discipline that can definitely benefit from collaboration with our ethics professionals. Thank you.
Greiner: Ivy, again, thank you so much for the opportunity.
##