Yonette F. Thomas, PhD: We have with us Dr. Teri Levitin, who is the Chief of NIDA's Office of Extramural Affairs. And we have Dr. Abe Bautista, who is the acting Director of the Office of Extramural Affairs at the NIAAA.

And we have, on the other side of the table here, Dr. Michael Micklin, who comes from the Center for Scientific Review. And as many of you may know, the Center for Scientific Review is the arm of NIH that controls and manages and ushers all of review across NIH. So Dr. Micklin is the Chief of Risk Prevention and Health Behavior in the Center for Scientific Review.

And we've got Dr. Jose Guerrier. And Dr. Guerrier is the Scientific Review Officer for the Division of the Behavioral and Social Science Approaches to Preventing HIV/AIDS study section. I know many of you will be submitting applications to that section.

We'll start off this morning with an overview presentation from Dr. Levitin to kind of give you a kind of orientation. And then each of our other participants will give you a flavor from their perspective, in case of Dr. Bautista, from NIAAA perspective on review. And then Drs. Micklin and Guerrier will talk a little bit about, give you an overview of specific study sections, give you an idea of a unique focus for their particular study sections.

And then we'll go to a question and answer dialogue about review, what review is, why it's important to pay attention to review, understanding the review process, what those pink sheets mean and what you do with them when you get them, and that sort of thing. And paying attention to them and being responsive in an elegant, professional, and careful way to those reviewers' comments.

So without further ado, Dr. Levitin.

Teresa Levitin, PhD: Hi. Thank you for the opportunity to speak with you. I am going to do an overview of what is going on at the NIH because there are some very profound changes going on that I wanted to share with you. I also wanted to say on behalf of my colleagues that everything we're going to tell you today is true, but it's true today, because so much is happening that we can't guarantee that what we say a week from now or a month from now or a year from now, certainly not a year from now, will be the same as what we have said to you today. So that underscores the importance of your staying in touch with staff at the institutes that are relevant for the kind of research you want to do.

So if there's one takeaway message that I want to give even before I start, is get in touch with us. It's always important to do so anyway because we can provide you with a lot of assistance and advice and help, and that's why we do these jobs because we really are interested in helping you to develop the best science possible for the public health.

But also it's even more important now than it has been in the past to get in touch with program officers, with review folks, with everyone who can assist you at the institutes because with all of the change going on and more change to come, it's confusing sometimes even to us. But we do want to be there for you and help walk you through some of those changes.

Teresa Levitin: So with that, let me begin by saying, as you can see from the slide, there has been now more than a year of effort, gathering information, gathering feedback, input from a wide variety of sources in order to help enhance peer review.

There was not the belief that it was broken, but rather the belief that there were opportunities for enhancing what we do because we do it very well, we believe. And you only need to look at the kinds of awards and Nobel prizes and other things that are awarded to our grantees to know that, yes, this system does work, but, yes, there is room for improvement.

And so we requested information. There were surveys. There were white papers. There were town meetings. There all sorts of activities during the diagnostic phase of this very extensive review of how we did peer review.

And this is peer review writ large, by the way, it's not simply how were the study sections operating, but how was the whole process operating from program, to review, to grants management, to make sure we really are funding the best science by the best scientists with the least amount of administrative burden.

That's what Dr. Zerhouni, the director of NIH has said is to be the umbrella under which all of this activity takes place. And so every time a change is recommended, we are asking ourselves, "Will that change enhance funding the best science by the best scientists with the least amount of administrative burden?"

Teresa Levitin: And so an oversight committee has established working groups, and those working groups are developing priorities and procedures that will help us accomplish the following three priorities. Actually, there's a fourth priority that's more an internal thing of continuous review of whatever it is we do.

But the first priority is to engage the best reviewers. And there are ways of, we hope, making the whole review experience more intellectually and scientifically more attractive and interesting.

Now one of the things, of course, in order to engage the best reviewers is having the people we contact and ask to participate, say yes. So the other takeaway message that I'm going to say now and that I'm sure my colleagues will also say and I'll probably say it again, too, is if we call on you to sit in on a peer review committee, please say yes. Don't say no, because our review committees are only as good as the reviewers who agree to serve on them. And it is a fantastic educational experience for you as well as a service to the scientific community to serve on a review committee, either at an ad hoc fashion or as a permanent member.

Anyhow, the first priority is to engage the best reviewers. And I'll go over how each of these is to be operationalized in just moment, but I want to just go through what they are to give you a sense of how comprehensive this review has been.

The second priority is to improve the quality and the transparency of review. Now, again, a lot of inroads have already been made on that now that so much is available in the Commons where you can find information about your review, about any number of other things.

It used to be before the electronic age that staff were gatekeepers of information. That's no longer the case. Really, you have access to information, but sometimes it's overwhelming. And what we want to do is simplify and modify our systems to make them more transparent, more effective, and more efficient.

The third priority is to ensure balanced and fair reviews no matter what the scientific field, no matter what the career stage. And so you can see we're very interested in supporting early stage investigators well as ensuring that different kinds of research, whether it's clinical research or basic research, is fairly and appropriately reviewed.

And then that fourth priority is to ensure that we do continuous evaluation and make sure that the changes we are recommending and implementing do, in fact, improve the process and enhance peer review as intended because you always have to watch for unintended consequences. And because they're unintended, sometimes you don't know until they show up, and then you have to stop and remedy if you've, in fact, made a problem that you didn't expect.

Teresa Levitin: Okay. So for priority area number one of engaging the best reviewers, let me just show you that these are some of the suggestions to improve reviewer retention, providing more flexible opportunities to serve.

We're talking about additional training for reviewers and allowing for more flexibility through virtual reviews. Sometimes it's just very difficult to get people to agree to come into the Washington, DC, area for a review, but we think that we can have more meetings and provide greater flexibility if they aren't face to face. And the question is, can we still guarantee the kind of thorough, thoughtful reviews through different media other than face-to-face?

Teresa Levitin: Second, implementation overview for priority area two, we're going to change the scale. As you know that one to five scale with increments of .1 was, as many psychometricians told us over and over again, much more complex than really was reasonable given the task at hand.

So now we're talking about an one through seven scale which will begin in May of 2009 where, again, there'll still be the critiques, but a different scale will be used. And reviewers will also be asked to scale each criterion because, again, psychometricians have told us that rather than simply giving an overall score, which will continue, if scores are also given for the individual criteria, such as approach or investigator or significance, that that leads to a richer and more psychometrically appropriate kind of scaling and review.

We are also talking about shortening the RO1, which, of course, is the standard against which other applications are measured. And so if that's a 12-page as opposed to what it is currently, a 25-page, application, then other smaller and different mechanisms such as the RO3, which is a small grant, will also follow suit in its changes in page length.

Again, all of this information is currently available on the Web and will be updated, but I did want to give you this overview now.

Teresa Levitin: And then priority area three, again, we have talked about these possibilities, including reviewing all of the ones from early stage investigators, those are basically new investigators, people just outside of their last training within a ten year period. And these are some of the recommendations that have come out of the diagnostic phase, which as I said a moment ago and showed you from the slide, involved people from all areas of the scientific community as well as staff.

Teresa Levitin: Okay. Now a plug for NIDA about what we can look for in NIDA. Many of you I know are familiar with our MIDARP program. It has now been changed to the D is for Diversity. There will be some changes in review criteria. We'll be looking still at the significance, approach, innovation.

And what I want to do because I shouldn't put in a plug solely for NIDA, but just encourage you to think about the possibility of this program and to tell you that program staff will be available to help you develop applications in this area.

Teresa Levitin: And now I'm going to close the way I began, which is to encourage you to be proactive, to encourage you to get in touch with us, to get involved with people at all levels in the Institute who can help you. I typically say you're three phone calls away from exactly the person you want to talk to. It may be three e-mails now or some combination.

But I urge you really talk to many different people because what will happen if you talk to somebody in review, somebody in programs, somebody in grants management, somebody else in program, and whatever, you'll hear different points of view and different perspectives.

That may be a little confusing initially, but what it really does is allow you to think through what is the research you want to do, what are the mechanisms that are most appropriate, what kind of assistance do you need, all of those kinds of things. And different folks can give you different perspectives on all of your questions.

And then last, but by no means least, when we call, don't say no. Please, just say yes. Because as I said earlier, and want to emphasize, the quality of review we can provide to you is only as good as the quality of people who come to our study sections and agree to participate in the process. So please, this is not original, but let me just say, be the change you want to see. Thank you.

Abraham Bautista, Ph.D. I'm not sure how many of you are funded by NIAAA, but I'd like to emphasize that NIAAA funds applications, of course, that deal with drug abuse and drug use disorder. And we have wide range of topics that any one of you can apply.

We have four major divisions. These divisions are the ones that are responsible for the programmatic issues of NIAAA. They are the Division of Metabolism and Health Effects. And this division funds applications that build with the biomedical action of alcohol on multiple organ systems other than the brain.

And then we also have the Division of Neuroscience and Behavior that deals with animal models and human studies focusing on the brain, and also focusing on the brain when it comes to fetal alcohol syndrome.

We also have a Division of Epidemiology Prevention, and we also have a Division of Clinical Treatment and Health Services. These are the four major divisions. And if you are interested in applying for a grant with any of the topics under the division, I would urge you to talk to the program officers under these divisions.

I'd also like to emphasize the terminologies between program officers and scientific review officers. They're all health scientist administrators, but they have two distinct responsibilities. Program officers are responsible for the programmatic issues of the NIAAA. And they'll give you advice on which area is the top priority for NIAAA.

Abraham Bautista: If you submit an application, and even if you get a score of 1.1 and if NIAAA is not interested in it, we will not fund it. So you have to talk to the program officer about the topics that we are interested in. We also issue a lot of RFAs and PAs, and you can go through this information in the website, and that will give you an idea on what type of research project you can do.

I'd also like to emphasize that although we are doing alcohol studies, we also accept applications that deal with drug abuse as a comorbidity factor because as you know, these two agents - one is legal, one is illegal - always go hand in hand together. But when you submit an application with NIAAA as the primary institute, of course, you should focus your studies more on the alcohol aspect rather than the drug abuse, and vice versa.

But sometimes we also co fund applications from NIDA. So that's also a very good opportunity. When you submit an application, you can request CSR to assign either NIDA or NIAAA as a primary institute and either NIDA or NIAAA as a secondary institute, because if the primary institute doesn't want to fund your application, we may consider that.

Abraham Bautista: All applications go to CSR for processing. And most of the applications, if not all, are reviewed in CSR. NIAAA also reviews applications with the exception of RO1s. All RO1s go to CSR for the review; however, there are certain RO1s that we review. And these are clinical, treatment, and health services. If you submit an RO1 on epidemiology, it will go to CSR. If you submit an R21 on epidemiology, NIAAA will review that.

Within NIAAA, we have four standing study sections which are AA1, AA2, AA3, and AA4. AA1 deals with biomedical action of alcohol except the brain. AA2 deals with prevention, epidemiology, and minority studies. AA3 deals with clinical, treatment, and health services. And AA4 deals with neuroscience and behavior, dealing with both animal models and also human studies.

And we also review all applications except RO1, as mentioned before. The training centers, P20, P50, these are alcohol research centers. And for the alcohol research centers, we also do a reverse site visit whereby the applicant comes to Washington, DC, to be interviewed by the reviewers. And we also review all of the RFA, which are specific subject matter applications. And in this instance, if you are responding to an RFA and applying for an RO1, then NIAAA will review that kind of application.

And I think that's about it. So if you have any other questions, we can do that later.

Yonette Thomas: Okay. Well, thank you. Before we transition to the totally SRO side - and SRO, Scientific Review Officer, for those of you who this is all alphabet soup, and CSR, Center for Scientific Review. And so on the right side of the table you have principals at two institutes at NIH who represent both the review side and to a certain extent a broader programmatic side, and on the left is all review, and the two doctors, Micklin and Guerrier, represent the Center for Scientific Review.

Audience: Thank you, Dr. Levitin, for the full review of anticipated changes. And I was wondering if either you or Dr. Bautista could comment on any anticipated changes in your visions of priorities in relation to the funding and support of new investigators? And I'm thinking chiefly for the R21 and the R34.

Teresa Levitin: The important thing to recognize is that the NIH is committed to its new investigators or young investigators, although we don't use that terms as much anymore - early investigators. And we are guaranteeing that we will continue to bring in a substantial number of new people into the institution.

I mean the next generation is our life's blood, so there may be changes in page length. There certainly will be changes in how these applications are scored. But in terms of commitment to the new generation of scientists, that's absolutely sacrosanct.

Abraham Bautista: We are trying to encourage a lot of new investigators or early stage investigators to apply to for an RO1 instead of an RO3 or an R21 because preference is given to those RO1 applications when it comes to consideration for funding when you submit an application.

As a new investigator, if you submit an R21, as a new investigator, we will not consider whether or not you are a new investigator for that when it comes to funding, so there will be no preferential treatment. But for RO1, you have a better chance of doing that because we are really trying to encourage a lot of new investigators to get into the RO1 mechanism as early as possible if you're ready to do one already.

Yonette Thomas: And I just want to add on the NIDA end, and I know Teri was intending to say this, that we do have a commitment to new investigators. It is the lifeblood, and we make every effort whenever possible to find,squeeze that extra dollar to make sure that a new investigator gets funded.

Yonette Thomas: I'd like to move on to Dr. Michael Micklin to give us a perspective on the CSR end. Michael?

Michael Micklin, Ph.D. Yeah. Good morning. I'd like to apologize for Gayle Boyd. She was supposed to be here, but has been ill for the last three days. And she just sent me an e-mail this morning saying that she didn't think she should operate any heavy machinery, meaning driving a car. So Gayle had planned to be here. And, in fact, she and I put together some handouts.

These are all available at OER or NIH Web sites. OER is the Office of Extramural Research. One's An Insider's Guide to Peer Review for Applicants. One is What Happens to Your Grant Application? A Primer for New Applicants. And then the third one is a set of Web sites that would be useful for those of you considering a submission of a research application.

Michael Micklin: Alright. I am the chief of behavioral social science IRG, or Integrated Review Group, at CSR. My group has five standing study sections, one dealing with addictions - that's Gayle Boyd's study section - one dealing with social psychology, personality, and interpersonal relations, one dealing with developmental psychology and family relationships, two of them dealing with behavioral medicine, one mainly in terms of prevention and the other in terms of outcomes and rehabilitation.

We also have two groups that over the past three years have been established. They're both special emphasis panels. One meets once a year, and one meets twice a year. The once a year group deals with applications that are responding to research on health disparities, and the other one, the one that meets twice a year, has to do with research on health communications. Both of these are program announcements that are coordinated by the Office of Behavioral and Social Science Research in the director's office. And they are going on for another two years, and I suspect they probably will be extended because of NIH's concern with the two topics.

I also have a group that deals with fellowships. We get maybe 30 to 50 fellowships each round. There's another group that deals with SBIRs, the Small Business Innovation Research grants, SNSTTRs. There is a fairly sizeable number of applications we review each round, somewhere between 70 and 90 of the SBIR/STTR variety. So it's a moderate size group within CSR. It is clearly focused on social and behavioral science. And for those of you who are interested in addictions, we do have a single study section that is focused specifically on that topic.

Michael Micklin: What kind of advice can I give you? Well, one of the first things I say when I talk to groups around the country in terms of how they ought to begin thinking about a research application is don't write anything down until you talk to your program officer. Now many of you will not have any idea who that program officer might be. But they don't bite. Call them on the phone. Give the person that you contact first, and you can get phone numbers off the Web sites, tell them what you're interested in, and they will guide you to the right program officer.

What you want to do with the program officer is first find out if the institute is interested in the topic at all and how much. Second, what sorts of wrinkles might you develop in terms of channeling your application theoretically or methodologically or in terms of substantive issues outcome variables that would be of more interest to the institute.

And as you develop your application, many - not all, but many program officers are willing to take a look of portions of it before you submit it. So I think it's really good advice to get as much help from the program officers as you can.

Now as was mentioned earlier, there is a division of labor at NIH between program and review. When you're developing an application, you should be in touch with program officers. When the application has been submitted and is under consideration by a review panel, then the SRO or the scientific review officer is the person you should contact if you have any questions. They cannot comment on the substance of your application. That's up to the program officers.

So if you've submitted an application, then get in touch with the scientific review officer.

Many, if not most scientific review officers, once they have a set of applications to be reviewed - and for standing study sections, we review applications three times a year - you will get an e-mail from that SRO indicating that your application is being considered in the next round of applications in X, Y, or Z study section, giving you information about what dates you need to keep in mind in terms of submitting supplementary information. They will advise you to please take a look at your application and make sure it has been received correctly. And they will tell you more or less when the review is scheduled. So that's the information that you can count on from the scientific review officer.

Jose Guerrier, Ph.D.: My name is Jose Guerrier and I'm the scientific review officer for the Behavioral and Social Science Approaches to Preventing HIV/AIDS. And that study section is under the aegis of the AIDS and AIDS-related research IRG, which used to be, until recently, under the Division of Biologic Basis of Disease.

One of the things I wanted to mention is that there are three review meetings usually per year. But the AIDS IRG is a little offset in that we are under what is called an expedited review approach. Applications, therefore, are submitted at different timeframes than for the rest of CSR.

So one of the things I would like to point out to investigators is to find out what the submission dates are for applications because as often occurs, people send an application at an off date and are shocked that it is not being reviewed when they expect. So for the AIDS IRG, for instance, applications are to be submitted January 7th, May 7th, September 7th, and they are reviewed in March, July, and November.

One of the other, I think, in terms of under our AIDS IRG we have I would say almost seven or eight study sections. I want to give you just generally what they deal with. As I told you, mine deals with behavioral and social science approaches to preventing HIV/AIDS. As such, most of the applications that are received in my study sections have to do with prevention, whether it be looking at how one can reduce substance abuse to also prevent HIV/AIDS, intimate partner violence, and all other correlates or hypothetically believed correlates of HIV/AIDS.

We have a study section, the other behavioral sister of the BSPH, is called the BSCH, which is the Behavioral and Social Science Consequences of HIV/AIDS. And most applications I get there are usually applications for individuals who are already HIV-positive, and it deals mostly with healthcare maintenance and things of the sort. We have a study section on vaccines, neuro-AIDS, opportunistic infections, and epidemiologic study section.

And I'm saying all this and Dr. Micklin mentioned that as well, is because under a specific IRG, we are the AIDS IRG, but it's important for the investigator to know pretty much something about a study section where he or she might wish to have an application reviewed. There is a blurb about each and every study section on the Internet. So the PI should also do some homework in trying to find out what each study section does. Standing sections have also what is called standing committees. There's a core group of reviewers who are nominated, and if accepted, serve for a maximum of four years, and the roster is on the Internet.

Jose Guerrier: One of the reasons I'm mentioning that is very often people send applications in, and they worry about expertise or mention that the expertise was missing, et cetera. If something arcane is being mentioned very deep into an application, while the SRO does not review your application for scientific merit, this is left to the reviewers. But we have to take a general look at things: read your abstract, maybe even your research design, to determine what sort of expertise is going to be needed.

As such, we don't read each and every page of your application. Therefore, if there is some arcane science or something that needs to be paid attention to, remember that the PI has the opportunity to also submit a cover letter. In the cover letter, it might be important to highlight that maybe some type of expertise might be considered. Do not provide us with names of particular people. The chances are we'll ignore them. You may speak about general expertise, maybe, that one may want to consider that may not be very evident at first.

The other aspect that I think is important to remember is also that one needs to understand the mechanisms for applications. We receive mostly, in our study sections, we review mostly RO1s, which has been mentioned before, the standard by which people also measure success and other sorts of things, but we also have the R21s which are exploratory in nature, the RO3s which are smaller, quick turnaround.

It is important that the PI understand the requirements for each of these. One of the reasons for that is, for instance, an R21, which is more of an exploratory mechanism, does not require any preliminary data. Very often, though, people may present preliminary data. While it's not required, if you provide it, it becomes a target. So it's important that one should know what is required for each mechanism as well, and, of course, remain within page limits and the like.

Jose Guerrier: The last thing I would like to point out, and then we can take questions, very often the job of the SRO generally is to oversee, not only to try to find individuals who are competent in the subject matter, to oversee that the review is conducted in fair manner. As has been stated before, our purview is scientific review. Consequently after the review, any question having to do with the outcome of review, is not to be directed to us, but to program, and we state it as such.

There have been instances where people call, even beforehand. True, one has to contact the SRO regarding supplemental materials that can be submitted. Now there are limits as to how much time. For these 30 days before the meeting, supplemental materials, if you have any, should be provided. Otherwise, we cannot take them in. And we don't provide paper anymore as such, so the electronic versions of them are presented, and they are placed in a grant folder.

So this is about the limit of what we can do. The job of the applicant is not to convince the SRO of the merit of his or her study, but most of the reviewers or the peers.

And the last thing I wanted to mention are some of the most frequent, I don't want to call them criticisms, but critiques, often that one hears of review, are such thing as, for instance, an application is presented, so one will see no theoretical framework presented. So scores get dinged on that. Poorly integrated theoretical framework, if one presents that say either one or even multiple ones.

Overly ambitious projects sometimes in vein of impressing reviewers, people may be promising things that are logistically challenging, if not impossible. So, again, another issue of things, or that the person may not have demonstrated any background or past experience in doing this matter.

So these are the general things that I have to say, and if you have any questions, we'll take them afterwards.

Yonette Thomas: What are supplemental materials? How do you know that you can submit them? What are they?

Jose Guerrier: Well, now actually one receives e mail to the effect that the PI is informed about the limits of time. But by supplemental material, we mean such things as, for instance, publication, a new publication that the PI thinks might enhance the strength of the application and also the strength of the PI, the capabilities of the PI.

Letters of support, for instance, that may have been received late. Because often you contact somebody that's in South Africa, and for some logistics, you don't get your letter on time. That's also supplemental material. Or noticing that a figure is missing that might be important.

So these are examples I've given you. But should you have questions regarding the nature of supplemental material, this would be a case where you would call me or an SRO to ask and we would guide you accordingly.

Michael Micklin: With publications, it's important to recognize that we don't accept manuscripts that have been submitted but not accepted. We will take a manuscript that has been accepted for publication by a journal or an actual reprint of a published article.

Teresa Levitin: I wanted to go back to Jim's point about new investigators because looking around the room, I suspect many of you are. It is true that when you're just stating out it can seem like a very formidable, if not impossible system, the NIH system, which is why there are so many people, program staff, available to you for assistance. And I want to stress that learning to write an application is a skill, and it's a teachable skill. And a lot of what people like us do is to help you learn that skill.

Now there's no substitute for a good idea. We can't provide you with a good idea. We can't tell you enough the importance of having a good idea that you feel passionate about. Because if you do get funded, you'll be spending three to five years of your life on that initial grant studying the topic. And so it should be a topic that you really care about, not one that your mentor or advisor has given you to do.

But assuming you've got a topic you really care about, assuming you've got a really good idea, but you don't know how to craft an application, come talk to us. But also talk to local people. Talk to your mentors. Talk to people in your department or wherever you happen to be who have been successful grantees.

Get on the NIH Web. There are a number of institutes, and NIAID is really one of the better ones, that has training kind of material that will walk you through, helping you to learn that skill of writing a good application. Because it is something you have to learn, and it is something that can be taught. And you can really take advantage of resources that are available to you, both at the NIH from people, and online, as well as from people that you work with, your friends and colleagues.

And what we always recommend is when you've drafted something, that you show it not just to your closest professional friend, but you show it to someone who really is not in your area of research and doesn't know the language of your particular discipline or sub-discipline, because the study sections are multidisciplinary. And it's important that you be able to write in a way that will be clear and persuasive to everybody on the committee. So always be prepared to show your applications to many different people before you submit them to the NIH.

Michael Micklin: That's really good advice. Many research institutes and university departments will have a required review before the application goes out. So it's always a good idea to get someone who knows the subject matter and has had successful experience with grant applications to review your work.

Michael Micklin: One other related point. Oft times, especially new investigators need help in conducting the work they're proposing. So they get well-known people to say, "Yeah, I'll come on as a consultant for this or that." Make sure that if you get a consultant that's going to help you with your project, make sure that they actually read what you wrote. You don't want to just throw some names out and assume that the reviewers will say, "Oh, yeah. He's got John Smith on his application, and that means it's going to be good.

Because oft times I find that applications that are heavily criticized are those that have not gotten the expertise integrated into the application sufficiently. You've just got the name without the expertise showing up in the application itself.

Jose Guerrier: Just to add to what Dr. Micklin has just mentioned, the advice of having someone with the expertise on your application to consult. Another aspect, also, is the fact that very often the name appears, but the time allotted to the individual, to the expert to contribute appears rather feeble. That is, the application may read well, but if the person is there for only .2 percent of the time, there will be some question as to what sort of substantive contribution will be made.

So, again, there has to be some validity, some consolidation of what one is presenting.

Audience: I have not had yet received NIH funding, and I recently submitted an RO3. And the issue that came up as far as paying someone because the grant's comparatively small, only $50,000.00 per year. So my mentor, I included her, and she's donating 6 percent of her time. But what is the expectation with small grants when the money isn't completely there to pay them?

Michael Micklin: There is no standard expectation. You do what you can with what you got. In other words, $50,000.00 doesn't go very far, obviously. But if you can get people to commit to a percentage of time, 6 percent you mentioned. That would seem to be important, 6 to 10 percent of their time. And especially if it's a two-year application or a three-year application, especially in the early phases to help you get started, I think that will make an impression on a review group.

Audience: I would appreciate some clarity about the role of the consultants on a proposal. I work at a research institute where we have a number of different grants. Our work's organized by grant and by research team. And there were a number of people we'd like to have on our proposals, but we're all under the same auspices of the same research institute even though they have different areas of expertise.

Would you advise that we have them on as a consultant, as a co-I status is something that we've also been considering. But could you talk about the pros and cons of kind of how we include people in our grants?

Abraham Bautista: You have to be very careful in doing that because if a consultant contributes significantly to the science of your application, then they should be included as a co-investigator. If they're just providing you with advice or providing you with patients or samples, then they should be a consultant.

Audience: Does it matter that we're all under the same research institute?

Abraham Bautista: It really doesn't matter. Yes, as long as you need the expertise, if you need the expertise. Because there are some reviewers that'll say, "Oh, this person is doing this, but there's no expertise." But if you have a certain expertise that is really required without having that person as a co-investigator then that person should be used as a consultant.

But don't put a name of a person as a consultant just to enhance the quality of your application if they have nothing to do with it, because there are some people who do that. They just put these big names attached to the application thinking that it will enhance. But the reviewer will look at that as a red flag: "Why? What is this person doing?"

Michael Micklin: One other caution I would add is when you're thinking about consultants and co investigators, you do want people that are necessary to help you conduct the work. But you don't want everybody out there on your application who might be a potential reviewer.

Because especially in very specialized areas, you can eliminate all the competent reviewers because you've got them on your application. Or the competent reviewers may be in conflict because they've published with the people that you've got on your application. So think about that as well. It's somewhat of a specialized concern, but it does appear every once in a while.

Audience: Good morning. My question is in relation to the pathway to independence award, the kangaroo. I know that I was not available for them. But it seems like a really good mechanism, so I was wondering if any of you could comment on that one.

Teresa Levitin: It is a really good mechanism. This is a combination K, which is a training mechanism, and R, which is a research mechanism. We call them kangaroo because they get the K99/ROO designation.

It's an opportunity to help people make the transition from training status into an independent scientist status. So it's just one of many kinds of mechanisms that are available to you. And detailed information, of course, is available on the Web. And I'll segue into my mantra for today, which is call a program officer, and you can get additional information as to whether or not that's an appropriate mechanism for you.

Abraham Bautista: Okay. I'd also like to add that if you receive a kangaroo award and when you apply for your independent RO1 later, you are still considered a new investigator.

Yonette Thomas: Yes, that's very true.

Yonette Thomas: Well, I'd like to ask of you to tell us a little bit about summary statements. What do you do with a summary statement? What is a summary statement? Why is it important, and how do you deal with it?

Teresa Levitin: You know what a summary statement is? No. Okay.

What the reviewers do is provide written critiques which are the basis for the discussion that takes place during the study section meeting as to the scientific and technical merit of your application. For those applications that are discussed during the meeting, the scientific review administrator, that's the staff person who is responsible for managing the meeting, that person puts together what's called a summary statement, which is the critiques that have been provided by the reviewers, plus a resume and summary of the discussion. And that is the official document of what happened with your application. Those summary statements are available to you.

And the question always is if you get a summary statement that indicates a poor score for a variety of reasons, what do you do? We can all talk about that. But all I will say at this point is, don't give up. It is a very difficult thing to read a summary statement that provides criticisms of something that you thought was really terrific because you wouldn't have sent it in if you didn't think it wasn't terrific.

So here you sent in a terrific application and you get back a summary statement, and there's this criticism and that problem identified and this criticism. And it can really be a very disheartening kind of thing. And it's important to realize that the summary statement is not a criticism of you as a scientist. It doesn't mean that you have no career in science or anything like that. All it means is that a particular group of people at a particular point in time had these particular things to say about this particular application that you sent in.

And so it's really important to be able to read a summary statement, to step back, perhaps to think evil thoughts about the review committee, but then after you've gotten past that, to look to see what they actually said. And if the reviewers suggested that the idea itself was problematic, then that's a more serious kind of criticism than if the review committee said, "This measure is of concern," or, "This particular sample may not be ideal." Those are the kinds of things you can remedy.

So the question is, were the concerns raised ones that you can remedy, or are they more serious in nature? And again, reading that summary statement, going to colleagues, getting their help, and also getting the help of NIH staff if you decide to revise and resubmit.

Abraham Bautista: There are normally at least two critiques, individual critiques in the summary statement, and these are virtually unedited. The SRO will edit it for grammar or some implementary statements, but there are some summary statements that will have two conflicting individual critiques. And in that case you should read the resume and summary of discussion because that will capture the outcome of the discussion, because the panel might be swayed either way.

And that's also very critical because there are some issues that are not in the body of the critique, but it may be in the resume and summary of discussion. And when the application comes back, the original reviewers may not be assigned to this application. It will still go to the same panel, and some of the members of the original panel would remember what was discussed. So you have to read the summary statement very carefully, especially the resume and summary of discussion.

Michael Micklin: I'll say a couple of things. I just want to repeat Teri's advice to step back and think about your summary statement before you fly off the handle and start charging around with a gun in your hand, figuratively, not literally. Think about it, and think about the critiques written by the various reviewers.

Actually, what you ought to do is go through each critique and try and summarize the principal points that are being made. Ask yourself, "Are these theoretical points? Are they methodological? Are they design points? Is there something to do with the outcome measures, independent variables, whatever?" That way you will have an idea about the consistency and the degree of agreement among reviewers about what's wrong with your application.

We advise reviewers not to use the term fatal flaw in their critiques because one person's fatal flaw might be another person's innovation. But do ask yourself, "Can this project be rectified? Can it be rejuvenated? Can I make the changes and resubmit it?" Sometimes, and perhaps with the relatively low pay lines that NIH has now, it might be better, more often than not, to actually start over again rather than to try to resuscitate an application that has gotten very poor comments on it.

Yonette Thomas: I think there was a question. Yes.

Audience: Yeah, more of a comment, actually. Just a comment on those summary statements for maybe people who haven't gone through it. I just went through that for the very, very first time. I submitted an RO3 to NIAAA in the spring, and I was a post doc at a top notch institution and had put together a great team, and I thought I had a great idea and felt very good,had two internal reviews before it was sent out. And it wasn't even scored.

I spent time going through the comments. And, in fact, all three reviewers really liked the ideas. They talked about the high public health significance and the terrific innovation of the idea, and gave me very, very specific comments on why the result was the result and it wasn't the methodology. And once I was able to take my ego out of the equation and sit down with some of my mentors and take a second look at, all of the comments were really, really good.

I have been in contact with my project officer and have identified a couple of additional team members, and we're going to be resubmitting it, actually, as an R21. And I feel very good about it.

So it is really devastating because we tend to get our ego involved in it. But when you actually take the time to look at those comments with the ego out of the equation, these are really smart people. They've been doing this for a very, very long time, and they may have seen things that you may not have seen. So it's important to do that, to take yourself out of the equation. I just wanted to share that.

Michael Micklin: Just one comment on un-scoring. We have a practice now of trying to discuss half of the applications that are submitted for a particular meeting. So we actually take the provisional scores that are provided by the reviewers before the meeting and look at where the cut line is. Where's the median?

Now we do this separately for RO1s and other mechanisms. And if we have a small number of mechanisms, small number of RO3s, small number of R21s, many of us typically make sure that at least half of those are discussed.

But the general point is about half of your applications are not going to be discussed. Doesn't mean they're bad. It just means that they didn't, in this particular set of applications, fall above the median.

Jose Guerrier: One thing I also wanted to add not only about that is, and I think this has been pointed out multiple times is, you need to take into consideration that a study section is multidisciplinary. Consequently what may appear very clear to you in your discipline may not be so to others. Therefore, when one reads a summary statement, what is expressed may be the reviewers' misunderstanding or a perception of a lack of clarity in a specific area.

And I think it was Dr. Levitin or someone else who mentioned when you are writing, you should be writing for somebody other than someone in your discipline. If someone else can understand it, then maybe it is clear.

Jose Guerrier:The other aspect is, and I think Dr. Thomas asked, how does one respond to a summary statement or to the critiques? Dr. Bautista mentioned, and it is a fact that, now we are asked more and more to provide shorter summary statements; therefore, the summary statement we provide will be a kernel of the discussion that occurred. Not what was written, but what must have been maybe the more critical points that the reviewers thought necessary to discuss.

This may appear sometimes contradictory or different from some other things people receive because often applicants will call and say, "But Reviewer No. 2 said he loved this. It was the best thing since sliced bread." So it may not have been what was discussed, and it was somebody else's different point of view.

Also, how does one address it? Does it mean that the PI needs to go back, and if a suggestion was made that the PI needs to change everything to please a study section? And I would state, you are the person doing the research. You need to think through the issue and determine, "Well, there's a lack of clarity, and I need to clarify that. They didn't understand that," One is not to believe that just because a reviewer states it is gospel either. One needs to consult with others.

One doesn't have to agree with the reviewers, but one cannot ignore what has been raised either, because ignoring this can be seen almost as an offense, I would imagine. So one needs to address a point, not necessarily agree with it, and clarify things accordingly.

Audience: Where is the level of acceptance, respect, however you want to say it, for qualitative versus quantitative, and/or mixed methods?

Jose Guerrier: I think what you're addressing is, I cannot say where is that level of respect. I think that this is where SROs and others are called upon to select individuals on their panel who have expertise in qualitative methods, others in quantitative methods. Because as you know, depending on the discipline you espouse, your tendency may be more to conduct one type of data analysis or data collection than another.

So we try to make sure that our panel represents that which we most likely receive. This is also the reason I think it's most important that the PI determine who is that on that study section. Go into the roster. See who these people are. If they all appear to be quantitative types, yeah, maybe we'll get some ad hocs, but is it the critical mass you might be pleased with?

So you need to also know who is on that panel and whether or not they may be the types who are best qualified to review your application.

Yonette Thomas: So as a follow up question, so if I'm submitting an application, I go online and I see that my research is a layered approach, I'm connecting qualitative and quantitative approaches. And I go to a particular section, and I read and see, "Well, there they have the information that I need. This is a study section that I would like to target," how do I do that? Do I make a note in my cover letter saying I'd like to target that study section? And what happens with that request?

Michael Micklin: Yeah. Everybody has the right to request a particular study section. And, in fact, we encourage it. If you've looked through the set of study sections that might be possible candidates for your application and you think one is better than another, you can certainly suggest that in your cover letter. Or you can say it might be A or B study sections. So, yes, that's perfect acceptable, and we encourage it.

Audience: And you'd do that in the cover letter?

Michael Micklin: Yes.

Audience: What are the options? If you are assigned for the study section that you requested and then they may say that if your study section was from the special study section where you don't have any idea of who's going to be the members of the study section. Then the application was changed leading up to the new ones. Do you have an option to reject that change?

Michael Micklin: I think given the speed with which new study sections are created, which is glacial, I think that's an unlikely event.

Special emphasis panels are created under only a certain set of conditions. One, conflicts, member conflict. If you're collaborating with somebody who is on the study section you requested, it has to go to a special emphasis panel. It's a member conflict.

The other is program announcements and RFAs are typically special emphasis panels.

Jose Guerrier: We had said initially that it is important that you know what the purview of a study section is and that it might be best that you indicate that in a cover letter. Having said that, what you are providing is a suggestion that can be of help both to receipt and referral as well as to the various IRGs. It is not, however, a dictum, a dictate or an edict that that's where it will go, because there are other decisions that need to be made as to where it should go.

As Dr. Micklin mentioned, it may be that an application ends up in a special emphasis panel because what one wants to obtain is a critical mass of other applications responding to the same program announcement so that you can then recruit a critical mass of reviewers to review these applications. Because if you were to leave it alone in a standing section, then you might recruit one, two reviewers, but the majority of them would not really be within that mold.

So other decisions are made, but I would like you to believe that at least that when these decisions are made, it is to favor the applicant as well in a sense, to make sure that the application gets the most competent and fair review.