The AWG was asked by the Code SR Discipline Scientists to comment on a number of potential changes that might improve the peer-review situation or at least ease the burden on the Discipline Scientists. Most of these suggestions arose in informal brain-storming sessions, and some may be difficult to implement, or may be entirely ruled out by NASA policies (or law). The responses below represent a more-or-less consensus response to various specific ideas on which the AWG was asked to comment.
Whenever possible, we almost unanimously feel that it would be best to not have Co-Is on review panels. However, several members feel that if the stakes are low and if the expertise of certain Co-Is is needed, Co-Is should be allowed to serve on review panels. They should leave the room when their own proposal or competing proposals are being discussed, and they cannot vote on either.
The creation of standing review panels was considered to be a poor idea. One objection was that the approval of certain types of science programs might be effectively precluded by having many of the same reviewers serve multi-year terms. Also, the committee felt that the performance of standing panels that they have dealt with was poorer than panels that were constituted anew each year. Finally, many committee members said that they would be reluctant to agree to serve on a multi-year review panel due to the time commitment and possible conflicts in future years.
Video conferencing offers advantages in some situations, especially for standing committees where a group dynamic has already been established. Some of us find that the most stimulating part of serving on a peer-review committee is a chance to meet and work with the other peer reviewers, and the impact of this is much diminished by video conferencing.
While there are obvious difficulties in actual video conferencing (e.g., it is unlikely that all participants will have available compatible hardware), there exists an alternative form of teleconferencing with some visual capabilities, as utilized by the SIRTF teams. This is called Showstation, and uses free software provided by JPL. Once this package is downloaded, view-graphs, images, and anything in digital form can be displayed by both the sender and the viewers. It also can be marked up or pointed at by the speaker or the viewers. Thus, while images of the speakers are not available, the materials presented are - and the telecon is held over a regular speaker phone. This technique cannot replace all meetings with their face-to-face advantages, but they are a cheaper and less disruptive alternative to frequent travel.
Mail-in reviews are a positive and integral part of the proposal review process. For those NASA grant programs which use solicit mail-in reviews (such as Origins of Planetary Systems and Planetary Atmospheres), these mail-in reviews serve as an important resource for the panel scientist who is the primary reviewer of a given proposal. The primary reviewer quotes from the mail-in reviews during the oral discussion of the panel review process, presenting these alternative views in addition to his or her own. A portion of the mail-in review is available as written comments to the proposal's principal investigator if requested. Generally, the use of mail-in reviews is felt to assure a fair panel review, yields a greater participation by the community, and provides additional written feedback to the principal investigator on the proposal.
The astrophysicists who are solicited to do the mail-in reviews feel these reviews are important. The reviewers dedicate several hours on average to carefully reading and checking proposal facts and background information. Also, mail-in reviewers provide expertise that the review panel might otherwise not have.
The AWG recommends strongly against this idea. Mail-in only reviews lack the group dynamics that form such an important part of the panel review process, and which we feel are extremely important. While we recognize the significant reduction in cost and time (both to the Discipline Scientists and to the community of reviewers) that could be realized by relying solely on mail-in reviews, we feel that the benefits of having the panel meet together to discuss each proposal far outweigh the cost and inconveniences that result.
The AWG does not believe this is a good way to improve the review and selection process. The establishment of such a limit would be somewhat arbitrary and could actively discriminate against some sub-disciplines where multiple small grants are used to support a research group. It also imposes artificial and arbitrary limits on creativity and productivity. The additional administration required to implement and monitor this requirement might easily outweigh any savings produced by the expected reduction in proposals.
Again, we believe that this is not a good idea, principally because it artificially discriminates against researchers with broad interests or who are especially industrious or productive. It is fundamentally anti-intellectual and we oppose it.
We oppose this as it places an arbitrary constraint on investigators. NASA should not concentrate on the number of investigators on any given project, but ought to concern itself with the roles played by each investigator to make sure that the support to be received by each co-investigator is commensurate with his or her role in the project.
While there are instances where action like this would be appropriate, we feel that this places an inappropriate burden and level of responsibility on the Discipline Scientists. From the user's point of view, it would be perceived as unfair if a proposal could be rejected on the basis of review by a single scientist who may or may not have the appropriate expertise and background. In any case, we believe that this action is likely to be taken only on the poorest proposals, which are a minority. We do not believe that this would expedite the review process, but it could certainly compromise its integrity.
In some cases, this effectively occurs already; many peer-review panels spend little time on proposals far below the cutoff based on preliminary rankings. We would discourage any further reduction in feedback to proposers of the least competitive proposals - indeed, these are not the difficult proposals to deal with. The hard reviews to write are of proposals which are not quite in the top tier, but are nevertheless deserving of support, and which cannot be supported on account of limited program resources.
Different programs have different needs. In some cases, funding for a 3-5 year period has already been implemented (e.g. LTSA, and hardware R&D). It is not recommended that all grant funding become 3-5 years; it may be appropriate for some programs, but not for others.(especially for GO programs, for example).
In some cases, this might be appropriate, even though proposers obviously prefer the opportunity to propose more often, in some cases for very good reasons that may not be obvious to NASA. For example, large research programs can sometimes survive a one-year lapse in funding by careful managment of resources. However, there are few if any programs that could survive a three-year lapse.
We understand that additional peer reviews place a substantial burden on the Office of Space Sciences, but this is an option that should be exercised only reluctantly and only in special cases.