JWST Cycle 1 Anonymous Proposal Review

STScI has implemented a dual-anonymous proposal review process, where the identities of the proposing team are concealed from reviewers. The primary goal in so doing is to reduce bias for all, not provide a fix for a specific group that currently appears to be under-performing.

STScI has a responsibility to simultaneously ensure that the community has equal opportunity for the use of JWST and that the best science is being done with the finite amount of observing time available. The Institute places a high value on the equity and integrity of the proposal review process. The goal is to enable each reviewer to focus on the science, not the scientist. Studies have shown that a reviewer's attitude toward a submission may be affected, even unconsciously, by the identity of the lead author or principal investigator (see the Anonymous-Double Blind Review Annotated Bibliography [.docx]).  HST proposals led by women have had systematically and successively lower success rates than those led by men. While the exact cause is unknown, independent studies (see On Gender Bias in Hubble Proposal Ratings [.pptx]) of our reviews suggest a double-anonymous process might help resolve this inequity, and may balance out other areas of potential bias including affiliation and country of origin. Such a process may also level the playing field between new and established researchers. The focus of the TAC review is to recommend the best science. The identity of the proposing team should not be a consideration in making this judgement.

In the spring of 2018, STScI convened a working group from the astronomy community to explore the idea of a dual-anonymous system and issue a set of recommendations to the STScI Director. The working group's report, along with detailed instructions to proposers and reviewers, and a list of FAQs, can be found on the Working Groups's Outerspace site. A summary of those guidelines, along with a description of how the review process will work, is given below.

The Review Process

Proposers submit their proposals through APT. The PDF attachment that includes the scientific and technical justifications must be anonymized following the guidelines below. Additionally, proposers must submit, via the Astronomer's Proposal Tool, a separate section titled "Team Expertise and Background." The review panels (and the TAC) will conduct their review without seeing any of the names associated with the proposal, and without seeing the information in the "Team Expertise and Background" section. The panels will discuss the proposals and generate a final ranked list of proposals that are recommended for selection. In addition to the Panel Chair, each review panel (including the TAC) will have a full-time "Leveler" present in the room during all panel discussions. The job of the Leveler is to ensure that discussions remain focused on the scientific merit of the proposal.

Once the ranked list is set, the panels will be given access to the "Team Expertise and Background" information associated with each proposal recommended for implementation. At this point, proposals may only be flagged for downgrade, where a downgrade would result in a non-selection of the proposal. Proposals may not be upgraded once the identity of the team is known. This flag, assigned by majority vote of the panel, should only be used in the most extreme circumstances of a team being clearly unqualified to undertake the work proposed. Should a proposal be suggested for downgrade, both the Panel Chair and the Leveler will participate in the discussion about why this recommendation is necessary. A detailed description of the reason for the flag must be given. This flag will then be passed on to the STScI Director, along with the proposal's initial ranking, and a statement by the panel on the rationale for flagging the proposal. The Director will make the final decision, in consultation with appropriate personnel from STScI, including the Science Mission Office (SMO), JWST Mission Office, ESA, CSA, and the operations and scheduling staff. Finally, any proposals that are downgraded will have the reasons for downgrade passed on to the proposers. The same process will be applied to Large proposals by the TAC.

Guidelines for Proposers

This section provides guidelines to assist proposers prepare their proposals, specifically their PDF Submissions, to avoid highlighting their identity and to support a fairer proposal evaluation process. The anonymous review does not mean proposals will be accepted from anonymous sources. As with previous cycles, proposers must still enter the names and affiliations of all investigators into the APT system. APT will not include names or affiliations in the versions generated for the reviews.

While APT will largely obscure the proposing teams identities in cover materials, it will not change or alter information contained in the PDF submission. Thus, it is necessary for proposers to take additional steps to further anonymize their PDF attachment before it is uploaded to APT. Below are some guidelines to accomplish this:

  • Do not include author names or affiliations anywhere in the PDF attachment. This includes but is not limited to, page headers, footers, diagrams, figures, or watermarks. This does not include references to past work, which should be included whenever relevant (see below).
  • Referencing is an essential part of demonstrating knowledge of the field and progress. When citing references within the proposal, use third person neutral wording. This especially applies to self-referencing. For example, replace phrases like “as we have shown in our previous work (Doe et al. 2010)” with “as Doe et al. (2010) showed...” Do not refer to previous campaigns using HST or other observatories in an identifying fashion. For instance, rather than write "we observed another cluster, similar to the one we are proposing under HST program #XXXXX," instead write "HST program #XXXXX has observed this target in the past..."
  • We encourage references to published work, including work citable by a DOI. It may be occasionally important to cite exclusive access datasets or non-public software that may reveal (or strongly imply) the investigators on the proposal. We suggest proposers use language like "obtained in private communication" or "from private consultation" when referring to such potentially revealing work.
  • Do not include acknowledgements, or the source of any grant funding. 

It takes some effort by authors to anonymize their PDF submissions. As the guidelines show, grammar and structure are expected to be different than in many other submissions of this type, including previous HST submissions. Some examples of re-worked text can be found on Example text for anonymous proposing. Proposers should take sufficient time to prepare the manuscript, especially if one plans to rework a proposal from other submissions

Proposers should make an effort to describe the past work in the field, and how this proposal will improve, build-upon, or complete that past work. Many successful proposals include a discussion of stated-sample goals or statistical completeness and how this proposed work fits into this broader context. Similarly, proposals may also discuss the uniqueness of the sample, and goals in comparison to similar work.

Guidelines for Reviewers

The primary objective of the time allocation review is to select the best science, not the best science teams. The TAC panels and chairs rank proposals in order of scientific merit, and recommend the resources that should be allocated to each. The experience of the team, with JWST or otherwise, is not a consideration. Do not spend time attempting to identify the team or the principal investigator. All accepted proposals are assigned a Program Coordinator who works with the PI to finalize the program for feasible observations. MAST provides "science ready" data for most uses, and there is help/documentation for further data processing. A reviewer's preliminary grading should be centered on the scientific merit of the proposal. This includes technical issues in the design of the study, as described in the technical justification and elsewhere. The discussion should focus on the scientific merit of the proposal. Chairs and Levelers should be quick to refocus or terminate discussion when it moves to PI or team. 

Team Expertise and Background Section

As part of the proposal submission, proposers should complete the "Team Expertise and Background" section in APT. This section should provide a brief description of the expertise, background, and roles of key team members, as they relate to the science proposed. This section should be limited in length; for most proposals, a paragraph or two will suffice. For proposals with a large number of Co-Investigators, it is not necessary to report on the qualifications of every team member; nor is it necessary to provide a bio of the team members. An example is provided on Proposer Guidelines in Anonymous Reviews

Please note: the text box will support ascii text. Special text markup and LaTex are not supported.

Compliance

Proposals must be anonymized in accordance with the guidelines above. Compliance with this policy is mandatory. Proposals received with flagrant violations will be subject to disqualification before the review-panel stage. Proposals with less serious violations (e.g., forgetting to change a reference from first person to third person) will be allowed to remain in contention, but will be flagged for review by SMO and the STScI Director for a final decision. Feedback will be provided to the proposers regarding any violations.

A possible concern that may arise is the following: "I've made every effort to anonymize my proposal, have followed all the guidelines, changed all my references to third-person, but I fear that my work is so specialized (or my analysis methods so unique) that panelists who know me will still be able to figure out who I am. Will my proposal be disqualified?" So long as the guidelines above are followed, the answer is NO, such a proposal will not be considered to be in violation. It is not necessary to "water down" or obscure your science, your methods, or your tools; it is simply your responsibility to write about them in the third-person, in a way that does not intentionally identify yourself. 


Next: JWST Cycle 1 Proposal Policies and Funding Support