Guidelines for Panel Support Scientists - Cycle 1
The James Webb Space Telescope science peer review process is new. Please review the documentation carefully. The Panel members have their own guidelines that will also be useful for your review. To minimize duplication, these guidelines describe only those issues unique to the panel support duties.
All Cycle 1 proposals will be reviewed by discussion during (virtual) review panels. Panel Support staff will be responsible for supporting the virtual discussion review.
The Cycle 1 review will comprise twenty topical panels:
- Solar System Astronomy (1 panel),
- Exoplanets and Exoplanet Formation (3 panels),
- Stellar Physics and Stellar Types (5 panels),
- Stellar Populations and the ISM (2 panels),
- Galaxies and the Intergalactic Medium (4 panels),
- Supermassive Black Holes and Active Galaxies (3 panels),
- and Large-Scale Structure of the Universe (2 panels).
Panelists are chosen based on their expertise in one or more of the areas under review by the panels. The discussion panels will assess and grade all of the proposals submitted in response to the Call for Proposals. Each panel will be managed by a panel chair, and there will be two TAC chairs, one representing Galactic science and the other representing Extragalactic science, overseeing the review process.
Confidentiality and conflicts of interest
The proprietary nature of the proposals and related documents is taken extremely seriously by STScI. Proposals are the intellectual property of the PIs, and the sensitivity of the process is enhanced by the heavy over-subscription of JWST time and the significant funding implications for US proposers. Review-related materials should not be left in the open when not in use, and should be locked away overnight or when otherwise inactive. Please purge any review files from your computer after the review.
Panel membership remains confidential except within a panel. The full TAC membership is published after the results have been announced, but without panel attributions.
It is critically important that conflicts of interest are avoided during the selection process. The conflict of interest rules are listed here. In general, Panel Support Scientists are not assigned to panels where there is the possibility of their having a conflict of interest. Please inform a member of the Science Policies Group if such an issue arises. (See Welcome Letter for SPG member contact information.)
Role of the panel support staff (PSS)
The PSSs are intermediaries between the external reviewers and STScI. They represent STScI and provide continuity through their insight into the operations of JWST, STScI and the review process. Their main tasks are to answer questions, to ensure that the panel meetings proceed smoothly, to record the results of the panel discussions, and to bring any critical issues promptly to the attention of Science Policies Group (SPG) and/or relevant STScI technical staff. The PSSs do not take part in the scientific discussions of the panel and do not grade.
- Reviewer assignments: Proposals are assigned to reviewers by SPG based on the information submitted by each panelist. If a panelist identifies a conflict after they receive their assignments, they should contact the PSS and the PSS should re-assign that proposal to another unconflicted reviewer. The PSS may consult with SPG if necessary. The PSS should inform SPG of the re-assignment to ensure that the appropriate information is updated in the database. The Panel Chair should also be informed about reassignments; the PSS should ascertain from the Chair whether they prefer to be updated with each reassignment or to receive the final assignment list after all swaps are made (just before the review).
- Proposal content: The PSS should read (at a minimum) each proposal's abstract and observation summary in order to acquire an overall familiarity with the proposal content of their panel, to assist the Chairs in organizing the material and running the meetings.
- Panelist questions prior to the meeting: Panelists should be directed to the JWST Help Desk (https://jwsthelp.stsci.edu) for all questions about science policy or technical performance. In the correspondence, panelist should identify themselves as a TAC member. If the question is about a specific proposal, the proposal ID should be stated in the question.
During the meetings, the PSS functions as an executive assistant to the Chair. They organize the materials, provide input on any technical issues noted before the meeting, coordinate real-time consultation with STScI specialists, monitor the grading and final ranking process as well as the reviewer comments status. More specifically, the principal activities during the meetings are as follows.
- Assist the Chair in the organization of the materials and discussion, and provide any other assistance requested by the Chair.
- Coordinate requests for consultations with STScI experts. A list of contacts and phone numbers for on-call experts will be available in each panel. A STScI specialist should be consulted on every significant feasibility or policy issue, and no proposal may be rejected on feasibility or policy grounds without such consultation. There have been problems on other missions with peer reviewers making their own judgments that later turned out to be wrong. Any consultation must be reported in SPIRIT.
- Panelists should record mandatory comments from the panel: duplications specifically allowed by the panel should be noted in the 'notification comments' of the database as being approved; verified technical issues should be entered into the 'Technical Comments' section of the Web-Reviewer tool.
- The PSS should keep careful notes on any controversies, conflicts, or other exceptional issues, and the resolutions of them recommended by the panels, for subsequent consideration during the Director's Review. These notes should be of a quality such that they would be intelligible to someone else (e.g., SPG staff responding months later to appeals from rejected proposers). An example of such information is keeping track of who leaves the room for each proposal, due to a conflict of interest. Another example would be noting where concerns were raised about the technical feasibility of a proposal, and how those concerns were addressed.
- Although it may be difficult, the PSS should completely refrain from making any scientific, technical, scheduling or policy comments, even if asked to do so by the Chair.
- Monitor comments (see below) progress via the Web-based system. Ensure that all comments are finished and signed off by the primary reviewers before they leave the meeting. Ensure that the Panel Chair is aware of the comments status and proof-read the submitted comments as time permits.
- Each panel member gives a numerical grade for each of the 3 grading criteria to the proposal in the web-based review system SPIRIT. (We strongly encourage panelists to review proposals as submitted, however, in exceptional cases, the panel may wish to change the time allocation, this should be voted on before grading.) The PSS must verify the hour allocations are the same as entered in the Web Tool. The importance of care and accuracy in this activity cannot be overemphasized, since it is the only record of the panel's intent.
- After the grading stage follows a ranking stage. The final product of each panel will be a ranked list for all GO, Survey, AR and Theory proposals. Ranks should be modified in the Web-Reviewer tool, as agreed upon by the panel. Grades can not be changed during the ranking stage.
Before the end of the Panel meetings, the PSS will be asked to proof-read and edit the proposal comments to make them suitable for dissemination to the proposers. Panel Chairs are also required to proof-read the final comments. Use the following guidelines:
- All rejected proposals (those below the Cutoff line), should have a clear statement on why the proposal was "rejected".
- Correct grammatical and typographical errors.
- Eliminate all offensive, gratuitous or personal remarks.
- Remove any explicit identifications of other Cycle 1 proposals or proposers, reviewers, or STScI staff.
- Comments that contain only insubstantial or superficial remarks should be removed, whether for approved or unapproved proposals.
- Remove remarks that do not make sense because the reviewers expected the proposal to be approved, while in the end it was not. However, it is okay if a rejected proposal has positive remarks that are useful; that possibility is covered in the notification letter as a consequence of the heavy over-subscription.
- Make sure you verify in the proposals, any references to Proposals that are over the page limits and correct comments as appropriate.
- Make sure you identify any proposals to SPG that violate the Anonymous Review Guidelines.
- Consult with SPG on any questionable cases.
- Inform SPG when a complete set of the final, edited comments are available.
Specific PSS duties
- Attend Panel Support Meetings as required.
- Familiarize yourself with the proposals before your panel.
- Identify any conflicts that you might have.
- Handle any review conflict reassignments.
- Experiment with the Web-Based Reviewer System.
- Answer and resolve any questions brought up by the reviewers as needed.
- Assist Panel Chair as required prior to the actual meetings.
- Monitor the voting process, document any issues about grading, conflicts of interest, etc., into the Web-Reviewer tool during the panel meetings.
- Summon assistance, as needed, from SPG (policy questions), Instrument Division (INS) (instrument questions) or OPB (implementation questions).
- Monitor adherence to all conflict-of-interest rules.
- Inform Brett Blacker that your panel is finished at the conclusion of your panel meeting.
- Review final Comments.
As a valuable member of the Cycle 1 Peer Review process, you are a critical part of the system and we value your comments and suggestions, and welcome feedback throughout the process.
Next: Guidelines for Levelers