Guidelines for Panel Chairs - Cycle 3

Dated material

You are viewing content from a previous JWST Call for Proposals (CfP). For information pertaining to the current observing cycle and future CfPs, please see JWST Opportunities and Policies.

A brief overview of the Virtual Review process for Panel Chairs is provided in this article. Overall, the virtual review meetings will proceed in the same way as on-site panel meetings; however, from experience, there are a few things to be aware of and we make some suggestions below.

On this page

Brief agenda

A very brief agenda for the review meeting is as follows:

  • Introductions of Chair, Panelists and Panel Support Scientist. Overview of how the meeting will run.
  • Panelists discuss and grade individual proposals in turn; grades are locked after each discussion.
  • STScI makes a ranked list from the grades. Panelists discuss the ranked list, and re-rank, if appropriate.
  • Panelists review team expertise statements for highly-ranked proposals.
  • Panelists discuss (though do not grade) a subset of Large proposals in the panel subject area and provide feedback to the Chair.
  • Panelists should enter comments for each proposal for which they are Primary reviewer by the end of the meeting; this may be done at the end or time set aside during the meeting. The Chair signs off on these once they are complete.



Guidelines and responsibilities

Understanding, patience & kindness

We have panelists spanning many time zones from Europe to Hawaii. The meeting times have been truncated to account for this, but we do understand it may be very late or very early for some.

It is also likely that many panelists will be working from home, with partners, children, housemates and pets around, possibly in small spaces. Be understanding about background noise and distractions. That said, we encourage panelists to mute when not speaking to minimise background noise.

Software and hardware

The meetings will be run using BlueJeans. We encourage all panelists to show their videos to allow for face-to-face communication, but recognise this may not be possible in cases where panelists have limited bandwidth. To that end, we encourage panelists to use a wired or stable wireless internet connection, but again recognise that may not be possible in all cases.

Set a schedule, include breaks

Chairs should set a schedule for the meeting and communicate this to the panelists, and PSS ahead of time. We recommend that Chairs schedule blocks of time for different activities (proposal discussions, comment writing). Don’t forget to include breaks, and stick to them!

When the panel takes a break, please arrange for a note to be projected on the screen with the time that the panel is expected to reconvene. Such a note is helpful for wandering observers and At-Large members to understand the panel's schedule.

Additionally, Chairs should decide and communicate the order in which proposals will be discussed ahead of time so panelists can make sure they are prepared.

Ensure everyone has an equal opportunity to contribute

Chairs should set a protocol for the order in which panelists will speak during each discussion and communicate this protocol at the start of the meeting.

Every proposal is assigned a Primary and Secondary reviewer. The Primary starts the discussion by briefly summarising and reviewing the proposal, then the Secondary gives supplementary comments before the discussion is opened up to all panelists. To make sure that everyone gets a chance to speak, and to mitigate the confusion of panelists talking over one another, we recommend that the chairs call on each panelist in turn to explicitly invite their contributions. To ensure that panelists play an equal role in the discussion, and that no one always goes last, we propose that the Chair calls on panelists alphabetically (after the Primary and Secondary reviewers have spoken), starting from the Secondary reviewer.

For example:
Cannon – 7
Herschel – 8
Leavitt (Secondary) – 2
Payne-Gaposchkin – 3
Roman – 4
Rubin – 5
Swope (Primary) – 1
Tinsley – 6

After all panelists have contributed once, the floor will be opened for general discussion. To avoid panelists talking over one another, we encourage panelists to raise their hands to indicate they wish to speak, and the Chair will call on them in turn.

Conflicts of interest

Any panelist who is conflicted on a proposal for any reason must leave the virtual meeting for the duration of the discussion of that proposal (both in the individual proposal discussion phase and the ranking phase if the conflicted proposal is being compared to another). The preferred solution is to add any conflicted panelists to a breakout room in BlueJeans and then recall them to the main meeting once the discussion is over. An alternative solution is for any conflicted panelists to disconnect from the meeting entirely and rejoin once the discussion is over. It is not allowed to turn off audio and video but remain connected to the main meeting.

Ahead of the meeting, panelists should identify the best way to contact them to tell them that the discussion has ended and they can rejoin the meeting. Panelists should also be vigilant during that time and aware that they could be called back at any time. We have set up a Slack channel for each panel and strongly recommend using this, but other methods (email, text, WhatsApp) could be used instead if Slack does not prove to work well. Panelists should make sure that notifications are turned on and/or alert sounds are turned on, as appropriate, for the method they choose.

If the Chair is conflicted on a proposal, the Chair should leave the meeting and the Chair will appoint a panelists to handle the discussion of that proposal. If necessary, an At-Large member can be asked to deputize as Chair; typically SPG will identify an At-Large member who will serve and the PSS will let that person know when the proposal is next up for discussion so they can join the panel.

Persons able to join the meeting

The meetings will be private, that is only people with a link may join. There will be a number of people beyond the panelists and panel support who will be given the links and will be able to join.

Each panel is comprised of a Chair and 8 -13 panelists. Each panel will also be assigned a Panel Support Scientist (PSS). The PSS facilitates the process and ensure everything runs smoothly. If there are problems or queries, the PSS can make sure that a member of SPG is aware and can address the issues or answer the questions. In addition, the PSS is present to ensure that the discussions remain focused on the scientific merits of the proposals, and not the proposing teams. Aside from cases of conflict, these people will be in the review meetings permanently.

Additionally, At-Large members of the EC and Observers will periodically drop in and out of the virtual meetings to oversee the discussions, just as they would come into and out of meeting rooms during an on-site meeting. Similarly, members of the Science Policies Group will also move between panels as needed. Occasionally, JWST instrument experts will join a meeting if a consultation is required by the panel.

Proposal comments

Panelists are required to provide Proposal Feedback Comments - Cycle 3 for all proposals for which they are the Primary reviewer, in consultation with the Secondary reviewer – incorporating comments from other panelists and from the discussions.

If time permits, we encourage Chairs to set aside time during the meeting to allow panelists to write their comments while other panelists are present to ask for additional feedback. With the truncated timeframe of the meetings, we recognize that this may not be possible.

We encourage panelists to use Slack to message the whole panel or individual panelists to assist in the writing of reviews.

Chairs should check the reviews submitted by the panelists and ask them to rewrite anything that is not clear or is not professional.



Resources

  • Example spreadsheet for managing meeting schedule, conflicts, discussion and comments: Schedules.xlsx (note this schedule was made for a 3-day meeting, and we now run 4-day meetings, but the template is still useful)
  • An alternative example spreadsheet with 3 schedule examples, and detailed breakdowns for tracking primaries/secondaries and conflicts: Draft Schedule for STScI TACs (link is to a google sheet)



Duplications

Duplications are observations of the same target or field using the same instrument, same mode, same optical elements and with an exposure time within a factor of four. The Primary goal is to maximize telescope efficiency by eliminating unnecessary observations. In general, duplications are only allowed if there is a clear scientific justification. 

  • In-panel duplications are easy to identify and should be handled during the panel discussions, but conflicts can lead to similar proposals being distributed to different panels.
    • Panel chairs should share their list of highest-ranked proposals with their mirror panel counterparts to check for science balance, identify duplications and recommend mitigations. These discussions will mostly occur during the Panel Chair Breakfast sessions held during the Executive Committee meeting.
    • SPG will compile Abstract (Dual-Anonymous version) and Exposure Catalogs for the recommended smaller proposals. They will share those catalogs with the appropriate Panel Chairs who should check for and resolve any duplications with proposals considered by their discussion panel.
  • If the panel chair(s) is unable to resolve the duplication, please bring that to the attention of your SPG Scientist so that the matter can be discussed and resolved.
  • Panel to Executive Committee Proposal duplications should be brought up during the Ranking phase of the EC meeting and resolved there if possible.



Next: Blue Jeans and Slack Guidelines - Cycle 3



Notable updates


Originally published