Guidelines for Panel Chairs - Cycle 1

This document provides a brief overview of the Virtual Review process for Panel Chairs. Overall, the virtual review meetings will proceed in the same way as on-site panel meetings; however there are a few things to be aware of as we switch to virtual meetings, and we make some suggestions below.

Brief agenda

A very brief agenda for the review meeting is as follows:

  • Introductions of Chair, Panelists, Panel Support Scientist and Leveler. Overview of how the meeting will run.
  • Panelists discuss and grade individual proposals in turn; grades are locked after each discussion.
  • STScI makes a ranked list from the grades. Panelists discuss the ranked list, and re-rank, if appropriate.
  • Panelists review team expertise statements for highly-ranked proposals.
  • Panelists discuss (though not grade) Large proposals related to the subject area of their panel and provide feedback to the Chair and Vice Chair.
  • Panelists should enter comments for each proposal for which they are Primary reviewer by the end of the meeting; the Chair should sign off on these.

Guidelines and responsibilities

Be understanding, be patient, be kind

We have panelists spanning many time zones from Europe to Hawaii. The meeting times have been truncated to account for this, but we do understand it may be very late or very early for some.

It is also likely that most panelists will be working from home, with partners, children, housemates and pets around, possibly in small spaces. Be understanding about background noise and distractions. That said, we encourage panelists to mute when not speaking to minimise background noise.

Software and hardware

The meetings will be run using BlueJeans. We encourage all panelists to show their videos to allow for face-to-face communication, but recognise this may not be possible in cases where panelists have limited bandwidth. To that end, we encourage panelists to use a wired or stable wireless internet connection, but again recognise that may not be possible in all cases.

Set a schedule

Chairs should set a schedule for the meeting and communicate this to the panelists, and PSS ahead of time. We recommend that Chairs schedule blocks of time for different activities (proposal discussions, comment writing). Don’t forget to include breaks!

Additionally, Chairs should decide and communicate the order in which proposals will be discussed ahead of time so panelists can make sure they are prepared.

Ensuring everyone has an equal opportunity to contribute

Chairs should set a protocol for the order in which panelists will speak during each discussion and communicate this protocol at the start of the meeting.

Every proposal is assigned a Primary and Secondary reviewer. The Primary starts the discussion by briefly summarising and reviewing the proposal, then the Secondary gives supplementary comments before the discussion is opened up to all panelists. To make sure that everyone gets a chance to speak, and to mitigate the confusion of panelists talking over one another, we recommend that the chairs call on each panelist in turn to explicitly invite their contributions. To ensure that panelists play an equal role in the discussion, and that no one always goes last, we propose that the Chair calls on panelists alphabetically (after the Primary and Secondary reviewers have spoken), starting from the Secondary reviewer.

For example:
Cannon – 7
Herschel – 8
Leavitt (Secondary) – 2
Payne-Gaposchkin – 3
Roman – 4
Rubin – 5
Swope (Primary) – 1
Tinsley – 6

After all panelists have contributed once, the floor will be opened for general discussion. To avoid panelists talking over one another, we encourage panelists to raise their hands to indicate they wish to speak, and the Chair will call on them in turn.

Conflicts of Interest

Any panelist who is conflicted on a proposal for any reason must leave the virtual meeting for the duration of the discussion of that proposal (both in the individual proposal discussion phase and the ranking phase if the conflicted proposal is being compared to another). They cannot turn off audio and video but remain connected to the call, they must leave the meeting. Once the discussion is over, they can rejoin the meeting again, using the same link as before.

Ahead of the meeting, panelists should identify the best way to contact them to tell them that the discussion has ended and they can rejoin the meeting. Panelists should also be vigilant during that time and aware that they could be called back at any time. We have set up a Slack channel for each panel and strongly recommend using this, but other methods (email, text, WhatsApp) could be used instead if Slack does not prove to work well. Panelists should make sure that notifications are turned on and/or alert sounds are turned on, as appropriate, for the method they choose.

If the Chair is conflicted on a proposal, the Chair should leave the meeting and designate either an At-Large TAC member or another panel member as Chair pro-tem.

Persons able to join the meeting

The meetings will be private, that is only people with a link may join, but there will be a number of people who will be given the links and will be able to join.

Each panel is comprised of a Chair and 9 panelists. Each panel will also be assigned a Panel Support Scientist (PSS) and a Leveler. The PSS facilitates the process and ensure everything runs smoothly. If there are problems or queries, the PSS can make sure that a member of the Science Policies Group is aware and can address the issues or answer the questions. The Leveler is present to ensure that the discussions remain focused on the scientific merits of the proposals, and not the proposing teams. Aside from cases of conflict, these people will be in the review meetings permanently.

Additionally, At-Large members of the TAC and Observers will periodically drop in and out of the virtual meetings to oversee the discussions, just as they would come into and out of meeting rooms during an on-site meeting. Similarly, members of the Science Policies Group will also move between panels as needed. Occasionally, JWST instrument experts will join a meeting if a consultation is required by the panel.

Cross-panel duplications

Wherever possible, proposals requesting closely similar observations for the same target(s) are assigned to the same panel. However, conflicts with panelists may result in such proposals directed to one of the other mirror panels. Once the panel has completed grading their proposals, chairs of mirror panels will be given an opportunity to compare the highly ranked proposals. If there are duplicate or closely similar proposals, that chairs should consider whether that should factor into the final ranking from their panel. They may consult with unconflicted members of both panels to decide whether or not to prioritise one proposal over another.

Proposal comments

Panelists are required to provide comments for all proposals for which they are the Primary reviewer, in consultation with the Secondary reviewer – incorporating comments from other panelists and from the discussions.

If time permits, we encourage Chairs to set aside time during the meeting to allow panelists to write their comments while other panelists are present to ask for additional feedback. With the truncated timeframe of the meetings, we recognize that this may not be possible.

We encourage panelists to use Slack to message the whole panel or individual panelists to assist in the writing of reviews.

Chairs should check the reviews submitted by the panelists and ask them to rewrite anything that is not clear or is not professional.

Next: Guidelines for Panelists

Latest updates

Originally published