A description of the final ranking process for discussion panels in the JWST peer review is provided in this page.

Goals of Ranking

The panel ranking uses the average normalized grades from all panelists to make a rank-ordered list of proposals. Panel members should review the ranked list to determine whether the highly-ranked proposals above the nominal cutoff line provide an appropriate science balance for the panel. There may be a consensus that some science areas have been unduly favored. There may also be cases where the chair identifies highly ranked proposals that have a science overlap with proposals highly ranked by another panel. The panel members can make a consensus decision to re-rank (but not re-grade) proposals to provide an appropriate reflection of the science topics reviewed by the panel. Panels should also take care to consider the relative ranking of two proposals with identical grades – by default, we will assign the higher rank to the proposal with the lower grade deviation.

The goal of ranking is not to play tetris. Panels should not try and manipulate the ranking to fit the proposals into the hour allocations.

Hour Allocations

Each Discussion panel will be given two separate hour allocations — a general allocation of N hours, and a medium allocation of M hours – which will be communicated by SPG. Our aim is that proposal success rates will be as even as possible for all science areas and proposal types. The M and N allocations are, thus, determined by the relative hour pressure across the science areas and proposal types and are different from each other and different for each panel. It is helpful to think of these allocations as guidelines to help frame the ranking discussions, rather than entitlements that are set in stone.

The general allocation N is the number that sets the nominal cutoff ("the 1N line"). We will ask the panels to rank carefully down to twice the general allocation ("the 2N line") in case we need to go deeper down the ranking, which is a common occurence!

Small and Medium Proposals

The general allocation, N, is used to fund Small proposals. The 1N line is set by calculating the cumulative sum of the hours requested by Small proposals. Proposals for which the cumulative sum is less than N are said to be "above the line" and will likely be recommended for approval. The next proposal down is said to be "on the line" or the straddler, this proposal may or may not be recommended for approval. All other proposals are said to be "below the line" and will likely not be recommended for approval.

The medium allocation, M, is used to fund Medium proposals. Only Medium proposals above the 1N line are eligible for this medium allotment. Any hours from the medium allocation that are not allocated may not be used for Small proposals. Panels should not inflate the rank of a Medium proposal in the case where they would not naturally use up this medium allocation.

If a panel has a Medium proposal above the 1N line that can only partially be filled by the medium allocation, remaining hours may be allocated from the general allocation, but this is not required. Similarly, the panel can keep additional Medium proposals in their ranked list to be filled by general allocation hours, but this is not required. Any Mediums removed from the ranked list should have their ranks adjusted by pairwise comparison with other proposals (see below); however, in making those comparisons the panelists are free to use valid considerations beyond the individual merits of each proposal, such as ensuring a scientific balance of the approved program. The only exception to this is if they duplicate science in which case they can be directly removed without any further pairwise comparison.

Some ranking examples are provided below and cover a hypothetical panel with only a general pool, and cases where a panel exhausts its medium allocation, under-allocates its medium allocation, or does not allocate any of its medium allocation.

Re-ranking process

In re-ranking proposals, panels may directly compare proposals, irrespective of their relative ranking, that are judged to have very similar science to the extent that the panel may recommend executing only one proposal. Panelists conflicted with either proposal may not vote on the re-ranking. If the panels choose to do only one proposal, the other proposal is moved to the 2N line and the proposal feedback comments for that proposal are adjusted to reflect the discussion.

In all other cases, panels may only compare and vote on adjacent proposals. This is to minimize conflicts. Thus, if a panelist advocates raising a proposal in position 14, it must be compared and voted against proposal 13—panelists with conflicts on #14 and #13 may not vote. If it is raised to position 13, it can be compared against proposal #12—again, panelists conflicted on either proposal may not vote. And so forth until the ranking is established. Note that the exact ranking is most important close to the 1N line for each panel.

Unallocated Hours

Panels rarely "use up" all of their hours. Sometimes they come pretty close and sometimes they under-allocate by a considerable amount. All the unallocated hours return to an overall pool. After all panels have ranked, the overall pool of unallocated hours will be redistributed back to the panels. In general, this redistribution seeks to ensure that the success rate is as even as possible across all science areas and proposal types. We also seek guidance from the Executive Committee in this process.

If a panel does not allocate all of its hours, they should make scientifically-motivated recommendations to the Panel Chair about what they would prefer if additional hours should come back to the panel at this stage. The Panel Chair should communicate these recommendations at the EC meeting and in their panel report. 

Final recommendation

Panelists are asked to rank proposals all the way down to twice the hour allocation (the 2N line); this is in case any changes need to be made to the top-ranked proposals (for example, an approved proposal in another panel proposes the same observations, thus making the proposal in the panel a duplication) or if it's necessary to add additional programs at a late stage in the cycle. Panelists are also asked to set a do-not-approve line, if they deem it appropriate.

Small and Medium Ranking Examples

A panel with only Small proposals might look like this. The General Allocation Used column is the cumulative sum of the hours requested. The proposal ranked #6 straddles the 1N line.

In the next example, the panel under-allocates its medium hours. The number of hours requested by Medium proposals above the 1N line, is less than the medium allocation. The remaining medium allocation returns to the overall pool.

In the next example, the panel fully allocates its medium hours and uses the general pool to supplement a Medium proposal.

In the next example, the panel does not allocate any of its medium hours because the highest-ranked Medium proposal is below the 1N line. The medium allocation returns to the overall pool.

Next: Large, Treasury and Legacy Proposals - Cycle 4