Selection Criteria and Scoring System - Cycle 3

Dated material

You are viewing content from a previous JWST Call for Proposals (CfP). For information pertaining to the current observing cycle and future CfPs, please see JWST Opportunities and Policies.

An explanation of the selection and scoring system for JWST proposals is provided in this page. 

On this page

Selection criteria

Proposals reviewed by external panelists are subject to a single-phase review; proposals reviewed by the virtual topical panels are subject to a two-stage review process: 1) preliminary grading and triage; and 2) the review meeting. In all cases, panelists use the same scoring system.

Each topical panel covers a very broad science category, and each science category contains a number of narrower sub-fields. Ideally a proposal will be impactful to both the narrow sub-field of the proposal and to other sub-fields within the science category or in other science categories. Proposals will be assessed on an absolute scale against three primary criteria described in the Call for Proposals with a separate grade given for each.

  • Impact within the sub-field:
    • The scientific merit of the program and its contribution to advancement of knowledge.
    • Will the proposed program improve our understanding of the objects, classes of object, or specialist topics under study in the proposal? By how much? How relevant is the proposed work to the immediate sub-field of the proposal?
    • The immediate sub-field of the proposal is the niche area of the program, not the whole broad science area of the topical panel to which it was assigned. The evaluation should be based on what is written in the proposal, not on the reviewer's broader knowledge, even if the reviewer is an expert in the sub-field. Though, in most cases, the reviewer will not be an expert in the sub-field of the proposal, and the proposal should have been written accordingly.
  • Out of field impact:
    • The program’s impact for astronomy in general.
    • Are there implications for other science areas and/or insights into larger-scale questions? Will the proposed program improve our understanding of science areas beyond the immediate sub-field of the proposal? How broad and how significant is this new understanding?
    • The proposal does not have to impact all of astronomy, but should ideally impact a number of other sub-fields or provide significant impacts in at least one other sub-field. The out-of-field impacts could be in other areas within the topical science panel of the proposal, or in other topical science areas. This evaluation should be based on what is written in the proposal, not on the reviewer's broader knowledge.
  • Suitability:
    • The necessity for JWST observations or relevance to JWST science.
    • For GO and AR programs: a demonstration that the unique capabilities of JWST are required to achieve the science goals; how much of an advantage does JWST data offer over other facilities?
    • For Theory programs: a demonstration of broad applicability to JWST observational programs.

The final grade is the straight average of these values.

AR and GO calibration proposals are required to provide an analysis plan; reviewers should also consider the strength of the analysis plan in assessing the first two criteria.

Descriptions of additional criteria by type of proposal are given in the Proposal Selection Procedures section of the Call for Proposals.

While reviewing the proposals if you notice and/or identify any issues with the proposal template formatting, page limit violations or resource request issues, please contact SPG to discuss before downgrading that proposal.


Scoring system

The full set of criteria to apply in assessing different types of proposals are described in the Proposal Selection Procedures section of the Call for Proposals. Those criteria should be taken into account when grading each proposal.

Preliminary grades

The preliminary scoring should be on an absolute scale with the framework set by the below criteria. Reviewers may submit grades in decimal form, but please limit to one decimal place. The preliminary grade for each proposal is determined by averaging the overall grade from each reviewer. The preliminary grades are used to create a rank order list for each panel and the lowest-ranked proposals (typically ~40%) are triaged from further discussion.

The grades for each virtual panelist are normalized by SPIRIT to a mean value of 3.0 and a standard deviation of 1.0. The reviewer's grade for that proposal is the average of the three normalized grades.

External grading

The grades from external panelists should be on an absolute scale with the framework set by the following criteria. Reviewers may submit grades in decimal form, but please limit to one decimal place. The grade for each proposal is determined by averaging the overall grade from each reviewer and these are not normalized, so you should use the full range. Once the grading is complete for all proposals, the rank order list is created.

Discussion panel meetings

In the discussion panel discussion, the panelists should use the same scale for their grading. Reviewers may submit grades in decimal form, but please limit to one decimal place. As with the preliminary grades, the final grades are normalized. The overall grade for each reviewer is the straight average of the three individual grades. The final grade for each proposal is determined by averaging the overall grade from each reviewer. Once the grading is complete for all proposals, the rank order list is created.

Grade

Impact within the sub-field

Out-of-field impact

Suitability

1

The proposed program has the potential for transformative results in the immediate sub-field of the proposal. The program will transform understanding of the objects, class of objects, or specialist topics under study.

The proposed program has transformative implications for one or more other sub-fields of astronomy. The impacts of the program are extremely broad and/or extremely significant.

GO: The program's science goals can only be achieved with the proposed observations. Only JWST is capable of collecting the required observations. Archival data may supplement the analysis, but is insufficient or does not exist in key areas, so new observations are needed.

AR: Analysis of archival JWST data is critical to reach the stated science goals. The analysis may be supplemented by data from other observatories but the majority of the data will come from JWST. 

Theory: The proposed theoretical or computational work will be transformative for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data.

2

The proposed program has the potential for major advancement in the immediate sub-field of the proposal. The program will provide major advances in understanding of the objects, class of objects, or specialist topics under study.

The proposed program has major implications for one or more other sub-fields of astronomy. The impacts of the program are broad and/or significant.

GO: The program's science goals will certainly be achieved with the proposed observations. JWST offers major advantages over other facilities for collecting the required observations. The need for new data instead of or to supplement archival data is clear.

AR: Analysis of archival JWST data offers major advantages over data from other facilities to reach the stated science goals. The analysis may be supplemented by data from other observatories but a significant fraction of the data will come from JWST. 

Theory: The proposed theoretical or computational work will offer major advances for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data.

3

The proposed program has the potential for moderate advancement in the immediate sub-field of the proposal. The program will provide moderate advances in understanding of the objects, class of objects, or specialist topics under study.

The proposed program has moderate implications for one or more other sub-fields of astronomy. The impacts of the program are moderate in scope and significance.

GO: The program's science goals will be achieved with the proposed observations. JWST offers some advantages over other facilities for collecting the required observations. The need for new data over archival data is discussed but not compelling.

AR: Analysis of archival JWST data offers some advantages over data from other facilities to reach the stated science goals. The analysis may be supplemented by data from other observatories but some of the data will come from JWST. 

Theory: The proposed theoretical or computational work will offer some advances for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data.

4

The proposed program has the potential for minor advancement in the immediate sub-field of the proposal. The program will provide minor advances in understanding of the objects, class of objects, or specialist topics under study.

The proposed program has minor implications for other sub-fields of astronomy. The impacts of the program are limited in scope and significance.

GO: The program's science goals will probably be achieved with the proposed observations. JWST offers minor advantages over other facilities for collecting the required observations. The need for new data over archival data is not discussed in depth or is not clear.

AR: Analysis of archival JWST data offers minor advantages over data from other facilities to reach the stated science goals. The analysis will mostly use data from other observatories but small fraction of the data will come from JWST.

Theory: The proposed theoretical or computational work will offer minor advances for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data.

5

The proposed program has limited or no potential for advancement in the immediate sub-field of the proposal. The program will not advance understanding of the objects, class of objects, or specialist topics under study.

The proposed program has little or no implications for other sub-fields of astronomy. The impacts of the program are extremely limited in scope and significance.

GO: The program's science goals will not be achieved with the proposed observations or JWST offers little to no advantage over other facilities for collecting the required observations. The need for new data over archival data is not discussed.

AR: Analysis of archival JWST data offers little to no advantage over data from other facilities to reach the stated science goals. The analysis will predominantly use data from other observatories with only very little or no data coming from JWST. 

Theory: The proposed theoretical or computational work will offer little or no advance in planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data.

Examples

The following examples aim to give guidance in applying these rubrics to grading proposals; reviewers should use their best judgement.

Case 1. IR spectroscopic observations of gas in young stars

In field

Highly significant improvement in our understanding of gas flow in young stars.

1-2

Out of field

Potential for significant changes in our understanding of shock excitation in a wide range of other environments.

1-2

Suitability

Warm H2 observations are essential to achieve the science goals and can only be acquired through JWST MIRI.

1


Case 2. Analysis of archival near-IR imaging of a nearby galaxy for star formation investigations

In field

Major advance in understanding of ionized and neutral PAHs in that galaxy.

2

Out of field

Some mention of the applicability to star formation studies in general.

3

Suitability

The increased spatial resolution offered by JWST provides some advantages over other facilities in addressing the science goals. The analysis offers significant improvements and/or additional value with respect to the original use of the data.

2-3


Case 3. Optical/near-IR spectroscopy of a quasar

In field

Moderate increase in understanding of the relative contribution of the AGN vs star formation to the ionized gas emission.

3

Out of field

Minor implications for the properties of other galactic systems, but no wider impact.

4

Suitability

Limited gains in performance at near-IR (rest frame optical) wavelengths as compared with larger ground-based facilities, does not justify the need for the rest frame IR lines

4-5


Case 4. Developing theoretical tools to characterize gas and dust in Galactic star-forming regions

In field

Potential significant increase in understanding of chemical composition in dusty environments.

1-2

Out of field

Results have significant implications for interpreting dust composition in other galaxies.

2

Suitability

The theoretical analysis will enable and support additional JWST observational programs.

2


Next: Proposal Feedback Comments - Cycle 3


Notable updates


Originally published