Selection Criteria and Scoring System
An explanation of the selection and scoring system for JWST proposals is provided in this page.
On this page
Selection criteria
Proposals reviewed by external panelists are subject to a single-phase review; proposals reviewed by the virtual topical panels are subject to a two-stage review process: 1) preliminary grading and triage; and 2) the review meeting. In all cases, panelists use the same scoring system.
Each topical panel covers a very broad science category, and each science category contains a number of narrower sub-fields. Ideally a proposal will be impactful to both the narrow sub-field of the proposal and to other sub-fields within the science category or in other science categories. Proposals will be assessed on an absolute scale against three primary criteria described in the Call for Proposals with a separate grade given for each.
(A) In-field Impact:
- The scientific merit of the program within its immediate sub-field, and its contribution to advancement of knowledge.
- The immediate sub-field of the proposal is the niche area of the program, not the whole broad science area (e.g. Trans-Neptunian Objects, not Solar System Astronomy).
- Will the proposed program improve our understanding of the objects, classes of object, or specialist topics under study in the proposal? By how much? Is the work relevant and timely?
- The evaluation should be based on what is written in the proposal, not on the reviewer's broader knowledge, even if the reviewer is an expert in the sub-field. Though, in most cases, the reviewer will not be an expert in the sub-field of the proposal, and the proposal should have been written accordingly.
(B) Out-of-field Impact:
- The program’s impact outside of its immediate sub-field.
- A proposal does not have to impact all of astronomy. The out-of-field impacts could be in other sub-fields within the broader science area of the proposal, or in other broad science areas (e.g., in the case of a TNO proposal, this could be solar system formation or planet formation in general, among others).
- Proposals should discuss implications for other fields or sub-fields, and their breadth, significance, and timeliness. Are there implications for other science areas and/or insights into larger-scale questions? Will the proposed program improve our understanding of science areas beyond the immediate sub-field of the proposal? How broad and how significant is this new understanding?
- This evaluation should be based on what is written in the proposal, not on the reviewer's broader knowledge.
(C) Suitability & Feasibility:
- The suitability of JWST observations or datasets, or relevance to JWST science. The necessity of special requirements. The feasibility of the science program.
- Proposals should demonstrate that the capabilities of JWST are required to achieve the scientific goals, or demonstrate the relevance of the work to JWST science. Technical issues will be adjudicated by STScI instrument scientists.
- For GO programs: Has the proposed program demonstrated that the unique capabilities of JWST are required to achieve the science goals? How much of an advantage does JWST data offer over other facilities? Has the time request been well justified? Have any special requirements been well justified? Have parallel observations been well justified? Have any duplications been well justified? If joint time was requested, have those additional observations been well justified? Does the observing plan make efficient use of resources? Is there a clear path to science?
- For AR programs: Has the proposed program demonstrated that the unique capabilities of JWST are required to achieve the science goals? How much of an advantage does JWST data offer over other facilities? Is JWST the predominant source of data for the program? If data from other facilities will be used, has it's use been well justified. Is there a clear path to science?
- For Theory programs: Has the proposed program demonstrated a broad applicability to JWST observational programs or data products? Will the proposed program provide results that assist in planning future JWST observations, improve analysis of JWST data or data products, or improve interpretation of JWST data or data products? Is there a clear path to science?
The final grade is the straight average of these values.
AR and GO calibration proposals are required to provide an analysis plan; reviewers should also consider the strength of the analysis plan in assessing the first two criteria.
Descriptions of additional criteria by type of proposal are given in the Proposal Selection Procedures section of the Call for Proposals.
While reviewing the proposals if you notice and/or identify any issues with the proposal template formatting, page limit violations or resource request issues, please contact SPG to discuss before downgrading that proposal.
Scoring system
The full set of criteria to apply in assessing different types of proposals are described in the Proposal Selection Procedures section of the Call for Proposals. Those criteria should be taken into account when grading each proposal.
Grading by external panelists
The grades from external panelists should be on an absolute scale defined below with the framework set by the aforementioned criteria. Reviewers may submit grades in decimal form, but please limit to one decimal place. The grade for each proposal is determined by averaging the grades given to the proposal by each of its assigned reviewers (which, as mentioned above, is calculated averaging the grades given to (A), (B) and (C) above). In this case the grades are not normalized, so you should use the full range. Once the grading is complete for all proposals, the rank order list is created.
Preliminary grades
The preliminary scoring should be on an absolute scale defined below with the framework set by the aforementioned criteria. Reviewers may submit grades in decimal form, but please limit to one decimal place. All the grades given by each panelist are normalized by SPIRIT to a mean value of 3.0 and a standard deviation of 1.0. The reviewer's grade for that proposal is the average of the three normalized grades (for (A), (B) and (C) above). The preliminary grade for each proposal is determined by averaging the grades given to the proposal by each of its assigned reviewers. The preliminary grades are used to create a rank order list for each panel and the lowest-ranked proposals (typically ~40%) are triaged from further discussion.
Discussion panel meetings
In the panel discussion, the panelists should use the same scale for their grading as in the preliminary grading. Reviewers may submit grades in decimal form, but please limit to one decimal place. As with the preliminary grades, the final grades are normalized for each individual reviewer. The overall grade for each reviewer is the straight average of the three individual grades for the criteria below. The final grade for each proposal is determined by averaging the overall normalized grade from each reviewer. Once the grading is complete for all proposals, the rank order list is created.
Grade | In-field impact | Out-of-field impact | Suitability & Feasibility |
|---|---|---|---|
1 | Transformative advancement in the sub-field. The proposed program has the potential for transformative results in the immediate sub-field of the proposal. The program will transform understanding of the objects, class of objects, or specialist topics under study. | Transformative implications beyond the sub-field. The proposed program has transformative implications for one or more other sub-fields of astronomy. The impacts of the program are extremely broad and/or extremely significant. | Extremely well-justified with a very clear path to science. GO: The program's science goals can only be achieved with the proposed observations. Only JWST is capable of collecting the required observations. Archival data may supplement the analysis, but is insufficient or does not exist in key areas, so new observations are needed. All aspects of the requested observations have been well justified and are necessary for the success of the science program. The observing plan uses resources very efficiently and provides a very clear path to science. AR: Analysis of archival JWST data is critical to reach the stated science goals. The analysis may be supplemented by data from other observatories but the majority of the data will come from JWST. The analysis plan is comprehensive and provides a very clear path to science. Theory: The proposed theoretical or computational work will be transformative for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data. The analysis plan is comprehensive and provides a very clear path to science. |
2 | Major advancement in the sub-field. The proposed program has the potential for major advancement in the immediate sub-field of the proposal. The program will provide major advances in understanding of the objects, class of objects, or specialist topics under study. | Major implications beyond the sub-field. The proposed program has major implications for one or more other sub-fields of astronomy. The impacts of the program are broad and/or significant. | Well-justified with a clear path to science. GO: The program's science goals will certainly be achieved with the proposed observations. JWST offers major advantages over other facilities for collecting the required observations. The need for new data instead of or to supplement archival data is clear. Most aspects of the requested observations have been reasonably well justified and they will contribute to the success of the science program. The observing plan uses resources efficiently and provides a clear path to science. AR: Analysis of archival JWST data offers major advantages over data from other facilities to reach the stated science goals. The analysis may be supplemented by data from other observatories but a significant fraction of the data will come from JWST. The analysis plan is adequate and provides a reasonably clear path to science. Theory: The proposed theoretical or computational work will offer major advances for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data. The analysis plan is adequate and provides a reasonably clear path to science. |
3 | Moderate advancement in the sub-field. The proposed program has the potential for moderate advancement in the immediate sub-field of the proposal. The program will provide moderate advances in understanding of the objects, class of objects, or specialist topics under study. | Moderate implications beyond the sub-field. The proposed program has moderate implications for one or more other sub-fields of astronomy. The impacts of the program are moderate in scope and significance. | Some justification lacking but path to science likely achievable. GO: The program's science goals will be achieved with the proposed observations. JWST offers some advantages over other facilities for collecting the required observations. The need for new data over archival data is discussed but not compelling. Some aspects of the requested observations have been justified but other aspects are lacking adequate justification and it is not clear that all are needed. The observing plan could use resources more efficiently but will likely lead to science. AR: Analysis of archival JWST data offers some advantages over data from other facilities to reach the stated science goals. The analysis may be supplemented by data from other observatories but some of the data will come from JWST. The analysis plan is lacking in some areas but the provided information gives confidence that the science program can be completed. Theory: The proposed theoretical or computational work will offer some advances for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data. The analysis plan is lacking in some areas but the provided information gives confidence that the science program can be completed. |
4 | Minor advancement in the sub-field. The proposed program has the potential for minor advancement in the immediate sub-field of the proposal. The program will provide minor advances in understanding of the objects, class of objects, or specialist topics under study. | Minor implications beyond the sub-field. The proposed program has minor implications for other sub-fields of astronomy. The impacts of the program are limited in scope and significance. | Many aspects poorly justified and path to science uncertain. GO: The program's science goals will probably be achieved with the proposed observations. JWST offers minor advantages over other facilities for collecting the required observations. The need for new data over archival data is not discussed in depth or is not clear. Many aspects of the proposed observations have been poorly justified. There are questions about many aspects of the observing set up. The observing plan uses resources inefficiently and the path to science in uncertain. AR: Analysis of archival JWST data offers minor advantages over data from other facilities to reach the stated science goals. The analysis will mostly use data from other observatories but small fraction of the data will come from JWST. The analysis plan leaves many unanswered questions and the path to science is uncertain. Theory: The proposed theoretical or computational work will offer minor advances for planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data. The analysis plan leaves many unanswered questions and the path to science is uncertain. |
5 | Limited advancement in the sub-field. The proposed program has limited or no potential for advancement in the immediate sub-field of the proposal. The program will not advance understanding of the objects, class of objects, or specialist topics under study. | Limited implications beyond the sub-field. The proposed program has little or no implications for other sub-fields of astronomy. The impacts of the program are extremely limited in scope and significance. | Very poorly justified with no clear path to science. GO: The program's science goals will not be achieved with the proposed observations or JWST offers little to no advantage over other facilities for collecting the required observations. The need for new data over archival data is not discussed. The proposed observations are poorly justified or not explained at all. The observing set up is unlikely to be successful. The observing plan is very inefficient and a path to science is lacking. AR: Analysis of archival JWST data offers little to no advantage over data from other facilities to reach the stated science goals. The analysis will predominantly use data from other observatories with only very little or no data coming from JWST. The analysis plan is extremely weak or missing altogether and leaves no confidence that the work can be completed successfully. Theory: The proposed theoretical or computational work will offer little or no advance in planning future JWST observing programs, for future analysis of JWST data, or interpreting results obtained from JWST data. The analysis plan is extremely weak or missing altogether and leaves no confidence that the work can be completed successfully. |
Examples
The following examples aim to give guidance in applying these rubrics to grading proposals; reviewers should use their best judgement.
Case 1. IR spectroscopic observations of gas in young stars
In field | Highly significant improvement in our understanding of gas flow in young stars. | 1-2 |
Out of field | Potential for significant changes in our understanding of shock excitation in a wide range of other environments. | 1-2 |
Suitability | Warm H2 observations are essential to achieve the science goals and can only be acquired through JWST MIRI. | 1 |
Case 2. Analysis of archival near-IR imaging of a nearby galaxy for star formation investigations
In field | Major advance in understanding of ionized and neutral PAHs in that galaxy. | 2 |
Out of field | Some mention of the applicability to star formation studies in general. | 3 |
Suitability | The increased spatial resolution offered by JWST provides some advantages over other facilities in addressing the science goals. The analysis offers significant improvements and/or additional value with respect to the original use of the data. | 2-3 |
Case 3. Optical/near-IR spectroscopy of a quasar
In field | Moderate increase in understanding of the relative contribution of the AGN vs star formation to the ionized gas emission. | 3 |
Out of field | Minor implications for the properties of other galactic systems, but no wider impact. | 4 |
Suitability | Limited gains in performance at near-IR (rest frame optical) wavelengths as compared with larger ground-based facilities, does not justify the need for the rest frame IR lines | 4-5 |
Case 4. Developing theoretical tools to characterize gas and dust in Galactic star-forming regions
In field | Potential significant increase in understanding of chemical composition in dusty environments. | 1-2 |
Out of field | Results have significant implications for interpreting dust composition in other galaxies. | 2 |
Suitability | The theoretical analysis will enable and support additional JWST observational programs. | 2 |
Next: Proposal Feedback Comments