Proposal Feedback Comments
Guidance for reviewers writing proposal feedback is provided in this page.
On this page
All proposals receive feedback.
- For each proposal in the Discussion Panels, its Primary Reviewer is responsible for compiling all the proposal feedback comments received by all its assigned reviewers, and for using those comments to write a final version of the proposal feedback comments, taking into account the panel discussion. For triaged proposals, consensus feedback comments should be based on the comments submitted by reviewers during preliminary grading.
- For each proposal in the External Panels, its reviewers provide proposal feedback comments that are appended together and sent verbatim. Reviewers must ensure that their comments are appropriate.
Reviewers are forbidden from using Generative Artificial Intelligence (GAI) tools to assist with any part of their reviews since this violates the confidentiality of the review process. This means that it is forbidden to upload proposal contents or review materials to GAI tools. It is also forbidden to upload your written comments to GAI tools for the purpose of constructing your final review or polishing the language.
Contents of the proposal feedback comments
Discussion and External Panels
The proposal feedback comments are broken down into different subsections. The first two are compulsory, the rest are optional (and will likely not be used for most proposals):
| Review comment tab name in SPIRIT | Required or optional? | Description |
|---|---|---|
| Strengths | Required | Describe the strengths of the proposal. Note that a summary of the proposal is not in itself a strength. |
| Weaknesses | Required | Describe the weaknesses of the proposal. |
| Resources | Optional, rarely used | Notes about the Time request or Targets, or instrument/configuration/filters, only needed if something is out of the ordinary |
| Comments | Optional, rarely used | Anything else that you want to pass on that doesn’t fit under Strengths, Weaknesses, or Resources. |
| Technical Notes | Optional, rarely used | Updated based on technical reviews, usually relevant for Joint Observatory programs or if reviewers have concerns about instruments. In the latter case they must request a consultation from an instrument expert via the Panel Support Scientist, and the notes from that consultation will be entered in this section. (This section will not be sent to the Proposers, it is for the review only, copy anything relevant to the Resources section.) |
| Instructions | Optional, rarely used | Additional instructions for the proposers to help them prepare their submission for implementation/scheduling should their proposal be approved. This can include mandatory recommendation for a change in the time allocation or the targets. Targets or observation time may be changed if two non-identical approved proposals contain duplicate observations; one of the proposals would be instructed to remove the duplicated observations, while preserving the unique elements of the proposal. (These recommendations will be made only by the Discussion Panel.) |
Guidance on how to enter this information in SPIRIT can be found in the SPIRIT WebReviewer Tool Guide.
Guidelines for writing proposal feedback comments
Discussion and External Panels
When writing proposal feedback comments, reviewers
- should be concise, constructive, and focus on the evaluation of the proposal by the reviewers;
- must focus on the scientific content and not the team,
- so please refer to the proposal / the observations / the analysis / the data / the work rather than the proposers / the team / the authors / they;
- must not ask “questions” in the comments;
- must not reference the reviewer or the panel, as this creates undue focus on the reviewers rather than the integrity of the process;
- must not leave comments that may be perceived as derogatory or insulting, in compliance with the Code of Conduct;
- must not phrase comments in a way that conveys certainty about whether or not the proposal will be accepted or the proposed observations scheduled for execution, as reviewers cannot be sure of this at the time of writing, even for very highly ranked proposals.
- The comments should be phrased in such a way that they are sensible and meaningful regardless of the final outcome.
- must not include any explicit reference to another proposal, such as the proposal ID;
- must avoid statements that create the impression that the low ranking of a proposal is due to a minor mistake. Many proposals do not have obvious weaknesses but are just less compelling than others: in such a case, acknowledge that the considered proposal is good but that there were others that were more compelling.
- must avoid giving the impression that following those suggestions guarantees that the proposal will be more successful in next cycle.
- Whenever possible, do make suggestions for possible improvements.
The Panel Chairs and SPD Panel Managers are responsible for checking for any such improprieties in the proposal feedback comments.
Examples
Below are some examples of proposal feedback comments that need revising and possible revisions compliant with the above guidelines.
| Problematic proposal feedback comment | Compliant proposal feedback comment |
|---|---|
| "It was not clear how the proposers/ the authors/ they are going to analyze the data." | "It was not clear how the data are going to be analyzed." |
"It was not clear to me/ this reviewer/ the panel how the data advance the science goals." | "It was not clear how the data advance the science goals." |
| "What is the rationale for this set of filters?" | "The rationale for this set of filters was not sufficiently explained." |
| "What about magnetic fields?" | "The impact of magnetic fields was not discussed in the proposal." |
| "This proposal was stupid." | (remove entirely) |
| "I really liked this proposal!" | "This was a compelling/ well-written/ well thought out proposal." |
| "This proposal is a must-do for JWST." | "This was a very compelling proposal." |
| "If the proposers can show a preliminary analysis of the archival data, this proposal will do better next time." | "The proposal would be strengthened by showing a preliminary analysis of the archival data." |
| "This was one of three proposals to observe this target evaluated by our panel." | (remove entirely) |
Process & workflow for finalizing proposal feedback comments
Discussion Panels Only
Guidance on how to enter this information in SPIRIT can be found in the SPIRIT WebReviewer Tool Guide.
The Primary Reviewer is responsible for submitting proposal feedback comments for each of their assigned proposals that are as specific, accurate, clear and informative as possible. In the week following the TAC meeting, the Panel Chairs shall review all the proposals feedback comments received by all the proposals in their panel, with a view to ensuring that all of them achieve the necessary standard of quality in providing the PIs with meaningful feedback. The SPD will undertake a final review of all the proposal feedback comments before they are distributed to the community.
Some guidelines on workflow for writing proposal feedback comments prior to and during the Discussion Panel meeting:
- Please do not discuss individual proposals in the Slack panel channel. It is OK to make general statements “I think I’ve incorporated your comments into the feedback for #123, let me know if it’s right” or “I could use some extra input on proposal #321”, but don’t discuss the merits or the science here.
- You can use the “Remarks” section in SPIRIT to collaborate on writing the reviews and contributing proposal feedback comments to the primary reviewer.
- DMs on Slack are allowed. Please use this Slack workspace instead of emails; we remind you that this is proprietary information. Use common sense here.
General Procedure for Reviewers:
- Both Primary and Secondary Reviewer add their proposal feedback comments. This can be done when the status of the proposal is in “Ready” or “Reviewing” status. (Saving often!)
- The Secondary Reviewer (typically) will Mark Completed their comments.
- The Primary Reviewer merges Secondary’s comments into their own, as well as any Remarks that have been left by other panelists. The Primary can hit the Mark Completed, to indicate they are done with making updates to their comments.
- The Primary or Secondary both have the ability to Reopen the Review to make edits to their comments.
- Once the virtual review has begun, either Reviewer, but it should generally be the Primary, can Submit as Final to indicate the comments are finalized. A Panel PSS or the Chair can also Submit as Final, if necessary. Whichever reviewer hits the Submit as Final button will have their comments pushed through as the “Submitted Review."
- Once the Submit as Final button has been pressed, two things happen:
- The review status for the proposal changes to “Submitted.”
- The tab “Submitted Review” can now be edited by the Chair/PSS to make edits for continuity (they should also Save often!).
The Chair, the PSS, and the SPD Panel Manager will proofread the proposal feedback comments. See the PSS guidelines for proposal feedback comments.
Once the Chair, PSS, and SPD Panel Manager are happy with the final proposal feedback comments, they will click the Review Signoff button. This will change the review status of the proposal to “Complete.”
Next: Discussion Panel Meetings