Proposal Feedback Comments - Cycle 1
All proposers receive feedback. Primary Reviewers are responsible for compiling reports for their assigned proposals. These feedback comments should take into account the panel discussion for proposals discussed at the in-person meetings. For triaged proposals, consensus reports should be based on the comments submitted by reviewers during preliminary grading.
Contents of the Feedback Comments
The review comments are broken down into different subsections. The first two are compulsory, the rest are optional (and will likely not be required for most proposals):
- Strengths: self-explanatory. Compulsory.
- Weaknesses: self-explanatory. Compulsory.
- Resources: notes about the Orbits or Targets or Funding levels, or instrument/configuration/filters, only needed if something is out of the ordinary. Optional.
- Comments: anything else that you want to pass on that doesn’t fit under Strengths, Weaknesses, or Resources. Optional.
- Technical Notes: updated based on technical reviews, usually relevant for Joint Observatory programs or if reviewers have concerns about instruments. In the latter case they must request a consultation from an instrument expert via the Panel Support Scientist, and the notes from that consultation will be entered in this section. (This section will not be sent to the Proposers, it is for the review only, copy anything relevant to the Resources section). Optional.
- Instructions: for approved programs only. Additional instructions for the proposers to modify their final submission. Changes may be required if two non-identical approved proposals contain duplicate observations; in that case, one PI will be instructed to remove the duplicated observations, while preserving the unique elements of the proposal. Optional.
Guidelines for Writing Feedback Comments
Panel reports should be concise. They should focus on the evaluation of the proposal by the reviewers. Please avoid “questions” in the comments. So, e.g., instead of “why have 6 targets instead of 5?” say, “the proposal did not sufficiently motivate the number of requested targets.” Likewise, refer to what the proposal says rather than what the proposers say.
Comments that may be perceived as derogatory or insulting must be avoided. The Panel Chairs are responsible for checking for any such improprieties in their review of the comments.
Except for triaged proposals, reviewers cannot be sure at the time of writing feedback comments whether the proposed observations will be scheduled for execution. The comments should be phrased in such a way that they are sensible and meaningful regardless of the final outcome.
Reviewers should avoid statements that create the impression that the low ranking of a proposal is due to a minor mistake. Many proposals do not have obvious weaknesses but are just less compelling than others: in such a case, acknowledge that the considered proposal is good but that there were others that were more compelling. However, never include in the consensus report an explicit reference to another proposal, such as the proposal ID. Whenever possible, make suggestions for possible improvements, but avoid giving the impression that following those suggestions guarantees that the proposal will be more successful in next cycle.
The Primary Reviewer is responsible for submitting a panel report that is as specific, accurate, clear and informative as possible. The Panel Chairs shall in the week following the TAC meeting review the panel reports of all the proposals that have been reviewed in their panel, with a view to ensuring that all of them achieve the necessary standard of quality in providing the PIs with meaningful feedback. The STScI Science Policies Group will undertake a final review of all comments before they are distributed to the community.
Process & Workflow for Finalizing Feedback Comments
The Primary Reviewer is responsible for submitting a feedback report that is as specific, accurate, clear and informative as possible. The Panel Chairs shall in the week following the TAC meeting review the panel reports of all the proposals that have been reviewed in their panel, with a view to ensuring that all of them achieve the necessary standard of quality in providing the PIs with meaningful feedback. The STScI Science Policies Group will undertake a final review of all comments before they are distributed to the community.
Some guidelines on workflow for writing reviews prior to and during the Virtual Meeting:
- Please do not discuss individual proposals in the Slack panel channel. It is OK to mention proposals a la “I think I’ve incorporated your comments into the feedback for #123, let me know if it’s right” or “I could use some extra input on proposal #321”, but don’t discuss the merits or the science here.
- You can use the “Remarks” section in SPIRIT to collaborate on writing the reviews and contributing comments to the primary reviewer.
- DMs on Slack are allowed. Please use this Slack workspace over emails; we remind you that this is proprietary information. Use common sense here.
General Procedure for Reviewers:
- Both Primary and Secondary Reviewer add their comments. This can be done when the status of the proposal is in “Ready” or “Reviewing” status. (Saving often!)
- The Secondary Reviewer (typically) will Mark Completed their comments.
- The Primary Reviewer merges Secondary’s comments into their own, as well as any Remarks that have been left by other panelists. The Primary can hit the Mark Completed, to indicate they are done with making updates to their comments.
- The Primary or Secondary both have the ability to Reopen the Review to make edits to their comments.
- Once the virtual review has begun, either Reviewer, but it should generally be the Primary, can Submit as Final to indicate the comments are finalized. A Panel Admin or the Chair can also Submit as Final, if necessary. Whichever reviewer hits the Submit as Final button will have their comments pushed through as the “Submitted Review."
- Once the Submit as Final button has been pressed, two things happen:
- The review status for the proposal changes to “Submitted.”
- The tab “Submitted Review” can now be edited by the Chair/PSS to make edits for continuity (they should also Save often!).
Both the Chair and the PSS will proofread the reviews. The PSS will look for the following, among other things:
- grammatical errors and typos
- anything contradicting the previously posted guidelines
- personal remarks about proposers
- explicit identifications of other Cycle 29 proposals or proposers, reviewers, or STScI staff
- comments that contain only insubstantial or superficial remarks
- remarks that do not make sense because the reviewers expected the proposal to be approved, while in the end it was not
Once the PSS/Panel Chair is happy with the final review they should click the Review Signoff button. This will change the review status of the proposal to “Complete.”