A Guide to Selecting Distinguished Paper Awards

Emery Berger
PC chair for PLDI ’16 and SIGPLAN EC Chair for SIGPLAN Research Highlights Awards
December 2017

We recommend that all SIGPLAN conferences adopt a formal procedure for selecting ACM SIGPLAN Distinguished Paper Awards (up to 10% of papers at a conference can be so designated) and recommend doing so via the process of establishing a Distinguished Paper Committee to make these decisions.

Here is how this process works at PLDI, starting with PLDI 2016. This is by no means the only way of setting up such a process, but we found it to be effective.

1. Stand up a Distinguished Paper Committee from the Program Committee, or, if possible, from the External Program Committee (see below regarding EPCs, which have been adopted by PLDI and OOPSLA). The Distinguished Paper Committee should comprise senior, established members of the community. (See https://pldi16.sigplan.org/committee/pldi-2016-distinguished-paper-committee for an example.)
2. Request nominations from the PC (and, if applicable, the EPC) from accepted papers with at least one A (generally all of them) and no D scores (since it’s a bit off to have a distinguished paper with a strong detractor). For PLDI 2016, we had 9 nominations and could select up to 4 (10%). We also recommend that if a paper was deemed to require an artifact, the fact that this artifact was or was not submitted to (and whether or not it passed) the Artifact Evaluation process should be considered as part of the decision-making process.
3. Set up a HotCRP instance with the nominated papers, and run like a mini-conference; meet via conference call. The PC chair should upload only the camera-ready papers to this site, without copying over reviews. Members should not be assigned to review papers they may have reviewed for the conference, since this is an entirely different set of criteria. Reviews should be much shorter (e.g., two paragraphs) and focused on whether they are deserving of distinction. During this period, reviews should be made unavailable on the main conference site; these can be opened up after distinguished paper reviews have been submitted. The timescale can be quite abbreviated: for example, for PLDI 2016, bidding was opened up on April 23, papers assigned on April 26, and May 11th was the deadline for reviews (the conference call was May 13).
4. Inform awardees to ensure that they are in attendance when the awards are presented, typically at the beginning of the conference. In addition, it is possible to have a specific Distinguished Papers session; for double-track conferences, this should be made a single-track so everyone can attend (connecting this session with a keynote makes this logistically straightforward).
5. Forward a suitable subset of Distinguished Papers as nominees for SIGPLAN Research Highlights (recipients of which are forwarded for consideration as CACM Research Highlights). Note that the criteria for CACM Research Highlights are special. Nominated papers should be of broad interest beyond the PL community. They must create excitement. Papers with surprising results that stir stimulating debate are particularly encouraged. The results must be recent, significant, and exciting, and of general interest (and accessible) to the CS research community.

For those unfamiliar with the notion of EPCs, this is from the Chair notes for PLDI 2016.

In addition to the Program Committee, we added an External Program Committee (EPC) consisting of senior members of the PL community who acted as a “light” program committee; their charge was to review roughly ten papers each, and in particular, to review all PC-authored papers. The EPC met via a day-long teleconference prior to the physical PC meeting. A subset of the EPC also formed the Distinguished Papers Committee, whose job it was to select the distinguished papers via a conference-like process.

The EPC is intended to be composed primarily of senior, established researchers (and is complementary to the ERC).