I EDITED THIS DOCUMENT TO CONFORM TO THE REQUIREMENTS OF OUR CLASS. PLEASE BE SURE TO READ EVERYTHING IN ITALICS

Detailed Instructions for Associate Chairs (ACs) from the CHI'2010 Papers Co-Chairs

Recall that you have three primary tasks for each of your primary (1AC) papers:

·         Assign reviewers (starts 11/5 reviewers due 11/10)

      In this class you will simply assign but in the PC you’d also have to recruit and manage them to be sure they finished their reviews

·         Prepare a review (due 11/15) and a meta-review of the paper (due 11/17, finalized by 11/24)

      In this class you will write a regular review by 11/15, a paragraph summarizing your opinion and that of the 2AC (by 11/17) and a final review (by 11/24). For CHI, the 1AC only writes a meta review, not an actual review. This is then updated after the author rebuttal period, and finalized after the PC meeting.

·         Lead discussion of the paper at the PC meeting (if paper is discussed)

      At CHI, a paper is only discussed if it has an average score of 2.5 or higher, or someone “Revives” it. In class, we will pick a similar cut off, but also we may not have time to discuss every paper.

You will also be asked to serve as a secondary AC for one paper. For this you will have two primary tasks:

·         Prepare a review of the paper (due 11/15)

·         Assist the 1AC in discussion of the paper at the PC meeting (11/19)

 If this were the CHI PC meeting, you would ignore initial 2AC allocations until later in the process when we have the first round of reviewer feedback, but for this class, you have to finish the review early (9/15) so the 1AC can write the meta review by 9/17.

Detailed Schedule for AC work (italics are things that don’t happen in class but do in real life)

   Date                Task

                         (Make travel arrangement for the PC meeting)

                         (Papers due from authors)

Wed 11/4        AC assignments released to you (and some 2AC assignments)

Thurs 11/5      Look for conflicts and other difficulties, optionally swap problem papers for others

Thurs 11/5      Recruit three high quality external reviewers for each paper

Tues 11/10      Target date for having all external reviewers signed up

                           Track review quality and follow up to ensure reviews are in on time as needed

Sun 11/15      Reviews due back from reviewers (1AC and 2AC)

                           Track missing reviews & quickly resolve (possibly with replacement reviews)

                           Initiate reviewer discussion as needed

Tue 11/17      Write meta-reviews

Tue 11/17      Hard deadline for all reviews (including replacement, and additional reviews)

                           Reviews and meta-reviews released to authors for rebuttal

                           Preliminary "discuss" status determined (based on score cut offs and AC input)

                           SCs assign "discuss" papers to 2ACs for review

                           Author rebuttals due

                           Initiate additional reviewer discussion as needed

                           Review rebuttals, possibly move papers to "discuss" and recruit 2AC reviewer

                           2AC reviews due

                           Updated meta-reviews and final "discuss" status due

Thur 11/19      PC meeting (in Atlanta)

Tue 11/24      Post-meeting finalization of feedback to authors (done in Atlanta if possible)

Tue 11/24      Materials certified by ACs as ready to go back to authors

                           Reviews and decisions returned to authors

Comments on detail items for ACs: These are verbatim from what was emailed to me. I chose not to change it so you get a feel for the dates & so on. You don’t have to read everything below, but you should DEFINITELY read the following items from the list below:

1.     Mon 9/28: Recruit three high quality external reviewers for each paper

2.     Tue 10/27: Write meta-reviews

Mon 9/28: AC assignments released to you
All work with papers for the conference is coordinated through the web-based PCS conference system which you will find at: https://precisionconference.com/~sigchi    As soon as paper assignments are released you should go to the site and look over each paper. 

Tue 9/28+: Initial 2AC assignments [ignore for now]
You may also have some papers assigned to you as 2AC. These are preliminary assignments while the Sub Committee chairs have a fresh overview of papers and ACs in their minds. Ignore these for now. 2AC work will happen after 11/5.

Mon 9/28: Look for conflicts and other difficulties and optionally swap problem papers for others
As a part of your initial review of each paper you should check for any hidden conflicts of interest that the subcommittee chair might have missed.  If you find a conflict you will need to "swap" the paper for another one.  Also, if you feel you really are not the right person to serve as 1AC for this paper and there is probably someone else on your subcommittee who would obviously be better suited, you may also attempt to swap that paper for another more suitable one (but we don't expect this to happen very frequently).  Swapping works on a "give one, take one" basis.  For non-conflict swaps, particularly a little later in the process, it might be best to identify and take a paper from the pool before you return yours.  For conflicted papers you must put your paper in regardless, so it's best to do that as soon as possible.  In that case you may need to return later to find the replacement, but please do try to find one (if papers remain in the pool past a certain point, the SC will have to force an assignment to a non-conflicting AC).

At this point you should also check each paper for anonymization and length limit violations.  Report back any problems to the papers co-chairs at papers@chi2010.org. For your reference the anonymization policy can be found here:  http://www.chi2010.org/authors/chi-anonymization-policy.html

Mon 9/28: Recruit three high quality external reviewers for each paper
Once you have determined that you do not have conflicts, and are going to handle a given paper, the most critical part of the review process -- recruiting of high quality reviewers from true experts -- begins.  Since you are picking those reviewers, your role in this is absolutely essential.  The most important part of your job lies in recruiting the right reviewers -- not just acceptable reviewers, but good reviewers, and not just some good reviewers, but as best you can, all good reviewers.  To help focus on the importance of this task, we are asking that you be prepared at the PC meeting to give a explanation to the rest of the subcommittee for why you picked the reviewers you picked. 

At the same time, experienced ACs also know that it's important to recruit reviewers fairly quickly; otherwise you may find that some of the more in-demand reviewers for any given topic may have already made commitments to others.  These two constraints can make this part of the job difficult. 

There are a number of strategies for finding good reviewers, but what you seek in the end is someone who really knows the subject matter, will make substantive insightful comments, and has the perspective to evaluate how interesting the results are and whether they are sufficiently relevant to (some part of) HCI as a field.  Your first line of attack for finding good reviewers may be your own knowledge -- if the paper is "in your area" (or close to it) you may be able to directly think of a good candidate that you already know to be an expert.  Another excellent way to isolate people knowledgeable and experienced in the area is to consider authors of previously published results on the topic.  You will likely find some of these publications in the references of the paper itself.  Searching for related work in the ACM Digital Library (http://portal.acm.org/advsearch.cfm) and other search engines is also typically very helpful.  Keep in mind that publishing a single paper on a topic, even at a good venue such as CHI, might not mean a person is an expert, and that different authors may have contributed different things to a particular paper.  For potential reviewers you are not familiar with in advance, it can be helpful to try to have a look at their overall research record through their web presence. 

We generally discourage the use of PhD student or convenient "friends down the hall", unless that person really is a highly qualified expert in the area (for example a late stage PhD student might well be extremely knowledgeable in the topic of the dissertation they are completing and mature enough to be a good reviewer).  This year there will be no explicit limits placed on use of reviewers from these categories.  But remember that you will be asked to publicly justify the choices that you have made and choosing a PhD student or more than one of your nearby colleagues may need extra justification.

Mon 10/5: Rough target date for having all external reviewers signed up
Each paper must have at least three external reviewers beyond yourself and the possible 2AC review (and please don't request reviews from other ACs as a part of this three.)  A good strategy is to be prepared for some of your reviewers to decline.  As you search for good reviewers, don't stop at a list of three, but when possible go on to identify one or two more candidates who can be held in reserve.  That way when/if a potential reviewer declines you will be prepared to immediately ask one of your backup candidates.  You may find that potential reviewers are slow to respond to requests.  You might want to set deadlines for a response based on the target date above, and you might consider sending a request to a backup choice in the case of non-responders.  (You may occasionally end up with more than three reviewers this way.  However, we ask that you do not seek more than three reviews as standard practice because good reviewers are a finite resource and we have many papers that need them.)  Note that this date is a rough target only.  The important thing is to ensure that you have three good quality completed reviews by the review deadline.

Mon 10/5: Track review quality and follow up to ensure reviews are in on time as needed
As the review process unfolds you should track your outstanding reviews, send a personal reminder about a week before the deadline, and read over the ones you have received.  If it looks like you may have lost a reviewer, you might consider trying to get a late replacement review (see below).

If you find reviews that you think are of lower quality than you will be proud to stand up at the meeting and state that you recruited, you may want to (very gently and diplomatically -- these are all volunteers giving of their scarce time) suggest to them that they e.g., extend the rationale given for their score.  This is important to ensure that all judgments are well justified, authors can understand them, and they can be weighed against opposing views.  Finally, if you find that reviews for a particular paper are widely divergent and it is unclear how to resolve this you may consider recruiting an additional reviewer as a "tie breaker".  You should not, however, recruit additional reviewers simply with the intent of moving the average away from a result you don't like.

Tue 10/27: Reviews due back from reviewers
Tue 10/27: Track missing reviews & quickly resolve (possibly with replacement reviews)
There is some extra time after the deadline for reviews to allow you to round up final missing reviewers from reviewers (but not much).  In rare cases it may be necessary to find last minute replacement reviews.  We really need 3 external reviews for every paper going into rebuttal on 11/5. 

Tue 10/27: Initiate reviewer discussion as needed
Tue 10/27: Write meta-reviews
Once reviews for a paper are available you can write a meta-review.  If there are significant disagreements between reviewers you may wish to initiate a discussion among them to see if this can assist you in resolving these.  Meta-reviews may contain some (preferably separate) discussion of how you personally see the paper, but are primarily a summary description of the external reviews indicating points where the reviews agree and disagree.  If reviews disagree you may wish to indicate which conflicting arguments you feel are more persuasive or points should be weighted more heavily (but remember that the reviewers will see your meta-review also).  The score you give for the meta-review should normally reflect the underlying reviews.  As a guide you should likely start with the average of the review scores and then think about what adjustments from that might be justified based on resolving conflicts and weighting of the importance of particular points raised.  We are asking you to use your expert judgment here, but you should not be completely substituting your opinion for that of your reviewers.

Wed 11/4: Hard deadline for all reviews (including meta-, replacement, and additional reviews)
Thur 11/5: Reviews and meta-reviews released to authors for rebuttal
All reviews and meta-reviews (with scores) will be released to authors on this date to help them prepare a rebuttal.  Rebuttals allow authors to counter factual errors in the reviews, so it's important that the authors are able to see all the external reviews at this point.  Note that after this point, since the authors will have seen your meta-review it is probably best to make all changes as additions to the review.  If you end up adjusting the score on the meta-review after this point (e.g., based on the author's rebuttal) you should remember that the author knows this is a change and add an explicit explanation of why this was done to the meta-review. 

Thur 11/5: Preliminary "discuss" status determined (based on score cut offs and AC input)
Thur 11/5: SCs revise final assignments of "discuss" papers to 2ACs for review
Just after releasing papers to authors for rebuttal, the papers chairs will determine cutoff lines for papers which will be by default accepted or rejected without discussion at the PC meeting.  Cutoffs will be based on score average and variance.  In addition you may request discussion for papers which wouldn't normally receive it if you feel this is justified.  All papers which are to be discussed at the PC meeting need to have an additional AC present who has also reviewed the paper.  A secondary AC for all these papers will have been assigned by your subcommittee chair.  2AC reviews are full reviews much like external reviews.

Thur 11/12: Author rebuttals due
Fri 11/13: Initiate reviewer discussion as needed
Fri 11/13: Review rebuttals, possibly move papers to "discuss" and ask for 2AC review
You should look at all rebuttals.  If you find that the rebuttal raises important points you may wish to change previously non-discussed paper to a discussed paper.  In that case you should ask the secondary AC to look at the paper.  (In general, any paper discussed at the PC meeting needs to have at least two people in the room who have read it -- it is your responsibility to ensure this is the case).  If there are points in the rebuttal that need discussion you may also initiate additional discussion among the reviewers at this point. 

Wed 11/25: 2AC reviews due
Mon 11/30: Updated meta-reviews and final "discuss" status due
Based on the newly available 2AC review, the rebuttal, and any additional discussion, you can make additional changes (most likely additions) to your meta-reviews.  Again, if you end up changing scores based on these updates it is important that you explicitly explain this in the meta-review.  The final status of whether the paper is to be discussed at the PC meeting should be established at this point.  This will happen either via the automatic cutoff points or your explicit request to discuss a paper (as marked on the meta-review form).

Wed evening 12/2 - Fri 12/4: PC meeting in Atlanta
Additional details about the conduct of the PC meeting will be provided separately.  Please plan to arrive Wednesday afternoon or early evening so we can start work first thing Thursday morning.  Work will continue until 1:30pm Friday.  Please don't plan a departure that would require you to leave the meeting early (Friday and Saturday night hotel costs will be covered where necessary due to travel arrangements).

Fri 12/4: Post-meeting finalization of feedback to authors (done in Atlanta if possible)
Tue 12/8: Materials certified by ACs as ready to go back to authors
Wed 12/9: Reviews and decisions returned to authors
After the meeting you will need to make final updates to the feedback that will be returned to the authors and mark in the system when this is ready to be released to the authors.  This needs to happen fairly quickly to meet our schedule, so you probably want to try to have most or all of this done before you leave Atlanta.  At this stage it is important that you provide information to the authors that enables them to understand why the decision that was made was made.  For conditionally accepted papers you need to provide clear feedback about the expectations of the committee about what changes may be needed in the paper. 

Scott Hudson and Geraldine Fitzpatrick, papers co-chairs
mailto:papers@chi2010.org