.jpg)
.jpg)
The purpose of this document is to define the Review Process for the MICCAI conference. The document has been initiated and is endorsed by the MICCAI 2017, 2018, 2019 and 2020 organizing committees, to be passed on each year, from successive MICCAI organizing committees, with the goal to refine and include feedback from the full set of future conference organising committees to create a "sliding window” and consistency between years.
The document is heavily inspired by the previous Review Process developed by S. Ourselin, D. Hawkes, N. Navab, P. Golland, N. Ayache, and G. Fichtinger. The current review process version was prepared jointly by the MICCAI 2017, 2018, 2019, 2020 committees with the following people involved: L. Joskowicz, M. Descoteaux, D.L. Collins, L. Meier-Hain, P. Jannin, S. Duchesne, C. Davatzikos, A.F. Frangi, J.A. Schnabel, D. Shen, T. Liu, A. Martel, D. Mateus, D. Stoyanov, M. Reyes, D. Racoceanu, P. Abolmasumi. We are grateful for all their contributions.
The policies set within are born of a desire to enhance the previous process, by specifically:
As in previous MICCAI conferences, the goal of the MICCAI conference review process is to select the best papers in each discipline. This selection should be fair, taking into consideration the specialized nature of our discipline and the size of our community; efficient, in not wasting valuable time, effort and funding from our peers; and just, in exploiting the consensus of peer comments.
This summary is made public so that all participants understand the review process and can plan accordingly.
The MICCAI 2020 review process will be overseen by the following individuals:
Program Chairs:
Submission Platform Manager:
General Chairs:
The program committee of MICCAI 2020 consists of two bodies of participants that are essential to the review process:
The following documents supplement the following guide, and will be made available on the MICCAI 2020 conference website:
The MICCAI Society should maintain a database of past Area Chairs and reviewers. Excellent ACs and reviewers should be identified after each event. The MICCAI 2020 organizing committee will build on the list provided by MICCAI 2019. The information about the number of times a member has been an AC will be kept indefinitely. The information on excellence of reviewers or ACs will be maintained in the system.
MICCAI 2020 also will issue a call for participation in the program committee via the newsletter and mailshots. Individuals will submit relevant information regarding their past participations in MICCAI and other similar conferences, as well as other biographical details.
Individuals will be chosen from the database, the received invitations and other sources, and invited to join the MICCAI 2020 program committee. When they apply, ACs will need to update their Toronto Paper Matching System entries, create/update their CMT profiles and associate themselves as being either MIC, CAI or MICCAI. They will need to use their institutional email address. The MICCAI Program Chairs will then select the program committee using the following guidelines:
A webinar may be held to explain the review process to ACs and secure their commitment early in the process.
Details and schedule for AC duties include the following:
You will be asked to update the keywords and CMT and upload your relevant papers to TPMS by the end of February. An initial teleconference will be held in March to explain the full review process and to discuss reviewing criteria. After the full paper submission deadline on 3rd March 2020, the first phase of your work will run from 10-15 March 2020 for the initial reviewer assignment period and it is essential that you are available during this week. The paper review period will run from 1-14 April 2020 during which time you are expected to monitor the progress of the reviewers. Between the 14-20 April you will need to assess the quality of the reviews, provide feedback for authors and recommend papers for accept, reject or rebuttal. The second phase will then start once the rebuttals have been submitted and between 13-26 May you will be asked to provide final rankings on your original paper allocations. In addition, you will be asked to act as a meta-reviewer and rank a further 16 papers. These are time intensive tasks and it is essential that you will be available during these crucial periods. Your responsibilities will end with the program committee teleconferences in June 2019. A detailed timeline is posted on the Conference webpage.
By accepting to serve on the program committee, you agree to this timeline.
The list of potential reviewers is primarily based on the list of reviewers who will respond to the Call for Reviewers, which will be disseminated to active reviewers of the last MICCAI conference, to which are added all authors at that conference. Further, student reviewers must be at least enrolled at Ph.D. and have two or more published articles in a related field to qualify as reviewers. ACs will contribute to updating and expanding the list of reviewers.
Each potential reviewer will be invited to review for MICCAI. By accepting to review:
Authors will provide an intention to submit two weeks prior to the paper submission deadline. Authors must provide: i) a list of co-authors and their affiliations, ii) Google scholar and DBPL profiles for each co-author where possible, iii) the title of the paper, and iv) the abstract of the paper. This information will be used to start building the assignment of papers to ACs and potential reviewers. Authors and co-authors must use their institutional/work email address.
Authors will submit papers in CMT. The MICCAI Conference review process will be double-blind, i.e., the names of the authors will be hidden from the Area Chairs and Reviewers, and the names of Reviewers and Area Chairs will not be revealed to the authors. To achieve this, papers must be properly anonymized before submission. At the discretion of the Program Chairs, a paper may undergo an outright rejection when it is in blatant breach of these anonymization rules. Area Chairs and Reviewers may bring concerns about such a breach to the attention of the Program Chairs.
Each paper must be submitted with Primary and Secondary areas selected from the CMT system, and should also indicate up top three relevant keywords in the "keyword” section of the paper template. Authors also must identify to which stream the paper belongs, i.e., either MIC, CAI, or MICCAI. These areas, the stream, and the paper itself, will be used to generate suggested reviewers using the automated TPMS paper matching system embedded in the CMT system (see Stages 5 and 6).
The goal of this step is to ascertain the breadth and depth of expertise required within the program committee in response to the initial submissions from authors. Given that it is almost impossible to accurately predict which domains will be most represented in any conference edition, this step will allow for the adjustment of the program committee composition.
The Program Chairs may then invite approximately 10 additional AC members having an identified expertise from the database of potential ACs and reviewers. Those who agree to join, must update their CMT profiles and associate themselves as being MIC, CAI or MICCAI, but at this point the balance between these three groups will be proportional to their representation in the submissions.
If required, a webinar will be held for all AC members to provide early statistics and explain the process once again.
Each paper will be assigned to an AC automatically by the CMT system, based on the ranked lists provided by the authors and the TPMS. (Note that the AC remains blind to the paper authorship.) The Program Chairs will check the assignments to make sure all papers received a good assignment.
The essential role of the AC is to move the paper through the review process, up until the decision by the program committee. ACs will use their knowledge of the topic and of the appropriate reviewers to ensure the best (most informative) reviews.
For MICCAI 2020, a maximum of 25 papers will be allocated to each AC. This is necessary to ensure a proper statistical distribution in the ranking to follow (cf. Stage 7). The number of ACs will be adjusted accordingly. Thus, for approximately 70 ACs, MICCAI 2020 is expecting to handle around 1,750 submitted papers.
The goal of the paper allocation is to find the most appropriate reviewers in terms of expertise for a given paper. This step is achieved in two phases using the CMT system:
In phase 1, the CMT system will provide a list of potential reviewers for each paper to the AC. This ordered list will be generated based on keywords, TPMS, and other tools use by the computer vision community. Using their expertise and judgement, the AC will re-order this list as needed and can add additional reviewers to the list (including those suggested by the authors, if deemed reasonable). The TPMS system will re-optimize matching of reviewers to papers based on the ACs suggested ranked list of reviewers (at least 6 reviewers on the list). The optimization will consider the ordered list from the AC, while load balancing across all papers, reviewers, and ACs. Each reviewer will be assigned an initial list of 25 papers.
In phase 2, reviewers will bid for papers by categorising them into "I am interested in reviewing this paper”, "I can review this paper if need be”, "I am not interested in reviewing this paper”, "I have a conflict with this paper”. The TPMS system will then re-optimize matching of reviewers to papers based on the ranked list provided by the reviewers, while load balancing across all papers, reviewers and ACs. Note that the reviewers and secondary ACs remain blind to the paper authorship at all times. CMT permitting, primary ACs will be blinded to the final set of reviewers.
This procedure has the following advantages over previous methods:
The Program Chairs will ensure ACs have re-ordered reviewers on time and can provide additional opinions if a borderline situation cannot be resolved by the AC members.
The goal of the review step is to provide constructive feedback of each submitted paper.
The reviewers will:
The AC will log in frequently and react to the reviewers' actions: remind reviewers to login and download the papers, and especially ask for a more detailed or fair review. This is critical. Should the AC be unsatisfied with the quality of a review, and failing to get further feedback from the reviewer, then the AC can ask for additional reviewer(s) input on the paper, beyond the original three reviewers. Any reviewer who does not provide a quality review will be identified in the reviewer database.
The Area Chairs will recommend clear acceptance, identify borderline papers for rebuttal, and clear rejection of papers based on consistent reviews and scores. The Program Chairs recommend that:
This year, the conference will aim for an acceptance rate of ~30%.
We anticipate that after the initial review in this stage, ~25% will be accepted outright; ~ 55% will be rejected, and the remaining borderline papers will be sent for rebuttal. Approximately 40% of these borderline papers will ultimately be accepted.
Borderline papers will enter Stage 9 (below). The goal of the rebuttal process is to provide authors of borderline papers the opportunity to highlight possible misinterpretations or inaccuracies in the reviewer's findings, and inform the AC's final recommendation for scoring. The primary AC will provide a meta review for all cases where they disagree with the reviewers comments.
The goal of this step is to provide a numerical score and rank for each paper assigned to the AC.
After rebuttals are entered, each AC will provide for the borderline papers a recommendation to either "accept” or "reject”. ACs will further provide a meta-review for each assigned paper, consistent with their recommendation and based on the assessment of the reviews and rebuttal.
In addition, each AC will:
The goal of the decision process is to establish the final list of accepted papers. Prior to this stage, the Program Chairs will update the ACs regarding the total number of orals and selected papers to be awarded at the conference after consultation with the Organizing Committee and the MICCAI Board.
Following rebuttal, in addition to the remaining ~5 papers in each AC's pool emerging from the first round (i.e. initial 25 minus the top 25% and bottom 55%), each AC will be assigned the remaining set of ~5 papers each from two other ACs (so each AC will look at ~15 papers). This assignment of borderline papers will be made by the Program Chairs. ACs will be asked to read the reviews, the rebuttal, and the papers if necessary, and recommend "Accept” or "Reject”. The final "Accept” will be by majority voting (i.e. if at least two ACs recommend "Accept”). By this process, and a wind-up conference call, we hope to add 40% of borderline papers to the "Accepted” pool.
After having switched to the CMT, and TPMS systems for MICCAI 2017, 2018, and 2019, and after having collected significant data before, during, and after the program committee meetings, we will again replace the program committee meeting by a set of dedicated teleconferences between the MICCAI 2020 Program Chairs and the ACs to inform about the overall results and statistics of the review process to discuss and resolve any arising issues, and to collect their feedback.
The goal of this step is to select the best submitted papers that justify an oral presentation.
The number and format of the oral sessions will be determined by the Organizing Committee, in consultation with the MICCAI Board and will reflect the overarching objectives of the conference. The recommendations of the ACs will be taken into consideration when deciding which papers to select for full oral presentation. There will be an emphasis on providing Young Investigators with the opportunity to present and there will minimally be 1 CAI oral session.
Three teleconferences will be held, in three time zones (GMT ±8h) to discuss the outcome of the MICCAI review process, statistics and any arising issues, and to collect feedback and recommendations for any further improvements of the process.
The goal of this step is to inform authors about the outcome of the review process.
The Program Chairs will issue the following, via email, to all authors:
A complete list of acceptance and orals will then be drafted for dissemination as a program. The Program Chairs will record each complaint and follow up.