Why This Influential ML Conference Banned Social Media Discussion During The Review Period?

The IEEE computer society technical committees on pattern analysis and machine learning (PAMI TC) has recently passed four motions with respect to reviewing papers at IEEE/CVF conferences.

The four motions passed at CVPR 2021 includes: 

• Conference and TC Ombud Positions  
• Diversity, equity and inclusion initiatives 
• Voluntary disclosure of conflicts of interest
• Social-media limitation during the review 

The new policy prohibits authors from issuing press releases or talking with the press about papers under review at IEEE/CVF conferences. It also facilitates double-blind review and helps preserve the public trust in the scientific peer-review process. 

"With the increasing use of social media to distribute scientific results, this policy needs updating," said PAMI TC.

arXiv excluded    

The research papers platform arXiv, which automatically posts research papers on social media platforms, has been excluded from this ban. 

The committee believes that dissemination on arXiv facilitates the rapid spread of information within the field.

"arXiv papers are often corrected and modified; the site is set up to support this scientific process of revision. Putting a paper on arXiv for early analysis by experts is very different from publicly promoting work on social media to a broad audience," said PAMI TC. 

But what about the tweets from arXiv platform? "arXiv tweets are largely followed by experts in the field and not the general public. The work is presented in its entirety and a pre-publication and can be judged as such," said PAMI TC.

Why now? 

The committees said the social media campaigns have largely taken over traditional press releases. The current policies are somewhat ambiguous about such posting but may be interpreted as allowing this as long as the authors do not say 'the paper is under review at an IEEE/CVF event.' "This loophole efficiently violates the principles of the original press ban," reads the document.

The committee said groups with large followings and the resources to mount visible social media promotions received significant attention for work under review. The committee believes it will create bias among the reviewers.

PAMI TC said groups with fewer followers are at a disadvantage. "This biases the peer review process and reduces trust in its fairness," said PAMI TC.

Further, the process helps them detect mistakes or false claims before the work appears in public. Also, this reduces the chance that work needs to be retracted and, hence, increases public trust in science and the scientific process. "Science depends on this trust both for funding and for its independence. Anything that undermines this trust can have long term negative consequences for basic research," according to the PAMI TC research committee. 

Machine learning expert Yannic Kilcher claimed he had exposed more false claims on his YouTube channel than the entire CVPR conference in the review process. "We have to get away from this notion that peer review is adequately constituted by three dudes sitting on the toilet while flicking through your paper on their smartphone and then giving a weak reject," he added. 

Christian Wolf, a researcher at INSA Lyon, said we sometimes get quick opinions on papers by social media influencers."But actually improving a paper requires a deep read and a long and detailed review, rebuttals, short, a real discussion," he said, in a Twitter post, replying to Black's tweet. 

At CVPR 2021, close to 7,093 papers were submitted. Out of these, 7,039 were assigned to reviewers, while 4,312 papers were rejected, 1,047 were withdrawn, and 19 were desk rejected. In total, only about 1,660 papers made it to the poster and oral presentation, a staggering acceptance rate of 0.236. 

It all comes down to peer versus popularity review of research papers. While some might look at this move as a measure towards eliminating biased peer reviews, others believe social media platforms act as a tool to offer the best peer-review possible.