Analysis on ICML & NIPS

6,163 papers issued from 2005 to 2016

Recently, artificial intelligence (AI) research has shown remarkable results in various fields. This is the result of the efforts of AI researchers who have been studying this field for a long time. The most important academic conferences on AI are ICML (International Conference on Machine Learning) and NIPS (Neural Information Processing Systems), based on the number of citations, the number of participants, the number of accepted papers and history. While ICML focuses on machine learning, NIPS spans a wider range of topics including cognitive science and applied machine learning. ICML held its first academic conference 37 years ago in 1980. and held the 34th conference in August this year in Sydney, Australia. Founded in 1987, NIPS is scheduled to hold the 30th [1] conference in December 2017 in California.
To examine the recent trend in AI study, the research team performed a meta-analysis of 6,163 scientific papers published by ICML and NIPS between 2005 and 2016. The analysis shed light on the recent trend and changes in AI research, by looking at the most cited papers for the past 12 years from various angles such as cumulative citations, collaboration networks of authors, and changes in keywords in papers.

[1. Change in the number of accepted papers for the past 12 years]

Change in the number of papers accepted by ICML for the past 12years

Between 2005 and 2016, a total of 2,315 papers have been accepted by ICML. The number of accepted papers in 2016 was 322, which is more than double the 134 accepted papers 11 years ago in 2005.

Change in the number of papers accepted by NIPS for the past 12years

As for NIPS, the number of accepted papers increased more than double-fold from 207 in 2005 to 568 in 2016.

The research team could confirm that the number of accepted papers in 2012 rose noticeably compared to that of 2011 for both conferences. This resulted from an important event in the history of AI study in 2012. In 2012, professor Geoffrey E. Hinton and his team (University of Toronto) demonstrated a brilliant result using deep neural network in the Imagenet Large Scale Visual Recognition Competition(ILSVRC)

[2. Number of citations by author]

[Top 20 Total Citations in ICML for the Past 12 Years]

[Top 20 Total Citations in NIPS for the Past 12 Years]

[Top 20 Cumulative Citations in ICML and NIPS for the Past 12 Years]

The following graphs show the total number of accepted papers and citations by top 30 most-cited authors (for NIPS and ICML respectively). (Download the original file)

Below is a graph of cumulative citations by year for top 15 most-cited authors.

Change in Cumulative Citations by Year of Top 15 Most Cited Authors

Change in Cumulative Accepted Papers by Year of Top 15 Most Cited Authors

The research team examined a 12-year trend of the top 15 most-cited authors in ICM and NIPS. The reason for selecting 15 authors is that the number can show the trend most effectively through data visualization. The above graphs show the number of cumulative citations and accepted papers for the 15 authors by year. They signify that authors who show a gradual change in the number of accepted papers also published papers continuously year by year. The entire list of the ICML and NIPS cumulative citations can be found in GitHub.

The following table lists most cited papers among those published by the 15 authors appearing in the above graphs. (Download the original file)

Most cited papers written by 15 authors

[3. Collaboration networks]

Collaboration networks of authors with high citation rates

The picture above shows the collaboration networks of authors who are most cited in papers accepted by ICML and NIPS. The bold line means that they often publish as co-authors. For analysis, the research team created a relationship network map of co-authors of papers published by the 15 most-cited authors. The data used in the analysis can be found in GitHub. (Download the original images)

[4. Keywords most often used in titles]

As a way to indirectly examine changes in research topics in the AI field for the past 12 years, the research team analyzed how keywords in paper titles have changed. First, the team used word clouds to get a big picture of the trend change in the past 12 years. The two keywords which were most often used in paper titles for the past 12 years on in both ICML and NIPS on average are “Learning” and “Model”. Since these two words appeared most frequently in paper titles over the target period, the team decided that other keywords than these two words would show the trend change in the AI research. The base years selected are 2006, 2011 and 2016. The interval of 5 years was selected to study changes more clearly.

Keywords appearing in paper titles accepted by ICML (Bigger words means they are used in the titles more often.)

ICML 2006

ICML 2011

ICML 2016

For papers accepted by ICML, the keywords which were most often used in titles in 2006 were “Bayesian”, “Kernel” and “classification”. However, papers accepted in 2016 showed keywords such as “network”, “algorithm”, “optimization” and “deep” most often.

Keywords appearing in paper titles accepted by NIPS (Bigger words means they are used in the titles more often.)

NIPS 2006

NIPS 2011

NIPS 2016

As for papers accepted by NIPS, the keywords which were most often used in titles in 2006 were “Bayesian”, “Kernel”, “classification” and “clustering”, which are quite similar to those appeared in papers accepted by ICML. In contrast, words such as “deep”, “natural”, “network” and “stochastic” appeared most frequently in papers accepted by NIPS in 2016. The examination of these keywords gives insight on how research topics have changed in the past 10 years.

To see the change in the AI research topics in the past 10 years, an analysis of most appeared title keywords was carried out.

Comparison of title keywords in ICML accepted papers: 2006 & 2016

Comparison of title keywords in NIPS accepted papers: 2006 & 2016

While the keyword “deep” appeared none in ICML and only once in NIPS in 2006, it became the most appeared keyword along with the word “networks” in 2016, appearing 22 times in ICML and 43 times in NIPS.

Comparison of the number of appearance of major keywords in paper titles by year

Change in the number of appearance of major keywords in ICML papers by year

Change in the number of appearance of major keywords in NIPS papers by year

[5. Researcher network in the AI field]

The main broker of the AI research network is Michael I. Jordan.

The research team also examined the researcher network based on accepted papers of NIPS and ICML. In particular, a special attention was paid to the status of researchers within the network. The base indices selected were degree centrality which measures how connected an entity is by counting the number of direct links each entity has to others in the network, and betweenness centrality which means the ability to act as brokers connecting other entities within the network. To apply these two concepts to the research network, degree centrality means the degree of direct connection between researchers, while betweenness centrality means the ability to broker between researchers. The number of researchers composing the network structure was 5,878 for NIPS and 3,949 for ICML. Based on the result of network analysis, top 20 researchers ranked by degree centrality and betweenness centrality are shown below. The indices in the graph are standardized values using the absolute value of the top number as the denominator, which make relative comparisons easier.
As for NIPS, Michael I. Jordan [4] who is a professor at the University of California, Berkeley ranked at the top for both degree centrality and betweenness centrality. Jordan is also the supervising professor of Andrew Y. Ng who is known as one of the four AI gurus. Jordan also ranked no. 1 in betweenness centrality in ICML. This result signifies that Michael I. Jordan acts as the hub of the AI research network. In other words, he is the crucial entity connecting researchers in the research network of both conferences. Generally, a broker has control over communications within the network, and the members of the network is dependent on the broker. The network analysis implies that Michael I. Jordan has been at the center of the AI research for the past 10 years.
Aside from Jordan, the so-called AI gurus such as Andrew Ng who is Jordan’s student, Geoffrey E. Hinton and Yoshua Bengio also occupy the top tiers of the network. Among Koreans, Honglak Lee [5], who is the professor of Computer Science & Engineering at the University of Michigan, takes the 19th place in ICML degree centrality. Lee completed his doctoral dissertation under the supervision of Andrew Ng.

The network of AI researchers with Michael I. Jordan at the center is shown below.

Analysis of Network Relationship among Authors in ICML

Analysis of Network Relationship among Authors in NIPS

1) Data Collecting
ICML Conferences http://www.machinelearning.org/icml.html
NIPS Proceeding https://papers.nips.cc/
Accepted papers between 2005 and 2016 were used.

2) Sampling
2,315 papers accepted at ICML and 3,848 papers accepted at NIPS between 2005 and 2016 were used.

3) Paper Citations
Citations by paper title were checked using the website https://scholar.google.co.kr. As new papers are accepted, the number of citations for existing papers are gradually increasing. For the purpose of this analysis, the number of citations are based on those of April 21.

4) Analysis Method
https://github.com/giallo41/Data_Science/tree/master/Conf
Data files collected by the research team and Python source codes used for analysis can be found here.
- ICML & NIPS paper titles, authors and the number of citations per paper for the past 12 years are organized in the Excel format, and analyzed using Python’s Pandas DataFrame.
- The number of cumulative citations for the past 12 years per author were added, and most cited authors were selected using the dataframe.sort() method.
- For paper title analysis, the words in titles were separated and converted into lowercase. Then, such words as ‘:’, ‘?’, ‘for’, ‘a’, ‘an’ ,’in’, ‘of’, ‘with’, ‘and’, ‘the’, ‘to’, ‘on’, ‘from’, ‘by’, ‘using’, ‘very’, ‘via’, ‘it’, ‘that’, ‘as’, ‘,’ ,’which’, ‘-’, ‘through’, ‘without’, ‘while’, ‘is’, ‘than’, ‘where’, ‘much’, ‘many’, ‘or’ and ‘so’ were discarded.
- The word cloud pack provided by Python was used in word cloud analysis which depicts the frequency of occurrence of each keyword in terms of relative typeface size.

5) Concept and Method of Network Analysis
(1) Concept
Researches are conducted by a single researcher sometimes and in collaboration other times. Let us assume that there is Researcher A. Research A may participate in Research (a) and also participate in Research (b). In this case, Researcher A may act as a bridge between both researches. As we can see from this example, a broker can have potential to connect different information or knowledge in one network. In many fields that have interest in network structure including organizational sociology, entities which (may) serve the role of a broker have been examined using the concept of “betweenness centrality”. This method was also applied to this paper. In addition, degree centrality, which looks at the degree of connection between entities, was also used as a measurement index. Degree centrality identifies how much influence one entity has over other entities connected to it.

(2) Method
The authors of paper accepted at NIPS and ICML from 2005 to 2016 were summarized. The number of researchers was 3,949 in ICML and 5,878 in NIPS. The list of authors was sorted into rows and columns, resulting in a square matrix was. ICML is a 3,949 × 3,949 matrix, and NIPS is a 5,878 × 5,878 matrix. We calculated the number of times each researcher wrote a paper with other researchers. If A and B wrote four papers together, then the value in row A, column B is 4. This matrix was analyzed using UCINET 6.0, a network analysis tool. From the results of the analysis, we extracted the top 20 in degree centrality and betweenness centrality respectively.