Comparison of journal cumulative impact factor and annual impact factor in the evaluation of science journals

The merit of science journals has always been judged by peer review, but it is a time-consuming burden on experts. Consequently, the impact factor (IF), as an accessible quantitative method, needs to be explored in the evaluation of journals in the academic community. The IF refers to the citation of a journal, invented in 1955 by the famous bibliometrician Eugene Garfiled, and is defined as the total number of citations received in a particular year to the source items published in a journal in the previous 2 y divided by the number of 'citable' items published during these 2 y^[1]. Since then, the IF has come to play a significant role in the assessment of scientific journals due to its convenience, sufficient accuracy and novelty. However, due to the increasing application of IF, a lot of people have recognised several flaws with impact factors, of which includes an inappropriate definition of citable items in the calculating formulas of IF, large gaps of IFs between disciplines, and strong bias in favour of U.S. journals^[2], inherent limitations to the SCI database, a too short of a time window for slowly developing research areas^[3] among other factors, so, in 1998, Dr. Garfield proposed the concept of cumulative IF (CIF) in order to modify the short-term time window of the 2-y IF^[4,5]. On the other hand, the original IF in 1955 sought by Dr. Garfield was an IF specific to a year according to its definition, for example, Nature has an IF of 41.456 in 2014, and the Lancet around 35 in 2006, which was named as the annual IF (AIF). And the IFs calculated by Thomson Scientific (Clarivate Analytics at present) every year belong to the scope of AIFs.

In fact, many associated studies have been conducted on the AIF and CIF at home and abroad. In his two papers, Dr. Garfield calculated 7-y CIF and 15-y CIF for the top 100 and 101−200 journals in the annual Journal Citation Report of 1995, followed by the comparison of the differences between 7-y/15-y CIF and the 2-y IF of these journals^[4,5]. In 2010, Haddow et al.^[6] of Curtin University of Technology in Australia proposed to modify the time window of CIF based on the concept of IF. After that, the notion of 'cumulative impact factor', mentioned by a large number of scholars was completely different from Garfield's CIF^[7−11] ; for example, Oelrich et al.^[7] made a comparison of the total number of publications and the cumulative impact factor (short for CuIF for distinction) that were determined for the first 15 E.U. member states (E.U.15), the U.S., and the world in 19 international urological journals in the Web of Science (WoS) database, and its CuIF was determined by the sum of the articles published multiplied with the IF of the individual journal and year. In China, Yang & Ye^[12] were the first to apply Garfield's CIF for journal evaluation in 2001, followed by Du & Tang^[13], who used the calculation of Garfield's CIF to conduct an empirical analysis of the CIF of the physical, chemical, pharmaceutical and surgical journals abstracted in China Scientific and Technical Papers and Citations Database (CSTPCD) of the Institute of Scientific and Technical Information of China. Meanwhile, some attention has been paid to CuIF in several other domestic research ^[14,15].

However, the then-current research is limited to the comparisons of CIF with IF, h-index and g-index, thus leading to the changeable ranking of journals. In view of this situation, we decided to perform a comparison analysis of AIF and CIF with peer review scores of U.S. ophthalmological journals between 2007 and 2016. This paper's aims were to: (1) analyze the efficacy of journal evaluation by AIF and CIF, and (2) to compare the CIF with different window times for the assessment of science journals.

A total of 25 ophthalmologic journals were included in the present study as they met the following the inclusive criteria, including (1) U.S. ophthalmological journals, (2) journals which were indexed in the Clarivate WoS database with citation data during 2007−2016, and (3) journals which had been given to peer review scores by U.S. ophthalmologists via questionnaires, while the exclusive criteria, including (i) journals which were scored by less than 60 U.S. ophthalmologists during questionnaire survey, (ii) journals which were included in the Clarivate WoS database less than 10 y until 2016. The journals which met the above criteria are presented in Table 1. These journals are arranged in alphabetical order.

Methods

Questionnaire implementation

Peer review is recognized as the golden criterion for testing the true impact of journals which can be directly reflected by the peer review scores obtained through filling in questionnaires by experts^[16,17]. Therefore, the self-designed questionnaire, in English, was implemented and issued only to U.S. ophthalmologists and researchers, who were allowed to give the credits to U.S. ophthalmologic journals based on their opinions about these journals' academic impact and quality, considering the fact that scholars in a certain country were not well acquainted with the journals of other countries. In this study, our time was limited so we had to adopt the previous peer review score from the results of the questionnaire survey we conducted in 2016. The procedures of this questionnaire were introduced briefly in the following manner. At first, the e-mail addresses of U.S. ophthalmologic authors (corresponding authors) whose publications were included in WoS-indexed journals were obtained due to the WoS database providing the corresponding author e-mail addresses. Secondly, a questionnaire, in English, was designed (see: www.askform.cn/survey) provided by the supplier of AskForm, and the e-mails were sent to the correspondents by politely informing them of the following information, including (1) a web site, to provide the questionnaire, where the questionnaire can be completed (see: http://app.askform.cn/b8e560ec-16ec-4b35-9267-f895e3915e51.aspx?Type=2), (2) the aim of this survey, which was to achieve the academic impact of U.S. ophthalmologic journals among U.S. ophthalmologic scholars, and (3) the strategies for filling in the questionnaire, which were taken by giving credit to each journal in the questionnaire according to its academic impact or quality in their mind ranging from 1.0 point to 10.0 points, (with 1.0 being the lowest and 10.0 being the highest, fractions with one decimal place were allowed, e.g., 1.1−9.9). Additionally, the journals in the questionnaire were ranked in alphabetical order to keep the questionnaire scoring untouched by the influence of journals' ranking. The 'academic impact' in the questionnaire did not equal the IF or any other indicators, and it solely reflects the journals' academic quality in the field of ophthalmology. Thirdly, the peer review scores were calculated. A total of 7,077 e-mail addresses were harvested, and we received 124 replies, of which the questionnaire in which only three journals or lower were scored, and all journals were given the highest or the lowest credits, as well as the journals that were scored according to the journals' ranking, were excluded. And finally, 112 questionnaires were valid a validity rate of 90.3%. Additionally, the journals that were not scored were recorded as 0 when calculated. The statistics for journal scores given by the ophthalmologic experts were sorted and calculated, and the sum of scores for each journal were recognized as the peer review scores; all the calculations were accurate to 1 decimal place. The survey was conducted between August 4^th, 2015 to September 15^th, 2015^[18].

The acquisition of metrics to the source items and the number of citable items

The database of WoS was searched for acquiring citations and citable items (the number of Review Article and Articles) to these journals via their ISSNs. After signing in to access WoS, we chose 'WoS Core Collection' on the tab of 'Select a database' and 'Advanced search', and then typed the query equations 'IS = XXXX − XXXX AND PY = 2007−2016', thus harvesting results in the search history table at the bottom of the page. Then, all the citation data needed in this study were available and could be downloaded by creating a citation report. On the other hand, the citable items including Review Article and Articles can be refined and their number sorted by publication year was obtained by using the 'Analyze Results' tool attached to the database. The research date was 27^th October, 2018.

AIF and CIF calculation

Two calculations were carried out based on the citations in the WoS for each journal: an AIF and a CIF. The calculation of Dr. Garfiled's IF is well known, and, in this present study, the AIFs with a 1-y time window to 10-y time window for a journal was calculated similarly to Garfiled's IF. For example, AIFs were calculated for all journals using the following equation of the form:

In the equation, n = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. According to the equations, 1-y AIF to 10-y AIF can be achieved.

While as for CIFs, we adopted the calculation of Haddow's extended impact factor^[6] for a journal in a particular year. The CIFs for all journals were calculated using the equation as follows:

In the equation, n = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Therefore, 1-y CIF to 10-y CIF can be computed according to the above equation.

Statistical analysis

Statistical analysis was performed with SPSS19.0 for Windows (SPSS Inc, USA). A Shapiro-Wilk test was used to assess the normality of the distribution of AIFs and CIFs of each journal. The correlation of peer review scores with AIFs and CIFs was performed using Spearman rank correlation. A P value of < 0.05 was considered statistically significant.

The AIFs of a total of 25 U.S. ophthalmologic journals were calculated based on the source data from the WoS database, and their AIFs with different time windows presented in Table 2. As shown in Table 2, we could see that the ranking of these journals by peer review was different from that obtain by sorting by the AIFs, and even differently ranked within AIFs at different time windows, the AIFs were greatly different with 1-y to 10-y time windows. Furthermore, the AIFs produced an interesting alteration in the journals that they initially increased and then decreased with a change in trend, and there were six journals reaching the maximum at the 4-y AIF. On the other hand, a Spearman rank correlation was conducted between the peer review scores and the AIFs, and the results presented in Table 3. The correlation results showed that the AIFs were positively correlated with peer review scores in the U.S. ophthalmologic journals (r > 0.664, all P = 0.000), and the 2-y AIF had the highest correlation with peer review score (r = 0.691, P = 0.000).

The CIF of each journal by time window and the correlation with peer review score

The CIFs of these 25 journals were calculated based on the equations described above and shown in Table 4. According to Table 4, we could make conclusions: (1) the CIFs were larger than the AIFs at the same time window, and this is because, at the condition of the same denominator in the both kinds of equations of AIF and CIF, the numerator in the calculation of AIF was the number of citations received in 2017, which was obviously less than that in the equation of CIFs, (2) with the lag in the time window becoming larger, the CIFs of all selected journals increased gradually. The relation between peer review scores and CIFs among the U.S. ophthalmologic journals was analyzed by Spearman rank correlation in Table 5. As shown in Table 5, we could see that the CIFs, ranging from 1-y CIF to 10-y CIF, were positively correlated with peer review scores of the U.S. ophthalmologic journals, and the 7-y CIF had the highest correlation with peer review score (r = 0.706, P = 0.000).

The correlation of peer review scores with CIF and AIF

To validate the research performance parameters, AIF and CIF, we took the peer review score as the 'golden criteria' for journal evaluation, and made a Spearman rank correlation to analyze the coefficient of correlation of peer review score with AIF and CIF, respectively, and we found that (1) either AIF or CIF was highly correlated with the peer review score, with the correlation coefficient above 0.646, (2) except for 3-y CIF and 6-y CIF, all CIFs with peer review score had the larger correlation coefficient than AIFs with peer review score at the same time window, and (3) the closer the time window, the higher correlation the CIFs had. However, one confusing point of the results was that there wasn't any regular changeable pattern in the correlation of peer review score with AIF and CIF in the present study.

CIFs of four journals with longer time windows

As described above, the CIFs increased as the time window went by, which depended on the calculation equation. And CIFs, only ranging from 1-y time window to 10-y time window were computed due to the limitation of the citation data of the 25 journals available in the WoS database. To investigate whether the CIF grew persistently or not, we conducted the calculations of CIF with longer time windows of four journals, including Ophthalmology, Surv Ophthalmol, Retina-J Ret Vit Dis, Am J Ophthalmol, which have been indexed in the WoS database for a longer time period, and the results are shown in Table 6, which indicated that the CIFs of the four journals did not grow persistently. We could see that the CIF of Am J Ophthalmol began to decrease at the 36-y time window, followed by a drop from 36.084 at the 38-y time window to 29.715 at the 72-y time window, while CIF of Surv Ophthalmol dwindling from 47.313 at the 36-y time window to 46.356 at the 40-y time window and CIF of Ophthalmology from 47.877 at the 36-y time window to 46.890 at the 39-y time window, respectively. Moreover, Retina-J Ret Vit Dis also presented a decline in CIFs at different time windows.

As presented in this study, the results showed although both AIFs and CIFs were positively correlated with the peer review score, the correlation coefficient of CIFs with peer review score overweighed that of AIFs with peer review score at the same time window excluding 3-y CIF and 6-y CIF. There are two possible explanations for this consequence. On the one hand, annual parameters, including 1-y AIF to 10-y AIF, solely involve the number of citations received in a particular year, such as the raw citation counts in 2017, resulting in the omission of a large number of citations in the previous years after the source items were published, which contributes considerably to the impact and scientific quality of journals, especially for journals in less highly active and rapidly developing research fields. On the other hand, the indicators of CIFs relate to all citations accumulating from the year when the source items were published to a particular year, which contributes to the merits of AIFs and the total citations for journal assessment. This consequence suggests that the accumulative total citation should be considered in journal evaluation.

7-y CIF is the optimum indicator

From the correlation analysis, the 7-y CIF had the highest correlation with peer review score (r = 0.706, P = 0.000), so CIF at the 7-y time window was the optimum parameter with regards to U.S. ophthalmologic journal evaluation in this research. However, the citation rate varied in different research fields, thus leading to the variable optimum time window^[19,20], of which the evidence can be studied further.

Changes of CIFs with citation time window

Testing the long-term impact of citation is one of the research objectives of this paper, so, in order to investigate whether the CIF grew persistently or not, we conducted the calculations of CIF with longer time windows of four journals, which have been indexed in the WoS database for a longer time period. The results showed that the CIFs of the four journals did not grow persistently as time went by. This phenomenon maybe due to the aging pattern, including rate of maturation and rate of decline in terms of citations, tending to be specific for individual journals, even in the same subject field^[21]. On the other hand, the CIFs were increasing continually until the 36-y time window in Am J Ophthalmol, whereas this occurred by the 15-y time window in Retina-J Ret Vit Dis. Why did this happen, and was it related to the high impact and quality of the journals or the cited half-life of journals? These hypotheses deserve further investigation.

In conclusion, the intention of the study is to weigh up the AIFs and CIFs of U.S. ophthalmologic journals using bibliometric methods, and this consequence results in the focus of the scholar's attention that should not only be given to the citation counts made in a particular year but the accumulative collection of citations received after the source items are published. More importantly, these results may be only appear in the U.S. ophthalmologic journals, and may be not consistent with journals in other fields.