Evaluation criteria in Tourism and its effects on the researchers’ publication profile

1. Introduction

Tourism, considered a recent field of studies and research in Brazil, had its beginning with undergraduate higher education in the 1970s (Rejowski; Aldrigui, 2007). The first researches on the tourist activity were influenced by the researchers formation areas, being carried out from the perspective of the different knowledge areas. This fact allows several approaches to Tourism that involves from economic and administrative aspects to sociological aspects related to the interaction between the individuals.

As it was expected, the scientific communication process has developed over time, and has shown growth in recent years, even for articles as for journals. However, it seems that the creation of domestic online journals in Tourism, as an initiative of many institutions, goes beyond the increase of the community – or the articles published by them –that raises many questions about their quality. On the other hand, we can see that there are 45 international journals indexed in the Journal Citation Reports (JCR) under the category Hospitality, leisure, sports and tourism – remaining 23, disregarding those specifically aimed at Sport – which points out that there are some options for international publications.

It is also worth mentioning the existence of an evaluation process of postgraduate programs carried on in Brazil by CAPES (Coordination for the Improvement of Higher Level Personnel) (Oliveira; Amaral, 2017), in which the Tourism subarea is evaluated with other subareas, in the so-called area ​​Administration, Accounting and Tourism (ADM / CON / TUR). Facing this, we question the fact of Tourism be evaluated among these subareas, given the risk of printing on it and its researchers, only the view of an economic activity, instead of looking at its social aspects, using appropriate parameters for its evaluation. Therefore, it is important to identify and measure the influences of the evaluation process on scientific production, which end up causing some areas to change their communicational practice in search of a better evaluation (Butler, 2007).

Qualis was proposed by CAPES in 1998 and consists of a journal classification system made with the participation of the academic scientific community. It is used to evaluate researcher’s scientific production from the postgraduate programs, according to a classification scale following the criteria defined by the evaluation commissions of the different areas – as is the case of ADM / CON / TUR –, being annually updated. For each stratum, a different score is defined based on the journal classification: A1, A2, B1, B2, B3, B4, B5 and C - being A1 and A2 the best journals, and C those that do not score.

In this sense, the scientific communication processes and the scientific knowledge must be considered within the context in which each community has different characteristics, being composed of departments and laboratories in universities and research institutions, including formal communication channels (scientific journals , societies and scientific events) (Kuhn, 2000). In addition, there are the growing complexity of the scientific communication system and the different types of interactions of its participants (HURD, 2000). Not to mention the existence of external factors, for example, of the research funders, who impose rules that may influence the choice of publication channels (Björk, 2007).

Each scientific community presents a behavior pattern that directly influences both the research activity and the way of communicating it. Thus, in the case of Social and Human Sciences, Hicks (2004) warns that evaluation processes, involving bibliometric indicators should consider four types of literature: journals, books, national literature (developed in a local context) and non-scholarly literature (knowledge in search of application). The author also highlights the differences between areas, such as economics and psychology whose literature is similar to the scientific literature, or sociology that belongs to a more paradigmatic Social Science literature, or history that represents the Humanities. Moreover, the author enlightens that citation indexes can favor the first two, but consequently may lead researchers to adapt their research to the evaluation models, which would result in a new social science, that leaves the three literatures in favor of indexed journals, marginalizing specific literatures, regardless of its purposes.

Among the marginalized literatures, domestic (or national) journals deserve attention, especially in countries where most researchers are not fluent in English, regardless of the development of a good publishing infrastructure, as is the case in some Latin American countries that are part of the SciELO project (Packer, 2011). Leite et al. (2011), analyzing curricula of about 51 million Brazilian doctors, based on their percentage of foreign-language publications, found out that in the Humanities and Social Sciences 80 % of the researchers presented a publication profile exclusively in Portuguese (publishing less than one-fifth of their production in another language). However, when comparing the variation of this index between two quadrennials – in the turn of the millennium –, they noticed that about 3 % of the PhD’s began to publish between 1/5 and 2/5 of their production in another language, in accordance with the inducement of the Brazilian scientific policy.

According to Beigel (2014), the pressure for publication in mainstream journals, and its relation with the evaluation, financing, and performance system– measured by the bibliometric indicators – has the serious implication of excluding from circulation the knowledge produced in peripheral circuits, non-integrating the mainstream. Colombian researchers, when interviewed by Chavarro et al. (2017), expressed that domestic journals are used for beginning researchers training, aiming at the next step, which is the mainstream publication. On the other hand, they pointed out the importance of these publications as knowledge bridging (by providing access to the national community through the publication in domestic journals to contents that normally are published exclusively in mainstream journals) or to fill gaps of knowledge (that consists on the publication of contents not presented in mainstream journals).

As shown, it is apparent that the scientific evaluation of production is a challenge that must be adequate to the subareas different specificities that when evaluated together can favor some subareas to the detriment of others, implying drastic consequences for its development. It is important to investigate possible changes in the researcher’s publication profile from the Tourism subarea – that appears to be mostly in domestic journals – which may be being forced to publish in specific mainstream journals from the Administration subarea.

2 Methodology

The research was carried out into three steps, two of which are directly related to the scientific production evaluation criteria of the Administration, Accounting and Tourism area done by CAPES: the area documents in which the classification criteria are established into a strata scale and the classified journal list used for publication with the respective stratum classification they received. The third step of the research was based on information from the curricula of the 108 accredited researchers in all nine Tourism postgraduate programs (PGPs).

In the first step were analyzed the area documents from the three years period 2007-2009 (that was here denominated T4 since it is the fourth triennium, considering that the first one was 1998-2000) and 2010-2012 (T5), to identify the classification criteria of the journals used for publication by the areas and the respective score for publication in each one.

In the second step, we analyzed a single list of the 2.187 journals that received a Qualis classification in the ADM/COM/TUR area in the periods T4 and T5. The journals were analyzed one by one using the following variables: knowledge area (one journal could be classified in more than one area), nationality, publication language, publisher and classification strata.

In the last step, information was extracted from the Lattes curricula regarding the publication of the Tourism researchers. The researchers accredited in the PGPs of the Tourism subarea were identified in the Sucupira Platform – information system that subsidizes the evaluation carried out by CAPES – and, from the list of researchers, the curricular information was obtained directly from the Lattes Platform, with the use of the Script Lattes tool (MENA-CHALCO; CESAR JR., 2009). The analysis considered the distribution of the articles in the several journals classified by the area in the period from 2010 to 2014 – this period extrapolates the end of T5 (2012), allowing the analysis of the effect of the criteria (established in T5) on the scientific production of subsequent years. The variables are the same as those listed in the second stage, except for the Qualis stratum of the journal which, by extrapolating the T5, was chosen the best classification between the years 2012, 2013 and 2014.

3 Results

3.1 Evaluation of scientific production in ​​ADM/CON/TUR area

3.1.1Evaluation criteria - analysis of area documents

Among all the evaluation areas, it is observed that the journals classification criteria can be divided into three types, by requiring: bibliometric indicators, mainly JCR Impact Factor (FI-JCR); databases indexing or journal characteristics (Mugnaini, 2015; Oliveira; Amaral, 2017). The last one consists of a guideline that determines models to be considered in the new journals establishment (Bonini, 2004).

Considering the higher strata (A1, A2 and B1), on T4 the ADM/CON/TUR area already based its criteria into indicators (setting FI-JCR levels for A1 and A2, and indexing on SciELO bases, EconLit, PsycInfo and Redalyc or journal characteristics for B1). In T5, the use of indicators for the three strata is intensified and the minimum FI-JCR was also established for B1.

Regarding the intellectual production valorization, weights are assigned for each publication vehicle (journal, book, event annals), related to the program total assessed items. Between the triennia, the ADM/CON/TUR area stops considering the events participation, besides reducing the score for publication in journals and books from 22.8 % to 17.5 %.

Another aspect is an assignment of points in journals publication on different strata. While most areas assign 100, 85 and 70 points for strata A1, A2 and B1 respectively, the ADM/CON/TUR area assigns 100, 80 and 60, denoting further devaluation of stratum below A1.

3.1.2 Journals Classification - analysis of their characteristics

In this stage of the research, the journals that received Qualis classification in the ADM/CON/TUR area were observed. Of the 2.187journals (T4 and T5), 608 are of ADM, 113 of CON and 50 of TUR. There are 1.471 journals from other areas, which were evaluated by this committee due to the publication of some researcher from the ADM/CON/TUR area – it is worth to mention that journals may have been classified according to more than one area. Among the areas, we highlight those with about 100 or more journals: Economy followed by Education and Multidisciplinary, either journals of Humanities and Engineering in general.

When we look at the journals publishers (Table 1) we realize that, when it comes to the A1 and A2 strata, most of them are concentrated among the large commercial publishers (this concentration is even more pronounced for the ADM/CON/TUR journals than for those from other areas). This phenomenon is influenced by the fact that these strata have the FI-JCR as classification criteria and these editors have the monopoly of most of these journals, as observed by Larivière et al. (2015). It is worth noting that in B1 stratum and other lower strata, the concentration in the large publishers is dissipated. Elsevier publishes the highest percentage of A1 journals, followed by TAYLOR and Springer; yet in A2 and B1 it is noticed that Emerald and INDERSCIENCE concentrate most of the ADM/CON/TUR journals.

Table 1 - Distribution of journals classified in ​​ADM/CON/TUR area,
according to the classification stratum, publisher and journal area

Analyzing the journals distribution into the classification strata of each area separately (Table 2), we observed that the ADM subarea has a different profile from the other two, presenting a significantly larger number of foreign journals in the T4 and T5 triennials (61.3 %, and of these, 85.3 % are in English) as well as the predominance into the high strata. In CON, it is noted that only in the T5 the foreign journals predominate in the upper strata, which also happens in TUR, but with smaller magnitude – it should be noted that the quantities are derisory in comparison with ADM. Also in the T4 and T5 triennials, CON and TUR present the majority of domestic journals (65.5 % and 62.0 %, respectively), and from foreign journals, 76.9 % and 47.4 %, respectively, are in English. When we look the other areas we can see the increase in the number of journals between the T4 and T5 triennials and at the same time, there is a slight increase in the percentage of these in the upper strata, denoting that the area is publishing more in qualified journals from other areas.

It is noticed the lack of A1 domestic journals in the three subareas, and in other areas this quantity is also small, which is due to the fact that the social sciences and humanities domestic journals are not well represented in the Web of Science (WoS) or have a low FI-JCR.

Table 2 - Distribution of journals classified in ​​ADM/CON/TUR area,
according to triennium, and journal area and nationality

The outlined scenario clearly favors the ADM subarea which held 100 % of A1 journals in T4 (29 foreign journals) and 90.6 % in T5 (58 foreign journals); in the A2 stratum it is not different, with 85.3 % in T4 (22 foreign and 7 domestic journals) and 87.5 % in T5 (60 foreign and 10 domestic journals). The insertion of journals from the other subareas is very timid, but in T5 they had some journals in these strata (9 of CON, and only 3 of TUR). The requirement for classification in stratum A1 was FI-JCR>0.5 or h-index>5 on Scopus database in T4, increasing to 1.0 and 20, respectively, in T5; and for stratum A2, the requirements were 0<FI-JCR ≤0.5 or 0<h-index≤5 in T4 and became 0.2 <FI-JCR ≤1.0 or 4<h-index≤20. Such impositions have prevented the classification of domestic journals from the three subareas in A1 stratum, and in the case of TUR in A2 as well, because they are not indexed in WoS or Scopus database.

For this reason, the analysis of the distribution of the Tourism subarea scientific production was proposed, aiming to identify the effects of the journals classification in the ADM/CON/TUR area.

3.2 Tourism researchers publication profile

From Table 3 it can be noted that 538 (or 61.2 %) of the articles are published in journals from the own subarea, 15.6 % from ADM/CON and 23.2 % from other areas. On the other hand, the increase in the number of articles in TUR journals is continuous throughout the T5 (2010-2012) period, presenting slightly lower number in subsequent years.

Considering the publication percentages in foreign journals, there is a slight oscillation in TUR journals throughout the period, totalizing about one third of the production in international journals. Worthy to highlight the years 2011 and 2014 that show the highest percentage observed in the period. In journals from other areas, the trend is to decrease the percentage over the period, although we note the beginning of the publication in foreign journals of ADM/CON in the two years following T5 (6 articles in 2013 and 5 in 2014).

Table 3 - Distribution of Tourism researchers articles according to year and journal area

Table 4 draws attention to the only publication in A1 in 2014, in ADM – Journal of Cleaner Production, edited by Elsevier since 1993 with FI-JCR 2014 of 3,844. On the other hand, publications in A2 are concentrated in TUR journals and increase in volume as the strata decrease until B3. There are only two TUR journals classified as A2, one Turkish (Anatolia turizm ve çevre kulturu dergisi), used only once, and another Argentine (Estúdios y Perspectivas del Turismo) with the 58 remaining articles.

The percentage of publications in TUR foreign journals is also significant in strata B1 and B3, denoting that the internationalization effort is occurring independent of the strata (a similar situation occurs when considering journals from other areas), but mainly in Spanish. However, when publishing in ADM/CON journals, the researchers seek for foreigners journals classified on higher strata in English.

Table 4 - Distribution of Tourism researchers articles according to journal stratum and area​​

The Tourism researcher’s publication profile showed a high concentration in B3 journals (39.4 %) and it should be noted that they are only in 2 foreign journals (one Spanish and one Portuguese) and 7 domestic journals.

From the above, it can be affirmed that the scientific production in Tourism is published mostly in Portuguese. In the case of foreign journals, when not published in the Portuguese journal, it is almost exclusively in Spanish. Of the 23 hospitality, leisure and tourism journals present at JCR 2016, all require manuscripts in English and most (19) would be classified as A1 (and the other 4 as A2) if they had been used for publication by this community. In the short term, the subarea will need to make efforts to publish in journals indexed in the JCR (in English) if it wants to have journals in A1 stratum; and in a longer period, needs to obtain indexing of domestic journals.

As Hurd (2000) and Beigel (2014) point out, the publication of articles in mainstream journals is the requirement that normally oriented evaluation systems around the world and in Brazil has not been different. In the evaluation carried out by CAPES, these criteria are used to determine the PGPs evaluation, resulting in what Butler (2007) calls the "goal shift". As it was seen, the Tourism researchers chose to adapt their research to the problems dealt in foreign journals from the Administration area in order to raise levels of excellence established by the area.

Hug, Ochsner and Daniel (2013) also warn of the importance of criteria and indicators of quality in the humanities that follow an inside-out approach – based on the specificities of the different subareas and then propose comprehensive humanities criteria in general. This would mean, in the case of the ADM/CON/TUR area, to complement its criteria in order to take into account the specificities of the Tourism subarea. In this sense, Larivière et al. (2015) clarifies that this observed capillarity between the human and social areas results from the absence of a larger scientific society that brings together the researchers/editors of the various disciplines, making them more vulnerable, leading journals from many countries to turn to large commercial publishers. In the case of Brazil – in addition to several Latin American countries –  the SciELO project offered a different path for most domestic journals, inducing quality aspects similar to those from the large publishers and international databases. However, one must consider the innumerous humanities journals that are not yet indexed.

The Tourism subarea suffers with criteria that favor Administration, since FI-JCR levels are determined by the journals that researchers in this subarea used for publication. However, although the tourism domestic journals do not enjoy mainstream status, the Revista brasileira de pesquisa em turismo – a bilingual journal published since 2007 by the National Association of Research and Postgraduate in Tourism – was the first one to be indexed in SciELO in 2016. This is an indicator of the maturation of the subarea at the domestic level, which could serve as a stimulus for a more appropriate evaluation of national journals.

4 Conclusions

When analyzing the joint assessment established in an evaluation area such as Administration, Accounting and Tourism, it was observed the predominance of criteria that favor Administration. It is estimated that small changes in the criteria could favor the Tourism subarea, such as requiring only indexation on specific bases, rather than minimum JCR Impact Factor or h-index in Scopus. However, such adjustments would not exempt researchers from the need to discuss their results in an international context, as it is happening, but to reach higher levels will inevitably require publication in English. This fact means a significant effort, given to the concentration of publications in domestic journals from strata B1 to B3 and the fact that publications in international journals are mainly in Spanish.

The situation of the Tourism subarea is not different from other areas, even those evaluated individually. The use of the JCR Impact Factor ignores some journals aspects that can show its importance to the area, as we could see in this study. On the other hand, the use of h-index, which is increasing (including when calculated in other bases, such as Google Scholar or SciELO), will not bring balance to the issue. It is very inadvisable for the journals classification because the indicator is not based on an average, which makes its performance depends on the volume of articles that the journal publishes (Waltman; Van Eck, 2012).

Finally, it is advised that bibliometric indicators should not be used exclusively, especially when dealing with the smallest number of journals on the higher strata (A1 and A2), in order to guarantee an evaluation that is more consistent with the specificities of the areas. In fact, even members of CAPES warn the scientific community that composes the evaluation commissions: "metrics should not be used indiscriminately or purely for accounting, without paying attention to its limitations. Moreover, there should not be single indicators for evaluation, overlapping experts’ analysis" (Oliveira; Amaral, 2017, p.163).

Acknowledgement

We thank the São Paulo Research Foundation (FAPESP, grant 2012/00255-6), for funding the research. We also thank Professor Jesús P. Mena-Chalco who gently extracted data from the Lattes curricula of Tourism researchers.

References

Beigel, F. (2014) "Publishing from the periphery: Structural heterogeneity and segmented circuits. The evaluation of scientific publications for tenure in Argentina’s CONICET". Current sociology, v. 62, n. 5, p. 743-765.<https://doi.org/10.1177/0011392114533977>. [Consulta: 06/11/2017].

Björk, B-C. A model of scientific communication as a global distributed information system. Information Research, v. 12, n. 2, 2007. <http://InformationR.net/ir/12-2/paper307.html>. [Consulta: 29/01/2018].

Bonini, A. (2004). "Qualis de Letras/Lingüística: uma análise de seus fundamentos". Revista brasileira de pós-graduação, vol. 1, no. 2, p. 141–159.<http://dx.doi.org/10.21713/2358-2332.2004.v1.45>. [Consulta: 06/11/2017].

Butler, L. (2007) "Assessing university research: a plea for a balanced approach". Science and public policy, vol. 34, no.8, p. 565-574.<https://doi.org/10.3152/030234207X254404>. [Consulta: 06/11/2017].

Chavarro, D. A., Tang, P., Ràfols, I. (2017). "Why researchers publish in non-mainstream journals: training, knowledge bridging, and gap filling". Research policy, v. 46, n. 9, p. 1666-1680.<https://doi.org/10.1016/j.respol.2017.08.002>. [Consulta: 06/11/2017].

Hicks, D. (2004)."The four literatures of Social Science". En: Moed H.F., Glänzel W., Schmoch U. (eds.)."Handbook of Quantitative Science and Technology Research". Springer, Dordrecht, p. 473-496.<https://doi.org/10.1007/1-4020-2755-9_22>. [Consulta: 06/11/2017].

Hug, S.; Ochsner, M.; Daniel, H-D. (2013). "Criteria for assessing research quality in the humanities: a Delphi study among scholars of English literature, German literature and art history". Research evaluation, vol. 22, p. 369–383.<https://doi.org/10.1093/reseval/rvt008>. [Consulta: 06/11/2017].

Hurd, J. M. (2000). "The Transformation of Scientific Communication: A Model for 2020". Journal of the American Society for Information Science, vol. 51, no. 14, p. 1279–1283.<http://dx.doi.org/10.1002/1097-4571(2000)9999:9999<::AID-ASI1044>3.0.CO;2-1>. [Consulta: 06/11/2017].

Kuhn, T. S. (2000). "A estrutura das revoluções cientificas". São Paulo: Editora Perspectiva.

Larivière, V.; Haustein, S.; Mongeon, P. (2015). "The Oligopoly of Academic Publishers in the Digital Era". PLoS ONE, 10 (June).<https://doi.org/10.1371/journal.pone.0127502>. [Consulta: 06/11/2017]

Leite, P.; Mugnaini, R.; Leta, J. (2011). "A new indicator for international visibility: exploring Brazilian scientific community". Scientometrics, vol. 88, p. 311–319.<https://doi.org/10.1007/s11192-011-0379-9>. [Consulta: 06/11/2017].

Mena-chalco, J. P.; Cesar Jr., R. M. (2009). "ScriptLattes: an open-source knowledge extraction system from the Lattes platform". Journal of the Brazilian Computer Society, vol. 15, no. 4, p. 31–39.<http://dx.doi.org/10.1007/BF03194511>.[Consulta: 06/11/2017].

Mugnaini, R. (2015)"Ciclo avaliativo de periódicos no Brasil: caminho virtuoso ou colcha de retalhos?".En: XVI ENANCIB-Encontro Nacional de Pesquisa em Ciência da Informação. João Pessoa: UFPB. <http://www.ufpb.br/evento/lti/ocs/index.php/enancib2015/enancib2015/paper/view/2984/1157>. [Consulta: 06/11/2017].

Oliveira, T. M. de; Amaral, L. (2017) "Public Policies in Science and Technology in Brazil: challenges and proposals for the use of indicators in evaluation". En: Mugnaini, R.; Fujino, A.; Kobashi, N. Y. (orgs.)."Bibliometrics and scientometrics in Brazil: scientific research assessment infrastructure in the era of Big Data". São Paulo: ECA/USP, p.189-217.<https://doi.org/10.11606/9788572051705>. [Consulta: 06/11/2017].

Packer, A. L. (2011) "Os periódicos brasileiros e a comunicação da pesquisa nacional". Revista USP, n. 89, p. 26-61.<http://dx.doi.org/10.11606/issn.2316-9036.v0i89p26-61>. [Consulta: 06/11/2017].

Rejowski, M.; Aldrigui, M.(2007) "Periódicos científicos em Turismo no Brasil: dos boletins técnico-informativos àss revistas científicas eletrônicas". Revista Turismo em Análise, v. 18, n. 2, p. 245-268.<http://dx.doi.org/10.11606/issn.1984-4867.v18i2p245-268>. [Consulta: 29/01/2018].

Waltman, L.; Van Eck, N.J. (2012) "The inconsistency of the h-index". Journal of the American Society for Information Science and Technology, v. 63, n. 2, p. 406-415.<http://dx.doi.org/10.1002/asi.21678>. [Consulta: 29/01/2018].