1 Introduction

To develop large annotated corpora is a challenging task in the area of Natural Language Processing (NLP). It is a common strategy to rely on automatic analysis tools to pre-annotate the data and then to manually review the annotation, as direct human annotation requires large amounts of time, a strict discipline, and an homogeneous system of annotation rules that must be followed by annotators. In the case of treebanks (i.e., databases of sentences and their corresponding syntactic trees), this must be carried out for both Part-of-Speech (PoS) tagging and syntactic parsing, at least. However, as the annotation of PoS tags can be considered input for syntactic annotation, the better the PoS annotation, the lower the risk of tagging errors leading to parsing errors.

In the pre-annotation of PoS, the hardest part is assigning tags to words from closed classes, as many of them may pertain to different classes, that is, they are ambiguous. These words are among the most frequent in any corpus, regardless of genre and domain. As they constitute marks for syntactic annotation decisions, it is important to annotate them accurately, thus offering safe annotation points to support the effort of producing large annotated treebanks.

Of particular interest to this paper is the initiative to perform PoS tagging for Portuguese language according to the widely adopted Universal Dependencies (UD) model [18, 20], which predicts the existence of “universal” PoS tags and syntactic relations that may be applied to all languages in the world. There are currently nearly 200 treebanks in over 100 languages affiliated to UD.

In Portuguese, we highlight the previous work of Rademaker et al. [22], that has produced a UD-annotated version of Bosque treebank [1], and the recent efforts of Silva et al. [25] that have explored PoS-tagging for tweets, Lopes et al. [17] that shows that there are non-ambiguous words and sequences of words that may significantly improve tagging accuracy, and Souza et al. [3] that creates a UD corpus for domain specific documents. Nevertheless, in practically all languages and domains there is a large number of challenging words that are ambiguous and require more efforts to be properly annotated.

We focus here on investigating and proposing methods for solving the problem of PoS tag disambiguation of words of a particular case in Portuguese: the closed class words. Such word classes include prepositions, conjunctions, determiners, numbers, pronouns and primitive adverbs (non-derived adverbs). The motivation for such choice relies on the fact that closed class words may be very ambiguous regarding PoS tagging and, once solved, may serve as solid evidence to help disambiguating words of other word classes, improving tagging accuracy overall. For instance, in PortiLexicon-UD [16], which is a UD-based lexicon for Portuguese, the word “que” (in English, it usually corresponds to “what”, “that” or “which”) may have up to 7 different word classes and represents a challenge for annotation, as discussed in the work of Duran et al. [6]. To exemplify the challenges, we reproduce some examples from this work: the word “que” is:

  • a pronoun in the sentence “As coisas com que sonhamos são pistas para o autoconhecimento”,

  • a subordinate conjunction in “É obvio que quero votar”,

  • a determiner in “ Que maravilha”, and

  • an adverb in “ Que horrível foi esse acidente”.

The work of Silva et al. [25] evidences the practical difficulties in PoS tagging. Some of these classes, as pronouns and subordinate conjunctions, achieve some of the lowest accuracy values (below 90%) in the reported experiments.

In our paper we explore three different methods to disambiguate the words of the selected classes:

  • a Markovian model similar to the one developed by Assunção et al. [2] to disambiguate each ambiguous word, considering the PoS tag sequences of a training set;

  • the tagger/parser UDPipe 2.0 [28] pretrained for Portuguese is used to tag the whole sentences, and we observe the decision took for ambiguous words;

  • a contextual word embedding-based solution using a pretrained Portuguese model to individually disambiguate the PoS tag for ambiguous words, in a similar approach to Vandenbussche et al. [30] that uses BERT (Bidirectional Encoder Representations from Transformers) [4] by a pretrained language model specific for Brazilian Portuguese, the BERTimbau [26].

We perform a detailed error analysis in comparison with two baselines and also explore an Ensemble solution considering a voting system taking into account the three presented models’ outcomes. Our results show accuracy values ranging from 90% to 98% for the three proposed methods, as well as the Ensemble one. While the BERTimbau-based approach delivers the higher accuracy values for the best cases experimented, the Ensemble solution presents more stable results since the accuracy of all experimented cases has a smaller standard deviation than all three methods applied individually. The achieved accuracy is relevant and the gains in terms of annotation effort are considerable, as we correct nearly 25% of all tokens of large Portuguese corpora.

This paper is organized as follows. The next section presents the main related work. The problem definition and the baselines for this study are presented in Sect. 3. Section 4 presents the three experienced models: the Markovian Model, the UDPipe 2.0 application, and our contextual word embedding BERTimbau-based approach, respectively. Section 5 shows our experiments over an example corpus and discusses the achieved effectiveness. Finally, the final section summarizes our contributions and suggests some future work.

2 Related Work

We can find in the literature related works using varied techniques to deal with PoS ambiguities. We comment on the main ones in what follows.

Ehsani et al. [7] perform the disambiguation of PoS tags for Turkish using Conditional Random Fields (CRF)  [14]. In such work the authors point out the processing burden associated using CRF to perform the task, but they deliver PoS tag accuracies around 97%. The authors also mention that this high PoS tag accuracy is very beneficial to support reliable lemma and morphological annotations.

A different approach is the work of Assunção et al. [2] that uses a Markovian model to perform PoS tag disambiguation based on Markovian models [13] and a partially annotated corpus. In this work, the disambiguation follows a basic probability computation of PoS tag sequences, and therefore delivers faster computations than more sophisticated approaches.

The work of Hoya Quecedo et al. [10] presents an effort to disambiguate PoS tag and lemma for morphological rich languages, which is the case of Portuguese, using a Neural Network model based on BiLSTM (Bidirectional Long Short-Term Memory) [8]. Hoya Quecedo et al. build word embeddings to submit to the bidirecional model and, thus, estimate the probability of the PoS tag and lemma options of each word. These experiments deliver a precision varying from 80% to 85% to corpora in Finnish, Russian, and Spanish.

Another interesting work published by Muñoz-Valero et al. [19] also employs an LSTM (Long Short-Term Memory) RNN (Recurrent Neural Networks) [5] to disambiguate words in the American National Corpus [11]. Similar to this one, the works of Shen et al. [24] and Zalmout and Habash [32] aim at the disambiguation of morphological features, both using LSTM and RNN.

3 Problem Definition and Baseline Approaches

The problem we are aiming to solve is how to disambiguate closed class words that can be tagged with different PoS tags, for example, the word “por” that in Portuguese can be either an ADP (in English, “by”) or a VERB (in English, “to put”). Our proposed methods are based on working with a training set of fully annotated sentences using Universal Dependencies (UD) in the CoNLL-U format [20] in order to disambiguate the PoS tag annotation of all ambiguous words of a test set.

The set of UD PoS tags is formed by:

  • ADJ - adjectives, as “bonito” (“beautiful” in English);

  • ADP - adpositions, as “de” (“of” in English);

  • ADV - adverbs, as “não” (“no” in English);

  • AUX - auxiliary verbs, as “foi” (“was” in English);

  • CCONJ - coordinating conjunctions, as “e” (“and” in English);

  • DET - determiners, as “cujo” (“whose” in English);

  • INTJ - interjections, as “tchau” (“goodbye” in English);

  • NOUN - nouns, as “vida” (“life” in English);

  • NUM - numerals, as “cinco” (“five” in English);

  • PART - particles, which is not employed in Portuguese;

  • PRON - pronouns, as “ele” (“he” in English);

  • PROPN - proper nouns, as “Brasil” (“Brazil” in English);

  • PUNCT - punctuations, as “?”;

  • SCONJ - subordinating conjunctions, as “porque” (“because” in English);

  • SYM - symbols, as “$”;

  • VERB - verbs, as “jogamos” (“(we) play” in English);

  • X - others, as foreign words.

The closed classes considered in this paper are ADP, CCONJ, DET, NUM, PRON and SCONJ, plus subsets of the classes ADVFootnote 1, AUXFootnote 2, and ADJFootnote 3. Table 1 shows the considered closed classes, the total number of words (#tot.), the number of ambiguous words (#amb.), and some examples. Considering that many ambiguous tokens belong to several classes, the overall sum of ambiguous tokens is 368 different words. A full list of the considered closed class words is available at https://sites.google.com/icmc.usp.br/poetisa/publications.

Table 1. Closed PoS tags considered in our work.
Full size table

The first baseline (Baseline 1) that we evaluate takes into account the number of occurrences of each ambiguous word in the training set and uses the most common PoS tag to all occurrences in the test set. For example, if the word “até” that can be an ADP (“until” in English) or an ADV (“even” in English) is found 189 times as ADP and 77 times as ADV, we will assume that all occurrences of the word “até” will be considered ADP. However, to prevent decisions based on too sparse data at the training set, we will only consider a prediction when there are at least 3 occurrences of a word with the most frequent PoS tag. Additionally, to prevent decisions based on too close differences, the second more frequent PoS tag has to be less than half frequent than the more frequent one.

The second baseline considered is the implementation following the definitions made by the work of Hoya Quecedo et al. [10] that uses a Neural Network model based on BiLSTM as described before. We employ this baseline using the same hyperparameters defined in [10] and we will refer to this as Baseline 2.

4 Prediction Methods

In this section we present the three methods implemented in these paper experiments. The first one is a traditional Markovian modeling approach inspired in the work of Assunção et al [2] and it is presented in Subsect. 4.1. The second method is the application of the UDPipe 2.0 tool [27] and it is presented at Subsect. 4.2. The third method is an approach based on BERTimbau pretrained neural language model [26] to build a contextual word embedding model and it is presented at Subsect. 4.3.

4.1 Prediction Through Markovian Models

Among the Markovian approaches to stochastically predict language information, we decided to employ the one based on Assunção et al. [2]. This approach is based on developing a Markov chain model to each sentence of the training set describing the probabilities of PoS tag sequences.

For an illustration of the method, consider the sentence “Com a rescisão, as provas apresentadas pelos delatores ainda podem ser usadas.”. This sentence will be annotated as described in Fig. 1. This sentence corresponds to the attached Markov chain that represents the sequences of PoS tags. For example, the two words tagged as ADP are followed by DET, therefore the probability of transition from ADP to DET is 100%. Similarly, the three occurrences of NOUN are followed by PUNCT, VERB, and ADV tags, therefore, each existing arc in the chain has a transition probability of 33.3%. Additionally, one additional node representing the boundary of the sentence (\(\star \)) links to the sentence start (before ADP) and the sentence end (after PUNCT).

Fig. 1.
figure 1

Annotated sentence and corresponding Markov chain of PoS tag sequences.

Full size image

This Markov chain creation process is repeated to all sentences of the training set. Once all sentences of the training set are used to create a Markov chain, each possible word with ambiguous PoS tags is disambiguated considering the PoS tags preceding and succeeding the ambiguous word.

4.2 Prediction Through UDPipe 2.0

The second method experimented is the use of the tagger/parser UDPipe 2.0 [28] using one of the available Portuguese training models, which is BOSQUE-PT [29] in our experiment, composed by 9,364 sentences and 210,957 tokens. UDPipe 2.0 uses a RNN based on BiLSTM to perform a full UD annotation based on a previously training set.

Our approach in this paper is to feed the whole sentences of the testing set to UDPipe 2.0 and pinpoint the ambiguous words to observe the accuracy of disambiguation of the target closed class words.

4.3 Prediction Through BERTimbau-Based Model

A more recent approach, which has achieved state of the art results in many NLP tasks [9, 15], is based on the usage of pretrained neural Language Models (LM), such as BERT [4]. A LM is capable of providing contextualized word embeddings, which can later be used in a downstream task. Specifically, we use the language model BERTimbau, a BERT model pretrained for Brazilian Portuguese [26], in the smallest version, BERTimbau-base, that, like BERT-base, has 12 layers.

All sentences are previously tokenized using the same BERTimbau model, as it requires the words to be split into subtokens. For words consisting of more than one subtoken, the embedding used during classification represents the first one.

For the PoS disambiguation, our model has two inputs: the sentence duly tokenized and the position of the token to disambiguate. BERTimbau is used to retrieve a sentence vector (which represents the whole context) and also the word embedding for the token to disambiguate (to include specific information about the token), which are later concatenated and passed through a linear layer in order to create a combined representation.

Finally, this vector is fed into another linear layer with Softmax for classification into PoS tags, plus a dropout layer between the combined representation and the final classification layer in order to tackle over fitting issues. Figure 2 depicts the general process for the BERTimbau-based method.

Fig. 2.
figure 2

BERTimbau-based method process.

Full size image

For the fine tuning of the model, the Cross Entropy loss function was optimized using Adam [12] with learning rate \(\alpha = 10^{-5}\) for 2 epochs. The dropout rate used was 0.2 with batch size 8. The whole model was implemented using HuggingFace’s Transformers library [31] alongside PyTorch [21] and it is available as supplemental materialFootnote 4.

5 Experiments

The conducted experiments were performed over an annotated corpus in Brazilian Portuguese language with 8,420 sentences (168,397 tokens) extracted from the first five thousand news from Folha Kaggle data bank [23]. This corpus was manually annotated using UD tags by a group of 10 annotators with redundancy and supervised by a chief linguist. It contains (among its 168,397 tokens) 44,208 occurrences of 368 distinct ambiguous words.

To allow performing the experiments and comparing all the methods, we randomly split the corpus into: train - a training data set with approximately 80% of the tokens of the full corpus; dev - a development data set with approximately 10% of the tokens of the full corpus; test - a testing data set with approximately 10% of the tokens of the full corpus. However, to prevent bias in our experiments, we created 10 of such random splits as folds to perform cross-validation in order to establish the accuracy of each method. The application of Baseline 1 and the Markov-based model employs the train+dev data sets to training and test data set to testing. The application of UDPipe 2.0 method employs test data set as testing and ignores the data sets train and dev. The application of Baseline 2 and BERTimbau-based model employ each split individually.

Table 2 presents the accuracy and F1 obtained for each fold, while Table 3 summarizes the results obtained by each method for each fold in terms of the averageFootnote 5, standard deviation, minimum, and maximum results of the ten folds individually. In these tables, the methods are abbreviated as Markov, for the Markov-based method (Subsect. 4.1), UDPipe for the UDPipe 2.0 application method (Subsect. 4.2), and BERT for the BERTimbau-based method (Subsect. 4.3). Finally, in these tables we included the accuracy and F1 results of an Ensemble method. The Ensemble is obtained applying the Markov-based, UDPipe 2.0, and BERTimbau-based methods, and deciding on the PoS tag by a simple voting strategy among them.

Table 2. Accuracy and F1 values of each method for each of the ten folds.
Full size table
Table 3. Summarization of accuracy and F1 values considering the ten folds.
Full size table

The first observation from the results in Table 3 is that the three methods are significantly more accurate than the baselines. It is also noticeable the high accuracy achieved by the BERTimbau-based approach that alone provides a better accuracy than the Ensemble method. The second observation is the stability of the Ensemble method delivering smaller standard deviation both for accuracy and F1. The accuracy achieved by the Ensemble method is slightly smaller than the BERTimbau-based one, but since the its variance of results is smaller, it is probably more reliable to employ the Ensemble method in a real annotation where the gold standard is not previously known.

To illustrate the difficulties of the phenomenon, we analyzed the number of mistaken decisions (wrong PoS tags) made by each of the three methods, stated in the first three rows of Table 4. We also present in the middle rows of this table the mistakes shared by the methods, two by two, and all together. The last rows stated the total mistakes by methods and the use of an Ensemble of the three methods application.

Table 4. Number of wrong PoS tags by method for each of the ten folds of the PoS tag errors.
Full size table
Fig. 3.
figure 3

Number of wrong PoS tags by each method for Fold 4

Full size image

Figure 3 shows a Venn diagram of the PoS tag errors made by the three methods for Fold 4 (which is a fold with number of errors nearest to the average of all folds, as highlighted in Table 4). For this fold, there were 319 PoS tag errors by the Markov-based method alone, 187 by the UDPipe 2.0 method alone, and 9 by the BERTimbau-based method alone. Similarly, 50 PoS tag errors were made by both Markov-based and UDPipe 2.0 methods, 18 by Markov-based and BERTimbau-based methods, and 16 by the UDPipe 2.0 and BERTimbau-based methods. Finally, 29 errors were made by all three methods. These lead to 416 errors by Markov-based (319+50+18+29), 282 by UDPipe 2.0 (187+50+16+29), and 72 by BERTimbau-based (9+18+16+29). Since the Ensemble method will always decide wrongfully when the three methods are wrong (29 errors), when any two methods are wrong (50+18+16), the errors of the Ensemble method will appear on 113 PoS tags (29+50+18+16).

Analyzing Fold 4 testing set outcome, we observe that it is composed by 842 sentences and a total of 16,938 tokens. From these, 4,354 tokens are ambiguous words from closed classes, and only 72 were incorrect after applying BERTimbau-based method, i.e., the method produced 4,282 correct tokens (25.21% of the total tokens), which correspond to 98.35% of the ambiguous tokens.

To illustrate the errors of the methods, we use two words from the 29 mistakes in the testing set for Fold 4. The token “segundo” (“second” in English) at the end of the sentence “Flamengo e São Paulo viraram 12 e 11 pontos de distância, mas não estavam em segundo.” is tagged as ADJ, but the methods delivered a NOUN prediction, because the methods had difficulty to handle the noun elipse, as the full form would be “segundo lugar” (“second place” in English). The first token “a” (“her” in English) at the sentence “Ela recebeu uma mensagem certo dia chamando a a ir a Cariri...” is tagged as PRON. The three methods delivered ADP prediction because a double token sequence “a a” is the usual tokenized form of crasis (“à”, in English “to the”), a very frequent form in Portuguese that is tagged as ADP DET, respectively. In the training set for Fold 4, the number of crasis occurrences is 276, while other occurrences of double “a” were never present, consequently the three methods were unable to learn tags PRON ADP for the sequence, assigning DET instead of PRON for it. One may see that both cases are difficult ones, which somehow explains why the methods failed.

6 Conclusion

We conducted an experiment with three proposed methods that show to be competitive and delivered an accuracy gain over the baselines, reaching average accuracy values ranging from 90.76% to 97.84%. The F1 average values were also impressive, ranging from 94.89% to 99.11%. It is important to mention that, since we experiment using a 10-fold cross-validation, our results are statistically sound, as the standard deviation of the results of each fold was always lowFootnote 6.

It is particularly noticeable that the BERTimbau-based approach brings a relevant correction power. Recalling Fold 4 testing set correction of 25.21% of the total tokens, we observe that, to a large corpus as the Folha-Kaggle [23] with around 84 million tokens, we can automatically deliver more than 20 million correct tokens for ambiguous words of closed classes.

It is also noticeable that combining the three presented methods in an Ensemble method has shown a small impact in the average accuracy with respect to BERTimbau-based method, respectively 97.23% and 97.84%, but the Ensemble solution offers more stable results, since the standard deviation of the 10 folds dropped from 0.0034 for BERTimbau-based solution to 0.0024 for the Ensemble one.

As future work, we intend to perform a deeper analysis of the Ensemble approach, perhaps adding other methods casting votes together with the three methods already developed. It is also possible to imagine the development of a similar technique to disambiguate other fields of the UD model, as lemma and morphological features, given the promising results achieved by our methods.