1 Introduction

The Relation Extraction (RE) task aims to identify and classify the semantic relationships that occur between (pairs of) recognized entities in a given text. Extracting relationships between named entities from text is a major challenge in Information Extraction (IE), given the necessary language knowledge and the sophistication of the language processing techniques employed. At the same time, this task can contribute to the development of several areas, such as Question and Answer Systems, summarization, among others [7].

There is a growing interest in RE, mainly motivated by the exponential growth of the information available on the Web, which can make the task of searching and using such a large amount of data manually unfeasible. This context makes RE an even more complex and relevant research area [10]. In relation to the financial market, a domain addressed in this work, the news brings information about sectors of the economy, industrial policies, acquisitions, and company partnerships, among others. Automating the process of analyzing this data, in the form of financial reports, headlines, and corporate announcements can support personal and corporate economic decision making [30].

Thus, for example, it is possible to extract an acquisition relationship between entities of the organization type, where a certain organization (the first entity) was purchased (relationship) by another organization (second entity) [20]. Therefore, RE research among financial entities is the basis for the automatic extraction of financial information that can be used to assist with individual economic activities and national economic decision making [30].

Research on RE for the English language is at a more advanced stage than for Portuguese. While much work can be found in the literature on RE for English [4, 12, 15, 16, 22, 25], very few papers focus on RE for Portuguese. Unlike the English language, which has a larger number of datasets available to produce research, the Portuguese language lacks this type of resource; in fact, we were unable to find any large dataset prepared for this task. Considering that we had to build such a dataset for the validation of our experiments, it should be clear the difficulties for advancement in this area of research.

There are numerous techniques aimed at the area of IE and among them, deep learning techniques have recently stood out, mainly due to their ability to discover patterns implicit in the data. The literature has presented deep learning algorithms such as Transformers [23], Recurrent Neural Networks [15, 24] and Convolutional Neural Networks [28, 29] as good alternatives, as they have been efficiently applied in several sequential text processing tasks, including the RE task.

In this context, this work aims to propose an approach to the extraction of any semantic relationship between Named Entities (NEs) in the Financial Market domain for the Portuguese language. To achieve this goal, we use BERT, a pre-trained model with the transformers [23] architecture, so it is possible to harness the power of BERT and get sentence semantics without the use of enhanced feature selection or other external resources. For the validation of the experiment, we also created a dataset composed of 4,641 sentences and semantic relations annotated manually. Therefore, the contributions of this work are:

  • An entity relation extraction approach for the Portuguese language in the financial context, based on BERT.

  • A corpus with more than 4,500 manually annotated tuples formed from financial market news, available on GitHubFootnote 1.

This work is organized as follows. Section 2 analyzes the context of automated information processing for the area of Competitive Intelligence (CI) and brings the concept about the transformers BERT. Section 3 discusses related work. Section 4 provides a detailed description of the proposed solution. The experimental process and its results are detailed in Sects. 5 and 6, respectively. Finally, Sect. 7 presents our conclusions, as well as possibilities for future work.

2 Background

This section presents a theoretical basis on the main concepts related to the work: Financial market news processing as a problem addressed, presented in Subsect. 2.1; BERT, presented in Subsect. 2.2.

2.1 Competitive Intelligence and News Processing

Nowadays, the largest companies in the financial segment have a CI sector and, through it, information from different sources is strategically worked, allowing them to anticipate market trends and enabling the evolution of the business in relation to its competitors. This sector is usually formed by one or more professionals dedicated specifically to monitoring the competitors’ moves.

CI is a structured survey model for unstructured facts and data analysis to support the company’s decision-making in its strategic planning. Essentially, CI, involves the legal collection of information on competitors and the overall business environment. The knowledge gained from this information is then used to enhance the organization’s own competitiveness [26]. In times of competitiveness based on knowledge and innovation, CI allows companies to exercise pro-activity. The conclusions obtained in this process allow the company to know if it really remains competitive and if there is sustainability for its business model. CI can bring some advantages to companies that use it, such as: minimizing surprises from competitors, identifying opportunities and threats, obtaining relevant knowledge for the formulation of strategic planning, understanding the repercussions of their actions on the market, among others.

The process of capturing information through news still requires a lot of manual effort. Most of the information in the financial field appears in the form of free text, relying on manual processing of these data, far from keeping up with the growth rate of data, and unable to make full use of this information [30]. It often depends on a professional responsible for carefully reading numerous news about organizations to highlight possible market movements, and this professional can also keep this knowledge for himself. It is expected then that, with a system that automatically filters the relationships between financial market entities, the effort and time spent on those activities can be reduced. Another benefit achieved is that this same system can feed Business Intelligence (BI) systems and establish a historical database with market events. Thus, knowledge about market movements can be stored and organized independently of the particular CI analysts currently working in the company.

2.2 BERT

BERT is a large pre-trained language model proposed by Google [9] in 2018. Since then, BERT has achieved leading-edge results in various Natural Language Processing (NLP) tasks [14, 27]. BERT consists of a bidirectional multi-layer Transformer encoder based on the original implementation described in Vaswani et al. [23]. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. Residual connections are applied to both sub-layers.

The model trains by combining two tasks: Masked LM and Next Sentence Prediction. The first, Masked LM, involves masking 15% of the words in a given sentence, whose original value must then be predicted, considering the context of the sentence in which they appear. Out of these 15%, 80% is replaced by the symbol [MASK], 10% by a random word, and 10% is kept with the original word.

In the BERT training process, the model takes pairs of sentences as input and learns to predict whether the second sentence in the pair is the subsequent sentence in the original document. During training, 50% of the entries are pairs where the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence. The assumption is that the random phrase will be disconnected from the first phrase.

3 Related Work

With the advancement of information and communication technology in recent times, it is possible to store data from different sources and massively process it, which can facilitate research. Due to its feasibility of use in several areas such as the one explored in this study, RE has become the subject of numerous studies in the literature [2, 3, 13, 17, 30]. Those studies differ in regards to the method used, the language of the trained dataset, and also the theme of the dataset itself. Thus, this section presents some of these works, exploring some peculiarities.

Recently, deep learning algorithms have been targeted for application in a wide range of areas, such as NLP, image processing, bioinformatics, crowd simulation, and many others. In the area of NLP, for example, there is work on named entities recognition (NER) (e.g., [19]) and also the tool used in this study for NER, SpaCyFootnote 2, which is based on deep learning.

Several approaches have been proposed for extracting relationships from unstructured sources. For some languages, such as English, there is extensive research and literature [8]. To get around the problem of lack of memory for very large sequences in Convolutional Neural Networks (CNN), some authors [12, 15, 16] adopted an approach using Long short-term memory (LSTM), which use memory cells for neural networks. In this direction, the work by Qingqing Li [15] uses recurrent neural networks Bidirectional Long short-term memory (Bi-LSTM), which are an extension of the traditional LSTMs for multitasking model, and presents a version with an attention mechanism that considerably improves the results in all tested datasets.

More recent work in the literature [13, 17, 30] uses attention mechanisms to improve the performance of their neural network models. These mechanisms assist in the automatic information filtering step that helps to find the most appropriate sentence passage to describe the relationship between the Named Entities (NEs). In [30], the authors also implement a model based on the Recurrent Neural Network (RNN) Bi-GRU with an attention mechanism to focus on the most important assumptions of the sentences for the financial market. Despite having great importance, the financial domain is little explored in the literature. The authors also created a corpus by manually collecting 3000 sentence records from major news sites, which was used for entity recognition and extraction of relationships such as learning and training as a whole.

Approaches to RE for the Portuguese language are still very limited and are often based on rules such as [2, 3]. Such approaches applied simple heuristics that explore fragments of evidence of relationships between NEs in texts, comprising lexical, syntactic, and semantic analysis, types of entities, and information from external sources. Chaves [3] proposes an extraction system that deals with the NER task only for the Local category and its relations. The work by Batista et al. [1] proposes a distance-supervised RE approach between two entities in Portuguese-written Wikipedia articles.

More recently Collovini et al. [5] developed a CI approach using RE to collect and organize unstructured data in the Portuguese language. The authors proposed a joint framework for Named Entity Recognition and also their semantic relationships. For the Entity Recognition task, a neural network was used, and for the relation extraction, a model based in Conditional Random Fields (CRF) algorithm was used. This approach has shown good results on a small dataset, but it needs some additional lexical-semantic features of the sentences.

Most studies present RE solutions for texts in English, thus it is possible to find a greater number of datasets in that language. There are few datasets available in the Portuguese language, such as the Golden Collection HAREM. HAREM is a joint Portuguese language assessment event, organized by Linguateca [18]. The lack of this type of resource forces researchers to develop their own research corpora. In most cases, it is necessary to first create a set of sentences and manually annotate them when the classification is supervised in order to proceed with the RE task. In addition, the lack of public datasets also makes it difficult to compare with related work fairly, as well as requiring more time and effort on the part of the researcher.

It can also be noted that there is work related the RE task which already uses machine learning techniques for this purpose. However, although we have found some papers on the RE task, few of them are for the Portuguese language, and only one of them is related to the financial context [5]. Considering other languages, the work by Zhou [30] has similar goals to ours but proposes a joint model for NER and classifies the type of relationship between such named entities. Thus, it is clear that there is a gap in the literature for work that address such tasks using deep learning techniques and having Portuguese as the main language, especially in the economic-financial context as addressed in this work.

4 Relationship Extractor Model

Through the analysis of the records collected and used in the experiment, it is possible to note that the data from the financial field contain relationships composed of many tokens, in addition to the fact that the sentences can be long. There are many ways to express relationships, and the same semantics can be expressed in many ways. Thus, the problem we address can benefit from the use of an attention mechanism.

In this section, we present our model based on transformers [23] BERT. We used the pre-trained model in Portuguese based on the brWaC corpus provided by Souza et al. [21]. The implementations of this version of transformers are provided by HuggingfaceFootnote 3; we have added and trained the last layers of the model in order to extract a sub-sequence of the input text.

As in Fig. 1, the model architecture is composed of two main components, encoders and decoders. This approach provides a slightly different representation for the words as they are used in the sentences and their relationship to other words. This allows the model to understand the word itself and the context in which it is inserted.

In general terms, the sentences enter through the encoder, which has two layers: an attention mechanism which then feeds into a Feed Forward Neural Network; the output of the encoder is then sent to the decoder. The decoder also has both those layers, but between them there is additionally an attention mechanism that helps focusing on the relevant parts of the input sentence. Below, we describe in more detail some of these concepts.

Fig. 1.
figure 1

Model architecture

Full size image

Attention Mechanism: An attention mechanism can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. As per Eq. 1, the output is calculated as a weighted sum of values, where the weight assigned to each value is calculated by a query matching function with the corresponding key. In the BERT pre-trained model, Multi-Head Attention is used, which allows the model to serve together information from different representations in different positions.

$$\begin{aligned} MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W^O, \end{aligned}$$
(1)

where

$$\begin{aligned} head_i = Attention(QW_1^Q, KW_i^K, VW_i^V). \end{aligned}$$
(2)

Feed-Forward Layer: In addition to the attention mechanism, each of the layers in our encoders and decoders contains a fully connected feed-forward layer, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation between them which are calculated according to Eq. 3.

$$\begin{aligned} FFN(x) = max(0, xW_1 + b_1)W_2 + b_2. \end{aligned}$$
(3)

5 Experiments

This section aims to evaluate the performance of the proposed model in experiments using the corpus for the financial domain. The proposed approach follows the Knowledge Discovery in Databases (KDD) process created by Fayyad [11]. This process contains five steps ranging from data collection and creation, as described in Sect. 5.1, to the evaluation of results. The next sections aim to explain how each of these steps was employed in this work.

5.1 Selection

As in the selection step it is necessary to indicate which data will be used during the experiments for the RE [11] task, we start by searching for corpora. However, there was no evidence of open datasets in the context of extracting relationships in the financial area in the Portuguese language. Therefore, for this work, a corpus with 4,641 manually annotated tuples was created.

More than 1,500 financial market news from 2018 was provided by a partner company. Those news articles were collected from various media outlets, such as financial market websites, newspapers, and corporate balance sheets. Those articles gave rise to more than 10,000 sentences that were analyzed manually.

Another 7,097 tweets with dates from January 2021 were also collected. Those tweets were selected from 10 communication media users focused on the financial market and the economy in general. The selected users were: infomoney, EstadaoEconomia, UOLEconomia, g1economia, OGlobo_Economia, folha_mercado, InvestingBrasil, leiamoneytimes, valoreconomico, br_economico. Sentences that contained co-references were removed because handling them would require additional processing.

5.2 Pre-processing

The pre-processing step is intended to apply cleaning, correction, or removal procedures for inconsistent or missing data. In this work, we carried out a manual spell-checking process for each sentence, using Excel’s spell-checker. Acronyms have also been expanded, such as BC being replaced by Banco Central.

Standardization can be done manually, but in a real work scenario, this task becomes massive and can be automated by creating a base of named entities and their acronyms. Thus, it is possible to design a process that validates the acronyms contained in the sentence and replaces them with their extensions or even an approach that focuses only on some specific entities informed by the CI analyst.

In this same phase, the named entities are also identified, through a single NER tool, called SpaCyFootnote 4, ensuring that the same criteria were used for all sentences.

The data cleaning process was done automatically through a script available at Github (See Footnote 1) that removes special characters and acronyms that follow the description itself. Phrases that have only one recognized named entity or none at all are removed from the dataset. At the end of this cleaning step, just over 5,000 sentences were selected for the next step.

The named entities in question are those related to the person, place, and organization categories, as they were considered the most relevant to the domain of organizations [6]. The focal point is information about organizations as well as their relationships with other organizations, people, and places.

5.3 Transformation

The data transformation or formatting step analyzes the data obtained from the previous step and reorganizes them in a specific way so that they can be analyzed and interpreted by the algorithm in the next step. In the case of this work, after identifying the Named Entities in the previous phase, the first step of the transformation step is to combine all entities present in the sentence in order to create a tuple (sentence, entity, entity) for each combination. It is important to emphasize that sentences with more than two entities generate more than one tuple for the same sentence, as it is necessary to create a pair for each combination of entities.

Then, the semantic relationships between the highlighted named entities are manually annotated. In this case, only positive tuples are considered and negative combinations are discarded. A tuple is considered positive when there is any semantic relationship between two named entities from the categories defined in Sect. 5.2. Finally, the two named entities are concatenated at the end of the sentence.

Finally, after removing sentences without the semantic relationship between the named entities, the corpus was left with a total of 4,641 records, containing a sentence, its entities, and the semantic relationship between them. Table 1 exemplifies some records that have combinations of entities that can generate more than one tuple per sentence and also examples of tuples with positive annotations that contain relationships between named entities of the organization type.

Table 1. Examples of positive tuples with annotations showing the relationships between named entities. The entities to be evaluated appear in bold font and the text representing the semantic relationship between them is underlined.
Full size table

As sentences are naturally composed of words and characters, the transformation step of the methodology also includes the transformation of tokens into numerical representations by the BERT encoder. As BERT is a pre-trained model that expects input data in a specific format, this step also adds the special tokens [CLS] and [SEP] properly encoded in each sentence, finishing the transformation of the natural language sentence into the input for the extractor model. The special token [SEP] is used to mark the end of a sentence or the separation between two sentences. The special token [CLS] is used to mark the beginning of our text.

5.4 Mining

The mining step contemplates the prediction task, in which a behavioral pattern of the data is searched in order to predict the behavior of a future entity [11]. The corpus was randomly divided into two parts, 90% of which were used to train the model and 10% for testing; This proportion rate was chosen so that it would be possible to provide more samples for training the two models used in the approach. The first set was used so that the algorithm could search for the particular pattern in the data related to the relationship label. Thus, after the training stage, from which the model is able to recognize this pattern, it is possible to apply it to the test set data.

The adjustment of the BERT hyper-parameters used was done using the combination of all the values indicated by Jacob Devlin when he proposed the new BERT language model in [9]. In that work, Jacob Devlin used most hyper-parameters with default values, except for batch size, learning rate, and a number of training epochs. We also added two more test values, referring to the optimizer parameter. The dropout rate was always kept at 0.1. Thus, the values analyzed for this task were:

  • Batch Size: 16, 32;

  • Learning Rate: 2e−5, 3e−5, 5e−5;

  • Epochs: 2, 3, 4, 5;

  • Optimizer: AdamW;

In the end, we ran a total of 24 experiments with all possible combinations of the parameters described above. After analyzing the results, the model that presented the best results was selected according to the parameters in Table 2.

Table 2. Combination of hyper-parameters that presented the best results.
Full size table

5.5 Evaluation

The last step of KDD methodology presented in this section aims to evaluate the performance of the model we built. The experimental evaluation was carried out by applying, in the test data, the model based on a deep neural network built in the learning stage based on the defined parameters. For the implementation of the neural network, the TensorFlow KerasFootnote 5 library was used to adapt the last layer of BERT.

The evaluation metric used in this work is the Jaccard Similarity Coefficient or simply Jaccard Coefficient. Jaccard’s coefficient measures the similarity between two sets and is defined as the size of the intersection divided by the size of the union of the sample sets. We chose this metric as it can present a more realistic view of the model’s efficiency when recognizing the tokens present in the relationship. If we only considered completely correct extractions, the model would be penalized by not extracting just a token for example. Equation 4 shows how the coefficient is calculated and Table 3 an example of evaluation.

$$\begin{aligned} jaccard(A, B) = \frac{|A \bigcap B|}{ |A\bigcup B|} \end{aligned}$$
(4)

For the above two relations, we get a Jaccard similarity of \(3 / (3 + 1) = 0.75\), which is the size of the intersection of the set divided by the total size of the set.

Table 3. Evaluation example for extracting relationships between Named Entities. The evaluated named entities are in bold font.
Full size table

6 Results and Discussion

After the training step, the model was applied to the test data set. In this evaluation step, the model obtained good results, indicating an overall Jaccard metric of 73.5% for all the prediction cases. In terms of correctly extracted relationships, it was possible to extract 212 completely correct relationships from a total of 465 samples available for testing; that is when the extracted relationship is exactly the same as that manually annotated. We consider a relationship partially correct when the extracted relationship reaches a Jaccard score of at least 50%. As for partially correct relationships, our approach was able to extract 176 relationships. In only 40 occurrences the model was not able to extract any token present in the relation to be extracted. These results are summarized in Table 4.

Table 4. Results obtained by the proposed model.
Full size table

In general, the extracted relationships express information relevant to the CI focus, for example, partnership formation relationships between Organizations; financial investment relationships between Organizations; relations between People and Organizations; entry of organizations into the market in certain locations, among others. Table 5 illustrates instances of extracted relationships that are completely correct. From this RE approach, it is possible to create a historical knowledge base, easily accessible and interpreted by CI analysts. Thus, it is possible to provide information so that everyone can extract value quickly and clearly.

Table 5. Completely correct relationships extracted by the proposed model.
Full size table
Fig. 2.
figure 2

Average score grouped by the number of words contained in the relation to be extracted.

Full size image

We claim that the model was able to respond well to the complexity of sentence length and the size of the relation to be extracted. It was able to recognize patterns and indicate which tokens belong to the semantic relations contained in the same sentence in the financial domain. Figure 2 illustrates the average score according to sentence length.

It can also be inferred that the model has greater difficulty when trying to extract relations composed of only one token, which ends up penalizing the model and its performance. This particular type of relationship is the third largest group within the test dataset, and in most cases there are only two score possibilities, 0 or 1. On these prediction errors, the model had more difficulty in inferring composite relationships just by a preposition (e.g. da, de, na, no) as shown in Table 6.

Table 6. Examples of relationships not extracted by the model. In bold and underlined, the semantic relationship to be extracted.
Full size table

7 Conclusion and Future Work

In this work, we provide an approach to the RE task and a manually annotated corpus from news provided by a market intelligence company and also news collected from Twitter on the subject of Competitive Intelligence. More than 1,500 news about the financial market and more than 7,000 tweets were selected to build a knowledge base. The model proved to be able to recognize semantic relationships and is therefore useful for discovering events and facts related to the financial market.

From the section on related work, it is possible to note that there is little research on techniques for extracting relationships between named entities for the financial domain in Portuguese. This domain lacks practical solutions, in part because of the large amount of information in the financial area, so manual analysis becomes difficult to meet the needs and make full use of this information. Due to this lack of approaches to the RE task, it is also difficult to present comparative results at this stage, since there is still no dataset available for comparison.

As the main contributions of this work, we can mention the development of a Relation Extraction model between named entities based on BERT, which replaces the explicit linguistic resources, required by other methods, such as rule-based approaches or that use Parts-Of-Speech (PoS). This approach becomes much simpler as it only needs the phrase information and the pair of concatenated entities. Thus, it allows more than one entry to be sent, as a sentence can have N pairs of named entities. Therefore, the approach adopted allows us to infer that the sentence and the pair of entities are sent separately. Another contribution of this work is found in the development of a large corpus related to the financial market, with manually annotated text from tweets and news provided by CI analysts to support decision making. The results demonstrate that the approach used has reasonable results, reaching a Jaccard score of 73.5%. This score is interesting since the model was able to extract relationships of different sizes. As shown in Sect. 6 it was more penalized when extracting relations from just a token, usually formed by prepositions.

As future work, a method in pipeline format containing two models for extracting relationships between named entities will be developed. The first model will determine whether an entry containing the phrase and entity pair has a semantic relationship between the named entities to be tested. The output of this model, when it indicates that there is a semantic relationship, will be the input of the model here presented, which will highlight the parts of the sentences that represent or describe the relationship between the selected named entities.