NER pipeline: Inconsistent entity grouping

[dav009]

🐛 Bug

Information

"mrm8488/bert-spanish-cased-finetuned-ner"

Language I am using the model on (English, Chinese ...): Spanish

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. create a ner pipeline
2. pass flag grouped_entities
3. entities are not grouped as expected see sample below

NER_MODEL = "mrm8488/bert-spanish-cased-finetuned-ner"
nlp_ner = pipeline("ner", model=NER_MODEL,
                   grouped_entities=True,
                   tokenizer=(NER_MODEL, {"use_fast": False}))

t = """Consuelo Araújo Noguera, ministra de cultura del presidente Andrés Pastrana (1998.2002) fue asesinada por las Farc luego de haber permanecido secuestrada por algunos meses."""
ner(t)
>>> 
[ {'entity_group': 'B-PER', 'score': 0.901019960641861, 'word': 'Consuelo'}, 
 {'entity_group': 'I-PER', 'score': 0.9990904808044434, 'word': 'Araújo Noguera'}, 
 {'entity_group': 'B-PER', 'score': 0.9998136162757874, 'word': 'Andrés'}, 
 {'entity_group': 'I-PER', 'score': 0.9996985991795858, 'word': 'Pastrana'}, 
 {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Far'}]

Expected behavior

Inconsistent grouping

I expect the first two items of the given sample( B-PER, and I-PER) to be grouped. As they are contiguous tokens and correspond to a single entity spot. It seems the current code does not take into account B and I tokens.

expected output:

 {'entity_group': 'I-PER', 'score': 0.9990904808044434, 'word': ' Consuelo Araújo Noguera'}, 
 {'entity_group': 'I-PER', 'score': 0.9998136162757874, 'word': 'Andrés Pastrana'}, 
 {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Farc'}]

Lost tokens?

for the same input, passing grouped_entities=False generates the following output:

[
{'word': 'Cons', 'score': 0.9994944930076599, 'entity': 'B-PER', 'index': 1},
{'word': '##uelo', 'score': 0.802545428276062, 'entity': 'B-PER', 'index': 2}, 
{'word': 'Ara', 'score': 0.9993102550506592, 'entity': 'I-PER', 'index': 3}, 
{'word': '##új', 'score': 0.9993743896484375, 'entity': 'I-PER', 'index': 4}, 
{'word': '##o', 'score': 0.9992871880531311, 'entity': 'I-PER', 'index': 5}, 
{'word': 'No', 'score': 0.9993029236793518, 'entity': 'I-PER', 'index': 6}, 
{'word': '##guera', 'score': 0.9981776475906372, 'entity': 'I-PER', 'index': 7}, 
{'word': 'Andrés', 'score': 0.9998136162757874, 'entity': 'B-PER', 'index': 15}, 
{'word': 'Pas', 'score': 0.999740719795227, 'entity': 'I-PER', 'index': 16}, 
{'word': '##tran', 'score': 0.9997414350509644, 'entity': 'I-PER', 'index': 17}, 
{'word': '##a', 'score': 0.9996136426925659, 'entity': 'I-PER', 'index': 18}, 
{'word': 'Far', 'score': 0.9989739060401917, 'entity': 'B-ORG', 'index': 28}, 
{'word': '##c', 'score': 0.7188423275947571, 'entity': 'I-ORG', 'index': 29}]

when using grouped_entities the last entity word (##c) got lost, it is not even considered as a different group

{'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Far'}]

Environment info

• transformers version: 2.11.0
• Platform: OSX
• Python version: 3.7
• PyTorch version (GPU?): 1.5.0
• Tensorflow version (GPU?):
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

[enzoampil]

@dav009 Thanks for posting this issue!

1. Inconsistent grouping - correct that B and I tokens are not yet considered. Will have to include this in a new PR.
2. Lost tokens - the skipped tokens are those with an entity type found in the ignore_labels argument for TokenClassificationPipeline, which is set as ["O"] by default. If you don't want to skip any token, you can just set ignore_labels=[].

I'm happy to work on 1 within the next week or so since I've already been planning to apply this fix.

[dav009]

@enzoampil 👋 thanks for your prompt answer

Lost tokens - the skipped tokens are those with an entity type found in the ignore_labels argument for TokenClassificationPipeline, which is set as ["O"] by default. If you don't want to skip any token, you can just set ignore_labels=[].

in the given sample, the missing entity is not tagged as O :

• ##c is tagged as I-ORG in (grouped_entities =False)
  {'word': '##c', 'score': 0.7188423275947571, 'entity': 'I-ORG', 'index': 29}]

however it did not get included in the grouping results (grouped_entities =True)

[enzoampil]

@dav009 Understand now! Thanks for clarifying. Yes, it does seem to be related to the I and B issue. Think can handle this in the same PR.

[sudharsan2020]

@dav009 I handled a similar scenario for grouping the Begin and Info tags.
The below code helps to merge the tokens between Begin and Info tags. Please adapt to your use

`def group_entities(self, prediction_results_list: List[Dict] ) -> List[RecordDataResponse]:
final_prediction_list = []

    # Group the prediction list by the last 3 characters of the tag
    # and group the results appropriately
    # B-PER-TAG -> TAG
    # B-PER -> PER
    tmp_dict = defaultdict(list)
    added_index = 0
    prev_index = 0
    for index, entity in enumerate(prediction_results_list):
        try:
            if entity['entity_group'].startswith("B") and \
                    prediction_results_list[index + 1]['entity_group'].startswith("I"):
                tmp_dict[index].append(entity)
                added_index = index
            elif entity['entity_group'].startswith("I"):
                if (1 == abs(added_index - index)) or (1 == abs(prev_index - index)):
                    tmp_dict[added_index].append(entity)
                    prev_index = index
            else:
                tmp_dict[index].append(entity)
        except IndexError:
            tmp_dict[index].append(entity)

    # Flatten the sub-lists
    final_grouped_list = list(map(list, map(itertools.chain, tmp_dict.values())))

    for entity_group_list in final_grouped_list:

        # Get the unique number of entities per list
        _entity_count = len(
            set(
                [
                    prediction_input["entity_group"]
                    for prediction_input in entity_group_list
                ]
            )
        )

        if entity_group_list:
            if len(entity_group_list) > 1:

                # Get the tag name
                tag_name = str(entity_group_list[0]["entity_group"][-3:])

                # Join the entities
                entity_value = " ".join(
                    [
                        prediction_input["word"]
                        for prediction_input in entity_group_list
                    ]
                )

                # Remove duplicate names
                _temp_entities = entity_value.split()
                entity_value = " ".join(
                    sorted(set(_temp_entities), key=_temp_entities.index)
                )

                # Compute the average of confidence scores
                mean_score = np.mean(
                    [
                        prediction_input["score"]
                        for prediction_input in entity_group_list
                    ]
                )

                # Frame the entities and ensure name is atleast has more than 1 character
                if len(entity_value) > 1:
                    final_prediction_list.append(
                        RecordDataResponse(
                            entity_group=tag_name,
                            score=mean_score,
                            word=entity_value,
                        )
                    )
            else:
                [
                    final_prediction_list.append(
                        RecordDataResponse(
                            entity_group=entity_group["entity_group"][-3:],
                            score=entity_group["score"],
                            word=entity_group["word"],
                        )
                    )
                    for entity_group in entity_group_list if len(re.sub(r"(?i)[^-0-9a-z\\s.,]+", "", entity_group["word"])) > 1
                ]

    # Sort the by the list by confidence score and return in descending order
    return sorted(final_prediction_list, key=lambda x: x.score, reverse=True)

`

The code is invoked from the pipeline:

   `prediction_results_list = [
        prediction
        for prediction_input in prediction_input_list
        for prediction in self.model_prediction_pipeline(prediction_input)
        if prediction
        and prediction["word"] not in self.stop_list
    ]

    # Return the predictions
    return (
        self.group_entities(prediction_results_list)
        if prediction_results_list
        else []
    )`

[enzoampil]

@dav009 Opened a PR (above) that should resolve this 😄

[sudharsan2020]

@enzoampil Just curious to know if your PR can handle the merging of multiple entities.

entities_list = [ {"word": "Patient", "score": 0.9977793097496033, "entity_group": "B-PER-TAG"}, {"word": "Name", "score": 0.9968074560165405, "entity_group": "I-PER-TAG"}, {"word": "Cecil", "score": 0.9995920658111572, "entity_group": "B-PER"}, {"word": "D . Thomas", "score": 0.9938908666372299, "entity_group": "I-PER"}, {"word": "Thomas", "score": 0.9993066191673279, "entity_group": "B-PER"} ]

In this case, I would expect the below output after the entities are grouped:
[ {"word": "Patient Name", "score": 0.9977793097496033, "entity_group": "PER-TAG"}, {"word": "Cecil D . Thomas", "score": 0.9995920658111572, "entity_group": "PER"}, {"word": "Thomas", "score": 0.9993066191673279, "entity_group": "PER"} ]

[dav009]

@enzoampil gonna check it out.

maybe part of another issue but do you get word fields containing ## is that expected?

[enzoampil]

@sudharsan2020 Setting grouped_entities=True should work for your example under the new PR, since similar entities w/ different prefixes are now grouped (e.g. "I-PER" and "B-PER") 😄

[enzoampil]

@dav009 This is even after grouping correct? I suspect this is possible when word pieces have different core entity types (e.g. ORG vs PER).

Can you give an example?

[Nighthyst]

Hello, I think @dav009 is refering to this :

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
from transformers import TokenClassificationPipeline

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

nlp = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=True
)

sequence = "In addition , the Blabla Group has completed the acquisition of ISO / TS16949 certification ."

res = nlp(sequence)
print(res)

I have this as a result :

[{'entity_group': 'I-ORG', 'score': 0.9988919496536255, 'word': 'Blabla Group'}, {'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO'}, {'entity_group': 'I-ORG', 'score': 0.6591967344284058, 'word': 'T'}, {'entity_group': 'I-MISC', 'score': 0.5822997689247131, 'word': '##S16'}, {'entity_group': 'I-MISC', 'score': 0.5067382454872131, 'word': '##9'}]

Some word fields still have ## in it. I have just installed transformers right now (version 2.11.0) with a pip install command then paste the pipelines.py fixed in my transformers folder.

[enzoampil]

@Nighthyst can you share the result when grouped_entities=False?

[Nighthyst]

Yes, here is a comparison of the resultats with grouped_entities=False or when grouped_entities=True :

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch
from transformers import TokenClassificationPipeline

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

nlp_not_grouped = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=False
)

nlp_grouped = TokenClassificationPipeline(
    model=model,
    tokenizer=tokenizer,
    grouped_entities=True
)

seq1 = "In addition , the Blabla Group has completed the acquisition of ISO / TS16949 certification ."
seq2 = "Directors and certain categories of personnel , who are all included in a regularly updated list"\
", must disclose any trades they carry out in Faurecia"
seq3 = "Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very" \
           "close to the Manhattan Bridge."

sequences = [seq1, seq2, seq3]

for i, seq in enumerate(sequences):
    ngrouped, grouped = nlp_not_grouped(seq), nlp_grouped(seq)
    print(f"===================== sentence n°{i+1}")
    print("---Not grouped entities---")
    print(ngrouped)
    print("---Grouped entities---")
    print(grouped)

This is the results:

===================== sentence n°1
---Not grouped entities---
[{'word': 'B', 'score': 0.9997261762619019, 'entity': 'I-ORG', 'index': 5}, 
{'word': '##la', 'score': 0.997683048248291, 'entity': 'I-ORG', 'index': 6}, 
{'word': '##bla', 'score': 0.99888014793396, 'entity': 'I-ORG', 'index': 7}, 
{'word': 'Group', 'score': 0.9992784261703491, 'entity': 'I-ORG', 'index': 8}, 
{'word': 'ISO', 'score': 0.9711909890174866, 'entity': 'I-MISC', 'index': 14},
{'word': 'T', 'score': 0.6591967344284058, 'entity': 'I-ORG', 'index': 16}, 
{'word': '##S', 'score': 0.658642053604126, 'entity': 'I-MISC', 'index': 17}, 
{'word': '##16', 'score': 0.5059574842453003, 'entity': 'I-MISC', 'index': 18}, 
{'word': '##9', 'score': 0.5067382454872131, 'entity': 'I-MISC', 'index': 21}]
---Grouped entities---
[{'entity_group': 'I-ORG', 'score': 0.9988919496536255, 'word': 'Blabla Group'}, 
{'entity_group': 'I-MISC', 'score': 0.9711909890174866, 'word': 'ISO'}, 
{'entity_group': 'I-ORG', 'score': 0.6591967344284058, 'word': 'T'}, 
{'entity_group': 'I-MISC', 'score': 0.5822997689247131, 'word': '##S16'}, 
{'entity_group': 'I-MISC', 'score': 0.5067382454872131, 'word': '##9'}]
===================== sentence n°2
---Not grouped entities---
[{'word': 'F', 'score': 0.6292181611061096, 'entity': 'I-ORG', 'index': 27}, 
{'word': '##au', 'score': 0.7241453528404236, 'entity': 'I-LOC', 'index': 28}, 
{'word': '##re', 'score': 0.49484530091285706, 'entity': 'I-LOC', 'index': 29}, 
{'word': '##cia', 'score': 0.6472106575965881, 'entity': 'I-LOC', 'index': 30}]
---Grouped entities---
[{'entity_group': 'I-ORG', 'score': 0.6292181611061096, 'word': 'F'}, 
{'entity_group': 'I-LOC', 'score': 0.6220671037832896, 'word': '##aurecia'}]
===================== sentence n°3
---Not grouped entities---
[{'word': 'Hu', 'score': 0.9995108246803284, 'entity': 'I-ORG', 'index': 1}, 
{'word': '##gging', 'score': 0.989597499370575, 'entity': 'I-ORG', 'index': 2}, 
{'word': 'Face', 'score': 0.9979704022407532, 'entity': 'I-ORG', 'index': 3}, 
{'word': 'Inc', 'score': 0.9993758797645569, 'entity': 'I-ORG', 'index': 4}, 
{'word': 'New', 'score': 0.9993405938148499, 'entity': 'I-LOC', 'index': 11}, 
{'word': 'York', 'score': 0.9991927742958069, 'entity': 'I-LOC', 'index': 12}, 
{'word': 'City', 'score': 0.9993411302566528, 'entity': 'I-LOC', 'index': 13}, 
{'word': 'D', 'score': 0.986336350440979, 'entity': 'I-LOC', 'index': 19}, 
{'word': '##UM', 'score': 0.9396238923072815, 'entity': 'I-LOC', 'index': 20}, 
{'word': '##BO', 'score': 0.9121386408805847, 'entity': 'I-LOC', 'index': 21}, 
{'word': 'Manhattan', 'score': 0.9839190244674683, 'entity': 'I-LOC', 'index': 29}, 
{'word': 'Bridge', 'score': 0.9924242496490479, 'entity': 'I-LOC', 'index': 30}]
---Grouped entities---
[{'entity_group': 'I-ORG', 'score': 0.9966136515140533, 'word': 'Hugging Face Inc'}, 
{'entity_group': 'I-LOC', 'score': 0.9992914994557699, 'word': 'New York City'}, 
{'entity_group': 'I-LOC', 'score': 0.9460329612096151, 'word': 'DUMBO'}, 
{'entity_group': 'I-LOC', 'score': 0.9881716370582581, 'word': 'Manhattan Bridge'}]

Everything is fine for seq3 but seq1 and seq2 have the issue.

[enzoampil]

@Nighthyst I see, you're bringing up a different issue now. This is the case where the entity type of a word's word piece, is different from other word pieces.

A fix I can apply here is to automatically group word pieces together regardless of entity type. I can apply this to a new PR after merging this existing one.

[dav009]

@Nighthyst @enzoampil indeed that's exactly the other issue I came accross. Thanks for digging a sample for it.