|
| 1 | +--- |
| 2 | +layout: model |
| 3 | +title: Clinical Deidentification Pipeline (Document Wise) |
| 4 | +author: John Snow Labs |
| 5 | +name: clinical_deidentification_docwise_wip |
| 6 | +date: 2024-10-18 |
| 7 | +tags: [deidentification, deid, en, licensed, clinical, pipeline, docwise] |
| 8 | +task: [De-identification, Pipeline Healthcare] |
| 9 | +language: en |
| 10 | +edition: Healthcare NLP 5.5.0 |
| 11 | +spark_version: 3.4 |
| 12 | +supported: true |
| 13 | +annotator: PipelineModel |
| 14 | +article_header: |
| 15 | + type: cover |
| 16 | +use_language_switcher: "Python-Scala-Java" |
| 17 | +--- |
| 18 | + |
| 19 | +## Description |
| 20 | + |
| 21 | +This pipeline can be used to deidentify PHI information from medical texts. The PHI information will be masked and obfuscated in the resulting text. |
| 22 | +The pipeline can mask and obfuscate `LOCATION`, `CONTACT`, `PROFESSION`, `NAME`, `DATE`, `ID`, `AGE`, `MEDICALRECORD`, `ORGANIZATION`, `HEALTHPLAN`, `DOCTOR`, `USERNAME`, |
| 23 | +`LOCATION-OTHER`, `URL`, `DEVICE`, `CITY`, `ZIP`, `STATE`, `PATIENT`, `COUNTRY`, `STREET`, `PHONE`, `HOSPITAL`, `EMAIL`, `IDNUM`, `BIOID`, `FAX`, `LOCATION_OTHER`, `DLN`, |
| 24 | +`SSN`, `ACCOUNT`, `PLATE`, `VIN`, `LICENSE`, `IP` entities. |
| 25 | + |
| 26 | +{:.btn-box} |
| 27 | +<button class="button button-orange" disabled>Live Demo</button> |
| 28 | +<button class="button button-orange" disabled>Open in Colab</button> |
| 29 | +[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/clinical/models/clinical_deidentification_docwise_wip_en_5.5.0_3.4_1729263282472.zip){:.button.button-orange.button-orange-trans.arr.button-icon.hidden} |
| 30 | +[Copy S3 URI](s3://auxdata.johnsnowlabs.com/clinical/models/clinical_deidentification_docwise_wip_en_5.5.0_3.4_1729263282472.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3} |
| 31 | + |
| 32 | +## How to use |
| 33 | + |
| 34 | + |
| 35 | + |
| 36 | +<div class="tabs-box" markdown="1"> |
| 37 | +{% include programmingLanguageSelectScalaPythonNLU.html %} |
| 38 | +```python |
| 39 | + |
| 40 | +from sparknlp.pretrained import PretrainedPipeline |
| 41 | + |
| 42 | +deid_pipeline = PretrainedPipeline("clinical_deidentification_docwise_wip", "en", "clinical/models") |
| 43 | + |
| 44 | +text = """Dr. John Lee, from Royal Medical Clinic in Chicago, attended to the patient on 11/05/2024. |
| 45 | +The patient’s medical record number is 56467890. |
| 46 | +The patient, Emma Wilson, is 50 years old, her Contact number: 444-456-7890 .""" |
| 47 | + |
| 48 | +deid_result = deid_pipeline.fullAnnotate(text) |
| 49 | + |
| 50 | +print(''.join([i.result for i in deid_result['mask_entity']])) |
| 51 | +print(''.join([i.result for i in deid_result['obfuscated']])) |
| 52 | + |
| 53 | + |
| 54 | +``` |
| 55 | +```scala |
| 56 | + |
| 57 | +import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline |
| 58 | + |
| 59 | +val deid_pipeline = PretrainedPipeline("clinical_deidentification_docwise_wip", "en", "clinical/models") |
| 60 | + |
| 61 | +val text = """Dr. John Lee, from Royal Medical Clinic in Chicago, attended to the patient on 11/05/2024. |
| 62 | +The patient’s medical record number is 56467890. |
| 63 | +The patient, Emma Wilson, is 50 years old, her Contact number: 444-456-7890 .""" |
| 64 | + |
| 65 | +val deid_result = deid_pipeline.fullAnnotate(text) |
| 66 | + |
| 67 | +println(deid_result("mask_entity").map(_("result").toString).mkString("")) |
| 68 | +println(deid_result("obfuscated").map(_("result").toString).mkString("")) |
| 69 | + |
| 70 | + |
| 71 | +``` |
| 72 | +</div> |
| 73 | + |
| 74 | +## Results |
| 75 | + |
| 76 | +```bash |
| 77 | + |
| 78 | +Masked with entity labels |
| 79 | +------------------------------ |
| 80 | +Dr. <DOCTOR>, from <HOSPITAL> in <CITY>, attended to the patient on <DATE>. |
| 81 | +The patient’s medical record number is <MEDICALRECORD> |
| 82 | +patient, <PATIENT>, is <AGE> years old, her Contact number: <PHONE> . |
| 83 | + |
| 84 | +Obfuscated |
| 85 | +------------------------------ |
| 86 | +Dr. Edwardo Graft, from MCBRIDE ORTHOPEDIC HOSPITAL in CLAMART, attended to the patient on 14/06/2024. |
| 87 | +The patient’s medical record number is 78295621. |
| 88 | +The patient, Nathaneil Bakes, is 43 years old, her Contact number: 308-657-8469 . |
| 89 | + |
| 90 | +``` |
| 91 | + |
| 92 | +{:.model-param} |
| 93 | +## Model Information |
| 94 | + |
| 95 | +{:.table-model} |
| 96 | +|---|---| |
| 97 | +|Model Name:|clinical_deidentification_docwise_wip| |
| 98 | +|Type:|pipeline| |
| 99 | +|Compatibility:|Healthcare NLP 5.5.0+| |
| 100 | +|License:|Licensed| |
| 101 | +|Edition:|Official| |
| 102 | +|Language:|en| |
| 103 | +|Size:|1.8 GB| |
| 104 | + |
| 105 | +## Included Models |
| 106 | + |
| 107 | +- DocumentAssembler |
| 108 | +- InternalDocumentSplitter |
| 109 | +- TokenizerModel |
| 110 | +- WordEmbeddingsModel |
| 111 | +- MedicalNerModel |
| 112 | +- MedicalNerModel |
| 113 | +- MedicalNerModel |
| 114 | +- NerConverterInternalModel |
| 115 | +- NerConverterInternalModel |
| 116 | +- NerConverterInternalModel |
| 117 | +- ChunkMergeModel |
| 118 | +- ContextualParserModel |
| 119 | +- ContextualParserModel |
| 120 | +- ContextualParserModel |
| 121 | +- ContextualParserModel |
| 122 | +- ContextualParserModel |
| 123 | +- ContextualParserModel |
| 124 | +- TextMatcherInternalModel |
| 125 | +- TextMatcherInternalModel |
| 126 | +- TextMatcherInternalModel |
| 127 | +- ContextualParserModel |
| 128 | +- RegexMatcherInternalModel |
| 129 | +- ContextualParserModel |
| 130 | +- ContextualParserModel |
| 131 | +- ContextualParserModel |
| 132 | +- RegexMatcherInternalModel |
| 133 | +- RegexMatcherInternalModel |
| 134 | +- RegexMatcherInternalModel |
| 135 | +- TextMatcherInternalModel |
| 136 | +- ChunkMergeModel |
| 137 | +- ChunkMergeModel |
| 138 | +- LightDeIdentification |
| 139 | +- LightDeIdentification |
0 commit comments