Improving textual medication extraction using combined conditional random fields and rule-based systems

Objective In the i2b2 Medication Extraction Challenge, medication names together with details of their administration were to be extracted from medical discharge summaries. Design The task of the challenge was decomposed into three pipelined components: named entity identification, context-aware filtering and relation extraction. For named entity identification, first a rule-based (FIB) method that was used in our overall fifth place-ranked solution at the challenge was investigated. Second, a conditional random fields (CRF) approach is presented for named entity identification (NEI) developed after the completion of the challenge. The CRF models are trained on the 17 ground truth documents, the output of the rule-based NEI component on all documents, a larger but potentially inaccurate training dataset. For both NEI approaches their effect on relation extraction performance was investigated. The filtering and relation extraction components are both rule-based. Measurements In addition to the official entry level evaluation of the challenge, entity level analysis is also provided. Results On the test data an entry level F-1-score of 80% was achieved for exact matching and 81% for inexact matching with the RB-NEI component. The CAF produces a significantly weaker result, but CRF outperforms the rule-based model with 81% exact and 82% inexact F-1-score (p<0.02). Conclusion This study shows that a simple rule-based method is on a par with more complicated machine learners; CAF models can benefit from the addition of the potentially inaccurate training data, when only very few training documents are available. Such training data could be generated using the outputs of rule-based methods.