Part of speech plays a very major role in NLP task as it is important to know how a word is used in every sentence. POS tagging is used mostly for Keyword Extractions, phrase extractions, Named Entity Recognition, etc. Before going further on POS tagging, I am assuming that you all know about the part of speech as we all have studied grammar during school. Didn’t we? But anyways let me give a brief explanation on it!
There are eight main Parts of Speech: Nouns(naming word), Pronouns(replaces a noun), Adjectives(describing word), Verbs(action word), Adverbs(describes a verb), Prepositions(shows relationships), Conjunctions(joining word) and Interjections(Expressive word). Most of it is further divided into sub-parts. A noun is divided into Proper Nouns, Common Nouns, Concrete Nouns etc.
Reminds you of school days?? Okay now let’s start with Hindi Part of Speech Tagging.
Hindi Part of Speech Tagging is something that people are still doing research on as we have various techniques and libraries available for English Text and rarely for Hindi Text. [1] Manish and Pushpak researched on Hindi POS using a simple HMM-based POS tagger with an accuracy of 93.12%. while [2]Nisheeth Joshi, Hemant Darbari and Iti Mathur also researched on Hindi POS using Hidden Markov Model with the frequency count of two tags seen together in the corpus divided by the frequency count of the previous tag seen independently in the corpus. [3] S Phani Kumar Gadde, Meher Vijay Yeleti used CRF based tagger and Brants TnT (Brants, 2000), an HMM-based tagger for Hindi POS Tag where they got an accuracy of 94.21%.
So today we’ll be using TNT tagger to tag Hindi words!
Let’s say we have a text to tag
text = “इराक के विदेश मंत्री ने अमरीका के उस प्रस्ताव का मजाक उड़ाया है , जिसमें अमरीका ने संयुक्त राष्ट्र के प्रतिबंधों को इराकी नागरिकों के लिए कम हानिकारक बनाने के लिए कहा है ।”Let’s use the already tagged data which is given in nltk to train the data.
from nltk.tag import tnt
from nltk.corpus import indiantrain_data = indian.tagged_sents(‘hindi.pos’)
tnt_pos_tagger = tnt.TnT()
tnt_pos_tagger.train(train_data)
Let’s Tag the text now!
tagged_words = (tnt_pos_tagger.tag(nltk.word_tokenize(text)))
print(tagged_words)[OUTPUT]:
[(‘इराक’, ‘NNP’), (‘के’, ‘PREP’), (‘विदेश’, ‘NNC’), (‘मंत्री’, ‘NN’), (‘ने’, ‘PREP’), (‘अमरीका’, ‘NNP’), (‘के’, ‘PREP’), (‘उस’, ‘PRP’), (‘प्रस्ताव’, ‘NN’), (‘का’, ‘PREP’), (‘मजाक’, ‘NVB’), (‘उड़ाया’, ‘VFM’), (‘है’, ‘VAUX’), (‘,’, ‘PUNC’), (‘जिसमें’, ‘PRP’), (‘अमरीका’, ‘NNP’), (‘ने’, ‘PREP’), (‘संयुक्त’, ‘NNC’), (‘राष्ट्र’, ‘NN’), (‘के’, ‘PREP’), (‘प्रतिबंधों’, ‘NN’), (‘को’, ‘PREP’), (‘इराकी’, ‘JJ’), (‘नागरिकों’, ‘NN’), (‘के’, ‘PREP’), (‘लिए’, ‘PREP’), (‘कम’, ‘INTF’), (‘हानिकारक’, ‘JJ’), (‘बनाने’, ‘VNN’), (‘के’, ‘PREP’), (‘लिए’, ‘PREP’), (‘कहा’, ‘VFM’), (‘है’, ‘VAUX’), (‘।’, ‘PUNC’)]
easy wasn’t it?
The main issue here is that the nltk data is not complete. So we tend to get the tag “Unk” most of the time while tagging the words ex: (‘वाशिंग’, ‘Unk’), (‘मशीन’, ‘Unk’).
How to overcome that?
- we can stem the word as purposed by [1]
- we can use probability with freq for next word as purposed by[2]
- we can try handling the compound words as purposed by [3]
- we can add more tagged sentences to NLTK hindi.pos
- we can also use Google translator to translate and get the tag
- You can try training the data and predict the “UNK” data using SVM or HMM Tagger.
I have tried using Google Translator API to handle the “Unk” tags by translating and getting the tags and then appending it to the NLTK Indian Corpus which gave a pretty good result. You can check in my Github.
REF:
[1] “Hindi POS Tagger Using Naive Stemming : Harnessing Morphological Information Without Extensive Linguistic Knowledge” by Manish and Pushpak https://www.cse.iitb.ac.in/~pb/papers/icon08-hindi-pos-tagger.pdf
[2] “HMM BASED POS TAGGER FOR HINDI” by Nisheeth Joshi, Hemant Darbari and Iti Mathur. https://airccj.org/CSCP/vol3/csit3639.pdf
[3] “Improving statistical POS tagging using linguistic features for Hindi and Telugu” by S Phani Kumar Gadde, Meher Vijay Yeleti. https://researchweb.iiit.ac.in/~mehervijay.yeleti/papers/icon08-pos.pdf
