NLP Project for clustering hashtags by concept

Included in the code:

Hashtag segmentation
Manually identified set of patterns for filtering query-type tags
Hand labeled dictionary of 6000 tags and their concepts
Test script for classifying new tags into respective clusters

Requirements

Need to have the following installed for the code to work correctly:

Python 3.0, or higher
NLTK, along with wordnet

Usage instructions

/pattern_filter/filter_query_type_tags.py <hashtagfile> Outputs a file containing segmented query type tags identified in the original hashtag file.
./benchmark_similarity.py A simple exact word matching algorithm to identify the cluster of the tag, using the hand labeled dictionary.
./wordnet_similarity.py Identify the cluster the tag belongs to, using the dictionary and word similarity measures from wordnet.

References

[Stanford Dependency Parser] (http://nlp.stanford.edu/software/lex-parser.shtml)
[Princeton University "About WordNet." WordNet. Princeton University. 2010] (http://wordnet.princeton.edu)
[Wordsegment python module] (https://github.com/grantjenks/wordsegment)