Layers of Core Text Mining

At the lowest level, linguistic analysis and NLP techniques aim to identify key concept descriptors (who/what/where/when) embedded in textual documents. Different types of linguistic analysis techniques have been developed. Word and inverted indexing can be combined with stemming, morphological analysis, Boolean, proximity, range and fuzzy search. The unit of analysis is 'word'. Phrasal analysis, on the other hand, aims to extract meaningful noun phrase units or entities (e.g., people names, organisation names, location names). Both linguistic and statistical analysis techniques are plausible. In addition, semantic analysis based on techniques, such as, semantic grammar and case grammar can be used to represent semantics (meaning) in sentences. Semantic analysis is domain specific and lacks scalability. This often requires a significant knowledge base or a domain lexicon creation effort and hence it may not be suitable for general-purpose text mining across a wide spectrum of domains.