sg.edu.nus.comp.nlp.ims.util
Interface ITokenizer

All Known Implementing Classes:
COpenNLPTokenizer, CPennTreeBankTokenizer

public interface ITokenizer

sentence tokenizer interface.

Author:
zhongzhi

Method Summary
 java.lang.String[] tokenize(java.lang.String input)
          tokenize an input sentence into tokens
 

Method Detail

tokenize

java.lang.String[] tokenize(java.lang.String input)
tokenize an input sentence into tokens

Parameters:
input - input sentence
Returns:
tokens