HFST - Helsinki Finite-State Transducer Technology API: libhfst/src/HfstTokenizer.h File Reference

HFST - Helsinki Finite-State Transducer Technology API version 3.7.1

All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages

Declaration of class hfst::HfstTokenizer. More...

#include "HfstSymbolDefs.h"
#include "HfstExceptionDefs.h"
#include <iostream>
#include <climits>
#include <string>

Go to the source code of this file.

Classes
class	HfstTokenizer
	A tokenizer for creating transducers from UTF-8 strings. More...

Namespaces
	hfst
	A namespace for HFST functions and datatypes.

Detailed Description

Declaration of class hfst::HfstTokenizer.