Lexical analysis

The process IR Expert uses to convert the characters in an IR query into a stream of "or" words or tokens.