In
computer science,
lexical analysis,
lexing or
tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (
strings with an assigned and thus identified meaning). A program that performs lexical analysis may be termed a
lexer,
tokenizer, or
scanner, though
scanner is also a term for the first stage of a lexer. A lexer is generally combined with a
parser, which together analyze the
syntax of programming languages, web pages, and so forth.
