Lexical Analysis, also known as scanning, is the first phase of the Front End (Compiler) in a compiler. It involves reading the source code as a stream of characters and grouping them into meaningful sequences called tokens.
Process:
- The lexer (or scanner) reads the source text.
- It identifies keywords, identifiers, literals, and operators.
- It produces a stream of tokens that are then passed to the Parser for the next stage, which is Parsing.
Relationship to other concepts:
- This process is fundamentally similar to Tokenization in the context of Large Language Model (LLM), where text is broken down into tokens for processing.
