alpaca/alpaca/alpaca.internal/alpaca.internal.lexer/Tokenization

Tokenization

alpaca.internal.lexer.Tokenization

abstract transparent class Tokenization[Ctx <: LexerCtx](using betweenStages: BetweenStages[Ctx], errorHandling: ErrorHandling[Ctx], empty: Empty[Ctx]) extends Selectable

The result of compiling a lexer definition.

This abstract class represents a compiled lexer that can tokenize input. It is generated by the lexer macro and provides methods to access tokens and perform tokenization.

Type parameters

Ctx: the global context type

Attributes

Experimental: true
Source: Tokenization.scala
Graph
Supertypes: trait Selectable

class Object

trait Matchable

class Any

Members list

Type members

Types

Attributes

Source: Tokenization.scala

Attributes

Source: Tokenization.scala

Attributes

Source: Tokenization.scala

Value members

Abstract methods

The compiled pattern that matches all defined tokens.

Attributes

Source: Tokenization.scala

Provides dynamic access to tokens by name.

This allows accessing tokens using dot notation, e.g., myLexer.PLUS.

Value parameters

fieldName: the token name

Attributes

Returns: the token definition
Source: Tokenization.scala

List of all tokens defined in this lexer, including ignored tokens.

Attributes

Source: Tokenization.scala

Concrete methods

Tokenizes the input character sequence.

Processes the input from start to finish, matching tokens and building a list of lexems. Throws a RuntimeException if an unexpected character is encountered.

Value parameters

input: the input to tokenize

Attributes

Returns: a list of lexems representing the tokenized input
Source: Tokenization.scala

In this article

Generated with

made with ❤️ and coffee