alpaca/alpaca/alpaca.internal/alpaca.internal.lexer/Tokenization

Tokenization

alpaca.internal.lexer.Tokenization

abstract transparent class Tokenization[Ctx <: LexerCtx](using betweenStages: OnTokenMatch[Ctx], errorHandling: ErrorHandling[Ctx], empty: Empty[Ctx]) extends Selectable

The result of compiling a lexer definition.

This abstract class represents a compiled lexer that can tokenize input. It is generated by the lexer macro and provides methods to access tokens and perform tokenization.

Type parameters

Ctx: the global context type

Attributes

Experimental: true
Source: Tokenization.scala
Graph
Supertypes: trait Selectable

class Object

trait Matchable

class Any

Members list

Type members

Types

Attributes

Source: Tokenization.scala

Attributes

Source: Tokenization.scala

Attributes

Source: Tokenization.scala

Value members

Abstract methods

The compiled pattern that matches all defined tokens.

Attributes

Source: Tokenization.scala

Provides dynamic access to tokens by name.

This allows accessing tokens using dot notation, e.g., myLexer.PLUS.

Value parameters

fieldName: the token name

Attributes

Returns: the token definition
Source: Tokenization.scala

Concrete methods

Tokenizes the input character sequence.

Processes the input from start to finish, matching tokens and building a list of lexemes. Throws a RuntimeException if an unexpected character is encountered.

Value parameters

input: the input to tokenize

Attributes

Returns: a tuple of (ctx, lexemes) where ctx is the final lexer context and lexemes is the list of matched tokens
Source: Tokenization.scala

In this article

Generated with

made with ❤️ and coffee