AlphaGitHub
Source/Packages

Compiler.Lexer

packages/compiler/src/Compiler/Lexer.alpha

219 lines31 declarations7.5 KiBSHA-256 62071dc752e3
Download exact fileView at revisionf5034773ad4bb52133

def · lines 166–192

lexTokens

Full file
166def lexTokens =
167  (lambda unrestricted input : Bytes .
168    (bytes-eliminate
169      (lambda unrestricted remaining : Bytes . (family TokenStream))
170      (constructor TokenStream TokenEnd)
171      (lambda unrestricted head : Byte .
172        (lambda unrestricted tail : Bytes .
173          (lambda unrestricted lexedTail : (family TokenStream) .
174            (eliminate
175              ByteClass
176              (lambda unrestricted value : (family ByteClass) . (family TokenStream))
177              (classifyByte head)
178              (branch
179                OpenDelimiter
180                .
181                (constructor TokenStream TokenNext (constructor LexToken TokenOpen) lexedTail))
182              (branch
183                CloseDelimiter
184                .
185                (constructor TokenStream TokenNext (constructor LexToken TokenClose) lexedTail))
186              (branch
187                Whitespace
188                .
189                (constructor TokenStream TokenNext (constructor LexToken TokenBoundary) lexedTail))
190              (branch DecimalDigit . (prependAtomByte head lexedTail))
191              (branch IdentifierByte . (prependAtomByte head lexedTail))))))
192      input))

The compiler supplied declaration spans and resolved links from this source snapshot. This page does not assert that this file belongs to a checked closure.