Splits FEEL source text into raw tokens (DMN 1.6, section 10.3.1.4).

The lexer is deliberately context-free. FEEL names may contain spaces and symbols, and whether a-b is a name or a subtraction depends on the names in scope, so the lexer cannot decide it. It emits words, numbers, strings and symbols, each with its text, position, and whether white space came before it; the parser groups them into names (see Feel.Syntax.Token). Scope names are lexed with the same lexer, so a name in scope and the same name in an expression produce the same tokens.

What the lexer does

• Words are maximal runs of name part characters (Feel.Syntax.Characters), so IncomeTaxesAmount and x1 are one token each. A word that equals a reserved word is flagged as a keyword; not, list, context, range, e, and E are ordinary words.
• Numbers are unsigned: - is always a separate token, and the parser turns it into a negation. The longest numeric literal wins (1.5, .5, 1e-3), unless name characters follow it directly. Then a shorter literal is tried, and if none fits the whole run is a word: 2nd is one word, and 1.5x is the number 1, a dot, and the word 5x. A number is read with Feel.Value.Number.parse/1, so one with more than 34 digits is rounded to 34, and one outside the decimal128 exponent range is an error.
• Strings are delimited by " only; typographic quotes are not delimiters. The escapes are backslash followed by ', ", \, n, r, or t, and \u with four hex digits or \U with six. A high and a low surrogate written as two \u escapes make one character. A backslash before any other character stays in the string literally, since the grammar allows a backslash as a plain character. A string cannot contain a raw line break (U+000A to U+000D).
• Comments are // to the end of the line and /* ... */ (not nested). They count as white space. Comment openers win over / and *, so a//b is a followed by a comment.
• White space is the set of grammar rule 61. It wins over name characters where the two overlap.
• Symbols are ... .. . ** * + - / ' <= >= != -> < > = ( ) [ ] { } , : @, longest match first, and ' or the typographic apostrophe U+2019 as the :quote symbol. -> is one token, so a->b is not a subtraction followed by a comparison.
• Names in backticks (D-83) are one :quoted_name token: `first name`. They are not DMN 1.6 FEEL; they say which tokens form a name without the scope having to, and the parser reports that one was used (Feel.Analysis.QuotedNames). What is between the backticks is lexed as the rest of the source is, and has to be a name in the canonical form of Feel.Name, which the lexer checks by writing the name out and comparing it with the text: `first name` (two spaces) is not one, nor is `a - b` (it is a-b), `a,b`, `2nd` or `if`. There is no escape for a backtick.

Anything else is an error. Tokenizing stops at the first error and never raises.

Errors

The error is a list with one Feel.Diagnostic whose code is one of :unterminated_string, :unterminated_comment, :invalid_escape, :unexpected_character, :number_out_of_range, :invalid_encoding, :unterminated_quoted_name, :invalid_quoted_name, or :source_too_large.