# `Feel.Syntax.Lexer`
[🔗](https://github.com/FeeJai/BDM/blob/feel-v0.1.0/apps/feel/lib/feel/syntax/lexer.ex#L1)

Splits FEEL source text into raw tokens (DMN 1.6, section 10.3.1.4).

The lexer is deliberately context-free. FEEL names may contain spaces and
symbols, and whether `a-b` is a name or a subtraction depends on the names in
scope, so the lexer cannot decide it. It emits words, numbers, strings and
symbols, each with its text, position, and whether white space came before it;
the parser groups them into names (see `Feel.Syntax.Token`). Scope names are
lexed with the same lexer, so a name in scope and the same name in an
expression produce the same tokens.

## What the lexer does

  * **Words** are maximal runs of name part characters (`Feel.Syntax.Characters`),
    so `IncomeTaxesAmount` and `x1` are one token each. A word that equals a
    reserved word is flagged as a keyword; `not`, `list`, `context`, `range`,
    `e`, and `E` are ordinary words.
  * **Numbers** are unsigned: `-` is always a separate token, and the parser
    turns it into a negation. The longest numeric literal wins (`1.5`, `.5`,
    `1e-3`), unless name characters follow it directly. Then a shorter literal
    is tried, and if none fits the whole run is a word: `2nd` is one word, and
    `1.5x` is the number `1`, a dot, and the word `5x`. A number is read with
    `Feel.Value.Number.parse/1`, so one with more than 34 digits is rounded to
    34, and one outside the decimal128 exponent range is an error.
  * **Strings** are delimited by `"` only; typographic quotes are not
    delimiters. The escapes are backslash followed by `'`, `"`, `\`, `n`, `r`,
    or `t`, and `\u` with four hex digits or `\U` with six. A high and a low
    surrogate written as two `\u` escapes make one character. A backslash before
    any other character stays in the string literally, since the grammar allows
    a backslash as a plain character. A string cannot contain a raw line break
    (U+000A to U+000D).
  * **Comments** are `//` to the end of the line and `/* ... */` (not nested).
    They count as white space. Comment openers win over `/` and `*`, so `a//b`
    is `a` followed by a comment.
  * **White space** is the set of grammar rule 61. It wins over name characters
    where the two overlap.
  * **Symbols** are `... .. . ** * + - / ' <= >= != -> < > = ( ) [ ] { } , : @`,
    longest match first, and `'` or the typographic apostrophe U+2019 as the
    `:quote` symbol. `->` is one token, so `a->b` is not a subtraction followed
    by a comparison.
  * **Names in backticks** (D-83) are one `:quoted_name` token: `` `first name` ``. They are not DMN 1.6
    FEEL; they say which tokens form a name without the scope having to, and the parser reports that
    one was used (`Feel.Analysis.QuotedNames`). What is between the backticks is lexed as the rest of the
    source is, and has to be a name in the canonical form of `Feel.Name`, which the lexer checks by
    writing the name out and comparing it with the text: `` `first  name` `` (two spaces) is not one,
    nor is `` `a - b` `` (it is `a-b`), `` `a,b` ``, `` `2nd` `` or `` `if` ``. There is no escape for a
    backtick.

Anything else is an error. Tokenizing stops at the first error and never raises.

## Errors

The error is a list with one `Feel.Diagnostic` whose code is one of
`:unterminated_string`, `:unterminated_comment`, `:invalid_escape`,
`:unexpected_character`, `:number_out_of_range`, `:invalid_encoding`, `:unterminated_quoted_name`,
`:invalid_quoted_name`, or `:source_too_large`.

# `result`

```elixir
@type result() ::
  {:ok, [Feel.Syntax.Token.t()]} | {:error, [Feel.Diagnostic.t(), ...]}
```

The result of tokenizing.

# `name_tokens`

```elixir
@spec name_tokens(String.t()) :: {:ok, [Feel.Syntax.Token.t(), ...]} | :error
```

Returns the tokens of a text that is a name, written the way a name can be: a word that can start a name,
then tokens that can continue it, with nothing but white space between them and around them.

A string used as a context key is a name when this gives its tokens (D-213), and the key is then the canonical
name of the tokens (`Feel.Name.join/1`). The parser and the analyzer both decide it here. A comment is no
white space for this: the lexer drops it, so the text would not be what it is read as. Anything the lexer
rejects is no name.

    iex> {:ok, tokens} = Feel.Syntax.Lexer.name_tokens("  first   name ")
    iex> Enum.map(tokens, & &1.text)
    ["first", "name"]
    iex> Feel.Syntax.Lexer.name_tokens("a - b = c")
    :error
    iex> Feel.Syntax.Lexer.name_tokens("a /* c */ b")
    :error

# `tokenize`

```elixir
@spec tokenize(String.t(), [{:limits, Feel.Limits.t()}]) :: result()
```

Tokenizes `source`.

## Options

  * `:limits` - a `t:Feel.Limits.t/0`; only `:max_source_bytes` is used here.

## Examples

    iex> {:ok, tokens} = Feel.Syntax.Lexer.tokenize("price * 2")
    iex> Enum.map(tokens, &{&1.kind, &1.text})
    [word: "price", star: "*", number: "2"]

---

*Consult [api-reference.md](api-reference.md) for complete listing*