# `Feel.Syntax.Token`
[🔗](https://github.com/FeeJai/BDM/blob/feel-v0.1.0/apps/feel/lib/feel/syntax/token.ex#L1)

A token of FEEL source text.

The lexer produces raw tokens and does not decide which of them form a name.
In FEEL a name is a sequence of tokens that may contain spaces and symbols
such as `-`, so `a - b` can be one name or a subtraction depending on the names
in scope, and only the parser knows the scope. The tokens carry what it needs:
the exact text, the position, and whether white space came before.

## Fields

  * `:kind` - see `t:kind/0`.
  * `:text` - the source text of the token, exactly as written.
  * `:value` - the decoded value: a `t:Feel.Value.Number.t/0` for `:number`, the
    string without quotes and with escapes resolved for `:string`, the tokens of the name as
    texts (`["first", "name"]`) for `:quoted_name`, `nil` otherwise.
  * `:span` - `{from, to}`, the byte range in the source (`to` exclusive).
  * `:space_before?` - whether white space or a comment immediately precedes the
    token. It tells `a-b` from `a - b`.
  * `:keyword?` - for a `:word`, whether it is a reserved word such as `and` or
    `if`. A keyword cannot start a name, but can be a later part of one.
  * `:name_start?` - for a `:word`, whether it begins with a name start
    character. A word like `2nd` does not, and can only be part of a name.

## Kinds

  * `:word` - a maximal run of name part characters, including keywords.
  * `:number` - a numeric literal, never with a sign.
  * `:string` - a string literal.
  * `:dot`, `:slash`, `:minus`, `:plus`, `:star`, `:quote` - the single-character
    symbols that can also be part of a name; `:quote` is `'` or the typographic
    apostrophe U+2019.
  * `:dot_dot`, `:dot_dot_dot`, `:star_star`, `:lt`, `:lte`, `:gt`, `:gte`, `:eq`,
    `:neq`, `:arrow`, `:at`, `:lparen`, `:rparen`, `:lbracket`, `:rbracket`,
    `:lbrace`, `:rbrace`, `:comma`, `:colon` - operators and punctuation.
  * `:quoted_name` - a name in backticks, `` `first name` `` (D-81). It is one token whatever is in scope,
    which is what it is for. What is between the backticks is a name as DMN 1.6 writes it, in its canonical
    form (`Feel.Name`): the words with one space between them, `-` and the other name symbols as in
    `tracking-id`, and nothing else. The lexer checks it and reports anything else; the token's `:text` is
    with the backticks, and its `:value` the texts of the tokens in it.
  * `:bang` - a `!` that is not the start of `!=`. FEEL itself has no use for it; it marks a
    type as not `null` in a type expression (`number!`, D-48).
  * `:pipe` - a `|` that is a union in a type that a host declares (`number | string`, D-178). The lexer does not make it, since
    `|` is not a character of FEEL: `Feel.Type.parse/2` cuts the source at the `|` the lexer stops at.

# `kind`

```elixir
@type kind() ::
  :word
  | :number
  | :string
  | :dot
  | :slash
  | :minus
  | :plus
  | :star
  | :quote
  | :dot_dot
  | :dot_dot_dot
  | :star_star
  | :lt
  | :lte
  | :gt
  | :gte
  | :eq
  | :neq
  | :arrow
  | :at
  | :lparen
  | :rparen
  | :lbracket
  | :rbracket
  | :lbrace
  | :rbrace
  | :comma
  | :colon
  | :quoted_name
  | :bang
  | :pipe
```

# `t`

```elixir
@type t() :: %Feel.Syntax.Token{
  keyword?: boolean(),
  kind: kind(),
  name_start?: boolean(),
  space_before?: boolean(),
  span: Feel.Diagnostic.span(),
  text: String.t(),
  value: Feel.Value.Number.t() | String.t() | [String.t(), ...] | nil
}
```

# `can_continue_name?`

```elixir
@spec can_continue_name?(t()) :: boolean()
```

Returns `true` for a token that can be a later part of a name: any word
(including keywords and words like `2nd`), a number, or one of the name
symbols `. / - + * '`.

    iex> Feel.Syntax.Token.can_continue_name?(%Feel.Syntax.Token{kind: :minus, text: "-", span: {1, 2}})
    true
    iex> Feel.Syntax.Token.can_continue_name?(%Feel.Syntax.Token{kind: :comma, text: ",", span: {1, 2}})
    false

# `can_start_name?`

```elixir
@spec can_start_name?(t()) :: boolean()
```

Returns `true` for a token that can begin a name: a word that starts with a
name start character and is not a keyword.

    iex> Feel.Syntax.Token.can_start_name?(%Feel.Syntax.Token{kind: :word, text: "age", span: {0, 3}, name_start?: true})
    true
    iex> Feel.Syntax.Token.can_start_name?(%Feel.Syntax.Token{kind: :word, text: "if", span: {0, 2}, name_start?: true, keyword?: true})
    false

---

*Consult [api-reference.md](api-reference.md) for complete listing*