Feel.Syntax.Token (feel v0.1.0)

Copy Markdown View Source

A token of FEEL source text.

The lexer produces raw tokens and does not decide which of them form a name. In FEEL a name is a sequence of tokens that may contain spaces and symbols such as -, so a - b can be one name or a subtraction depending on the names in scope, and only the parser knows the scope. The tokens carry what it needs: the exact text, the position, and whether white space came before.

Fields

  • :kind - see kind/0.
  • :text - the source text of the token, exactly as written.
  • :value - the decoded value: a Feel.Value.Number.t/0 for :number, the string without quotes and with escapes resolved for :string, the tokens of the name as texts (["first", "name"]) for :quoted_name, nil otherwise.
  • :span - {from, to}, the byte range in the source (to exclusive).
  • :space_before? - whether white space or a comment immediately precedes the token. It tells a-b from a - b.
  • :keyword? - for a :word, whether it is a reserved word such as and or if. A keyword cannot start a name, but can be a later part of one.
  • :name_start? - for a :word, whether it begins with a name start character. A word like 2nd does not, and can only be part of a name.

Kinds

  • :word - a maximal run of name part characters, including keywords.
  • :number - a numeric literal, never with a sign.
  • :string - a string literal.
  • :dot, :slash, :minus, :plus, :star, :quote - the single-character symbols that can also be part of a name; :quote is ' or the typographic apostrophe U+2019.
  • :dot_dot, :dot_dot_dot, :star_star, :lt, :lte, :gt, :gte, :eq, :neq, :arrow, :at, :lparen, :rparen, :lbracket, :rbracket, :lbrace, :rbrace, :comma, :colon - operators and punctuation.
  • :quoted_name - a name in backticks, `first name` (D-81). It is one token whatever is in scope, which is what it is for. What is between the backticks is a name as DMN 1.6 writes it, in its canonical form (Feel.Name): the words with one space between them, - and the other name symbols as in tracking-id, and nothing else. The lexer checks it and reports anything else; the token's :text is with the backticks, and its :value the texts of the tokens in it.
  • :bang - a ! that is not the start of !=. FEEL itself has no use for it; it marks a type as not null in a type expression (number!, D-48).

  • :pipe - a | that is a union in a type that a host declares (number | string, D-178). The lexer does not make it, since | is not a character of FEEL: Feel.Type.parse/2 cuts the source at the | the lexer stops at.

Summary

Types

kind()
t()

Functions

can_continue_name?(token)

Returns true for a token that can be a later part of a name: any word (including keywords and words like 2nd), a number, or one of the name symbols . / - + * '.

can_start_name?(token)

Returns true for a token that can begin a name: a word that starts with a name start character and is not a keyword.

Types

kind()

@type kind() ::
  :word
  | :number
  | :string
  | :dot
  | :slash
  | :minus
  | :plus
  | :star
  | :quote
  | :dot_dot
  | :dot_dot_dot
  | :star_star
  | :lt
  | :lte
  | :gt
  | :gte
  | :eq
  | :neq
  | :arrow
  | :at
  | :lparen
  | :rparen
  | :lbracket
  | :rbracket
  | :lbrace
  | :rbrace
  | :comma
  | :colon
  | :quoted_name
  | :bang
  | :pipe

t()

@type t() :: %Feel.Syntax.Token{
  keyword?: boolean(),
  kind: kind(),
  name_start?: boolean(),
  space_before?: boolean(),
  span: Feel.Diagnostic.span(),
  text: String.t(),
  value: Feel.Value.Number.t() | String.t() | [String.t(), ...] | nil
}

Functions

can_continue_name?(token)

@spec can_continue_name?(t()) :: boolean()

Returns true for a token that can be a later part of a name: any word (including keywords and words like 2nd), a number, or one of the name symbols . / - + * '.

iex> Feel.Syntax.Token.can_continue_name?(%Feel.Syntax.Token{kind: :minus, text: "-", span: {1, 2}})
true
iex> Feel.Syntax.Token.can_continue_name?(%Feel.Syntax.Token{kind: :comma, text: ",", span: {1, 2}})
false

can_start_name?(token)

@spec can_start_name?(t()) :: boolean()

Returns true for a token that can begin a name: a word that starts with a name start character and is not a keyword.

iex> Feel.Syntax.Token.can_start_name?(%Feel.Syntax.Token{kind: :word, text: "age", span: {0, 3}, name_start?: true})
true
iex> Feel.Syntax.Token.can_start_name?(%Feel.Syntax.Token{kind: :word, text: "if", span: {0, 2}, name_start?: true, keyword?: true})
false