.. Copyright (c) 2017-2026 Juancarlo Añez (apalala@gmail.com)
.. SPDX-License-Identifier: BSD-4-Clause

.. include:: links.rst
.. highlight:: none

Grammar Syntax
--------------

|TatSu| grammars use an extension of the classic `EBNF`_ syntax. The classic
variations of EBNF_ (Tomassetti, EasyExtend, Wirth) and `ISO EBNF`_ are also
supported as input grammar format.

Rules
~~~~~

A grammar consists of a sequence of one or more rules of the form:

.. code:: ebnf
    :force:

    rulename: <expre>

If a *name* collides with a `Python`_ keyword or builtin, an underscore (``_``)
will be appended to it on the generated parser.

Rule names that start with an uppercase character:

.. code:: ebnf
    :force:

    FRAGMENT: /[a-z]+/

*do not* advance over whitespace before beginning to parse. This feature
becomes handy when defining complex lexical elements, as it allows
breaking them into more than one rule.

The parser returns an `AST`_ value for each rule depending on what was
parsed:

-  A single value
-  A list of `AST`_
-  A dict-like object for rules with named elements
-  An object, when ModelBuilderSemantics is used
-  None

See the `Abstract Syntax Trees`_ and `Building Models`_ sections for more
details.

.. _Abstract Syntax Trees: ast.html
.. _Building Models: models.html

Expressions
~~~~~~~~~~~

The expressions, in reverse order of operator precedence, can be any of the following.

.. note::
   Because |TatSu| now supports EBNF_, there must not be empty lines in expressions.

``# comment ...``
^^^^^^^^^^^^^^^^^

`Python`_-style end-of-line comments are allowed.


``// comment ...``
^^^^^^^^^^^^^^^^^^

`Java`_-style end-of-line comments are allowed.


``/* ... */``
^^^^^^^^^^^^^

`EBNF`_-style multi-line comments are allowed.


``e1 | e2``
^^^^^^^^^^^

Choice. Match either ``e1`` or ``e2``.

A ``|`` may be used before the first option if desired:

.. code:: ebnf
    :force:

    choices:
      | e1
      | e2
      | e3


``e1 e2``
^^^^^^^^^

Sequence. Match ``e1`` and then match ``e2``.


``( e )``
^^^^^^^^^

Grouping. Match ``e``. For example: ``('a' | 'b')``.

``(?: e )``
^^^^^^^^^^^

A non-capturing group. Like in a ``()`` group, match ``e``, but this time do not
capture what was parsed. For example:

.. code:: apl
    :force:

    start: header (?: delimiter ) body

    header: /[A-Z]+/

    delimiter: /[:,-]+/

    body: /[a-z]+/


``[ e ]``
^^^^^^^^^

Optionally match ``e``.


``{ e }`` or ``{ e }*``
^^^^^^^^^^^^^^^^^^^^^^^

Closure. Match ``e`` zero or more times. The `AST`_ returned for a
closure is always a ``list``.


``{ e }+``
^^^^^^^^^^

Positive closure. Match ``e`` one or more times. The `AST`_ is always
a ``list``.


``{}``
^^^^^^

Empty closure. Match nothing and produce an empty ``list`` as `AST`_.


``~``
^^^^^

The *cut* expression: commit to the current option and prevent
other options from being considered even when what follows fails to
parse.

In this example, other options won't be considered if a parenthesis is
parsed:

.. code:: ebnf
    :force:

      atom:
          | '(' ~ @:expre ')'
          | int
          | bool

Cut expression may be used anywhere. The effect of ``~`` is scoped to the nearest
enclosing brackets (*group*, *optional*, *closure*), the enclosing *choice*, or
the enclosing *rule*.

On the scoping of *cut*, consider these theoretical equivalences about implicit
choices in some expressions:

.. code:: apl
    :force:

    A →  α   ≡ A → α | ⊥
    A → [x]  ≡ A → B, B → x | ε
    A → {x}  ≡ A → B, B → xB | ε
    A → {x}+ ≡ A → B, B → xB | x

This is a common use of ``~``. The *closure* doesn't allow a partial assignment
expressions to escape it:

.. code:: ebnf
    :force:

    parameters: ','.{name '=' ~ expression}


``s%{ e }+``
^^^^^^^^^^^^

Positive join. Inspired by `Python`_'s ``str.join()``, it parses the
same as this expression:

.. code:: ebnf
    :force:

    e {s ~ e}

yet the result is a single list of the form:

.. code:: ebnf
    :force:

    [e, s, e, s, e, ...]

Use grouping if `s` is more complex than a *token* or a *pattern*:

.. code:: ebnf
    :force:

    (s t)%{ e }+


``s%{ e }`` or ``s%{ e }*``
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Join. Parses the list of ``s``-separated expressions, or the empty
closure. It's equivalent to:

.. code:: ebnf
    :force:

    s%{e}+|{}


``s.{ e }+``
^^^^^^^^^^^^

Positive *gather*. Like *positive join*, but the separator is not
included in the resulting `AST`_.

``s.{ e }`` or ``s.{ e }*``
^^^^^^^^^^^^^^^^^^^^^^^^^^^

*Gather*. Like the *join*, but the separator is not included in the
resulting `AST`_. It's equivalent to:

.. code:: ebnf
    :force:

    s.{e}+|{}


``&e``
^^^^^^

Positive lookahead. Succeed if ``e`` can be parsed, but do not consume
any input.


``!e``
^^^^^^

Negative lookahead. Fail if ``e`` can be parsed, and do not consume
any input.


``'text'`` or ``"text"``
^^^^^^^^^^^^^^^^^^^^^^^^

Match the token *text* within the quotation marks.

Note that if *text* is alphanumeric, then |TatSu| will check that the
character following the token is not alphanumeric. This is done to
prevent tokens like *IN* matching when the text ahead is
*INITIALIZE*. This feature can be turned off by passing
``nameguard=False`` to the ``Parser`` or the ``Buffer``, or by using a
pattern expression (see below) instead of a token expression.
The ``@@nameguard`` and ``@@namechars`` directives may be specified
in the grammar for the same effect:

.. code:: ebnf
    :force:

    @@nameguard :: False

or to specify additional characters that should also be considered
part of names:

.. code:: ebnf
    :force:

    @@namechars :: '$-.'


``r'text'`` or ``r"text"``
^^^^^^^^^^^^^^^^^^^^^^^^^^

Match the token *text* within the triple quotation marks, interpreting *text*
like `Python`_'s `raw string literal`_\ s. This is useful


``'''text'''`` or ``"""text"""``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A multi-line version of ``token``. The text within the triple quotation marks
is stripped of trailing whitespace, and the common indentation is removed as to
help with pretty formatting grammars without having to worry about indentation.
The resulting text is matched as a token.



``/regexp/``
^^^^^^^^^^^^

Also  ``?"regexp"`` or ``?'regexp'``, The *pattern* expression.
Match the `Python`_ regular expression
``regexp`` at the current text position. Unlike other expressions,
this one does not advance over whitespace or comments. For that, place
the ``regexp`` as the only term in its own rule.

The *regex* is interpreted as a Python_ `raw string literal`_ and
passed the Python_ re_ module using ``match()`` at the current
position in the text. The returned AST_ has the semantics of
``re.findall(pattern, text)[0]`` (a `tuple` if there is more than one
group), so use ``(?:)`` for groups that should not be in the resulting
AST_.

Consecutive *patterns* are concatenated to form a single one.


``/./``
^^^^^^^

The *any* expression, matches the next position in the input. It works
exactly like the ``?'.'`` pattern, but is implemented at the lexical
level, without regular expressions.


``->e``
^^^^^^^

The "*skip to*" expression, useful for writing *recovery* rules.

The parser will advance over input, one character at time, until ``e``
matches. Whitespace and comments will be skipped at each
step. Advancing over input is done efficiently, with no regular
expressions involved.

The expression is equivalent to:

.. code:: ebnf
    :force:

    { !e /./ } e

A common form of the expression is ``->&e``, which is equivalent to:

.. code:: ebnf
    :force:

    { !e /./ } &e

This is an example of the use of the "*skip to*" expression for
recovery:

.. code:: ebnf
    :force:

    statement:
        | if_statement
        # ...

    if_statement:
        | 'if' condition 'then' statement ['else' statement]
        | 'if' statement_recovery

    statement_recovery: ->&statement


```constant```
^^^^^^^^^^^^^^

Match nothing, but behave as if ``constant`` had been parsed.

Constants can be used to inject elements into the concrete and
abstract syntax trees, perhaps avoiding having to write a
semantic action. For example:

.. code:: ebnf
    :force:

    boolean_option: name ['=' (boolean|`true`) ]


If the text evaluates to a Python literal (with ``ast.literal_eval()``), that will be the returned value. Otherwise, string interpolation in the style of ``str.format()`` over the names in the current `AST`_ is applied for *constant* elements. Occurrences of the ``{`` character must be escaped to ``\{`` if they are not intended for interpolation. A *constant* expression that hast type ``str`` is evaluated using:

.. code:: python
    :force:

    eval(f'{"f" + repr(text)}', {}, ast)


`````constant`````
^^^^^^^^^^^^^^^^^^

A multi-line version of ```constant```.


``^`constant```
^^^^^^^^^^^^^^^

Also  ``^```constant`````. An alert. There will be no token returned by
the parser, but an alert will be registed in the parse context and added
to the current node's ``parseinfo``.

The ``^`` character may appear more than once to encode the *alert
level*:

.. code:: ebnf
    :force:

    assignment: identifier '=' (
        | value
        | -> &';' ^^^`could not parse value in assignment to {identifier}`
    )


``rulename``
^^^^^^^^^^^^

Invoke the rule named ``rulename``. To help with lexical aspects of
grammars, rules with names that begin with an uppercase letter will
not advance the input over whitespace or comments.


``>rulename``
^^^^^^^^^^^^^

The include operator. Include the *right hand side* of rule
``rulename`` at this point. This is useful when some fragment of the
expression is complicated but still must provide named element to
the current context. the |TatSu| grammar uses it to parse rule parameters,
a complex expression that deliver only ``params`` and ``kwparams``.

The following set of declarations:

.. code:: ebnf
    :force:

    includable: exp1

    expanded: exp0 >includable exp2

Has the same effect as defining *expanded* as:

.. code:: ebnf
    :force:

    expanded: exp0 exp1 exp2

Note that the included rule must be defined before the rule that
includes it.


``()``
^^^^^^

The empty expression. Succeed without advancing over input. Its value
is the empty tuple ``()``.


``!()``
^^^^^^^

The *fail* expression. This is ``!`` applied to ``()``, which always fails, and
thus has no value. This is useful to prevent the parser from accepting certain
input, for example in placeholders for rules that are not yet defined.


``name=e`` or ``name:e``
^^^^^^^^^^^^^^^^^^^^^^^^

Add the result of ``e`` to the `AST`_ using ``name`` as key. If
``name`` collides with any attribute or method of ``dict``, or is a
`Python`_ keyword, an underscore (``_``) will be appended to the name.

.. note::

    ``name`` is bound in the *option* in which it appears, or in the rule when
    there are no options. When options define different names, only the names
    in the option that parses will be present in the resulting AST_. A ``name``
    will be bound to ``None`` when the expression ``e`` fails to parse. For ``name``
    used in enclosing expressions like *group*, *optional*, or *closure*, ``name``
    will be bound in the rule-level AST_ only if the complete enclosure parses
    (they have local scope, and are transferred to the outer scope only on success).
    The same criteria applies to expressions nested to any level.

When there are no named items in a rule or choice, the `AST`_ consists of the
elements parsed by the rule, either a single item or a ``list``. This
default behavior makes it easier to write simple rules:

.. code:: ebnf
    :force:

    number: /[0-9]+/

Without having to write:

.. code:: ebnf
    :force:

    number: number=/[0-9]+/

When a rule has named elements, the unnamed ones are excluded from the
`AST`_ (they are ignored).


``name+=e`` or ``name+:e``
^^^^^^^^^^^^^^^^^^^^^^^^^^

Add the result of ``e`` to the `AST`_ using ``name`` as key. Force the
AST entry to be a ``list`` even if only one element is added. Collisions
with ``dict`` attributes or `Python`_ keywords are resolved by
appending an underscore to ``name``.


``=e`` or ``@:e``
^^^^^^^^^^^^^^^^^

The override operator. Make the `AST`_ for the complete rule or choice be
the `AST`_ for ``e``.

.. note::

    As with ``name=e``, the effect of ``=e`` is scoped to the enclosing
    *option*, *group*, *optional*, or *closure*, and will apply only when the
    enclosure parses successfully.

This expression is useful to recover only part of the right hand side of a rule
without the need to name it, or add a semantic action. This is a typical use
of the override operator:

.. code:: ebnf
    :force:

    subexp: '(' =expre ')'

The `AST`_ returned for the ``subexp`` rule will be the `AST`_
recovered from invoking ``expre``.


``+=e`` or ``@+:e``
^^^^^^^^^^^^^^^^^^^

Like ``=e``, but make the `AST`_ always be a ``list``.

This operator is convenient in cases such as this, in which the delimiting
tokens are of no interest.

.. code:: ebnf
    :force:

    arglist: '(' +=arg {',' +=arg}* ')'



``$``
^^^^^

The *end of text* symbol. Verify that the end of the input text has
been reached.

``$->``
^^^^^^^

The *end of line* symbol. Verify that the end of the current line has
been reached. This is useful for parsing line-based formats, such as
configuration files, or for parsing comments.

The ``$->`` (EOL) expression will consume the whitespace up to and including the next line break, using the Python semantics of ``os.linesep``. The match interprets whitespace using the Python definition as implemented by ``str .isspace()``, so beware when a particular definition of *whitespace* is part of the language being parsed.

Comments, as defined for the grammar, will also be skipped by the ``$->`` expression in search of a newline, which means that newlines consummed by the comments patterns will not be *"seen"* by ``$->``.


Deprecated Expressions
~~~~~~~~~~~~~~~~~~~~~~

The following expressions are still recognized in grammars, but they are
considered deprecated, and will be removed in a future version of
|TatSu|.

The alternative syntax with no *keyword parameters* is deprecated:

.. code:: ebnf
    :force:

    addition[Add, '+']: addend '+' addend



``?/regexp/?``
^^^^^^^^^^^^^^

Another form of the pattern expression that can be used when there
are slashes ( ``/`` ) in the pattern is *deprecated*. Use the ``?"regexp"``
or ``?'regexp'`` forms instead.


``+/regexp/``
^^^^^^^^^^^^^

Also  ``+?"regexp"`` or ``+?'regexp'``, concatenate the given pattern with
the preceding one is *deprecated*. Use ``?" ..."`` string concatenations
instead.


``(* comment *)``
^^^^^^^^^^^^^^^^^

`Pascal`_-style multi-line comments are *deprecated*. Use `Java`_-style
comments instead.


``op<{ e }+``
^^^^^^^^^^^^^

Left join. Like the *join expression*, but the result is a
left-associative tree built with ``tuple()``, in which the first
element is the separator (``op``), and the other two elements are the
operands.

The expression:

.. code:: ebnf
    :force:

    '+'<{/\d+/}+

Will parse this input:

.. code:: python
    :force:

    1 + 2 + 3 + 4

To this tree:

.. code:: python
    :force:

    (
        '+',
        (
            '+',
            (
                '+',
                '1',
                '2'
            ),
            '3'
        ),
        '4'
    )


``op>{ e }+``
^^^^^^^^^^^^^

Right join. Like the *join expression*, but the result is a
right-associative tree built with ``tuple()``, in which the first
element is the separator (``op``), and the other two elements are the
operands.

The expression:

.. code:: ebnf
    :force:

    '+'>{/\d+/}+

Will parse this input:

.. code:: ebnf
    :force:

    1 + 2 + 3 + 4

To this tree:

.. code:: python

   (
      '+',
      '1',
      (
          '+',
          '2',
          (
              '+',
              '3',
              '4'
          )
      )
   )


Rules with Arguments
~~~~~~~~~~~~~~~~~~~~

Rules may specify a list of arguments and keyword arguments by enclosing
them in square brackets right after the rule name:

.. code:: ebnf
    :force:

    addition[Add, op='+']: addend '+' addend

Arguments within parenthesis are also available:

.. code:: ebnf
    :force:

    addition(Add, op='+'): addend '+' addend

The arguments values are fixed at grammar-compilation time.

Semantic methods for rules with arguments can be ready to receive the
arguments declared in the rule:

.. code:: python

   def addition(self, ast, name, op=None):
       ...

When working with rule arguments, it's good to define a ``_default()`` method
that is ready to take any combination of standard and keyword arguments:

.. code:: python

   def _default(self, ast, *args, **kwargs):
       ...


Based Rules
~~~~~~~~~~~

Rules may extend rules defined before by using the ``<`` operator.

The following set of declarations:

.. code:: ebnf
    :force:

    base[Param]: exp1

    extended < base: exp2

Has the same effect as defining *extended* as:

.. code:: ebnf
    :force:

    extended[Param]: exp1 exp2

Parameters from the *base rule* are copied to the new rule if the new
rule doesn't define its own. Repeated inheritance should be possible,
but it *hasn't been tested*.


Memoization
~~~~~~~~~~~

|TatSu| generates *packrat* parsers. The result of parsing a rule at a
given position in the input is *memoized*. The next time the parser
visits the same input position, with the same rule, the memoized result
is returned and the input advanced accordingly.

Some rules should not be memoized. For example, rules that
may succeed or not depending on the associated semantic action *should
not* be memoized.

The ``@nomemo`` decorator turns off memoization for a particular rule:

.. code:: ebnf
    :force:

    @nomemo
    INDENT: ()

    @nomemo
    DEDENT: ()


Rule Overrides
~~~~~~~~~~~~~~

A grammar rule may be redefined by using the ``@override`` decorator:

.. code:: ebnf
    :force:

    start: ab $

    ab: 'xyz'

    @override
    ab: @:'a' {@:'b'}


Grammar Name
~~~~~~~~~~~~

The prefix to be used in classes generated by |TatSu| can be passed to
the command-line tool using the ``-m`` option:

.. code:: bash

   $ tatsu -m MyLanguage mygrammar.tatsu

will generate:

.. code:: python

   class MyLanguageParser(Parser):
       ...

The name can also be specified within the grammar using the
``@@grammar`` directive:

.. code:: ebnf
    :force:

    @@grammar :: MyLanguage


Whitespace
~~~~~~~~~~

By default, |TatSu| generated parsers skip the usual whitespace characters with
the regular expression ``r'\s+'``, but you can change that behavior.

Whitespace may be specified within the grammar using the
``@@whitespace`` directive in the grammar:

.. code:: ebnf
    :force:

    @@whitespace :: /[\t ]+/

or:

.. code:: ebnf
    :force:

    @@whitespace :: None

If no ``whitespace`` or ``@@whitespace`` is specified, |TatSu| will use
``r'(?m)\s+'`` as a default. Use ``None`` to have *no whitespace definition*.

You can also pass a ``whitespace`` parameter to your parser, overriding any
directive setting in the grammar. For example, the following will skip over
*tab* (``\t``) and *space* characters, but not so with other typical whitespace
characters such as *newline* (``\n``):

.. code:: python

   parser: tatsu.parse(grammar, text, whitespace='\t ')

The character string is converted into a regular expression character
set before starting to parse.

You can also provide a regular expression directly instead of a string.
The following is equivalent to the previous example:

.. code:: python

   parser: tatsu.parse(grammar, text, whitespace=re.compile(r'[\t ]+'))

Note that the regular expression must be pre-compiled to let |TatSu|
distinguish it from plain string.

If you do not define any whitespace characters, then you will have to
handle whitespace in your grammar rules (as it's often done in `PEG`_
parsers):

.. code:: python

   parser: tatsu.parse(grammar, text, whitespace='')


.. code:: python

   parser: tatsu.parse(grammar, text, whitespace=None)

Case Sensitivity
~~~~~~~~~~~~~~~~

You may specify case insensitivity within the grammar using the
``@@ignorecase`` directive:

.. code:: ebnf
    :force:

    @@ignorecase :: True

The change will affect token matching, but not pattern matching. Use ``(?i)``
in patterns that should ignore case.

Case sensitivity can also be specified in the parser by using the ``ignorecase``
parameter when instantiating a parser:

.. code:: python

   parser: tatsu.parse(grammar, text, ignorecase=True)

Comments
~~~~~~~~

Parsers will skip over comments specified as a regular expression using
the ``comments`` parameter:

.. code:: python

   parser: tatsu.parse(grammar, text, comments="\(\*.*?\*\)")

For more complex comment handling, you can override the
``Buffer.eat_comments()`` method.

For flexibility, it's also possible to specify a pattern for end-of-line
comments:

.. code:: python

   parser: tatsu.compile(
       grammar,
       comments="\(\*.*?\*\)",
       eol_comments="#.*?$"
   )

Both patterns may also be specified within a grammar using the
``@@comments`` and ``@@eol_comments`` directives:

.. code:: ebnf
    :force:

    @@comments :: /\(\*.*?\*\)/
    @@eol_comments :: /#.*?$/


Reserved Words and Keywords
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some languages must reserve the use of certain tokens as valid
identifiers because the tokens are used to mark particular constructs in
the language. Those reserved tokens are known as `Reserved Words`_ or
`Keywords`_

|TatSu| provides support for preventing the use of `keywords`_ as
identifiers though the ``@@keyword`` directive,and the ``@name``
decorator.

A grammar may specify reserved tokens providing a list of them in one or
more ``@@keyword`` directives:

.. code:: ebnf
    :force:

    @@keyword :: if endif
    @@keyword :: else elseif

The ``@name`` decorator checks that the result of a grammar rule does
not match a token defined as a `keyword`_:

.. code:: ebnf
    :force:

    @name
    identifier: /(?!\d)\w+/

Note that the rule decorated with ``@name`` must produce a single string as result
(no named expressions that will produce a dict, and no rule arguments).

In some situations a token is reserved only in a specific context. In those
cases, a negative lookahead will prevent the use of that token:

.. code:: ebnf
    :force:

    statements: {!'END' statement}+


..
    Include Directive
    ~~~~~~~~~~~~~~~~~

    |TatSu| grammars support file inclusion through the include directive:

    .. code:: ebnf
        :force:

        #include :: "filename"

    The resolution of the *filename* is relative to the directory/folder of
    the source. Absolute paths and ``../`` navigation are honored.

    The base for implementing includes is available to |TatSu|-generated
    parsers through the ``Buffer`` class. See the ``EBNFBuffer`` class in
    ``tatsu.parser`` module for an example.


Left Recursion
~~~~~~~~~~~~~~

|TatSu| supports left recursion in `PEG`_ grammars. The algorithm used is
`Warth et al`_'s. Left recursion support is enabled by default.

Left recursion can be turned *on* or *off* from within the grammar using the
``@@left_recursion`` directive:

.. code:: ebnf
    :force:

    @@left_recursion :: False

Sometimes, while debugging a grammar, it's useful to turn left-recursion
support *on* or *off* in the code:

.. code:: python

   parser: tatsu.parse(grammar, text, left_recursion=True)