[3.14] gh-132661: Document t-strings and templatelib (GH-135229) (#136974)

Co-authored-by: Dave Peck <davepeck@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Loïc Simon <loic.pano@gmail.com>
Co-authored-by: pauleveritt <pauleveritt@me.com>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

Doc/glossary.rst

@@ -462,7 +462,7 @@ Glossary
       core and with user code.
 
    f-string
-      String literals prefixed with ``'f'`` or ``'F'`` are commonly called
+      String literals prefixed with ``f`` or ``F`` are commonly called
       "f-strings" which is short for
       :ref:`formatted string literals <f-strings>`.  See also :pep:`498`.
 
@@ -1322,6 +1322,11 @@ Glossary
 
       See also :term:`borrowed reference`.
 
+   t-string
+      String literals prefixed with ``t`` or ``T`` are commonly called
+      "t-strings" which is short for
+      :ref:`template string literals <t-strings>`.
+
    text encoding
       A string in Python is a sequence of Unicode code points (in range
       ``U+0000``--``U+10FFFF``). To store or transfer a string, it needs to be

Doc/library/ast.rst

@@ -290,9 +290,9 @@ Literals
    * ``conversion`` is an integer:
 
      * -1: no formatting
-     * 115: ``!s`` string formatting
+     * 115 (``ord('s')``): ``!s`` string formatting
-     * 114: ``!r`` repr formatting
+     * 114 (``ord('r')``): ``!r`` repr formatting
-     * 97: ``!a`` ascii formatting
+     * 97 (``ord('a')``): ``!a`` ASCII formatting
 
    * ``format_spec`` is a :class:`JoinedStr` node representing the formatting
      of the value, or ``None`` if no format was specified. Both
@@ -326,6 +326,54 @@ Literals
                                 Constant(value='.3')]))]))
 
 
+.. class:: TemplateStr(values)
+
+   A t-string, comprising a series of :class:`Interpolation` and :class:`Constant`
+   nodes.
+
+   .. doctest::
+
+        >>> print(ast.dump(ast.parse('t"{name} finished {place:ordinal}"', mode='eval'), indent=4))
+        Expression(
+            body=TemplateStr(
+                values=[
+                    Interpolation(
+                        value=Name(id='name', ctx=Load()),
+                        str='name',
+                        conversion=-1),
+                    Constant(value=' finished '),
+                    Interpolation(
+                        value=Name(id='place', ctx=Load()),
+                        str='place',
+                        conversion=-1,
+                        format_spec=JoinedStr(
+                            values=[
+                                Constant(value='ordinal')]))]))
+
+   .. versionadded:: 3.14
+
+
+.. class:: Interpolation(value, str, conversion, format_spec)
+
+   Node representing a single interpolation field in a t-string.
+
+   * ``value`` is any expression node (such as a literal, a variable, or a
+     function call).
+   * ``str`` is a constant containing the text of the interpolation expression.
+   * ``conversion`` is an integer:
+
+     * -1: no conversion
+     * 115: ``!s`` string conversion
+     * 114: ``!r`` repr conversion
+     * 97: ``!a`` ascii conversion
+
+   * ``format_spec`` is a :class:`JoinedStr` node representing the formatting
+     of the value, or ``None`` if no format was specified. Both
+     ``conversion`` and ``format_spec`` can be set at the same time.
+
+   .. versionadded:: 3.14
+
+
 .. class:: List(elts, ctx)
            Tuple(elts, ctx)
 

Doc/library/dis.rst

@@ -1120,6 +1120,48 @@ iterations of the loop.
    .. versionadded:: 3.12
 
 
+.. opcode:: BUILD_TEMPLATE
+
+   Constructs a new :class:`~string.templatelib.Template` from a tuple
+   of strings and a tuple of interpolations and pushes the resulting instance
+   onto the stack::
+
+      interpolations = STACK.pop()
+      strings = STACK.pop()
+      STACK.append(_build_template(strings, interpolations))
+
+   .. versionadded:: 3.14
+
+
+.. opcode:: BUILD_INTERPOLATION (format)
+
+   Constructs a new :class:`~string.templatelib.Interpolation` from a
+   value and its source expression and pushes the resulting instance onto the
+   stack.
+
+   If no conversion or format specification is present, ``format`` is set to
+   ``2``.
+
+   If the low bit of ``format`` is set, it indicates that the interpolation
+   contains a format specification.
+
+   If ``format >> 2`` is non-zero, it indicates that the interpolation
+   contains a conversion. The value of ``format >> 2`` is the conversion type
+   (``0`` for no conversion, ``1`` for ``!s``, ``2`` for ``!r``, and
+   ``3`` for ``!a``)::
+
+      conversion = format >> 2
+      if format & 1:
+          format_spec = STACK.pop()
+      else:
+          format_spec = None
+      expression = STACK.pop()
+      value = STACK.pop()
+      STACK.append(_build_interpolation(value, expression, conversion, format_spec))
+
+   .. versionadded:: 3.14
+
+
 .. opcode:: BUILD_TUPLE (count)
 
    Creates a tuple consuming *count* items from the stack, and pushes the

Doc/library/stdtypes.rst

@@ -2675,9 +2675,9 @@ For example:
    lead to a number of common errors (such as failing to display tuples and
    dictionaries correctly).  Using the newer :ref:`formatted string literals
    <f-strings>`, the :meth:`str.format` interface, or :ref:`template strings
-   <template-strings>` may help avoid these errors.  Each of these
+   ($-strings) <template-strings-pep292>` may help avoid these errors.
-   alternatives provides their own trade-offs and benefits of simplicity,
+   Each of these alternatives provides their own trade-offs and benefits of
-   flexibility, and/or extensibility.
+   simplicity, flexibility, and/or extensibility.
 
 String objects have one unique built-in operation: the ``%`` operator (modulo).
 This is also known as the string *formatting* or *interpolation* operator.

Doc/library/string.rst

@@ -198,8 +198,9 @@ Format String Syntax
 The :meth:`str.format` method and the :class:`Formatter` class share the same
 syntax for format strings (although in the case of :class:`Formatter`,
 subclasses can define their own format string syntax).  The syntax is
-related to that of :ref:`formatted string literals <f-strings>`, but it is
+related to that of :ref:`formatted string literals <f-strings>` and
-less sophisticated and, in particular, does not support arbitrary expressions.
+:ref:`template string literals <t-strings>`, but it is less sophisticated
+and, in particular, does not support arbitrary expressions.
 
 .. index::
    single: {} (curly brackets); in string formatting
@@ -264,6 +265,8 @@ Some simple format string examples::
    "Weight in tons {0.weight}"       # 'weight' attribute of first positional arg
    "Units destroyed: {players[0]}"   # First element of keyword argument 'players'.
 
+.. _formatstrings-conversion:
+
 The *conversion* field causes a type coercion before formatting.  Normally, the
 job of formatting a value is done by the :meth:`~object.__format__` method of the value
 itself.  However, in some cases it is desirable to force a type to be formatted
@@ -306,7 +309,7 @@ Format Specification Mini-Language
 
 "Format specifications" are used within replacement fields contained within a
 format string to define how individual values are presented (see
-:ref:`formatstrings` and :ref:`f-strings`).
+:ref:`formatstrings`, :ref:`f-strings`, and :ref:`t-strings`).
 They can also be passed directly to the built-in
 :func:`format` function.  Each formattable type may define how the format
 specification is to be interpreted.
@@ -789,10 +792,20 @@ Nesting arguments and more complex examples::
 
 
 
-.. _template-strings:
+.. _template-strings-pep292:
 
-Template strings
+Template strings ($-strings)
-----------------
+----------------------------
+
+.. note::
+
+   The feature described here was introduced in Python 2.4.  It is unrelated
+   to, and should not be confused with, the newer
+   :ref:`template strings <template-strings>` and
+   :ref:`t-string literal syntax <t-strings>` introduced in Python 3.14.
+   T-string literals evaluate to instances of a different
+   :class:`~string.templatelib.Template` class, found in the
+   :mod:`string.templatelib` module.
 
 Template strings provide simpler string substitutions as described in
 :pep:`292`.  A primary use case for template strings is for

Doc/library/string.templatelib.rst

@@ -0,0 +1,313 @@
+:mod:`!string.templatelib` --- Support for template string literals
+===================================================================
+
+.. module:: string.templatelib
+   :synopsis: Support for template string literals.
+
+**Source code:** :source:`Lib/string/templatelib.py`
+
+--------------
+
+.. seealso::
+
+   * :ref:`Format strings <f-strings>`
+   * :ref:`T-string literal syntax <t-strings>`
+
+
+.. _template-strings:
+
+Template strings
+----------------
+
+.. versionadded:: 3.14
+
+Template strings are a formatting mechanism that allows for deep control over
+how strings are processed. You can create templates using
+:ref:`t-string literal syntax <t-strings>`, which is identical to
+:ref:`f-string syntax <f-strings>` but uses a ``t`` instead of an ``f``.
+While f-strings evaluate to ``str``, t-strings create a :class:`Template`
+instance that gives you access to the static and interpolated (in curly braces)
+parts of a string *before* they are combined.
+
+
+.. _templatelib-template:
+
+Template
+--------
+
+The :class:`!Template` class describes the contents of a template string.
+
+:class:`!Template` instances are immutable: their attributes cannot be
+reassigned.
+
+.. class:: Template(*args)
+
+   Create a new :class:`!Template` object.
+
+   :param args: A mix of strings and :class:`Interpolation` instances in any order.
+   :type args: str | Interpolation
+
+   The most common way to create a :class:`!Template` instance is to use the
+   :ref:`t-string literal syntax <t-strings>`. This syntax is identical to that of
+   :ref:`f-strings <f-strings>` except that it uses a ``t`` instead of an ``f``:
+
+   >>> name = "World"
+   >>> template = t"Hello {name}!"
+   >>> type(template)
+   <class 'string.templatelib.Template'>
+
+   Templates ars stored as sequences of literal :attr:`~Template.strings`
+   and dynamic :attr:`~Template.interpolations`.
+   A :attr:`~Template.values` attribute holds the interpolation values:
+
+   >>> template.strings
+   ('Hello ', '!')
+   >>> template.interpolations
+   (Interpolation('World', ...),)
+   >>> template.values
+   ('World',)
+
+   The :attr:`!strings` tuple has one more element than :attr:`!interpolations`
+   and :attr:`!values`; the interpolations “belong” between the strings.
+   This may be easier to understand when tuples are aligned::
+
+      template.strings:  ('Hello ',          '!')
+      template.values:   (          'World',    )
+
+   While literal syntax is the most common way to create :class:`!Template`
+   instances, it is also possible to create them directly using the constructor:
+
+   >>> from string.templatelib import Interpolation, Template
+   >>> name = "World"
+   >>> template = Template("Hello, ", Interpolation(name, "name"), "!")
+   >>> list(template)
+   ['Hello, ', Interpolation('World', 'name', None, ''), '!']
+
+   If two or more consecutive strings are passed, they will be concatenated
+   into a single value in the :attr:`~Template.strings` attribute. For example,
+   the following code creates a :class:`Template` with a single final string:
+
+   >>> from string.templatelib import Template
+   >>> template = Template("Hello ", "World", "!")
+   >>> template.strings
+   ('Hello World!',)
+
+   If two or more consecutive interpolations are passed, they will be treated
+   as separate interpolations and an empty string will be inserted between them.
+   For example, the following code creates a template with empty placeholders
+   in the :attr:`~Template.strings` attribute:
+
+   >>> from string.templatelib import Interpolation, Template
+   >>> template = Template(Interpolation("World", "name"), Interpolation("!", "punctuation"))
+   >>> template.strings
+   ('', '', '')
+
+   .. attribute:: strings
+       :type: tuple[str, ...]
+
+       A :ref:`tuple <tut-tuples>` of the static strings in the template.
+
+       >>> name = "World"
+       >>> t"Hello {name}!".strings
+       ('Hello ', '!')
+
+       Empty strings *are* included in the tuple:
+
+       >>> name = "World"
+       >>> t"Hello {name}{name}!".strings
+       ('Hello ', '', '!')
+
+       The ``strings`` tuple is never empty, and always contains one more
+       string than the ``interpolations`` and ``values`` tuples:
+
+       >>> t"".strings
+       ('',)
+       >>> t"".values
+       ()
+       >>> t"{'cheese'}".strings
+       ('', '')
+       >>> t"{'cheese'}".values
+       ('cheese',)
+
+   .. attribute:: interpolations
+       :type: tuple[Interpolation, ...]
+
+       A tuple of the interpolations in the template.
+
+       >>> name = "World"
+       >>> t"Hello {name}!".interpolations
+       (Interpolation('World', 'name', None, ''),)
+
+       The ``interpolations`` tuple may be empty and always contains one fewer
+       values than the ``strings`` tuple:
+
+       >>> t"Hello!".interpolations
+       ()
+
+   .. attribute:: values
+       :type: tuple[Any, ...]
+
+       A tuple of all interpolated values in the template.
+
+       >>> name = "World"
+       >>> t"Hello {name}!".values
+       ('World',)
+
+       The ``values`` tuple always has the same length as the
+       ``interpolations`` tuple. It is equivalent to
+       ``tuple(i.value for i in template.interpolations)``.
+
+   .. describe:: iter(template)
+
+       Iterate over the template, yielding each string and
+       :class:`Interpolation` in order.
+
+       >>> name = "World"
+       >>> list(t"Hello {name}!")
+       ['Hello ', Interpolation('World', 'name', None, ''), '!']
+
+       Empty strings are *not* included in the iteration:
+
+       >>> name = "World"
+       >>> list(t"Hello {name}{name}")
+       ['Hello ', Interpolation('World', 'name', None, ''), Interpolation('World', 'name', None, '')]
+
+   .. describe:: template + other
+                 template += other
+
+       Concatenate this template with another, returning a new
+       :class:`!Template` instance:
+
+       >>> name = "World"
+       >>> list(t"Hello " + t"there {name}!")
+       ['Hello there ', Interpolation('World', 'name', None, ''), '!']
+
+       Concatenation between a :class:`!Template` and a ``str`` is *not* supported.
+       This is because it is ambiguous whether the string should be treated as
+       a static string or an interpolation. If you want to concatenate a
+       :class:`!Template` with a string, you should either wrap the string
+       directly in a :class:`!Template` (to treat it as a static string) or use
+       an :class:`!Interpolation` (to treat it as dynamic):
+
+       >>> from string.templatelib import Template, Interpolation
+       >>> template = t"Hello "
+       >>> # Treat "there " as a static string
+       >>> template += Template("there ")
+       >>> # Treat name as an interpolation
+       >>> name = "World"
+       >>> template += Template(Interpolation(name, "name"))
+       >>> list(template)
+       ['Hello there ', Interpolation('World', 'name', None, '')]
+
+
+.. class:: Interpolation(value, expression="", conversion=None, format_spec="")
+
+   Create a new :class:`!Interpolation` object.
+
+   :param value: The evaluated, in-scope result of the interpolation.
+   :type value: object
+
+   :param expression: The text of a valid Python expression, or an empty string.
+   :type expression: str
+
+   :param conversion: The optional :ref:`conversion <formatstrings>` to be used, one of r, s, and a.
+   :type conversion: ``Literal["a", "r", "s"] | None``
+
+   :param format_spec: An optional, arbitrary string used as the :ref:`format specification <formatspec>` to present the value.
+   :type format_spec: str
+
+   The :class:`!Interpolation` type represents an expression inside a template string.
+
+   :class:`!Interpolation` instances are immutable: their attributes cannot be
+   reassigned.
+
+   .. attribute:: value
+
+       :returns: The evaluated value of the interpolation.
+       :type: object
+
+       >>> t"{1 + 2}".interpolations[0].value
+       3
+
+   .. attribute:: expression
+
+       :returns: The text of a valid Python expression, or an empty string.
+       :type: str
+
+       The :attr:`~Interpolation.expression` is the original text of the
+       interpolation's Python expression, if the interpolation was created
+       from a t-string literal. Developers creating interpolations manually
+       should either set this to an empty string or choose a suitable valid
+       Python expression.
+
+       >>> t"{1 + 2}".interpolations[0].expression
+       '1 + 2'
+
+   .. attribute:: conversion
+
+       :returns: The conversion to apply to the value, or ``None``.
+       :type: ``Literal["a", "r", "s"] | None``
+
+       The :attr:`!Interpolation.conversion` is the optional conversion to apply
+       to the value:
+
+       >>> t"{1 + 2!a}".interpolations[0].conversion
+       'a'
+
+       .. note::
+
+         Unlike f-strings, where conversions are applied automatically,
+         the expected behavior with t-strings is that code that *processes* the
+         :class:`!Template` will decide how to interpret and whether to apply
+         the :attr:`!Interpolation.conversion`.
+
+   .. attribute:: format_spec
+
+       :returns: The format specification to apply to the value.
+       :type: str
+
+       The :attr:`!Interpolation.format_spec` is an optional, arbitrary string
+       used as the format specification to present the value:
+
+       >>> t"{1 + 2:.2f}".interpolations[0].format_spec
+       '.2f'
+
+       .. note::
+
+         Unlike f-strings, where format specifications are applied automatically
+         via the :func:`format` protocol, the expected behavior with
+         t-strings is that code that *processes* the :class:`!Template` will
+         decide how to interpret and whether to apply the format specification.
+         As a result, :attr:`!Interpolation.format_spec` values in
+         :class:`!Template` instances can be arbitrary strings, even those that
+         do not necessarily conform to the rules of Python's :func:`format`
+         protocol.
+
+   Interpolations support pattern matching, allowing you to match against
+   their attributes with the :ref:`match statement <match>`:
+
+   >>> from string.templatelib import Interpolation
+   >>> interpolation = Interpolation(3.0, "1 + 2", None, ".2f")
+   >>> match interpolation:
+   ...     case Interpolation(value, expression, conversion, format_spec):
+   ...         print(value, expression, conversion, format_spec)
+   ...
+   3.0 1 + 2 None .2f
+
+
+Helper functions
+----------------
+
+.. function:: convert(obj, /, conversion)
+
+   Applies formatted string literal :ref:`conversion <formatstrings-conversion>`
+   semantics to the given object *obj*.
+   This is frequently useful for custom template string processing logic.
+
+   Three conversion flags are currently supported:
+
+   * ``'s'`` which calls :func:`str` on the value,
+   * ``'r'`` which calls :func:`repr`, and
+   * ``'a'`` which calls :func:`ascii`.
+
+   If the conversion flag is ``None``, *obj* is returned unchanged.

Doc/library/text.rst

@@ -16,6 +16,7 @@ Python's built-in string type in :ref:`textseq`.
 .. toctree::
 
    string.rst
+   string.templatelib.rst
    re.rst
    difflib.rst
    textwrap.rst

Doc/reference/compound_stmts.rst

@@ -852,8 +852,8 @@ A literal pattern corresponds to most
 
 The rule ``strings`` and the token ``NUMBER`` are defined in the
 :doc:`standard Python grammar <./grammar>`.  Triple-quoted strings are
-supported.  Raw strings and byte strings are supported.  :ref:`f-strings` are
+supported.  Raw strings and byte strings are supported.  :ref:`f-strings`
-not supported.
+and :ref:`t-strings` are not supported.
 
 The forms ``signed_number '+' NUMBER`` and ``signed_number '-' NUMBER`` are
 for expressing :ref:`complex numbers <imaginary>`; they require a real number

Doc/reference/lexical_analysis.rst

@@ -561,9 +561,9 @@ escapes are not treated specially.
    single: f'; formatted string literal
    single: f"; formatted string literal
 
-A string literal with ``'f'`` or ``'F'`` in its prefix is a
+A string literal with ``f`` or ``F`` in its prefix is a
-:dfn:`formatted string literal`; see :ref:`f-strings`.  The ``'f'`` may be
+:dfn:`formatted string literal`; see :ref:`f-strings`.  The ``f`` may be
-combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw
+combined with ``r``, but not with ``b`` or ``u``, therefore raw
 formatted strings are possible, but formatted bytes literals are not.
 
 In triple-quoted literals, unescaped newlines and quotes are allowed (and are
@@ -756,7 +756,7 @@ f-strings
 .. versionadded:: 3.6
 
 A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal
-that is prefixed with ``'f'`` or ``'F'``.  These strings may contain
+that is prefixed with ``f`` or ``F``.  These strings may contain
 replacement fields, which are expressions delimited by curly braces ``{}``.
 While other string literals always have a constant value, formatted strings
 are really expressions evaluated at run time.
@@ -913,6 +913,48 @@ See also :pep:`498` for the proposal that added formatted string literals,
 and :meth:`str.format`, which uses a related format string mechanism.
 
 
+.. _t-strings:
+.. _template-string-literals:
+
+t-strings
+---------
+
+.. versionadded:: 3.14
+
+A :dfn:`template string literal` or :dfn:`t-string` is a string literal
+that is prefixed with ``t`` or ``T``.  These strings follow the same
+syntax and evaluation rules as :ref:`formatted string literals <f-strings>`, with
+the following differences:
+
+- Rather than evaluating to a ``str`` object, t-strings evaluate to a
+  :class:`~string.templatelib.Template` object from the
+  :mod:`string.templatelib` module.
+
+- The :func:`format` protocol is not used. Instead, the format specifier and
+  conversions (if any) are passed to a new :class:`~string.templatelib.Interpolation`
+  object that is created for each evaluated expression. It is up to code that
+  processes the resulting :class:`~string.templatelib.Template` object to
+  decide how to handle format specifiers and conversions.
+
+- Format specifiers containing nested replacement fields are evaluated eagerly,
+  prior to being passed to the :class:`~string.templatelib.Interpolation` object.
+  For instance, an interpolation of the form ``{amount:.{precision}f}`` will
+  evaluate the expression ``{precision}`` before setting the ``format_spec``
+  attribute of the resulting :class:`!Interpolation` object; if ``precision``
+  is (for example) ``2``, the resulting format specifier will be ``'.2f'``.
+
+- When the equal sign ``'='`` is provided in an interpolation expression, the
+  resulting :class:`~string.templatelib.Template` object will have the expression
+  text along with a ``'='`` character placed in its
+  :attr:`~string.templatelib.Template.strings` attribute. The
+  :attr:`~string.templatelib.Template.interpolations` attribute will also
+  contain an ``Interpolation`` instance for the expression. By default, the
+  :attr:`~string.templatelib.Interpolation.conversion` attribute will be set to
+  ``'r'`` (that is, :func:`repr`), unless there is a conversion explicitly
+  specified (in which case it overrides the default) or a format specifier is
+  provided (in which case, the ``conversion`` defaults to ``None``).
+
+
 .. _numbers:
 
 Numeric literals

Doc/tutorial/inputoutput.rst

@@ -95,10 +95,11 @@ Some examples::
    >>> repr((x, y, ('spam', 'eggs')))
    "(32.5, 40000, ('spam', 'eggs'))"
 
-The :mod:`string` module contains a :class:`~string.Template` class that offers
+The :mod:`string` module also contains support for so-called
-yet another way to substitute values into strings, using placeholders like
+:ref:`$-strings <template-strings-pep292>` that offer yet another way to
-``$x`` and replacing them with values from a dictionary, but offers much less
+substitute values into strings, using placeholders like ``$x`` and replacing
-control of the formatting.
+them with values from a dictionary. This syntax is easy to use, although
+it offers much less control of the formatting.
 
 .. index::
    single: formatted string literal