PEP 750: Tag Strings For Writing Domain-Specific Languages #3858

jimbaker · 2024-07-08T23:12:52Z

This PEP introduces tag strings for custom, repeatable string processing. Tag strings are an extension to f-strings, with a custom function -- the "tag" -- in place of the f prefix. This function can then provide rich features such as safety checks, lazy evaluation, domain specific languages (DSLs) for web templating, and more.

Tag strings are similar to JavaScript tagged template literals and related ideas in other languages.

📚 Documentation preview 📚: https://pep-previews--3858.org.readthedocs.build/

cpython-cla-bot · 2024-07-08T23:12:54Z

All commit authors signed the Contributor License Agreement.

.github/CODEOWNERS

peps/pep-0750.rst

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Lambda-wrapped expressions use class scope

peps/pep-0750.rst

Carreau · 2024-07-25T08:45:49Z

This is great. I hope we'll get log.debug"", log.info"" and alike. I've so many project use fstrings in logging, which prevents doing proper structured logging later. Though I do have the impression that tags having side-effects is not mentioned in the pep (beyond memoizing), but most example and discussions are around pure tags I think.

I'm also guessing projects like mypy will automatically detect (*args: Decoded | Interpolation) -> Any as potential tags.

jimbaker · 2024-07-26T13:52:16Z

This is great. I hope we'll get log.debug"", log.info"" and alike.

In this PEP, we don't support dotted names as prefixes for tag strings. We looked into it, and it required a fairly large number of parser changes. We decided to keep it simple instead. But dotted names should be supportable in a future PEP, if there's interest.

I've so many project use fstrings in logging, which prevents doing proper structured logging later. Though I do have the impression that tags having side-effects is not mentioned in the pep (beyond memoizing), but most example and discussions are around pure tags I think.

So the way I would think we would support logging integration is as follows, and it would enable structured logging:

Tag function. Create a tag function struct_log (it could be imported with a short name like sl). It has a signature like so:

def struct_log(*args: str | Interpolation) -> LogRecordMessage:
    ...

where LogRecordMessage holds the strings and interpolations for evaluation, if and when written to a log. It should be a simple wrapper, and is used to support mixed usage with the usual strings and args passed into log.info, etc.

@dataclass
class LogRecordMessage:
    args: tuple[str | Interpolation, ...]

A few things:

Logging most likely doesn't need Decoded.raw, so str can be used.
The specific tag function could be built by some sort of configurator. Example: provide a tag function that expands all usages of {expr} to {expr=value}.
Likewise, this can be configured such that structured logging can use Interpolation.expr when writing out a JSON structured log.
Another possible configuration would be Interpolation.getvalue when evaluated returns a Pydantic model, it can use that library's support for serializing to JSON. Lots of possibilities.

Now add the following to the above class:

    def __str__(self):
         # evaluate as if a lazy f-string

With __str__, this provides instant integration with logging given that LogRecord.getMessage calls str on the message.

Logging integration. The most straightforward way to do this is with setLogRecordFactory on a logger; this factory will be a subclass of LogRecord; it will have its own getMessage. This then works with a subclass of Formatter to build the final text representation, whether that's output of the above as if it were a f-string; or a JSON structured log version. I haven't used this library, but https://github.com/madzak/python-json-logger is one possible example for supporting such structured logging. There are also examples in the Logging Cookbook.

I'm also guessing projects like mypy will automatically detect (*args: Decoded | Interpolation) -> Any as potential tags.

That's an interesting idea. Definitely it goes the opposite way.

…ple, explain *args.) (#5) Co-authored-by: pauleveritt <[email protected]>

jimbaker · 2024-07-31T18:06:42Z

This PEP should be ready for approval, now that all earlier questions have been addressed.
@python/pep-editors

peps/pep-0750.rst

willingc

Overall, I like this PEP a lot. I've made some suggestions to help with clarity and distinction from f-strings.

peps/pep-0750.rst

Co-authored-by: Carol Willing <[email protected]>

@Carreau

* Include three comments from @Carreau (drop two bullets, *args in example, explain *args.) * Lambda-wrapped expressions use annotation scope * Clarify use of annotation scope * Mention what happens to named unicodes followed by text * Use DecodedConcrete in assertion * Rewrite why annotation scope is needed (#4) * Rewrite why annotation scope is needed * Minor copyediting * PEP 747: Fix rules related to UnionType (T1 | T2). Contrast TypeExpr with TypeAlias. Apply other feedback. (python#3856) * PEP 694: Fix typo (python#3859) * PEP 2026: Update following discussion (python#3860) Co-authored-by: Erlend E. Aasland <[email protected]> * PEP 101: Remove outdated info and add new info (python#3863) * PEP 101: Remove outdated info * PEP 101: Update make command for running tests * PEP 101: Replace '#python-dev and/or python-committers' with 'Discord and/or Discourse * PEP 101: Add Hugo as 3.14 RM * PEP 101: Add to PSRT * PEP 11: Add Russell as an iOS contact (python#3865) * Meta: Document the PEPs API (python#3864) Co-authored-by: Adam Turner <9087854 [email protected]> * PEP 719: Update for today's release of 3.13.0b4 (python#3868) * PEP 740: Mark as Provisional (python#3848) Signed-off-by: William Woodruff <[email protected]> * PEP 749: Add section on metaclasses (python#3847) Co-authored-by: Carl Meyer <[email protected]> * PEP 8: Update a Wikipedia link (python#3552) * PEP 635: Minor typo fix in code sample (python#3871) Looks like an unclosed f-string. * PEP 751: A file format to list Python dependencies for installation reproducibility (python#3870) Co-authored-by: Hugo van Kemenade <1324225 [email protected]> Co-authored-by: Adam Turner <9087854 [email protected]> Co-authored-by: Jelle Zijlstra <[email protected]> Co-authored-by: Carol Willing <[email protected]> * PEP 743: Rewrite to hide (soft-)deprecated API (pythonGH-3869) Co-authored-by: Victor Stinner <[email protected]> * PEP 751: Add Discussions-To and Post-History (python#3872) * PEP 639: Incorporate the latest discussion feedback (python#3866) * Remove the requirement of license-files defaults * Cover all rejected subkeysideas in one paragraph * Change the deprecation policy around classifiers * Flatten the value of the license-files key, only globs are specified * Update the Rejected ideas to match the current license-files proposal --------- Co-authored-by: Miro Hrončok <[email protected]> * PEP 715: clarify what `[package.tool]` is (python#3873) * PEP 665: Superseded-By: 751 (python#3875) * PEP 751: update based on feedback (python#3877) * PEP 751: update based on feedback * Fix a section underline * Include three comments from @Carreau (drop two bullets, *args in example, explain *args.) * From Carol, move the point about import to the following paragraph. * Per Carol: Remove paragraph about lifecycles as that is about *a* DSL, not DSLs in general. --------- Signed-off-by: William Woodruff <[email protected]> Co-authored-by: pauleveritt <[email protected]> Co-authored-by: Jim Baker <[email protected]> Co-authored-by: Lysandros Nikolaou <[email protected]> Co-authored-by: David Foster <[email protected]> Co-authored-by: Barry Warsaw <[email protected]> Co-authored-by: Hugo van Kemenade <1324225 [email protected]> Co-authored-by: Erlend E. Aasland <[email protected]> Co-authored-by: Adam Turner <9087854 [email protected]> Co-authored-by: T. Wouters <[email protected]> Co-authored-by: William Woodruff <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]> Co-authored-by: Carl Meyer <[email protected]> Co-authored-by: Lavrentiy Rubtsov <[email protected]> Co-authored-by: Mariatta <[email protected]> Co-authored-by: Brett Cannon <[email protected]> Co-authored-by: Carol Willing <[email protected]> Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Victor Stinner <[email protected]> Co-authored-by: Karolina Surma <33810531 [email protected]> Co-authored-by: Miro Hrončok <[email protected]>

willingc

Thanks for the changes.

lysnikolaou · 2024-08-09T14:42:21Z

Thanks everyone for the reviews! I'll merge this to get the conversation started and we can always come back to fix more issues.

jimbaker added 2 commits July 8, 2024 16:30

Add PEP authors and sponsor

0d36f4d

Add tagstr PEP

aacd309

jimbaker requested a review from a team as a code owner July 8, 2024 23:12

Fix PEP header ordering

ff3b6ee

JelleZijlstra reviewed Jul 8, 2024

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

Only include active core developers

9203527

JelleZijlstra reviewed Jul 8, 2024

View reviewed changes

peps/pep-0750.rst Show resolved Hide resolved

JelleZijlstra reviewed Jul 8, 2024

View reviewed changes

peps/pep-0750.rst Outdated Show resolved Hide resolved

jimbaker added 2 commits July 8, 2024 18:03

Fix link to reference implementation

08a74da

Fix markup of code from Markdown

9e70734

hugovk reviewed Jul 9, 2024

View reviewed changes

jimbaker and others added 17 commits July 9, 2024 16:22

Remove obsolete Content-Type header

7c53408

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Update creation date

aa1f9ed

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Better example code

951fc66

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

End the sentence with a period.

0223def

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

mytag -> my_tag

c460461

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

word choice

8ad4959

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Style

4938cdf

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Style

9fc2cea

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Typo

9fd3269

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Style

20991c9

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Mark up code

d2772a0

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Don't mark tool as code.

ea0251c

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Mark code as such

3a3d307

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Mark code as such

f37602b

Co-authored-by: Hugo van Kemenade <1324225 [email protected]>

Do not start f-string at the beginning of the sentence.

8e6b8c2

Lambda-wrapped expressions use class scope

7f42f89

Merge pull request #1 from jimbaker/class-scope-lambda-wrapped-expr

7c18570

Lambda-wrapped expressions use class scope

JelleZijlstra reviewed Jul 17, 2024

View reviewed changes

peps/pep-0750.rst Outdated Show resolved Hide resolved

Carreau reviewed Jul 23, 2024

View reviewed changes

peps/pep-0750.rst Outdated Show resolved Hide resolved

JelleZijlstra and others added 3 commits July 26, 2024 09:39

Merge branch 'main' into pep-750-tagstr

db0e420

Merge branch 'main' into pep-750-tagstr

477474e

Include three comments from @Carreau (drop two bullets, *args in exam…

ed7aabb

…ple, explain *args.) (#5) Co-authored-by: pauleveritt <[email protected]>

jimbaker added 2 commits July 31, 2024 12:07

Merge branch 'main' into pep-750-tagstr

7f3fda5

Merge branch 'main' into pep-750-tagstr

4c411fe