Improved types for Literal[<some string>].format

[rsokl]

I work with a code base where we often pass around many different string literals that contain format fields. In order to resolve the strings with .format we have to look at the string's definition and fill out the method manually. This is quite unwieldy and hard to maintain.

Would it be possible / not-to-difficult for pyright to provide enhanced typed information for Literal[...].format? E.g. given

some_str: Final = "{}, {x}, {}, {y}"

then some_str.format would have the following overloads

@overload
def format(a: LiteralString, b: LiteralString, /, *, x: LiteralString, y: LiteralString) -> LiteralString: ...

@overload
def format(a: object, b: object, /, *, x: object, y: object) -> str: ...

[erictraut]

Thanks for the suggestion.

I can think of various types of support for str.format, so let me ask some follow-up questions to understand which of these is most important to you.

1. More precise type checking for arguments passed to str.format. In other words, you want pyright to report type violations when you pass arguments that don't correspond to symbols within the format string literal or you omit a required argument.
2. Signature help in the language server. As you start to type the arguments in the call expression, you want to see the "signature help" tooltip change to indicate which keyword arguments need to be supplied.
3. Hover text in the language server. When you hover over the word format (e.g. in the statement some_str.format(...)) you want the reported type to be def format(a: LiteralString, b: LiteralString, /, *, x: LiteralString, y: LiteralString.
4. Literal math. You want pyright to "compute" the literal string that the runtime will create as a result of calling str.format and use the literal result for type narrowing on assignment or in other literal math operations, such as "x = " + {}.format(x).
5. Validation of the format string itself. Format strings must follow certain rules and are rejected if they don't. For example, "{x }" is rejected because it has an extra space after the x, and "{0}, {}" is rejected because you cannot switch from manual field specification to automatic field number.

I'm guessing that 1 is most important to you, 2 is a "nice to have", and 3, 4 and 5 are less important. Is that a good guess?

Item 1 would require some special-casing of the str.format function within pyright. We generally don't like to special-case functions because it's not scalable and often leads to problems when we ignore or override the declarative definitions in typeshed stubs. However, there is precedent for providing special casing in specific cases, such as for functools.partial. I think there's a compelling case to be made for special-casing str.format. It's pretty commonly used (although waning somewhat since the introduction of f-fstrings). Also, mypy has special-case support for it.

Items 2 and 3 would require special-casing in the language server (with potential support from the type checking logic). This is probably not something we'd do in pyright, but it's something the pylance team might consider adding. If you want one or both of these, I recommend filing a feature request in the pylance-release issue tracker.

Item 4 would require special-casing in pyright. I'm reluctant to add this functionality because of the complexities involved in format specifiers. The edge cases for format specifiers are not well documented even for built-in classes like str. So I think this would be difficult to get correct in the general case, and it would be a source of a long tail of bugs.

Item 5 would be possible if the rules were documented, but I don't think they are. One could look at the implementation in Python to try to reverse engineer them, but if they're not documented as part of the interface contract for str.format, these rules could vary across Python versions.

[rsokl]

I'm guessing that 1 is most important to you, 2 is a "nice to have", and 3, 4 and 5 are less important. Is that a good guess?

Yep, that is right! 1 is by far the most important, but 2 would also be a major improvement to UX; to such an extent that I could imagine exposing strings-to-be-formatted directly to users if I knew that they would have signature help to guide them. If you decide to move forward with 1, I will open an issue with pylance about 2 (unless these are independent of one another).

I think there's a compelling case to be made for special-casing str.format. It's pretty commonly used (although waning somewhat since the introduction of f-fstrings).

In my case I am working with large language models and need to work with many prompts, each of which have different format fields that will be filled out downstream. So f-string aren't helpful here. I imagine that there is probably a surge in this style of workflow right now.

Item 4 would definitely be neat, but I can't say that it would be all that impactful for me and my colleagues.

Regarding Item 5, it seems like implementing Item 1 would necessarily catch some of the rules for Item 5. E.g. if I write '{1x}'.format, then the signature would ask for a parameter named 1x, which would be detected as an illegal name. Is that correct? Regardless, I don't think item 5 is important compared to 1.

[erictraut]

Thanks for the additional information.

It would be a fair amount of work to add this support (even limited to item 1). I've searched through all of the issues and discussion posts in the pyright and pylance issue trackers, and this is the first time we've received such a request. I'm open to investing in a solution here if we receive signal that it will be more broadly useful to other pyright or pylance users. Mypy has support for it, which lends it some support, but I'd like to see additional requests (in the form of thumbs up reactions) before committing to make this investment.