To all Ren'Py users! We need your help! - Full Ren'Py grammar definition for 'intellisense'

[duckdoom4]

Let's add 'intellisense' features to the extension!

We're looking to improve this extension even further and would like your help!
Our next goal is to include behaviours you've come to expect from most IDEs which support intellisense features for popular languages. We want to add these features to the plugin as well.

As you know we already support some of these features, but these are limited in it's current state.
Some examples people have been asking for are:

• Rename symbols (F2)
  • Rename symbol (F2) #169
• A properly working document outliner
  • Empty outline when opening folder #151
• Properly working go to definition (F12)
• Better error/warning detection and syntax errors
  • False error in comments #171
  • [Bug (Minor)] False warning flag with persistent variables defaulted in other source files #189
  • Incorrect indentation error when spanning several lines #201
  • Suggestion: Generate a warning for unmatched {color=#......} tags #255
  • False identification of '"$" starts a one-line Python statement' with multi-line say statement #289
• Support highlighting of custom keywords
  • Missing highlighting for custom keywords #203
• All features working on a single .rpy file (without the need for Ren'Py static data)
  • Old issue #180/#204 has resurfaced - Infinate "Initializing Ren'Py static data..." #332
  • I get infinity status "Initializing Ren'Py static data..." #180
  • Question: Statusbar keeps showing "Initializing Ren'Py static data..."  #204

How can you help

We are currently just two people building/maintaining this large extension.
With some help from other contributors (Special thanks to @a2937, @seanj29, @Booplicate, @pjpollina, and @Gouvernathor for your recent help! ❤️ 🚀)

To implement all these features we need a full formal grammar definition of the Ren'Py language. This will ensure that we implement these features as they should be implemented.

Just like how there already exists a Full Python grammar specification, we will need to build a Full Ren'Py grammar specification.

Now for some good news:
Ren'Py already has some formal specification! (ATL is fully specified)

Here is an example of the Transform Statement in BNF notation

atl_transform ::=  "transform" qualname ( "(" parameters ")" )? ":"
                      atl_block

Just like ATL, we will need this specification for all other renpy features.

Any help is greatly appreciated! ⭐
You can help us by adding or reviewing to the grammar specification or simply supplying us with renpy code samples showing how each language feature is used in your project.
If you want to help us out, please submit your contributions to this branch. Or leave a comment down below.

[duckdoom4]

@renpytom / @Gouvernathor does such a specification already exist? That would be very helpfull :)

[Gouvernathor]

It came up some time ago for some reason and the answer was no, I doubt it has been done since. There were some bugs (in renpy/renpy#3991 for example) where a grammar would certainly have helped flagging them, but that example also means we are changing it occasionally. And it's possibly a bag of knots already, though I'm not a connaisseur of BNF so I may be wrong as compared to other languages. Our parser code is not that long and quite regular, so who knows.

In any case I guess that would require warning you when we make changes to the grammar, preferably before release. It will require some discipline on our end.

[duckdoom4]

In any case I guess that would require warning you when we make changes to the grammar, preferably before release. It will require some discipline on our end.

Maybe we could build an official specification together? That would help on both ends I suppose.

[duckdoom4]

@Gouvernathor, I used Bing Chat (🤣) to extract some EBNF definitions from the official Ren'Py parser source. It wasn't 100% accurate, so after it had processed most of the statements I manually went over them.
I've uploaded the very rough draft here.

The entries at the bottom of the file (after # Renpy Expressions) are the ones I manually reviewed and improved based on the source code. A review would be appreciated, but feel free to do so later :). I will continue to work on this for the next few days

[Gouvernathor]

Some notes about those :

• what we consider to be a string doesn't always follow python rules. For example, we allow backquotes as delimiters for dialogue lines, line breaking behavior is... let's say weird (we allow it for not-triple strings), and we usually don't support f, r and u as prepended. All that, at least for dialogue lines, I'm not sure if it's specific or if shared among several statements parsing.
• the python block is not exactly as you describe : "hide" and "in x" are mutually exclusive, and I doubt "early" takes either of the two (gotta check)
• "end translate" is something we implicitly generate in AST, but it's not represented with characters in the file : the end translate of a non-None language is a block ending, and of a None translate (implicit too), it's just the end of a dialogue line. So it should probably not be in your list.
• del is not a renpy statement, it's a python statement. Not sure why it ended there.
• the SCREEN_ section looks wrong

Ill increment this list.

[duckdoom4]

Ha, yes. I see it did a really bad job on the python statement. There is a lot more to it then this. Good catch

[duckdoom4]

@Gouvernathor Do I read correctly that the current parser will allow any statement to start with init?
https://github.com/renpy/renpy/blob/a33f891d65d3fd2b59a533c365800a59f8e15f0c/renpy/parser.py#L1167

So that would mean things like:

init $ <python_expression>
init init label
init return
init init init init ...

Would all be valid? (Note that this is not an init block!)

Bug? Yes it is :) renpy/renpy#4713

[Gouvernathor]

The init $ is probably intended, sort of. And archaic. The others, probably not.

[duckdoom4]

The init $ is probably intended, sort of. And archaic. The others, probably not.

Yeah, kinda makes sense.

I've updated the grammar file again. With the exception of screen and translate, all 'normal' (meaning non screen- or ATL blocks) are now in there. Another review would be appreciated :)

(You can ignore the bits below (* GENERATED *), I'm gonna manually convert the code anyway)

[Gouvernathor]

I think LABEL_NAME should be [[NAME], "."], NAME. Assuming I understand the syntax correctly, there can't be label .name.be.

image_specifier seems wrong in that it seems to allow several behind_expression (for example) to be parsed in a row, as different image_specifier_clause. It needs to allow at most one of each, in any order. I'm not sure how to specify that, though.
Same with menu_block, possibly. If not it's our fault enabling set and caption to come before choices. But in any case there can be only one caption/say in a given menu statement.
Same with style_clause and probably others. Maybe I don't understand the bnf syntax though.

say_attributes should be {["-"], IMAGE_NAME_COMPONENT_NO_DASH}, I believe. The dashes can only be at the beginning of image attributes, and there can be any number of negated attributes.

The layeredimage statement is missing, as well as a number of statements defined in renpy/common : the audio statements (some of them at least) are part of those I think. They are UserStatements, technically, but they're also built-in.

It's just cosmetic maybe, but I wouldn't put the if_expression in common between the if clause and the menu statement, for me it's just a common keyword just as the in operator and the for x in y syntax in python. But it's just my view of things, it works the same.

rpy_python is just "rpy", "python", "3". Until python 4 comes out, I guess. Though I may want a rpy python annotations, but we'll see.
testcase is still work in progress, not merged yet and the semantics and grammar may still evolve.

[renpytom]

Question - where is the grammar located? I'd be interested in taking a look at it, and seeing what I can do.

[duckdoom4]

Question - where is the grammar located? I'd be interested in taking a look at it, and seeing what I can do.

I have the grammars located on this branch
(renpy-grammar-specification/grammars/).

Thanks in advance for taking the time to help :)

[multimokia]

Can I put in a request as well, kinda like typescript has or even pylint allowing @ignore lines? When working on mods the undefined storemodule stuff, while it's technically correct, in practice isn't due to the presence of base scripts (inside .rpa files)

As such it leads to unnecessary warnings, and it'd be great to kinda have a # @renpy-ignore-next-line or some more fine tuned diagnostic rules, if not just # @renpy-expect-error <description> i.e. TS

[duckdoom4]

Can I put in a request as well, kinda like typescript has or even pylint allowing @ignore lines? When working on mods the undefined storemodule stuff, while it's technically correct, in practice isn't due to the presence of base scripts (inside .rpa files)

As such it leads to unnecessary warnings, and it'd be great to kinda have a # @renpy-ignore-next-line or some more fine tuned diagnostic rules, if not just # @renpy-expect-error <description> i.e. TS

Yeah we can implement this feature when the other features are implemented. I think this is a nice to have

[duckdoom4]

Something to look into:
https://code.visualstudio.com/api/extension-guides/virtual-documents
https://code.visualstudio.com/api/language-extensions/embedded-languages

Basically we could already add python's language server features by building a virtual document of the python code and then adding the results to our own providers. This would be incomplete though, since we'd be missing a lot of the python context. It might still be nice enough to be worth adding already