Support for common editors

[mzizzi]

How about trying to flesh this repository out with a few more grammars that support common editors? Having a few solid grammars for common editors in this repository might help aenea/dragonly newbies ease into development.

I hacked up an Intellij IDEA grammar this week and am open to tidying it up if we think it's worth the effort.

https://github.com/mzizzi/aenea-grammars/blob/mzizzi/_idea/_idea.py

[djr7C4]

Part of the problem is that anything generic will be unsuitable for a particular user's workflow. I have a fairly extensive Emacs grammar, but it is heavily dependent on my own customizations to Emacs. I imagine the same is true of other programmable editors.

[mzizzi]

I agree with you 100%. It would be very tough to get a single grammar for
any one editor that fit every workflow. Perhaps offering a few grammars
for different editors as "kick off" points for other users would ease new
users into the system.

Or maybe the target audience of aenea is just the type that is willing to
roll their own solution every time. It would be nice to start to see more
grammar reuse between users though. I've attempted to cut down on duplicate
code in my own grammars with a 'base' module for commonly repeated
elements. Maybe offering something as basic as that would prove helpful for
newcomers.
On May 25, 2015 3:35 PM, "djr7C4" notifications@github.com wrote:

Part of the problem is that anything generic will be unsuitable for a
particular user's workflow. I have a fairly extensive Emacs grammar, but it
is heavily dependent on my own customizations to Emacs. I imagine the same
is true of other programmable editors.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

[djr7C4]

I think the main problem is that the current facilities for customizing grammars are not powerful enough to allow code reuse. Basically, we need a way to add new mappings (this shouldn't be too hard to add) and override or disable existing ones dynamically from separate python or config files. The Aenea configuration system allows renaming commands, but it seems a bit fragile (e.g. if the author of the grammar changes the name of the command) and doesn't allow one to change the associated action or add new arguments to the command.

Given something like the above, I think we could have a common core for all grammars, and this would be a very good thing.

I also have my own base library, and adding something like this to aenea or aenea-grammars seems like a good idea.

[mzizzi]

Let's start with the what is easiest then. A solid base module would be easy enough to add.

Mind sharing yours? I believe I mentioned mine in a previous issue but here it is in its current state: https://github.com/mzizzi/aenea-grammars/tree/f2639d5dab654ceac4aeb592f3ffeee715e8e8e5/base

[djr7C4]

Yes, I remember that issue.

I've been meaning to release my code, but it's not ready yet and I do not have much time to work on it now. It will probably be as a separate project since it is already much larger than aenea-grammars, does not respect aenea conventions such as the Aenea config system (which I don't use for reasons mentioned above) and also so that I can have commit access to it.

[djr7C4]

I see customization as more important than a good base module. I worked a bit on something called polymorphic grammars that allow all of the above plus overriding rules in more specific contexts. Unfortunately, my initial version was unsatisfactory and I have not had time to rewrite it yet.

[calmofthestorm]

Have you looked at the vocabulary system? https://github.com/dictation-toolbox/aenea#taking-advantage-of-the-vocabulary-system? That and related are my "base". I created that as an attempt to unify my base system with that of @poppe1219 and @nirvdrum , but we were unable to come up with a system that made everyone happy.

The basis of this system was to decouple grammars (which would then be specific to a given editor/program/etc) and vocabulary (whether specific language keywords or IDE shortcuts, a similar principle). Vocabulary can be dynamically enabled or disabled (currently this is manual but it could very much be automated if one desired). So for example you could write an emacs vocabulary and then use it with the multiedit grammar. It is even possible to use simple rules in this manner, though to the best of my knowledge it is not possible to dynamically enable/disable subrules using Dragonfly/Natlink.

Eg, see:
https://github.com/dictation-toolbox/aenea-grammars/tree/master/vocabulary_config
https://github.com/dictation-toolbox/aenea-grammars/blob/master/vocabulary_config/dynamic/eclipse.json

This change also greatly complicated the system -- where before you could interuse Dragonfly and Aenea grammars, now you were dependent on the whole system to use anything. It might be possible to write them in a modular way, but that would greatly increase the effort.

My takeaway from that was that trying to come up with a common base was unlikely to succeed -- if only three of us couldn't do it for problems we'd each already solved, how could we also write a system that would work for new people and problems we were not yet aware of. You yourself have already mentioned not wanting to use the Aenea configuration system (and Aenea itself doesn't use the Dragonfly configuration system:P)I don't particularly like the world where everyone has their own system, but the problem is that in order to force consensus you need to be willing and able to break some people's workflow, and for a project like this this is neither necessary nor really feasible. Furthermore, very few other users even seemed aware of these systems -- to a great degree this seems like a reimplementation thereof. I concluded that the reason for this is that for something as simple as grammar configuration, learning to use an existing system with all its complexity was less attractive to users than simply reimplementing. I'm happy to be proven wrong here, but this isn't the first time I've had this discussion, and every time it happens without a discussion of the existing vocabulary system I feel it reinforces my point:P

I'm definitely open to improving the existing base, vocuablary and configuration system if it doesn't meet your needs, but I'd encourage you to take a look before reimplementing it/something like it. The grammar constructions @mzizzi posted are interesting and I'd be happy to consider standardizing on those, with the exception of format.py which already exists in nearly identical form: https://github.com/dictation-toolbox/aenea/blob/master/client/aenea/format.py

[calmofthestorm]

Also take a look at https://github.com/dictation-toolbox/dragonfly-scripts/tree/master/dynamics. It was my intent (dictation-toolbox/dragonfly-scripts#27) to integrate this into the new vocabulary system, but there were a few edge cases that would not have been supported, so this effort failed.

[calmofthestorm]

TL;DR I wrote an awesome vocabulary system that does the best it can within the constraints of the problem and tries to solve the general case of a number of things other people have tried to do, no one but me has ever cared, I'm a bit pouty:-)

[calmofthestorm]

@djr7C4 Regardless of whether we can come to an understanding in terms of making a combined base system, if you decide to go your own way, would you be interested in hosting the repo under dictation-toolbox? I'd have to clear it with my co-owners obviously, but it should be fine -- our goal was to have a one-stop-shop with related projects rather than a top-down cathedral of interoperability:-) All memebers have push access to all repos but there's an understanding that you only push directly to your own barring exceptional circumstances (eg, someone disappears)

[djr7C4]

@calmofthestorm I have looked at your vocabulary system, but if I understand it correctly, your goals are a bit different from what I am trying to do. Your main goal seems to me to be able to share terms (like language keywords) between different grammars. While I am sure that this is useful in some cases and could (and probably should be used in some of my langauge specific grammars), I don't think it quite does what I want.

For example, if there is some MappingRule in a grammar that contains a mapping like

"command arg" : action_that_requires_one_arg

then I'd like to be able to redefine it to

"command arg arg2" : action_that_requires_two_args

I would also like to be able to do things like the above, but only in more specific contexts. For example, the command might take one argument in the global context but use two within Emacs. I wrote some code a while ago to test that rules could be overridden in this way using facilities within dragonfly, but actually getting it to work automatically is more work, though I don't think there is any fundamental obstacle.

I do think that my polymorphic grammars would work with your vocabulary system without problems, so it should be possible to use both.

I would be happy to have it in the dictation-toolbox repository. That would probably also give it more visibility to new users, which would be benificial to my project. In any case, this probably won't happen for at least a while since things are crazy now, and I won't release it without consulting you first.

[djr7C4]

@calmofthestorm Also, you can enable/disable rules dynamically, but you cannot enable/disable subrules dynamically (as you say). My code for polymorphic grammars gets around this by defining its own grammar and rule classes that allow everything to be overriden or enabled/disabled. However, currently it has a bunch of problems and is in need of a rewrite (which I haven't had time to do yet).

[dylan-chong]

@calmofthestorm Just saw this thread. Regarding the vocabulary system, Does chaining commands work between vocabulary system and normal grammars? For example, I might want to say something like abbreviate def natword hello where the first two words are part of the static vocabulary, and the second two words are part of the Vim grammar.

[calmofthestorm]

Yes and no: basically the degree of support for vocabs depends on the grammar. Most of mine that support them do it by having a "top level" rule for them in the chaining rule that just enters them as plain text. What I don't think will work in any of mine would be if you wanted to compose them -- using the vocabularies inside another command (e.g., as an argument to a rule). You might be able to make a grammar that worked like that, but I'd worry about complexity issues.

Does that make sense? I feel like I'm kind of grasping for the right terms here:-)

[dylan-chong]

I didn't know you could have vocabulary as part of a rule! Is there an example with that somewhere?

…

On 21/12/2018, at 4:46 AM, Alex Roper ***@***.***> wrote: Yes and no: basically the degree of support for vocabs depends on the grammar. Most of mine that support them do it by having a "top level" rule for them in the chaining rule that just enters them as plain text. What I don't think will work in any of mine would be if you wanted to compose them -- using the vocabularies inside another command (e.g., as an argument to a rule). You might be able to make a grammar that worked like that, but I'd worry about complexity issues. Does that make sense? I feel like I'm kind of grasping for the right terms here:-) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[calmofthestorm]

It's been awhile, but I'd take a look at how multiedit integrates it:

https://github.com/dictation-toolbox/aenea-grammars/blob/master/_multiedit/_multiedit.py#L161

It's pretty complicated, but the basic intuition is that a grammar defines a tree of elements that compromise the grammar, and the vocabularies can be any node in that grammar. Typically this would work by just hooking into the chain rule, but you can do it anywhere in theory.

Be aware that in practice, depending on how large the vocabulary is, and how you use it in your grammar, you may run into grammar complexity issues with Natlink/Dragon.