Implementation Discussion for Message Filtering

[96-LB]

I'm opening this discussion thread to explore possible approaches to implementing a solution for #160. The ability to filter which messages are exported and downloaded is perhaps one of the most requested features.

Relevant issues include: #28 #178 #219 #336 #343 #361 #384 #430 #566

---

As the scope of these requests varies from "export the messages of a specific user" to "compound filters with boolean operators", we should probably first determine to what extent the feature would encompass. I'm personally in favor of adding a query parameter which uses syntax similar to the search bar in the Discord client (e.g. from:236257776421175296 before:2021), but as far as I know, Discord doesn't expose their search function in their API. Discord's search is also complex (e.g. supports spaces like from:96 LB#7383), quirky (e.g. searching ring brings up results for ringing but not r), and limited (no support for "OR" or "NOT" queries). As such, implementing a close copy would be basically impossible.

I think we could reasonably implement a simple key:value query syntax. I did an extremely cursory search for useful libraries and nothing stood out, but I'm sure something like that exists. It probably wouldn't be difficult to build on our own either; though it may be a little more of a challenge to search for text the way Discord does unless we add our own properties such as contains: or startswith: (which I don't think is a bad idea).

There's a lot to discuss with this feature, so hopefully this gets the ball rolling.

[Tyrrrz]

Hello!

I'm personally in favor of adding a query parameter which uses syntax similar to the search bar in the Discord client

Yes, I think it's the way to go. This means we won't have to come up with something entirely new ourselves and most users will be familiar with the syntax.

Discord doesn't expose their search function in their API. Discord's search is also complex (e.g. supports spaces like from:96 LB#7383), quirky (e.g. searching ring brings up results for ringing but not r), and limited (no support for "OR" or "NOT" queries).

1. I believe the from:96 LB#7383 thing works when you type and select the user from a dropdown. I think in such case Discord treats the object you selected as a token and not just as a string of text, so it disregards the spaces. In our case we won't have a dropdown, so escaping spaces with double quotes can be a reasonable workaround.
2. Fuzzy search (i.e. ringing/ring) can definitely be a stretch goal. We don't have to make a 100% copy of Discord's search query syntax, but the closer we can get, the better.

I think we could reasonably implement a simple key:value query syntax. I did an extremely cursory search for useful libraries and nothing stood out, but I'm sure something like that exists. It probably wouldn't be difficult to build on our own either; though it may be a little more of a challenge to search for text the way Discord does unless we add our own properties such as contains: or `startswith: (which I don't think is a bad idea).

From the outset it looks like even simple regex expressions can do because there aren't any nested tokens, so a recursive-descent parser (like the one I built for markdown) is not necessary.

[haiitsstrawberry]

i think it would also be helpful if you had something like user:bot or user:webhook and the startswith: is useful because it can filter out bot commands(which is what i need in my situation) also for the logical operators i think if it isn't specified you can just take what discord defaults when searched e.g. strand (user:foo user:bar) searches the word strand from either user foo or user bar

[96-LB]

Yeah, I'm in support of eventually integrating bot/user/webhook filters and some more advanced text matching at some point in the future. Also, I just realized that Discord actually uses the from: keyword instead of user: like we've been saying 😅. Should we include an alias for that?

[haiitsstrawberry]

Woops, i mixed up from: and user:, how about from: specifies which user(LB, Tyrrrz) and user: specifies which kind of user(bot/webhook/user). For aliases idk

[96-LB]

I think user: could easily get mixed up with from: if we do that. I'd prefer something naming it something like type:.

[haiitsstrawberry]

sounds good