Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize filter queries #37

Open
yawor opened this issue Jan 4, 2012 · 1 comment
Open

Optimize filter queries #37

yawor opened this issue Jan 4, 2012 · 1 comment

Comments

@yawor
Copy link

yawor commented Jan 4, 2012

The way sunburnt uses filter queries (fq parameter) is far from optimal from caching point of view.

Solr allows to use more than one fq parameter to provide many filter queries which are executed and cached separately and then intersected using very fast algorithms.

Sunburnt does not give possibility to build a query with more than one fq parameter.

For example:

`````` si.query().filter(field1=True).filter(field2=False)will produce something like this:?q=:&fq=field1:true AND field2:false```
but, from caching point of view, it would be much better to produce something like this:
```?q=:&fq=field1:true&fq=field2:false```

Both cases will give the same results, but in first case Solr would cache and reuse resultset for field1:true AND field2:false as one. In second case there would be 2 separate resultsets cached and reused for field1 and field2.

After changing the query like this:
si.query().filter(field1=True).filter(field2=True)
the cache for field1:true would be already populated and reused requiring only executing query and populating cache for field2:true.

This change would still allow to build a query with "AND" in fq like this:
si.query().filter(si.Q(field1=True) & si.Q(field2=False))

@nidico
Copy link

nidico commented Feb 7, 2013

Allowing multiple fq query parts would indeed be very useful. Apart from performance optimization, this would be a first step in order to implement LocalParamas (#6).

nidico added a commit to liqd/adhocracy that referenced this issue Feb 8, 2013
The solution to calculate facet item counts for facets with exclusive
values in 9e7c7c4 was wrong and only
worked for simple cases.

The general solution in this commit instead does the following:

For each facet with exclusive values, create another solr query, which
is the same as the current main query (facet_query), but without the
filter on that same facet.

This would normally be done in solr with `LocalParams`, but this isn't
yet implemented in sunburnt (see tow/sunburnt#6). A workaround (as
described in the latter issue) would be possible if sunburnt would allow
to create multiple filter query parts instead of putting them together
in one filter query with `AND` constructs (see tow/sunburnt#37).
nidico added a commit to liqd/adhocracy that referenced this issue Feb 22, 2013
The solution to calculate facet item counts for facets with exclusive
values in 9e7c7c4 was wrong and only
worked for simple cases.

The general solution in this commit instead does the following:

For each facet with exclusive values, create another solr query, which
is the same as the current main query (facet_query), but without the
filter on that same facet.

This would normally be done in solr with `LocalParams`, but this isn't
yet implemented in sunburnt (see tow/sunburnt#6). A workaround (as
described in the latter issue) would be possible if sunburnt would allow
to create multiple filter query parts instead of putting them together
in one filter query with `AND` constructs (see tow/sunburnt#37).
nidico added a commit to liqd/adhocracy that referenced this issue Mar 10, 2013
The solution to calculate facet item counts for facets with exclusive
values in 9e7c7c4 was wrong and only
worked for simple cases.

The general solution in this commit instead does the following:

For each facet with exclusive values, create another solr query, which
is the same as the current main query (facet_query), but without the
filter on that same facet.

This would normally be done in solr with `LocalParams`, but this isn't
yet implemented in sunburnt (see tow/sunburnt#6). A workaround (as
described in the latter issue) would be possible if sunburnt would allow
to create multiple filter query parts instead of putting them together
in one filter query with `AND` constructs (see tow/sunburnt#37).
nidico added a commit to liqd/adhocracy that referenced this issue Mar 10, 2013
The solution to calculate facet item counts for facets with exclusive
values in 3b27c3a was wrong and only
worked for simple cases.

The general solution in this commit instead does the following:

For each facet with exclusive values, create another solr query, which
is the same as the current main query (facet_query), but without the
filter on that same facet.

This would normally be done in solr with `LocalParams`, but this isn't
yet implemented in sunburnt (see tow/sunburnt#6). A workaround (as
described in the latter issue) would be possible if sunburnt would allow
to create multiple filter query parts instead of putting them together
in one filter query with `AND` constructs (see tow/sunburnt#37).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants