Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include Vespa Lexical Search as an option to BEIR benchmark #76

Open
wants to merge 32 commits into
base: main
Choose a base branch
from

Conversation

thigm85
Copy link

@thigm85 thigm85 commented Mar 1, 2022

@NThakur20 could you take a look at this PR? The idea is to make it easier to benchmark Vespa applications using the BEIR datasets and framework. We started with Lexical Search but will make it more general later.

thigm85 and others added 30 commits January 24, 2022 08:38
Introduce Vespa Lexical experiment into Beir
@thakur-nandan
Copy link
Member

Hi @thigm85, thank you for providing this PR!

in the next upcoming days, I will have a look at the PR. I'm happy to see Vespa being included within BEIR!

I was busy shifting the repository last few weeks. Will soon merge with the development branch and later release with the next version of beir updates!

Kind Regards,
Nandan Thakur

@thakur-nandan thakur-nandan self-assigned this Mar 30, 2022
Copy link
Member

@thakur-nandan thakur-nandan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @thigm85, I went through your PR. Thanks for all the code added and it looks good.

I have a few updates mentioned here.

  1. in setup.py can you add pyvespa and tenacity as optional dependencies something similar to what I have done for tensorflow? Something like:
optional_packages = {
    "tf" : ['tensorflow>=2.2.0', 'tensorflow-text', 'tensorflow-hub'],
    "vespa": ["pyvespa", "tenacity"]
}
  1. Can you create a tests folder within the main directory of BEIR and place test_retrieval_lexical_vespa.py inside this tests folder. I currently do not have any unittests implemented for other methods, this will unify in future all unittests at one place.

  2. could you shift your example benchmark_lexical_vespa.py from examples/benchmarking to examples/retrieval/evaluation/lexical because the other place contains all sample scripts to evaluate different lexical search and will be easier for the user to find it.

  3. Could you mention briefly on top of benchmark_lexical_vespa.py a few steps on how to run vespa lexical search? What must a user should have in place to run vespa search? Or how to download and run the vespa application? You can have a look at evaluate_bm25.py for reference.

Thank you! Will merge once it the small updates mentioned has been resolved.

Kind Regards,
Nandan Thakur

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants