Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User study bugfix #41

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

User study bugfix #41

wants to merge 4 commits into from

Conversation

Ar4l
Copy link
Collaborator

@Ar4l Ar4l commented Mar 20, 2024

Fixes/updates the following server-side components:

  • Upgrade CUDA drivers 11.4 -> 12.2 and NVIDIA to gpgpu (not actually part of this PR code-wise; but was necessary)

  • use vLLM for batching requests and Paged Attention. Engines at 0.9 fractional GPU utilisation; 20GB swap space.

  • Add StarCoder2-3b as a backend model, replacing CodeGPT and UniXCoder.

    • Load in float16.
    • Ensure infilling-mode works correctly (see hf thread)
  • Why do we store ground truths only for accepted completions?

  • Store v1 user requests under data/user_uuid/json_uuid.json, to avoid counting all invocations on every request. However, this brings two issues:

    • Need to convert previous data from user_uuid-json_uuid.json to user_uuid/json_uuid.json; but this can be done with a simple replacement command on the server.
    • Mark's data analysis scripts may need to be updated to follow this convention. (the first thing mine do is sort the data into this user/json structure to make processing locally manageable).
  • Fix User Study passthrough filter; I forgot to save before amending my last commit on the aral_user_study branch.

Client side (vsc):

  • Fix shown_times is used before declared.

@FrankHeijden
Copy link
Collaborator

FrankHeijden commented Mar 20, 2024

You can also cache the counts per user for the current runtime; that way the user's folder is only globbed once. This can be done since there's only one instance of the app running on the backend, so we don't run into the split-brain problem. The counts can then be incremented in memory when a new completion is done. Ideally, all this data should actually be stored in a proper database instead of a flat file system, but I am not sure if that's a transition it is worthwhile to make at this stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants