Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement on Spot batch VLM predicate evaluation pipeline #285

Open
wants to merge 48 commits into
base: master
Choose a base branch
from
Open

Implement on Spot batch VLM predicate evaluation pipeline #285

wants to merge 48 commits into from

Commits on Apr 23, 2024

  1. try to visualize sokoban - may need nore neat way

    @lf-zhao
    lf-zhao committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    0824ae7 View commit details
    Browse the repository at this point in the history

  2. add assert for fast downward planner in running once func

    @lf-zhao
    lf-zhao committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    bf4000d View commit details
    Browse the repository at this point in the history

  3. try to use rich package for more structured console output!

    @lf-zhao
    lf-zhao committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    19d9d53 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. upload a naive way to store images

    @lf-zhao
    lf-zhao committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    feaa264 View commit details
    Browse the repository at this point in the history

  2. debug

    @lf-zhao
    lf-zhao committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    dd1fbf6 View commit details
    Browse the repository at this point in the history

  3. upload - manual copy from Nishanth's VLM interface in LIS predicators… 

    …; we need to figure out a better way to sync later!
    @lf-zhao
    lf-zhao committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    32a06a3 View commit details
    Browse the repository at this point in the history

  4. add OpenAI vlm - in progress

    @lf-zhao
    lf-zhao committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    ee3f634 View commit details
    Browse the repository at this point in the history

  5. update config setting for using vlm

    @lf-zhao
    lf-zhao committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    a1a67fd View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. add package

    @lf-zhao
    lf-zhao committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    5d9f12e View commit details
    Browse the repository at this point in the history

  2. another missed one

    @lf-zhao
    lf-zhao committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    cf3dbf7 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. manually add Nishanth's new pretrained model interface for now, see L… 

    …IS predicators!
    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    3001f32 View commit details
    Browse the repository at this point in the history

  2. add new OpenAI VLM class, add example to use

    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    4d47211 View commit details
    Browse the repository at this point in the history

  3. add a flag for caching

    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e45a0e9 View commit details
    Browse the repository at this point in the history

  4. now the example working - fix requesting vision messages, update test

    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    8786dcd View commit details
    Browse the repository at this point in the history

  5. update; add choosing img detail quality

    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    71d0b3e View commit details
    Browse the repository at this point in the history

  6. include the test image I used for now, not sure what I should do with… 

    … this image
    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    112b690 View commit details
    Browse the repository at this point in the history

  7. remove original vlm interface, already merged into latest pretrained … 

    …py one
    @lf-zhao
    lf-zhao committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    f884df7 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Merge branch 'refs/heads/master' into lis-spot/implement-vlm-predicat… 

    …e-eval

# Conflicts:
#	predicators/pretrained_model_interface.py
#	setup.py
    @lf-zhao
    lf-zhao committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    8c17c42 View commit details
    Browse the repository at this point in the history

  2. found a way to use VLM to evaluate; add current images and also visib… 

    …le objects
    @lf-zhao
    lf-zhao committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    aec70de View commit details
    Browse the repository at this point in the history

  3. found a way to use VLM to evaluate; check if visible in current scene… 

    …, only update these predicates
    @lf-zhao
    lf-zhao committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    94e6a4c View commit details
    Browse the repository at this point in the history

  4. update State struct; adding to Spot specific subclass doesn't work, n… 

    …o idea why
    @lf-zhao
    lf-zhao committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    3ee2ba9 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. add detail option

    @lf-zhao
    lf-zhao committed May 1, 2024
    Configuration menu
    Copy the full SHA
    1abf488 View commit details
    Browse the repository at this point in the history

  2. working; implement On predicate with VLM classifier pipeline! add cal… 

    …l VLM and more
    @lf-zhao
    lf-zhao committed May 1, 2024
    Configuration menu
    Copy the full SHA
    1c82c44 View commit details
    Browse the repository at this point in the history

  3. make a separate function for vlm predicate classifier evaluation

    @lf-zhao
    lf-zhao committed May 1, 2024
    Configuration menu
    Copy the full SHA
    eeb1583 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. add test

    @lf-zhao
    lf-zhao committed May 3, 2024
    Configuration menu
    Copy the full SHA
    acbdb0a View commit details
    Browse the repository at this point in the history

  2. update example, move to test, move img

    @lf-zhao
    lf-zhao committed May 3, 2024
    Configuration menu
    Copy the full SHA
    01498f6 View commit details
    Browse the repository at this point in the history

  3. remove

    @lf-zhao
    lf-zhao committed May 3, 2024
    Configuration menu
    Copy the full SHA
    0774354 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2024

  1. format

    @lf-zhao
    lf-zhao committed May 4, 2024
    Configuration menu
    Copy the full SHA
    68ef57d View commit details
    Browse the repository at this point in the history

  2. update

    @lf-zhao
    lf-zhao committed May 4, 2024
    Configuration menu
    Copy the full SHA
    6560a94 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. batch VLM classifier working on Spot!! add field to State, add VLMPre… 

    …dicate and VLMGroundAtom
    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    1596f68 View commit details
    Browse the repository at this point in the history

  2. batch VLM classifier eval: add vlm predicates fields to observation

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    98ae2a9 View commit details
    Browse the repository at this point in the history

  3. batch VLM classifier eval: function on batch query and parse

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    2683a6b View commit details
    Browse the repository at this point in the history

  4. batch VLM classifier eval: provide VLM predicates to object finding, … 

    …evaluate all VLM predicates on all objects to build init obs for env reset
    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    b0b7568 View commit details
    Browse the repository at this point in the history

  5. batch VLM classifier eval: add VLM predicate fields to state+obs, bui… 

    …ld init obs that query all objects + all VLM predicates, update VLM ground atoms for visible objects, update VLM predicate classifiers
    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    b4718f7 View commit details
    Browse the repository at this point in the history

  6. formatting

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    3ca9032 View commit details
    Browse the repository at this point in the history

  7. remove some comments

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    1ab3d36 View commit details
    Browse the repository at this point in the history

  8. remove some comments

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    11f91ff View commit details
    Browse the repository at this point in the history

  9. fix, add tenacity

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    c98e2fe View commit details
    Browse the repository at this point in the history

  10. fix structs

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    4973f1f View commit details
    Browse the repository at this point in the history

  11. more fix

    @lf-zhao
    lf-zhao committed May 8, 2024
    Configuration menu
    Copy the full SHA
    26239e8 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. update

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    e78e342 View commit details
    Browse the repository at this point in the history

  2. some clean

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    ce43a76 View commit details
    Browse the repository at this point in the history

  3. fix no VLM case

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    93fb574 View commit details
    Browse the repository at this point in the history

  4. add predicate prompt; fix and clean

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    cb0e1ee View commit details
    Browse the repository at this point in the history

  5. add predicate prompt & some logging; fix and clean

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    63a1429 View commit details
    Browse the repository at this point in the history

  6. overwrite vlm predicate classifier; reformat

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    ba2575f View commit details
    Browse the repository at this point in the history

  7. update

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    a11e5a1 View commit details
    Browse the repository at this point in the history

  8. update vlm query in obj finding

    @lf-zhao
    lf-zhao committed May 9, 2024
    Configuration menu
    Copy the full SHA
    8ba4c5d View commit details
    Browse the repository at this point in the history