Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: experimental_eval.py cannot be successfully run with 0.0.11 or 0.1.0-rcx #200

Open
lwaekfjlk opened this issue Sep 26, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@lwaekfjlk
Copy link
Member

lwaekfjlk commented Sep 26, 2024

Description of the bug

class name including EvaluationForTwoAgents is not supported

Steps To Reproduce

run experimental_eval.py with sotopia==0.0.11

independently install sotopia

python experimental_eval.py
--gin_file sotopia_conf/generation_utils_conf/generate.gin
--gin_file sotopia_conf/server_conf/server.gin
--gin_file sotopia_conf/run_async_server_in_batch.gin
'--gin.ENV_IDS=["01H7VFHNV13MHN97GAH73E3KM8", "01H7VFHN5WVC5HKKVBHZBA553R", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHN9W0WAFZCBT09PKJJNK", "01H7VFHPDZVVCDZR3AARA547CY", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHPQQQY6H4DNC6NBQ8XTG", "01H7VFHN7WJK7VWVRZZTQ6DX9T", "01H7VFHN7A1ZX5KSMT2YN9RXC4", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHPS5WJW2694R1MNC8JFY", "01H7VFHNN7XTR99319DS8KZCQM", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHQ11NAMZS4A2RDGDB01V", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHPSWGDGEYRP63H2DJKV0", "01H7VFHNF4G18PC9JHGRC8A1R6", "01H7VFHNNYH3W0VRWVY178K2TK", "01H7VFHP8AN5643B0NR0NP00VE", "01H7VFHN7A1ZX5KSMT2YN9RXC4"]'
'--gin.AGENT1_MODEL="gpt-3.5-turbo"'
'--gin.AGENT2_MODEL="gpt-3.5-turbo"'
'--gin.BATCH_SIZE=1'
'--gin.PUSH_TO_DB=True'
'--gin.TAG="gpt-3.5-turbo_vs_gpt-3.5-turbo-0927"'

Additional Information

No response

@lwaekfjlk lwaekfjlk added the bug Something isn't working label Sep 26, 2024
@lwaekfjlk lwaekfjlk changed the title [BUG]: experimental_eval.py cannot be successfully run with 0.0.11 [BUG]: experimental_eval.py cannot be successfully run with 0.0.11 or 0.1.0-rcx Sep 27, 2024
@XuhuiZhou
Copy link
Member

@lwaekfjlk is this fixed or still there? imo it's bc we removed the retrieved prompt there and we are still figuring out a way to fix this.

@lwaekfjlk
Copy link
Member Author

still there I believe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants