Paper writing

[ft2023]

Outline:

1 Introduction

2 Related Work

Research agent/tools
Human-AI Interaction

3 Research Town Pipeline and Dataset

3.1 Overview of Research Town

3.2 Academic Paper Writing
Idea Generation
Paper Discussion and Writing

3.3 Paper Submission and Review
Reviewer Assignment
Reviewer Discussion

4 Experiments

4.1 Baselines

4.2 Evaluation
Metric Eval
AI Eval
Human Eval

4.3 Result Analysis

4.4 New Findings from Research Town
Research Idea Verification/Prediction

[lwaekfjlk]

RQ1: Can we use multi-agent system to simulate research community process?

RQ2: Can simulated research communtiy generate useful information for academic research?

[ft2023]

Let us discuss the five questions for our paper.
1.What is the problem?

2.Why is it interesting and important?

3.Why is it hard? (E.g., why do naive approaches fail?)

4.Why hasn‘t it been solved before? (Or, what’s wrong with previous proposed solutions? How does mine differ?)

5.What are the key components of my approach and results? Also include any specific limitations.

[ft2023]

Accelerate

Goal: Accelerate knowledge/info creation and sharing for researchers

What we have done

1 Get “Researcher Avatar” for researchers (10K)

2 Simulate researchers reading latest papers

Idea 1: Traditional research pipeline (NeurIPS benchmark)

• Read paper
• Brainstorm (ideation)
• Write paper
  • Write paper abstract
• Review
• Rebuttal
• Meta-review
• Presentation
• Sharing

Back-testing: for researchers

• Dynamic

Suggestion:

• Text2PPT
• Community-level analysis
• We are building a research town simulation framework
  • Input: list of researcher names
  • Case study
    • Focus on a smaller domain
    • General domain
    • Interdisciplinary domain
  • Output/Evaluation
    • General quality: Human, GPT score
    • Hold-out testing Truthfully: verify existing hold-out ideas
    • Insightful: can research town propose verifiable new ideas
• Simulate a “Twitter/Zhihu” platform for researchers
  • Graph metrics: measure knowledge sharing speed
• CHI: LLM conflict
  • Teach researcher how to do research
  • Accelerate / improve research skills

Research town framework - involved platforms:

• Research writing workflow: read, social, collaborate, compete, write
• Peer-review workflow: submit, review, rebuttal, meta, decision
  • Assumption: over-confident?
• PR/social workflow: social, PR, feedback
• Additional analysis: show our env is realistic
  • ICLR keyword ranking
  • Acceptance rate vs popularity

Release a realistic multi-agent environment

class SingleAgent

class MultiAgent

class RAG

Todo

Data:

• Input arxiv paper (Done)
• Openreview: review rebuttal

Code:

• Single agent
• Multi agent
• Environment / evaluation
  • Fake conference
  • Fake collaboration
    • How to construct graph: retrieval / embed Agent state. Rule: cap node degree (max 5)

Idea 2: Next-gen research town

• Paper necessary?
• Multi-round necessary / better?

May 2

1. What is the problem?

   1. Simulate research community

2. Why is it interesting and important?

   1. Inefficiency in existing human-driven research community
   2. (Multi-agent) AI agent environment: not realistic(?) / real-world use case enough
   3. Future potential: accelerate knowledge/idea generation and propagation

3. Why is it hard? (E.g., why do naive approaches fail?)

   1. Data: how to get realistic data from real-world researchers
   2. Model: AI agent, capable enough for simulation?
      1. E.g., review is a challenging task (PhD, Prof) → Agent need specialized knowledge (general LLM may not be sufficient, need RAG or finetuning)
   3. Environment: Dynamic, open-ended
      1. [Our potential]: Citation, impact

4. Why hasn't it been solved before? (Or, what's wrong with previous proposed solutions? How does mine differ?)

   1. Focus on 1 or few aspect of research community: paper writing, summarization, coding, paper review
   2. Mostly single agent/participant: fail to consider the communication/sharing of idea and knowledge
   3. Realistic and up-to-date: fail to reflect latest real-world research

5. What are the key components of my approach and results? Also include any specific limitations.

   Deliverable:

   • Simulator for research community
     • Input:
       • Real-time data stream
         • Research paper
         • [Optional] review, Twitter
       • A group of agents, originated from humans
         • E.g., list of authors in LLM decoding
     • Output - human users:
       • Data valuable for humans
         • Summary of latest research
         • Novel research ideas from agents
         • Fun/educational: Opportunity to “talk” with top research
         • Get instant feedback from AI agent
       • Benchmark metrics: valuable for evaluating Agent/LLM
         • Simulate full research community component - evaluation the quality and relevance
         • What metrics?
           • Quality: LLM-as-Judge: GPT-4
           • Truthfulness/Value: hold-out data
           • Insightful: Human evaluation
         • Analysis
           • Knowledge propagation metrics (on graphs)

   Research town Framework

   Components (mostly prompt engineering + RAG):

   • Single researcher agent
     • Read paper (RAG for data/knowledge chunk)
       • Both real-world papers
       • And synthetic papers from our simulation
     • Ideation
     • Find collaborator
       • RAG for other agent profile
       • Discuss with co-author
     • Write paper
       • Based on idea and author discussion
     • Review role:
       • Review
       • Discuss with author
       • Discuss with AC
     • Author role:
       • Rebuttal
       • Discuss with reviewer
     • AC role:
       • Discuss with AC
       • Decision
     • Social role: share the paper
   • Environment
     • Conference
       • Submission, review, rebuttal, meta-review, decision, public sharing
     • Social
       • Share work, find collaborator
   • Multi-agent mechanism

Note:

• Access control for AI agents

Our value:

• Multi-agent research

Todo:

• Literature review - summarize related work, find our difference and their shared limitations

class basesingleagent(object)

type of methods

• def method(input) - LLM inference

  return response

• def method(input_list) - Multi-agent (message passing)

  return response

• def method(input, external_data) - RAG

  return response

• def memory()

  • short-term: serve as in-context
  • long-term: serve as RAG
  • [Optional] Access control: We can tag the knowledge when being added to the memory

• [Optional] def method(input_list, external_data) - RAG + Multi-agent (Discussion on a paper)

• def feedback():

  • Reward

• [?]def method(self, input) - LLM inference

  • def get_profile()

• def method(self, input_dict: Dict[agent, message]) - Multi-agent (message passing)

  • def communicate(agent_instance)
    • with author, review, AC

[lwaekfjlk]

Why hasn't it been solved before?

the whole simulation of the overall research pipeline (not just single stage or single stage evaluation)

[lwaekfjlk]

unique point, we can simulate a small conference (paper factory)

[lwaekfjlk]

use real-world data, might have real-world social impact instead of just simple simulation.

[lwaekfjlk]

AgentAI hospital (bad) Sotopia (bad) mincraft simulation (bad) general game simulation (bad)
ChatDev (good)

[lwaekfjlk]

dynamic environment (arxiv is daily updated.)

[timsanders256]

AgentVerse

[lwaekfjlk]

What are the key components of my approach and results?

paper and agent db --> how to get high-quality and useful information.

reearch environment --> how to prompt (how to save activity log)

research evaluation --> how to evaluate