This repository has been archived by the owner on Jun 9, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 563
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #204 from serpapi/add-serpapi-plugin
Add SerpApi plugin
- Loading branch information
Showing
6 changed files
with
515 additions
and
0 deletions.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
# Auto-GPT SerpApi Search Plugin | ||
|
||
This search plugin integrates [SerpApi](https://serpapi.com) into Auto-GPT, allowing users to choose a broader range of | ||
search engines supported by SerpApi, and get much more information than the default search engine in Auto-GPT. | ||
|
||
## Key Features: | ||
- Perform search queries with engine of your choice supported by SerpApi, including Google, Bing, Baidu, Yahoo, DuckDuckGo, Yandex and so on. | ||
|
||
## Installation | ||
|
||
- Follow the instructions as per the [Auto-GPT-Plugins/README.md](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/blob/master/README.md) | ||
|
||
- Append the following configuration settings to the `.env` file within AutoGPT, see [Configuration](#configuration) for details: | ||
|
||
```ini | ||
################################################################################ | ||
### SerpApi | ||
################################################################################ | ||
|
||
SERPAPI_API_KEY= | ||
SERPAPI_ENGINE= | ||
SERPAPI_NO_CACHE= | ||
SERPAPI_RESULT_FILTER= | ||
``` | ||
|
||
|
||
- In the `.env` file, search for `ALLOWLISTED_PLUGINS` and add this plugin: | ||
|
||
```ini | ||
################################################################################ | ||
### ALLOWLISTED PLUGINS | ||
################################################################################ | ||
|
||
#ALLOWLISTED_PLUGINS - Sets the listed plugins that are allowed (Example: plugin1,plugin2,plugin3) | ||
ALLOWLISTED_PLUGINS=AutoGPTSerpApiSearch | ||
``` | ||
|
||
## Configuration | ||
|
||
| Variable | Required | Description | | ||
| ---- | ---- | ---- | | ||
| SERPAPI_API_KEY | Yes | Your API key for the SerpApi. You can obtain a key by following the steps:<br>- Sign up for a free account at [SerpApi](https://serpapi.com).<br>- Navigate to the [Dashboard](https://serpapi.com/dashboard) page and find "Your Private API Key". | | ||
| SERPAPI_ENGINE | No | The engine you want to use for web searches performed by Auto-GPT.<br>- You can find valid engine values from [SerpApi Documentation](https://serpapi.com/search-api).<br>- Typical values are: `google`, `bing`, `baidu`, `yahoo`, `duckduckgo`, `yandex`, ...<br>- The default value is `google` if not set. | | ||
| SERPAPI_NO_CACHE | No | Set to `true` if you want to force SerpApi to fetch the results even if a cached version is already present. Defaulted to `false`. | | ||
| SERPAPI_RESULT_FILTER | No | SerpApi can return JSON results that is too large for Auto-GPT to process. This variable allows you to pick certain fields from the returned JSON to reduce the size. Defaulted to `organic_results(title,link,snippet)`. See [Result Filter](#result-filter) for details.| | ||
|
||
### Result Filter | ||
This plugin supports filtering fields up to a depth of 2. The syntax of the filter is `<t>(<s>,<s>,...),<t>(<s>,<s>,...),...`, where `<t>` is top level field, and `<s>` is second level field. `<s>` is optional. Set to `<none>` to disable filtering. Here are some examples: | ||
- `<none>` | ||
- Filter disabled. The whole JSON output will be the input of the current command. | ||
- `organic_results`: | ||
- Pick only `organic_results` from the top level fields of JSON output. | ||
- `organic_results, knowledge_graph`: | ||
- Pick only `organic_results` and `knowledge_graph` from the top level fields of JSON output. | ||
- `organic_results(title, link, snippet)`: | ||
- Pick only `organic_results` from the top level fields of JSON output. | ||
- Pick only `title`, `link` and `snippet` from `organic_results`. | ||
- If `organic_results` is an object, applies to itself. | ||
- If `organic_results` is an array, applies to all its containing objects. | ||
- Otherwise, the second level filter is ignored. | ||
- `organic_results(title,link,snippet), knowledge_graph(website, description)`: | ||
- Pick only `organic_results` and `knowledge_graph` from the top level fields of JSON output. | ||
- Pick only `title`, `link` and `snippet` from `organic_results`. | ||
- If `organic_results` is an object, applies to itself. | ||
- If `organic_results` is an array, applies to all its containing objects. | ||
- Otherwise, the second level filter is ignored. | ||
- Pick only `website`, and `description` from `knowledge_graph`. | ||
- If `knowledge_graph` is an object, applies to itself. | ||
- If `knowledge_graph` is an array, applies to all its containing objects. | ||
- Otherwise, the second level filter is ignored. | ||
|
||
### Filter Tuning | ||
Sometimes too much input can make Auto-GPT confused, failing to extract the correct information. Other than [organic_results](https://serpapi.com/organic-results), SerpApi extracts more fields such as [answer_box](https://serpapi.com/direct-answer-box-api), [knowledge_graph](https://serpapi.com/knowledge-graph) and [related_questions](https://serpapi.com/related-questions), which are more straightforward and easier to make sense of, but not always present. You can always check if those exist through the [Dashboard](https://serpapi.com/searches) and add/remove fields to the filter according to your needs. | ||
|
||
### Example | ||
Here's an example to let Auto-GPT search on Google and get information from "Answer Box" and "Knowledge Graph" | ||
```ini | ||
SERPAPI_API_KEY=your_api_key | ||
SERPAPI_ENGINE=google | ||
SERPAPI_RESULT_FILTER=answer_box,knowledge_graph | ||
``` | ||
## How it works | ||
When `SERPAPI_API_KEY` is set. The plugin will add a new command `serpapi_search` to Auto-GPT. The `google` command will be intercepted to use `serpapi_search` instead. Auto-GPT can also use `serpapi_search` command directly. Therefore, all web searches performed by Auto-GPT are routed to SerpApi. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
"""This is the SerpApi search engines plugin for Auto-GPT.""" | ||
import os | ||
from typing import Any, Dict, List, Optional, Tuple, TypedDict, TypeVar | ||
|
||
from auto_gpt_plugin_template import AutoGPTPluginTemplate | ||
|
||
from .serpapi_search import serpapi_search | ||
|
||
PromptGenerator = TypeVar("PromptGenerator") | ||
|
||
|
||
class Message(TypedDict): | ||
role: str | ||
content: str | ||
|
||
|
||
class AutoGPTSerpApiSearch(AutoGPTPluginTemplate): | ||
def __init__(self): | ||
super().__init__() | ||
self._name = "SerpApi-Search-Plugin" | ||
self._version = "0.1.0" | ||
self._description = ( | ||
"This plugin performs SerpApi searches using the provided query." | ||
) | ||
self.load_commands = ( | ||
os.getenv("SERPAPI_API_KEY") | ||
) | ||
|
||
def can_handle_post_prompt(self) -> bool: | ||
return True | ||
|
||
def post_prompt(self, prompt: PromptGenerator) -> PromptGenerator: | ||
if self.load_commands: | ||
# Add SerpApi Search command | ||
prompt.add_command( | ||
"SerpApi Search", | ||
"serpapi_search", | ||
{"query": "<query>"}, | ||
serpapi_search, | ||
) | ||
else: | ||
print( | ||
"Warning: SerpApi-Search-Plugin is not fully functional. " | ||
"Please set the SERPAPI_API_KEY environment variable." | ||
) | ||
return prompt | ||
|
||
def can_handle_pre_command(self) -> bool: | ||
return True | ||
|
||
def pre_command( | ||
self, command_name: str, arguments: Dict[str, Any] | ||
) -> Tuple[str, Dict[str, Any]]: | ||
if command_name == "google" and self.load_commands: | ||
return "serpapi_search", {"query": arguments["query"]} | ||
else: | ||
return command_name, arguments | ||
|
||
def can_handle_post_command(self) -> bool: | ||
return False | ||
|
||
def post_command(self, command_name: str, response: str) -> str: | ||
pass | ||
|
||
def can_handle_on_planning(self) -> bool: | ||
return False | ||
|
||
def on_planning( | ||
self, prompt: PromptGenerator, messages: List[Message] | ||
) -> Optional[str]: | ||
pass | ||
|
||
def can_handle_on_response(self) -> bool: | ||
return False | ||
|
||
def on_response(self, response: str, *args, **kwargs) -> str: | ||
pass | ||
|
||
def can_handle_post_planning(self) -> bool: | ||
return False | ||
|
||
def post_planning(self, response: str) -> str: | ||
pass | ||
|
||
def can_handle_pre_instruction(self) -> bool: | ||
return False | ||
|
||
def pre_instruction(self, messages: List[Message]) -> List[Message]: | ||
pass | ||
|
||
def can_handle_on_instruction(self) -> bool: | ||
return False | ||
|
||
def on_instruction(self, messages: List[Message]) -> Optional[str]: | ||
pass | ||
|
||
def can_handle_post_instruction(self) -> bool: | ||
return False | ||
|
||
def post_instruction(self, response: str) -> str: | ||
pass | ||
|
||
def can_handle_chat_completion( | ||
self, messages: Dict[Any, Any], model: str, temperature: float, max_tokens: int | ||
) -> bool: | ||
return False | ||
|
||
def handle_chat_completion( | ||
self, messages: List[Message], model: str, temperature: float, max_tokens: int | ||
) -> str: | ||
pass | ||
|
||
def can_handle_text_embedding( | ||
self, text: str | ||
) -> bool: | ||
return False | ||
|
||
def handle_text_embedding( | ||
self, text: str | ||
) -> list: | ||
pass | ||
|
||
def can_handle_user_input(self, user_input: str) -> bool: | ||
return False | ||
|
||
def user_input(self, user_input: str) -> str: | ||
return user_input | ||
|
||
def can_handle_report(self) -> bool: | ||
return False | ||
|
||
def report(self, message: str) -> None: | ||
pass |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
import os | ||
import re | ||
import requests | ||
|
||
_engine_query_key = { | ||
"ebay": "_nkw", | ||
"google_maps_reviews": "data_id", | ||
"google_product": "product_id", | ||
"google_lens": "url", | ||
"google_immersive_product": "page_token", | ||
"google_scholar_author": "author_id", | ||
"google_scholar_profiles": "mauthors", | ||
"google_related_questions": "next_page_token", | ||
"google_finance_markets": "trend", | ||
"google_health_insurance": "provider_id", | ||
"home_depot_product": "product_id", | ||
"walmart": "query", | ||
"walmart_product": "product_id", | ||
"walmart_product_reviews": "product_id", | ||
"yahoo": "p", | ||
"yahoo_images": "p", | ||
"yahoo_videos": "p", | ||
"yandex": "text", | ||
"yandex_images": "text", | ||
"yandex_videos": "text", | ||
"youtube": "search_query", | ||
"google_play_product": "product_id", | ||
"yahoo_shopping": "p", | ||
"apple_app_store": "term", | ||
"apple_reviews": "product_id", | ||
"apple_product": "product_id", | ||
"naver": "query", | ||
"yelp": "find_desc", | ||
"yelp_reviews": "place_id", | ||
} | ||
|
||
|
||
def _filter_dict(obj, filter): | ||
if not isinstance(obj, dict): | ||
return obj | ||
|
||
return dict([(k, v) for k, v in obj.items() if k in filter]) | ||
|
||
|
||
def _filter_results(json, filterstr): | ||
if not filterstr or filterstr == "<none>": | ||
return json | ||
|
||
filter = {} | ||
matches = re.findall(r"(\w+)(?:\((.*?)\))*", filterstr) | ||
for match in matches: | ||
first_level = match[0] | ||
second_levels = [x.strip() for x in match[1].split(",") if x.strip() != ""] | ||
filter[first_level] = second_levels | ||
|
||
filtered_json = _filter_dict(json, list(filter.keys())) | ||
for k, v in filtered_json.items(): | ||
inner_filter = filter[k] | ||
if len(inner_filter) > 0: | ||
if isinstance(v, list): | ||
filtered_json[k] = [ | ||
_filter_dict(x, inner_filter) for x in filtered_json[k] | ||
] | ||
elif isinstance(v, dict): | ||
filtered_json[k] = _filter_dict(filtered_json[k], inner_filter) | ||
|
||
if len(filtered_json) == 1: | ||
return filtered_json[list(filtered_json.keys())[0]] | ||
return filtered_json | ||
|
||
|
||
def _get_params(query: str): | ||
engine = os.getenv("SERPAPI_ENGINE") or "google" | ||
no_cache = os.getenv("SERPAPI_NO_CACHE") | ||
api_key = os.getenv("SERPAPI_API_KEY") | ||
params = { | ||
"engine": engine, | ||
"api_key": api_key, | ||
"source": "serpapi-auto-gpt-plugin-1st", | ||
} | ||
|
||
if no_cache and no_cache != "false": | ||
params["no_cache"] = "true" | ||
|
||
query_key = _engine_query_key[engine] if engine in _engine_query_key else "q" | ||
params[query_key] = query | ||
|
||
return params | ||
|
||
|
||
def serpapi_search(query: str): | ||
""" | ||
Perform a SerpApi search and return the JSON results. | ||
""" | ||
|
||
response = requests.get("https://serpapi.com/search", params=_get_params(query)) | ||
response.raise_for_status() | ||
|
||
result_json = response.json() | ||
|
||
filter = os.getenv("SERPAPI_RESULT_FILTER") or "organic_results(title,link,snippet)" | ||
return _filter_results(result_json, filter) |
Oops, something went wrong.