Merge pull request #204 from serpapi/add-serpapi-plugin

Add SerpApi plugin

.github/CODEOWNERS

@@ -12,3 +12,4 @@
 /src/autogpt_plugins/twitter @desojo
 /src/autogpt_plugins/wikipedia_search @pierluigi-failla
 /src/autogpt_plugins/wolframalpha_search @pierluigi-failla
+/src/autogpt_plugins/serpapi @zyc9012

README.md

@@ -102,6 +102,7 @@ You can see the first-party plugins below. These are included in this Auto-GPT-P
 | Planner          | Simple Task Planner Module for Auto-GPT  | [autogpt_plugins/planner](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/blob/master/src/autogpt_plugins/planner/) |
 | Random Values    | Enable Auto-GPT to generate various random numbers and strings.                                                    | [autogpt_plugins/random_values](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/src/autogpt_plugins/random_values) |
 | SceneX           | Explore image storytelling beyond pixels with the Auto-GPT SceneX Plugin.                                        | [autogpt_plugins/scenex](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/src/autogpt_plugins/scenex)               |
+| SerpApi          | Search on a broad range of search engines supported by SerpApi and get rich information from the results.        | [autogpt_plugins/serpapi](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/src/autogpt_plugins/serpapi)|
 | Telegram |  A smoothly working Telegram bot that gives you all the messages you would normally get through the Terminal. | [autogpt_plugins/telegram](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/src/autogpt_plugins/telegram) |
 | Twitter          | Auto-GPT is capable of retrieving Twitter posts and other related content by accessing the Twitter platform via the v1.1 API using Tweepy.               | [autogpt_plugins/twitter](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/src/autogpt_plugins/twitter)           |
 | Wikipedia Search | This allows Auto-GPT to use Wikipedia directly.                                                                    | [autogpt_plugins/wikipedia_search](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/tree/master/src/autogpt_plugins/wikipedia_search) |

src/autogpt_plugins/serpapi/README.md

@@ -0,0 +1,85 @@
+# Auto-GPT SerpApi Search Plugin
+
+This search plugin integrates [SerpApi](https://serpapi.com) into Auto-GPT, allowing users to choose a broader range of
+search engines supported by SerpApi, and get much more information than the default search engine in Auto-GPT.
+
+## Key Features:
+- Perform search queries with engine of your choice supported by SerpApi, including Google, Bing, Baidu, Yahoo, DuckDuckGo, Yandex and so on.
+
+## Installation
+
+- Follow the instructions as per the [Auto-GPT-Plugins/README.md](https://github.com/Significant-Gravitas/Auto-GPT-Plugins/blob/master/README.md)
+
+- Append the following configuration settings to the `.env` file within AutoGPT, see [Configuration](#configuration) for details:
+
+    ```ini
+    ################################################################################
+    ### SerpApi
+    ################################################################################
+
+    SERPAPI_API_KEY=
+    SERPAPI_ENGINE=
+    SERPAPI_NO_CACHE=
+    SERPAPI_RESULT_FILTER=
+    ```
+
+
+- In the `.env` file, search for `ALLOWLISTED_PLUGINS` and add this plugin:
+
+    ```ini
+    ################################################################################
+    ### ALLOWLISTED PLUGINS
+    ################################################################################
+
+    #ALLOWLISTED_PLUGINS - Sets the listed plugins that are allowed (Example: plugin1,plugin2,plugin3)
+    ALLOWLISTED_PLUGINS=AutoGPTSerpApiSearch
+    ```
+
+## Configuration
+
+| Variable | Required | Description |
+| ---- | ---- | ---- |
+| SERPAPI_API_KEY | Yes | Your API key for the SerpApi. You can obtain a key by following the steps:<br>- Sign up for a free account at [SerpApi](https://serpapi.com).<br>- Navigate to the [Dashboard](https://serpapi.com/dashboard) page and find "Your Private API Key". |
+| SERPAPI_ENGINE | No | The engine you want to use for web searches performed by Auto-GPT.<br>- You can find valid engine values from [SerpApi Documentation](https://serpapi.com/search-api).<br>- Typical values are: `google`, `bing`, `baidu`, `yahoo`, `duckduckgo`, `yandex`, ...<br>- The default value is `google` if not set. |
+| SERPAPI_NO_CACHE | No | Set to `true` if you want to force SerpApi to fetch the results even if a cached version is already present. Defaulted to `false`. |
+| SERPAPI_RESULT_FILTER | No | SerpApi can return JSON results that is too large for Auto-GPT to process. This variable allows you to pick certain fields from the returned JSON to reduce the size. Defaulted to `organic_results(title,link,snippet)`. See [Result Filter](#result-filter) for details.|
+
+### Result Filter
+This plugin supports filtering fields up to a depth of 2. The syntax of the filter is `<t>(<s>,<s>,...),<t>(<s>,<s>,...),...`, where `<t>` is top level field, and `<s>` is second level field. `<s>` is optional. Set to `<none>` to disable filtering. Here are some examples:
+- `<none>`
+  - Filter disabled. The whole JSON output will be the input of the current command.
+- `organic_results`:
+  - Pick only `organic_results` from the top level fields of JSON output.
+- `organic_results, knowledge_graph`:
+  - Pick only `organic_results` and `knowledge_graph` from the top level fields of JSON output.
+- `organic_results(title, link, snippet)`:
+  - Pick only `organic_results` from the top level fields of JSON output.
+  - Pick only `title`, `link` and `snippet` from `organic_results`.
+    - If `organic_results` is an object, applies to itself.
+    - If `organic_results` is an array, applies to all its containing objects.
+    - Otherwise, the second level filter is ignored.
+- `organic_results(title,link,snippet), knowledge_graph(website, description)`:
+  - Pick only `organic_results` and `knowledge_graph` from the top level fields of JSON output.
+  - Pick only `title`, `link` and `snippet` from `organic_results`.
+    - If `organic_results` is an object, applies to itself.
+    - If `organic_results` is an array, applies to all its containing objects.
+    - Otherwise, the second level filter is ignored.
+  - Pick only `website`, and `description` from `knowledge_graph`.
+    - If `knowledge_graph` is an object, applies to itself.
+    - If `knowledge_graph` is an array, applies to all its containing objects.
+    - Otherwise, the second level filter is ignored.
+
+### Filter Tuning
+Sometimes too much input can make Auto-GPT confused, failing to extract the correct information. Other than [organic_results](https://serpapi.com/organic-results), SerpApi extracts more fields such as [answer_box](https://serpapi.com/direct-answer-box-api), [knowledge_graph](https://serpapi.com/knowledge-graph) and [related_questions](https://serpapi.com/related-questions), which are more straightforward and easier to make sense of, but not always present. You can always check if those exist through the [Dashboard](https://serpapi.com/searches) and add/remove fields to the filter according to your needs.
+
+### Example
+Here's an example to let Auto-GPT search on Google and get information from "Answer Box" and "Knowledge Graph"
+
+```ini
+SERPAPI_API_KEY=your_api_key
+SERPAPI_ENGINE=google
+SERPAPI_RESULT_FILTER=answer_box,knowledge_graph
+```
+
+## How it works
+When `SERPAPI_API_KEY` is set. The plugin will add a new command `serpapi_search` to Auto-GPT. The `google` command will be intercepted to use `serpapi_search` instead. Auto-GPT can also use `serpapi_search` command directly. Therefore, all web searches performed by Auto-GPT are routed to SerpApi.

src/autogpt_plugins/serpapi/__init__.py

@@ -0,0 +1,133 @@
+"""This is the SerpApi search engines plugin for Auto-GPT."""
+import os
+from typing import Any, Dict, List, Optional, Tuple, TypedDict, TypeVar
+
+from auto_gpt_plugin_template import AutoGPTPluginTemplate
+
+from .serpapi_search import serpapi_search
+
+PromptGenerator = TypeVar("PromptGenerator")
+
+
+class Message(TypedDict):
+    role: str
+    content: str
+
+
+class AutoGPTSerpApiSearch(AutoGPTPluginTemplate):
+    def __init__(self):
+        super().__init__()
+        self._name = "SerpApi-Search-Plugin"
+        self._version = "0.1.0"
+        self._description = (
+            "This plugin performs SerpApi searches using the provided query."
+        )
+        self.load_commands = (
+            os.getenv("SERPAPI_API_KEY")
+        )
+
+    def can_handle_post_prompt(self) -> bool:
+        return True
+
+    def post_prompt(self, prompt: PromptGenerator) -> PromptGenerator:
+        if self.load_commands:
+            # Add SerpApi Search command
+            prompt.add_command(
+                "SerpApi Search",
+                "serpapi_search",
+                {"query": "<query>"},
+                serpapi_search,
+            )
+        else:
+            print(
+                "Warning: SerpApi-Search-Plugin is not fully functional. "
+                "Please set the SERPAPI_API_KEY environment variable."
+            )
+        return prompt
+
+    def can_handle_pre_command(self) -> bool:
+        return True
+
+    def pre_command(
+        self, command_name: str, arguments: Dict[str, Any]
+    ) -> Tuple[str, Dict[str, Any]]:
+        if command_name == "google" and self.load_commands:
+            return "serpapi_search", {"query": arguments["query"]}
+        else:
+            return command_name, arguments
+
+    def can_handle_post_command(self) -> bool:
+        return False
+
+    def post_command(self, command_name: str, response: str) -> str:
+        pass
+
+    def can_handle_on_planning(self) -> bool:
+        return False
+
+    def on_planning(
+        self, prompt: PromptGenerator, messages: List[Message]
+    ) -> Optional[str]:
+        pass
+
+    def can_handle_on_response(self) -> bool:
+        return False
+
+    def on_response(self, response: str, *args, **kwargs) -> str:
+        pass
+
+    def can_handle_post_planning(self) -> bool:
+        return False
+
+    def post_planning(self, response: str) -> str:
+        pass
+
+    def can_handle_pre_instruction(self) -> bool:
+        return False
+
+    def pre_instruction(self, messages: List[Message]) -> List[Message]:
+        pass
+
+    def can_handle_on_instruction(self) -> bool:
+        return False
+
+    def on_instruction(self, messages: List[Message]) -> Optional[str]:
+        pass
+
+    def can_handle_post_instruction(self) -> bool:
+        return False
+
+    def post_instruction(self, response: str) -> str:
+        pass
+
+    def can_handle_chat_completion(
+        self, messages: Dict[Any, Any], model: str, temperature: float, max_tokens: int
+    ) -> bool:
+        return False
+
+    def handle_chat_completion(
+        self, messages: List[Message], model: str, temperature: float, max_tokens: int
+    ) -> str:
+        pass
+
+    def can_handle_text_embedding(
+        self, text: str
+    ) -> bool:
+        return False
+
+    def handle_text_embedding(
+        self, text: str
+    ) -> list:
+        pass
+
+    def can_handle_user_input(self, user_input: str) -> bool:
+        return False
+
+    def user_input(self, user_input: str) -> str:
+        return user_input
+
+    def can_handle_report(self) -> bool:
+        return False
+
+    def report(self, message: str) -> None:
+        pass

src/autogpt_plugins/serpapi/serpapi_search.py

@@ -0,0 +1,102 @@
+import os
+import re
+import requests
+
+_engine_query_key = {
+    "ebay": "_nkw",
+    "google_maps_reviews": "data_id",
+    "google_product": "product_id",
+    "google_lens": "url",
+    "google_immersive_product": "page_token",
+    "google_scholar_author": "author_id",
+    "google_scholar_profiles": "mauthors",
+    "google_related_questions": "next_page_token",
+    "google_finance_markets": "trend",
+    "google_health_insurance": "provider_id",
+    "home_depot_product": "product_id",
+    "walmart": "query",
+    "walmart_product": "product_id",
+    "walmart_product_reviews": "product_id",
+    "yahoo": "p",
+    "yahoo_images": "p",
+    "yahoo_videos": "p",
+    "yandex": "text",
+    "yandex_images": "text",
+    "yandex_videos": "text",
+    "youtube": "search_query",
+    "google_play_product": "product_id",
+    "yahoo_shopping": "p",
+    "apple_app_store": "term",
+    "apple_reviews": "product_id",
+    "apple_product": "product_id",
+    "naver": "query",
+    "yelp": "find_desc",
+    "yelp_reviews": "place_id",
+}
+
+
+def _filter_dict(obj, filter):
+    if not isinstance(obj, dict):
+        return obj
+
+    return dict([(k, v) for k, v in obj.items() if k in filter])
+
+
+def _filter_results(json, filterstr):
+    if not filterstr or filterstr == "<none>":
+        return json
+
+    filter = {}
+    matches = re.findall(r"(\w+)(?:\((.*?)\))*", filterstr)
+    for match in matches:
+        first_level = match[0]
+        second_levels = [x.strip() for x in match[1].split(",") if x.strip() != ""]
+        filter[first_level] = second_levels
+
+    filtered_json = _filter_dict(json, list(filter.keys()))
+    for k, v in filtered_json.items():
+        inner_filter = filter[k]
+        if len(inner_filter) > 0:
+            if isinstance(v, list):
+                filtered_json[k] = [
+                    _filter_dict(x, inner_filter) for x in filtered_json[k]
+                ]
+            elif isinstance(v, dict):
+                filtered_json[k] = _filter_dict(filtered_json[k], inner_filter)
+
+    if len(filtered_json) == 1:
+        return filtered_json[list(filtered_json.keys())[0]]
+    return filtered_json
+
+
+def _get_params(query: str):
+    engine = os.getenv("SERPAPI_ENGINE") or "google"
+    no_cache = os.getenv("SERPAPI_NO_CACHE")
+    api_key = os.getenv("SERPAPI_API_KEY")
+    params = {
+        "engine": engine,
+        "api_key": api_key,
+        "source": "serpapi-auto-gpt-plugin-1st",
+    }
+
+    if no_cache and no_cache != "false":
+        params["no_cache"] = "true"
+
+    query_key = _engine_query_key[engine] if engine in _engine_query_key else "q"
+    params[query_key] = query
+
+    return params
+
+
+def serpapi_search(query: str):
+    """
+    Perform a SerpApi search and return the JSON results.
+    """
+
+    response = requests.get("https://serpapi.com/search", params=_get_params(query))
+    response.raise_for_status()
+
+    result_json = response.json()
+
+    filter = os.getenv("SERPAPI_RESULT_FILTER") or "organic_results(title,link,snippet)"
+    return _filter_results(result_json, filter)