Merge pull request #16 from dbkinder/main

publish latest

latest/_sources/GenAIComps/comps/agent/langchain/README.md.txt

@@ -1,39 +1,70 @@
-# langchain Agent Microservice
+# Agent Microservice
 
-The langchain agent model refers to a framework that integrates the reasoning capabilities of large language models (LLMs) with the ability to take actionable steps, creating a more sophisticated system that can understand and process information, evaluate situations, take appropriate actions, communicate responses, and track ongoing situations.
+## 1. Overview
 
-![Architecture Overview](agent_arch.jpg)
+This agent microservice is built on Langchain/Langgraph frameworks. Agents integrate the reasoning capabilities of large language models (LLMs) with the ability to take actionable steps, creating a more sophisticated system that can understand and process information, evaluate situations, take appropriate actions, communicate responses, and track ongoing situations.
+
+### 1.1 Supported agent types
+
+We currently support the following types of agents:
+
+1. ReAct: use `react_langchain` or `react_langgraph` as strategy. First introduced in this seminal [paper](https://arxiv.org/abs/2210.03629). The ReAct agent engages in "reason-act-observe" cycles to solve problems. Please refer to this [doc](https://python.langchain.com/v0.2/docs/how_to/migrate_agent/) to understand the differences between the langchain and langgraph versions of react agents.
+2. RAG agent: `rag_agent` strategy. This agent is specifically designed for improving RAG performance. It has the capability to rephrase query, check relevancy of retrieved context, and iterate if context is not relevant.
+3. Plan and execute: `plan_execute` strategy. This type of agent first makes a step-by-step plan given a user request, and then execute the plan sequentially (or in parallel, to be implemented in future). If the execution results can solve the problem, then the agent will output an answer; otherwise, it will replan and execute again.
+   For advanced developers who want to implement their own agent strategies, please refer to [Section 5](#5-customize-agent-strategy) below.
+
+### 1.2 LLM engine
+
+Agents use LLM for reasoning and planning. We support 2 options of LLM engine:
+
+1. Open-source LLMs served with TGI-gaudi. To use open-source llms, follow the instructions in [Section 2](#222-start-microservices) below. Note: we recommend using state-of-the-art LLMs, such as llama3.1-70B-instruct, to get higher success rate.
+2. OpenAI LLMs via API calls. To use OpenAI llms, specify `llm_engine=openai` and `export OPENAI_API_KEY=<your-openai-key>`
+
+### 1.3 Tools
 
-## 🚀1. Start Microservice with Python（Option 1）
+The tools are registered with a yaml file. We support the following types of tools:
 
-### 1.1 Install Requirements
+1. Endpoint: user to provide url
+2. User-defined python functions. This is usually used to wrap endpoints with request post or simple pre/post-processing.
+3. Langchain tool modules.
+   Examples of how to register tools can be found in [Section 4](#-4-provide-your-own-tools) below.
+
+### 1.4 Agent APIs
+
+Currently we have implemented OpenAI chat completion compatible API for agents. We are working to support OpenAI assistants APIs.
+
+# 🚀2. Start Agent Microservice
+
+## 2.1 Option 1: with Python
+
+### 2.1.1 Install Requirements
 
 ```bash
 cd comps/agent/langchain/
 pip install -r requirements.txt
 ```
 
-### 1.2 Start Microservice with Python Script
+### 2.1.2 Start Microservice with Python Script
 
 ```bash
 cd comps/agent/langchain/
 python agent.py
 ```
 
-## 🚀2. Start Microservice with Docker (Option 2)
+## 2.2 Option 2. Start Microservice with Docker
 
-### Build Microservices
+### 2.2.1 Build Microservices
 
 ```bash
 cd GenAIComps/ # back to GenAIComps/ folder
 docker build -t opea/comps-agent-langchain:latest -f comps/agent/langchain/docker/Dockerfile .
 ```
 
-### start microservices
+### 2.2.2 Start microservices
 
 ```bash
 export ip_address=$(hostname -I | awk '{print $1}')
-export model=meta-llama/Meta-Llama-3-8B-Instruct
+export model=mistralai/Mistral-7B-Instruct-v0.3
 export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
 export HF_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
 
@@ -53,10 +84,10 @@ docker logs comps-langchain-agent-endpoint
 > debug mode
 >
 > ```bash
-> docker run --rm --runtime=runc --name="comps-langchain-agent-endpoint" -v ./comps/agent/langchain/:/home/user/comps/agent/langchain/ -p 9090:9090 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} --env-file ${agent_env} opea/comps-agent-langchain:latest
+> docker run --rm --runtime=runc --name="comps-langchain-agent-endpoint" -v ./comps/agent/langchain/:/home/user/comps/agent/langchain/ -p 9090:9090 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} -e model=${model} -e ip_address=${ip_address} -e strategy=react -e llm_endpoint_url=http://${ip_address}:8080 -e llm_engine=tgi -e recursion_limit=5 -e require_human_feedback=false -e tools=/home/user/comps/agent/langchain/tools/custom_tools.yaml opea/comps-agent-langchain:latest
 > ```
 
-## 🚀3. Validate Microservice
+# 🚀 3. Validate Microservice
 
 Once microservice starts, user can use below script to invoke.
 
@@ -73,7 +104,7 @@ data: [DONE]
 
 ```
 
-## 🚀4. Provide your own tools
+# 🚀 4. Provide your own tools
 
 - Define tools
 
@@ -148,3 +179,8 @@ data: 'The weather information in Austin is not available from the Open Platform
 
 data: [DONE]
 ```
+
+# 5. Customize agent strategy
+
+For advanced developers who want to implement their own agent strategies, you can add a separate folder in `src\strategy`, implement your agent by inherit the `BaseAgent` class, and add your strategy into the `src\agent.py`. The architecture of this agent microservice is shown in the diagram below as a reference.
+![Architecture Overview](agent_arch.jpg)

latest/_sources/GenAIComps/comps/agent/langchain/src/strategy/ragagent/README.md.txt

@@ -0,0 +1,31 @@
+# RAG Agent
+
+This agent is specifically designed to improve answer quality over conventional RAG.
+This agent strategy includes steps listed below:
+
+1. QueryWriter
+   This is an llm with tool calling capability, it decides if tool calls are needed to answer the user query or it can answer with llm's parametric knowledge.
+
+   - Yes: Rephrase the query in the form of a tool call to the Retriever tool, and send the rephrased query to 'Retriever'. The rephrasing is important as user queries may be not be clear and simply using user query may not retrieve relevant documents.
+   - No: Complete the query with Final answer
+
+2. Retriever:
+
+   - Get related documents from a retrieval tool, then send the documents to 'DocumentGrader'. Note: The retrieval tool here is broad-sense, which can be a text retriever over a proprietary knowledge base, a websearch API, knowledge graph API, SQL database API etc.
+
+3. DocumentGrader
+   Judge retrieved info relevance with respect to the user query
+
+   - Yes: Go to TextGenerator
+   - No: Go back to QueryWriter to rewrite query.
+
+4. TextGenerator
+   - Generate an answer based on query and last retrieved context.
+   - After generation, go to END.
+
+Note:
+
+- Currently the performance of this RAG agent has been tested and validated with only one retrieval tool. If you want to use multiple retrieval tools, we recommend a hierarchical multi-agent system where a supervisor agent dispatches requests to multiple worker RAG agents, where individual worker RAG agents uses one type of retrieval tool.
+- The max number of retrieves is set at 3.
+- You can specify a small `recursion_limit` to stop early or a big `recursion_limit` to fully use the 3 retrieves.
+- The TextGenerator only looks at the last retrieved docs.

latest/_sources/GenAIComps/comps/cores/telemetry/README.md.txt

@@ -0,0 +1,121 @@
+# Telemetry for OPEA
+
+OPEA Comps currently provides telemetry functionalities for metrics and tracing using Prometheus, Grafana, and Jaeger. Here’s a basic introduction to these tools:
+
+![opea telemetry](https://raw.githubusercontent.com/Spycsh/assets/main/OPEA%20Telemetry.jpg)
+
+## Metrics
+
+OPEA microservice metrics are exported in Prometheus format and are divided into two categories: general metrics and specific metrics.
+
+General metrics, such as `http_requests_total `, `http_request_size_bytes`, are exposed by every microservice endpoint using the [prometheus-fastapi-instrumentator](https://github.com/trallnag/prometheus-fastapi-instrumentator).
+
+Specific metrics are the built-in metrics exposed under `/metrics` by each specific microservices such as TGI, vLLM, TEI and others. Both types of the metrics adhere to the Prometheus format.
+
+### General Metrics
+
+To access the general metrics of each microservice, you can use `curl` as follows:
+
+```bash
+curl localhost:{port of your service}/metrics
+```
+
+Then you will see Prometheus format metrics printed out as follows:
+
+```yaml
+HELP http_requests_total Total number of requests by method, status and handler.
+# TYPE http_requests_total counter
+http_requests_total{handler="/metrics",method="GET",status="2xx"} 3.0
+http_requests_total{handler="/v1/chatqna",method="POST",status="2xx"} 2.0
+...
+# HELP http_request_size_bytes Content length of incoming requests by handler. Only value of header is respected. Otherwise ignored. No percentile calculated.
+# TYPE http_request_size_bytes summary
+http_request_size_bytes_count{handler="/metrics"} 3.0
+http_request_size_bytes_sum{handler="/metrics"} 0.0
+http_request_size_bytes_count{handler="/v1/chatqna"} 2.0
+http_request_size_bytes_sum{handler="/v1/chatqna"} 128.0
+...
+```
+
+### Specific Metrics
+
+To access the metrics exposed by each specific microservice, ensure that you check the specific port and your port mapping to reach the `/metrics` endpoint correctly.
+
+For example, you can `curl localhost:6006/metrics` to retrieve the TEI embedding metrics, and the output should look like follows:
+
+```yaml
+# TYPE te_embed_count counter
+te_embed_count 7
+
+# TYPE te_request_success counter
+te_request_success{method="batch"} 2
+
+# TYPE te_request_count counter
+te_request_count{method="single"} 2
+te_request_count{method="batch"} 2
+
+# TYPE te_embed_success counter
+te_embed_success 7
+
+# TYPE te_queue_size gauge
+te_queue_size 0
+
+# TYPE te_request_inference_duration histogram
+te_request_inference_duration_bucket{le="0.000015000000000000002"} 0
+te_request_inference_duration_bucket{le="0.000022500000000000005"} 0
+te_request_inference_duration_bucket{le="0.00003375000000000001"} 0
+```
+
+These metrics can be scraped by the Prometheus server into a time-series database and further visualized using Grafana.
+
+Below are some default metrics endpoints for specific microservices:
+
+| component     | port  | endpoint | metircs doc                                                                                             |
+| ------------- | ----- | -------- | ------------------------------------------------------------------------------------------------------- |
+| TGI           | 80    | /metrics | [link](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/monitoring)             |
+| milvus        | 9091  | /metrics | [link](https://milvus.io/docs/monitor.md)                                                               |
+| vLLM          | 18688 | /metrics | [link](https://docs.vllm.ai/en/v0.5.0/serving/metrics.html)                                             |
+| TEI embedding | 6006  | /metrics | [link](https://huggingface.github.io/text-embeddings-inference/#/Text%20Embeddings%20Inference/metrics) |
+| TEI reranking | 8808  | /metrics | [link](https://huggingface.github.io/text-embeddings-inference/#/Text%20Embeddings%20Inference/metrics) |
+
+## Tracing
+
+OPEA use OpenTelemetry to trace function call stacks. To trace a function, add the `@opea_telemetry` decorator to either an async or sync function. The call stacks and time span data will be exported by OpenTelemetry. You can use Jaeger UI to visualize this tracing data.
+
+By default, tracing data is exported to `http://localhost:4318/v1/traces`. This endpoint can be customized by editing the `TELEMETRY_ENDPOINT` environment variable.
+
+```py
+from comps import opea_telemetry
+
+
+@opea_telemetry
+async def your_async_func():
+    pass
+
+
+@opea_telemetry
+def your_sync_func():
+    pass
+```
+
+## Visualization
+
+### Visualize metrics
+
+Please refer to [OPEA grafana](https://github.com/opea-project/GenAIEval/tree/main/evals/benchmark/grafana) to get the details of Prometheus and Grafana server setup. The Grafana dashboard JSON files are also provided under [OPEA grafana](https://github.com/opea-project/GenAIEval/tree/main/evals/benchmark/grafana) to visualize the metrics.
+
+### Visualize tracing
+
+Run the following command to start the Jaeger server.
+
+```bash
+docker run -d --rm \
+-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
+-p 16686:16686 \
+-p 4317:4317 \
+-p 4318:4318 \
+-p 9411:9411 \
+jaegertracing/all-in-one:latest
+```
+
+Access the dashboard UI at `localhost:16686`.