Merge pull request #28 from dbkinder/main

publish latest

latest/_sources/GenAIComps/comps/embeddings/README.md.txt

@@ -31,3 +31,7 @@ For details, please refer to this [readme](multimodal/README.md).
 ## Embeddings Microservice with Multimodal Clip
 
 For details, please refer to this [readme](multimodal_clip/README.md).
+
+## Embeddings Microservice with Prediction Guard
+
+For details, please refer to this [readme](predictionguard/README.md).

latest/_sources/GenAIComps/comps/embeddings/predictionguard/README.md.txt

@@ -0,0 +1,39 @@
+# Embedding Generation Prediction Guard Microservice
+
+[Prediction Guard](https://docs.predictionguard.com) allows you to utilize hosted open access LLMs, LVMs, and embedding functionality with seamlessly integrated safeguards. In addition to providing a scalable access to open models, Prediction Guard allows you to configure factual consistency checks, toxicity filters, PII filters, and prompt injection blocking. Join the [Prediction Guard Discord channel](https://discord.gg/TFHgnhAFKd) and request an API key to get started.
+
+This embedding microservice is designed to efficiently convert text into vectorized embeddings using the [BridgeTower model](https://huggingface.co/BridgeTower/bridgetower-large-itm-mlm-itc). Thus, it is ideal for both RAG or semantic search applications.
+
+**Note** - The BridgeTower model implemented in Prediction Guard can actually embed text, images, or text + images (jointly). For now this service only embeds text, but a follow on contribution will enable the multimodal functionality.
+
+# 🚀 Start Microservice with Docker
+
+## Setup Environment Variables
+
+Setup the following environment variables first
+
+```bash
+export PREDICTIONGUARD_API_KEY=${your_predictionguard_api_key}
+```
+
+## Build Docker Images
+
+```bash
+cd ../../..
+docker build -t opea/embedding-predictionguard:latest -f comps/embeddings/predictionguard/Dockerfile .
+```
+
+## Start Service
+
+```bash
+docker run -d --name="embedding-predictionguard" -p 6000:6000 -e PREDICTIONGUARD_API_KEY=$PREDICTIONGUARD_API_KEY opea/embedding-predictionguard:latest
+```
+
+# 🚀 Consume Embeddings Service
+
+```bash
+curl localhost:6000/v1/embeddings \
+     -X POST \
+     -d '{"text":"Hello, world!"}' \
+     -H 'Content-Type: application/json'
+```

latest/_sources/GenAIComps/comps/guardrails/README.md.txt

@@ -7,5 +7,6 @@ The Guardrails service enhances the security of LLM-based applications by offeri
 | [Llama Guard](./llama_guard/langchain/README.md)     | Provides guardrails for inputs and outputs to ensure safe interactions                                                   |
 | [PII Detection](./pii_detection/README.md)           | Detects Personally Identifiable Information (PII) and Business Sensitive Information (BSI)                               |
 | [Toxicity Detection](./toxicity_detection/README.md) | Detects Toxic language (rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion) |
+| [Bias Detection](./bias_detection/README.md)         | Detects Biased language (framing bias, epistemological bias, and demographic bias)                                       |
 
 Additional safety-related microservices will be available soon.

latest/_sources/GenAIComps/comps/guardrails/bias_detection/README.md.txt

@@ -0,0 +1,93 @@
+# Bias Detection Microservice
+
+## Introduction
+
+Bias Detection Microservice allows AI Application developers to safeguard user input and LLM output from biased language in a RAG environment. By leveraging a smaller fine-tuned Transformer model for bias classification (e.g. DistilledBERT, RoBERTa, etc.), we maintain a lightweight guardrails microservice without significantly sacrificing performance making it readily deployable on both Intel Gaudi and Xeon.
+
+Bias erodes our collective trust and fuels social conflict. Bias can be defined as inappropriate subjectivity in the form of one of the following:
+
+- Framing bias -- using subjective words or phrases linked with a particular point of view
+- Epistemological bias -- linguistic features that subtly modify the believability of a proposition
+- Demographic bias -- text with presuppositions about particular genders, races, or other demographic categories
+
+## Future Development
+
+- Add a "neutralizing bias" microservice to neutralizing any detected bias in the RAG serving, guarding the RAG usage.
+
+## 🚀1. Start Microservice with Python（Option 1）
+
+### 1.1 Install Requirements
+
+```bash
+pip install -r requirements.txt
+```
+
+### 1.2 Start Bias Detection Microservice with Python Script
+
+```bash
+python bias_detection.py
+```
+
+## 🚀2. Start Microservice with Docker (Option 2)
+
+### 2.1 Prepare bias detection model
+
+export HUGGINGFACEHUB_API_TOKEN=${HP_TOKEN}
+
+### 2.2 Build Docker Image
+
+```bash
+cd ../../../ # back to GenAIComps/ folder
+docker build -t opea/guardrails-bias-detection:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/guardrails/bias_detection/Dockerfile .
+```
+
+### 2.3 Run Docker Container with Microservice
+
+```bash
+docker run -d --rm --runtime=runc --name="guardrails-bias-detection" -p 9092:9092 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN} -e HF_TOKEN=${HUGGINGFACEHUB_API_TOKEN} opea/guardrails-bias-detection:latest
+```
+
+## 🚀3. Get Status of Microservice
+
+```bash
+docker container logs -f guardrails-bias-detection
+```
+
+## 🚀4. Consume Microservice Pre-LLM/Post-LLM
+
+Once microservice starts, users can use examples (bash or python) below to apply bias detection for both user's query (Pre-LLM) or LLM's response (Post-LLM)
+
+**Bash:**
+
+```bash
+curl localhost:9092/v1/bias
+    -X POST
+    -d '{"text":"John McCain exposed as an unprincipled politician"}'
+    -H 'Content-Type: application/json'
+```
+
+Example Output:
+
+```bash
+"\nI'm sorry, but your query or LLM's response is BIASED with an score of 0.74 (0-1)!!!\n"
+```
+
+**Python Script:**
+
+```python
+import requests
+import json
+
+proxies = {"http": ""}
+url = "http://localhost:9092/v1/bias"
+data = {"text": "John McCain exposed as an unprincipled politician"}
+
+
+try:
+    resp = requests.post(url=url, data=data, proxies=proxies)
+    print(resp.text)
+    resp.raise_for_status()  # Raise an exception for unsuccessful HTTP status codes
+    print("Request successful!")
+except requests.exceptions.RequestException as e:
+    print("An error occurred:", e)
+```

latest/_sources/GenAIComps/comps/llms/text-generation/predictionguard/README.md.txt

@@ -0,0 +1,54 @@
+# Introduction
+
+[Prediction Guard](https://docs.predictionguard.com) allows you to utilize hosted open access LLMs, LVMs, and embedding functionality with seamlessly integrated safeguards. In addition to providing a scalable access to open models, Prediction Guard allows you to configure factual consistency checks, toxicity filters, PII filters, and prompt injection blocking. Join the [Prediction Guard Discord channel](https://discord.gg/TFHgnhAFKd) and request an API key to get started.
+
+# Get Started
+
+## Build Docker Image
+
+```bash
+cd ../../..
+docker build -t opea/llm-textgen-predictionguard:latest -f comps/llms/text-generation/predictionguard/Dockerfile .
+```
+
+## Run the Predictionguard Microservice
+
+```bash
+docker run -d -p 9000:9000 -e PREDICTIONGUARD_API_KEY=$PREDICTIONGUARD_API_KEY  --name llm-textgen-predictionguard opea/llm-textgen-predictionguard:latest
+```
+
+# Consume the Prediction Guard Microservice
+
+See the [Prediction Guard docs](https://docs.predictionguard.com/) for available model options.
+
+## Without streaming
+
+```bash
+curl -X POST http://localhost:9000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Hermes-2-Pro-Llama-3-8B",
+        "query": "Tell me a joke.",
+        "max_new_tokens": 100,
+        "temperature": 0.7,
+        "top_p": 0.9,
+        "top_k": 50,
+        "stream": false
+    }'
+```
+
+## With streaming
+
+```bash
+curl -N -X POST http://localhost:9000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "Hermes-2-Pro-Llama-3-8B",
+        "query": "Tell me a joke.",
+        "max_new_tokens": 100,
+        "temperature": 0.7,
+        "top_p": 0.9,
+        "top_k": 50,
+        "stream": true
+    }'
+```

latest/_sources/GenAIExamples/ChatQnA/README.md.txt

@@ -10,16 +10,15 @@ ChatQnA architecture shows below:
 
 The ChatQnA example is implemented using the component-level microservices defined in [GenAIComps](https://github.com/opea-project/GenAIComps). The flow chart below shows the information flow between different microservices for this example.
 
-```{mermaid}
+```mermaid
 ---
 config:
   flowchart:
-    nodeSpacing: 100
+    nodeSpacing: 400
     rankSpacing: 100
     curve: linear
-  theme: base
   themeVariables:
-    fontSize: 42px
+    fontSize: 50px
 ---
 flowchart LR
     %% Colors %%
@@ -28,67 +27,60 @@ flowchart LR
     classDef orchid fill:#C26DBC,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5
     classDef invisible fill:transparent,stroke:transparent;
     style ChatQnA-MegaService stroke:#000000
+
     %% Subgraphs %%
-    subgraph ChatQnA-MegaService["ChatQnA-MegaService"]
+    subgraph ChatQnA-MegaService["ChatQnA MegaService "]
         direction LR
-        EM([Embedding <br>]):::blue
-        RET([Retrieval <br>]):::blue
-        RER([Rerank <br>]):::blue
-        LLM([LLM <br>]):::blue
+        EM([Embedding MicroService]):::blue
+        RET([Retrieval MicroService]):::blue
+        RER([Rerank MicroService]):::blue
+        LLM([LLM MicroService]):::blue
     end
-    subgraph User Interface
-        direction TB
+    subgraph UserInterface[" User Interface "]
+        direction LR
         a([User Input Query]):::orchid
         Ingest([Ingest data]):::orchid
         UI([UI server<br>]):::orchid
     end
-    subgraph ChatQnA GateWay
-        direction LR
-        invisible1[ ]:::invisible
-        GW([ChatQnA GateWay<br>]):::orange
-    end
-    subgraph .
-        X([OPEA Micsrservice]):::blue
-        Y{{Open Source Service}}
-        Z([OPEA Gateway]):::orange
-        Z1([UI]):::orchid
-    end
+
+
 
     TEI_RER{{Reranking service<br>}}
     TEI_EM{{Embedding service <br>}}
     VDB{{Vector DB<br><br>}}
     R_RET{{Retriever service <br>}}
-    DP([Data Preparation<br>]):::blue
+    DP([Data Preparation MicroService]):::blue
     LLM_gen{{LLM Service <br>}}
+    GW([ChatQnA GateWay<br>]):::orange
 
     %% Data Preparation flow
     %% Ingest data flow
     direction LR
-    Ingest[Ingest data] -->|a| UI
-    UI -->|b| DP
-    DP <-.->|c| TEI_EM
+    Ingest[Ingest data] --> UI
+    UI --> DP
+    DP <-.-> TEI_EM
 
     %% Questions interaction
     direction LR
-    a[User Input Query] -->|1| UI
-    UI -->|2| GW
-    GW <==>|3| ChatQnA-MegaService
-    EM ==>|4| RET
-    RET ==>|5| RER
-    RER ==>|6| LLM
+    a[User Input Query] --> UI
+    UI --> GW
+    GW <==> ChatQnA-MegaService
+    EM ==> RET
+    RET ==> RER
+    RER ==> LLM
 
 
     %% Embedding service flow
-    direction TB
-    EM <-.->|3'| TEI_EM
-    RET <-.->|4'| R_RET
-    RER <-.->|5'| TEI_RER
-    LLM <-.->|6'| LLM_gen
+    direction LR
+    EM <-.-> TEI_EM
+    RET <-.-> R_RET
+    RER <-.-> TEI_RER
+    LLM <-.-> LLM_gen
 
     direction TB
     %% Vector DB interaction
-    R_RET <-.->|d|VDB
-    DP <-.->|d|VDB
+    R_RET <-.->VDB
+    DP <-.->VDB
 
 
 

latest/_sources/GenAIExamples/CodeGen/docker_compose/intel/cpu/xeon/README.md.txt

@@ -76,7 +76,7 @@ The CodeGen megaservice manages a single microservice called LLM within a Direct
 
 The mega flow of the CodeGen application, from user's input query to the application's output response, is as follows:
 
-```{mermaid}
+```mermaid
 flowchart LR
     subgraph CodeGen
         direction LR

latest/_sources/GenAIExamples/CodeGen/docker_compose/intel/hpu/gaudi/README.md.txt

@@ -62,7 +62,7 @@ The CodeGen megaservice manages a single microservice called LLM within a Direct
 
 The mega flow of the CodeGen application, from user's input query to the application's output response, is as follows:
 
-```{mermaid}
+```mermaid
 flowchart LR
     subgraph CodeGen
         direction LR

latest/_sources/GenAIExamples/DocSum/README.md.txt

@@ -104,7 +104,7 @@ Refer to the [DocSum helm chart](https://github.com/opea-project/GenAIInfra/tree
 
 The workflow of the Document Summarization Service, from user's input query to the application's output response, is as follows:
 
-```{mermaid}
+```mermaid
 flowchart LR
     subgraph DocSum
         direction LR

latest/_sources/GenAIExamples/MultimodalQnA/README.md.txt

@@ -10,7 +10,7 @@ The MultimodalQnA architecture shows below:
 
 MultimodalQnA is implemented on top of [GenAIComps](https://github.com/opea-project/GenAIComps), the MultimodalQnA Flow Chart shows below:
 
-```{mermaid}
+```mermaid
 ---
 config:
   flowchart:

latest/_sources/GenAIExamples/VideoQnA/README.md.txt

@@ -4,7 +4,7 @@ VideoQnA is a framework that retrieves video based on provided user prompt. It u
 
 VideoQnA is implemented on top of [GenAIComps](https://github.com/opea-project/GenAIComps), with the architecture flow chart shows below:
 
-```{mermaid}
+```mermaid
 ---
 config:
   flowchart:

latest/_sources/GenAIInfra/proxy/deployments/guardrails/README.md.txt

@@ -6,7 +6,7 @@ Guardrails in OPEA Pipeline Proxy supports both rule-based detection and AI-powe
 
 ## Architecture
 
-```{mermaid}
+```mermaid
 flowchart LR
     subgraph Filter[HTTP Filter]
         direction BT