A curated list of solutions that secure GenAI systems and LLMs
A curated list of awesome solutions to hard AI security problems and risks. This stands in contrast to using AI for security (for offense or defense) and is explicitly meant to link to software and solutions that help solve the problems
Other lists cover the many excellent frameworks, papers, attacks, safety, and so forth. But new GenAI systems are subject to all new classes of attacks and it has been difficult to find projects and companies that solve these problems as they rarely shine through the noise of "AI security" as a phrase.
If you want to contribute, create a PR.
- Related Awesome Lists
- Infographic
- Confidential Computing
- Encryption and Data Protection
- Governance
- Model Testing
- Prompt Firewall and Redaction
- QA
- Training Data Protection
- ottosulin/awesome-ai-security - AI security related frameworks, attacks, tools and papers.
- deepspaceharbor/awesome-ai-security - AI security resources including attacks, examples, and code.
- awesome-ai-for-cybersecurity - Research roundup on AI's use in classic security tools.
- awesome-llm-security - A curation of awesome tools, documents and projects about LLM Security.
- awesome-ml-privacy-attacks - An awesome list of papers on privacy attacks against machine learning.
- awesome-ml-security - Trail of Bits' machine learning security references, guidance, and tools.
As an experiment, we'll try to keep an infographic of Awesome AI security solutions up-to-date as an open source SVG file. This will be a fast visual overview with a mix of logos and text.
Solutions that use secure enclaves and confidential computing to keep the data in parts of AI workflows private.
- Fortanix Confidential AI - Run AI models inside Intel SGX and other enclave technologies.
Products that protect AI data privacy by encrypting the data. Infrastructure-layer encryption like filesystem encryption do not belong here. Only application-layer encryption solutions are eligible.
-
IronCore Labs' Cloaked AI - Encrypt vector embeddings before sending to a vector database to secure the data in RAG workflows and other AI workflows.
-
Enveil Secure AI - Train encrypted models and do encrypted inferences over them.
Products that specifically track AI projects and produce reports to meet various AI and privacy regulations and/or frameworks.
- OneTrust AI Governance - Track projects and apply frameworks to them.
- Cranium AI Exposure Management Solution - Provide visibility into an AI system, characterize attack surfaces, and assess vulnerabilities in an organization.
- CredoAI - AI governance, risk, and compliance for the AI-powered enterprise.
- DynamoEval - Provides automated stress testing of AI systems and autogenerates documentation needed for regulatory audits.
Products that examine or test models for security issues of various kinds.
- HiddenLayer Model Scanner - Scan models for vulnerabilities and supply chain issues.
- Plexiglass - A toolkit for detecting and protecting against vulnerabilities in Large Language Models (LLMs).
- PurpleLlama - Set of tools from Meta to assess and improve LLM security.
- Garak - A LLM vulnerability scanner.
- CalypsoAI Platform - Platform for testing and launching LLM applications securely.
- Lakera Red - Automated safety and security assessments for your GenAI applications.
- jailbreak-evaluation - Python package for language model jailbreak evaluation.
- Patronus AI - Automated testing of models to detect PII, copyrighted materials, and sensitive information in models.
- Adversa Red Teaming - Continuous AI red teaming for LLMs.
- Advai - Automates the tasks of stress-testing, red-teaming, and evaluating your AI systems for critical failure.
- Mindgard AI - Identifies and remediates risks across AI models, GenAI, LLMs along with AI-powered apps and chatbots.
- Protect AI ModelScan - Scan models for serialization attacks.
- Protect AI Guardian - Scan models for security issues or policy violations with auditing and reporting.
Products that intercept prompts and responses and apply security or privacy rules to them. We've blended two categories here because some prompt firewalls just redact private data (and then reidentify in the response) while others focus on identifying and blocking attacks like injection attacks or stopping data leaks. Many of the products in this category do all of the above, which is why they've been combined.
- Protect AI Rebuff - A LLM prompt injection detector.
- Protect AI LLM Guard - Suite of tools to protect LLM applications by helping you detect, redact, and sanitize LLM prompts and responses.
- HiddenLayer AI Detection and Response - Proactively defend against threats to your LLMs.
- Robust Intelligence AI Firewall - Real-time protection, automatically configured to address the vulnerabilities of each model.
- Vigil LLM - Detect prompt injections, jailbreaks, and other potentially risky Large Language Model (LLM) inputs.
- Lakera Guard - Protection from prompt injections, data loss, and toxic content.
- Arthur Shield - Built-in, real-time firewall protection against the biggest LLM risks.
- Prompt Security - SDK and proxy for protection against common prompt attacks.
- Private AI - Detect, anonymize, and replace PII with less than half the error rate of alternatives.
- DynamoGuard - Identify / defend against any type of non-compliance as defined by your specific AI policies and catch attacks.
- Skyflow LLM Privacy Vault - Redacts PII from prompts flowing to LLMs.
- Guardrails AI - Guardrails runs Input/Output Guards in your application that detect, quantify and mitigate the presence of specific types of risks.
Products that add quality controls and testing for GenAI workflows other than express testing of models. These tend to focus more on prompts and APIs.
- Freeplay AI - Test and track prompts, their performance and versions over time.
- Prompt Security Fuzzer - Open-source tool to help you harden your GenAI applications.
- LLMFuzzer - Open-source fuzzing framework specifically designed for LLMs, especially for their integrations in applications via APIs.
Products that specifically address private data in the training of models and sometimes also in prompts to models that have been trained using their protections.
- Synthesis AI - Simulation and synthetic data for computer vision training.
- Protopia AI - "Stained glass transforms" of text and image data when training preserves privacy in model and inferences.
- Mostly AI - Use existing data and the power of Generative AI for synthetic data generation.
- DynamoEnhance - Use differential privacy and PII sanitization on training data.
Contributions are welcome. Add new items, suggest changes to categories or descriptions, etc. We're not aiming to be comprehensive, but to provide a short list of the most notable solutions in each category.
That said, there are some rules as there is an established format and approach. Please carefully read the guidelines for contributing in the contributing.md file in this repo.