ToolsFrameworks.md

1. Langchain: LangChain is a framework for developing applications powered by language models. It enables applications that:

   • Are context-aware: connect a language model to sources of context (prompt instructions, few shot examples, content to ground its response in, etc.)
   • Reason: rely on a language model to reason (about how to answer based on provided context, what actions to take, etc.)

2. llamaindex: LlamaIndex is a data framework for LLM-based applications which benefit from context augmentation. Such LLM systems have been termed as RAG systems, standing for “Retrieval-Augmented Generation”. LlamaIndex provides the essential abstractions to more easily ingest, structure, and access private or domain-specific data in order to inject these safely and reliably into LLMs for more accurate text generation. It’s available in Python (these docs) and Typescript.

3. Haystack: Haystack is the open source Python framework by deepset for building custom apps with large language models (LLMs). It lets you quickly try out the latest models in natural language processing (NLP) while being flexible and easy to use. Our inspiring community of users and builders has helped shape Haystack into what it is today: a complete framework for building production-ready NLP apps.

4. FlowiseAI: FlowiseAI is a drag-and-drop UI for building LLM flows and developing LangChain apps. It's an excellent choice for developers who want to construct large language models. At the same time, it's aimed at organizations that want to develop LLM apps but lack the means to employ a developer. You can use Flowise AI to build apps such as chatbots, virtual assistants, and data analysis tools.

5. AgentGPT: AgentGPT is designed for organizations that wish to deploy autonomous AI agents in their browsers. While Auto-GPT operates independently and generates its own prompts, Agent GPT depends on user inputs and works by interacting with humans to achieve tasks. Though still in the beta stage, AgentGPT currently provides long-term memory and web browsing capabilities.

6. BabyAGI: BabyAGI is a Python script that acts as an AI-powered task manager. It uses OpenAI, LangChain, and vector databases, such as Chroma and Pinecone, to create, prioritize, and execute tasks. It does this by selecting a task from a list and sending the task to an agent, which uses OpenAI to complete the task based on context. The vector database then enriches and stores the result. BabyAGI then goes on to create new tasks and reprioritizes the list according to the result and objective of the previous task.

7. Langdock: LangDock was built for developers searching for an all-in-one product suite for creating, testing, deploying, and monitoring their LLM plugins. It lets you add your API documentation manually or import an existing OpenAPI specification.

8. GradientJ: GradientJ is a tool for developers looking to build and manage large language model applications. It lets you orchestrate and manage complex applications by chaining prompts and knowledge bases into complex APIs and enhances the accuracy of your models by integrating them with your proprietary data.

9. Semantic Kernel: Semantic Kernel is an SDK that integrates Large Language Models (LLMs) like OpenAI, Azure OpenAI, and Hugging Face with conventional programming languages like C#, Python, and Java. Semantic Kernel achieves this by allowing you to define plugins that can be chained together in just a few lines of code.

10. MetaGPT: Created by a team of researchers from Chinese and US universities, MetaGPT is a new LLM-based meta programming framework aiming to enable collaboration in multi-agent systems by leveraging human procedural knowledge to enhance robustness, reduce errors, and engineer software solutions for complex tasks

11. AutoChain: AutoChain takes inspiration from LangChain and AutoGPT and aims to solve both problems by providing a lightweight and extensible framework for developers to build their own agents using LLMs with custom tools and automatically evaluating different user scenarios with simulated conversations. Experienced user of LangChain would find AutoChain is easy to navigate since they share similar but simpler concepts.

12. Fastchat: FastChat is an open platform for training, serving, and evaluating large language model based chatbots.

13. Skypilot: SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, highest GPU availability, and managed execution.

14. Swiss Army Llama: The Swiss Army Llama is designed to facilitate and optimize the process of working with local LLMs by using FastAPI to expose convenient REST endpoints for various tasks, including obtaining text embeddings and completions using different LLMs via llama_cpp, as well as automating the process of obtaining all the embeddings for most common document types, including PDFs (even ones that require OCR), Word file, etc; it even allows you to submit an audio file and automatically transcribes it with the Whisper model, cleans up the resulting text, and then computes the embeddings for it. To avoid wasting computation, these embeddings are cached in SQlite and retrieved if they have already been computed before. To speed up the process of loading multiple LLMs, optional RAM Disks can be used, and the process for creating and managing them is handled automatically for you. With a quick and easy setup process, you will immediately get access to a veritable "Swiss Army Knife" of LLM related tools, all accessible via a convenient Swagger UI and ready to be integrated into your own applications with minimal fuss or configuration required.

15. Langstream: angStream is a lighter alternative to LangChain for building LLMs application, instead of having a massive amount of features and classes, LangStream focuses on having a single small core, that is easy to learn, easy to adapt, well documented, fully typed and truly composable, with Streams instead of chains as the building block.

16. Magentic: asily integrate Large Language Models into your Python code. Simply use the @prompt decorator to create functions that return structured output from the LLM. Mix LLM queries and function calling with regular Python code to create complex logic.

17. [Promptfoo:](https://github.com/promptfoo/promptfoo Promptfoo is a tool for testing and evaluating LLM output quality.With promptfoo, you can:

    • Systematically test prompts, models, and RAGs with predefined test cases
    • Evaluate quality and catch regressions by comparing LLM outputs side-by-side
    • Speed up evaluations with caching and concurrency
    • Score outputs automatically by defining test cases
    • Use as a CLI, library, or in CI/CD
    • Use OpenAI, Anthropic, Azure, Google, HuggingFace, open-source models like Llama, or integrate custom API providers for any LLM API

18. Agenta: he open-source LLM developer platform for prompt-engineering, evaluation, human feedback, and deployment of complex LLM apps.

19. Langdroid: angroid is an intuitive, lightweight, extensible and principled Python framework to easily build LLM-powered applications, from ex-CMU and UW-Madison researchers. You set up Agents, equip them with optional components (LLM, vector-store and tools/functions), assign them tasks, and have them collaboratively solve a problem by exchanging messages. This Multi-Agent paradigm is inspired by the Actor Framework.

20. Embedchain: mbedchain is an Open Source Framework for personalizing LLM responses. It makes it easy to create and deploy personalized AI apps. At its core, Embedchain follows the design principle of being "Conventional but Configurable" to serve both software engineers and machine learning engineers. Embedchain streamlines the creation of personalized LLM applications, offering a seamless process for managing various types of unstructured data. It efficiently segments data into manageable chunks, generates relevant embeddings, and stores them in a vector database for optimized retrieval. With a suite of diverse APIs, it enables users to extract contextual information, find precise answers, or engage in interactive chat conversations, all tailored to their own data.

21. CometLLM: CometLLM is a tool to log and visualize your LLM prompts and chains. Use CometLLM to identify effective prompt strategies, streamline your troubleshooting, and ensure reproducible workflows!

22. OpenLLM: OpenLLM is an open-source platform designed to facilitate the deployment and operation of large language models (LLMs) in real-world applications. With OpenLLM, you can run inference on any open-source LLM, deploy them on the cloud or on-premises, and build powerful AI applications.

23. TensorRT-LLM: TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. It also includes a backend for integration with the NVIDIA Triton Inference Server; a production-quality system to serve LLMs. Models built with TensorRT-LLM can be executed on a wide range of configurations going from a single GPU to multiple nodes with multiple GPUs (using Tensor Parallelism and/or Pipeline Parallelism).

24. Chainlit: Chainlit is an open-source Python package to build production ready Conversational AI.

25. DeepSpeed: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

26. BMTrain: MTrain is an efficient large model training toolkit that can be used to train large models with tens of billions of parameters. It can train models in a distributed manner while keeping the code as simple as stand-alone training.

27. LM Evaluation Harness: The project provides a unified framework to test generative language models on a large number of different evaluation tasks.

28. Lighteval: LightEval is a lightweight LLM evaluation suite that Hugging Face has been using internally with the recently released LLM data processing library datatrove and LLM training library nanotron.

29. InstructEval: InstructEval is an ICL evaluation suite to conduct a thorough assessment of these techniques. The suite includes 13 open-sourced LLMs of varying scales from four model families, and covers nine tasks across three categories.

30. Bonito: Bonito is an open-source model for conditional task generation: the task of converting unannotated text into task-specific training datasets for instruction tuning. This repo is a lightweight library for Bonito to easily create synthetic datasets built on top of the Hugging Face transformers and vllm libraries.

31. LitGPT: Hackable implementation of state-of-the-art open-source large language models released under the Apache 2.0 license.

32. DSPy: DSPy is a framework for algorithmically optimizing LM prompts and weights, especially when LMs are used one or more times within a pipeline. To use LMs to build a complex system without DSPy, you generally have to: (1) break the problem down into steps, (2) prompt your LM well until each step works well in isolation, (3) tweak the steps to work well together, (4) generate synthetic examples to tune each step, and (5) use these examples to finetune smaller LMs to cut costs. Currently, this is hard and messy: every time you change your pipeline, your LM, or your data, all prompts (or finetuning steps) may need to change.

33. Guidance: Guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting and chaining. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditional, loops) and generation seamlessly.

34. Mergekit: mergekit is a toolkit for merging pre-trained language models. mergekit uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention.

35. Turkish NLP Resources: urkish NLP (Türkçe Doğal Dil İşleme) related Tools, Libraries, Models, Datasets and other resources.

36. nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs. It is a rewrite of minGPT that prioritizes teeth over education.

37. OpenGPT: A framework for creating grounded instruction based datasets and training conversational domain expert Large Language Models (LLMs).

38. OpenAGI: The project presents OpenAGI, an open-source AGI research platform, specifically designed to offer complex, multi-step tasks and accompanied by task-specific datasets, evaluation metrics, and a diverse range of extensible models. OpenAGI formulates complex tasks as natural language queries, serving as input to the LLM. The LLM subsequently selects, synthesizes, and executes models provided by OpenAGI to address the task. Furthermore, the project presents the Reinforcement Learning from Task Feedback (RLTF) mechanism, which uses the task-solving result as feedback to improve the LLM's task-solving ability. Thus, the LLM is responsible for synthesizing various external models for solving complex tasks, while RLTF provides feedback to improve its task-solving ability, enabling a feedback loop for self-improving AI. We believe that the paradigm of LLMs operating various expert models for complex task-solving is a promising approach towards AGI.

39. PromptAppGPT: PromptAppGPT is a low-code prompt-based rapid app development framework. PromptAppGPT contains features such as low-code prompt-based development, GPT text generation, DALLE image generation, online prompt editer+compiler+runer, automatic user interface generation, support for plug-in extensions, etc. PromptAppGPT aims to enable natural language app development based on GPT.

40. OpenPrompt: Prompt-learning is the latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks, which modifies the input text with a textual template and directly uses PLMs to conduct pre-trained tasks. This library provides a standard, flexible and extensible framework to deploy the prompt-learning pipeline. OpenPrompt supports loading PLMs directly from huggingface transformers. In the future, we will also support PLMs implemented by other libraries.

41. DeepSparse: DeepSparse is a CPU inference runtime that takes advantage of sparsity to accelerate neural network inference. Coupled with SparseML, our optimization library for pruning and quantizing your models, DeepSparse delivers exceptional inference performance on CPU hardware.

42. Vllm: vLLM is a fast and easy-to-use library for LLM inference and serving.

43. txtai: txtai is an all-in-one embeddings database for semantic search, LLM orchestration and language model workflows.

44. Fairscale: FairScale is a PyTorch extension library for high performance and large scale training. This library extends basic PyTorch capabilities while adding new SOTA scaling techniques. FairScale makes available the latest distributed training techniques in the form of composable modules and easy to use APIs. These APIs are a fundamental part of a researcher’s toolbox as they attempt to scale models with limited resources.

45. OLMO-Eval: The olmo_eval framework is a way to run evaluation pipelines for language models on NLP tasks. The codebase is extensible and contains task_sets and example configurations, which run a series of tango steps for computing the model outputs and metrics.

46. IntelliServer: Intelliserver is a microservice providing unified access to multiple AI models, allowing you to easily integrate cutting-edge AI into your project.

47. YiVal: YiVal is a state-of-the-art tool designed to streamline the tuning process for your GenAI app prompts and ANY configs in the loop. With YiVal, manual adjustments are a thing of the past. This data-driven and evaluation-centric approach ensures optimal prompts, precise RAG configurations, and fine-tuned model parameters. Empower your applications to achieve enhanced results, reduce latency, and minimize inference costs effortlessly with YiVal!

48. Outlines: The first step towards reliability of systems that include large language models is to ensure that there is a well-defined interface between their output and user-defined code. Outlines provides ways to control the generation of language models to make their output more predictable.

49. LMQL: LMQL is a programming language for LLMs. Robust and modular LLM prompting using types, templates, constraints and an optimizing runtime.

50. ModelFusion: ModelFusion is an abstraction layer for integrating AI models into JavaScript and TypeScript applications, unifying the API for common operations such as text streaming, object generation, and tool usage. It provides features to support production environments, including observability hooks, logging, and automatic retries. You can use ModelFusion to build AI applications, chatbots, and agents.

51. Flappy: Flappy is a production-ready Language Language Model (LLM) Application/Agent SDK designed to simplify AI integration in your projects. It is an easy-to-use, universally compatible, and production-ready solution that brings the power of AI to developers regardless of their preferred programming language.

52. QAnything: QAnything(Question and Answer based on Anything) is a local knowledge base question-answering system designed to support a wide range of file formats and databases, allowing for offline installation and use. With QAnything, you can simply drop any locally stored file of any format and receive accurate, fast, and reliable answers.

53. EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models.

54. LangFlow: UI for LangChain

55. Gorilla: Gorilla enables LLMs to use tools by invoking APIs. Given a natural language query, Gorilla comes up with the semantically- and syntactically- correct API to invoke. With Gorilla, we are the first to demonstrate how to use LLMs to invoke 1,600+ (and growing) API calls accurately while reducing hallucination. We also release APIBench, the largest collection of APIs, curated and easy to be trained on! Join us, as we try to expand the largest API store and teach LLMs how to write them! Hop on our Discord, or open a PR, or email us if you would like to have your API incorporated as well.

56. DatasetGPT: DatasetGPT is a command-line interface and a Python library for inferencing Large Language Models to generate textual datasets.

57. Langforge: LangForge is an open-source toolkit designed to make it easy to create and deploy LangChain applications.

58. Zep: ep is an open source platform for productionizing LLM apps. Zep summarizes, embeds, and enriches chat histories and documents asynchronously, ensuring these operations don't impact your user's chat experience. Data is persisted to database, allowing you to scale out when growth demands. As drop-in replacements for popular LangChain components, you can get to production in minutes without rewriting code.

59. LocalGPT: LocalGPT is an open-source initiative that allows you to converse with your documents without compromising your privacy. With everything running locally, you can be assured that no data ever leaves your computer. Dive into the world of secure, local document interactions with LocalGPT.

60. ThinkGPT: ThinkGPT is a Python library aimed at implementing Chain of Thoughts for Large Language Models (LLMs), prompting the model to think, reason, and to create generative agents.

61. GPT-Agent: Camel AutoGPT allows you to configure and deploy communicating Autonomous AI agents. Name your own custom AI characters and have them embark on any goal imaginable.

62. PrivateGPT: PrivateGPT is a production-ready AI project that allows you to ask questions about your documents using the power of Large Language Models (LLMs), even in scenarios without an Internet connection. 100% private, no data leaves your execution environment at any point.

63. SkyAGI: SkyAGI is a python package that demonstrates LLM's emerging capability in simulating believable human behaviors. Specifically, SkyAGI implements the idea of Generative Agents and delivers a role-playing game that creates a very interesting user experience Different from previous AI based NPC systems, SkyAGI's NPC generates very believable human responses. The interesting observations in this demo show a huge potential for rethinking game development in many aspects, such as NPC script writing.

64. SuperAgent: uperagent allows any developer to add powerful AI assistants to their applications. These assistants use large language models (LLM), retrieval augmented generation (RAG), and generative AI to help users.

65. Voyager: An Open-Ended Embodied Agent with Large Language Models

66. IX: IX is a platform for designing and deploying autonomous and [semi]-autonomous LLM powered agents and workflows. IX provides a flexible and scalable solution for delegating tasks to AI powered agents. Agents created with the platform can automate a wide variety of tasks, while running in parallel and communicating with each other.

67. DuetGPT: DuetGPT is an experimental AI powered CLI tool and semi-autonomous agent that helps developers with coding tasks and file system tasks. The developer describes tasks to the AI who then issues commands or follow-up questions for clarification. After approval by the developer, DuetGPT automatically executes the commands issued by the AI.

68. DemoGPT: Autonomous AI Agent for Effortless App Creation

69. SuperAGI: Open-source framework to build, manage and run useful Autonomous AI Agents

70. BlockAGI: BlockAGI is an open-source research agent built with Python3, utilizing the capabilities of LangChain and OpenAI. BlockAGI conducts iterative, domain-specific research, primarily focused on cryptocurrency but customizable to other domains. It outputs detailed narrative reports to showcase its findings. The progress of the AI agent's work is presented interactively through a user-friendly web interface, allowing users to watch the progress in real-time.

71. Waggle-Dance: waggledance.ai is an experimental application focused on achieving user-specified goals. It provides a friendly but opinionated user interface for building agent-based systems. The project focuses on explainability, observability, concurrent generation, and exploration.

72. Quivr: Quivr, your second brain, utilizes the power of GenerativeAI to be your personal assistant ! Think of it as Obsidian, but turbocharged with AI capabilities.

73. DocsGPT: DocsGPT is a cutting-edge open-source solution that streamlines the process of finding information in the project documentation. With its integration of the powerful GPT models, developers can easily ask questions about a project and receive accurate answers.

74. llamafile: llamafile lets you distribute and run LLMs with a single file.

75. firecrawl: Crawl and convert any website into LLM-ready markdown or structured data

76. exo: Run your own AI cluster at home with everyday devices

77. ragflow: RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data.

78. torchchat: torchchat is a small codebase showcasing the ability to run large language models (LLMs) seamlessly. With torchchat, you can run LLMs using Python, within your own (C/C++) application (desktop or server) and on iOS and Android.

79. prompt-poet: Prompt Poet streamlines and simplifies prompt design for both developers and non-technical users with its low code approach. Using a mix of YAML and Jinja2, Prompt Poet allows for flexible, dynamic prompt creation, enhancing the efficiency and quality of interactions with AI models. It saves time on engineering string manipulations, enabling everyone to focus more on crafting the optimal prompts for their users.

80. LMDeploy: LMDeploy is a toolkit for compressing, deploying, and serving LLM

81. spark-nlp: Spark NLP is a state-of-the-art Natural Language Processing library built on top of Apache Spark. It provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment.

82. composer: Composer is an open-source deep learning training library by MosaicML. Built on top of PyTorch, the Composer library makes it easier to implement distributed training workflows on large-scale clusters.

83. baselines: OpenAI Baselines is a set of high-quality implementations of reinforcement learning algorithms.

84. instructlab: InstructLab uses a novel synthetic data-based alignment tuning method for Large Language Models (LLMs.)

85. MemGPT: MemGPT allows you to build LLM agents with long term memory & custom tools

86. phidata: Build AI Assistants with memory, knowledge and tools