Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

🐢 Open-Source Evaluation & Testing for LLMs and ML models

AI Observability & Evaluation

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

DataChain 🔗 AI-dataframe to enrich, transform and analyze data from cloud storages for ML training and LLM apps

Python SDK for running evaluations on LLM generated responses

Generate ideal question-answers for testing RAG

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

Бенчмарк сравнивает русские аналоги ChatGPT: Saiga, YandexGPT, Gigachat

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

An open source library for asynchronous querying of LLM endpoints

Code for "Prediction-Powered Ranking of Large Language Models", Arxiv 2024.

Realign is a AI testing and simulation framework for multi-turn AI applications. It simulates user interactions, evaluates AI performance, and generates adversarial scenarios to test LLM vulnerabilities.

TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

The prompt engineering, prompt management, and prompt evaluation tool for Python

Generative agents — computational software agents that simulate believable human behavior and OpenAI LLM models. Our main focus was to develop a game - “Werewolves of Miller’s Hollow”, aiming to replicate human-like behavior.

The prompt engineering, prompt management, and prompt evaluation tool for TypeScript, JavaScript, and NodeJS.

LLM Security Platform Docs

Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.