Create your own private LLM leaderboard! 📊

There's no one-fit-all leaderboard. FM-Leaderboard-er will allow you to find the best LLM for your own business use case based on your own tasks, prompts, and data.

1. Tasks - Example notebooks for common tasks like Summarization, Classification, and RAG (coming soon).
2. Models - Amazon Bedrock, OpenAI, any API (with a code integration).
3. Metrics - Built-in metrics per task + custom metrics (via a code integration).
4. Latency - Latency metric per model
5. Cost - comparison.
6. Prompt - You could compare several prompts across one model

1. AWS account with Amazon Bedrock access to selected models.

1. Clone the repository:

   git clone https://github.com/aws-samples/fm-leaderboarder.git
   

To get started, open the example-1 notebook and follow the instructions provided.

Coming soon.

This code can interact with the OpenAI service which has terms published here and pricing described here. You should be familiar with the pricing and confirm that your use case complies with the terms before proceeding.

This repository makes use of aws/fmeval Foundation Model Evaluations Library. Please review any license terms applicable to the dataset with your legal team and confirm that your use case complies with the terms before proceeding.

See CONTRIBUTING for more information.

Contributions to FM-Leaderboarder are welcome! Please refer to the CONTRIBUTING.md file for guidelines on how to contribute.

This project is licensed under the Apache-2.0 License.