OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
AI Evaluation & Observability

FMBench

FMBench is a Python package and command-line tool for benchmarking foundation models on Amazon SageMaker, Bedrock, EKS, EC2, or a user-provided endpoint. It measures performance and model accuracy to help teams compare deployment configurations and candidate models.

Explore the practical guide ↓

Independent OpenViST project profile, based on linked sources.

FMBench — OpenViST project profile in AI Evaluation & Observability.
OpenViST project illustration. See the linked sources for product details.
Understand the project

Practical overview

When it fits

Use FMBench when a deployment or model choice needs workload-relevant performance and accuracy comparisons across supported AWS options.

Who it is for

  • Teams selecting an AWS serving configuration for a foundation model.
  • Developers comparing model performance across instance types or inference stacks.
  • Evaluators measuring model accuracy against ground truth.

Documented uses

  • Compare inference latency and transactions per minute across model-serving configurations.
  • Measure candidate-model accuracy against ground truth or use a panel of LLM evaluators.
  • Benchmark a model already deployed behind a bring-your-own endpoint.
Before you choose

Requirements & limits

FMBench is distributed as a Python package and used from the command line. The project is published under the MIT-0 license.

  • Runs on an AWS environment that supports Python; the README recommends this to avoid internet round-trip time being counted in end-to-end latency.
  • A run requires a configuration file describing the model, deployment steps, and tests.
  • Performance and evaluation choices depend on the capabilities supported by the underlying AWS platform.
  • Metrics, reports, and results are stored in an Amazon S3 bucket.

Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.

Source-based practical guide

Run the FMBench SageMaker quickstart

Benchmark Llama2-7b performance on two SageMaker instance types using the provided quickstart configuration.

Before you start

  • An AWS account and access to a supported AWS Region listed in the quickstart.
  • A SageMaker notebook created by the FMBench CloudFormation template.
  • The quickstart configuration and datasets provided in the template’s read S3 bucket.

Steps

  1. Choose the provided Llama2-7b quickstart configuration, which tests ml.g5.xlarge and ml.g5.2xlarge, or adapt a configuration for your requirements.

  2. Launch the FMBench CloudFormation template in a supported Region and wait for the stack to create its S3 buckets, IAM role, and SageMaker notebook.

  3. Open the fmbench-notebook, start a Terminal, and create and activate the Python 3.11 environment, then install FMBench: conda create --name fmbench_python311 -y python=3.11 ipykernel followed by source activate fmbench_python311; pip install -U fmbench.

  4. Run the sample configuration with account=$(aws sts get-caller-identity | jq .Account | tr -d '"'); region=$(aws configure get region); fmbench --config-file s3://sagemaker-fmbench-read-${region}-${account}/configs/llama2/7b/config-llama2-7b-g5-quick.yml > fmbench.log 2>&1.

  5. Review the generated report and metrics in the write S3 bucket and the local results directory; the Markdown report is named report.md.

Expected outcome

FMBench produces a performance benchmark and report for Llama2-7b on the two specified SageMaker instance types, with metrics and reports stored in S3 and locally.

Guide limitations

  • The quickstart test is estimated to take about 30 minutes and cost about $0.20; actual results may vary.
  • The default token-count estimate uses 750 words per 1,000 tokens. The guide recommends using a tokenizer specific to the tested model for more accurate token-throughput results.
  • Not all models available through Bedrock or other services may be available in GovCloud.
Documentation 1 ↗
Source-based practical guide

Prepare a shared FMBench client-server setup across AWS accounts

Set up the documented arrangement in which a platform team hosts model endpoints on EC2 and other teams run FMBench tests from a separate AWS account.

Before you start

  • A server AWS account with EC2 instances hosting model endpoints.
  • A client AWS account with FMBench installed on EC2 instances.
  • Security groups configured with an inbound rule that permits access to the model endpoint, typically on TCP port 8080.

Steps

  1. Deploy the models on EC2 in the server account using FMBench, configuring the serving stack for the platform team’s needs.

  2. Configure the server-side security groups to allow the client installation to access the model endpoint; the documentation gives TCP port 8080 as typical.

  3. Install FMBench in the client AWS account and use it to run tests against the hosted endpoints with the teams’ datasets, performance criteria, and inference parameters.

Expected outcome

Platform teams can keep model endpoints available for other teams to benchmark without each team first deploying the models.

Guide limitations

  • Endpoint access depends on security groups permitting inbound traffic from the client account.
  • This setup describes model hosting on EC2 and client-side testing from another AWS account.
Documentation 1 ↗
Evidence and transparency

Sources & review

Official repository ↗Guide documentation ↗Guide documentation ↗
Source record details
Source availability
Active
Sources checked
17 October 2026
Original website title
aws-samples/foundation-model-benchmarking-tool · Official homepage title · retained in its source language
How we review sourcesSuggest a correction