Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker.

[prasad-nair-amd]

For example i want to run an endpoint which is running on intel/amd/nvidia instances which is generally available. ie, instances which are not available on Sagemaker but available on AWS.

[aarora79]

We are working on a new FMBench release which supports hosting both the LLMs and FMBench on the same EC2 instance (so no SageMaker, no S3 dependancy) and this should enable you to test on AMD instances. Note that you can already do this today in a "bring your own endpoint" kind of way, see instructions here and config file examples here. What the next release would enable doing is to deploy the model on EC2 also through FMBench (although we have not yet to tried this on AMD instances so there would be some trial and error involved, in the meantime please take a look at the links above).

[prasadwrites]

@aarora79 Is there an update on the FMBench release you had pointed out which works on EC2 instance ?

[aarora79]

Yes, the latest version of fmbench supports deploying models on ec2. Please let us connect offline so that we can get this moving faster. Cc @madhurprash @4mp

[herothedog]

Hello @aarora79 I can help with @prasadwrites to eval FMBench on EC2 AMD instances. Could we connect offline?

[aarora79]

@herothedog absolutely, please send me an email at [email protected] and I would setup a meeting. Looking forward.