Issue when executing Compatibility test in EC2

[anushkai]

I am trying to experiment medperf in EC2 environment and following the documentation. When executing the command
medperf test run \ --benchmark 1 \ --model "medperf_tutorial/model_mobilenetv2/mlcube/mlcube.yaml"
in https://docs.medperf.org/getting_started/model_owner_demo/ to execute the compatibility test. I am getting some errors which I have included the screenshots. Can you help in troubleshooting?

[anushkai]

@aristizabal95

[anushkai]

@aristizabal95 @hasan7n I was using 8Gb of storage. How much of a storage I should use? What is your suggestion?

[hasan7n]

Hi @anushkai

You can try 12GB.

[anushkai]

@hasan7n @aristizabal95 Thanks, it worked after increasing the storage. Another question: Where can I find some example tutorials to start experimenting?. I am using MedPerf 0.1.0 and the it is stated in the examples that they are not compatible with latest medperf version.

[anushkai]

@hasan7n and @aristizabal95

[hasan7n]

Hi @anushkai

We are sorry that the other examples are not compatible with the current MedPerf version. In the current time we only have the chest x-ray example.

[anushkai]

HI @hasan7n even in ChestXray example https://github.com/mlcommons/medperf/tree/main/examples/ChestXRay it is stated that example is not compatible. Are you referring to the chestXray_tutorial. If that so can you kindly add some description on the execution steps. Specially on model evaluation?

[anushkai]

@hasan7n

[hasan7n]

Hi @anushkai
Yes, I am referring to the chestXray_tutorial.

This example is the one used within the tutorials found here: https://docs.medperf.org/getting_started/overview/, where we describe in details how to run such benchmarking example using MedPerf.

I am not sure what do you mean by "description on the execution steps". Did you run the tutorials?

[anushkai]

Hi @hasan7n Yes, I followed the descriptions upto Creating Model MLCubes from Scratch but Creating metrics part is still in progress. I would be grateful if you can advise how to execute metrics which is used to evaluate models.

[anushkai]

@hasan7n @aristizabal95

[aristizabal95]

Hey @anushkai, If you have already followed the Hands-on Tutorial for Data Owners and gotten results then you've already evaluated a model using the ChestXray_tutorial. Let us know if you've done so, and if that wasn't clear. Any feedback on the clarity of our documentation is greatly appreciated!

[aristizabal95]

Hi @anushkai, we haven't heard from you on this for a while. Were you able to run the ChestXray tutorial? Do you need any help from our side? We're currently in the process of doing some heavy work for future MedPerf goals, so example compatibility is not currently on our radar, but we would be happy to help make sure you're able to run medperf with the one example we expect to work!

[hasan7n]

closing for inactivity, feel free to re-open if you have any questions