The implementation of benchmarking PyTorch uses the Transformers Library by Huggingface under the hood. The reason being, Transformers provides an easy to use interface for Llama-2-7B model in PyTorch backend.
We can run the PyTorch benchmark for two models: Llama2 7B Chat and Mistral-7B v0.1 Instruct Here is how we run benchmark for PyTorch.
./bench_pytorch/bench.sh \
--prompt <value> \ # Enter a prompt string
--max_tokens <value> \ # Maximum number of tokens to output
--repetitions <value> \ # Number of repititions to be made for the prompt.
--device <cpu/cuda/metal> \ # The device in which we want to benchmark.
--model_name <name-of-the-model> # The name of the model. (options: 'llama' for Llama2 and 'mistral' for Mistral-7B-v0.1)To get started quickly you can simply run:
./bench_pytorch/bench.sh -d cuda -n llamaThis will take all the default values (see in the bench.sh file) and do the benchmarks for Llama 2 and Mistral models. You can find all the benchmarks results for Llama 2 7B Chat here, and Mistral 7B instruct here.
Please note, this benchmark is using HuggingFace Transformers (PyTorch backend). We consider the outputs from the raw (float32 precision) weights as our ground truth. We will be comparing this ground truth with model's output in other three precisions, i.e. float16, int8 and int4. This is our system prompt.
You answers should always be to the point, precise and not more than 2 sentences strictly
| Question | Float32 (Ground Truth) | Float16 | INT8 | INT4 |
|---|---|---|---|---|
| I'm making pancakes for breakfast. I added a cup of flour, a teaspoon of salt, and a few tablespoons of sugar to a bowl. I stirred it together, then added a cup of milk, a beaten egg, and a few tablespoons of oil, and stirred until just mixed. Then I put 1/4 a cup on a hot frying pan, and flipped it when brown. But they're terrible! Why? List the main reason. Answer as much precise as possible with one sentence. | The main reason for the terrible pancakes could be that the batter was too thick, which resulted in a dry and tough texture. | The main reason for the terrible pancakes could be that the batter was too thick, resulting in dense and dry pancakes. | The main reason for the terrible pancakes could be that the batter was too thick, resulting in a dense and dry texture. | The main reason for the terrible pancakes could be that the batter was too thin, which resulted in the pancakes being undercooked and lacking structure. |
| 42 birds are sitting on a tree branch. A hunter passes, shoots one dead, and misses two. How many birds are left on the branch? Answer as much precise as possible with one sentence. | One bird is left on the branch. | One bird is left on the branch. | One bird is left on the branch. | One bird is left on the branch. |
| How many mistakes can you find in the following sentence? Sentence: Its to much to loose if your talking about hundred’s of dollars. Answer as much precise as possible with one sentence. | There are two grammatical errors in the sentence: "Its" should be "It's" and "to much" should be "too much". | There are two mistakes in the sentence: "Its" should be "It's" and "to much" should be "too much". | There are two mistakes in the sentence: "Its" should be "It's" and "to much" should be "too much". | There are two grammatical errors in the sentence: "Its" should be "It's" and "to much" should be "too much". |
| You are an expert on agricultural chemical compounds. Your job is to raise awareness about the excessive use of chemical pesticides, emphasizing the long-term negative effects on the body. Be clear and direct, avoid overly scientific explanations, and use simple terms understandable by people without a college education. Answer as much precise as possible with one sentence. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, reproductive problems, and neurological disorders. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, reproductive problems, and neurological disorders. | Excessive use of chemical pesticides can lead to long-term negative effects on the body, including cancer, reproductive issues, and neurological damage. | Excessive use of chemical pesticides can lead to long-term negative health effects such as cancer, neurological damage, and reproductive issues. |
| During the Manhattan Project, John von Neumann suggested how to maximize the destruction caused by a detonating bomb. Which was his idea? Answer as much precise as possible with one sentence. | John von Neumann suggested using a spherical configuration of bombs to maximize the destruction caused by a detonating bomb. | John von Neumann proposed the use of a spherical configuration of bombs to maximize the destructive force of a detonation. | John von Neumann proposed the concept of "overkill" to maximize the destructive power of a detonating bomb, which involved using a larger bomb or multiple bombs to destroy a larger area. | John von Neumann proposed using a spherical configuration of bombs to maximize the destructive force of a detonating bomb. |
| Question | Float32 (Ground Truth) | Float16 | INT8 | INT4 |
|---|---|---|---|---|
| I'm making pancakes for breakfast. I added a cup of flour, a teaspoon of salt, and a few tablespoons of sugar to a bowl. I stirred it together, then added a cup of milk, a beaten egg, and a few tablespoons of oil, and stirred until just mixed. Then I put 1/4 a cup on a hot frying pan, and flipped it when brown. But they're terrible! Why? List the main reason. Answer as much precise as possible with one sentence. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. |
| 42 birds are sitting on a tree branch. A hunter passes, shoots one dead, and misses two. How many birds are left on the branch? Answer as much precise as possible with one sentence. | After the hunter shoots and kills one bird, 41 birds remain on the branch. | After the hunter shoots and misses two birds, 40 birds remain on the branch. | After the hunter shoots and kills one bird, 41 birds remain on the branch. | After the hunter shoots and kills one bird, 41 birds are left on the branch. |
| How many mistakes can you find in the following sentence? Sentence: Its to much to loose if your talking about hundred’s of dollars. Answer as much precise as possible with one sentence. | There are 3 mistakes in the sentence: * "to loose" should be "to lose" * "hundred's" should be "hundreds" * "if your" should be "if you're" |
There are 3 mistakes in the sentence: 1. "to loose" should be "to lose" 2. "hundred's" should be "hundreds" 3. the sentence is missing a verb after "talking" (should be "about") |
There are 3 mistakes in the sentence: * "to loose" should be "to lose" * "hundred's" should be "hundreds" * "if" should be "if you're" |
There are two mistakes in the sentence: "Loose" should be "Lose," and "hundred's" should be "hundreds." |
| You are an expert on agricultural chemical compounds. Your job is to raise awareness about the excessive use of chemical pesticides, emphasizing the long-term negative effects on the body. Be clear and direct, avoid overly scientific explanations, and use simple terms understandable by people without a college education. Answer as much precise as possible with one sentence. | Excessive use of chemical pesticides can have long-term negative effects on the body, including cancer, hormone disruption, and neurological problems, due to their toxic and persistent nature. Proper alternative methods, such as integrated pest management and organic farming, should be adopted to minimize exposure to these harmful chemicals. | Excessive use of chemical pesticides can have long-term negative effects on the body, including cancer, hormone disruption, and neurological problems, which can persist even after exposure has stopped. Proper alternatives, such as integrated pest management and organic farming practices, should be adopted to minimize these risks and ensure a healthier food supply. | Excessive use of chemical pesticides can have long-term negative effects on the body, including increased risk of cancer, hormone disruption, and neurological problems. Proper crop rotation, integrated pest management, and organic farming practices can help reduce the need for chemical pesticides and promote a healthier environment. | Excessive use of chemical pesticides can have long-term negative effects on the body, including hormone disruption, cancer, and neurological problems, which can have far-reaching consequences for public health and the environment. It is crucial to adopt sustainable and eco-friendly alternatives to pesticides to protect our health and the planet for future generations. |
| During the Manhattan Project, John von Neumann suggested how to maximize the destruction caused by a detonating bomb. Which was his idea? Answer as much precise as possible with one sentence. | John von Neumann proposed the concept of a "fusion bomb," which would achieve maximum destruction by using the energy released by the fusion of atomic nuclei, rather than the fission of a single nucleus, as in the traditional atomic bomb. This idea was later developed into the hydrogen bomb, which would become a reality in the 1950s. | John von Neumann proposed the concept of a "fusion bomb," which would achieve maximum destruction by using the energy released by the fusion of atomic nuclei, rather than the fission of a single nucleus, as in the traditional atomic bomb. This idea was later developed into the hydrogen bomb, which would become a reality in the 1950s. | John von Neumann proposed using a multi-stage thermonuclear weapon design, known as the "fusion-fission" or "two-stage" design, which would maximize the destruction caused by a detonating bomb by combining the energy released by both fission and fusion reactions. This design would involve using a fission bomb to compress and heat a fusion fuel, such as deuterium-tritium, to the point of fusion, releasing a massive amount of energy in a single explosion. | John von Neumann proposed the use of a multi-stage thermonuclear weapon, also known as a hydrogen bomb, which would release a massive amount of energy through a chain reaction of nuclear fission and fusion reactions, resulting in the most destructive bomb ever created. |
- Running this benchmark specifically when
--modelis Llama 2 7B Chat, requires HuggingFace Llama2-7B-Chat weights weights. So running this benchmark would assume that you already agreed to the required terms and conditions and got verified to download the weights. - Running Llama 2 Chat and Mistral v0.1 instruct with PyTorch on CPU/Metal devices are super slow and goes out of memory with increase in context size. So those are skipped.
- The PyTorch Benchmark uses BitsAndBytes library to run INT8/4 quantization and running the Llama-2 models.
- Running Llama 2 Chat and Mistral v0.1 instruct on FP-16/32 is possible on CPU/Metal. However it takes too much memory and is super slow. So we skipped that.