Memory leak in the multimodal processor of Phi-3-vision

[hiro28844]

Describe the bug
When calling the Phi-3-vision multimodal processor, a memory leak appears to occur, causing memory usage to continuously increase.

To Reproduce
Run the following script:

from io import BytesIO
from tempfile import mkstemp

import requests
from PIL import Image

import onnxruntime_genai as og


print("Loading model...")
model = og.Model("/path/to/Phi-3-vision-128k-instruct-onnx-cpu/cpu-int4-rtn-block-32-acc-level-4")
processor = model.create_multimodal_processor()

response = requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", timeout=30)
_, image_path = mkstemp(suffix=".png")
Image.open(BytesIO(response.content)).convert("RGB").save(image_path)
image = og.Images.open(image_path)

while True:
    prompt = "<|user|>\n"
    prompt += "<|image_1|>\n"
    text = "Please describe the image in detail."
    prompt += f"{text}<|end|>\n<|assistant|>\n"
    print("Processing image and prompt...")
    r = processor(prompt, images=image)
    del r

Expected behavior
Memory usage remains constant no matter how many times the multimodal processor is called.

Desktop (please complete the following information):

• OS: Ubuntu 20.04.6 LTS
• onnxruntime-genai==0.4.0

[natke]

Hi @hiro28844, does it remain constant or grow? We do pre-allocate memory for the KV-cache to improve performance

[hiro28844]

Hi @natke ,

does it remain constant or grow?

It continues to grow. Attach a video when I run the script above.

memory_leak.mp4

[i-dubits]

Confirm memory leak in Phi-3-vision. Probably this is related to the following issue: #590
But fixing 'max_length' parameter suggested by @PatriceVignola does not change anything for me. Probably the image-text processing is somewhat different from text alone.
I am using cuda onnx-genai version. The GPU memory remains constant but CPU memory increases every iteration

[hiro28844]

@natke
Any update on this?