Local LLM issue

[chemical-accuracy]

Hi there,
I am trying to use Llama so I downloaded llava-v1.5-7b-q4.llamafile (4.29 GB), added x permission and executed
./llava-v1.5-7b-q4.llamafile -cb -np 4 -a my-llm-model --embedding
I can see the LLM running in the browser and it seems to work.

However, when I try to run the example code:

from paperqa import Settings, ask

local_llm_config = dict(
    model_list=dict(
        model_name="my_llm_model",
        litellm_params=dict(
            model="my-llm-model",
            api_base="http://localhost:8080/v1",
            api_key="sk-no-key-required",
            temperature=0.1,
            frequency_penalty=1.5,
            max_tokens=512,
        ),
    )
)

answer = ask(
    "What manufacturing challenges are unique to bispecific antibodies?",
    settings=Settings(
        llm="my-llm-model",
        llm_config=local_llm_config,
        summary_llm="my-llm-model",
        summary_llm_config=local_llm_config,
    ),
)

I get the following error:

Traceback (most recent call last):
  File "/MyFolder/pqa.py", line 17, in <module>
    answer = ask(
             ^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/__init__.py", line 92, in ask
    return get_loop().run_until_complete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/main.py", line 81, in agent_query
    response = await run_agent(docs, query, agent_type, **env_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/main.py", line 206, in run_agent
    answer, agent_status = await run_fake_agent(query, docs, **env_kwargs)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/main.py", line 238, in run_fake_agent
    _, tools = await env.reset()
               ^^^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/env.py", line 134, in reset
    self.state, self.tools = self.make_initial_state_and_tools()
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/env.py", line 124, in make_initial_state_and_tools
    self.tools = settings_to_tools(
                 ^^^^^^^^^^^^^^^^^^
  File "MyPathl/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/agents/env.py", line 40, in settings_to_tools
    llm_model = llm_model or settings.get_llm()
                             ^^^^^^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/settings.py", line 497, in get_llm
    return LiteLLMModel(
           ^^^^^^^^^^^^^
  File "MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/pydantic/main.py", line 209, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File MyPath/miniconda3/envs/pqa/lib/python3.12/site-packages/paperqa/llms.py", line 392, in maybe_set_config_attribute
    and len({m["model_name"] for m in data["config"]["model_list"]}) > 1
             ~^^^^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'

[dosubot[bot]]

Hey there, @chemical-accuracy!

I'm here to help you with any bugs, questions, or contributions while we wait for a human maintainer.

I found a similar issue that might help you resolve the error you're encountering. The error TypeError: string indices must be integers, not 'str' typically occurs when attempting to access elements of a string using a string key instead of an integer index. This can happen if the config dictionary is not structured correctly.

Here is the expected structure of the local_llm_config dictionary in the paperqa library:

local_llm_config = dict(
    model_list=[
        dict(
            model_name="my_llm_model",
            litellm_params=dict(
                model="my-llm-model",
                api_base="http://localhost:8080/v1",
                api_key="sk-no-key-required",
                temperature=0.1,
                frequency_penalty=1.5,
                max_tokens=512,
            ),
        )
    ],
    router_kwargs=dict(
        some_router_param="value"
    )
)

Notice that model_list should be a list of dictionaries, not a single dictionary. This might be the cause of the error you're seeing.

Additionally, there was a similar closed issue where the solution involved ensuring the correct structure and parameters for the local LLM configuration ^[1].

Try updating your local_llm_config dictionary to match the expected structure and see if that resolves the issue.

To continue talking to Dosu, mention @dosu.

[chemical-accuracy]

This didn't help. I am getting the same error!

[jamesbraza]

Sorry for this @chemical-accuracy, taking a look now.

It looks like firstly, you had a bad schema for Settings's summary_llm_config. I made #383 just now to ensure our validators catch this in the future.

As for this one:

litellm.exceptions.BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=my-llm-model
 Pass model as E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/starcoder',..)` Learn more: https://docs.litellm.ai/docs/providers

You will need to write a real model name instead of my-llm-model, unless I am missing something here.

As far as getting local models working, a new issue for us (#378) is open to document this area better

[chemical-accuracy]

Thanks for jumping in on this right away! I was following the README instructions. Can you update them?

Also, I am not sure what real model name to give it. As per these README instructions I used the alias flag when executing llava as follows: -a my-llm-model, so I expected that to be the model name.

[mskarlin]

Hey @chemical-accuracy -- I think you need to tell litellm the protocol for your model as part of the name (i.e. openai/gpt-4 for an OpenAI model and ollama/my-llm-model for an ollama model, per the litellm docs example like the one below:

from litellm import completion

response = completion(
    model="ollama/llama2", 
    messages=[{ "content": "respond in 20 words. who are you?","role": "user"}], 
    api_base="http://localhost:11434"
)
print(response)

So in your case, you'd need to use both model_name and model (within litellm_config) as ollama/my-llm-model.

Could you let ys know if that works?

[whitead]

More info in this Issue #390