What are the instructions to uninstall LLM_Web_search and dependencies?

[ShaunCassidyPoster]

Hi! I have correctly installed LLM_Web_search in the webui environment. I think it's pretty cool, but I can't get it to work for me reliably, perhaps because I am limited to 13B and 7B models. These don't like to use the command no matter how I craft the system prompt. When they do, they are unable to digest the results.

In short I'd like to remove the extension and uninstall its dependencies (like langchain) from my Ooba environment. I saw when I was installing that conda took a long time to carefully check dependencies in my text-generation-webui\installer_files\env environment. I am hoping that there is a similar way to uninstall the late additions.

What is the proper method to uninstall the LLM_Web_search extension and its dependencies that would otherwise be orphaned?

Thank you!

[mamei16]

Hey, sorry to hear that the extension doesn't really work for you.
I am also limited to max. 13B models, so it should work with those (but of course, bigger models will likely be better at using the command and extracting info from the results). Feel free to to reach out if you'd like me to take a look with you together.

Now, regarding your actual request for uninstall instructions: Assuming you used the one_click install option of the text-generation-webui and are using MacOs or Linux, the following four steps should remove the extension files and all its dependencies:

1. Remove the text-generation-webui/extensions/LLM_Web_search folder, which contains all files of the extension
2. Run conda remove -p <path_to_your_environment> faiss-cpu, where you need to replace <path_to_your_environment> with the path to the /installer_files/env subfolder within the text-generation-webui folder
3. Run conda activate -p <path_to_your_environment>
4. Finally, run python -m pip uninstall duckduckgo-search beautifulsoup4 langchain unstructured rank_bm25 sentence-transformers

I haven't tried this yet, so let me know if that works!

[ShaunCassidyPoster]

Thank you! I will try that shortly. I think it would be easier for non-developers if it had some working defaults for various models. Also, I use chat-instruct mode, which may not be optimal.

I tried Mistral-7B-Instruct-0.2, Toppy-M-7B (could only repeat a greeting over and over), Mythomax-L2-13B, and LLAMA2-13B-Tiefighter. I could only coax 13B models to use the command, and only after I put it in brackets in the system prompt and asked them to use the "bracketed command", but much more often they would pretend to search the web and confabulate, even after they'd used it successfully once.

The regex was kind of difficult. I worked with it on debuggex as advised, and then had Bard help so it wouldn't truncate at apostrophes. I am guessing the statement I ended up with is still not great. It would be cool to select from descriptively-named settings (obscuring well-crafted regex statements) instead of tinkering.

I may have noticed one non-critical bug. I was having trouble getting the models to use the command, so I tried to force it using the "Start reply with..." text box, but it yielded an error: UnboundLocalError: cannot access local variable 'reply' where it is not associated with a value. Fixing this would allow one to reliably produce search results when their bots are recalcitrant.

When my models successfully used the command, they did not use the results to inform a response. Right after the results printout, in the same response, they would continue with a stock greeting, apparently having lost context completely. Their context would be subsequently refreshed with my next reply.

I'd be happy to hear any ideas you might have, and I will keep trying things throughout the week. I can tell everything you done is put together well, however. Thanks!

[mamei16]

Thank you for your detailed answer, especially regarding the regex. Let me try to address the issues you have faced, based on my experience.

Models

Regarding good models to try, here are some that I found can consistently use the command and the search results:

1. Mistral-7B-OpenOrca
   A good 7B model wrt. using the command and not excessively hallucinating, but will at times claim to have the searched the web without actually having used the command (and you will not convince it otherwise).
2. SOLAR-10.7B-Instruct-v1.0
   Good at using the command and extracting information, but it is very confident in its hallucinations being correct, when they occur.
3. Starling-LM-11B-alpha
   Extremely verbose and prone to hallucinations, but when it works, it gives by far the best looking output (formatted in markdown and including clickable references to each source it uses in its answer)
4. Orca-2-13B
   So far the most consistent, but is strongly limited by the fact that I can only fit it in conjunction with a max. 2048 token context in VRAM.

Defaults for non-developers

This is a great suggestion and something I have thought about, but also quite difficult to get right. Here are some things I have considered:

1. Being able to save multiple regex strings and loading them (similar to custom system messages). Perhaps the name of each regex string file could match the custom system message file it should be used with.
2. Highlighting the parts in the custom system message that are matched by the current regex in green and otherwise showing a warning if the current regex doesn't match anything in the current system message.
3. Saving custom system messages and regex strings together. For example, if you load a custom system message, the current "Search" and "Open URL" command regex strings will be overwritten with ones matching the custom system message. Could be useful, but also confusing if the UI design doesn't convey how it works in an intuitive manner.

Concerning "working" defaults for various models: Since LLMs are non-deterministic in their output, a custom system message and matching regex that work for a particular model in one chat session may fail to work for the same model in the next session, so this is hard. In my experience, one of the websearch_gpt4_prompt websearch_python_syntax prompts almost always works.

Models not using the results to inform their response

This happens when the length of the results can not fit within the max. context length. Based on my understanding, this truncation of the context is performed by the model loader (in my case "llama.cpp"). You can trigger the same behavior by, e.g., pasting a very long text into a normal chat reply, so unfortunately there is not much the extension can do in that regard. You could try reducing the Max. search results to return per query setting under Advanced Settings.

Chat-instruct mode

This mode can indeed be sub-optimal, if you use a specific character prompt in conjunction with the system prompt. Any extra instructions in the initial prompt can make the model (especially smaller ones) less likely to focus on the instruction related to using the search command. Personally, I use chat-instruct with an empty context, simply to keep the greeting at the start.

Using "Start reply with" to force the model

The text you enter in this field will not be processed as model output (since it is handled by a different function), so you can't use the command here directly. However, you can try something like "Let me search the web for you!" and hope that the model will output the search command right after. If you want full control over when to search the web, I recommend trying out another web UI like the hugging face chat ui, which has a button to toggle web search on/off for each LLM reply.
BTW: If you can consistently reproduce the UnboundLocalError, I would be very grateful if you opened an issue for it, even if it isn't critical.

[ShaunCassidyPoster]

Hi! Today, I had some success with SOLAR-10.7B-Instruct-v1.0-uncensored-GPTQ:gptq-4bit-32g-actorder_True (I have 12 GB VRAM). I made the system prompt minimal, but still included the command in brackets. I find the models can find "bracketed" stuff easier and are less likely to break the command at the space. The plain Assistant in chat-instruct was able to repeatedly search for info. I reduced the specificity of my request with each try. It was even able to respond correctly to a generic ask: "Would you please look up Dave Mustaine's birthday?"

It is nearly working in chat-instruct with a character card that has ~1000-token context. The Oobabooga extension AllTalk TTS requires the greeting message to be in quotes. I believe this may cause the model to escape the quotes of the web search command: the character says, "[Search_web: \"Nicole Kidman height\"]". I did get the character to unescape the search command once, but not repeatedly. But it does try to search repeatedly, which is good.

The model is able to summarize its results without "Display search results in chat" and "Display extracted URL content in chat" selected. I assume the former would be okay for me to select without overfilling context, but not the latter, so I will turn that on.

I think I can also produce another UnboundLocalError by selecting "Continue" on a truncated message in which a search has been successful. I will submit these when I am able to repeat them, but I am usually in chat-instruct with a character loaded.

Thanks for your thorough reply! I will keep at it since I'm almost there. Please let me know if you have any ideas how to get it to work with the quoted greeting (Parameters > Character> "Greeting") as AllTalk requires. HTML character entities or something. I'm not sure yet.

[mamei16]

Hey, great to hear you had some success!
I have also found the command syntax involving a space to be lackluster, so I have now updated the default syntax to use parentheses and no longer any space.

If the character likes to sometimes escape the quotes around the search query, you can try to use a regex that allows both escaped and non-escaped quotes. For your bracketed syntax, try:

\[Search_web: \\?"(.*?)(?:\\"|")\]