tools.python.CallFunction macro

[alextremblay]

Request

I'm very happy to see that carapace has bridges for some of the more popular python cli framework completion tools (click, argcomplete)

Every time one of these bridges is invoked, it executes the named python cli tool (ie watson in the example documentation) in a zsh completion context and lets the python cli framework's own cli completion logic produce completion candidates for carapace to use

The only problem i have with this approach is the heavy performance cost. every time completion is requested for one of these python tools, it executes the python cli app in a subprocess, which imports the whole tree of python modules depended on by that cli tool (most of which won't be needed), constructs the entire argument parser tree generated by the CLI framework (ie click or argparse/argcomplete) and then executes completion code in python to generate completions.

This is fine for small CLI apps, but quickly becomes VERY noticable to end users for larger CLI apps with nested trees of commands (sometimes >3 second delay between user hitting <TAB> and carapace returning values)

My idea to solve this is to pre-generate carapace Specs for each of my python cli tools, to be installed as part of the cli tool installation process. I'm writing a tool at the moment which takes a click.Group object, introspects it, and generates a carapace Spec yaml file

So far, this approach is proving very promising! every feature / capability expressible in a click CLI parser can be directly mapped to a carapace Spec option.

The only place where this falls short is in custom completion functions. Many python CLI frameworks allow you to specify python functions to execute during shell completions. These functions take in some object representing the current state of the parser (ie. which flags have been parsed so far, what positional arguments have been provided so far, their values, etc), as well as a string representing the current word being completed. These functions are expected to return a list of strings representing viable completion candidates.

here's a concrete example:

def completion_function(ctx, param, incomplete):
    return [k for k in os.environ if k.startswith(incomplete)]

@click.command()
@click.argument("name", shell_complete=completion_function)
def cli(name):
    click.echo(f"Name: {name}")
    click.echo(f"Value: {os.environ[name]}")

Proposed solution

I would love to see a custom macro that could be invoked something like this:

name: example-python
completion:
  positional:
    - ["$carapace.tools.python.CallFunction(my_package.my_module:completion_function)"]

which, when invoked, would do the following:

1. serialize all carapace variables (flags, args, everything available via https://carapace-sh.github.io/carapace-spec/carapace-spec/variables.html) into a JSON string
2. exec something like python -c 'from python_package.python_module import completion_function; print(completion_function("<carapace variables json string>"))
3. collect the results in much the same way you would with the "exec" macro (ie. ["$(echo -e 'a\nb\nc')"])

Anything else?

we can already very nearly get there with:

name: example-python
completion:
  positional:
    - ["$(python -c 'from python_package.python_module import completion_function; print(completion_function(\"${C_VALUE}\")))"]

but:

1. there's no effective way to pass in all carapace values
2. the nested brackets and double-nested double-quotes are gnarly, hard reason about, and all too easy to mess up.

Also, as a side note, i think this feature would be of value to anyone using carapace-bin, not just python CLI app authors.

With this feature, anyone writing custom carapace spec files could leverage this to write custom completion functions

Polar

[rsteube]

Yeah. the slow startup time of python is an issue. But it's also how click works.
The spec generation is a good approach, i've already got a couple scrapers for other frameworks.

Not sure about the function call though. There's also Context and Paramater in shell_complete so it's a bit more complicated.

In theory the application could be bridged at a given position with the full command line to let the click parser do it's job.
But that's not available anymore at that point.

[rsteube]

For simple stuff this should work (just needs to get rid of positional args for flags with shift):

# yaml-language-server: $schema=https://carapace.sh/schemas/command.json
name: example
commands:
  - name: sub
    flags:
      -s, --string=: some string
    completion:
      flag:
        string: ["$carapace.bridge.Click([example, sub, --string]) | shift(99)"]
      positional:
        - [one, two]

[alextremblay]

Thanks, That's very helpful!

I still wish we could have a macro of some kind to pass parsed args and opts into the called program.
It would be incredibly useful to enable shell completion in CLI frameworks that don't currently have / support shell completion, like for example the cyclopts project, which I've had my eye on for a while

[rsteube]

Those are avalable as environment variables and positional arguments ($@):

# yaml-language-server: $schema=https://carapace.sh/schemas/command.json
name: context
persistentflags:
  -p, --persistent: persistent flag
commands:
  - name: sub
    flags:
      -s, --string=: string flag
      -b, --bool: bool flag
      --custom=: custom flag
    completion:
      flag:
        custom: ["$(env)"]

context --persistent sub --string one -b arg1 arg2 --custom C_[TAB]
# C_ARG0=arg1                                                                                                                              
# C_ARG1=arg2                                                                                                                              
# C_FLAG_BOOL=true                                                                                                                         
# C_FLAG_STRING=one                                                                                                                        
# C_VALUE=C_

[alextremblay]

Wow, that is incredibly useful undocumented behaviour!

id be happy to close this issue and submit a PR to add this to the exec macro’s documentation, if you’d like

[rsteube]

Sure, you can also link this in the documentation.

[alextremblay]

Excellent, will do!

btw, one thing I’m not clear on in the variables document: “… during multipart completion“

What is multipart completion?

[rsteube]

https://carapace-sh.github.io/carapace/carapace/defaultActions/actionMultiParts.html

There's still a lot to do on the specs, which is why the documentation is so lacking.

[rsteube]

see also the examples at https://carapace-sh.github.io/carapace/carapace/context.html#examples

[alextremblay]

Fantastic! Thank you for the extra resources! As a non-golang-developer, I hadn’t thought to look at the carapace library docs for extra info. I’ll be sure to link to those pages as well where appropriate

[alextremblay]

Also, for adding to exec documentation, I plan to add a more thorough “real-world” example for how to use the exec macro with parsed arg environment variables.

Note to self since I’m on mobile and can’t test code right now:

# yaml-language-server: $schema=https://carapace.sh/schemas/command.json
name: context
persistentflags:
  -p, --persistent: persistent flag
commands:
  - name: sub
    flags:
      -s, --string=: string flag
      -b, --bool: bool flag
      --custom=: custom flag
    completion:
      flag:
        custom: ["$(python -c 'from my_package.my_module import completion_function; completion_function())"]

# my_package/my_module.py

import os

def completion_function():
    # all opts and args already parsed by carapace are provided as env vars
    # if user typed `context --persistent sub --string one -b arg1 arg2 --custom unfinished-word-` and hit <TAB>,  then os.environ would contain
    # C_ARG0=arg1                                                                                                                              
    # C_ARG1=arg2                                                                                                                              
    # C_FLAG_BOOL=true                                                                                                                         
    # C_FLAG_STRING=one                                                                                                                        
    # C_VALUE=unfinished-word-
   completion_candidates = … # completion logic goes here
    for candidate in completion_candidates:
        print(candidate)
        # or if you’re feeling fancy
        print(f”{candidate}\tcandidate description”)

[alextremblay]

btw, while testing the newly-documented behaviour discussed here, i noticed that caparace doesn't include any flags "outside the current scope" in the context that it passes into executed commands.

ie: in your example above,

context --persistent sub --string one -b arg1 arg2 --custom C_[TAB]
# C_ARG0=arg1                                                                                                                              
# C_ARG1=arg2                                                                                                                              
# C_FLAG_BOOL=true                                                                                                                         
# C_FLAG_STRING=one                                                                                                                        
# C_VALUE=C_

i would have expected C_FLAG_PERSISTENT=true to be included in the environment, but it's not...

Is this a bug, or intended behaviour?

[rsteube]

Sounds like bug 🐛 .
Best guess is the flagset isn't merged 🤔 - an issue I had here as well: https://github.com/carapace-sh/carapace/blob/master/traverse.go#L25

Workaround is to just call LocalFlags first. Think it's time to open an issue in the cobra repo for it.