Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement autofix for 91x #158

Merged
merged 1 commit into from
Mar 20, 2023

Conversation

jakkdl
Copy link
Member

@jakkdl jakkdl commented Mar 15, 2023

does still not finish #70 😅 It "merely" inserts checkpoints wherever needed.

TODO:

  • insert checkpoints
  • add import if necessary
  • insert anyio checkpoint if anyio is the preferred library (and anyio import)
  • handle multiple yields on the same line (but they're not autofixed)
  • save state on new class members
  • add some more comments
  • add review comments

And added a couple more tasks to #124 before this is reasonably usable by an end user. That includes checks for whether autofix toggling actually is toggled, which I kinda know isn't done properly by this anyway (but atm it mostly doesn't matter cause the new file doesn't get written without the autofix flag). But I'll leave those for another PR.

@jakkdl jakkdl force-pushed the autofix_91x_forealthistime branch 3 times, most recently from 0bf4add to b700100 Compare March 15, 2023 15:16
@Zac-HD
Copy link
Member

Zac-HD commented Mar 15, 2023

Exciting!

@jakkdl jakkdl force-pushed the autofix_91x_forealthistime branch from b700100 to 49166b6 Compare March 16, 2023 11:14
@jakkdl jakkdl marked this pull request as ready for review March 16, 2023 11:14
Comment on lines -145 to -151
@property
def artificial_errors(self) -> set[cst.Return | cst.FunctionDef | cst.Yield]:
return self.loop_state.artificial_errors

@artificial_errors.setter
def artificial_errors(self, value: set[cst.Return | cst.FunctionDef | cst.Yield]):
self.loop_state.artificial_errors = value # pragma: no cover
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

randomly got irritated by this one as I had to modify the typing on it so I removed it, it was barely used anyway and you might not even noticed the diff access below.

for statement in self.uncheckpointed_statements:
self.error_91x(node, statement)
self.restore_state(original_node)
return updated_node # noqa: R504
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

R504 irritated me so much, but realized it wasn't just me writing esoteric code and opened an issue afonasev/flake8-return#133

Comment on lines +259 to +279
# TODO: generate an error in these two if transforming+visiting is done in a single
# pass and emit-error-on-transform can be enabled/disabled. The error can't be
# generated in the yield/return since it doesn't know if it will be autofixed.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unlikely I'll go with that design, transform+visit in single pass I'm pretty sure would guarantee that errors are output with incorrect line markers so I think you have to first transform, then revisit without transform. But would be substantial runtime reduction if you can do in single pass ofc, so will have to play around a bit with it.

@@ -116,3 +119,22 @@ def visit_Import(self, node: ast.Import):
name = alias.name
if name in ("trio", "anyio") and alias.asname is None:
self.add_library(name)


@utility_visitor_cst
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty much just a copy of the ast version

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remaining todo that [selectively] disabling autofix works, but now testing autofix for both trio and anyio.

@@ -494,10 +589,31 @@ def leave_While_orelse(self, node: cst.While | cst.For):

# reset break & continue in case of nested loops
self.outer[node]["uncheckpointed_statements"] = self.uncheckpointed_statements
self.restore_state(node)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needed to be moved to leave_While since it has to be done after inserting checkpoints in the loop body.

flake8_trio/visitors/visitor91x.py Show resolved Hide resolved
@jakkdl jakkdl force-pushed the autofix_91x_forealthistime branch from 49166b6 to 6f20d79 Compare March 16, 2023 13:00
# returns a bool indicating if any real (i.e. not artificial) errors were raised
# so caller can insert checkpoint before statement (if yield/return) or at end
# of body (functiondef)
def check_function_exit(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved all artificial statement logic from error_91x to check_function_exit

@@ -394,6 +493,8 @@ def visit_While(self, node: cst.While | cst.For):
self.loop_state = LoopState()
self.infinite_loop = self.body_guaranteed_once = False

visit_For = visit_While
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooooops, pretty nasty line to have forgotten about. Added a test that would've caught it.

flake8_trio/visitors/visitor91x.py Show resolved Hide resolved
Comment on lines +728 to +729
This is *slightly* different enough from what the utility visitor does
that it's added here, but the functionality is maybe better placed in there.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'd probably just need another bit saved in self.library on whether it's been explicitly imported, or just assumed/specified with --anyio. But I'll procrastinate on that with the on-the-spot-made-up-reasoning that there's ast visitors that use the current self.library functionality and I neither wanna update both nor have different library logic between them.

Comment on lines +306 to +319
async def foo_while_endless_3():
while True:
...
yield # type: ignore[unreachable]
await foo()


async def foo_while_endless_4():
await foo()
while True:
yield
while True:
await foo()
yield
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember why I added these 😇

tests/test_flake8_trio.py Show resolved Hide resolved
@jakkdl
Copy link
Member Author

jakkdl commented Mar 16, 2023

Exciting indeed! Although not sure how you can be excited about reviewing modifications to a visitor that's up to 800 lines 😱 And that's before adding remove-unnecessary-checkpoint-logic!!!
Thankfully 1850 of the 2365 added lines are in generated autofix files, so you "only" have to review a diff of +501/-58 😅

Comment on lines +35 to +36
### autofix files
Checks that have autofixing can have a file in the `tests/autofix_files` directory matching the filename in `tests/eval_files`. The result of running the checker on the eval file with autofix enabled will then be compared to the content of the autofix file and will print a diff (if `-s` is on) and assert that the content is the same. `--generate-autofix` is added as a pytest flag to ease development, which will print a diff (with `-s`) and overwrite the content of the autofix file. Also see the magic line marker `pass # AUTOFIX_LINE ` below
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of spitting out the resulting .py files, can we save the diff between before and after as a .py.diff? It's much easier to get a sense of "what it's doing" in this format, and I'm concerned that without that code review is not going to be effective.

(although instead of constructing a three-way diff, I'd just let Pytest's assertion helper show the diff between the actual and expected diffs)

((diff diff diff, it hardly seems to diff mean anything anymore. diff)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the generated files for checking that they're valid and passes linters, and when developing they're easier to read and throw into cst.parse_module, but I'll generate diff files as well 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yep, both sounds good!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wohoo, bigger commits! 😁
They're still not perfect, as they're not showing e.g. where checkpoints aren't inserted - but that will be covered by future tests that autofixed files don't generate any errors. (and will have to figure out a solution for errors that are expected not to get autofixed)

@@ -112,7 +125,7 @@ def __init__(self, options: Namespace, module: Module):
def run(self) -> Iterable[Error]:
if not self.visitors:
return
for v in self.visitors:
for v in chain(self.utility_visitors, self.visitors):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for v in chain(self.utility_visitors, self.visitors):
for v in (*self.utility_visitors, *self.visitors):

Could also use + because they're both tuples, but the "anything iterable" syntax is visibly safe.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought the chain was nice and explicit, but don't mind either variant. The plus solution might also be slower, no?

@jakkdl jakkdl force-pushed the autofix_91x_forealthistime branch from 6f20d79 to 249eb2b Compare March 17, 2023 10:14
@jakkdl
Copy link
Member Author

jakkdl commented Mar 17, 2023

fixed comments

@jakkdl jakkdl force-pushed the autofix_91x_forealthistime branch from 249eb2b to cc43270 Compare March 17, 2023 10:18
@jakkdl
Copy link
Member Author

jakkdl commented Mar 17, 2023

except I had untracked files ... updated my aliases so that won't happen again ^^
This is so handy it's crazy:

alias untracked="git ls-files -o --directory --exclude-standard | sed q1 > /dev/null"
alias gfpush='untracked && pytest --quiet -x && git commit -a --amend --no-edit && git push -f'

@jakkdl
Copy link
Member Author

jakkdl commented Mar 17, 2023

Oh and I realized that the pass # AUTOFIX_INSERT was a big brainfart yesterday, autofixing is all about inserting lines - I'm not explicitly marking each checkpoint that should be inserted, they're just added/generated to the autofix file. So completely removed that.
Still need the isort: skip_file though since autofix doesn't insert an empty line before the added import, and trying to worry about that isn't really viable.

@jakkdl
Copy link
Member Author

jakkdl commented Mar 17, 2023

ahahaha, trio.lowlevel.checkpoint() sure doesn't checkpoint - you kiiiinda need to do await trio.lowlevel.checkpoint()

@jakkdl jakkdl force-pushed the autofix_91x_forealthistime branch from cc43270 to 220c405 Compare March 17, 2023 10:47
@jakkdl
Copy link
Member Author

jakkdl commented Mar 17, 2023

I'm writing the extra tests now, and this implementation still does some dumb stuff (yield inside boolops and stuff), but I'll leave (not) handling those for the next PR.

@Zac-HD
Copy link
Member

Zac-HD commented Mar 17, 2023

alias gfpush='untracked && pytest --quiet -x && git commit -a --amend --no-edit && git push -f'

--force-with-lease!

ahahaha, trio.lowlevel.checkpoint() sure doesn't checkpoint - you kiiiinda need to do await trio.lowlevel.checkpoint()

lint all the things 😜

@jakkdl
Copy link
Member Author

jakkdl commented Mar 20, 2023

Ooh, --force-with-lease is nice!

Well, there's no linter - other than flake8-trio itself, that catches unawaited trio.lowlevel.checkpoint()! Unless we change the pyright dependency from trio to trio-typing. That might definitely be good though, but I'm getting 59 errors when doing that - so needs a separate PR.

Copy link
Member

@Zac-HD Zac-HD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whew! Big diff to review, but this looks great. Thanks again @jakkdl!

@Zac-HD Zac-HD merged commit 033317a into python-trio:main Mar 20, 2023
@jakkdl jakkdl deleted the autofix_91x_forealthistime branch March 21, 2023 07:25
@jakkdl
Copy link
Member Author

jakkdl commented Mar 21, 2023

I'll try to get the diffs to be smaller, but it's hard to avoid at this stage - and especially when working with the monster that is 91x 😅

@Zac-HD
Copy link
Member

Zac-HD commented Mar 21, 2023

Oh yeah, that's not a criticism! Just an expression of relief that I made it through 😋

@jakkdl
Copy link
Member Author

jakkdl commented Mar 21, 2023

Haha, just meant it's worth trying for. If I don't keep it in the back of my mind the scope creep of PR's is very real 😇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants