[Adapter implementation] Is it possible to listen to test output stream?

[sidlatau]

Problem

When running test command for a file, all tests in that file get an "in progress" status, and only when all tests are finished we get the results. So from the user's perspective one second all tests are in progress and the other second all tests are finished.
When this command is being run in the terminal, I see results are streamed, some test result JSON arrives before other results. It would be nice to show the test results to the user as soon as they arrive.

Question

Is it possible somehow listen to the test output stream and process results as they arrive?

Ouput example

``` ❯ fvm flutter test test/app/manage/contact_form/manage_contact_form_page_test.dart --reporter json {"protocolVersion":"0.1.1","runnerVersion":null,"pid":4111,"type":"start","time":0} {"suite":{"id":0,"platform":"vm","path":"/Users/ts/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dar t"},"type":"suite","time":0} {"test":{"id":1,"name":"loading /Users/ts/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart","suite ID":0,"groupIDs":[],"metadata":{"skip":false,"skipReason":null},"line":null,"column":null,"url":null},"type":"testStart","time":1} {"count":1,"time":11,"type":"allSuites"} ../../../../.pub-cache/hosted/pub.dartlang.org/flutter_web_auth-0.3.2/lib/flutter_web_auth.dart:33:20: Warning: Operand of null-aware operation '?.' has type 'WidgetsBinding' which excludes null. - 'WidgetsBinding' is from 'package:flutter/src/widgets/binding.dart' ('../../../../fvm/versions/3.0.2/packages/flutter/lib/src/widgets/binding.dart '). WidgetsBinding.instance?.removeObserver(_resumedObserver); // safety measure so we never add this observer twice ^ ../../../../.pub-cache/hosted/pub.dartlang.org/flutter_web_auth-0.3.2/lib/flutter_web_auth.dart:34:20: Warning: Operand of null-aware operation '?.' has type 'WidgetsBinding' which excludes null. - 'WidgetsBinding' is from 'package:flutter/src/widgets/binding.dart' ('../../../../fvm/versions/3.0.2/packages/flutter/lib/src/widgets/binding.dart '). WidgetsBinding.instance?.addObserver(_resumedObserver); ^ ../../../../.pub-cache/hosted/pub.dartlang.org/flutter_web_auth-0.3.2/lib/flutter_web_auth.dart:47:20: Warning: Operand of null-aware operation '?.' has type 'WidgetsBinding' which excludes null. - 'WidgetsBinding' is from 'package:flutter/src/widgets/binding.dart' ('../../../../fvm/versions/3.0.2/packages/flutter/lib/src/widgets/binding.dart '). WidgetsBinding.instance?.removeObserver(_resumedObserver); ^ {"testID":1,"result":"success","skipped":false,"hidden":true,"type":"testDone","time":6821} {"group":{"id":2,"suiteID":0,"parentID":null,"name":"","metadata":{"skip":false,"skipReason":null},"testCount":8,"line":null,"column":null,"url":null },"type":"group","time":6826} {"group":{"id":3,"suiteID":0,"parentID":2,"name":"Manage contact form page","metadata":{"skip":false,"skipReason":null},"testCount":8,"line":22,"colu mn":3,"url":"file:///Users/ts/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"group","t ime":6827} {"test":{"id":4,"name":"Manage contact form page add - show add title","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false,"skipReason":null},"line ":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":48,"root_column":5,"root_url":"file:///Users/ts/Documents/github/mwo rker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":6828} {"testID":4,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":9847} {"test":{"id":5,"name":"Manage contact form page add, first contact - main contact set to ON","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false," skipReason":null},"line":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":54,"root_column":5,"root_url":"file:///Users/ ts/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":9848} {"testID":5,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":10076} {"test":{"id":6,"name":"Manage contact form page add, not the first contact - main contact set to OFF","suiteID":0,"groupIDs":[2,3],"metadata":{"skip ":false,"skipReason":null},"line":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":62,"root_column":5,"root_url":"file: ///Users/ts/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":10076} {"testID":6,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":10271} {"test":{"id":7,"name":"Manage contact form page edit - show add title","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false,"skipReason":null},"lin e":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":74,"root_column":5,"root_url":"file:///Users/ts/Documents/github/mw orker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":10271} {"testID":7,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":10470} {"test":{"id":8,"name":"Manage contact form page name not set, click save - show validation","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false,"s kipReason":null},"line":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":80,"root_column":5,"root_url":"file:///Users/t s/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":10471} {"testID":8,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":10841} {"test":{"id":9,"name":"Manage contact form page invalid email, click save - show validation","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false," skipReason":null},"line":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":89,"root_column":5,"root_url":"file:///Users/ ts/Documents/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":10842} {"testID":9,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":11575} {"test":{"id":10,"name":"Manage contact form page fill fields - fields saved in model","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false,"skipRea son":null},"line":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":105,"root_column":5,"root_url":"file:///Users/ts/Doc uments/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":11575} {"testID":10,"messageType":"print","message":"MissingStubError: 'createCustomerContact'\nNo stub was found which matches the arguments of this method call:\ncreateCustomerContact(\n ManageContactCreateDto {\n customerId=1,\n fullName=Name,\n title=Msr.,\n isMainContact=false ,\n phone=phone,\n mobilePhone=mobilePhone,\n [email protected],\n })\n\nAdd a stub for this method using Mockito's 'when' API, or generate the WebApiMock mock with a MockSpec with 'returnNullOnMissingStub: true' (see https://pub.dev/documentation/mockito/latest/annotations/MockS pec-class.html).","type":"print","time":12300} {"testID":10,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":12367} {"test":{"id":11,"name":"Manage contact form page update - update endpoint called","suiteID":0,"groupIDs":[2,3],"metadata":{"skip":false,"skipReason" :null},"line":153,"column":5,"url":"package:flutter_test/src/widget_tester.dart","root_line":139,"root_column":5,"root_url":"file:///Users/ts/Documen ts/github/mworker/flutter_mw/test/app/manage/contact_form/manage_contact_form_page_test.dart"},"type":"testStart","time":12368} {"testID":11,"messageType":"print","message":"MissingStubError: 'updateCustomerContact'\nNo stub was found which matches the arguments of this method call:\nupdateCustomerContact(\n ManageContactUpdateDto {\n contactId=1,\n customerId=1,\n fullName=Contact 1,\n isMainContact =false,\n [email protected],\n })\n\nAdd a stub for this method using Mockito's 'when' API, or generate the WebApiMock mock with a Mock Spec with 'returnNullOnMissingStub: true' (see https://pub.dev/documentation/mockito/latest/annotations/MockSpec-class.html).","type":"print","time": 12663} {"testID":11,"result":"success","skipped":false,"hidden":false,"type":"testDone","time":12723} {"success":true,"type":"done","time":12781} ❯

</details>

[rcarriga]

This is not possible right now but I'd definitely be open to adding a way to do it the adapter interface. What would you need to make this work? Just the stdout/stderr?

[sidlatau]

I am quite new to NeoVim API/Lua, so not sure what would be the best solution. But I imagine a callback in the adapter interface that is being called each 1s (or any configurable time) in the loop with output from stdout would allow implementing this functionality.

[stevearc]

This is something I've thought about as well. I'm away from my computer, but my initial idea would be an optional adapter method that will be called repeatedly with partial stdout/stderr, and maybe it could either return results or call directly into neotest with results as they appear in the output. We would still want the final results method to be able to override whatever progressive results exist because there are some test runners that print out incremental pass/fail, but the detailed test output/stack traces are saved until the entire suite has completed.

[rcarriga]

I do like the idea of the "progressive" results not being the final result.
Right now, I'm thinking we can use Lua iterators for a simple API:

function NeotestClient:run()
  local run_spec = adapter.build_spec(args)

  self:mark_running(args.tree)
  local process = strategy(run_spec)

  if run_spec.stream then
    -- Set this to run off, we don't need to block here
    async.run(function()
      for partial_results in run_spec.stream(process.output_stream) do
        -- These results can contain "preliminary" results which may change. I'm thinking mainly for namespaces/files/dirs
        -- which are aggregated. Would probably want to have a "still running" check in case a badly behaved adapter
        -- sent results after stopping.
        self:broadcast_results(partial_results)
      end
    end)
  end

  local strategy_result = process.result()

  local final_results = adapter.results(run_spec, strategy_result, args.tree)
  self:mark_not_running(args.tree)
  self:broadcast_results(final_results)
end

function MyAdapter.build_spec(args)
  return {
    -- Output stream is an async iterator that'll return data as it is read by the strategy.
    stream = function(output_stream)
      return function()
        local partial_output = output_stream()
        if not partial_output then
          return
        end
        return parse_partial_output(partial_output)
      end
    end,
  }
end

The only new code here would be within the if run_spec.stream. I've gone with a property of the run spec rather than a separate adapter method just as a counter thought, not sure which I'd prefer. Could also have the build_spec function return two values, to keep client logic separate from the strategies.

[stevearc]

I see, so the strategy would expose output_stream as an iterator? This seems fine to me. I don't have a strong opinion either way about where to put the adapter "stream" method. We'd also have to come up with the expected format for the output_stream data. The easiest may be the :help channel-lines format, but I have found that to be a bit of a pain to deal with if you're just trying to do line-by-line parsing. Maybe the stream could pass entire lines as they are completed?

[rcarriga]

I don't think channel lines would be work well because then strategies other than integrated would have to make it from their data (e.g. DAP just gets raw strings). On the other hand though it would be a pain for adapters to each have to split raw data. I do think passing split lines would be the best route. This could be done in the client or there could be a lib function used by the strategies. I think I'm leaning towards the former, and the client just takes raw strings from the strategy.

[rcarriga]

Linking this for history because GitHub doesn't show PRs that link a discussion #70

[rcarriga]

That looks correct to me. How much time after the result is streamed does the process exit? If it is short, the results may not be emitted before the final ones. Also please make sure you're on latest neotest, there was a small rendering issue for the summary (in case that's how you were watching results)

[sidlatau]

I can describe the problem more precisely now: status is not being updated in signColumn, when batch of tests are running. Here in the screenshot you can see that state - in summary one test is finished, but in signColumn status for test widget test1 is not green:

Should I register an issue for this, or this comment is good enough?

[rcarriga]

Ah yep I found the issue, the latest master contains a fix

[sidlatau]

Thanks, @rcarriga, I was able to implement results streaming in https://github.com/sidlatau/neotest-dart. . I am getting command output results line by line, so I need to buffer these lines myself. But it works :)

[rcarriga]

Great to hear!