FlakeGuard fails to run integration tests for Electron app (Swell) with "Unexpected end of JSON input" error

[12mv2]

Description

When attempting to run FlakeGuard on integration tests for the Swell Electron application, I consistently receive a "SyntaxError: Unexpected end of JSON input" error. This occurs for all integration tests, while unit tests run successfully with FlakeGuard.

Thanks in advance for your help (as I know Y'all are NOT Flakes!)

Reproduction

https://github.com/open-source-labs/Swell

System information

System:
OS: macOS 14.5
CPU: (8) arm64 Apple M3
Memory: 46.84 MB / 16.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 16.15.0 - ~/.nvm/versions/node/v16.15.0/bin/node
npm: 8.19.4 - ~/Dev/Codesmith/GitHub/Others-Repos/Swell_15_04/node_modules/.bin/npm
Browsers:
Chrome: 127.0.6533.120

Additional information

Things Ive tried:

I have tried using nvm 16, 18, 19, 22

Running integration tests directly with Mocha also fails, which might suggests that the issue might be related to how the integration tests are set up or how they're interacting with the Electron environment, rather than a simple configuration issue with FlakeGuard.:

➜ test git:(master) ✗ npx flake-guard "npx mocha ./test/IntegrationTests/httpIntegrationTests.js"
Number of runs: 10
Error in run number 1: SyntaxError: Unexpected end of JSON input
Error in run number 2: SyntaxError: Unexpected end of JSON input
Error in run number 3: SyntaxError: Unexpected end of JSON input
Error in run number 4: SyntaxError: Unexpected end of JSON input
Error in run number 5: SyntaxError: Unexpected end of JSON input
Error in run number 6: SyntaxError: Unexpected end of JSON input
Error in run number 7: SyntaxError: Unexpected end of JSON input
Error in run number 8: SyntaxError: Unexpected end of JSON input
Error in run number 9: SyntaxError: Unexpected end of JSON input
Error in run number 10: SyntaxError: Unexpected end of JSON input
Total FlakeGuard runtime: 3.666 seconds
Results Summary:
{}
Results successfully sent to FlakeGuard server
Press Enter to open results in the dashboard, or Ctrl+C to exit...

Link to integration tests:
https://github.com/open-source-labs/Swell/tree/f1d1f71c4bffc46cb8963d71b010e256e6ff2c43/test

Pic of terminal with successful and failing tests:

package.json (from beginning til not pertinent):
{
"name": "swell",
"version": "1.18.0",
"description": "Swell",
"main": "main.js",
"repository": "https://github.com/open-source-labs/Swell",
"scripts": {
"report": "istanbul report --dir ./test/coverage/total-coverage html",
"test": "jest && webpack --mode=production --config ./webpack.production.js && cross-env process.env.NODE_ENV=test nyc --report-dir ./test/coverage/mocha-coverage --reporter json --reporter text --reporter html mocha --timeout 15000 --exit",
"server-sse": "node ./test/SSE_HTTP1_server.js",
"server-gql": "node ./test/graphqlServer.mjs",
"server-grpc": "node ./test/grpcServer.js",
"server-http": "node ./test/httpServer.js",
"server-http2": "node ./test/HTTP2_server.js",
"server-trpc": "ts-node ./test/tRPC_Test_Servers/Express_Test_Server/server/api.ts",
"server-websocket": "node ./test/websocketServer.js",
"server-webrtc": "node ./test/webrtcWSServer.js",
"test-jest": "jest",
"test-mocha": "webpack --mode=production --config ./webpack.production.js && cross-env process.env.NODE_ENV=test nyc --report-dir ./test/coverage/mocha-coverage --reporter json --reporter text --reporter html mocha --timeout 15000 --exit",
"test-mocha-zero": "webpack --mode=production --config ./webpack.production.js && cross-env process.env.NODE_ENV=test mocha --timeout 0 --exit",
"format": "prettier --write "**/.+(js|jsx| tsx| json|css|md)"",
"lint": "eslint .",
"lint:fix": "eslint --fix . ",
"dev": "concurrently --success first "webpack-dev-server --mode=development --config ./webpack.development.js" "nodemon --legacy-watch ./src/server/server.js"",
"build": "concurrently --success first "webpack --mode=production --config ./webpack.production.js"",
"prod": "concurrently --success first "webpack --mode=production --config ./webpack.production.js && electron ."",
"package-mac": "webpack --mode=production --config ./webpack.production.js && electron-builder build --x64 --mac",
"package-all": "webpack --mode=production --config ./webpack.production.js && electron-builder build -mwl",
"package-win": "webpack --mode=production --config ./webpack.production.js && electron-builder build --x64 --win",
"package-linux": "webpack --mode=production --config ./webpack.production.js && electron-builder build --linux",
"gh-publish-mac": "electron-builder build --x64 --mac -p always",
"gh-publish": "electron-builder build -mwl -p always",
"check-types": "tsc",
"postinstall": "patch-package"
},
"build": {
"npmRebuild": false,
"productName": "Swell",
"appId": "com.getswell.Swell.app",
"extends": null,
"files": [
"dist/",
"menu/mainMenu.js",
"package.json",
"main.js",
"main_process/",
"preload.js",
"src/server/"
],
"nodeVersion": "16.15.0",
"nsis": {
"createDesktopShortcut": "always"
},

👨‍👧‍👦 Contributing

• 🙋‍♂️ Yes, I'd love to make a PR to fix this bug!

[willsuto]

Thanks so much for opening this issue! I think you've hit on two salient points here:

1. Flake Guard currently only works with test suites using the Jest framework. In the Swell repo, the unit tests that you were able to run successfully with flake-guard use the Jest framework. The other tests that weren't working use other test-running frameworks, so they are not compatible with flake-guard currently. This is something we would like to address going forward by expanding the existing FlakeGuard logic to accommodate other testing frameworks, particularly popular E2E testing tools such as Playwright and Cypress, especially since the types of tests run in these frameworks tend to be the ones most prone to test flake.

2. I think we can strive for better documentation and error handling in the NPM package to make the type of issue you encountered easier to understand.

Thank you so much for using FlakeGuard and providing feedback! We truly appreciate it and will always welcome any other feedback or contributions.