FlashbackMacroRecorder is a tool that allows you to record and playback macros on Windows applications. It uses image data and basic computer vision to recover in scenarios with variable lag and to add branching behaviors to macros.
This is particularly useful for Android applications being run through Windows.
FBMR can be used for:
- Automating repetitive tasks, like QA workflows.
- Building apps that perform complex UI interactions, e.g. a bot that takes community input to perform actions in a game like 'twitchplayspokemon'.
- Records mouse clicks and drag events along with image data, timings, and positional data.
- Can playback macros with the correct timing even if the application is lagging or if there is a network delay.
- Can add branching behaviors to macros by using image data to identify different states of the application.
- Supports Android applications that can be run on Windows (via ADB, Bluestacks, or Scrcpy).
- Only supports left clicks and drags on Windows applications.
- Supported Windows applications vary based on how the application is rendered.
Install python if you don't have it; tested on 3.10.
- Clone the GitHub repository:
git clone https://github.com/alac/flashbackmacros.git
- Install the dependencies:
pip install -r requirements.txt
- (For Android automation) Setup ADB Debugging.
If you plan on using this with an Android device or an emulator like Bluestacks. Turn on ADB Debugging.
On Bluestacks, you would go to Settings -> Advanced -> Enable ADB
With an Android device, you're likely going to use scrcpy. Head over there for the instructions.
python -m tools.runner [config name] [device name] --ui
This runs the macro with default settings a user interface with live screenshots and logging.
If you run into issues, check out debugging
For advanced behaviors like branching, check out advanced usage
- Add a scheduler to run macros on a user-defined schedule.
- Add a graph visualizer for the flow of a FlashbackMacro.
- Integrate image segmentation and object detection.
- Add an HTTP server so that FlashbackMacros can interface with non-Python applications.
- Add support for right clicks and keyboard actions on Windows applications.
Test are standard pytest fare.
Run them with pytest .\tests.
This project leverages several libraries for the Android side of things.
https://github.com/leng-yue/py-scrcpy-client