Imported from pokipoi/visorn (
extensions/computer-use/skills/computer-use/SKILL.md). Install upstream withnpx skills add pokipoi/visorn --skill computer-use. Copyright stays with the author.
Computer Use Skill
This skill gives you eyes and hands on the user's desktop. You can see the screen, move the mouse, type text, and press keys — just like a human sitting at the computer.
Available tools
| Tool | When to use |
|---|---|
computer_screenshot |
See what's on screen. Always screenshot first before any interaction, and again after each action to verify the result. |
computer_click |
Click at (x, y) with left/right/middle/back/forward button. |
computer_double_click |
Double-click at (x, y). Opens files, selects words, etc. |
computer_type |
Type text character by character. For text input fields. |
computer_keypress |
Press key combos: ["ctrl","c"], ["enter"], ["alt","tab"], etc. |
computer_scroll |
Scroll the mouse wheel at a position. |
Operating principles
1. Always screenshot before acting
Before clicking or typing anything, take a screenshot to see the current state. This is your "eyes" — you cannot see the screen without it.
2. Work step-by-step
Follow this loop:
screenshot → analyze → decide action → perform action → screenshot → verify → repeat
Never batch multiple actions without screenshots in between. The screen state may have changed unexpectedly.
3. Coordinates are pixels
All coordinates (x, y) are in screen pixels measured from the top-left corner (0, 0). Analyze the screenshot carefully to determine the right coordinates for buttons, text fields, etc.
4. Be patient with animations
After clicking something that triggers an animation (opening a menu, switching tabs, loading a page), wait a moment before taking the next screenshot. The tool has a configurable delay (screenshotDelay setting).
5. Handle failures gracefully
If a click doesn't seem to work (the next screenshot shows no change):
- Try again — the coordinates may have been slightly off
- Try a different approach (e.g., keyboard shortcut instead of mouse click)
- Report the issue to the user
6. Key names reference
Common key names for computer_keypress:
- Modifiers:
ctrl,alt,shift,super/meta/win - Navigation:
up,down,left,right,home,end,pageup,pagedown - Actions:
enter/return,tab,escape/esc,backspace,delete/del,space - Function keys:
f1throughf12 - Other:
printscreen
Combine multiple keys in an array for shortcuts: ["ctrl", "shift", "t"] reopens a closed browser tab.
7. Platform notes
- Wayland (modern Linux): Uses
ydotoolfor input. May needsudoorinputgroup membership. - X11: Uses
xdotoolfor input. Works out of the box. - Screenshots: Uses Electron's built-in
desktopCapturer(always available). Fallbacks:grim(Wayland),scrot,gnome-screenshot, ImageMagickimport.
Example workflow
User: "Open Firefox and go to google.com"
computer_screenshot— see the desktopcomputer_click({ x: 45, y: 120 })— click the Firefox icon (from screenshot analysis)computer_screenshot— verify Firefox openedcomputer_click({ x: 600, y: 50 })— click the address barcomputer_type({ text: "google.com" })— type the URLcomputer_keypress({ keys: ["enter"] })— navigatecomputer_screenshot— verify the page loaded
Troubleshooting
| Problem | Solution |
|---|---|
| "No input automation backend available" | Install ydotool (Wayland) or xdotool (X11): sudo apt install ydotool or sudo apt install xdotool |
| Screenshot is black/blank | Some apps block screen capture. Try a different screenshot method or disable hardware acceleration. |
| ydotool doesn't respond | Ensure ydotoold daemon is running: sudo systemctl start ydotool |
| Coordinates seem off | Screen scaling may affect coordinates. Take a fresh screenshot and re-analyze. |