Microsoft Skill Recorder

1. Project Overview

Skill Recorder's one-sentence positioning is:

Let the user do the task in person, and then turn the demonstration into a reusable skill or automated process for AI agents.

It is not the "coordinate click playback" of traditional RPA, nor is it a simple screen recording tool. It is more like an Agent skill acquisition tool: it records a real working session of a person, then lets GitHub Copilot CLI understand "what exactly the person wants to accomplish and what key actions he has done in what order", and finally generates a reusable 'SKILL.md' or triggered/timed Automation.

2. What does it mostly do?

2.1 record a real work process

After the user clicks record, the task is completed as usual. Skill Recorder will be recorded locally:

-Screen video and low frequency keyframes.

-Active application and window switching.

-Browser URL,README clear macOS support.

-Clipboard short text preview.

-Optional voice narration, transcribe using the local Whisper model.

This allows the Agent to not only see the final result, but also understand how business people complete tasks across multiple tools such as web pages, terminals, forms, emails, CRM, and expense systems.

2.2 Refactor actions into intents and steps

After the user clicks on the Analyze, the Skill Recorder will send the event timeline, window/document title, URL, clipboard preview, key screen image and voice narrative text to GitHub cloud, which will be reconstructed by GitHub Copilot CLI:

-overall intent: The overall goal of the mission.

-ordered steps: Orderly steps to complete the task.

This analysis does not repeat clicks frame by frame, but attempts to abstract "generalizable" business actions.

2.3 Generate Reusable Skill

Generate a Skill, which is a 'SKILL.md' procedure, from the approved analysis results. The Agent can perform similar tasks in this procedure.

Key point: README emphasizes that the generated Skill will give priority to using Agent native tools, such as' gh' CLI, 'web_fetch,' instead of mechanically replaying UI clicks.

This is very important before the sale. Customers often worry about "whether the screen recording automation is very fragile". The idea Recorder by Skill is that recording the screen is only a collection example, and the final output is a more stable tool-based process.

2.4 Generate Automation

It can also turn the process into a Automation, I .e. run the same procedure at the scheduled time or trigger condition.

Suitable examples:

-Compile release notes from GitHub PR every week.

-Fetch prices from the web every day and write to the table.

-Check customer leads regularly and write to CRM.

-Organize monthly expense reimbursement materials.

2.5 Local Voice Narrative Transcrimation

The user can say "why am I doing this" while operating ". README description narration is optional, voice transfer is completed on this machine, Whisper the model will download about 252MB for the first time, supporting 99 languages of the Whisper.

This ability is valuable because many business steps cannot be understood just by looking at the screen. For example, "This supplier is chosen here because the amount exceeds the 5000 and requires advanced approval". This implicit rule can be supplemented by oral statement.

3. Workflow understanding

This process can be seen as a pipeline of "from human demonstration to Agent skills:

1. Demonstration: People do it once.
2. Observation: The system records the screen and context.
3. Interpretation:Copilot the refactoring intent and steps.
4. Generalization: Generalize a single sample into a reusable process.
5. Execution: The agent is run using native tools.

4. Core Competence List

5. Applicable Scenario

5.1 Agent Skill Collection and Precipitation

One of the biggest problems with enterprise landing agents is that business processes exist in a large number of employees' minds, are not fully documented, and often span multiple systems. The Skill Recorder allows the expert to "do it again" and then turn it into a draft of the skill that the Agent can understand.

Suitable for:

-Presales consultant to demonstrate standard documentation process.

-Operation and maintenance personnel demonstrate the troubleshooting process.

-Finance staff demonstrates the reimbursement review process.

-Operations staff to demonstrate the data collection and form update process.

-R & D Lead demonstrates GitHub issue/PR triage process.

5.2 RPA Upgrade to Agentic Automation

Traditional RPA relies heavily on UI coordinates, selectors, and fixed paths, and is prone to failure due to page changes. The selling point of the Skill Recorder is to "record the example, but generate the generalization step" and prefer to use CLI/API/native tools.

This is suitable for explaining to the customer:

-Instead of putting the human mouse track directly back.

-Recording only helps the Agent understand the process.

-Try to take a more stable tool interface when actually executing.

5.3 business process standardization

Many enterprises hope to standardize the practices of excellent employees, but it is expensive to write SOP. The Skill Recorder can assist in generating the first version of the SOP/Skill:

-Record the best practice first.

-Copilot generate intents and steps.

-Expert review and revision.

-Sedimentation as Skill or Automation.

5.4 Copilot/Agent Platform Pre-sales Demo

If the customer has no idea about "how Agent learns to do internal tasks", Skill Recorder is a very good Demo entrance.

Demonstrable Routes:

1. Record "Extract data from web page to table" once ".
2. Analyze intentions and steps.
3. Generate Skill.
4. Let the Agent execute on another similar page.

This is easier for the business side to understand than directly talking about the abstract Agent architecture.

5.5 Developer/Operation and Maintenance Process Automation

GitHub and CLI tasks are mentioned several times in README and evals, such as GitHub issue triage, stale PR nudge, release notes, Azure deploy. This kind of task is very suitable for Agent, because the data source and operation interface are relatively clear.

Suitable for:

-GitHub issue collation and reply.

-PR summary and release notes.

-CLI deploys and logs the URL.

-Web/terminal/form combined DevOps process.

6. Not quite the scene

7. Privacy and Security Boundaries

This part must be made clear before sale.

7.1 Local Phase

README descriptions: Recording, storage, frame extraction, and optional voice transcription all take place on this machine. It will not be uploaded automatically during recording.

Local collection includes:

-app/window switches

-macOS Browser URL

-Screen video and low frequency screenshots

-Clipboard Short Preview

-Optional narration text

7.2 Analyze Phase

Only after the user clicks on the Analyze will the data be sent to GitHub cloud for Copilot processing. Sent content includes event timeline, window/document title, URL, clipboard preview, extracted screen image, and narration text.

7.3 pre-sales advice

Rules must be established before customer PoC:

-Do not record passwords, tokens, API keys, and customer privacy data.

-Use mock account numbers and test data.

-Don't show real customer lists, financial data, contract information on the screen.

-Turn off unnecessary notifications to prevent sensitive information from entering screenshots.

-If the enterprise does not allow cloud analysis, the project is not suitable for direct use.

8. Installation and deployment methods

The Skill Recorder is currently the source-only release. In other words, GitHub Release does not come with pre-built app/installer, but provides commands: download the source code of the specified commit, and build it locally after pulling Node.js 24, Electron, GitHub Copilot CLI and dependencies.

The commit of the latest release v0.3.1 is:

32fd0b57e02c3ea1e016cca0d64e59052e93a9b9

macOS / Ubuntu:

commit="32fd0b57e02c3ea1e016cca0d64e59052e93a9b9"; curl -fsSL "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.sh" | SKILL_RECORDER_COMMIT="$commit" bash

Windows PowerShell:

$commit="32fd0b57e02c3ea1e016cca0d64e59052e93a9b9"; $env:SKILL_RECORDER_COMMIT=$commit; irm "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.ps1" | iex

Officials suggest a safer inspect-first method: download the script, verify the SHA-256, read the script, and then execute it.

Default installation location:

8.1 Developer Run

Node.js 24:

npm ci
npm run compliance:licenses
npm run dev

Common Scripts:

9. Engineering and Quality Signals

9.1 Eval Design

The project has a relatively complete eval harness, covering three types of phases:

Typical scenarios include:

-web-to-spreadsheet

-invoice-extract

-research-compile

-directory-lookup

-expense-report

-release-notes

-lead-to-crm

-windows-deploy

-GitHub issue triage

-Cowork Teams digest / Outlook reply / Calendar schedule

This shows that the project is not just doing UI prototyping, but is carefully evaluating the generalization quality "from demonstration to skill.

9.2 Windows ARM64 Verification

The project has a 'WINDOWS-VALIDATION.md, 'indicating that both Windows 11 x 64 and ARM64 have validation plans. It removes the "ffmpeg-static" and uses Chromium MediaRecorder and AudioContext processing for screen recording/voice, avoiding the lack of ffmpeg-static binary Windows ARM64.

9.3 Compliance Boundary

The installation script runs license compliance, preserving MIT license, third-party notices, Electron/Chromium runtime notices, GPL/LGPL/MPL text, and so on. Release notes also emphasizes that local assembled apps are only for local execution and should not be redistributed as release artifact.

10. Compare with relevant schemes

11. What can I say before sales

11.1 for Business Owner

"Many key processes can only be done by skilled employees, but there is no reusable automation. Skill Recorder lets employees do it again, and the system automatically organizes them into steps that can be understood by the Agent, which are then reviewed and solidified by experts."

11.2 for CIO / IT Leader

"It is not a simple UI playback, but a desktop operation into a more stable tool flow, and prioritizes the use of CLI, API, web_fetch and other native tools. This is closer to Agentic Automation, rather than traditional RPA."

11.3 for Security/Compliance

"The data during the recording phase is saved locally and will not be sent to GitHub cloud until it is Analyze. PoC must use test data to avoid entering passwords, Token, customer privacy or sensitive business information. If the enterprise does not allow cloud analysis, this plan needs to be evaluated separately."

11.4 for development/operations team

"It is especially suitable for cross-tool processes such as GitHub, CLI, web pages, forms, CRM and deployment records. Record release notes or issue triage once, and then generate Agent Skill to reduce repetitive knowledge work."

12. Frequently Asked Customer Questions

13. PoC Recommendation

13.1 PoC Target

It is recommended to define PoC objectives:

1. Verify that the intent and key steps of a cross-application task can be accurately restored.
2. Verify whether the generated Skill can be generalized to similar tasks.
3. Verify that the privacy/compliance process is acceptable.
4. Verify the workload of the business expert to review and revise the Skill.

13.2 Recommended PoC Scenarios

13.3 PoC Considerations

-All use test accounts.

-Use of falsified customer data.

-Real passwords and tokens are not recorded.

-Turn off message notification before recording.

-Manual check of Analyze bundle after recording.

-After Skill is generated, it is executed in the sandbox environment.

-Compare the time savings of "Handwriting Skill" with "Recording Skill.

13.4 acceptance index

14. Risks and Considerations

14.1 Privacy Disclosure Risk

This is the biggest risk. Screenshots, URLs, clipboard previews, and narration may contain sensitive information. Presentation directly on real customer data must be avoided before sales.

14.2 cloud analysis dependencies

Analyze rely on the GitHub Copilot CLI and GitHub cloud. It may not be acceptable for customers with complete intranet, strong financial supervision and confidential government affairs.

14.3 Early Project Maturity

The warehouse was created in 2026-07-29. The latest release is v0.3.1, which is still an early project. Suitable for PoC and research, should not directly promise production stability.

14.4 platform and permission dependency

macOS requires the Screen Recording permission; Windows and Ubuntu also have different system limitations. Enterprise endpoint security policies may prevent screen recording, Electron, local build, or script execution.

The quality of 14.5 generation is uncertain

Copilot describer and builders have evals, real business processes are more complex and may appear:

-Steps that are too fine or too coarse.

-Take irrelevant detour as a step.

-Error selecting browser automation instead of CLI/API.

-Incomplete understanding of business rules.

Therefore, a manual review must be retained.

15. My Pre-Sales Judgment

The Skill Recorder is a very "Agent Age" tool: it solves not the model capability itself, but the more difficult problem of "where does the skill come from" in the enterprise Agent landing. A large number of processes in the enterprise have no API documents and no standard SOP, but skilled employees will do it. The Skill Recorder provides a way to extract Agent Skill from a human demonstration.

I think it fits best in the following pre-sales story:

-Agent Capacity Building: Precipitate employees' repetitive knowledge work into a skill base.

-RPA upgrade: Agentic Automation from UI playback to tool priority.

-Copilot Eco Extension: Connect Microsoft Scout, Copilot Cowork, and Copilot Studio to build skills/automation.

-Process Standardization: Quickly generate the first draft of SOP/Skill and then review by experts.

But it is not suitable for direct as a "risk-free automation artifact" to sell. It involves screen recording and cloud analysis, and data security and authorization issues must be resolved first; generating results also requires review, testing, and desensitization. My suggestion is to do PoC with low-risk, cross-tool and repetitive processes first, such as GitHub issue triage, web page to form, release notes, CRM lead sorting. As long as it can be proved that "record once-> generate the first draft of available Skill-> Agent can be reused", it can easily become the methodology entrance for the landing of the enterprise Agent platform.

16. REFERENCE

-GitHub repository:microsoft/skill-recorder

-README:README.md

-Installation documentation:INSTALL.md

-Latest Release:Skill Recorder v0.3.1

-Evals documentation:evals/README.md

-Windows verification:WINDOWS-VALIDATION.md

-Third Party License:THIRD-PARTY-NOTICES.md

-Security Disclosure:SECURITY.md