|
<< Click to Display Table of Contents >> Navigation: 3. Script Language > AI - Artificial Intelligence Commands > AIU. - OpenAI API > AIU. - Artificial Intelligence Utility |
Operator Mode |
Top |
MiniRobotLanguage (MRL)
Operator Mode
Enabling AI to use tools and control the computer.
Intention
Operator Mode transforms the AI from a simple chatbot into an active agent capable of performing tasks on your computer. By providing the AI with a set of "tools" (like taking a screenshot, reading a file, or clicking the mouse), your script can give it complex goals and let the AI determine the sequence of steps needed to achieve them. This is the foundation of advanced Robotic Process Automation (RPA) and building autonomous agents.
The process is a loop: you send the user's request and a screenshot to the AI, the AI responds with a list of tool actions to perform, your script executes those actions, you add the results back to the conversation history, and repeat until the task is complete.
Illustration
┌──────────────────────────┐
│ User Goal: "Read email" │
└─────────────┬────────────┘
▼
┌──────────────────────────┐
│ AI Plan (Tool Calls): │
│ 1. mouse_click(100, 200) │
│ 2. type_text("password") │
│ 3. screenshot() │
└─────────────┬────────────┘
▼
┌──────────────────────────┐
│ SPR Script Executes Calls│
└─────────────┬────────────┘
▼
┌──────────────────────────┐
│ Results Sent Back to AI │
└──────────────────────────┘
The core loop of an Operator Mode session.
Operator Mode Commands
Session Management
• AIU.OperatorBeginSession - Initializes the session and sets up the required arrays.
• AIU.OperatorAddTool - Adds a custom tool definition to the AI's available actions.
• AIU.OperatorSetScreen - Configures the screen resolution for mouse click mapping.
Execution Loop
• AIU.Screenshot - Captures a screen image to provide visual context to the AI.
• AIU.OperatorRunStep - The main command that sends the history/tools to the AI and gets back a list of actions.
• AIU.OperatorAddResult - Adds the outcome of an executed tool back into the conversation history.
Example Workflow
'***********************************
' Operator Mode - Sample Workflow: Open Notepad and type
'***********************************
AIU.SetModel|gpt-4o
AIU.OperatorBeginSession|1|2|3
AIU.OperatorSetScreen
' Add the user's initial request to the history array (ARR 3)
VAR.$$GOAL=user*Please open Notepad, type "Hello, Operator!" and then tell me you are finished.
ARS.Add|3|$$GOAL
:Loop
AIU.Screenshot|desktop|$$IMG
ARS.Add|3|$$IMG
AIU.OperatorRunStep|4
' $$TOS now contains the number of actions. Actions are in ARR 4.
JIZ.$$TOS|Loop
' (Here you would add a loop to execute the actions in ARR 4)
' (And then call AIU.OperatorAddResult for each action's outcome)
' For this example, we just show the first action and stop
ARS.Get|4|1|$$ACT
MBX.AI wants to perform this action:|$$ACT
END.