Annotation Agents

Precision in Data Labeling: The Role of Agents

Annotation Agents in GPTBoost offer you the power to automate labelling for user prompts and LLM completions based on predefined criteria. These intelligent agents act as your vigilant assistants, flagging requests for privacy concerns, relevance, potential harm, and more.

The benefits of automatic annotation are plenty, including gaining insights, refining models, saving time even with smaller datasets, and more. Plus, as a project scales, the volume of requests can skyrocket, making manual labelling impractical.

Explore the specifics and the capabilities of Annotation Agents to efficiently manage your language model interactions.

GPTBoost Agents

Provided with the right set of instructions these Agents are irreplaceable helpers in automatic labelling of the prompt of your users or the model's completion. Every GPTBoost Annotation agent has a Profile, Instructions and Settings.

Profile

Much like our human profiles, these cute helpers have a name, avatar, and job description, and are usually in a good mood.

Profile - Smile 😄

Annotation Instructions

The annotation instructions are essential. They should provide the right info so the agent can perform at its best. Choose the annotation name and provide a clear and concise definition of the criteria.

• Annotation Name - The name of the field that will contain the label. Choose a simple and concrete name, such as is_irrelevan, is_harmful, or has_sensitive_data. You see, even without any context you can assume what this field is about.

• Definition - What the Agent will evaluate - Is it an irrelevant question from the user, a potential sales opportunity, or perhaps an uncertain response from the model? Or you may have a specific use case and for example, want to flag particular words or phrases. Simply provide a clear definition, and the agent will handle the rest.

• Strictness Level - Another cool option is selecting the level of strictness for the agent. In other words - how harsh text should be evaluated.

The Annotation Instructions

Settings

The Agent's settings are for managing the setup - choose will the agent evaluate each request automatically or you'll decide for which requests to get it working and add such info in the Headers? Or maybe you want to asses only completion that has received some sort of feedback? The choice is yours.

• Auto Triggering - By default enabled. Once you create an Agent it starts annotating any new requests made. In case this mode is disabled, the agent can be also triggered with an optional header in the request to the LLM. To enable Agents manually use the "GPTBoost-Run-Agents" key in the headers. The value should be a string with Agents' names, divided by a comma.

Important! - When you manually enable Agents, pass the name all lowercase. If

the is a space in the Agent name, replace it with a hyphen E.g:

# If the Agents name is 'Sales Radar' use:
headers = {
    "GPTBoost-Run-Agents": "sales-radar"
    }

• Trigger Rule - Agents start working from the moment of their creation and by default are checking every single request. For the times that the bot should evaluate requests only with feedback, use the Trigger Rule.

E.g. If you want to detect if there's frustration in requests with negative feedback,

select the Sentiment Analysis from the ready-to-use agents. Next from the

newly created agent's Settings-> Trigger Rule choose 'With Negative Feedback'

• Danger Zone - You can 'retire' an agent here.

Whether you prefer the simplicity of ready-to-use agents or crave a more tailored approach, GPTBoost gives you the flexibility to choose.

Select Ready to Use Agent

GPTBoost provides you with an army of helpers that can help in various scenarios. You can enable them with a few clicks from the dashboard.

1. Go to your GPTBoost Dashboard -> Annotation Agents -> Ready to Use Agents.

2. Decide which agent best matches your case and click the Use This Agent button.

3. Confirm that you want to use this agent.

4. Provide Annotation Instructions to the Agent. Make sure to provide a clear explanation of what the agent should check out. Let's say you have a music application that has a feature to provide info about artists and music in general, try the following annotation definition to check the relevance of the text: "Does the input contain any topic not related to music, music videos, songs, singers, musical instruments"

5. Set the Mood 😜

Enabling ready to use agent to mark if a prompt is related to music or irrelevant

6. Once the agent is created, you'll see annotation per each subsequent request. The example below shows how the agent's work looks in the JSON object.

"annotations": {
        "is_irrelevant": true
    },
    "annotations_agents": [
        "guardian"
    ],    

Build a Custom Agent

Similarly to the Ready-to-Use Agents, you can create your own custom-tailored helper to label requests based on the application's specific needs.

• Go to your GPTBoost Dashboard -> Annotation Agents -> My Agents.

• Click on Build Your Agent.

• Pick the most suitable name for your agent's Profile and provide a short job description of how it will benefit your work. Press next to continue providing more precise instructions.

• Provide Annotation Instructions to the Agent. Think of an annotation name that best matches your agent's task and provide a clear Description of what the agent should look out for. Let's say you want to asses possible sale opportunities, try something like the following annotation definition: "Does the input contain anything similar to buy, purchase, cost?. You may further enhance the description with info tailored to your specific case.

• If needed, change the Settings to match your needs.

GPTBoost Army - Binary Squad

Each ready-to-use agent serves a distinct purpose, offering a streamlined approach to enhance user safety, filter irrelevant content, and gain valuable insights into user sentiment.

Some of the Ready-to-Use binary GPTBoost Agents

• Privacy Officer - The Privacy Guardian agent automatically scans user prompts for sensitive information, such as names, addresses, phone numbers, etc. ensuring compliance with data protection regulations and safeguarding user privacy. Suppose you operate a platform where users seek mental health advice. The Privacy Guardian agent flags any messages containing personally identifiable mental health information, helping you handle these cases with care and sensitivity.

• Relevancy Detector (Irrelevant Topics) - The Relevancy Detector agent filters out requests with irrelevant topics, allowing you to prioritize meaningful interactions by eliminating prompts unrelated to your service's scope or objectives. For example, in a customer support chatbot service, the agent identifies off-topic requests, ensuring that the language model focuses on resolving customer queries related to your product or service.

• Guardian (Harm Identification) - The Guardian agent identifies and addresses potential harm in user prompts or completions, helping ensure the safety and well-being of users by preventing the generation of harmful or inappropriate content. Imagine how in a community forum, the Ethicist agent flags messages that may involve harassment or harmful language, allowing you to moderate content and maintain a positive and respectful online environment.

• Sentiment Analysis - The Sentiment Analysis agent detects chats where users express signs of frustration or negative emotions, providing insights into user sentiment to improve overall user experience. Thus, in a customer service application, the Sentiment Analysis agent can flag interactions where users exhibit frustration, allowing you to prioritize and address these concerns promptly, enhancing customer satisfaction.

• No Answers - The No Answers agent labels user prompts where the language model provides no response or indicates uncertainty, enabling you to review and address instances where the model may need further training or refinement. For instance, in an educational assistance application, the No Answers agent flags prompt where the model struggles to provide relevant information, helping you enhance the model's training data for more accurate responses.

Currently, all GPTBoost agents are binary and label requests based on whether the instruction provided is true or false. Still, we understand the need for a more sophisticated approach and category labelling, so the next squad of agents is on its way. Don't miss out!