AutoGPT (part 2): monitoring and securing with Blueteam AI

This is the second post of a two-part series on AutoGPT. Our previous post explained how AutoGPT works, how its improved capabilities also increase its risks, and how monitoring its safety and costs is critical for productive operations of autonomous AGIs like AutoGPT. This post shows how Blueteam AI can be used to gain visibility and insight into the costs, latencies, and message contents of an AutoGPT system. We will also see how we can configure redaction policies to prevent data leaks and protect privacy when using AutoGPT for applications involving sensitive data.

Blueteam AI: the best monitoring and privacy solution for LLM applications like AutoGPT

Given the importance of ensuring safety and managing costs, it's no surprise that many are asking for a robust monitoring solution. Fortunately, at Blueteam AI we've been building with LLMs before transformers were invented and our monitoring solutions can easily be integrated with AutoGPT.

Add monitoring with Blueteam AI to AutoGPT in two easy steps:

1. On Blueteam AI, create a new endpoint, configure an OpenAI chat completions upstream, and allocate a service token for it.
2. Configure AutoGPT to use your newly created endpoint by adding the following snippet to the end of autogpt/__init__.py:

   import openai
   
   openai.api_base = "https://fmops.ai/api/v1/endpoints/<your_endpoint_name>/openai/v1"
   openai.api_key = "<your_endpoint_service_token>"

   replacing the endpoint name and service token from step 1.

After making these changes, you can run AutoGPT as usual and your traffic will now be monitored by the Blueteam AI platform, bringing you a number of easy wins.

Real-time performance and cost monitoring

When you created an endpoint, Blueteam AI automatically created an application performance monitoring (APM) dashboard to visualize and analyze that endpoint's traffic. This provides you with detailed breakdowns of your AutoGPT's latencies and costs at a granularity unavailable from OpenAI's usage reports.

For example, here we see that AutoGPT's requests to GPT-4 can be pretty slow with response times ranging from 12-30 seconds:

In addition, we can also see that operating AutoGPT requires around 3000 tokens every 30 seconds. At GPT-4's current pricing of $0.03 / 1k tokens, this means that it costed about $0.20 / minute to operate AutoGPT during our tests:

Finally, we can drill down into each individual step taken by AutoGPT to look at what data was sent, what was returned, and the corresponding latencies and costs.

Privacy and security measures

When using AutoGPT to work with sensitive data or private intellectual property, protection of data security is paramount in order to ensure compliance and maintain a competitive advantage.

For example, suppose we were using AutoGPT as an email assistant. We just discovered that Feynman's SSN was leaked on the internet and need to draft him an email. Without any privacy measures in place, the following request is sent to OpenAI and results in a leak of private information to a third party data processor:

Fortunately, the Blueteam AI platform makes it easy to prevent such problems. Without writing any code, we can configure a data redaction policy on our Blueteam AI endpoint to redact these entities:

Afterwards, any private information detected in the traffic arriving at an endpoint is automaticaly redacted. Retrying the same AutoGPT as before now results in a properly redacted request which is free of data leaks:

Conclusions

Monitoring and data policy enforcement can be powerful tools to help to ensure safe, secure, and compliant operations of LLM applications. As we've previously explored, the increased capabilities of LLM applications like AutoGPT also results in a larger risk factor that must be addressed before these applications can generate real enterprise value. In this post, we showed how the Blueteam AI platform can be used to both monitor LLM application cost/latency/traffic as well as enforce data protection policies so that those responsible have peace of mind that their applications are operating safely, securely, and cost efficiently.

Time to add monitoring and security to your LLM applications? Book a demo at blueteam.ai to learn more.