Traffic sent through Blueteam AI is recorded and ingested into an OpenSearch service. Under the dashboards tab, you can inspect the recorded traffic and create custom charts and dashboards.
In addition, Blueteam AI enriches the data ingested with some useful metrics. Currently, OpenAI doesn't provide usage statistics such as prompt and completion tokens for streaming completions. Blueteam AI overcomes this limitation by manually tokenizing and computing usage asynchronously after requests are sent. In addition, Blueteam AI' data enrichment also includes:
- Time to first chunk, which is often a better measure of user experience in interactive chat applications implementing streaming completions
- Token velocity, which captures the consistency of chunk output during streaming completion and can detect irregular pauses and hiccups
- Custom headers, which allow you to attach arbitrary header metadata and filter or segment on them for analysis and visualization