Datadog announced new capabilities that help customers monitor and troubleshoot issues in their generative AI-based applications.

Datadog announced a broad set of generative AI observability capabilities to help teams deploy LLM-based applications to production with confidence and help them troubleshoot health, cost and accuracy in real time.

These capabilities include integrations for the end-to-end AI stack:

- AI Infrastructure and compute: NVIDIA, CoreWeave, AWS, Azure and Google Cloud

- Embeddings and data management: Weaviate, Pinecone and Airbyte

- Model serving and deployment: Torchserve, VertexAI and Amazon Sagemaker

- Model layer: OpenAI and Azure OpenAI

- Orchestration framework: LangChain

Additionally, Datadog released in beta a complete solution for LLM observability, which brings together data from applications, models and various integrations to help engineers quickly detect and resolve real-world application problems like model cost spikes, performance degradations, drift, hallucinations and more to ensure positive end user experiences.

LLM observability includes:

- Model catalog: Monitor and alert on model usage, costs and API performance.

- Model performance: Identify model performance issues based on different data characteristics provided out of the box, such as prompt and response lengths, API latencies and token counts.

- Model drift: Categorization of prompts and responses into clusters enabling performance tracking and drift detection over time.

"It's essential for teams to measure the time and resources they are investing in their AI models, especially as tech stacks continue to modernize," said Yrieix Garnier, VP of Product at Datadog. "These latest LLM monitoring capabilities and integrations for the AI stack will help organizations monitor and improve their LLM-based applications and capabilities while also making them more cost efficient."