Summary
AgentOps currently reads evaluation datasets from the local file system. Many enterprise deployments store evaluation assets in Azure Storage and would benefit from a native mechanism to load datasets directly from storage without requiring a local copy.
Problem
In enterprise AI solutions, evaluation assets are often centrally managed and stored in Azure Storage, including:
Excel datasets
CSV files
JSON evaluation sets
Markdown reference content
Golden datasets
Test case repositories
Today, these assets must be downloaded locally before they can be used by AgentOps evaluations.
Current flow:
Azure Storage
↓
Download files locally
↓
AgentOps reads local files
↓
Run evaluations
This introduces:
Dataset duplication
Additional preprocessing steps
CI/CD complexity
Synchronization challenges between source data and evaluation data
Proposed Enhancement:
Allow AgentOps evaluation datasets to be loaded directly from Azure Storage.
Possible sources:
Azure Blob Storage
Azure Data Lake Storage Gen2
Potential authentication methods:
Managed Identity
Azure CLI authentication
Service Principal
SAS Token (optional)
Business Value
Benefits include:
Single source of truth for evaluation assets
Simplified CI/CD pipelines
Better enterprise governance
Reduced data duplication
Easier management of large evaluation datasets
Improved alignment with Azure-native architectures
This would allow AgentOps to support both local and cloud-native evaluation workflows while maintaining backward compatibility.
Summary
AgentOps currently reads evaluation datasets from the local file system. Many enterprise deployments store evaluation assets in Azure Storage and would benefit from a native mechanism to load datasets directly from storage without requiring a local copy.
Problem
In enterprise AI solutions, evaluation assets are often centrally managed and stored in Azure Storage, including:
Excel datasets
CSV files
JSON evaluation sets
Markdown reference content
Golden datasets
Test case repositories
Today, these assets must be downloaded locally before they can be used by AgentOps evaluations.
Current flow:
Azure Storage
↓
Download files locally
↓
AgentOps reads local files
↓
Run evaluations
This introduces:
Dataset duplication
Additional preprocessing steps
CI/CD complexity
Synchronization challenges between source data and evaluation data
Proposed Enhancement:
Allow AgentOps evaluation datasets to be loaded directly from Azure Storage.
Possible sources:
Azure Blob Storage
Azure Data Lake Storage Gen2
Potential authentication methods:
Managed Identity
Azure CLI authentication
Service Principal
SAS Token (optional)
Business Value
Benefits include:
Single source of truth for evaluation assets
Simplified CI/CD pipelines
Better enterprise governance
Reduced data duplication
Easier management of large evaluation datasets
Improved alignment with Azure-native architectures
This would allow AgentOps to support both local and cloud-native evaluation workflows while maintaining backward compatibility.