Observability: Expand tracing for APIs #802

Describe the current behavior
A clear description of how it currently works and what the limitations are.
Currently, observability and tracing are implemented primarily for LLM call/experiment flows. Other API endpoints lack consistent tracing, metrics collection, and visibility into request performance or failures.

Describe the enhancement you'd like
A clear and concise description of the improvement you want to see.
Extend observability (tracing, metrics, and logging) to all major endpoints, not just LLM-related flows.

Why is this enhancement needed?
Explain the benefits (e.g., performance, usability, maintainability, scalability).
This will improve system-wide visibility, help identify bottlenecks outside LLM calls, and enable better debugging and performance optimization. It also ensures consistency in monitoring and supports scalability as more endpoints are added.

Additional context
Add any other context, metrics, screenshots, or examples about the enhancement here.