Client-side query timing is a lot more slower than dashboard measurement

[ishank-dev]

Description

When I run a simple SELECT via the Python SDK and measure wall‑clock from .execute() through .fetchall(), I consistently see ~17 s latency for a query that the Firebolt UI reports as ~0.5 s server time. This large discrepancy makes it hard to understand where time is spent.

Steps to reproduce

Create benchmark_query.py with:

from time import perf_counter
from firebolt.client.auth import ClientCredentials
from firebolt.db import connect
import os; from dotenv import load_dotenv
load_dotenv()

creds = ClientCredentials(
    client_id=os.getenv("FIREBOLT_CLIENT_ID"),
    client_secret=os.getenv("FIREBOLT_CLIENT_SECRET")
)
conn = connect(
    engine_name=os.getenv("FIREBOLT_ENGINE_NAME"),
    database=os.getenv("FIREBOLT_DATABASE"),
    account_name=os.getenv("FIREBOLT_ACCOUNT"),
    auth=creds
)
cur = conn.cursor()

query = """
SELECT … FROM accidentdata
WHERE to_date(start_time) >= '2022-12-01'
  AND to_date(start_time) <= '2023-12-31'
"""
t0 = perf_counter()
cur.execute(query)
rows = cur.fetchall()
elapsed = perf_counter() - t0
print(f"Client RTT: {elapsed:.3f}s")

Run:

python benchmark_query.py

Compare the printed Client RTT to the dashboard’s “Query Duration” for that same statement.

Observed behavior

Python SDK: ~17.7 s

Firebolt Dashboard: ~0.5 s

Expected behavior
The SDK’s .execute() + .fetchall() should more closely align with the dashboard’s server‑side execution time

Python version: 3.12

OS: macOS 15.3.2

[stepansergeevitch]

Hey @ishank-dev
Thanks for reporting this issue. Did you try to measure the time taken separately by .execute() and .fetchall()? That could be really helpful here to understand the cause

[ishank-dev]

Yes I checked them separately.

The bottleneck seems to be present in the execute() method.
These results are consistent on all consecutive runs after engine has started successfully.

Time to create credentials: 0.027s
Time to connect: 0.847s
Time to create cursor: 0.000s
Query execution time: 13.899s
Result fetch time: 1.269s
Total elapsed time (including setup): 16.042s

Here is a script to reproduce the results:

from time import perf_counter
from firebolt.client.auth import ClientCredentials
from firebolt.db import connect
import os
from dotenv import load_dotenv

load_dotenv()

t_total_start = perf_counter()

t0 = perf_counter()
creds = ClientCredentials(
    client_id=os.getenv("FIREBOLT_CLIENT_ID"),
    client_secret=os.getenv("FIREBOLT_CLIENT_SECRET")
)
t_creds = perf_counter() - t0

t0 = perf_counter()
conn = connect(
    engine_name=os.getenv("FIREBOLT_ENGINE_NAME"),
    database=os.getenv("FIREBOLT_DATABASE"),
    account_name=os.getenv("FIREBOLT_ACCOUNT"),
    auth=creds
)
t_conn = perf_counter() - t0

# Cursor creation (negligible, but for completeness)
t0 = perf_counter()
cur = conn.cursor()
t_cursor = perf_counter() - t0

query = """
SELECT
    id,
    severity,
    start_time,
    description,
    weather_condition,
    distance_mi
FROM accidentdata
WHERE to_date(start_time) >= '2022-12-01'
  AND to_date(start_time) <= '2023-12-31'
"""
t0 = perf_counter()
cur.execute(query)
t_execute = perf_counter() - t0

t0 = perf_counter()
rows = cur.fetchall()
t_fetch = perf_counter() - t0

t_total = perf_counter() - t_total_start

print(f"Time to create credentials: {t_creds:.3f}s")
print(f"Time to connect: {t_conn:.3f}s")
print(f"Time to create cursor: {t_cursor:.3f}s")
print(f"Query execution time: {t_execute:.3f}s")
print(f"Result fetch time: {t_fetch:.3f}s")
print(f"Total elapsed time (including setup): {t_total:.3f}s")

[stepansergeevitch]

Hey @ishank-dev
Really sorry for such a delay in answering. As I see from the results you've shared, the query execution itself still takes 13.9 seconds (compared to 0.5 seconds for the Dashboard)
Do I understand correctly that by Dashboard you mean the Firebolt UI? At go.firebolt.io?
If so, I would ask you to check the following points

1. Are you running the query on the same engine in both Python and UI? Some engines might be much faster that the others, so that can be the cause
2. Does your result include 10000+ rows? In such case, the reason can be that the UI automatically trims the result to 10k rows, so that can be causing the discrepancy too.

Please check and let me know if the issue still persists for you