Skip to content

Make CSV importer configurable via ConfigArgParse - #166

Open
MrlixiangWE wants to merge 2 commits into
apache:masterfrom
MrlixiangWE:fix/119-csv-configargparse
Open

Make CSV importer configurable via ConfigArgParse#166
MrlixiangWE wants to merge 2 commits into
apache:masterfrom
MrlixiangWE:fix/119-csv-configargparse

Conversation

@MrlixiangWE

Copy link
Copy Markdown

Problem

CSV is the only importer whose delimiter and quote character cannot be set through ConfigArgParse. They can currently only be configured per test in YAML.

Before / after

Before: CSV delimiter and quote character can only be configured through per-test csv_options in YAML.

After: they can also be set through --csv-delimiter and --csv-quote-char, the matching environment variables, or a csv section in the config file. Explicit global values override per-test options; otherwise existing behavior is unchanged.

Verification

uv run pytest -q --ignore=tests/graphite_e2e_test.py --ignore=tests/postgres_e2e_test.py

uv run tox -e lint

uv build --no-create-gitignore

Closes #119

@Gerrrr Gerrrr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Please revert the breaking yaml config change, otherwise LGTM.

Comment thread docs/CSV.md Outdated
csv_options:
delimiter: ','
quotechar: "'"
quote_char: "'"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep the existing name and not break existing configs unnecessarily?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I restored quotechar for the existing per-test YAML configuration and added regression coverage for it. The new top-level ConfigArgParse setting remains csv.quote_char, matching --csv-quote-char and CSV_QUOTE_CHAR.

@MrlixiangWE
MrlixiangWE force-pushed the fix/119-csv-configargparse branch from bc0ee05 to 70fdcad Compare August 9, 2026 04:01
Comment thread docs/CSV.md Outdated
```yaml
csv:
delimiter: ';'
quote_char: "'"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another one

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, I missed the new top-level YAML example. I've changed it to quotechar too and added regression coverage for the top-level config path.

Comment thread otava/config.py
BigQueryConfig.NAME,
]

CONFIG_KEY_ALIASES = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am really on the fence with CONFIG_KEY_ALIASES. @MrlixiangWE can you please elaborate your thoughts on why do we want to introduce them?

My take is, if we really dislike quotechar - now is the time to break backwards compatibility and rename it to quote-char everywhere (although, I'd like to know the justification for it). OTOH if we can live with quotechar, why don't we just add csv-quotechar and call it a day?

@MrlixiangWE MrlixiangWE Aug 12, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alias table goes away either way — it was bridging a divergence this PR itself created. But let me give the justification you asked for, which I should have led with in the first round instead of just renaming things.

Across the config surface, multi-word names are separated: time_column, csv_options, update_statement, test_groups, base_branch, bigquery.project_id in YAML; --bigquery-project-id, --since-commit, --config-file on the CLI; BIGQUERY_PROJECT_ID and friends in the environment. quotechar is the one concatenated exception (not counting hostname and username, which are ordinary words). It comes from Python's csv API, but someone writing an otava config shouldn't need to know that — it's just one more special spelling to memorize. That's what the original rename was about — and it matched the names the code itself already used (CsvOptions.quote_char, and the per-test loader key). When you asked to keep the existing name, I took that to mean the per-test files specifically, restored quotechar there, and named the new flag --csv-quote-char to match the other options — then papered over the gap with the alias. In hindsight I should have raised the naming question openly instead of doing it in halves.

One thing worth knowing before we decide: on current master the per-test loader reads only quote_char:

otava/otava/test_config.py

Lines 257 to 259 in cd5bc1e

if test_info.get("csv_options"):
csv_options.delimiter = test_info["csv_options"].get("delimiter", ",")
csv_options.quote_char = test_info["csv_options"].get("quote_char", '"')

so the quotechar in docs/CSV.md and examples/csv/config/otava.yaml is silently ignored today — the example never notices because its CSV has no quoted fields. The docs and the code already disagree, and a full rename to quote_char wouldn't break any config that actually works.

So my vote is the deliberate rename: quote_char in YAML, --csv-quote-char, CSV_QUOTE_CHAR, docs and examples updated to match, no aliases and no fallback. If you'd rather keep quotechar, I'll do that instead — --csv-quotechar and done. Either way the PR ends with exactly one public spelling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make CSV importer configurable via ConfigArgParse

2 participants