Skip to content

Ignore quotes when detecting the row separator - #365

Open
gaoflow wants to merge 1 commit into
ruby:mainfrom
gaoflow:quote-aware-row-sep-detection
Open

Ignore quotes when detecting the row separator#365
gaoflow wants to merge 1 commit into
ruby:mainfrom
gaoflow:quote-aware-row-sep-detection

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 28, 2026

Copy link
Copy Markdown

:auto row-separator detection scans the sample for the first CR/LF without tracking quote state, so a CR inside a quoted field is mistaken for the row separator. CSV therefore cannot parse its own generate output whenever a field contains \r:

CSV.parse(CSV.generate {|c| c << ["a\rb", "x"]})
# => [["a\rb", "x\n"]]   # wrong: the \n was swallowed into the last field
CSV.parse(CSV.generate {|c| c << ["a\r\nb", "x"]})
# => CSV::MalformedCSVError: Unquoted fields do not allow new line
CSV.parse(CSV.generate {|c| c << ["\r"]})
# => CSV::MalformedCSVError: Any value after quoted field isn't allowed

All three parse correctly with an explicit row_sep: "\n", which isolates the fault to :auto detection. detect_row_separator now skips CR/LF that fall inside a quoted field. An unquoted CR is still honored as a separator, so old Mac line endings keep working, and explicit row_sep paths are untouched.

Added round-trip and detection tests in test/csv/parse/test_row_separator.rb; the full suite stays green (527 tests, 4050 assertions). This is distinct from #107, which is about an explicit non-standard row_sep.

`:auto` row separator detection scanned the sample for the first CR/LF
without tracking quote state, so a CR inside a quoted field was mistaken
for the row separator. As a result CSV could not parse its own output:
`CSV.parse(CSV.generate)` corrupted or crashed on any field containing
a carriage return. For example `["a\rb", "x"]` round-tripped to
`["a\rb", "x\n"]`, and fields with `\r\n` or a lone `\r` raised
CSV::MalformedCSVError.

Make detection skip CR/LF that fall inside a quoted field. Unquoted CR
is still honored as a separator, so old Mac line endings keep working.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant