[BugFix] Shadow stale mapped leaves when an override replaces an object parent (#5718)

[Ystk-hsn]

Description

Problem

spath input=body output=log on an index where log is a mapped object field silently answers log.<key> references from the stale mapped leaves instead of the extracted JSON (issue #5718). Within a single row, each leaf independently reads whichever source happens to be mapped — and where log.level = 'ERROR' on the extracted value silently matches nothing.

Root cause

Three interacting behaviors:

1. OpenSearch exposes an object field to the row schema as a struct-parent column (log) plus flattened leaf columns (log.level, log.src) side by side.
2. projectPlusOverriding replaces only exact-name matches, so spath's rewrite (eval log = json_extract_all(body)) replaces the parent column but leaves the stale leaf columns in the schema.
3. QualifiedNameResolver prefers an exact-name column over map access, so log.level resolves to the stale leaf while log.msg (unmapped) falls through to the fresh map.

Fix

Add the mirror of the existing dropStructParentsFor step: when an override replaces a column that was an object/nested parent (MAP/ARRAY-typed), also drop its flattened leaf columns (dropStructChildrenFor). The replacement value then shadows the entire <name>.* subtree — issue #5718's preferred option 1.

The children-drop is type-gated on the replaced column being container-typed. A scalar column that merely shares a dotted prefix with user-created literal columns (eval x.y = 1 | eval x = 2) keeps its children, preserving the SPL1 literal-field semantics that PR #5351 restored. Same gating style as the existing parent-drop: it only fires when the override actually replaced a container column.

This also fixes a companion defect found while reproducing: with stale leaves present, a subsequent eval log.level = 'patched' fired the override path and dropStructParentsFor removed the freshly extracted map (Field [log] not found). With the stale leaves gone, the assignment creates a literal column and the parent survives, consistent with the #5185 semantics.

Behavior changes (only when an assignment collides with a mapped object/nested parent)

Query pattern	Before	After
spath output=<object parent> → leaf reference	stale mapped value, silent	extracted value
same → where on leaf	silently matches stale value	filters on extracted value
same → unrelated doc triggers dynamic mapping	extraction silently retires to null	unaffected
spath ... path=... / eval <parent> = <scalar> → leaf reference	stale value, silent	clear resolution error (mirrors the flat-keyword collision behavior)
spath collision → eval a dotted leaf	Field not found error	works; parent map survives
user literal dotted columns under a scalar prefix	preserved	preserved (type gate)

Default (fields *) output shape is unchanged — the stale leaves were already hidden by tryToRemoveNestedFields; they were only reachable by explicit reference.

Known accepted corner: a literal dotted column created under a mapped object parent is dropped together with the mapped leaves when the parent itself is overridden (provenance of individual columns is not tracked).

Testing

• New CalcitePPLSpathCollisionIT (10 tests) written against the expected semantics before the fix: 7 failed on the unfixed build reproducing every symptom (A/B/C and path-mode) plus the companion defect; 3 guard tests pinned the already-correct behaviors. All 10 pass with the fix.
• Reproduced on main (issue was reported against 3.8; confirms main is affected).
• Regression suites all green: Spath/Eval/Bin/Rex/Parse/Trendline/Flatten/Patterns/MapPath ITs (278+ tests), :ppl:test spath units, :core:test, :integ-test:yamlRestTest (includes the issues/5185.yml regression).
• Documented the collision behavior in docs/user/ppl/cmd/spath.md (prose only; no new doctest blocks).

Related Issues

Resolves #5718

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed. (N/A — not a new command)
• API changes companion pull request created. (N/A)
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created. (N/A — in-repo user manual updated)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit a29ce5b)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to a29ce5b
Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Add null-safety for field lookup Add null-safety check before calling getType() on the field. If getField() returns null for a non-existent field name, calling getType() will throw a NullPointerException. Filter out null fields before accessing their type. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [1432-1435] Set<String> overriddenContainerParents = overriddenNames.stream() - .filter(name -> isContainerType(originalRowType.getField(name, true, false).getType())) + .map(name -> originalRowType.getField(name, true, false)) + .filter(field -> field != null && isContainerType(field.getType())) + .map(field -> field.getName()) .collect(Collectors.toSet()); Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a potential NullPointerException if getField() returns null. The overriddenNames are derived from filtering against originalFieldNameSet, which should guarantee existence, but defensive programming is warranted. The improved code properly handles the null case and maintains the field names in the set.	Medium
General	Validate parent name before processing Validate that parentName is not null or empty before constructing the prefix. An empty or null parent name would create an invalid prefix that could match unintended fields or cause unexpected behavior in the filtering logic. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [1491-1501] private void dropStructChildrenFor(String parentName, CalcitePlanContext context) { + if (parentName == null || parentName.isEmpty()) { + return; + } String prefix = parentName + "."; List<String> fieldNames = context.relBuilder.peek().getRowType().getFieldNames(); List<RexNode> childrenToDrop = fieldNames.stream() .filter(f -> f.startsWith(prefix)) .map(f -> (RexNode) context.relBuilder.field(f)) .toList(); if (!childrenToDrop.isEmpty()) { context.relBuilder.projectExcept(childrenToDrop); } } Suggestion importance[1-10]: 7 __ Why: Valid defensive programming suggestion. While parentName comes from overriddenNames which should be non-null/non-empty, adding validation prevents potential issues if the method is called from other contexts in the future. The early return is a clean way to handle invalid input.	Medium

---

Previous suggestions

Suggestions up to commit 20a0604

Category	Suggestion	Impact
Possible issue	Add null-safety for field lookup Add null-safety check before calling getType() on the field. If getField() returns null for a non-existent field name, calling getType() will throw a NullPointerException. Filter out null fields before accessing their type. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [1432-1435] Set<String> overriddenContainerParents = overriddenNames.stream() - .filter(name -> isContainerType(originalRowType.getField(name, true, false).getType())) + .map(name -> originalRowType.getField(name, true, false)) + .filter(field -> field != null && isContainerType(field.getType())) + .map(field -> field.getName()) .collect(Collectors.toSet()); Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies a potential NullPointerException when getField() returns null. The overriddenNames are derived from filtering against originalFieldNameSet, so fields should exist, but defensive programming is valuable here. The improved code properly handles the null case.	Medium
General	Validate field resolution before casting Verify that context.relBuilder.field(f) doesn't return null before casting to RexNode. If a field name exists in the row type but cannot be resolved by the builder, this could cause issues. Add validation or filter out unresolvable fields. core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java [1491-1501] private void dropStructChildrenFor(String parentName, CalcitePlanContext context) { String prefix = parentName + "."; List<String> fieldNames = context.relBuilder.peek().getRowType().getFieldNames(); List<RexNode> childrenToDrop = fieldNames.stream() .filter(f -> f.startsWith(prefix)) - .map(f -> (RexNode) context.relBuilder.field(f)) + .map(f -> context.relBuilder.field(f)) + .filter(node -> node != null) + .map(node -> (RexNode) node) .toList(); if (!childrenToDrop.isEmpty()) { context.relBuilder.projectExcept(childrenToDrop); } } Suggestion importance[1-10]: 7 __ Why: This suggestion asks to verify that context.relBuilder.field(f) doesn't return null. While the field names come from the row type, adding null-safety is a reasonable defensive measure. However, since this is primarily a verification request rather than fixing a confirmed bug, the score is moderate.	Medium

[Ystk-hsn]

The 6 failing unit jobs all hit the licenseHeaders / IBM424_rtl flake on HighlightFunctionIT.java, which was fixed on main by #5733 after this PR's CI ran. A re-run should pick up the fix — could someone trigger one? (All integration, doc, BWC, and security jobs are green.)

[dai-chen]

The 6 failing unit jobs all hit the licenseHeaders / IBM424_rtl flake on HighlightFunctionIT.java, which was fixed on main by #5733 after this PR's CI ran. A re-run should pick up the fix — could someone trigger one? (All integration, doc, BWC, and security jobs are green.)

Hi @Ystk-hsn , I think you need to rebase to include the fix from main if you haven't.

[github-actions]

Persistent review updated to latest commit a29ce5b

[Ystk-hsn]

Thanks @dai-chen — rebased onto latest main. All tests pass locally on the rebased branch. A review would also be much appreciated.

[dai-chen]

np: if you want, you can try out yaml test which is more readable I think. Ref: https://github.com/opensearch-project/sql/blob/main/integ-test/src/yamlRestTest/resources/rest-api-spec/test/issues/5702.yml

[dai-chen]

Could you double check whether this should happen earlier — right after overriddenContainerParents is computed — rather than here? Step 5's rename runs in between, so by this point the row already carries the new newNames?

[dai-chen]

The gate is a pure type test, so any MAP/ARRAY column counts as a mapped parent, including ones a function just built, which can't have stale flattened leaves. Could you check are both of these returning a row on main and 400 with the changes?

 ... | eval arr = array(1,2) | eval `arr.x` = 5 | eval arr = array(3,4) | fields arr, `arr.x`
       main → [[3,4], 5]     → Field [arr.x] not found?

 ... | spath input=body output=data | eval `data.custom` = 'kept'
     | spath input=body output=data | fields data, `data.custom`
       main → (map, "kept")  → Field [data.custom] not found?