Repository navigation
Conversation
When the collector's DNS is unreachable, teardown of the curl transport could block host process exit for the system resolver timeout (tens of seconds). curl_easy_perform() does not run the progress callback while a name is resolving, so Abort() waited for the 5s connect timeout, and curl_easy_cleanup()/curl_multi_cleanup() then joined the resolver thread still blocked in getaddrinfo(). Set CURLOPT_QUICK_EXIT on every handle, and run modern-libcurl transfers on a private multi handle with CURLMOPT_QUICK_EXIT, polling the abort flag every 100ms. Option ids are passed numerically so a runtime libcurl newer than the build headers still receives them. A Linux unit test interposes getaddrinfo() to stall deterministically; it hangs 60s without this change. Files changed: - lib/http/HttpClient_Curl.hpp - tests/unittests/CMakeLists.txt - tests/unittests/HttpClientCurlTests.cpp Co-authored-by: Copilot <[email protected]> Copilot-Session: ff620832-ec83-4315-b3e0-6f5d825d45e1
Include the integer type dependency in ICdsFactory.hpp and avoid the racy mkdir implementation in build-runner-tests.py. Co-authored-by: Copilot <[email protected]> Copilot-Session: ea7f5226-1c17-476f-abaa-90cfa4b32e3c
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The curl transport (Linux) could hang the host process at exit when the collector's DNS is unreachable:
curl_easy_perform()doesn't run the progress/xferinfo callback while a name is resolving. AnAbort()during a stalled lookup therefore waited for the fullCURLOPT_CONNECTTIMEOUT(5s).curl_easy_cleanup()(curl 7.87–8.19) andcurl_multi_cleanup()(curl 8.20+, where the resolver thread pool belongs to the multi handle) join the threaded-resolver thread. That thread stays blocked ingetaddrinfo()for the system DNS timeout, which is tens of seconds when DNS is blackholed.HttpClient_Curl::CancelAllRequests()drains every operation. Hosts that tear telemetry down from static destructors therefore blocked at process exit; ONNX Runtime is one example. I reproduced this with curl 8.21 (the SDK's fetched version) and curl 8.5 (Ubuntu 24.04).Fix
CURLOPT_QUICK_EXITon every easy handle so cleanup doesn't join a stalled resolver thread.CURLMOPT_QUICK_EXIT. The loop usescurl_multi_pollin 100ms slices and checksisAbortedbetween them. A completed transfer always reports its real result. The legacyCONNECT_ONLYpath is unchanged.static_asserts whenever the headers define them. A runtime libcurl newer than the build headers then still receives the options; this is common with distro upgrades. An older runtime rejects them harmlessly.Trade-off: a resolver thread abandoned this way keeps running until
getaddrinfo()returns. If a library that statically links libcurl is unloaded while such a thread is still blocked, the thread would return into unmapped code. Normal process exit is unaffected.Test
The new
HttpClientCurlDnsStallTests.DestroyingClientDuringStalledResolutionDoesNotHangis Linux-only. It interposesgetaddrinfo()fordns-stall.invalidonly and forwards every other lookup throughdlsym(RTLD_NEXT). It asserts that destroying the client during the stall returns within 3s with oneAbortedresponse.It skips when the runtime libcurl is older than 7.87, uses a synchronous resolver, or doesn't resolve through
getaddrinfo(c-ares). Each run uses its own generation, so--gtest_repeatis safe.Validation (WSL Ubuntu 24.04, GCC 13)
UnitTests --gtest_filter=HttpClientCurl*, system curl 8.5UnitTests --gtest_filter=HttpClientCurl*,MATSDK_CURL_PROVIDER=FETCH(8.21)--gtest_repeat=3, both curl versionsUnitTests, both curl versionsclang++compile ofHttpClientCurlTests.cpp