Repository navigation
fix: paginate getVilageFcst/getUltraSrtFcst/getUltraSrtNcst instead of failing on a second page - #30
Merged
Merged
Conversation
…f failing on a second page
_fetch_items requested one 1,000-row page and raised a non-retryable
KmaParseError the moment the response reported a next page, rather than
fetching it. getVilageFcst alone publishes up to a dozen categories over
a 3-day, 3-hour-step forecast, and how many rows that produces depends on
how many categories the issuing office actually populated for a given
base_time -- comfortably under 1,000 most of the time, but not always. A
downstream consumer (kor-travel-weather) observed this failing for
several consecutive hourly runs at a time, clustered in the evening KST
hours, going stale for hours before the next base_time happened to fit.
_fetch_items now walks pageNo forward, accumulating items.item across
pages, until has_next_page() says there is nothing left, with a
_MAX_FETCH_PAGES=20 backstop in case an upstream response never reports
itself as the last page (turning that into a clear, bounded error instead
of an infinite request loop). The three call sites (now, forecast_short,
_forecast_vilage) share this method already, so all three benefit
uniformly; none of them exposed a page-size knob to fix this from the
caller's side, so this had to be fixed here.
No test previously exercised the has_next_page=True branch at all --
covered now with a two-page fetch and a page-cap test, both confirmed to
fail against the pre-fix code with exactly the production error text
("KMA response has more items than the requested page size").
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01YChhZnHkrqPpfZ3NehtB58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
무엇
_fetch_items()가 1,000행짜리 페이지 하나만 요청하고, 응답에 다음 페이지가 있으면 재시도 불가능한KmaParseError("KMA response has more items than the requested page size")로 즉시 실패시키고 있었습니다.왜
getVilageFcst(단기예보)는 3일치를 3시간 간격으로 최대 십수 개 카테고리에 걸쳐 발표합니다. base_time마다 실제로 채워지는 카테고리 수에 따라 응답 row 수가 달라지는데, 대부분은 1,000행 안에 들어오지만 항상 그런 건 아닙니다.이 라이브러리를 쓰는 다운스트림(kor-travel-weather, 시간당 실행)에서 저녁 KST 시간대에 몰려 여러 시간 연속으로 실패하는 게 관측됐습니다 — 매번 정상적인 fresh forecast 응답이었는데 두 번째 페이지가 있다는 이유만으로 버려졌습니다.
무엇을 고쳤는지
_fetch_items()가 이제pageNo를 1부터 늘려가며has_next_page()가 거짓이 될 때까지items.item을 계속 모읍니다. 응답이 절대 "마지막 페이지"라고 알려주지 않는 이상 상황에 대비해_MAX_FETCH_PAGES=20상한을 뒀습니다 — 무한 루프 대신 명확한 에러로 끝납니다.now(),forecast_short(),_forecast_vilage()세 endpoint가 전부 이 메서드를 공유하므로 셋 다 동일하게 이득을 봅니다. 호출부 어디에도 page size를 조절할 파라미터가 없어서(forecast.vilage(nx=, ny=)), 호출 측에서 우회할 방법이 없었습니다 — 여기서 고치는 게 유일한 경로입니다.테스트
has_next_page=True분기를 검사하는 테스트가 이전에 전혀 없었습니다. 이번에 2개 추가:두 테스트 모두 수정 전 코드에 대해 먼저 돌려서, 프로덕션에서 실제로 난 것과 정확히 같은 에러 문구로 실패하는 것을 확인했습니다.
검증
python -m pytest -q -m "not integration"— 180 passed (CI와 동일한 호출 방식)python -m ruff check .— 통과python -m mypy src/kma— 통과 (25 files, no issues)🤖 Generated with Claude Code
https://claude.ai/code/session_01YChhZnHkrqPpfZ3NehtB58