SPEC: what the servers read from a feedback's info (episode r/l/s/mask) - #6
Merged
Merged
Conversation
…, mask} on a finished transition, and nothing else A client written from this document trained normally but logged return 0 (plugrl-server E44): the servers' rollout metrics come only from info["episode"], which the document did not mention.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SPEC.md says a feedback's
infois free-form and that{}is valid. It never says what the servers read from it. They read one key:episode= {r,l,s,mask}, each batched tom.terminatedortruncatedis set.rollout/reward,rollout/lengthandrollout/successcome from.A client that leaves it out trains exactly the same, but those metrics read 0. plugrl-server's E44 found this the hard way. Its C++ client, written from this document, trained Pendulum normally and logged a return of 0 until it sent
episode.This adds a subsection to 5.4, "What the servers read from
info", which covers:evaluationlogs each episode.maskis treated: an entry whosemaskis false is skipped, and a missingmaskcounts as true.plugrl-env-clientsends besides: a full-lengthepisode,mean_success_rate, andis_step_success.{"episode": {...}}alone is unbatched correctly, because the server looks one level into nested maps form.The one-line pointer from the
infoparagraph is the only other change. The reference clients inexamples/still send{}, which stays valid.