diff --git a/README.md b/README.md index 700fbbc..88b13f5 100644 --- a/README.md +++ b/README.md @@ -24,14 +24,15 @@ has been measured: | [coverage](#what-runs-on-it) | Which policies and algorithms learn through the split? | **Every one tried** - sixteen combinations on four tasks, all learning, below | | [E2](experiments/e2-cross-language/) | Does an env client have to be this codebase, or Python? | **No** - a C++ client with no third-party libraries drove this server | | [E12](experiments/e12-cuda-free-rollout/) | Does a rollout machine need CUDA, or a GPU? | **No** - LIBERO's env client goes from 7.8G to 3.4G with no nvidia wheels, and renders on the CPU at 1.91x the wall clock ([E13](experiments/e13-gpu-free-rendering/)) | -| [E7](experiments/e7-cross-machine/) | What does the boundary cost once packets leave the machine? | **+0.52 ms** on a 184 KiB observation, measured from a VM to its host | -| [E10](experiments/e10-vla-forward-cost/) | Is that cheap beside a VLA forward pass? | **Yes** - 1.3-3.6% of a step | +| [E43](experiments/e43-cross-machine-training/) | Does training still work with the env clients on another physical machine? | **Yes** - the quickstart pair learns with its env clients on a Windows laptop over campus Wi-Fi | +| [E43](experiments/e43-cross-machine-training/) | What does crossing cost? | About **3 ms plus twice the observation's bytes over the link** per exchange: 21 ms for 184 KiB at 18 MB/s | +| [E10](experiments/e10-vla-forward-cost/) | Is that cheap beside a VLA forward pass? | **On a fast link.** Over that Wi-Fi a 184 KiB observation is 21% of pi0.5's 100 ms forward, not E10's 1.3-3.6% | | [E11](experiments/e11-vla-rl-libero/) | Does a full-size VLA go through it? | **Yes** - pi0.5 on LIBERO scores what openpi publishes, and the server's episode and step counts match the clients' exactly | Fine-tuning pi0.5 with reinforcement learning through it has not made the policy better yet; that record is on [its own page](https://plugrl.github.io/vla/). -[`experiments/`](experiments/) holds forty experiments, thirty-nine of them +[`experiments/`](experiments/) holds forty-one experiments, forty of them run. Each carries its data and a `FINDINGS.md` stating what the result does **not** support. The documentation, including a quickstart that trains FPO on HalfCheetah with no GPU and nothing to download, is at diff --git a/experiments/README.md b/experiments/README.md index cfcce9a..6cf8d13 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -173,10 +173,12 @@ rather than an achievement. ## What is missing -- **A true cross-machine number.** E7 got as far as one virtual machine to - its host, which separates the network stack from the machine boundary but - still shares a CPU and a hypervisor. A physical NIC and a switch will cost - more; how much more is unmeasured, and needs a second computer. +- **A cross-machine number on a fast link.** E7 got as far as one virtual + machine to its host. E43 then crossed two physical machines - a wired Linux + workstation and a Windows laptop on campus Wi-Fi - and found the cost is a + fixed latency (about 3 ms) plus twice the observation's bytes over the + link's bandwidth (18 MB/s there), with training unchanged. A wired LAN or a + datacenter link is still unmeasured; the model says how it scales. - **E3 was never run.** Its protocol is pre-registered, including an explicit declaration of the familiarity bias that would have favoured PlugRL and three ranked mitigations, but no data exists. diff --git a/experiments/e43-cross-machine-training/AMENDMENT.md b/experiments/e43-cross-machine-training/AMENDMENT.md new file mode 100644 index 0000000..a3ac518 --- /dev/null +++ b/experiments/e43-cross-machine-training/AMENDMENT.md @@ -0,0 +1,32 @@ +# E43 amendments + +## 1 - 2026-09-28, after the registered run: three more seeds per arm + +**What prompted it.** The registered run holds on every prediction. But the +two arms ended apart: +- The cross arm's last ten iterations averaged 2,060, 2,127 and 946. +- The local arm's averaged 1,262, 695 and 725. + +The figure shows that gap from about iteration 40 on. PROTOCOL.md predicted +nothing about it. The two arms should draw from the same distribution: +- the same code +- the same settings +- the same seeds +- trajectories that differ only by floating-point bits (known item 4) + +A real gap would then be a defect to find, not a result. With three seeds per +arm, luck cannot be told from a difference. + +**What is added.** Seeds 3, 4 and 5 in both arms, with the same scripts, the +same concurrent layout and the same machines. They are written to +`results/local-b`, `results/cross-b` and `results/cross-clients-b`. + +**How it is read.** The additions are reported, not predicted. P1-P6 stay as +registered, on seeds 0-2. The six seeds per arm are compared on: +- iteration 91-100's mean +- the status rule +- the extra time, as P4 measured it + +**What would count as a gap worth chasing:** all six cross seeds above all +six local seeds, or the arms' six-seed means further apart than the spread +within either arm. Anything less is read as seed variation. diff --git a/experiments/e43-cross-machine-training/FINDINGS.md b/experiments/e43-cross-machine-training/FINDINGS.md new file mode 100644 index 0000000..4c87b66 --- /dev/null +++ b/experiments/e43-cross-machine-training/FINDINGS.md @@ -0,0 +1,158 @@ +# E43: with its env clients on another physical machine, the cell still learns; crossing costs a fixed latency plus the observation's bytes over the link + +2026-09-28 · server on guangzhao (Linux workstation, wired); env clients on +a Windows 11 laptop on Wi-Fi 6, joined by Tailscale peer-to-peer on one +campus network · protocol: [`PROTOCOL.md`](PROTOCOL.md) (`46bce03`, after the +pilot in `results/pilot.txt`, before the registered run) · amendment: +[`AMENDMENT.md`](AMENDMENT.md) + +--- + +## The result + +**A. Training.** The coverage figure's `fpo-policy` · FPO · HalfCheetah +cell, with E16's command lines, ran as two arms at once. In the local arm, +servers and env clients were on guangzhao. In the cross arm, the servers +were on guangzhao and the env clients on the laptop. The table gives the mean +return over iterations 91-100 against iteration 1: + +| seed | local | cross | +| --- | --- | --- | +| 0 | -293 -> 1,262 (**+1,555**) | -345 -> 2,060 (**+2,405**) | +| 1 | -325 -> 695 (**+1,020**) | -318 -> 2,127 (**+2,445**) | +| 2 | -251 -> 725 (**+976**) | -250 -> 946 (**+1,196**) | + +* **P1 holds.** All six seeds logged 100 iterations and five checkpoints, + with no traceback, and every client exited 0. +* **V1 passes.** The cross clients connected to guangzhao's Tailscale + address and the local ones to 127.0.0.1. All six servers logged the same + configuration. +* **P2 and P3 hold.** Both arms learn, 3 of 3 seeds each, against a bar of + +200. +* **P4 holds.** The cross arm took 1,253, 1,288 and 1,356 s longer from + iteration 1 to 100, against a predicted 1,119 s ±25%. All three are inside + the band, but all three sit above the point prediction, by 12-21%. + +**B. The boundary's cost**, as the median of five runs of 1,000 exchanges. +Throughput was 20.5, 18.0, 19.6, 15.9 and 14.4 MB/s (median 18.0). + +| payload | `lo` | `ts` | `phys` | `phys` predicted | +| --- | --- | --- | --- | --- | +| states only | 0.099 ms | 0.107 ms | 2.82 ms | | +| 48 KiB | 0.161 | 0.172 | 8.01 | 8.28 | +| 184 KiB | 0.293 | 0.306 | 21.38 | 23.72 | +| 588 KiB | 0.947 | 0.700 | 65.80 | 69.72 | + +* **V2 passes.** All 60 ladder runs completed. +* **P5 holds.** `lo` and `ts` agree within 0.25 ms at every payload. The + 588 KiB row is the exception in kind, if not in verdict: 0.248 ms apart, + just inside the tolerance, and with `lo` the slower of the two - as in the + pilot. That is not explained. +* **P6 holds.** The crossing is a fixed latency plus twice the observation's + bytes over the link's throughput. Measured is 3-10% under predicted at + all three payloads. + +--- + +## What crossing costs, and what it is made of + +On one machine the boundary is a tenth of a millisecond with states only, +and under a millisecond at 588 KiB. Tailscale adds nothing measurable there: +a machine's own Tailscale address is short-circuited like any local one, +as E7 found for a machine's own Ethernet address. + +Between two machines it becomes two terms: +- **A fixed one**: about 2.8 ms here, which is Wi-Fi plus WireGuard plus two + operating systems. +- **One proportional to size**: each exchange sends the observation twice, + once in the previous feedback and once in the infer, and both queue on the + same link. At 18 MB/s that is about 0.11 ms per KiB of observation, more + than fifty times E7's VM-to-host 2 µs. + +A state-only task like this cell pays almost only the fixed term. Per call, +the env client waited 5.0 ms for an action across machines against 2.7 ms +locally. Stepping MuJoCo on the laptop took 0.50 ms against guangzhao's +0.10, and sending feedback 0.41 ms against 0.07. That doubled the wall clock, +from 20 minutes to 41, and changed nothing else; an hour later, with a busier +link and laptop, it was 52 minutes (below). + +An image-based policy pays the second term. **This corrects E10**, whose +"1.3-3.6% of a step" used E7's 1.3 ms VM-to-host crossing. Over this link a +184 KiB observation costs 21.4 ms per exchange, which is 21% of the +full-size pi0.5's 100 ms forward. On a faster link the second term shrinks +in proportion to the bandwidth, but no faster link was measured here. + +P4's three misses on the high side fit a link that was slowing during the +session: the five throughput runs fell from 20.5 to 14.4 MB/s. This is an +observation, not a test. + +--- + +## The two arms' returns + +The registered run left the cross arm's seeds higher: 2,060, 2,127 and 946, +against 1,262, 695 and 725. PROTOCOL.md did not predict that. The two arms +draw from one distribution, so a real gap would have been a defect to find. +[`AMENDMENT.md`](AMENDMENT.md) added seeds 3-5 to both arms, and fixed how +they would be read, before they ran. Iterations 91-100: + +| seed | local | cross | +| --- | --- | --- | +| 0 | 1,262 | 2,060 | +| 1 | 695 | 2,127 | +| 2 | 725 | 946 | +| 3 | 1,967 | 2,267 | +| 4 | 2,336 | 1,587 | +| 5 | 857 | 1,082 | +| mean (sd) | 1,307 (694) | 1,678 (564) | + +All twelve seeds learn by the status rule. Two of the new local seeds end +above four of the cross seeds, and the means are 371 apart, against a spread +of 694 within the local arm. By the amendment's rule this is seed variation +(`results/amendment.txt`). That is why the figure draws each arm's band, from +its lowest seed to its highest, and no mean line. + +![Both arms' seeds within one band](results/cross-machine.png) + +The amendment's cross arm also took longer: 1,875-1,985 s more than its local +seeds, against 1,253-1,356 s in the registered run. Two things were slower +an hour later. +- The link: the env clients waited 6.0 ms per call for an action instead of + 5.0. +- The laptop, which was doing other work: stepping MuJoCo took 0.73 ms + instead of 0.50, and sending feedback 0.56 ms instead of 0.41. + +Together that is about 1.4 ms per step, or 570 s over the run, which is the +difference. On a shared Wi-Fi link the crossing's cost is not a constant. + +--- + +## What E43 does not show + +* **Any other link, or this one at a steady cost.** Campus Wi-Fi through + Tailscale is slower than two machines on one wired switch, and it changed + within the evening (throughput 20.5 down to 14.4 MB/s; per-call wait 5.0 + then 6.0 ms). The model says how the cost scales, but only this link was + measured. +* **Any other second machine**, or more than one. This is one laptop and + one server. +* **Image-based training across machines.** The cell sends states. Part B + measures what images cost on this link; no image policy was trained + across it. +* **Identical trajectories.** The two machines' MuJoCo differ in the last + bit from the first step (8.9e-16) and HalfCheetah amplifies it (order 1 + by step 100, `results/pilot.txt`). On one machine a seed reproduces + exactly. So the arms are compared by the status rule, not trajectory by + trajectory. +* **Anything about an environment that does not wait.** PlugRL's env client + stops and waits for each action, so latency slows training without + changing its data. A robot, or any environment that keeps moving, turns + latency into a different problem, and E43 does not touch it. + +The runs' checkpoints and tensorboards stay on guangzhao. Kept here: +- the logs, of both arms and of the amendment's seeds +- both machines' client summaries +- `ladder.tsv` and `throughput.tsv` +- `curves.json` and `cross-machine.png` (`figure.py`) +- `verdicts.txt` and `summary.tsv` (`summarise.py`) +- `amendment.txt` (`amendment.py`) diff --git a/experiments/e43-cross-machine-training/PROTOCOL.md b/experiments/e43-cross-machine-training/PROTOCOL.md new file mode 100644 index 0000000..64527d4 --- /dev/null +++ b/experiments/e43-cross-machine-training/PROTOCOL.md @@ -0,0 +1,169 @@ +# E43 measurement protocol (pre-registered) + +**Written 2026-09-28, after the pilot in `results/pilot.txt`, before the +registered run.** + +This file must not be edited after the first registered data point. Anything +learned afterwards goes in `AMENDMENT.md`, dated. + +--- + +## The questions + +PlugRL's home page says the env clients can run on another machine. Every +cell of the coverage figure was trained with both sides on one machine, and +E7's crossing was a VM to its own host. So: + +**A. With the env clients on another physical machine, does training still +learn?** + +**B. What does crossing to that machine cost per exchange, and what is the +cost made of?** + +--- + +## Declared in advance: what was already known + +1. **The cell.** `fpo-policy` · FPO · HalfCheetah-v5 from the coverage + figure, with E16 Phase A's command lines (`server_side.sh`): 409,600 + steps, buffer 4,096, so 100 iterations. One server per seed, and the same + seed for the server and the client. Its status rule, as the figure has + it: it **learns** if the mean return over iterations 91-100 is at least + **+200** over iteration 1, on at least **2 of 3** seeds. +2. **The machines** (`results/pilot.txt`). + - The server runs on guangzhao, a wired Linux workstation. + - The second machine is a Windows 11 laptop on Wi-Fi 6. Its env client + runs natively on Windows, not in WSL. + - Both are on the same campus network and joined by Tailscale + peer-to-peer, not relayed. guangzhao's public address refuses inbound + connections, so Tailscale is the path. + - The env client is at the same commit on both machines, with the same + mujoco, gymnasium and numpy versions. +3. **The pilot's training.** Over 8,192 steps both arms ran, and the clients + exited 0. + - Per step on the server's clock: 2.94 ms local, 5.70 ms cross. + - Per call on the client's side, local then cross: + - infer wait: 2.5 ms, 4.6 ms + - env step: 0.09 ms, 0.55 ms (the laptop steps MuJoCo slower) + - feedback send: 0.07 ms, 0.45 ms +4. **Determinism.** On guangzhao, seed 0 run three times gave the same return + at every iteration. Across the two machines the returns differ from the + first iteration. The reason is the physics, not the transport: + - With no network involved, the same seed and actions give observations + that differ by 8.9e-16 after one step, 4e-7 by step 50 and order 1 by + step 100. + - This is the last bit of the floating-point math on two platforms, + amplified by a chaotic system. + - So the arms are compared by the status rule, not trajectory by + trajectory. +5. **The pilot's ladder** (one repetition, 200 exchanges). + - `lo` and `ts` are within 0.25 ms of each other at every payload. + - `phys` is 3.8 ms with states only, then 9.1, 21.8 and 55.7 ms at 48, 184 + and 588 KiB. + - Pilot throughput was 19.0, 21.7 and 23.5 MB/s. With it, `phys` fits a + fixed cost plus twice the observation's bytes over the link's + throughput. The observation crosses twice per exchange: once in the + previous feedback, once in the infer. + - **This model was suggested by the pilot, and it is declared as such**. + P6 tests whether it holds on fresh measurements. + +--- + +## Design + +**Part B first**, so that the training does not share the link with it: + +- **Throughput.** Five transfers of 64 MiB, laptop to guangzhao + (`throughput.sh`). +- **The ladder.** `ladder.sh`, with plugrl-protocol's conformance server on + guangzhao and `bench_client.py` on either machine: + - three rungs: `lo` and `ts` from guangzhao itself, `phys` from the laptop + - four payloads: states only, 48, 184 and 588 KiB + - five repetitions of 1,000 exchanges each; the rung order reverses on + even repetitions + +**Then Part A.** The local arm and the cross arm run at the same time, seeds +0-2, with 409,600 steps each (`server_side.sh local`, `server_side.sh cross`, +`client_side.sh`): + +- **The local arm**: servers and clients on guangzhao. +- **The cross arm**: servers on guangzhao, clients on the laptop. + +`summarise.py` reads everything. + +--- + +## Checks + +* **V1 - the arms are what they say.** + - Every cross client connected to guangzhao's Tailscale address, and every + local client to 127.0.0.1. + - The six servers logged the same algorithm configuration apart from the + port and the name. +* **V2 - a ladder run counts** only if its client completed its 1,000 + exchanges and exited 0. + +--- + +## Predictions, and what falsifies each + +**P1 - both arms run end to end**: on all six seeds, 100 iterations logged, +five checkpoints, no traceback, and the client exited 0. + +**P2 - the cross arm learns**, by the status rule. **P3 - the local arm +learns**, by the same rule. + +> Grounds: known item 1. E16's control learned this cell on one machine, from +> about -300 to 1,346-2,050. Where the environment is stepped changes the +> timing, not the data a step produces; item 4's differences are +> floating-point, not systematic. Each is falsified if fewer than two seeds +> clear the bar. + +**P4 - the cross arm's extra time is the pilot's per-step cost.** For each +seed, the cross arm's span from iteration 1 to iteration 100 on the server's +clock minus the local arm's, against 99 x 4,096 x 2.76 ms = **1,119 s**. + +> Grounds: known item 3. This fails if crossing costs something that grows +> over a run and that two iterations do not show, such as queueing or +> backlog. Holds if the difference is within **±25%** (839-1,399 s) on at +> least 2 of 3 seeds. + +**P5 - `lo` and `ts` agree.** At every payload, the medians of the five +repetitions are within **0.25 ms**. + +> Grounds: known item 5, and E7's `self` rung. On one machine a Tailscale +> address is short-circuited like any other local one. + +**P6 - the crossing cost is fixed latency plus bytes over bandwidth.** Take +B = the median throughput of the five transfers. For 48, 184 and 588 KiB, +the median `phys` round trip is predicted as the median `phys` round trip +with states only, plus 2 x (the observation's image bytes) / B. It holds if +the measured median is within **±25%** of the prediction at all three +payloads. + +> Grounds: known item 5, and suggested by it (declared above). + +**Reported, not predicted:** +- every rung's pack, round-trip and unpack times +- the throughputs +- the env clients' timing breakdown in both arms +- the returns at iterations 1, 50 and 91-100 for every seed +- wall clock +- E10's ratio recomputed with `phys` at 184 KiB against the full-size + pi0.5's 100 ms forward + +--- + +## Declared deviations allowed in advance + +1. One restart of any cross-arm seed that dies for a reason outside the + experiment, such as the laptop's Wi-Fi dropping or the laptop sleeping. + It is recorded in `AMENDMENT.md`. The local arm's matching seed is rerun + with it, so that the two arms stay concurrent. + +--- + +## Reading order + +P1, V1, the status rule, P2, P3, P4; then V2, P5, P6; then the reported +figures. diff --git a/experiments/e43-cross-machine-training/amendment.py b/experiments/e43-cross-machine-training/amendment.py new file mode 100644 index 0000000..fcddcf4 --- /dev/null +++ b/experiments/e43-cross-machine-training/amendment.py @@ -0,0 +1,85 @@ +"""Read AMENDMENT.md's six seeds per arm: seeds 0-2 from the registered run, +3-5 from the amendment. + + python amendment.py [results-dir] + +Per seed: iterations 91-100's mean, the gain over iteration 1, and the span +from iteration 1 to 100 on the server's clock. Then AMENDMENT.md's test for +a gap worth chasing: all six cross seeds above all six local seeds, or the +arms' six-seed means further apart than the standard deviation within +either arm. +""" + +import pathlib +import statistics +import sys + +from tensorboard.backend.event_processing.event_accumulator import EventAccumulator + +HERE = pathlib.Path(__file__).resolve().parent +R = pathlib.Path(sys.argv[1]).resolve() if len(sys.argv) > 1 else HERE / "results" +LAYOUT = {("local", s): "local" for s in (0, 1, 2)} | { + ("cross", s): "cross" for s in (0, 1, 2) +} +LAYOUT |= {("local", s): "local-b" for s in (3, 4, 5)} | { + ("cross", s): "cross-b" for s in (3, 4, 5) +} + + +def read(arm: str, seed: int): + run = ( + R / LAYOUT[(arm, seed)] / "fpo" / "fpo-policy" / f"halfcheetah-{arm}-seed{seed}" + ) + acc = EventAccumulator( + str(sorted((run / "tensorboard").glob("events.*"))[-1]), + size_guidance={"scalars": 0}, + ) + acc.Reload() + r = acc.Scalars("rollout/reward") + v = [e.value for e in r] + return len(v), v[0], statistics.mean(v[90:100]), r[99].wall_time - r[0].wall_time + + +def main() -> int: + last, gains, spans = {"local": [], "cross": []}, {"local": [], "cross": []}, {} + print("seed arm iterations first last10 gain span_s") + for seed in range(6): + for arm in ("local", "cross"): + n, first, last10, span = read(arm, seed) + last[arm].append(last10) + gains[arm].append(last10 - first) + spans[(arm, seed)] = span + print( + f"{seed:4d} {arm:5s} {n:10d} {first:6.0f} {last10:7.0f} {last10 - first:+7.0f} {span:8.0f}" + ) + + print("\nstatus rule over six seeds (+200 over iteration 1)") + for arm in ("local", "cross"): + print(f" {arm}: {sum(g >= 200 for g in gains[arm])} of 6") + + print("\nextra time, cross minus local, iteration 1 -> 100 (P4 predicted 1,119 s)") + for seed in range(6): + print( + f" seed {seed}: {spans[('cross', seed)] - spans[('local', seed)]:.0f} s" + ) + + lo, cr = last["local"], last["cross"] + ml, mc = statistics.mean(lo), statistics.mean(cr) + sl, sc = statistics.stdev(lo), statistics.stdev(cr) + separated = min(cr) > max(lo) + apart = abs(mc - ml) > max(sl, sc) + print("\niterations 91-100, six seeds per arm") + print(f" local mean {ml:.0f} sd {sl:.0f} range {min(lo):.0f}-{max(lo):.0f}") + print(f" cross mean {mc:.0f} sd {sc:.0f} range {min(cr):.0f}-{max(cr):.0f}") + print(f" all six cross above all six local: {'yes' if separated else 'no'}") + print( + f" means further apart ({abs(mc - ml):.0f}) than either arm's sd ({max(sl, sc):.0f}): {'yes' if apart else 'no'}" + ) + print( + f" -> {'a gap worth chasing' if separated or apart else 'read as seed variation'}" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/experiments/e43-cross-machine-training/bench_client.py b/experiments/e43-cross-machine-training/bench_client.py new file mode 100644 index 0000000..f14d1db --- /dev/null +++ b/experiments/e43-cross-machine-training/bench_client.py @@ -0,0 +1,164 @@ +"""E43's boundary-cost client: plugrl-protocol's examples/raw_client.py with a clock. + +raw_client.py speaks the protocol with nothing but msgpack and websockets, and +so runs unchanged on Windows and Linux alike. E7 timed the boundary with the +C++ client, which is Linux-only; E43 needs the same measurement from a Windows +laptop, so this keeps raw_client.py's encoding and adds what E7's client +reported: per-exchange pack, round-trip and unpack time. + + python bench_client.py --host H --port P --steps 1000 --img 224 --cameras 2 + +--img 0 sends states only (HalfCheetah's 17 floats), the observation E43's +training run sends. Otherwise a base camera of IMG x IMG, and with --cameras 2 +a wrist camera of half that, as E7: 128/1 is 48 KiB, 224/2 184 KiB, 448/1 +588 KiB. Prints one tab-separated line: + + exchanges pack_ms rtt_ms unpack_ms wall_s exchanges_per_s + +where each *_ms is the mean over the exchanges, as E7's were. +""" + +from __future__ import annotations + +import argparse +import sys +import time +from array import array + +import msgpack +import websockets.sync.client + +INFER, FEEDBACK, METADATA, ACTION = "infer", "feedback", "metadata", "action" +_ARRAY_CODE = {("f", 4): "f", ("f", 8): "d", ("i", 8): "q", ("u", 1): "B"} + + +def encode_array(values, typestr: str, shape: tuple[int, ...]) -> dict: + order, kind, size = typestr[0], typestr[1], int(typestr[2:]) + if kind == "b": + raw = bytes(1 if v else 0 for v in values) + else: + buf = array(_ARRAY_CODE[(kind, size)], values) + if (sys.byteorder == "little") != (order in "<|"): + buf.byteswap() + raw = buf.tobytes() + return { + b"__ndarray__": True, + b"data": raw, + b"dtype": typestr, + b"shape": list(shape), + } + + +def decode_array(obj: dict) -> list: + typestr = obj[b"dtype"] + typestr = typestr.decode() if isinstance(typestr, bytes) else typestr + order, kind, size = typestr[0], typestr[1], int(typestr[2:]) + buf = array(_ARRAY_CODE[(kind, size)]) + buf.frombytes(obj[b"data"]) + if (sys.byteorder == "little") != (order in "<|"): + buf.byteswap() + return list(buf) + + +def image(h: int) -> dict: + n = h * h * 3 + raw = (bytes(range(256)) * (n // 256 + 1))[:n] + return {b"__ndarray__": True, b"data": raw, b"dtype": "|u1", b"shape": [1, h, h, 3]} + + +def observation(img: int, cameras: int) -> dict: + images = {} + if img: + images["base"] = image(img) + if cameras == 2: + images["wrist"] = image(img // 2) + return { + "images": images, + "states": {"obs": encode_array([0.1] * 17, " int: + packer = msgpack.Packer() + env_indices = encode_array([0], " int: + p = argparse.ArgumentParser(description=__doc__) + p.add_argument("--host", default="127.0.0.1") + p.add_argument("--port", type=int, default=8000) + p.add_argument("--steps", type=int, default=1000) + p.add_argument("--img", type=int, default=0) + p.add_argument("--cameras", type=int, default=1, choices=(1, 2)) + a = p.parse_args() + return run(a.host, a.port, a.steps, a.img, a.cameras) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/e43-cross-machine-training/client_side.sh b/experiments/e43-cross-machine-training/client_side.sh new file mode 100644 index 0000000..c3b39de --- /dev/null +++ b/experiments/e43-cross-machine-training/client_side.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# E43's env clients for the cross arm, on the second machine: a Windows laptop +# that reaches guangzhao over Tailscale. The same command line as the local +# arm's clients, with guangzhao's address in place of 127.0.0.1. +# +# bash client_side.sh STEPS [OUT_DIR] +# +# PLUGRL_CLIENT_DIR is an env-client checkout at guangzhao's commit, with the +# mujoco extra and guangzhao's mujoco and gymnasium versions. +set -uo pipefail + +STEPS="$1" +HERE="$(cd "$(dirname "$0")" && pwd)" +OUT="${2:-$HERE/results/cross-clients}" +mkdir -p "$OUT" +HOST="${SERVER_HOST:-100.75.226.89}" +PORT_BASE="${PORT_BASE:-9790}" +SEEDS="${SEEDS:-0 1 2}" +CLIENT_DIR="${PLUGRL_CLIENT_DIR:?set PLUGRL_CLIENT_DIR to the env-client checkout}" +CLIENT_PY="$CLIENT_DIR/.venv/Scripts/python.exe" +[ -x "$CLIENT_PY" ] || CLIENT_PY="$CLIENT_DIR/.venv/bin/python" + +echo "client: $CLIENT_PY" +echo "server: $HOST ports $PORT_BASE+seed" +echo "out: $OUT" +date '+start %F %T' + +run_seed() { + local seed="$1" port=$((PORT_BASE + $1)) + (cd "$CLIENT_DIR" && "$CLIENT_PY" -m plugrl_env_client.cli mujoco-v1 \ + --server-port "$port" --server-host "$HOST" \ + --num-envs 1 --num-episodes $((STEPS / 1000 + 10)) \ + --runner.replan-steps 1 --runner.seed "$seed" \ + --exp-name "e43-cross-seed$seed" \ + > "$OUT/client-seed$seed.log" 2>&1) + echo "seed $seed client rc=$? at $(date '+%T')" +} + +PIDS=() +for seed in $SEEDS; do + run_seed "$seed" & + PIDS+=($!) + sleep 5 +done +for pid in "${PIDS[@]}"; do wait "$pid"; done +date '+end %F %T' +echo "CLIENTS_DONE" diff --git a/experiments/e43-cross-machine-training/figure.py b/experiments/e43-cross-machine-training/figure.py new file mode 100644 index 0000000..233b5c2 --- /dev/null +++ b/experiments/e43-cross-machine-training/figure.py @@ -0,0 +1,106 @@ +"""E43's figure: the same cell trained on one machine and across two. + + python figure.py export [results-dir] # on guangzhao: tensorboards -> results/curves.json + python figure.py plot OUT.png # anywhere with matplotlib + +Every seed's return per iteration, over its arm's band from the lowest seed to +the highest. No mean line: with six seeds the arms' means differ by less than +either arm's spread (AMENDMENT.md). The legend carries each arm's range of +wall clock from iteration 1 to 100. +""" + +import json +import pathlib +import sys + +HERE = pathlib.Path(__file__).resolve().parent +CURVES = HERE / "results" / "curves.json" +# Seeds 0-2 are the registered run; 3-5 are AMENDMENT.md's, in the -b directories. +LAYOUT = [(0, ""), (1, ""), (2, ""), (3, "-b"), (4, "-b"), (5, "-b")] + + +def export(results: pathlib.Path) -> None: + from tensorboard.backend.event_processing.event_accumulator import EventAccumulator + + data = {} + for arm in ("local", "cross"): + data[arm] = {"returns": [], "span_s": []} + for seed, suffix in LAYOUT: + run = ( + results + / f"{arm}{suffix}" + / "fpo" + / "fpo-policy" + / f"halfcheetah-{arm}-seed{seed}" + ) + if not run.exists(): + continue + acc = EventAccumulator( + str(sorted((run / "tensorboard").glob("events.*"))[-1]), + size_guidance={"scalars": 0}, + ) + acc.Reload() + r = acc.Scalars("rollout/reward") + data[arm]["returns"].append([round(e.value, 1) for e in r]) + data[arm]["span_s"].append(round(r[-1].wall_time - r[0].wall_time, 1)) + CURVES.write_text(json.dumps(data) + "\n", encoding="utf-8") + print("wrote", CURVES) + + +def plot(out: pathlib.Path) -> None: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + data = json.loads(CURVES.read_text(encoding="utf-8")) + style = { + "local": ("#9aa5d8", "one machine"), + "cross": ("#4051b5", "two machines, over campus Wi-Fi"), + } + fig, ax = plt.subplots(figsize=(6.4, 3.4), dpi=200) + for arm in ("local", "cross"): + colour, label = style[arm] + runs = data[arm]["returns"] + n = min(len(r) for r in runs) + x = list(range(1, n + 1)) + # The spread across seeds, and each seed: no mean line, since the + # arms' means differ by less than either arm's spread (AMENDMENT.md). + lo = [min(r[i] for r in runs) for i in range(n)] + hi = [max(r[i] for r in runs) for i in range(n)] + ax.fill_between(x, lo, hi, color=colour, alpha=0.18, linewidth=0) + for r in runs: + ax.plot(x, r[:n], color=colour, linewidth=0.8, alpha=0.7) + spans = [s / 60 for s in data[arm]["span_s"]] + ax.plot( + [], + [], + color=colour, + linewidth=6, + alpha=0.5, + label=f"{label} ({min(spans):.0f}-{max(spans):.0f} min)", + ) + ax.set_xlabel("iteration (4,096 steps each)") + ax.set_ylabel("episode return") + seeds = len(data["local"]["returns"]) + ax.set_title( + f"fpo-policy · FPO · HalfCheetah-v5, {seeds} seeds per arm", fontsize=10 + ) + for side in ("top", "right"): + ax.spines[side].set_visible(False) + ax.grid(axis="y", color="#e3e6f0", linewidth=0.6) + ax.legend(frameon=False, loc="upper left", fontsize=8.5) + fig.tight_layout() + fig.savefig(out, facecolor="white") + print("wrote", out) + + +if __name__ == "__main__": + if sys.argv[1] == "export": + export( + pathlib.Path(sys.argv[2]).resolve() + if len(sys.argv) > 2 + else HERE / "results" + ) + else: + plot(pathlib.Path(sys.argv[2])) diff --git a/experiments/e43-cross-machine-training/ladder.sh b/experiments/e43-cross-machine-training/ladder.sh new file mode 100644 index 0000000..a6cfff1 --- /dev/null +++ b/experiments/e43-cross-machine-training/ladder.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# E43's boundary cost between two physical machines: E7's ladder with the +# server fixed on guangzhao and the client moved. +# +# lo client on guangzhao -> server on guangzhao, 127.0.0.1 +# ts client on guangzhao -> server on guangzhao, its own Tailscale address +# phys client on the laptop -> server on guangzhao, over Tailscale +# +# The server is plugrl-protocol's conformance server, as in E7; the client is +# bench_client.py on both machines. Run from the laptop: +# +# bash ladder.sh REPS STEPS OUT_TSV +set -uo pipefail + +REPS="$1" +STEPS="$2" +TSV="$3" +HERE="$(cd "$(dirname "$0")" && pwd)" +REMOTE="${REMOTE:-guangzhao-memeseeks}" +TS_IP="${TS_IP:-100.75.226.89}" +LOCAL_PY="${PLUGRL_CLIENT_DIR:?set PLUGRL_CLIENT_DIR}/.venv/Scripts/python.exe" +R_PY='~/zuogou/plugrl/plugrl-server/.venv/bin/python' +R_PROTO='~/zuogou/plugrl/e43-protocol' +R_BENCH='~/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/bench_client.py' + +declare -A PAYLOADS=( [0]="0 1" [48]="128 1" [184]="224 2" [588]="448 1" ) +ORDER=(0 48 184 588) + +printf 'rung\tpayload_kib\trep\texchanges\tpack_ms\trtt_ms\tunpack_ms\twall_s\texchanges_per_s\n' > "$TSV" +date '+start %F %T' + +start_server() { # $1 = port; returns once it listens + ssh "$REMOTE" "cd $R_PROTO && (nohup $R_PY examples/conformance_server.py --host 0.0.0.0 --port $1 --steps $STEPS --timeout 300 > /tmp/e43srv_$1.log 2>&1 &) ; for i in \$(seq 100); do grep -q 'conformance server on' /tmp/e43srv_$1.log 2>/dev/null && exit 0; sleep 0.1; done; exit 1" +} +stop_server() { # $1 = port + ssh "$REMOTE" "pkill -f 'conformance_server.py --host 0.0.0.0 --port $1 ' ; true" +} + +port=9800 +for rep in $(seq "$REPS"); do + if [ $((rep % 2)) -eq 1 ]; then RUNGS=(lo ts phys); else RUNGS=(phys ts lo); fi + for kib in "${ORDER[@]}"; do + read -r img cams <<< "${PAYLOADS[$kib]}" + args="--port $port --steps $STEPS --img $img --cameras $cams" + for rung in "${RUNGS[@]}"; do + start_server "$port" || { echo "server on $port did not start" >&2; continue; } + case "$rung" in + lo) out=$(ssh "$REMOTE" "$R_PY $R_BENCH --host 127.0.0.1 $args") ;; + ts) out=$(ssh "$REMOTE" "$R_PY $R_BENCH --host $TS_IP $args") ;; + phys) out=$("$LOCAL_PY" "$HERE/bench_client.py" --host "$TS_IP" $args) ;; + esac + rc=$? + stop_server "$port" + if [ "$rc" -eq 0 ] && [ -n "$out" ]; then + printf '%s\t%s\t%s\t%s\n' "$rung" "$kib" "$rep" "$out" >> "$TSV" + printf '%s\t%s\t%s\t%s\n' "$rung" "$kib" "$rep" "$out" + else + echo "FAILED $rung $kib rep $rep rc=$rc" >&2 + fi + port=$((port + 1)) + [ "$port" -ge 9850 ] && port=9800 + args="--port $port --steps $STEPS --img $img --cameras $cams" + done + done +done +date '+end %F %T' +echo LADDER_DONE diff --git a/experiments/e43-cross-machine-training/results/amendment.txt b/experiments/e43-cross-machine-training/results/amendment.txt new file mode 100644 index 0000000..75a8bba --- /dev/null +++ b/experiments/e43-cross-machine-training/results/amendment.txt @@ -0,0 +1,32 @@ +seed arm iterations first last10 gain span_s + 0 local 100 -293 1262 +1555 1223 + 0 cross 100 -345 2060 +2404 2476 + 1 local 100 -325 695 +1019 1187 + 1 cross 100 -318 2127 +2445 2475 + 2 local 100 -250 725 +976 1175 + 2 cross 100 -250 946 +1196 2530 + 3 local 100 -321 1967 +2288 1277 + 3 cross 100 -283 2267 +2550 3151 + 4 local 100 -312 2336 +2647 1204 + 4 cross 100 -314 1587 +1901 3189 + 5 local 100 -346 857 +1204 1188 + 5 cross 100 -270 1082 +1352 3151 + +status rule over six seeds (+200 over iteration 1) + local: 6 of 6 + cross: 6 of 6 + +extra time, cross minus local, iteration 1 -> 100 (P4 predicted 1,119 s) + seed 0: 1253 s + seed 1: 1288 s + seed 2: 1356 s + seed 3: 1875 s + seed 4: 1985 s + seed 5: 1963 s + +iterations 91-100, six seeds per arm + local mean 1307 sd 694 range 695-2336 + cross mean 1678 sd 564 range 946-2267 + all six cross above all six local: no + means further apart (371) than either arm's sd (694): no + -> read as seed variation diff --git a/experiments/e43-cross-machine-training/results/cross-b.out b/experiments/e43-cross-machine-training/results/cross-b.out new file mode 100644 index 0000000..5af31f5 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-b.out @@ -0,0 +1,14 @@ +arm: cross ports 9790+seed +server: /home/guangzhao/zuogou/plugrl/e43-reg/../plugrl-server/.venv/bin/python (src /home/guangzhao/zuogou/plugrl/e43-reg/src) +out: /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b +steps: 409600 seeds: 3 4 5 +start 2026-09-28 18:23:56 +READY seed 3 port 9793 at 18:23:59 +READY seed 4 port 9794 at 18:24:04 +READY seed 5 port 9795 at 18:24:09 +seed 3 server rc=0 at 19:17:11 +seed 5 server rc=0 at 19:17:20 +seed 4 server rc=0 at 19:17:53 +end 2026-09-28 19:17:53 +failed seeds: 0 +ARM_DONE cross diff --git a/experiments/e43-cross-machine-training/results/cross-b/server-seed3.log b/experiments/e43-cross-machine-training/results/cross-b/server-seed3.log new file mode 100644 index 0000000..9ab94db --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-b/server-seed3.log @@ -0,0 +1,23 @@ +18:23:58|INFO|plugrl_server version: 0.1.0 +18:23:58|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +18:23:58|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +18:23:58|INFO|Seeded python, numpy and torch with 3. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +18:23:58|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed3 +18:23:59|INFO|Policy created... +18:23:59|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +18:23:59|INFO|Algorithm created: + +18:23:59|INFO|Agent Server is listening on 0.0.0.0:9793 +18:35:11|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed3/81921 +18:44:44|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed3/163841 +18:53:42|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed3/245761 +19:03:37|INFO|Checkpoint saved at step 327680 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed3/327680 +19:17:11|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed3/409601 +19:17:11|INFO|Stopping server as the algorithm signaled to stop. +19:17:11|INFO|Shutdown started: aborting pending infer requests. +19:17:11|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +19:17:11|INFO|Shutdown closing 1 websocket connection(s). +19:17:11|INFO|Shutdown interrupted an in-flight inference request from ('100.65.224.34', 58498). +19:17:11|INFO|WebSocket server closed. +19:17:11|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/cross-b/server-seed4.log b/experiments/e43-cross-machine-training/results/cross-b/server-seed4.log new file mode 100644 index 0000000..f9c5abb --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-b/server-seed4.log @@ -0,0 +1,23 @@ +18:24:03|INFO|plugrl_server version: 0.1.0 +18:24:03|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +18:24:03|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +18:24:03|INFO|Seeded python, numpy and torch with 4. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +18:24:03|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed4 +18:24:04|INFO|Policy created... +18:24:04|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +18:24:04|INFO|Algorithm created: + +18:24:04|INFO|Agent Server is listening on 0.0.0.0:9794 +18:35:16|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed4/81921 +18:44:49|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed4/163841 +18:53:55|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed4/245761 +19:04:04|INFO|Checkpoint saved at step 327680 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed4/327680 +19:17:53|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed4/409601 +19:17:53|INFO|Stopping server as the algorithm signaled to stop. +19:17:53|INFO|Shutdown started: aborting pending infer requests. +19:17:53|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +19:17:53|INFO|Shutdown closing 1 websocket connection(s). +19:17:53|INFO|Shutdown interrupted an in-flight inference request from ('100.65.224.34', 52623). +19:17:53|INFO|WebSocket server closed. +19:17:53|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/cross-b/server-seed5.log b/experiments/e43-cross-machine-training/results/cross-b/server-seed5.log new file mode 100644 index 0000000..e39079c --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-b/server-seed5.log @@ -0,0 +1,23 @@ +18:24:09|INFO|plugrl_server version: 0.1.0 +18:24:09|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +18:24:09|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +18:24:09|INFO|Seeded python, numpy and torch with 5. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +18:24:09|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed5 +18:24:09|INFO|Policy created... +18:24:09|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +18:24:09|INFO|Algorithm created: + +18:24:09|INFO|Agent Server is listening on 0.0.0.0:9795 +18:35:19|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed5/81921 +18:44:51|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed5/163841 +18:53:49|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed5/245761 +19:03:45|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed5/327681 +19:17:19|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross-b/fpo/fpo-policy/halfcheetah-cross-seed5/409601 +19:17:19|INFO|Stopping server as the algorithm signaled to stop. +19:17:19|INFO|Shutdown started: aborting pending infer requests. +19:17:19|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +19:17:19|INFO|Shutdown closing 1 websocket connection(s). +19:17:19|INFO|Shutdown interrupted an in-flight inference request from ('100.65.224.34', 52637). +19:17:19|INFO|WebSocket server closed. +19:17:19|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed3.log b/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed3.log new file mode 100644 index 0000000..0fdcb8e --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed3.log @@ -0,0 +1,115 @@ +2026-09-28 18:25:00.316 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 18:25:00.319 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 18:25:00.322 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 18:25:00.326 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 18:25:00.333 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 18:25:00.557 | INFO  | __main__:main:83 - Starting env client exp_name=e43-cross-seed3 output_dir=runs\e43-cross-seed3 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 18:25:00.716 | INFO  | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://100.75.226.89:9793... +2026-09-28 18:25:00.752 | INFO  | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 18:25:30.757 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=4692 infer_calls=4692 feedback_calls=4692 infer_wait=23.531s infer_obs_pack=0.446s env_step=2.618s feedback_total=2.068s feedback_obs_pack=0.513s feedback_info_pack=0.076s effective_fps=163.69 +2026-09-28 18:26:00.762 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=8863 infer_calls=8863 feedback_calls=8863 infer_wait=46.739s infer_obs_pack=0.933s env_step=5.374s feedback_total=4.249s feedback_obs_pack=1.072s feedback_info_pack=0.162s effective_fps=154.69 +2026-09-28 18:26:30.769 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=12547 infer_calls=12547 feedback_calls=12547 infer_wait=69.423s infer_obs_pack=1.507s env_step=8.385s feedback_total=6.508s feedback_obs_pack=1.674s feedback_info_pack=0.248s effective_fps=146.20 +2026-09-28 18:27:00.775 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=16528 infer_calls=16528 feedback_calls=16528 infer_wait=92.339s infer_obs_pack=2.044s env_step=11.239s feedback_total=8.785s feedback_obs_pack=2.246s feedback_info_pack=0.342s effective_fps=144.47 +2026-09-28 18:27:30.783 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=19046 infer_calls=19046 feedback_calls=19046 infer_wait=115.414s infer_obs_pack=2.565s env_step=14.131s feedback_total=10.992s feedback_obs_pack=2.763s feedback_info_pack=0.428s effective_fps=133.09 +2026-09-28 18:28:00.785 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=22484 infer_calls=22484 feedback_calls=22484 infer_wait=137.780s infer_obs_pack=3.146s env_step=17.338s feedback_total=13.301s feedback_obs_pack=3.363s feedback_info_pack=0.526s effective_fps=131.05 +2026-09-28 18:28:30.785 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=26664 infer_calls=26664 feedback_calls=26664 infer_wait=160.849s infer_obs_pack=3.651s env_step=20.195s feedback_total=15.444s feedback_obs_pack=3.917s feedback_info_pack=0.609s effective_fps=133.23 +2026-09-28 18:29:00.790 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=30248 infer_calls=30248 feedback_calls=30248 infer_wait=184.168s infer_obs_pack=4.144s env_step=22.944s feedback_total=17.563s feedback_obs_pack=4.449s feedback_info_pack=0.688s effective_fps=132.19 +2026-09-28 18:29:30.795 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=34667 infer_calls=34667 feedback_calls=34667 infer_wait=207.390s infer_obs_pack=4.638s env_step=25.683s feedback_total=19.742s feedback_obs_pack=5.012s feedback_info_pack=0.771s effective_fps=134.65 +2026-09-28 18:30:00.796 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=38071 infer_calls=38071 feedback_calls=38071 infer_wait=229.961s infer_obs_pack=5.164s env_step=28.836s feedback_total=22.012s feedback_obs_pack=5.628s feedback_info_pack=0.864s effective_fps=133.13 +2026-09-28 18:30:30.804 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41490 infer_calls=41490 feedback_calls=41490 infer_wait=252.421s infer_obs_pack=5.709s env_step=32.032s feedback_total=24.327s feedback_obs_pack=6.260s feedback_info_pack=0.951s effective_fps=131.93 +2026-09-28 18:31:00.806 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=44713 infer_calls=44713 feedback_calls=44713 infer_wait=275.123s infer_obs_pack=6.216s env_step=35.044s feedback_total=26.549s feedback_obs_pack=6.833s feedback_info_pack=1.039s effective_fps=130.38 +2026-09-28 18:31:30.817 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=47293 infer_calls=47293 feedback_calls=47293 infer_wait=297.428s infer_obs_pack=6.747s env_step=38.330s feedback_total=28.771s feedback_obs_pack=7.422s feedback_info_pack=1.113s effective_fps=127.38 +2026-09-28 18:32:00.850 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=50500 infer_calls=50500 feedback_calls=50500 infer_wait=320.347s infer_obs_pack=7.303s env_step=41.359s feedback_total=30.957s feedback_obs_pack=7.972s feedback_info_pack=1.224s effective_fps=126.26 +2026-09-28 18:32:30.855 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=53840 infer_calls=53840 feedback_calls=53840 infer_wait=343.651s infer_obs_pack=7.789s env_step=44.214s feedback_total=33.031s feedback_obs_pack=8.531s feedback_info_pack=1.310s effective_fps=125.59 +2026-09-28 18:33:00.856 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=57719 infer_calls=57719 feedback_calls=57719 infer_wait=366.152s infer_obs_pack=8.315s env_step=47.372s feedback_total=35.391s feedback_obs_pack=9.118s feedback_info_pack=1.402s effective_fps=126.24 +2026-09-28 18:33:30.858 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=61615 infer_calls=61615 feedback_calls=61615 infer_wait=389.267s infer_obs_pack=8.815s env_step=50.201s feedback_total=37.570s feedback_obs_pack=9.667s feedback_info_pack=1.484s effective_fps=126.82 +2026-09-28 18:34:00.858 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=66478 infer_calls=66478 feedback_calls=66478 infer_wait=412.626s infer_obs_pack=9.283s env_step=52.905s feedback_total=39.720s feedback_obs_pack=10.208s feedback_info_pack=1.564s effective_fps=129.20 +2026-09-28 18:34:30.868 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=70752 infer_calls=70752 feedback_calls=70752 infer_wait=435.829s infer_obs_pack=9.750s env_step=55.692s feedback_total=41.871s feedback_obs_pack=10.743s feedback_info_pack=1.643s effective_fps=130.26 +2026-09-28 18:35:00.882 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=75100 infer_calls=75100 feedback_calls=75100 infer_wait=458.986s infer_obs_pack=10.240s env_step=58.583s feedback_total=43.973s feedback_obs_pack=11.290s feedback_info_pack=1.723s effective_fps=131.34 +2026-09-28 18:35:30.892 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=78805 infer_calls=78805 feedback_calls=78805 infer_wait=481.496s infer_obs_pack=10.765s env_step=61.729s feedback_total=46.285s feedback_obs_pack=11.876s feedback_info_pack=1.827s effective_fps=131.28 +2026-09-28 18:36:00.892 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=82199 infer_calls=82199 feedback_calls=82199 infer_wait=504.014s infer_obs_pack=11.226s env_step=64.953s feedback_total=48.625s feedback_obs_pack=12.507s feedback_info_pack=1.919s effective_fps=130.72 +2026-09-28 18:36:30.905 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=86027 infer_calls=86027 feedback_calls=86027 infer_wait=526.934s infer_obs_pack=11.735s env_step=67.863s feedback_total=50.887s feedback_obs_pack=13.105s feedback_info_pack=2.011s effective_fps=130.86 +2026-09-28 18:37:01.831 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=90114 infer_calls=90114 feedback_calls=90114 infer_wait=550.996s infer_obs_pack=12.196s env_step=70.685s feedback_total=53.113s feedback_obs_pack=13.689s feedback_info_pack=2.095s effective_fps=131.17 +2026-09-28 18:37:31.830 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=93640 infer_calls=93640 feedback_calls=93640 infer_wait=573.777s infer_obs_pack=12.707s env_step=73.774s feedback_total=55.346s feedback_obs_pack=14.273s feedback_info_pack=2.180s effective_fps=130.85 +2026-09-28 18:38:01.831 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=96212 infer_calls=96212 feedback_calls=96212 infer_wait=596.460s infer_obs_pack=13.236s env_step=76.966s feedback_total=57.592s feedback_obs_pack=14.853s feedback_info_pack=2.279s effective_fps=129.27 +2026-09-28 18:38:31.834 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=99885 infer_calls=99885 feedback_calls=99885 infer_wait=619.157s infer_obs_pack=13.762s env_step=79.978s feedback_total=59.895s feedback_obs_pack=15.469s feedback_info_pack=2.365s effective_fps=129.25 +2026-09-28 18:39:01.847 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=103444 infer_calls=103444 feedback_calls=103444 infer_wait=642.067s infer_obs_pack=14.264s env_step=82.963s feedback_total=62.115s feedback_obs_pack=16.047s feedback_info_pack=2.448s effective_fps=129.08 +2026-09-28 18:39:31.852 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=107252 infer_calls=107252 feedback_calls=107252 infer_wait=665.339s infer_obs_pack=14.712s env_step=85.733s feedback_total=64.249s feedback_obs_pack=16.612s feedback_info_pack=2.532s effective_fps=129.21 +2026-09-28 18:40:01.859 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=111755 infer_calls=111755 feedback_calls=111755 infer_wait=688.846s infer_obs_pack=15.148s env_step=88.423s feedback_total=66.301s feedback_obs_pack=17.128s feedback_info_pack=2.609s effective_fps=130.14 +2026-09-28 18:40:31.858 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=115640 infer_calls=115640 feedback_calls=115640 infer_wait=711.887s infer_obs_pack=15.639s env_step=91.263s feedback_total=68.541s feedback_obs_pack=17.756s feedback_info_pack=2.691s effective_fps=130.32 +2026-09-28 18:41:01.864 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=119953 infer_calls=119953 feedback_calls=119953 infer_wait=734.943s infer_obs_pack=16.130s env_step=94.069s feedback_total=70.747s feedback_obs_pack=18.314s feedback_info_pack=2.781s effective_fps=130.97 +2026-09-28 18:41:31.866 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=124514 infer_calls=124514 feedback_calls=124514 infer_wait=758.356s infer_obs_pack=16.585s env_step=96.747s feedback_total=72.835s feedback_obs_pack=18.852s feedback_info_pack=2.862s effective_fps=131.83 +2026-09-28 18:42:01.868 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=129691 infer_calls=129691 feedback_calls=129691 infer_wait=781.894s infer_obs_pack=17.041s env_step=99.360s feedback_total=74.918s feedback_obs_pack=19.380s feedback_info_pack=2.942s effective_fps=133.26 +2026-09-28 18:42:31.868 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=134723 infer_calls=134723 feedback_calls=134723 infer_wait=805.809s infer_obs_pack=17.474s env_step=101.768s feedback_total=76.921s feedback_obs_pack=19.888s feedback_info_pack=3.021s effective_fps=134.46 +2026-09-28 18:43:01.872 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=139402 infer_calls=139402 feedback_calls=139402 infer_wait=829.857s infer_obs_pack=17.898s env_step=104.190s feedback_total=78.835s feedback_obs_pack=20.375s feedback_info_pack=3.098s effective_fps=135.24 +2026-09-28 18:43:31.876 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=144054 infer_calls=144054 feedback_calls=144054 infer_wait=853.477s infer_obs_pack=18.333s env_step=106.797s feedback_total=80.859s feedback_obs_pack=20.889s feedback_info_pack=3.182s effective_fps=135.97 +2026-09-28 18:44:01.875 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=149120 infer_calls=149120 feedback_calls=149120 infer_wait=877.316s infer_obs_pack=18.770s env_step=109.272s feedback_total=82.847s feedback_obs_pack=21.394s feedback_info_pack=3.259s effective_fps=137.03 +2026-09-28 18:44:31.879 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=153882 infer_calls=153882 feedback_calls=153882 infer_wait=901.156s infer_obs_pack=19.192s env_step=111.764s feedback_total=84.819s feedback_obs_pack=21.910s feedback_info_pack=3.334s effective_fps=137.77 +2026-09-28 18:45:01.883 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=158948 infer_calls=158948 feedback_calls=158948 infer_wait=924.959s infer_obs_pack=19.637s env_step=114.255s feedback_total=86.817s feedback_obs_pack=22.415s feedback_info_pack=3.416s effective_fps=138.74 +2026-09-28 18:45:32.500 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163842 infer_calls=163842 feedback_calls=163842 infer_wait=949.367s infer_obs_pack=20.083s env_step=116.728s feedback_total=88.821s feedback_obs_pack=22.928s feedback_info_pack=3.496s effective_fps=139.44 +2026-09-28 18:46:02.503 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=168684 infer_calls=168684 feedback_calls=168684 infer_wait=972.900s infer_obs_pack=20.529s env_step=119.309s feedback_total=90.941s feedback_obs_pack=23.443s feedback_info_pack=3.576s effective_fps=140.14 +2026-09-28 18:46:32.505 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=173255 infer_calls=173255 feedback_calls=173255 infer_wait=996.354s infer_obs_pack=20.997s env_step=121.951s feedback_total=93.043s feedback_obs_pack=23.985s feedback_info_pack=3.656s effective_fps=140.59 +2026-09-28 18:47:02.505 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=178211 infer_calls=178211 feedback_calls=178211 infer_wait=1020.107s infer_obs_pack=21.442s env_step=124.441s feedback_total=95.072s feedback_obs_pack=24.495s feedback_info_pack=3.740s effective_fps=141.32 +2026-09-28 18:47:32.511 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=183316 infer_calls=183316 feedback_calls=183316 infer_wait=1044.040s infer_obs_pack=21.868s env_step=126.855s feedback_total=97.085s feedback_obs_pack=25.001s feedback_info_pack=3.819s effective_fps=142.12 +2026-09-28 18:48:02.513 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=188018 infer_calls=188018 feedback_calls=188018 infer_wait=1067.534s infer_obs_pack=22.322s env_step=129.432s feedback_total=99.237s feedback_obs_pack=25.568s feedback_info_pack=3.905s effective_fps=142.60 +2026-09-28 18:48:32.711 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=192514 infer_calls=192514 feedback_calls=192514 infer_wait=1091.414s infer_obs_pack=22.778s env_step=131.957s feedback_total=101.287s feedback_obs_pack=26.092s feedback_info_pack=3.984s effective_fps=142.87 +2026-09-28 18:49:02.714 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=197303 infer_calls=197303 feedback_calls=197303 infer_wait=1114.777s infer_obs_pack=23.240s env_step=134.637s feedback_total=103.418s feedback_obs_pack=26.628s feedback_info_pack=4.068s effective_fps=143.38 +2026-09-28 18:49:32.723 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=200860 infer_calls=200860 feedback_calls=200860 infer_wait=1137.608s infer_obs_pack=23.732s env_step=137.641s feedback_total=105.695s feedback_obs_pack=27.232s feedback_info_pack=4.174s effective_fps=142.99 +2026-09-28 18:50:02.724 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=204667 infer_calls=204667 feedback_calls=204667 infer_wait=1159.713s infer_obs_pack=24.307s env_step=140.939s feedback_total=108.131s feedback_obs_pack=27.851s feedback_info_pack=4.260s effective_fps=142.82 +2026-09-28 18:50:32.730 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=208420 infer_calls=208420 feedback_calls=208420 infer_wait=1183.802s infer_obs_pack=24.726s env_step=143.341s feedback_total=110.002s feedback_obs_pack=28.321s feedback_info_pack=4.331s effective_fps=142.57 +2026-09-28 18:51:02.740 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=212949 infer_calls=212949 feedback_calls=212949 infer_wait=1206.639s infer_obs_pack=25.248s env_step=146.267s feedback_total=112.243s feedback_obs_pack=28.921s feedback_info_pack=4.422s effective_fps=142.88 +2026-09-28 18:51:32.741 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=217817 infer_calls=217817 feedback_calls=217817 infer_wait=1230.366s infer_obs_pack=25.710s env_step=148.741s feedback_total=114.293s feedback_obs_pack=29.437s feedback_info_pack=4.505s effective_fps=143.38 +2026-09-28 18:52:02.745 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=223129 infer_calls=223129 feedback_calls=223129 infer_wait=1254.125s infer_obs_pack=26.155s env_step=151.226s feedback_total=116.331s feedback_obs_pack=29.962s feedback_info_pack=4.585s effective_fps=144.16 +2026-09-28 18:52:32.746 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=227984 infer_calls=227984 feedback_calls=227984 infer_wait=1277.577s infer_obs_pack=26.610s env_step=153.848s feedback_total=118.450s feedback_obs_pack=30.513s feedback_info_pack=4.667s effective_fps=144.62 +2026-09-28 18:53:02.750 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=232360 infer_calls=232360 feedback_calls=232360 infer_wait=1300.627s infer_obs_pack=27.109s env_step=156.665s feedback_total=120.665s feedback_obs_pack=31.071s feedback_info_pack=4.765s effective_fps=144.77 +2026-09-28 18:53:32.752 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=237005 infer_calls=237005 feedback_calls=237005 infer_wait=1323.853s infer_obs_pack=27.575s env_step=159.397s feedback_total=122.833s feedback_obs_pack=31.625s feedback_info_pack=4.851s effective_fps=145.08 +2026-09-28 18:54:03.572 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=241666 infer_calls=241666 feedback_calls=241666 infer_wait=1348.066s infer_obs_pack=28.039s env_step=162.069s feedback_total=124.932s feedback_obs_pack=32.164s feedback_info_pack=4.930s effective_fps=145.31 +2026-09-28 18:54:33.573 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=246325 infer_calls=246325 feedback_calls=246325 infer_wait=1372.102s infer_obs_pack=28.460s env_step=164.456s feedback_total=126.865s feedback_obs_pack=32.651s feedback_info_pack=5.002s effective_fps=145.59 +2026-09-28 18:55:03.588 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=250879 infer_calls=250879 feedback_calls=250879 infer_wait=1395.268s infer_obs_pack=28.940s env_step=167.235s feedback_total=129.034s feedback_obs_pack=33.210s feedback_info_pack=5.086s effective_fps=145.82 +2026-09-28 18:55:33.590 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=255145 infer_calls=255145 feedback_calls=255145 infer_wait=1418.913s infer_obs_pack=29.393s env_step=169.798s feedback_total=131.058s feedback_obs_pack=33.734s feedback_info_pack=5.163s effective_fps=145.87 +2026-09-28 18:56:03.590 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=258845 infer_calls=258845 feedback_calls=258845 infer_wait=1442.234s infer_obs_pack=29.805s env_step=172.596s feedback_total=133.122s feedback_obs_pack=34.296s feedback_info_pack=5.251s effective_fps=145.60 +2026-09-28 18:56:33.592 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=263137 infer_calls=263137 feedback_calls=263137 infer_wait=1465.147s infer_obs_pack=30.307s env_step=175.421s feedback_total=135.410s feedback_obs_pack=34.887s feedback_info_pack=5.349s effective_fps=145.68 +2026-09-28 18:57:03.596 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=267642 infer_calls=267642 feedback_calls=267642 infer_wait=1488.445s infer_obs_pack=30.776s env_step=178.113s feedback_total=137.563s feedback_obs_pack=35.447s feedback_info_pack=5.431s effective_fps=145.86 +2026-09-28 18:57:33.598 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=272832 infer_calls=272832 feedback_calls=272832 infer_wait=1512.185s infer_obs_pack=31.244s env_step=180.591s feedback_total=139.595s feedback_obs_pack=35.964s feedback_info_pack=5.511s effective_fps=146.40 +2026-09-28 18:58:03.600 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=276364 infer_calls=276364 feedback_calls=276364 infer_wait=1535.945s infer_obs_pack=31.702s env_step=183.173s feedback_total=141.586s feedback_obs_pack=36.454s feedback_info_pack=5.583s effective_fps=146.04 +2026-09-28 18:58:33.601 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=279124 infer_calls=279124 feedback_calls=279124 infer_wait=1558.227s infer_obs_pack=32.242s env_step=186.507s feedback_total=143.953s feedback_obs_pack=37.089s feedback_info_pack=5.687s effective_fps=145.31 +2026-09-28 18:59:03.625 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=283321 infer_calls=283321 feedback_calls=283321 infer_wait=1581.163s infer_obs_pack=32.778s env_step=189.319s feedback_total=146.263s feedback_obs_pack=37.709s feedback_info_pack=5.773s effective_fps=145.33 +2026-09-28 18:59:33.627 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=287540 infer_calls=287540 feedback_calls=287540 infer_wait=1604.005s infer_obs_pack=33.283s env_step=192.213s feedback_total=148.571s feedback_obs_pack=38.331s feedback_info_pack=5.863s effective_fps=145.36 +2026-09-28 19:00:03.630 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=291966 infer_calls=291966 feedback_calls=291966 infer_wait=1627.097s infer_obs_pack=33.778s env_step=194.961s feedback_total=150.807s feedback_obs_pack=38.917s feedback_info_pack=5.946s effective_fps=145.50 +2026-09-28 19:00:33.631 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=296656 infer_calls=296656 feedback_calls=296656 infer_wait=1650.426s infer_obs_pack=34.256s env_step=197.618s feedback_total=152.960s feedback_obs_pack=39.480s feedback_info_pack=6.028s effective_fps=145.76 +2026-09-28 19:01:03.635 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=301432 infer_calls=301432 feedback_calls=301432 infer_wait=1674.217s infer_obs_pack=34.692s env_step=200.101s feedback_total=154.988s feedback_obs_pack=39.998s feedback_info_pack=6.106s effective_fps=146.04 +2026-09-28 19:01:33.636 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=305331 infer_calls=305331 feedback_calls=305331 infer_wait=1696.800s infer_obs_pack=35.230s env_step=203.105s feedback_total=157.356s feedback_obs_pack=40.639s feedback_info_pack=6.190s effective_fps=145.92 +2026-09-28 19:02:03.638 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=309319 infer_calls=309319 feedback_calls=309319 infer_wait=1719.627s infer_obs_pack=35.781s env_step=205.937s feedback_total=159.674s feedback_obs_pack=41.226s feedback_info_pack=6.299s effective_fps=145.84 +2026-09-28 19:02:33.642 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=313566 infer_calls=313566 feedback_calls=313566 infer_wait=1742.903s infer_obs_pack=36.259s env_step=208.584s feedback_total=161.878s feedback_obs_pack=41.800s feedback_info_pack=6.383s effective_fps=145.87 +2026-09-28 19:03:03.646 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=317873 infer_calls=317873 feedback_calls=317873 infer_wait=1766.209s infer_obs_pack=36.733s env_step=211.249s feedback_total=164.050s feedback_obs_pack=42.362s feedback_info_pack=6.466s effective_fps=145.93 +2026-09-28 19:03:33.650 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=320871 infer_calls=320871 feedback_calls=320871 infer_wait=1789.378s infer_obs_pack=37.245s env_step=214.141s feedback_total=166.149s feedback_obs_pack=42.916s feedback_info_pack=6.542s effective_fps=145.39 +2026-09-28 19:04:03.653 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=324588 infer_calls=324588 feedback_calls=324588 infer_wait=1812.985s infer_obs_pack=37.693s env_step=216.771s feedback_total=168.161s feedback_obs_pack=43.437s feedback_info_pack=6.630s effective_fps=145.19 +2026-09-28 19:04:33.661 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=328882 infer_calls=328882 feedback_calls=328882 infer_wait=1836.992s infer_obs_pack=38.117s env_step=219.181s feedback_total=170.075s feedback_obs_pack=43.927s feedback_info_pack=6.712s effective_fps=145.24 +2026-09-28 19:05:03.662 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=333054 infer_calls=333054 feedback_calls=333054 infer_wait=1860.861s infer_obs_pack=38.549s env_step=221.626s feedback_total=172.046s feedback_obs_pack=44.435s feedback_info_pack=6.786s effective_fps=145.24 +2026-09-28 19:05:33.666 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=337255 infer_calls=337255 feedback_calls=337255 infer_wait=1884.327s infer_obs_pack=39.023s env_step=224.227s feedback_total=174.126s feedback_obs_pack=44.985s feedback_info_pack=6.867s effective_fps=145.26 +2026-09-28 19:06:03.673 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=340939 infer_calls=340939 feedback_calls=340939 infer_wait=1907.529s infer_obs_pack=39.553s env_step=226.975s feedback_total=176.283s feedback_obs_pack=45.564s feedback_info_pack=6.948s effective_fps=145.06 +2026-09-28 19:06:33.674 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=344625 infer_calls=344625 feedback_calls=344625 infer_wait=1931.075s infer_obs_pack=40.025s env_step=229.440s feedback_total=178.493s feedback_obs_pack=46.143s feedback_info_pack=7.049s effective_fps=144.86 +2026-09-28 19:07:03.675 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=348425 infer_calls=348425 feedback_calls=348425 infer_wait=1954.578s infer_obs_pack=40.508s env_step=232.021s feedback_total=180.618s feedback_obs_pack=46.738s feedback_info_pack=7.134s effective_fps=144.71 +2026-09-28 19:07:33.682 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=350399 infer_calls=350399 feedback_calls=350399 infer_wait=1977.158s infer_obs_pack=41.075s env_step=235.061s feedback_total=183.004s feedback_obs_pack=47.411s feedback_info_pack=7.215s effective_fps=143.82 +2026-09-28 19:08:03.704 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=351912 infer_calls=351912 feedback_calls=351912 infer_wait=2000.115s infer_obs_pack=41.581s env_step=238.013s feedback_total=185.341s feedback_obs_pack=48.030s feedback_info_pack=7.317s effective_fps=142.76 +2026-09-28 19:08:33.715 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=353504 infer_calls=353504 feedback_calls=353504 infer_wait=2022.357s infer_obs_pack=42.109s env_step=241.446s feedback_total=187.689s feedback_obs_pack=48.721s feedback_info_pack=7.400s effective_fps=141.76 +2026-09-28 19:09:03.719 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=355829 infer_calls=355829 feedback_calls=355829 infer_wait=2043.835s infer_obs_pack=42.740s env_step=245.093s feedback_total=190.267s feedback_obs_pack=49.453s feedback_info_pack=7.508s effective_fps=141.09 +2026-09-28 19:09:33.719 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=358417 infer_calls=358417 feedback_calls=358417 infer_wait=2065.587s infer_obs_pack=43.356s env_step=248.620s feedback_total=192.712s feedback_obs_pack=50.121s feedback_info_pack=7.601s effective_fps=140.54 +2026-09-28 19:10:03.721 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=360838 infer_calls=360838 feedback_calls=360838 infer_wait=2087.031s infer_obs_pack=43.968s env_step=252.297s feedback_total=195.332s feedback_obs_pack=50.807s feedback_info_pack=7.694s effective_fps=139.93 +2026-09-28 19:10:33.722 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=362503 infer_calls=362503 feedback_calls=362503 infer_wait=2108.964s infer_obs_pack=44.563s env_step=255.789s feedback_total=197.844s feedback_obs_pack=51.484s feedback_info_pack=7.771s effective_fps=139.04 +2026-09-28 19:11:03.734 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=364087 infer_calls=364087 feedback_calls=364087 infer_wait=2131.168s infer_obs_pack=45.219s env_step=259.039s feedback_total=200.302s feedback_obs_pack=52.201s feedback_info_pack=7.863s effective_fps=138.14 +2026-09-28 19:11:33.752 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=366926 infer_calls=366926 feedback_calls=366926 infer_wait=2154.115s infer_obs_pack=45.730s env_step=261.974s feedback_total=202.484s feedback_obs_pack=52.809s feedback_info_pack=7.967s effective_fps=137.72 +2026-09-28 19:12:03.754 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=370015 infer_calls=370015 feedback_calls=370015 infer_wait=2177.842s infer_obs_pack=46.168s env_step=264.568s feedback_total=204.460s feedback_obs_pack=53.327s feedback_info_pack=8.039s effective_fps=137.40 +2026-09-28 19:12:33.757 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=373463 infer_calls=373463 feedback_calls=373463 infer_wait=2201.364s infer_obs_pack=46.644s env_step=267.208s feedback_total=206.510s feedback_obs_pack=53.850s feedback_info_pack=8.117s effective_fps=137.22 +2026-09-28 19:13:03.761 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=376390 infer_calls=376390 feedback_calls=376390 infer_wait=2223.722s infer_obs_pack=47.146s env_step=270.369s feedback_total=209.000s feedback_obs_pack=54.506s feedback_info_pack=8.229s effective_fps=136.86 +2026-09-28 19:13:33.768 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=379215 infer_calls=379215 feedback_calls=379215 infer_wait=2246.354s infer_obs_pack=47.685s env_step=273.536s feedback_total=211.248s feedback_obs_pack=55.079s feedback_info_pack=8.322s effective_fps=136.47 +2026-09-28 19:14:03.772 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=382232 infer_calls=382232 feedback_calls=382232 infer_wait=2268.851s infer_obs_pack=48.246s env_step=276.700s feedback_total=213.565s feedback_obs_pack=55.658s feedback_info_pack=8.416s effective_fps=136.15 +2026-09-28 19:14:34.613 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=385025 infer_calls=385025 feedback_calls=385025 infer_wait=2291.878s infer_obs_pack=48.842s env_step=279.933s feedback_total=216.062s feedback_obs_pack=56.299s feedback_info_pack=8.505s effective_fps=135.73 +2026-09-28 19:15:04.613 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=388817 infer_calls=388817 feedback_calls=388817 infer_wait=2314.009s infer_obs_pack=49.428s env_step=283.139s feedback_total=218.552s feedback_obs_pack=56.972s feedback_info_pack=8.598s effective_fps=135.71 +2026-09-28 19:15:34.616 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=391371 infer_calls=391371 feedback_calls=391371 infer_wait=2337.056s infer_obs_pack=49.940s env_step=286.057s feedback_total=220.689s feedback_obs_pack=57.514s feedback_info_pack=8.694s effective_fps=135.25 +2026-09-28 19:16:04.616 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=395703 infer_calls=395703 feedback_calls=395703 infer_wait=2360.675s infer_obs_pack=50.384s env_step=288.577s feedback_total=222.817s feedback_obs_pack=58.028s feedback_info_pack=8.770s effective_fps=135.40 +2026-09-28 19:16:34.617 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=400136 infer_calls=400136 feedback_calls=400136 infer_wait=2384.198s infer_obs_pack=50.837s env_step=291.184s feedback_total=224.901s feedback_obs_pack=58.540s feedback_info_pack=8.852s effective_fps=135.59 +2026-09-28 19:17:04.618 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=404632 infer_calls=404632 feedback_calls=404632 infer_wait=2408.046s infer_obs_pack=51.294s env_step=293.621s feedback_total=226.898s feedback_obs_pack=59.060s feedback_info_pack=8.930s effective_fps=135.79 +2026-09-28 19:17:34.621 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=407629 infer_calls=407629 feedback_calls=407629 infer_wait=2431.178s infer_obs_pack=51.762s env_step=296.380s feedback_total=229.249s feedback_obs_pack=59.717s feedback_info_pack=9.022s effective_fps=135.49 +2026-09-28 19:17:59.065 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=2448.158s infer_obs_pack=52.213s env_step=298.774s feedback_total=230.968s feedback_obs_pack=60.168s feedback_info_pack=9.081s effective_fps=135.18 +2026-09-28 19:17:59.065 | INFO  | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed4.log b/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed4.log new file mode 100644 index 0000000..e13e20e --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed4.log @@ -0,0 +1,116 @@ +2026-09-28 18:25:05.293 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 18:25:05.296 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 18:25:05.301 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 18:25:05.306 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 18:25:05.312 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 18:25:05.497 | INFO  | __main__:main:83 - Starting env client exp_name=e43-cross-seed4 output_dir=runs\e43-cross-seed4 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 18:25:05.653 | INFO  | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://100.75.226.89:9794... +2026-09-28 18:25:05.680 | INFO  | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 18:25:35.689 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=4725 infer_calls=4725 feedback_calls=4725 infer_wait=23.684s infer_obs_pack=0.455s env_step=2.528s feedback_total=2.067s feedback_obs_pack=0.525s feedback_info_pack=0.082s effective_fps=164.43 +2026-09-28 18:26:05.691 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=8827 infer_calls=8827 feedback_calls=8827 infer_wait=46.875s infer_obs_pack=0.937s env_step=5.260s feedback_total=4.264s feedback_obs_pack=1.082s feedback_info_pack=0.166s effective_fps=153.95 +2026-09-28 18:26:35.700 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=12392 infer_calls=12392 feedback_calls=12392 infer_wait=69.719s infer_obs_pack=1.431s env_step=8.243s feedback_total=6.505s feedback_obs_pack=1.657s feedback_info_pack=0.263s effective_fps=144.26 +2026-09-28 18:27:05.710 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=16557 infer_calls=16557 feedback_calls=16557 infer_wait=92.829s infer_obs_pack=2.001s env_step=11.039s feedback_total=8.660s feedback_obs_pack=2.201s feedback_info_pack=0.350s effective_fps=144.57 +2026-09-28 18:27:35.718 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=18960 infer_calls=18960 feedback_calls=18960 infer_wait=115.330s infer_obs_pack=2.622s env_step=14.095s feedback_total=11.038s feedback_obs_pack=2.840s feedback_info_pack=0.479s effective_fps=132.51 +2026-09-28 18:28:05.773 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=22596 infer_calls=22596 feedback_calls=22596 infer_wait=137.881s infer_obs_pack=3.174s env_step=17.177s feedback_total=13.386s feedback_obs_pack=3.450s feedback_info_pack=0.586s effective_fps=131.66 +2026-09-28 18:28:35.777 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=26722 infer_calls=26722 feedback_calls=26722 infer_wait=161.016s infer_obs_pack=3.653s env_step=20.014s feedback_total=15.555s feedback_obs_pack=4.008s feedback_info_pack=0.686s effective_fps=133.45 +2026-09-28 18:29:05.781 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=30459 infer_calls=30459 feedback_calls=30459 infer_wait=184.089s infer_obs_pack=4.128s env_step=22.859s feedback_total=17.801s feedback_obs_pack=4.589s feedback_info_pack=0.781s effective_fps=133.08 +2026-09-28 18:29:35.783 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=34649 infer_calls=34649 feedback_calls=34649 infer_wait=207.064s infer_obs_pack=4.631s env_step=25.734s feedback_total=20.075s feedback_obs_pack=5.156s feedback_info_pack=0.891s effective_fps=134.56 +2026-09-28 18:30:05.839 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=38039 infer_calls=38039 feedback_calls=38039 infer_wait=229.736s infer_obs_pack=5.114s env_step=28.892s feedback_total=22.365s feedback_obs_pack=5.756s feedback_info_pack=0.985s effective_fps=132.95 +2026-09-28 18:30:35.838 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41714 infer_calls=41714 feedback_calls=41714 infer_wait=252.397s infer_obs_pack=5.619s env_step=31.911s feedback_total=24.703s feedback_obs_pack=6.350s feedback_info_pack=1.096s effective_fps=132.58 +2026-09-28 18:31:05.851 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=44376 infer_calls=44376 feedback_calls=44376 infer_wait=275.067s infer_obs_pack=6.218s env_step=35.010s feedback_total=27.036s feedback_obs_pack=6.923s feedback_info_pack=1.209s effective_fps=129.25 +2026-09-28 18:31:35.855 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=47333 infer_calls=47333 feedback_calls=47333 infer_wait=297.431s infer_obs_pack=6.752s env_step=38.271s feedback_total=29.397s feedback_obs_pack=7.557s feedback_info_pack=1.307s effective_fps=127.29 +2026-09-28 18:32:05.872 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=50211 infer_calls=50211 feedback_calls=50211 infer_wait=320.203s infer_obs_pack=7.298s env_step=41.249s feedback_total=31.671s feedback_obs_pack=8.151s feedback_info_pack=1.395s effective_fps=125.40 +2026-09-28 18:32:35.878 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=53868 infer_calls=53868 feedback_calls=53868 infer_wait=343.573s infer_obs_pack=7.781s env_step=43.994s feedback_total=33.770s feedback_obs_pack=8.739s feedback_info_pack=1.487s effective_fps=125.53 +2026-09-28 18:33:05.884 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=57691 infer_calls=57691 feedback_calls=57691 infer_wait=366.155s infer_obs_pack=8.323s env_step=47.067s feedback_total=36.087s feedback_obs_pack=9.354s feedback_info_pack=1.584s effective_fps=126.06 +2026-09-28 18:33:35.891 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=61843 infer_calls=61843 feedback_calls=61843 infer_wait=389.343s infer_obs_pack=8.800s env_step=49.825s feedback_total=38.295s feedback_obs_pack=9.921s feedback_info_pack=1.683s effective_fps=127.18 +2026-09-28 18:34:05.894 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=66831 infer_calls=66831 feedback_calls=66831 infer_wait=412.609s infer_obs_pack=9.275s env_step=52.557s feedback_total=40.473s feedback_obs_pack=10.469s feedback_info_pack=1.779s effective_fps=129.79 +2026-09-28 18:34:35.896 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=70715 infer_calls=70715 feedback_calls=70715 infer_wait=435.755s infer_obs_pack=9.786s env_step=55.425s feedback_total=42.610s feedback_obs_pack=10.998s feedback_info_pack=1.873s effective_fps=130.09 +2026-09-28 18:35:05.897 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=75013 infer_calls=75013 feedback_calls=75013 infer_wait=458.919s infer_obs_pack=10.304s env_step=58.207s feedback_total=44.768s feedback_obs_pack=11.546s feedback_info_pack=1.960s effective_fps=131.10 +2026-09-28 18:35:35.912 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=78668 infer_calls=78668 feedback_calls=78668 infer_wait=481.478s infer_obs_pack=10.829s env_step=61.206s feedback_total=47.122s feedback_obs_pack=12.144s feedback_info_pack=2.069s effective_fps=130.97 +2026-09-28 18:36:05.917 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=82206 infer_calls=82206 feedback_calls=82206 infer_wait=504.000s infer_obs_pack=11.357s env_step=64.261s feedback_total=49.546s feedback_obs_pack=12.782s feedback_info_pack=2.165s effective_fps=130.66 +2026-09-28 18:36:35.921 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=86114 infer_calls=86114 feedback_calls=86114 infer_wait=527.066s infer_obs_pack=11.825s env_step=67.096s feedback_total=51.769s feedback_obs_pack=13.381s feedback_info_pack=2.272s effective_fps=130.92 +2026-09-28 18:37:06.078 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=90114 infer_calls=90114 feedback_calls=90114 infer_wait=550.658s infer_obs_pack=12.257s env_step=69.818s feedback_total=53.894s feedback_obs_pack=13.947s feedback_info_pack=2.370s effective_fps=131.24 +2026-09-28 18:37:36.083 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=93397 infer_calls=93397 feedback_calls=93397 infer_wait=573.271s infer_obs_pack=12.802s env_step=72.900s feedback_total=56.218s feedback_obs_pack=14.534s feedback_info_pack=2.465s effective_fps=130.59 +2026-09-28 18:38:06.085 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=95897 infer_calls=95897 feedback_calls=95897 infer_wait=596.257s infer_obs_pack=13.342s env_step=75.860s feedback_total=58.441s feedback_obs_pack=15.132s feedback_info_pack=2.571s effective_fps=128.91 +2026-09-28 18:38:36.097 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=99749 infer_calls=99749 feedback_calls=99749 infer_wait=618.947s infer_obs_pack=13.858s env_step=78.846s feedback_total=60.768s feedback_obs_pack=15.739s feedback_info_pack=2.689s effective_fps=129.14 +2026-09-28 18:39:06.100 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=103201 infer_calls=103201 feedback_calls=103201 infer_wait=642.074s infer_obs_pack=14.323s env_step=81.715s feedback_total=62.942s feedback_obs_pack=16.298s feedback_info_pack=2.775s effective_fps=128.83 +2026-09-28 18:39:36.103 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=107176 infer_calls=107176 feedback_calls=107176 infer_wait=665.484s infer_obs_pack=14.794s env_step=84.375s feedback_total=65.049s feedback_obs_pack=16.835s feedback_info_pack=2.867s effective_fps=129.17 +2026-09-28 18:40:06.107 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=111594 infer_calls=111594 feedback_calls=111594 infer_wait=688.943s infer_obs_pack=15.231s env_step=87.102s feedback_total=67.100s feedback_obs_pack=17.346s feedback_info_pack=2.959s effective_fps=130.01 +2026-09-28 18:40:36.109 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=115529 infer_calls=115529 feedback_calls=115529 infer_wait=712.256s infer_obs_pack=15.681s env_step=89.784s feedback_total=69.292s feedback_obs_pack=17.880s feedback_info_pack=3.051s effective_fps=130.24 +2026-09-28 18:41:06.111 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=119792 infer_calls=119792 feedback_calls=119792 infer_wait=735.309s infer_obs_pack=16.182s env_step=92.587s feedback_total=71.513s feedback_obs_pack=18.471s feedback_info_pack=3.161s effective_fps=130.84 +2026-09-28 18:41:36.117 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=124398 infer_calls=124398 feedback_calls=124398 infer_wait=758.739s infer_obs_pack=16.632s env_step=95.274s feedback_total=73.600s feedback_obs_pack=19.006s feedback_info_pack=3.258s effective_fps=131.74 +2026-09-28 18:42:06.120 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=129575 infer_calls=129575 feedback_calls=129575 infer_wait=782.311s infer_obs_pack=17.086s env_step=97.846s feedback_total=75.699s feedback_obs_pack=19.537s feedback_info_pack=3.349s effective_fps=133.18 +2026-09-28 18:42:36.124 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=134532 infer_calls=134532 feedback_calls=134532 infer_wait=806.165s infer_obs_pack=17.528s env_step=100.297s feedback_total=77.717s feedback_obs_pack=20.049s feedback_info_pack=3.439s effective_fps=134.30 +2026-09-28 18:43:07.751 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=139266 infer_calls=139266 feedback_calls=139266 infer_wait=831.675s infer_obs_pack=17.949s env_step=102.770s feedback_total=79.713s feedback_obs_pack=20.544s feedback_info_pack=3.529s effective_fps=134.93 +2026-09-28 18:43:37.754 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=144155 infer_calls=144155 feedback_calls=144155 infer_wait=855.221s infer_obs_pack=18.396s env_step=105.370s feedback_total=81.785s feedback_obs_pack=21.066s feedback_info_pack=3.620s effective_fps=135.90 +2026-09-28 18:44:07.756 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=149406 infer_calls=149406 feedback_calls=149406 infer_wait=879.127s infer_obs_pack=18.835s env_step=107.835s feedback_total=83.740s feedback_obs_pack=21.561s feedback_info_pack=3.695s effective_fps=137.13 +2026-09-28 18:44:37.758 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=154083 infer_calls=154083 feedback_calls=154083 infer_wait=903.199s infer_obs_pack=19.238s env_step=110.258s feedback_total=85.645s feedback_obs_pack=22.036s feedback_info_pack=3.782s effective_fps=137.78 +2026-09-28 18:45:07.759 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=159123 infer_calls=159123 feedback_calls=159123 infer_wait=926.907s infer_obs_pack=19.672s env_step=112.802s feedback_total=87.673s feedback_obs_pack=22.553s feedback_info_pack=3.872s effective_fps=138.72 +2026-09-28 18:45:37.762 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163851 infer_calls=163851 feedback_calls=163851 infer_wait=950.737s infer_obs_pack=20.103s env_step=115.301s feedback_total=89.672s feedback_obs_pack=23.058s feedback_info_pack=3.961s effective_fps=139.35 +2026-09-28 18:46:07.763 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=168581 infer_calls=168581 feedback_calls=168581 infer_wait=974.161s infer_obs_pack=20.545s env_step=117.985s feedback_total=91.793s feedback_obs_pack=23.589s feedback_info_pack=4.055s effective_fps=139.96 +2026-09-28 18:46:37.763 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=173146 infer_calls=173146 feedback_calls=173146 infer_wait=997.676s infer_obs_pack=20.990s env_step=120.594s feedback_total=93.911s feedback_obs_pack=24.149s feedback_info_pack=4.150s effective_fps=140.41 +2026-09-28 18:47:07.767 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=178010 infer_calls=178010 feedback_calls=178010 infer_wait=1021.476s infer_obs_pack=21.434s env_step=123.084s feedback_total=95.920s feedback_obs_pack=24.654s feedback_info_pack=4.246s effective_fps=141.06 +2026-09-28 18:47:37.771 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=183047 infer_calls=183047 feedback_calls=183047 infer_wait=1045.454s infer_obs_pack=21.869s env_step=125.506s feedback_total=97.894s feedback_obs_pack=25.150s feedback_info_pack=4.336s effective_fps=141.82 +2026-09-28 18:48:07.772 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=187497 infer_calls=187497 feedback_calls=187497 infer_wait=1068.872s infer_obs_pack=22.348s env_step=128.150s feedback_total=100.053s feedback_obs_pack=25.702s feedback_info_pack=4.437s effective_fps=142.11 +2026-09-28 18:48:37.775 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=192447 infer_calls=192447 feedback_calls=192447 infer_wait=1092.366s infer_obs_pack=22.821s env_step=130.759s feedback_total=102.201s feedback_obs_pack=26.259s feedback_info_pack=4.536s effective_fps=142.75 +2026-09-28 18:49:08.428 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=196610 infer_calls=196610 feedback_calls=196610 infer_wait=1116.726s infer_obs_pack=23.264s env_step=133.315s feedback_total=104.188s feedback_obs_pack=26.768s feedback_info_pack=4.621s effective_fps=142.73 +2026-09-28 18:49:38.428 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=200185 infer_calls=200185 feedback_calls=200185 infer_wait=1138.958s infer_obs_pack=23.846s env_step=136.524s feedback_total=106.636s feedback_obs_pack=27.403s feedback_info_pack=4.738s effective_fps=142.38 +2026-09-28 18:50:08.434 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=204184 infer_calls=204184 feedback_calls=204184 infer_wait=1161.825s infer_obs_pack=24.353s env_step=139.507s feedback_total=108.869s feedback_obs_pack=27.984s feedback_info_pack=4.837s effective_fps=142.33 +2026-09-28 18:50:38.438 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=207861 infer_calls=207861 feedback_calls=207861 infer_wait=1185.880s infer_obs_pack=24.769s env_step=141.885s feedback_total=110.772s feedback_obs_pack=28.483s feedback_info_pack=4.925s effective_fps=142.05 +2026-09-28 18:51:08.441 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=212398 infer_calls=212398 feedback_calls=212398 infer_wait=1208.763s infer_obs_pack=25.274s env_step=144.783s feedback_total=113.034s feedback_obs_pack=29.067s feedback_info_pack=5.032s effective_fps=142.37 +2026-09-28 18:51:38.447 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=217277 infer_calls=217277 feedback_calls=217277 infer_wait=1232.806s infer_obs_pack=25.711s env_step=147.133s feedback_total=115.005s feedback_obs_pack=29.570s feedback_info_pack=5.121s effective_fps=142.88 +2026-09-28 18:52:08.452 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=222236 infer_calls=222236 feedback_calls=222236 infer_wait=1256.428s infer_obs_pack=26.160s env_step=149.683s feedback_total=117.062s feedback_obs_pack=30.086s feedback_info_pack=5.214s effective_fps=143.44 +2026-09-28 18:52:38.458 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=226954 infer_calls=226954 feedback_calls=226954 infer_wait=1280.161s infer_obs_pack=26.591s env_step=152.246s feedback_total=119.071s feedback_obs_pack=30.590s feedback_info_pack=5.306s effective_fps=143.82 +2026-09-28 18:53:08.463 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=231131 infer_calls=231131 feedback_calls=231131 infer_wait=1303.178s infer_obs_pack=27.050s env_step=155.165s feedback_total=121.254s feedback_obs_pack=31.148s feedback_info_pack=5.407s effective_fps=143.86 +2026-09-28 18:53:38.464 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=236036 infer_calls=236036 feedback_calls=236036 infer_wait=1326.551s infer_obs_pack=27.508s env_step=157.824s feedback_total=123.391s feedback_obs_pack=31.713s feedback_info_pack=5.504s effective_fps=144.34 +2026-09-28 18:54:08.468 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=240592 infer_calls=240592 feedback_calls=240592 infer_wait=1349.804s infer_obs_pack=27.965s env_step=160.660s feedback_total=125.487s feedback_obs_pack=32.259s feedback_info_pack=5.597s effective_fps=144.59 +2026-09-28 18:54:38.474 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=245269 infer_calls=245269 feedback_calls=245269 infer_wait=1373.897s infer_obs_pack=28.375s env_step=163.013s feedback_total=127.423s feedback_obs_pack=32.753s feedback_info_pack=5.684s effective_fps=144.90 +2026-09-28 18:55:08.476 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=249576 infer_calls=249576 feedback_calls=249576 infer_wait=1396.925s infer_obs_pack=28.881s env_step=165.885s feedback_total=129.608s feedback_obs_pack=33.316s feedback_info_pack=5.783s effective_fps=144.99 +2026-09-28 18:55:38.476 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=253725 infer_calls=253725 feedback_calls=253725 infer_wait=1420.728s infer_obs_pack=29.296s env_step=168.487s feedback_total=131.573s feedback_obs_pack=33.810s feedback_info_pack=5.871s effective_fps=144.98 +2026-09-28 18:56:08.482 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=257262 infer_calls=257262 feedback_calls=257262 infer_wait=1444.055s infer_obs_pack=29.750s env_step=171.201s feedback_total=133.805s feedback_obs_pack=34.366s feedback_info_pack=5.997s effective_fps=144.63 +2026-09-28 18:56:38.491 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=261561 infer_calls=261561 feedback_calls=261561 infer_wait=1466.990s infer_obs_pack=30.253s env_step=174.067s feedback_total=136.077s feedback_obs_pack=34.984s feedback_info_pack=6.103s effective_fps=144.72 +2026-09-28 18:57:08.495 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=266072 infer_calls=266072 feedback_calls=266072 infer_wait=1490.338s infer_obs_pack=30.715s env_step=176.829s feedback_total=138.175s feedback_obs_pack=35.535s feedback_info_pack=6.200s effective_fps=144.91 +2026-09-28 18:57:38.500 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=270709 infer_calls=270709 feedback_calls=270709 infer_wait=1514.486s infer_obs_pack=31.138s env_step=179.171s feedback_total=140.087s feedback_obs_pack=36.022s feedback_info_pack=6.290s effective_fps=145.16 +2026-09-28 18:58:08.530 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=273754 infer_calls=273754 feedback_calls=273754 infer_wait=1537.727s infer_obs_pack=31.686s env_step=181.955s feedback_total=142.207s feedback_obs_pack=36.596s feedback_info_pack=6.386s effective_fps=144.57 +2026-09-28 18:58:38.534 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=277048 infer_calls=277048 feedback_calls=277048 infer_wait=1560.076s infer_obs_pack=32.224s env_step=185.238s feedback_total=144.595s feedback_obs_pack=37.254s feedback_info_pack=6.495s effective_fps=144.14 +2026-09-28 18:59:08.535 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=280730 infer_calls=280730 feedback_calls=280730 infer_wait=1582.904s infer_obs_pack=32.768s env_step=188.056s feedback_total=146.892s feedback_obs_pack=37.838s feedback_info_pack=6.610s effective_fps=143.92 +2026-09-28 18:59:38.543 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=285053 infer_calls=285053 feedback_calls=285053 infer_wait=1605.812s infer_obs_pack=33.271s env_step=190.928s feedback_total=149.184s feedback_obs_pack=38.442s feedback_info_pack=6.717s effective_fps=144.02 +2026-09-28 19:00:08.545 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=289735 infer_calls=289735 feedback_calls=289735 infer_wait=1629.160s infer_obs_pack=33.754s env_step=193.633s feedback_total=151.311s feedback_obs_pack=39.003s feedback_info_pack=6.813s effective_fps=144.30 +2026-09-28 19:00:38.548 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=294379 infer_calls=294379 feedback_calls=294379 infer_wait=1652.667s infer_obs_pack=34.223s env_step=196.244s feedback_total=153.413s feedback_obs_pack=39.544s feedback_info_pack=6.907s effective_fps=144.55 +2026-09-28 19:01:08.552 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=298714 infer_calls=298714 feedback_calls=298714 infer_wait=1676.461s infer_obs_pack=34.675s env_step=198.710s feedback_total=155.429s feedback_obs_pack=40.091s feedback_info_pack=7.005s effective_fps=144.64 +2026-09-28 19:01:38.557 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=302752 infer_calls=302752 feedback_calls=302752 infer_wait=1699.239s infer_obs_pack=35.182s env_step=201.637s feedback_total=157.758s feedback_obs_pack=40.722s feedback_info_pack=7.109s effective_fps=144.59 +2026-09-28 19:02:08.562 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=306813 infer_calls=306813 feedback_calls=306813 infer_wait=1722.348s infer_obs_pack=35.693s env_step=204.373s feedback_total=159.983s feedback_obs_pack=41.288s feedback_info_pack=7.208s effective_fps=144.56 +2026-09-28 19:02:38.563 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=310929 infer_calls=310929 feedback_calls=310929 infer_wait=1745.718s infer_obs_pack=36.155s env_step=207.060s feedback_total=162.136s feedback_obs_pack=41.849s feedback_info_pack=7.306s effective_fps=144.55 +2026-09-28 19:03:08.564 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=315012 infer_calls=315012 feedback_calls=315012 infer_wait=1769.105s infer_obs_pack=36.641s env_step=209.741s feedback_total=164.218s feedback_obs_pack=42.384s feedback_info_pack=7.403s effective_fps=144.52 +2026-09-28 19:03:38.565 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=317765 infer_calls=317765 feedback_calls=317765 infer_wait=1792.383s infer_obs_pack=37.094s env_step=212.569s feedback_total=166.325s feedback_obs_pack=42.980s feedback_info_pack=7.502s effective_fps=143.89 +2026-09-28 19:04:08.564 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=321850 infer_calls=321850 feedback_calls=321850 infer_wait=1816.258s infer_obs_pack=37.513s env_step=215.109s feedback_total=168.224s feedback_obs_pack=43.466s feedback_info_pack=7.605s effective_fps=143.87 +2026-09-28 19:04:38.568 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=326097 infer_calls=326097 feedback_calls=326097 infer_wait=1840.285s infer_obs_pack=37.932s env_step=217.510s feedback_total=170.160s feedback_obs_pack=43.967s feedback_info_pack=7.689s effective_fps=143.92 +2026-09-28 19:05:08.569 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=329997 infer_calls=329997 feedback_calls=329997 infer_wait=1864.238s infer_obs_pack=38.353s env_step=219.976s feedback_total=172.085s feedback_obs_pack=44.465s feedback_info_pack=7.778s effective_fps=143.81 +2026-09-28 19:05:38.570 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=333863 infer_calls=333863 feedback_calls=333863 infer_wait=1887.654s infer_obs_pack=38.830s env_step=222.597s feedback_total=174.223s feedback_obs_pack=45.006s feedback_info_pack=7.871s effective_fps=143.70 +2026-09-28 19:06:08.573 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=337653 infer_calls=337653 feedback_calls=337653 infer_wait=1911.155s infer_obs_pack=39.316s env_step=225.159s feedback_total=176.375s feedback_obs_pack=45.558s feedback_info_pack=7.978s effective_fps=143.56 +2026-09-28 19:06:38.579 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=341311 infer_calls=341311 feedback_calls=341311 infer_wait=1934.723s infer_obs_pack=39.784s env_step=227.807s feedback_total=178.445s feedback_obs_pack=46.120s feedback_info_pack=8.063s effective_fps=143.36 +2026-09-28 19:07:08.587 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=344598 infer_calls=344598 feedback_calls=344598 infer_wait=1958.089s infer_obs_pack=40.309s env_step=230.554s feedback_total=180.601s feedback_obs_pack=46.678s feedback_info_pack=8.167s effective_fps=143.01 +2026-09-28 19:07:38.608 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=346519 infer_calls=346519 feedback_calls=346519 infer_wait=1980.729s infer_obs_pack=40.906s env_step=233.588s feedback_total=182.924s feedback_obs_pack=47.405s feedback_info_pack=8.252s effective_fps=142.12 +2026-09-28 19:08:08.623 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=347931 infer_calls=347931 feedback_calls=347931 infer_wait=2003.326s infer_obs_pack=41.393s env_step=236.921s feedback_total=185.199s feedback_obs_pack=48.039s feedback_info_pack=8.407s effective_fps=141.04 +2026-09-28 19:08:38.630 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=349589 infer_calls=349589 feedback_calls=349589 infer_wait=2025.370s infer_obs_pack=41.953s env_step=240.422s feedback_total=187.681s feedback_obs_pack=48.702s feedback_info_pack=8.500s effective_fps=140.09 +2026-09-28 19:09:08.637 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=352067 infer_calls=352067 feedback_calls=352067 infer_wait=2046.636s infer_obs_pack=42.563s env_step=244.222s feedback_total=190.288s feedback_obs_pack=49.389s feedback_info_pack=8.606s effective_fps=139.50 +2026-09-28 19:09:38.641 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=354573 infer_calls=354573 feedback_calls=354573 infer_wait=2068.220s infer_obs_pack=43.184s env_step=247.791s feedback_total=192.900s feedback_obs_pack=50.044s feedback_info_pack=8.725s effective_fps=138.93 +2026-09-28 19:10:08.642 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=356879 infer_calls=356879 feedback_calls=356879 infer_wait=2089.828s infer_obs_pack=43.759s env_step=251.476s feedback_total=195.462s feedback_obs_pack=50.771s feedback_info_pack=8.831s effective_fps=138.30 +2026-09-28 19:10:38.646 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=358399 infer_calls=358399 feedback_calls=358399 infer_wait=2112.037s infer_obs_pack=44.305s env_step=254.865s feedback_total=197.863s feedback_obs_pack=51.470s feedback_info_pack=8.952s effective_fps=137.37 +2026-09-28 19:11:08.650 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=360036 infer_calls=360036 feedback_calls=360036 infer_wait=2134.214s infer_obs_pack=44.983s env_step=258.259s feedback_total=200.134s feedback_obs_pack=52.101s feedback_info_pack=9.061s effective_fps=136.50 +2026-09-28 19:11:38.658 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=362872 infer_calls=362872 feedback_calls=362872 infer_wait=2157.647s infer_obs_pack=45.449s env_step=260.997s feedback_total=202.231s feedback_obs_pack=52.634s feedback_info_pack=9.157s effective_fps=136.09 +2026-09-28 19:12:08.659 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=366187 infer_calls=366187 feedback_calls=366187 infer_wait=2181.236s infer_obs_pack=45.977s env_step=263.527s feedback_total=204.322s feedback_obs_pack=53.152s feedback_info_pack=9.246s effective_fps=135.87 +2026-09-28 19:12:38.666 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=369495 infer_calls=369495 feedback_calls=369495 infer_wait=2204.618s infer_obs_pack=46.454s env_step=266.212s feedback_total=206.436s feedback_obs_pack=53.687s feedback_info_pack=9.356s effective_fps=135.66 +2026-09-28 19:13:08.674 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=372201 infer_calls=372201 feedback_calls=372201 infer_wait=2226.660s infer_obs_pack=47.059s env_step=269.574s feedback_total=208.894s feedback_obs_pack=54.345s feedback_info_pack=9.467s effective_fps=135.24 +2026-09-28 19:13:38.675 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=375206 infer_calls=375206 feedback_calls=375206 infer_wait=2249.351s infer_obs_pack=47.581s env_step=272.648s feedback_total=211.195s feedback_obs_pack=54.908s feedback_info_pack=9.557s effective_fps=134.93 +2026-09-28 19:14:08.679 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=378072 infer_calls=378072 feedback_calls=378072 infer_wait=2271.743s infer_obs_pack=48.114s env_step=275.902s feedback_total=213.619s feedback_obs_pack=55.542s feedback_info_pack=9.662s effective_fps=134.58 +2026-09-28 19:14:38.680 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=380790 infer_calls=380790 feedback_calls=380790 infer_wait=2293.544s infer_obs_pack=48.661s env_step=279.330s feedback_total=216.199s feedback_obs_pack=56.232s feedback_info_pack=9.777s effective_fps=134.19 +2026-09-28 19:15:08.683 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=384544 infer_calls=384544 feedback_calls=384544 infer_wait=2316.635s infer_obs_pack=49.172s env_step=282.142s feedback_total=218.400s feedback_obs_pack=56.808s feedback_info_pack=9.886s effective_fps=134.16 +2026-09-28 19:15:38.696 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=387036 infer_calls=387036 feedback_calls=387036 infer_wait=2340.155s infer_obs_pack=49.619s env_step=284.921s feedback_total=220.406s feedback_obs_pack=57.295s feedback_info_pack=9.976s effective_fps=133.69 +2026-09-28 19:16:08.698 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=391306 infer_calls=391306 feedback_calls=391306 infer_wait=2364.416s infer_obs_pack=50.042s env_step=287.196s feedback_total=222.289s feedback_obs_pack=57.780s feedback_info_pack=10.066s effective_fps=133.83 +2026-09-28 19:16:38.706 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=395327 infer_calls=395327 feedback_calls=395327 infer_wait=2388.262s infer_obs_pack=50.494s env_step=289.682s feedback_total=224.265s feedback_obs_pack=58.285s feedback_info_pack=10.153s effective_fps=133.89 +2026-09-28 19:17:08.706 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=399787 infer_calls=399787 feedback_calls=399787 infer_wait=2412.117s infer_obs_pack=50.952s env_step=292.182s feedback_total=226.208s feedback_obs_pack=58.788s feedback_info_pack=10.239s effective_fps=134.09 +2026-09-28 19:17:38.714 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=402600 infer_calls=402600 feedback_calls=402600 infer_wait=2435.292s infer_obs_pack=51.423s env_step=295.130s feedback_total=228.375s feedback_obs_pack=59.353s feedback_info_pack=10.341s effective_fps=133.74 +2026-09-28 19:18:08.715 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=405598 infer_calls=405598 feedback_calls=405598 infer_wait=2458.764s infer_obs_pack=51.865s env_step=297.897s feedback_total=230.436s feedback_obs_pack=59.884s feedback_info_pack=10.467s effective_fps=133.47 +2026-09-28 19:18:40.920 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=2481.450s infer_obs_pack=52.357s env_step=300.707s feedback_total=232.633s feedback_obs_pack=60.448s feedback_info_pack=10.553s effective_fps=133.54 +2026-09-28 19:18:40.920 | INFO  | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed5.log b/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed5.log new file mode 100644 index 0000000..6426f16 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client-seed5.log @@ -0,0 +1,115 @@ +2026-09-28 18:25:10.170 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 18:25:10.174 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 18:25:10.175 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 18:25:10.180 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 18:25:10.186 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 18:25:10.367 | INFO  | __main__:main:83 - Starting env client exp_name=e43-cross-seed5 output_dir=runs\e43-cross-seed5 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 18:25:10.511 | INFO  | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://100.75.226.89:9795... +2026-09-28 18:25:10.568 | INFO  | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 18:25:40.581 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=4823 infer_calls=4823 feedback_calls=4823 infer_wait=23.749s infer_obs_pack=0.449s env_step=2.531s feedback_total=2.002s feedback_obs_pack=0.516s feedback_info_pack=0.076s effective_fps=167.86 +2026-09-28 18:26:10.582 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=8634 infer_calls=8634 feedback_calls=8634 infer_wait=46.696s infer_obs_pack=0.947s env_step=5.434s feedback_total=4.236s feedback_obs_pack=1.113s feedback_info_pack=0.179s effective_fps=150.65 +2026-09-28 18:26:40.591 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=12294 infer_calls=12294 feedback_calls=12294 infer_wait=69.513s infer_obs_pack=1.487s env_step=8.347s feedback_total=6.524s feedback_obs_pack=1.688s feedback_info_pack=0.283s effective_fps=143.17 +2026-09-28 18:27:11.126 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=16385 infer_calls=16385 feedback_calls=16385 infer_wait=93.179s infer_obs_pack=2.114s env_step=11.043s feedback_total=8.713s feedback_obs_pack=2.230s feedback_info_pack=0.371s effective_fps=142.42 +2026-09-28 18:27:41.143 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=18751 infer_calls=18751 feedback_calls=18751 infer_wait=115.635s infer_obs_pack=2.678s env_step=14.253s feedback_total=11.120s feedback_obs_pack=2.846s feedback_info_pack=0.488s effective_fps=130.50 +2026-09-28 18:28:11.147 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=22703 infer_calls=22703 feedback_calls=22703 infer_wait=138.390s infer_obs_pack=3.219s env_step=17.149s feedback_total=13.412s feedback_obs_pack=3.451s feedback_info_pack=0.581s effective_fps=131.86 +2026-09-28 18:28:41.149 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=26589 infer_calls=26589 feedback_calls=26589 infer_wait=161.445s infer_obs_pack=3.724s env_step=19.981s feedback_total=15.618s feedback_obs_pack=4.031s feedback_info_pack=0.660s effective_fps=132.44 +2026-09-28 18:29:11.151 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=30475 infer_calls=30475 feedback_calls=30475 infer_wait=184.512s infer_obs_pack=4.232s env_step=22.758s feedback_total=17.876s feedback_obs_pack=4.565s feedback_info_pack=0.757s effective_fps=132.86 +2026-09-28 18:29:41.155 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=34456 infer_calls=34456 feedback_calls=34456 infer_wait=207.561s infer_obs_pack=4.747s env_step=25.663s feedback_total=20.029s feedback_obs_pack=5.126s feedback_info_pack=0.839s effective_fps=133.55 +2026-09-28 18:30:11.163 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=37949 infer_calls=37949 feedback_calls=37949 infer_wait=230.049s infer_obs_pack=5.323s env_step=28.758s feedback_total=22.368s feedback_obs_pack=5.761s feedback_info_pack=0.927s effective_fps=132.46 +2026-09-28 18:30:41.170 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41595 infer_calls=41595 feedback_calls=41595 infer_wait=252.982s infer_obs_pack=5.825s env_step=31.690s feedback_total=24.566s feedback_obs_pack=6.313s feedback_info_pack=1.021s effective_fps=132.02 +2026-09-28 18:31:11.171 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=43905 infer_calls=43905 feedback_calls=43905 infer_wait=275.545s infer_obs_pack=6.370s env_step=35.012s feedback_total=26.837s feedback_obs_pack=6.861s feedback_info_pack=1.093s effective_fps=127.72 +2026-09-28 18:31:41.181 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=47236 infer_calls=47236 feedback_calls=47236 infer_wait=297.974s infer_obs_pack=6.980s env_step=38.148s feedback_total=29.219s feedback_obs_pack=7.464s feedback_info_pack=1.184s effective_fps=126.87 +2026-09-28 18:32:11.274 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=50179 infer_calls=50179 feedback_calls=50179 infer_wait=321.403s infer_obs_pack=7.539s env_step=40.907s feedback_total=31.333s feedback_obs_pack=7.981s feedback_info_pack=1.254s effective_fps=125.08 +2026-09-28 18:32:41.275 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=53900 infer_calls=53900 feedback_calls=53900 infer_wait=344.165s infer_obs_pack=8.028s env_step=43.931s feedback_total=33.616s feedback_obs_pack=8.551s feedback_info_pack=1.354s effective_fps=125.42 +2026-09-28 18:33:11.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=57784 infer_calls=57784 feedback_calls=57784 infer_wait=367.072s infer_obs_pack=8.523s env_step=46.879s feedback_total=35.869s feedback_obs_pack=9.117s feedback_info_pack=1.437s effective_fps=126.07 +2026-09-28 18:33:41.283 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=62191 infer_calls=62191 feedback_calls=62191 infer_wait=390.537s infer_obs_pack=8.978s env_step=49.585s feedback_total=37.920s feedback_obs_pack=9.639s feedback_info_pack=1.518s effective_fps=127.70 +2026-09-28 18:34:11.287 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=67051 infer_calls=67051 feedback_calls=67051 infer_wait=413.719s infer_obs_pack=9.455s env_step=52.404s feedback_total=40.092s feedback_obs_pack=10.191s feedback_info_pack=1.598s effective_fps=130.03 +2026-09-28 18:34:41.291 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=71147 infer_calls=71147 feedback_calls=71147 infer_wait=436.987s infer_obs_pack=9.945s env_step=55.174s feedback_total=42.221s feedback_obs_pack=10.738s feedback_info_pack=1.683s effective_fps=130.71 +2026-09-28 18:35:11.301 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=75369 infer_calls=75369 feedback_calls=75369 infer_wait=459.948s infer_obs_pack=10.440s env_step=58.112s feedback_total=44.419s feedback_obs_pack=11.278s feedback_info_pack=1.775s effective_fps=131.55 +2026-09-28 18:35:41.313 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=78801 infer_calls=78801 feedback_calls=78801 infer_wait=482.404s infer_obs_pack=11.019s env_step=61.209s feedback_total=46.826s feedback_obs_pack=11.878s feedback_info_pack=1.858s effective_fps=131.02 +2026-09-28 18:36:11.317 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=82406 infer_calls=82406 feedback_calls=82406 infer_wait=505.000s infer_obs_pack=11.525s env_step=64.321s feedback_total=49.154s feedback_obs_pack=12.483s feedback_info_pack=1.947s effective_fps=130.80 +2026-09-28 18:36:41.316 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=86555 infer_calls=86555 feedback_calls=86555 infer_wait=527.942s infer_obs_pack=11.987s env_step=67.246s feedback_total=51.356s feedback_obs_pack=13.076s feedback_info_pack=2.037s effective_fps=131.44 +2026-09-28 18:37:11.330 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=90196 infer_calls=90196 feedback_calls=90196 infer_wait=551.446s infer_obs_pack=12.465s env_step=69.946s feedback_total=53.434s feedback_obs_pack=13.583s feedback_info_pack=2.133s effective_fps=131.23 +2026-09-28 18:37:41.333 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=93795 infer_calls=93795 feedback_calls=93795 infer_wait=574.169s infer_obs_pack=12.973s env_step=72.991s feedback_total=55.700s feedback_obs_pack=14.187s feedback_info_pack=2.250s effective_fps=131.03 +2026-09-28 18:38:11.334 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=96316 infer_calls=96316 feedback_calls=96316 infer_wait=597.093s infer_obs_pack=13.515s env_step=76.040s feedback_total=57.890s feedback_obs_pack=14.779s feedback_info_pack=2.350s effective_fps=129.36 +2026-09-28 18:38:41.336 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=100137 infer_calls=100137 feedback_calls=100137 infer_wait=619.973s infer_obs_pack=14.030s env_step=79.005s feedback_total=60.111s feedback_obs_pack=15.379s feedback_info_pack=2.442s effective_fps=129.52 +2026-09-28 18:39:11.346 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=103811 infer_calls=103811 feedback_calls=103811 infer_wait=643.127s infer_obs_pack=14.513s env_step=81.817s feedback_total=62.295s feedback_obs_pack=15.946s feedback_info_pack=2.533s effective_fps=129.48 +2026-09-28 18:39:41.347 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=107799 infer_calls=107799 feedback_calls=107799 infer_wait=666.700s infer_obs_pack=14.965s env_step=84.430s feedback_total=64.362s feedback_obs_pack=16.477s feedback_info_pack=2.619s effective_fps=129.81 +2026-09-28 18:40:11.351 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=112176 infer_calls=112176 feedback_calls=112176 infer_wait=690.043s infer_obs_pack=15.453s env_step=87.172s feedback_total=66.466s feedback_obs_pack=16.999s feedback_info_pack=2.714s effective_fps=130.57 +2026-09-28 18:40:41.355 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=116045 infer_calls=116045 feedback_calls=116045 infer_wait=713.163s infer_obs_pack=15.933s env_step=90.058s feedback_total=68.583s feedback_obs_pack=17.563s feedback_info_pack=2.806s effective_fps=130.72 +2026-09-28 18:41:11.356 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=120386 infer_calls=120386 feedback_calls=120386 infer_wait=736.361s infer_obs_pack=16.403s env_step=92.827s feedback_total=70.734s feedback_obs_pack=18.116s feedback_info_pack=2.900s effective_fps=131.38 +2026-09-28 18:41:41.359 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=125100 infer_calls=125100 feedback_calls=125100 infer_wait=759.790s infer_obs_pack=16.844s env_step=95.501s feedback_total=72.852s feedback_obs_pack=18.666s feedback_info_pack=2.992s effective_fps=132.38 +2026-09-28 18:42:11.363 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=130160 infer_calls=130160 feedback_calls=130160 infer_wait=783.479s infer_obs_pack=17.292s env_step=98.038s feedback_total=74.924s feedback_obs_pack=19.188s feedback_info_pack=3.089s effective_fps=133.67 +2026-09-28 18:42:43.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=135170 infer_calls=135170 feedback_calls=135170 infer_wait=809.167s infer_obs_pack=17.733s env_step=100.531s feedback_total=76.962s feedback_obs_pack=19.706s feedback_info_pack=3.181s effective_fps=134.58 +2026-09-28 18:43:13.286 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=139732 infer_calls=139732 feedback_calls=139732 infer_wait=832.955s infer_obs_pack=18.180s env_step=103.068s feedback_total=78.921s feedback_obs_pack=20.204s feedback_info_pack=3.272s effective_fps=135.25 +2026-09-28 18:43:43.288 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=144620 infer_calls=144620 feedback_calls=144620 infer_wait=856.441s infer_obs_pack=18.621s env_step=105.750s feedback_total=80.999s feedback_obs_pack=20.741s feedback_info_pack=3.367s effective_fps=136.20 +2026-09-28 18:44:13.287 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=150011 infer_calls=150011 feedback_calls=150011 infer_wait=880.430s infer_obs_pack=19.057s env_step=108.140s feedback_total=82.961s feedback_obs_pack=21.232s feedback_info_pack=3.458s effective_fps=137.55 +2026-09-28 18:44:43.289 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=154689 infer_calls=154689 feedback_calls=154689 infer_wait=904.367s infer_obs_pack=19.459s env_step=110.603s feedback_total=84.927s feedback_obs_pack=21.727s feedback_info_pack=3.545s effective_fps=138.19 +2026-09-28 18:45:13.290 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=159725 infer_calls=159725 feedback_calls=159725 infer_wait=928.029s infer_obs_pack=19.896s env_step=113.175s feedback_total=86.961s feedback_obs_pack=22.237s feedback_info_pack=3.634s effective_fps=139.13 +2026-09-28 18:45:43.295 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=164349 infer_calls=164349 feedback_calls=164349 infer_wait=951.814s infer_obs_pack=20.340s env_step=115.686s feedback_total=88.989s feedback_obs_pack=22.747s feedback_info_pack=3.725s effective_fps=139.65 +2026-09-28 18:46:13.299 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=169229 infer_calls=169229 feedback_calls=169229 infer_wait=975.572s infer_obs_pack=20.791s env_step=118.193s feedback_total=91.015s feedback_obs_pack=23.247s feedback_info_pack=3.814s effective_fps=140.37 +2026-09-28 18:46:43.300 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=173796 infer_calls=173796 feedback_calls=173796 infer_wait=998.944s infer_obs_pack=21.248s env_step=120.879s feedback_total=93.152s feedback_obs_pack=23.797s feedback_info_pack=3.908s effective_fps=140.81 +2026-09-28 18:47:13.302 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=178653 infer_calls=178653 feedback_calls=178653 infer_wait=1022.670s infer_obs_pack=21.692s env_step=123.409s feedback_total=95.172s feedback_obs_pack=24.307s feedback_info_pack=3.998s effective_fps=141.46 +2026-09-28 18:47:43.301 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=183808 infer_calls=183808 feedback_calls=183808 infer_wait=1046.502s infer_obs_pack=22.135s env_step=125.885s feedback_total=97.183s feedback_obs_pack=24.819s feedback_info_pack=4.089s effective_fps=142.30 +2026-09-28 18:48:13.318 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=188287 infer_calls=188287 feedback_calls=188287 infer_wait=1069.844s infer_obs_pack=22.613s env_step=128.574s feedback_total=99.362s feedback_obs_pack=25.359s feedback_info_pack=4.184s effective_fps=142.60 +2026-09-28 18:48:43.320 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=193011 infer_calls=193011 feedback_calls=193011 infer_wait=1093.560s infer_obs_pack=23.081s env_step=131.096s feedback_total=101.409s feedback_obs_pack=25.883s feedback_info_pack=4.286s effective_fps=143.06 +2026-09-28 18:49:13.323 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=197548 infer_calls=197548 feedback_calls=197548 infer_wait=1116.774s infer_obs_pack=23.550s env_step=133.868s feedback_total=103.594s feedback_obs_pack=26.458s feedback_info_pack=4.391s effective_fps=143.38 +2026-09-28 18:49:43.326 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=200785 infer_calls=200785 feedback_calls=200785 infer_wait=1139.300s infer_obs_pack=24.096s env_step=136.995s feedback_total=105.937s feedback_obs_pack=27.105s feedback_info_pack=4.486s effective_fps=142.77 +2026-09-28 18:50:13.328 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=204976 infer_calls=204976 feedback_calls=204976 infer_wait=1162.087s infer_obs_pack=24.632s env_step=139.943s feedback_total=108.150s feedback_obs_pack=27.692s feedback_info_pack=4.588s effective_fps=142.86 +2026-09-28 18:50:44.239 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=208898 infer_calls=208898 feedback_calls=208898 infer_wait=1186.755s infer_obs_pack=25.078s env_step=142.480s feedback_total=110.141s feedback_obs_pack=28.206s feedback_info_pack=4.674s effective_fps=142.65 +2026-09-28 18:51:14.243 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=213537 infer_calls=213537 feedback_calls=213537 infer_wait=1209.439s infer_obs_pack=25.578s env_step=145.474s feedback_total=112.473s feedback_obs_pack=28.807s feedback_info_pack=4.768s effective_fps=143.03 +2026-09-28 18:51:44.248 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=218926 infer_calls=218926 feedback_calls=218926 infer_wait=1232.895s infer_obs_pack=26.062s env_step=148.083s feedback_total=114.610s feedback_obs_pack=29.362s feedback_info_pack=4.853s effective_fps=143.87 +2026-09-28 18:52:14.250 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=223838 infer_calls=223838 feedback_calls=223838 infer_wait=1256.405s infer_obs_pack=26.507s env_step=150.752s feedback_total=116.674s feedback_obs_pack=29.877s feedback_info_pack=4.941s effective_fps=144.38 +2026-09-28 18:52:44.255 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=228767 infer_calls=228767 feedback_calls=228767 infer_wait=1279.926s infer_obs_pack=26.944s env_step=153.404s feedback_total=118.743s feedback_obs_pack=30.406s feedback_info_pack=5.023s effective_fps=144.88 +2026-09-28 18:53:14.258 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=233152 infer_calls=233152 feedback_calls=233152 infer_wait=1302.930s infer_obs_pack=27.433s env_step=156.309s feedback_total=120.908s feedback_obs_pack=30.967s feedback_info_pack=5.112s effective_fps=145.03 +2026-09-28 18:53:44.261 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=237715 infer_calls=237715 feedback_calls=237715 infer_wait=1326.449s infer_obs_pack=27.892s env_step=158.957s feedback_total=122.980s feedback_obs_pack=31.490s feedback_info_pack=5.211s effective_fps=145.28 +2026-09-28 18:54:14.264 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=242295 infer_calls=242295 feedback_calls=242295 infer_wait=1349.734s infer_obs_pack=28.348s env_step=161.741s feedback_total=125.060s feedback_obs_pack=32.019s feedback_info_pack=5.302s effective_fps=145.53 +2026-09-28 18:54:44.273 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=246954 infer_calls=246954 feedback_calls=246954 infer_wait=1373.558s infer_obs_pack=28.789s env_step=164.237s feedback_total=127.050s feedback_obs_pack=32.516s feedback_info_pack=5.399s effective_fps=145.81 +2026-09-28 18:55:14.277 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=251351 infer_calls=251351 feedback_calls=251351 infer_wait=1396.541s infer_obs_pack=29.249s env_step=167.106s feedback_total=129.322s feedback_obs_pack=33.099s feedback_info_pack=5.495s effective_fps=145.95 +2026-09-28 18:55:44.285 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=255686 infer_calls=255686 feedback_calls=255686 infer_wait=1420.354s infer_obs_pack=29.684s env_step=169.652s feedback_total=131.303s feedback_obs_pack=33.600s feedback_info_pack=5.583s effective_fps=146.02 +2026-09-28 18:56:14.289 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=259176 infer_calls=259176 feedback_calls=259176 infer_wait=1443.430s infer_obs_pack=30.188s env_step=172.609s feedback_total=133.483s feedback_obs_pack=34.168s feedback_info_pack=5.673s effective_fps=145.63 +2026-09-28 18:56:44.293 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=263670 infer_calls=263670 feedback_calls=263670 infer_wait=1466.222s infer_obs_pack=30.672s env_step=175.572s feedback_total=135.771s feedback_obs_pack=34.779s feedback_info_pack=5.779s effective_fps=145.82 +2026-09-28 18:57:14.295 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=268321 infer_calls=268321 feedback_calls=268321 infer_wait=1489.667s infer_obs_pack=31.138s env_step=178.225s feedback_total=137.890s feedback_obs_pack=35.313s feedback_info_pack=5.874s effective_fps=146.07 +2026-09-28 18:57:44.298 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=273350 infer_calls=273350 feedback_calls=273350 infer_wait=1513.129s infer_obs_pack=31.607s env_step=180.836s feedback_total=140.049s feedback_obs_pack=35.855s feedback_info_pack=5.973s effective_fps=146.52 +2026-09-28 18:58:14.303 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=275511 infer_calls=275511 feedback_calls=275511 infer_wait=1536.503s infer_obs_pack=32.189s env_step=183.756s feedback_total=142.043s feedback_obs_pack=36.371s feedback_info_pack=6.068s effective_fps=145.43 +2026-09-28 18:58:44.308 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=279526 infer_calls=279526 feedback_calls=279526 infer_wait=1558.812s infer_obs_pack=32.734s env_step=186.889s feedback_total=144.519s feedback_obs_pack=37.004s feedback_info_pack=6.184s effective_fps=145.36 +2026-09-28 18:59:14.312 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=283270 infer_calls=283270 feedback_calls=283270 infer_wait=1581.629s infer_obs_pack=33.259s env_step=189.852s feedback_total=146.822s feedback_obs_pack=37.584s feedback_info_pack=6.289s effective_fps=145.15 +2026-09-28 18:59:44.312 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=287861 infer_calls=287861 feedback_calls=287861 infer_wait=1604.347s infer_obs_pack=33.787s env_step=192.858s feedback_total=149.132s feedback_obs_pack=38.171s feedback_info_pack=6.389s effective_fps=145.38 +2026-09-28 19:00:14.314 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=292774 infer_calls=292774 feedback_calls=292774 infer_wait=1627.660s infer_obs_pack=34.274s env_step=195.568s feedback_total=151.278s feedback_obs_pack=38.717s feedback_info_pack=6.485s effective_fps=145.75 +2026-09-28 19:00:44.316 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=297309 infer_calls=297309 feedback_calls=297309 infer_wait=1651.185s infer_obs_pack=34.727s env_step=198.218s feedback_total=153.335s feedback_obs_pack=39.243s feedback_info_pack=6.579s effective_fps=145.92 +2026-09-28 19:01:14.322 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=301734 infer_calls=301734 feedback_calls=301734 infer_wait=1674.559s infer_obs_pack=35.202s env_step=200.952s feedback_total=155.450s feedback_obs_pack=39.770s feedback_info_pack=6.682s effective_fps=146.04 +2026-09-28 19:01:44.334 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=305566 infer_calls=305566 feedback_calls=305566 infer_wait=1697.073s infer_obs_pack=35.748s env_step=204.083s feedback_total=157.759s feedback_obs_pack=40.409s feedback_info_pack=6.789s effective_fps=145.88 +2026-09-28 19:02:14.339 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=309917 infer_calls=309917 feedback_calls=309917 infer_wait=1720.032s infer_obs_pack=36.233s env_step=206.935s feedback_total=160.035s feedback_obs_pack=40.977s feedback_info_pack=6.897s effective_fps=145.96 +2026-09-28 19:02:44.339 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=314068 infer_calls=314068 feedback_calls=314068 infer_wait=1743.244s infer_obs_pack=36.717s env_step=209.650s feedback_total=162.235s feedback_obs_pack=41.566s feedback_info_pack=7.001s effective_fps=145.95 +2026-09-28 19:03:14.342 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=317691 infer_calls=317691 feedback_calls=317691 infer_wait=1766.453s infer_obs_pack=37.193s env_step=212.518s feedback_total=164.320s feedback_obs_pack=42.113s feedback_info_pack=7.099s effective_fps=145.70 +2026-09-28 19:03:44.345 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=320908 infer_calls=320908 feedback_calls=320908 infer_wait=1789.788s infer_obs_pack=37.710s env_step=215.280s feedback_total=166.390s feedback_obs_pack=42.678s feedback_info_pack=7.184s effective_fps=145.26 +2026-09-28 19:04:14.349 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=325127 infer_calls=325127 feedback_calls=325127 infer_wait=1813.530s infer_obs_pack=38.137s env_step=217.863s feedback_total=168.343s feedback_obs_pack=43.185s feedback_info_pack=7.283s effective_fps=145.28 +2026-09-28 19:04:44.364 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=329409 infer_calls=329409 feedback_calls=329409 infer_wait=1837.492s infer_obs_pack=38.555s env_step=220.349s feedback_total=170.276s feedback_obs_pack=43.691s feedback_info_pack=7.367s effective_fps=145.33 +2026-09-28 19:05:14.366 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=333424 infer_calls=333424 feedback_calls=333424 infer_wait=1861.190s infer_obs_pack=38.996s env_step=222.923s feedback_total=172.259s feedback_obs_pack=44.210s feedback_info_pack=7.456s effective_fps=145.26 +2026-09-28 19:05:44.371 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=337324 infer_calls=337324 feedback_calls=337324 infer_wait=1884.289s infer_obs_pack=39.495s env_step=225.729s feedback_total=174.459s feedback_obs_pack=44.799s feedback_info_pack=7.559s effective_fps=145.15 +2026-09-28 19:06:14.377 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=341228 infer_calls=341228 feedback_calls=341228 infer_wait=1907.473s infer_obs_pack=39.983s env_step=228.442s feedback_total=176.678s feedback_obs_pack=45.388s feedback_info_pack=7.667s effective_fps=145.04 +2026-09-28 19:06:44.380 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=345009 infer_calls=345009 feedback_calls=345009 infer_wait=1931.135s infer_obs_pack=40.437s env_step=230.999s feedback_total=178.731s feedback_obs_pack=45.894s feedback_info_pack=7.759s effective_fps=144.88 +2026-09-28 19:07:14.400 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=348046 infer_calls=348046 feedback_calls=348046 infer_wait=1953.828s infer_obs_pack=40.934s env_step=234.188s feedback_total=180.941s feedback_obs_pack=46.509s feedback_info_pack=7.843s effective_fps=144.42 +2026-09-28 19:07:44.443 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=349866 infer_calls=349866 feedback_calls=349866 infer_wait=1976.701s infer_obs_pack=41.522s env_step=237.153s feedback_total=183.257s feedback_obs_pack=47.196s feedback_info_pack=7.934s effective_fps=143.47 +2026-09-28 19:08:14.453 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=351328 infer_calls=351328 feedback_calls=351328 infer_wait=1998.938s infer_obs_pack=42.130s env_step=240.489s feedback_total=185.641s feedback_obs_pack=47.789s feedback_info_pack=8.012s effective_fps=142.40 +2026-09-28 19:08:44.453 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=352991 infer_calls=352991 feedback_calls=352991 infer_wait=2020.971s infer_obs_pack=42.690s env_step=243.944s feedback_total=188.119s feedback_obs_pack=48.425s feedback_info_pack=8.129s effective_fps=141.44 +2026-09-28 19:09:14.456 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=355730 infer_calls=355730 feedback_calls=355730 infer_wait=2042.009s infer_obs_pack=43.339s env_step=247.823s feedback_total=190.816s feedback_obs_pack=49.120s feedback_info_pack=8.250s effective_fps=140.94 +2026-09-28 19:09:44.464 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=358321 infer_calls=358321 feedback_calls=358321 infer_wait=2063.487s infer_obs_pack=43.942s env_step=251.565s feedback_total=193.299s feedback_obs_pack=49.778s feedback_info_pack=8.353s effective_fps=140.39 +2026-09-28 19:10:14.482 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=360466 infer_calls=360466 feedback_calls=360466 infer_wait=2084.916s infer_obs_pack=44.563s env_step=255.320s feedback_total=195.917s feedback_obs_pack=50.473s feedback_info_pack=8.486s effective_fps=139.68 +2026-09-28 19:10:44.491 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=362028 infer_calls=362028 feedback_calls=362028 infer_wait=2106.760s infer_obs_pack=45.248s env_step=258.899s feedback_total=198.429s feedback_obs_pack=51.220s feedback_info_pack=8.659s effective_fps=138.74 +2026-09-28 19:11:14.493 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=363952 infer_calls=363952 feedback_calls=363952 infer_wait=2129.084s infer_obs_pack=45.828s env_step=262.113s feedback_total=200.866s feedback_obs_pack=51.815s feedback_info_pack=8.740s effective_fps=137.97 +2026-09-28 19:11:44.498 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=366869 infer_calls=366869 feedback_calls=366869 infer_wait=2152.123s infer_obs_pack=46.313s env_step=265.075s feedback_total=203.008s feedback_obs_pack=52.389s feedback_info_pack=8.834s effective_fps=137.58 +2026-09-28 19:12:14.500 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=370361 infer_calls=370361 feedback_calls=370361 infer_wait=2175.262s infer_obs_pack=46.811s env_step=267.893s feedback_total=205.169s feedback_obs_pack=52.960s feedback_info_pack=8.916s effective_fps=137.42 +2026-09-28 19:12:44.507 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=373522 infer_calls=373522 feedback_calls=373522 infer_wait=2198.868s infer_obs_pack=47.248s env_step=270.600s feedback_total=207.145s feedback_obs_pack=53.497s feedback_info_pack=9.007s effective_fps=137.13 +2026-09-28 19:13:14.510 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=376121 infer_calls=376121 feedback_calls=376121 infer_wait=2221.041s infer_obs_pack=47.802s env_step=273.975s feedback_total=209.513s feedback_obs_pack=54.124s feedback_info_pack=9.140s effective_fps=136.66 +2026-09-28 19:13:44.509 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=379327 infer_calls=379327 feedback_calls=379327 infer_wait=2243.628s infer_obs_pack=48.354s env_step=277.051s feedback_total=211.883s feedback_obs_pack=54.721s feedback_info_pack=9.245s effective_fps=136.40 +2026-09-28 19:14:14.511 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=382064 infer_calls=382064 feedback_calls=382064 infer_wait=2265.791s infer_obs_pack=48.870s env_step=280.423s feedback_total=214.345s feedback_obs_pack=55.345s feedback_info_pack=9.353s effective_fps=135.99 +2026-09-28 19:14:44.513 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=384811 infer_calls=384811 feedback_calls=384811 infer_wait=2287.340s infer_obs_pack=49.492s env_step=284.038s feedback_total=216.947s feedback_obs_pack=56.076s feedback_info_pack=9.480s effective_fps=135.60 +2026-09-28 19:15:14.523 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=388317 infer_calls=388317 feedback_calls=388317 infer_wait=2310.703s infer_obs_pack=49.919s env_step=286.872s feedback_total=219.017s feedback_obs_pack=56.589s feedback_info_pack=9.582s effective_fps=135.47 +2026-09-28 19:15:44.525 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=390962 infer_calls=390962 feedback_calls=390962 infer_wait=2334.005s infer_obs_pack=50.406s env_step=289.733s feedback_total=221.071s feedback_obs_pack=57.104s feedback_info_pack=9.676s effective_fps=135.04 +2026-09-28 19:16:14.528 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=395630 infer_calls=395630 feedback_calls=395630 infer_wait=2357.945s infer_obs_pack=50.841s env_step=292.161s feedback_total=223.040s feedback_obs_pack=57.595s feedback_info_pack=9.763s effective_fps=135.31 +2026-09-28 19:16:44.528 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=399959 infer_calls=399959 feedback_calls=399959 infer_wait=2381.212s infer_obs_pack=51.325s env_step=294.940s feedback_total=225.131s feedback_obs_pack=58.146s feedback_info_pack=9.852s effective_fps=135.46 +2026-09-28 19:17:14.534 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=404354 infer_calls=404354 feedback_calls=404354 infer_wait=2404.921s infer_obs_pack=51.766s env_step=297.567s feedback_total=227.111s feedback_obs_pack=58.685s feedback_info_pack=9.938s effective_fps=135.63 +2026-09-28 19:17:44.538 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=406876 infer_calls=406876 feedback_calls=406876 infer_wait=2427.832s infer_obs_pack=52.291s env_step=300.678s feedback_total=229.283s feedback_obs_pack=59.225s feedback_info_pack=10.024s effective_fps=135.17 +2026-09-28 19:18:07.712 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=2444.007s infer_obs_pack=52.693s env_step=302.855s feedback_total=230.878s feedback_obs_pack=59.617s feedback_info_pack=10.088s effective_fps=135.16 +2026-09-28 19:18:07.713 | INFO  | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed3.json b/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed3.json new file mode 100644 index 0000000..ec84d71 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed3.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-cross-seed3", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 2096.526611328125, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 2448.1576471005683, + "infer_obs_pack_s": 52.213102603913285, + "env_step_s": 298.774276682816, + "feedback_s": 230.9675520976598, + "feedback_obs_pack_s": 60.16772568714805, + "feedback_info_pack_s": 9.080838402092922, + "collect_time_s": 3030.1125784849573, + "effective_fps": 135.17682574183388, + "infer_wait_frac": 0.807942802021117, + "infer_obs_pack_frac": 0.01723140683770225, + "env_step_frac": 0.09860170833395299, + "feedback_frac": 0.07622408280722776 + } +} diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed4.json b/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed4.json new file mode 100644 index 0000000..52ded14 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed4.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-cross-seed4", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 1432.5323486328125, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 2481.450054302579, + "infer_obs_pack_s": 52.35691609972855, + "env_step_s": 300.70661359018413, + "feedback_s": 232.6325633918459, + "feedback_obs_pack_s": 60.447664595471, + "feedback_info_pack_s": 10.552917202177923, + "collect_time_s": 3067.1461473843374, + "effective_fps": 133.54466344856368, + "infer_wait_frac": 0.8090420003033634, + "infer_obs_pack_frac": 0.01707023845093868, + "env_step_frac": 0.09804117545772208, + "feedback_frac": 0.0758465857879759 + } +} diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed5.json b/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed5.json new file mode 100644 index 0000000..bdc4e49 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client-summary-seed5.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-cross-seed5", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 1246.04248046875, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 2444.0065643891867, + "infer_obs_pack_s": 52.69297790527344, + "env_step_s": 302.8548518976313, + "feedback_s": 230.87841510627186, + "feedback_obs_pack_s": 59.61658298198017, + "feedback_info_pack_s": 10.087548698007595, + "collect_time_s": 3030.4328092983633, + "effective_fps": 135.16254138458692, + "infer_wait_frac": 0.8064876267476289, + "infer_obs_pack_frac": 0.01738793803432766, + "env_step_frac": 0.09993782108231311, + "feedback_frac": 0.0761866141357304 + } +} diff --git a/experiments/e43-cross-machine-training/results/cross-clients-b/client_side.out b/experiments/e43-cross-machine-training/results/cross-clients-b/client_side.out new file mode 100644 index 0000000..0c33fab --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients-b/client_side.out @@ -0,0 +1,9 @@ +client: C:/Users/75128/.claude/jobs/1fea619d/tmp/e43/client/.venv/Scripts/python.exe +server: 100.75.226.89 ports 9790+seed +out: D:/75128/Desktop/plugrl-work/plugrl-server/.claude/worktrees/fix-pi0-image-mask-batching/experiments/e43-cross-machine-training/results/cross-clients-b +start 2026-09-28 18:24:59 +seed 3 client rc=0 at 19:17:59 +seed 5 client rc=0 at 19:18:08 +seed 4 client rc=0 at 19:18:41 +end 2026-09-28 19:18:41 +CLIENTS_DONE diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client-seed0.log b/experiments/e43-cross-machine-training/results/cross-clients/client-seed0.log new file mode 100644 index 0000000..8c400a7 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client-seed0.log @@ -0,0 +1,94 @@ +2026-09-28 17:38:50.627 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 17:38:50.631 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 17:38:50.635 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 17:38:50.641 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 17:38:50.648 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 17:38:50.870 | INFO  | __main__:main:83 - Starting env client exp_name=e43-cross-seed0 output_dir=runs\e43-cross-seed0 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 17:38:51.727 | INFO  | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://100.75.226.89:9790... +2026-09-28 17:38:51.796 | INFO  | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 17:39:21.804 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=689 infer_calls=689 feedback_calls=689 infer_wait=28.929s infer_obs_pack=0.086s env_step=0.419s feedback_total=0.349s feedback_obs_pack=0.084s feedback_info_pack=0.012s effective_fps=23.13 +2026-09-28 17:39:51.817 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=2830 infer_calls=2830 feedback_calls=2830 infer_wait=54.283s infer_obs_pack=0.494s env_step=2.344s feedback_total=1.757s feedback_obs_pack=0.467s feedback_info_pack=0.059s effective_fps=48.07 +2026-09-28 17:40:21.821 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=7578 infer_calls=7578 feedback_calls=7578 infer_wait=77.488s infer_obs_pack=0.990s env_step=5.112s feedback_total=3.927s feedback_obs_pack=1.028s feedback_info_pack=0.147s effective_fps=86.59 +2026-09-28 17:40:51.822 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=12124 infer_calls=12124 feedback_calls=12124 infer_wait=100.872s infer_obs_pack=1.463s env_step=7.772s feedback_total=6.074s feedback_obs_pack=1.591s feedback_info_pack=0.228s effective_fps=104.35 +2026-09-28 17:41:21.823 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=16606 infer_calls=16606 feedback_calls=16606 infer_wait=124.542s infer_obs_pack=1.957s env_step=10.319s feedback_total=8.080s feedback_obs_pack=2.114s feedback_info_pack=0.306s effective_fps=114.60 +2026-09-28 17:41:51.826 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=21357 infer_calls=21357 feedback_calls=21357 infer_wait=148.961s infer_obs_pack=2.361s env_step=12.540s feedback_total=9.913s feedback_obs_pack=2.576s feedback_info_pack=0.378s effective_fps=122.90 +2026-09-28 17:42:21.827 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=26484 infer_calls=26484 feedback_calls=26484 infer_wait=172.915s infer_obs_pack=2.806s env_step=14.934s feedback_total=11.896s feedback_obs_pack=3.083s feedback_info_pack=0.457s effective_fps=130.75 +2026-09-28 17:42:51.832 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=31441 infer_calls=31441 feedback_calls=31441 infer_wait=196.511s infer_obs_pack=3.278s env_step=17.494s feedback_total=13.990s feedback_obs_pack=3.637s feedback_info_pack=0.540s effective_fps=135.95 +2026-09-28 17:43:21.855 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=36384 infer_calls=36384 feedback_calls=36384 infer_wait=220.189s infer_obs_pack=3.731s env_step=20.044s feedback_total=16.074s feedback_obs_pack=4.178s feedback_info_pack=0.621s effective_fps=139.92 +2026-09-28 17:43:52.632 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=40962 infer_calls=40962 feedback_calls=40962 infer_wait=244.789s infer_obs_pack=4.177s env_step=22.523s feedback_total=18.045s feedback_obs_pack=4.674s feedback_info_pack=0.697s effective_fps=141.48 +2026-09-28 17:44:22.633 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=45949 infer_calls=45949 feedback_calls=45949 infer_wait=267.824s infer_obs_pack=4.672s env_step=25.355s feedback_total=20.282s feedback_obs_pack=5.246s feedback_info_pack=0.787s effective_fps=144.43 +2026-09-28 17:44:52.633 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=50826 infer_calls=50826 feedback_calls=50826 infer_wait=291.394s infer_obs_pack=5.129s env_step=27.931s feedback_total=22.374s feedback_obs_pack=5.776s feedback_info_pack=0.874s effective_fps=146.55 +2026-09-28 17:45:22.634 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=55890 infer_calls=55890 feedback_calls=55890 infer_wait=315.347s infer_obs_pack=5.562s env_step=30.372s feedback_total=24.345s feedback_obs_pack=6.280s feedback_info_pack=0.953s effective_fps=148.79 +2026-09-28 17:45:52.637 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=60777 infer_calls=60777 feedback_calls=60777 infer_wait=339.124s infer_obs_pack=6.019s env_step=32.905s feedback_total=26.329s feedback_obs_pack=6.792s feedback_info_pack=1.036s effective_fps=150.30 +2026-09-28 17:46:24.269 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=65538 infer_calls=65538 feedback_calls=65538 infer_wait=364.176s infer_obs_pack=6.483s env_step=35.581s feedback_total=28.431s feedback_obs_pack=7.322s feedback_info_pack=1.118s effective_fps=150.78 +2026-09-28 17:46:54.275 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=70188 infer_calls=70188 feedback_calls=70188 infer_wait=387.684s infer_obs_pack=6.952s env_step=38.236s feedback_total=30.528s feedback_obs_pack=7.859s feedback_info_pack=1.204s effective_fps=151.46 +2026-09-28 17:47:24.276 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=74829 infer_calls=74829 feedback_calls=74829 infer_wait=411.367s infer_obs_pack=7.411s env_step=40.807s feedback_total=32.561s feedback_obs_pack=8.371s feedback_info_pack=1.286s effective_fps=152.05 +2026-09-28 17:47:54.280 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=79866 infer_calls=79866 feedback_calls=79866 infer_wait=434.762s infer_obs_pack=7.878s env_step=43.506s feedback_total=34.675s feedback_obs_pack=8.917s feedback_info_pack=1.371s effective_fps=153.35 +2026-09-28 17:48:24.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=84723 infer_calls=84723 feedback_calls=84723 infer_wait=459.319s infer_obs_pack=8.278s env_step=45.682s feedback_total=36.494s feedback_obs_pack=9.395s feedback_info_pack=1.453s effective_fps=154.11 +2026-09-28 17:48:54.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=89740 infer_calls=89740 feedback_calls=89740 infer_wait=483.379s infer_obs_pack=8.698s env_step=48.111s feedback_total=38.411s feedback_obs_pack=9.885s feedback_info_pack=1.531s effective_fps=155.10 +2026-09-28 17:49:24.285 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=94512 infer_calls=94512 feedback_calls=94512 infer_wait=507.574s infer_obs_pack=9.110s env_step=50.495s feedback_total=40.276s feedback_obs_pack=10.356s feedback_info_pack=1.605s effective_fps=155.59 +2026-09-28 17:49:54.292 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=99323 infer_calls=99323 feedback_calls=99323 infer_wait=532.137s infer_obs_pack=9.491s env_step=52.712s feedback_total=42.062s feedback_obs_pack=10.804s feedback_info_pack=1.676s effective_fps=156.07 +2026-09-28 17:50:24.293 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=104694 infer_calls=104694 feedback_calls=104694 infer_wait=556.124s infer_obs_pack=9.921s env_step=55.139s feedback_total=44.021s feedback_obs_pack=11.297s feedback_info_pack=1.753s effective_fps=157.39 +2026-09-28 17:50:54.299 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=109729 infer_calls=109729 feedback_calls=109729 infer_wait=579.866s infer_obs_pack=10.372s env_step=57.706s feedback_total=46.026s feedback_obs_pack=11.805s feedback_info_pack=1.835s effective_fps=158.12 +2026-09-28 17:51:24.534 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=114690 infer_calls=114690 feedback_calls=114690 infer_wait=604.791s infer_obs_pack=10.762s env_step=59.847s feedback_total=47.770s feedback_obs_pack=12.248s feedback_info_pack=1.907s effective_fps=158.59 +2026-09-28 17:51:54.537 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=119604 infer_calls=119604 feedback_calls=119604 infer_wait=629.591s infer_obs_pack=11.133s env_step=61.940s feedback_total=49.484s feedback_obs_pack=12.684s feedback_info_pack=1.976s effective_fps=159.02 +2026-09-28 17:52:24.543 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=123806 infer_calls=123806 feedback_calls=123806 infer_wait=653.793s infer_obs_pack=11.537s env_step=64.292s feedback_total=51.379s feedback_obs_pack=13.169s feedback_info_pack=2.053s effective_fps=158.52 +2026-09-28 17:52:54.543 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=128925 infer_calls=128925 feedback_calls=128925 infer_wait=677.753s infer_obs_pack=11.973s env_step=66.747s feedback_total=53.319s feedback_obs_pack=13.660s feedback_info_pack=2.130s effective_fps=159.21 +2026-09-28 17:53:24.550 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=133720 infer_calls=133720 feedback_calls=133720 infer_wait=702.270s infer_obs_pack=12.356s env_step=68.995s feedback_total=55.091s feedback_obs_pack=14.112s feedback_info_pack=2.199s effective_fps=159.44 +2026-09-28 17:53:54.552 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=138458 infer_calls=138458 feedback_calls=138458 infer_wait=725.254s infer_obs_pack=12.871s env_step=71.854s feedback_total=57.351s feedback_obs_pack=14.703s feedback_info_pack=2.287s effective_fps=159.64 +2026-09-28 17:54:24.554 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=143351 infer_calls=143351 feedback_calls=143351 infer_wait=749.352s infer_obs_pack=13.291s env_step=74.224s feedback_total=59.306s feedback_obs_pack=15.201s feedback_info_pack=2.361s effective_fps=159.96 +2026-09-28 17:54:54.556 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=148141 infer_calls=148141 feedback_calls=148141 infer_wait=773.686s infer_obs_pack=13.698s env_step=76.495s feedback_total=61.163s feedback_obs_pack=15.672s feedback_info_pack=2.437s effective_fps=160.15 +2026-09-28 17:55:24.562 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=153375 infer_calls=153375 feedback_calls=153375 infer_wait=797.705s infer_obs_pack=14.150s env_step=78.873s feedback_total=63.139s feedback_obs_pack=16.174s feedback_info_pack=2.522s effective_fps=160.79 +2026-09-28 17:55:54.565 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=158173 infer_calls=158173 feedback_calls=158173 infer_wait=821.434s infer_obs_pack=14.584s env_step=81.386s feedback_total=65.191s feedback_obs_pack=16.710s feedback_info_pack=2.606s effective_fps=160.97 +2026-09-28 17:56:24.568 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=162655 infer_calls=162655 feedback_calls=162655 infer_wait=845.019s infer_obs_pack=15.041s env_step=84.015s feedback_total=67.252s feedback_obs_pack=17.251s feedback_info_pack=2.689s effective_fps=160.83 +2026-09-28 17:56:54.568 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=166761 infer_calls=166761 feedback_calls=166761 infer_wait=868.540s infer_obs_pack=15.482s env_step=86.746s feedback_total=69.329s feedback_obs_pack=17.776s feedback_info_pack=2.769s effective_fps=160.33 +2026-09-28 17:57:24.572 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=171483 infer_calls=171483 feedback_calls=171483 infer_wait=892.447s infer_obs_pack=15.924s env_step=89.203s feedback_total=71.310s feedback_obs_pack=18.271s feedback_info_pack=2.850s effective_fps=160.43 +2026-09-28 17:57:56.087 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=176130 infer_calls=176130 feedback_calls=176130 infer_wait=917.704s infer_obs_pack=16.397s env_step=91.684s feedback_total=73.353s feedback_obs_pack=18.792s feedback_info_pack=2.931s effective_fps=160.24 +2026-09-28 17:58:26.090 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=180901 infer_calls=180901 feedback_calls=180901 infer_wait=941.122s infer_obs_pack=16.885s env_step=94.325s feedback_total=75.482s feedback_obs_pack=19.338s feedback_info_pack=3.016s effective_fps=160.40 +2026-09-28 17:58:56.094 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=186025 infer_calls=186025 feedback_calls=186025 infer_wait=965.361s infer_obs_pack=17.308s env_step=96.590s feedback_total=77.409s feedback_obs_pack=19.837s feedback_info_pack=3.099s effective_fps=160.83 +2026-09-28 17:59:26.095 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=191385 infer_calls=191385 feedback_calls=191385 infer_wait=989.353s infer_obs_pack=17.744s env_step=98.985s feedback_total=79.399s feedback_obs_pack=20.339s feedback_info_pack=3.178s effective_fps=161.44 +2026-09-28 17:59:57.243 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=196610 infer_calls=196610 feedback_calls=196610 infer_wait=1014.691s infer_obs_pack=18.169s env_step=101.302s feedback_total=81.310s feedback_obs_pack=20.828s feedback_info_pack=3.255s effective_fps=161.76 +2026-09-28 18:00:27.245 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=202110 infer_calls=202110 feedback_calls=202110 infer_wait=1039.027s infer_obs_pack=18.589s env_step=103.550s feedback_total=83.190s feedback_obs_pack=21.310s feedback_info_pack=3.332s effective_fps=162.42 +2026-09-28 18:00:57.245 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=207686 infer_calls=207686 feedback_calls=207686 infer_wait=1063.233s infer_obs_pack=19.012s env_step=105.845s feedback_total=85.125s feedback_obs_pack=21.806s feedback_info_pack=3.409s effective_fps=163.12 +2026-09-28 18:01:27.529 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=212994 infer_calls=212994 feedback_calls=212994 infer_wait=1088.071s infer_obs_pack=19.417s env_step=107.980s feedback_total=86.953s feedback_obs_pack=22.271s feedback_info_pack=3.482s effective_fps=163.54 +2026-09-28 18:01:57.528 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=218140 infer_calls=218140 feedback_calls=218140 infer_wait=1112.722s infer_obs_pack=19.817s env_step=110.081s feedback_total=88.741s feedback_obs_pack=22.729s feedback_info_pack=3.557s effective_fps=163.85 +2026-09-28 18:02:27.534 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=223743 infer_calls=223743 feedback_calls=223743 infer_wait=1136.786s infer_obs_pack=20.265s env_step=112.400s feedback_total=90.737s feedback_obs_pack=23.234s feedback_info_pack=3.638s effective_fps=164.49 +2026-09-28 18:02:57.539 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=228785 infer_calls=228785 feedback_calls=228785 infer_wait=1160.941s infer_obs_pack=20.695s env_step=114.716s feedback_total=92.678s feedback_obs_pack=23.729s feedback_info_pack=3.718s effective_fps=164.71 +2026-09-28 18:03:27.544 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=233513 infer_calls=233513 feedback_calls=233513 infer_wait=1185.111s infer_obs_pack=21.109s env_step=117.053s feedback_total=94.601s feedback_obs_pack=24.209s feedback_info_pack=3.793s effective_fps=164.69 +2026-09-28 18:03:57.545 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=239007 infer_calls=239007 feedback_calls=239007 infer_wait=1209.112s infer_obs_pack=21.548s env_step=119.408s feedback_total=96.616s feedback_obs_pack=24.723s feedback_info_pack=3.874s effective_fps=165.21 +2026-09-28 18:04:27.545 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=244683 infer_calls=244683 feedback_calls=244683 infer_wait=1233.271s infer_obs_pack=21.983s env_step=121.692s feedback_total=98.577s feedback_obs_pack=25.229s feedback_info_pack=3.954s effective_fps=165.83 +2026-09-28 18:04:57.546 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=249877 infer_calls=249877 feedback_calls=249877 infer_wait=1257.765s infer_obs_pack=22.388s env_step=123.871s feedback_total=100.412s feedback_obs_pack=25.699s feedback_info_pack=4.030s effective_fps=166.09 +2026-09-28 18:05:27.550 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=255311 infer_calls=255311 feedback_calls=255311 infer_wait=1281.718s infer_obs_pack=22.837s env_step=126.253s feedback_total=102.416s feedback_obs_pack=26.211s feedback_info_pack=4.111s effective_fps=166.52 +2026-09-28 18:05:57.550 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=260599 infer_calls=260599 feedback_calls=260599 infer_wait=1305.739s infer_obs_pack=23.275s env_step=128.616s feedback_total=104.393s feedback_obs_pack=26.719s feedback_info_pack=4.190s effective_fps=166.83 +2026-09-28 18:06:27.550 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=265136 infer_calls=265136 feedback_calls=265136 infer_wait=1329.146s infer_obs_pack=23.756s env_step=131.286s feedback_total=106.528s feedback_obs_pack=27.292s feedback_info_pack=4.276s effective_fps=166.68 +2026-09-28 18:06:57.553 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=269647 infer_calls=269647 feedback_calls=269647 infer_wait=1352.840s infer_obs_pack=24.223s env_step=133.838s feedback_total=108.549s feedback_obs_pack=27.820s feedback_info_pack=4.353s effective_fps=166.51 +2026-09-28 18:07:27.554 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=274563 infer_calls=274563 feedback_calls=274563 infer_wait=1377.123s infer_obs_pack=24.641s env_step=136.102s feedback_total=110.437s feedback_obs_pack=28.306s feedback_info_pack=4.429s effective_fps=166.57 +2026-09-28 18:07:57.556 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=279072 infer_calls=279072 feedback_calls=279072 infer_wait=1400.551s infer_obs_pack=25.143s env_step=138.678s feedback_total=112.641s feedback_obs_pack=28.899s feedback_info_pack=4.519s effective_fps=166.41 +2026-09-28 18:08:28.258 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=282626 infer_calls=282626 feedback_calls=282626 infer_wait=1424.318s infer_obs_pack=25.653s env_step=141.522s feedback_total=114.897s feedback_obs_pack=29.408s feedback_info_pack=4.619s effective_fps=165.63 +2026-09-28 18:08:58.259 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=287380 infer_calls=287380 feedback_calls=287380 infer_wait=1447.773s infer_obs_pack=26.142s env_step=144.146s feedback_total=117.017s feedback_obs_pack=29.951s feedback_info_pack=4.710s effective_fps=165.63 +2026-09-28 18:09:28.268 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=292048 infer_calls=292048 feedback_calls=292048 infer_wait=1471.090s infer_obs_pack=26.635s env_step=146.846s feedback_total=119.167s feedback_obs_pack=30.528s feedback_info_pack=4.795s effective_fps=165.58 +2026-09-28 18:09:58.270 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=296651 infer_calls=296651 feedback_calls=296651 infer_wait=1494.572s infer_obs_pack=27.090s env_step=149.435s feedback_total=121.310s feedback_obs_pack=31.073s feedback_info_pack=4.875s effective_fps=165.50 +2026-09-28 18:10:28.274 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=301679 infer_calls=301679 feedback_calls=301679 infer_wait=1518.087s infer_obs_pack=27.567s env_step=152.035s feedback_total=123.404s feedback_obs_pack=31.624s feedback_info_pack=4.964s effective_fps=165.66 +2026-09-28 18:10:58.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=306498 infer_calls=306498 feedback_calls=306498 infer_wait=1541.432s infer_obs_pack=28.056s env_step=154.697s feedback_total=125.544s feedback_obs_pack=32.175s feedback_info_pack=5.056s effective_fps=165.70 +2026-09-28 18:11:29.276 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=311298 infer_calls=311298 feedback_calls=311298 infer_wait=1565.869s infer_obs_pack=28.555s env_step=157.279s feedback_total=127.677s feedback_obs_pack=32.723s feedback_info_pack=5.148s effective_fps=165.64 +2026-09-28 18:11:59.277 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=315868 infer_calls=315868 feedback_calls=315868 infer_wait=1589.483s infer_obs_pack=29.014s env_step=159.844s feedback_total=129.716s feedback_obs_pack=33.243s feedback_info_pack=5.226s effective_fps=165.54 +2026-09-28 18:12:29.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=320355 infer_calls=320355 feedback_calls=320355 infer_wait=1613.339s infer_obs_pack=29.486s env_step=162.254s feedback_total=131.722s feedback_obs_pack=33.764s feedback_info_pack=5.308s effective_fps=165.40 +2026-09-28 18:12:59.287 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=325080 infer_calls=325080 feedback_calls=325080 infer_wait=1637.729s infer_obs_pack=29.885s env_step=164.528s feedback_total=133.549s feedback_obs_pack=34.235s feedback_info_pack=5.381s effective_fps=165.38 +2026-09-28 18:13:29.293 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=330171 infer_calls=330171 feedback_calls=330171 infer_wait=1661.835s infer_obs_pack=30.303s env_step=166.907s feedback_total=135.471s feedback_obs_pack=34.733s feedback_info_pack=5.460s effective_fps=165.54 +2026-09-28 18:13:59.296 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=335372 infer_calls=335372 feedback_calls=335372 infer_wait=1685.873s infer_obs_pack=30.747s env_step=169.241s feedback_total=137.457s feedback_obs_pack=35.243s feedback_info_pack=5.543s effective_fps=165.75 +2026-09-28 18:14:29.298 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=340016 infer_calls=340016 feedback_calls=340016 infer_wait=1709.714s infer_obs_pack=31.219s env_step=171.678s feedback_total=139.463s feedback_obs_pack=35.766s feedback_info_pack=5.625s effective_fps=165.69 +2026-09-28 18:14:59.300 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=345167 infer_calls=345167 feedback_calls=345167 infer_wait=1733.513s infer_obs_pack=31.678s env_step=174.133s feedback_total=141.505s feedback_obs_pack=36.289s feedback_info_pack=5.708s effective_fps=165.88 +2026-09-28 18:15:29.301 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=350531 infer_calls=350531 feedback_calls=350531 infer_wait=1757.727s infer_obs_pack=32.107s env_step=176.387s feedback_total=143.454s feedback_obs_pack=36.786s feedback_info_pack=5.787s effective_fps=166.15 +2026-09-28 18:15:59.310 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=355978 infer_calls=355978 feedback_calls=355978 infer_wait=1782.033s infer_obs_pack=32.534s env_step=178.624s feedback_total=145.354s feedback_obs_pack=37.268s feedback_info_pack=5.867s effective_fps=166.46 +2026-09-28 18:16:29.310 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=360975 infer_calls=360975 feedback_calls=360975 infer_wait=1806.888s infer_obs_pack=32.920s env_step=180.650s feedback_total=147.076s feedback_obs_pack=37.710s feedback_info_pack=5.936s effective_fps=166.54 +2026-09-28 18:16:59.313 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=366355 infer_calls=366355 feedback_calls=366355 infer_wait=1831.333s infer_obs_pack=33.335s env_step=182.839s feedback_total=148.928s feedback_obs_pack=38.185s feedback_info_pack=6.012s effective_fps=166.80 +2026-09-28 18:17:29.317 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=371052 infer_calls=371052 feedback_calls=371052 infer_wait=1855.127s infer_obs_pack=33.797s env_step=185.273s feedback_total=150.929s feedback_obs_pack=38.701s feedback_info_pack=6.093s effective_fps=166.76 +2026-09-28 18:17:59.318 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=376482 infer_calls=376482 feedback_calls=376482 infer_wait=1879.468s infer_obs_pack=34.216s env_step=187.490s feedback_total=152.827s feedback_obs_pack=39.186s feedback_info_pack=6.170s effective_fps=167.03 +2026-09-28 18:18:29.322 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=381563 infer_calls=381563 feedback_calls=381563 infer_wait=1903.892s infer_obs_pack=34.632s env_step=189.674s feedback_total=154.698s feedback_obs_pack=39.665s feedback_info_pack=6.245s effective_fps=167.14 +2026-09-28 18:18:59.326 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=386788 infer_calls=386788 feedback_calls=386788 infer_wait=1927.250s infer_obs_pack=35.149s env_step=192.305s feedback_total=156.881s feedback_obs_pack=40.230s feedback_info_pack=6.332s effective_fps=167.33 +2026-09-28 18:19:29.330 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=392102 infer_calls=392102 feedback_calls=392102 infer_wait=1951.354s infer_obs_pack=35.585s env_step=194.614s feedback_total=158.851s feedback_obs_pack=40.728s feedback_info_pack=6.412s effective_fps=167.54 +2026-09-28 18:20:00.737 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=397314 infer_calls=397314 feedback_calls=397314 infer_wait=1976.717s infer_obs_pack=36.028s env_step=196.989s feedback_total=160.854s feedback_obs_pack=41.241s feedback_info_pack=6.492s effective_fps=167.60 +2026-09-28 18:20:30.740 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=401918 infer_calls=401918 feedback_calls=401918 infer_wait=2000.269s infer_obs_pack=36.500s env_step=199.583s feedback_total=162.938s feedback_obs_pack=41.776s feedback_info_pack=6.574s effective_fps=167.52 +2026-09-28 18:21:00.741 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=406984 infer_calls=406984 feedback_calls=406984 infer_wait=2023.727s infer_obs_pack=36.972s env_step=202.190s feedback_total=165.089s feedback_obs_pack=42.320s feedback_info_pack=6.659s effective_fps=167.62 +2026-09-28 18:21:16.420 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=2034.719s infer_obs_pack=37.188s env_step=203.391s feedback_total=166.086s feedback_obs_pack=42.576s feedback_info_pack=6.699s effective_fps=167.77 +2026-09-28 18:21:16.422 | INFO  | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client-seed1.log b/experiments/e43-cross-machine-training/results/cross-clients/client-seed1.log new file mode 100644 index 0000000..0331aab --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client-seed1.log @@ -0,0 +1,94 @@ +2026-09-28 17:38:54.806 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 17:38:54.808 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 17:38:54.810 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 17:38:54.814 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 17:38:54.819 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 17:38:54.991 | INFO  | __main__:main:83 - Starting env client exp_name=e43-cross-seed1 output_dir=runs\e43-cross-seed1 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 17:38:55.173 | INFO  | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://100.75.226.89:9791... +2026-09-28 17:38:55.254 | INFO  | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 17:39:25.275 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=720 infer_calls=720 feedback_calls=720 infer_wait=28.870s infer_obs_pack=0.074s env_step=0.465s feedback_total=0.395s feedback_obs_pack=0.091s feedback_info_pack=0.025s effective_fps=24.16 +2026-09-28 17:39:55.278 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=3403 infer_calls=3403 feedback_calls=3403 infer_wait=53.853s infer_obs_pack=0.423s env_step=2.466s feedback_total=2.062s feedback_obs_pack=0.508s feedback_info_pack=0.105s effective_fps=57.87 +2026-09-28 17:40:25.280 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=8086 infer_calls=8086 feedback_calls=8086 infer_wait=76.883s infer_obs_pack=0.937s env_step=5.308s feedback_total=4.267s feedback_obs_pack=1.068s feedback_info_pack=0.195s effective_fps=92.52 +2026-09-28 17:40:55.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=12441 infer_calls=12441 feedback_calls=12441 infer_wait=100.712s infer_obs_pack=1.393s env_step=7.754s feedback_total=6.294s feedback_obs_pack=1.595s feedback_info_pack=0.275s effective_fps=107.11 +2026-09-28 17:41:25.282 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=17246 infer_calls=17246 feedback_calls=17246 infer_wait=124.048s infer_obs_pack=1.947s env_step=10.411s feedback_total=8.399s feedback_obs_pack=2.143s feedback_info_pack=0.359s effective_fps=119.10 +2026-09-28 17:41:55.286 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=22031 infer_calls=22031 feedback_calls=22031 infer_wait=148.479s infer_obs_pack=2.354s env_step=12.621s feedback_total=10.241s feedback_obs_pack=2.617s feedback_info_pack=0.432s effective_fps=126.84 +2026-09-28 17:42:25.287 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=27081 infer_calls=27081 feedback_calls=27081 infer_wait=172.120s infer_obs_pack=2.829s env_step=15.166s feedback_total=12.313s feedback_obs_pack=3.147s feedback_info_pack=0.515s effective_fps=133.78 +2026-09-28 17:42:55.289 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=32093 infer_calls=32093 feedback_calls=32093 infer_wait=195.953s infer_obs_pack=3.292s env_step=17.654s feedback_total=14.308s feedback_obs_pack=3.657s feedback_info_pack=0.597s effective_fps=138.81 +2026-09-28 17:43:25.293 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=36797 infer_calls=36797 feedback_calls=36797 infer_wait=219.580s infer_obs_pack=3.790s env_step=20.252s feedback_total=16.336s feedback_obs_pack=4.167s feedback_info_pack=0.676s effective_fps=141.55 +2026-09-28 17:43:55.295 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41472 infer_calls=41472 feedback_calls=41472 infer_wait=243.343s infer_obs_pack=4.233s env_step=22.813s feedback_total=18.330s feedback_obs_pack=4.678s feedback_info_pack=0.755s effective_fps=143.64 +2026-09-28 17:44:25.301 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=46493 infer_calls=46493 feedback_calls=46493 infer_wait=266.337s infer_obs_pack=4.729s env_step=25.715s feedback_total=20.549s feedback_obs_pack=5.239s feedback_info_pack=0.844s effective_fps=146.51 +2026-09-28 17:44:55.310 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=51263 infer_calls=51263 feedback_calls=51263 infer_wait=289.825s infer_obs_pack=5.194s env_step=28.387s feedback_total=22.636s feedback_obs_pack=5.770s feedback_info_pack=0.924s effective_fps=148.14 +2026-09-28 17:45:25.311 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=56464 infer_calls=56464 feedback_calls=56464 infer_wait=313.828s infer_obs_pack=5.615s env_step=30.821s feedback_total=24.579s feedback_obs_pack=6.265s feedback_info_pack=1.001s effective_fps=150.63 +2026-09-28 17:45:55.316 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=61312 infer_calls=61312 feedback_calls=61312 infer_wait=337.446s infer_obs_pack=6.064s env_step=33.433s feedback_total=26.626s feedback_obs_pack=6.804s feedback_info_pack=1.083s effective_fps=151.92 +2026-09-28 17:46:25.318 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=65770 infer_calls=65770 feedback_calls=65770 infer_wait=361.281s infer_obs_pack=6.503s env_step=35.963s feedback_total=28.593s feedback_obs_pack=7.314s feedback_info_pack=1.173s effective_fps=152.13 +2026-09-28 17:46:55.322 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=70428 infer_calls=70428 feedback_calls=70428 infer_wait=384.797s infer_obs_pack=6.974s env_step=38.609s feedback_total=30.665s feedback_obs_pack=7.844s feedback_info_pack=1.256s effective_fps=152.76 +2026-09-28 17:47:25.326 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=75120 infer_calls=75120 feedback_calls=75120 infer_wait=408.386s infer_obs_pack=7.440s env_step=41.222s feedback_total=32.726s feedback_obs_pack=8.371s feedback_info_pack=1.334s effective_fps=153.38 +2026-09-28 17:47:55.329 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=80127 infer_calls=80127 feedback_calls=80127 infer_wait=431.713s infer_obs_pack=7.907s env_step=43.977s feedback_total=34.846s feedback_obs_pack=8.914s feedback_info_pack=1.417s effective_fps=154.55 +2026-09-28 17:48:25.331 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=85033 infer_calls=85033 feedback_calls=85033 infer_wait=456.190s infer_obs_pack=8.305s env_step=46.216s feedback_total=36.649s feedback_obs_pack=9.369s feedback_info_pack=1.492s effective_fps=155.35 +2026-09-28 17:48:55.337 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=89991 infer_calls=89991 feedback_calls=89991 infer_wait=480.307s infer_obs_pack=8.727s env_step=48.585s feedback_total=38.566s feedback_obs_pack=9.857s feedback_info_pack=1.567s effective_fps=156.18 +2026-09-28 17:49:25.340 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=94799 infer_calls=94799 feedback_calls=94799 infer_wait=504.595s infer_obs_pack=9.144s env_step=50.876s feedback_total=40.435s feedback_obs_pack=10.335s feedback_info_pack=1.640s effective_fps=156.68 +2026-09-28 17:49:55.341 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=99639 infer_calls=99639 feedback_calls=99639 infer_wait=529.227s infer_obs_pack=9.523s env_step=53.023s feedback_total=42.228s feedback_obs_pack=10.779s feedback_info_pack=1.715s effective_fps=157.16 +2026-09-28 17:50:25.342 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=105052 infer_calls=105052 feedback_calls=105052 infer_wait=553.187s infer_obs_pack=9.955s env_step=55.440s feedback_total=44.214s feedback_obs_pack=11.280s feedback_info_pack=1.794s effective_fps=158.50 +2026-09-28 17:50:55.344 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=110078 infer_calls=110078 feedback_calls=110078 infer_wait=576.994s infer_obs_pack=10.432s env_step=57.906s feedback_total=46.226s feedback_obs_pack=11.795s feedback_info_pack=1.874s effective_fps=159.17 +2026-09-28 17:51:25.349 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=114996 infer_calls=114996 feedback_calls=114996 infer_wait=601.737s infer_obs_pack=10.814s env_step=59.992s feedback_total=47.977s feedback_obs_pack=12.237s feedback_info_pack=1.943s effective_fps=159.60 +2026-09-28 17:51:55.352 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=119970 infer_calls=119970 feedback_calls=119970 infer_wait=626.493s infer_obs_pack=11.193s env_step=62.083s feedback_total=49.706s feedback_obs_pack=12.680s feedback_info_pack=2.011s effective_fps=160.07 +2026-09-28 17:52:25.357 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=124195 infer_calls=124195 feedback_calls=124195 infer_wait=650.669s infer_obs_pack=11.611s env_step=64.488s feedback_total=51.584s feedback_obs_pack=13.155s feedback_info_pack=2.081s effective_fps=159.56 +2026-09-28 17:52:55.362 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=129286 infer_calls=129286 feedback_calls=129286 infer_wait=674.629s infer_obs_pack=12.039s env_step=66.934s feedback_total=53.533s feedback_obs_pack=13.651s feedback_info_pack=2.161s effective_fps=160.18 +2026-09-28 17:53:25.361 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=134071 infer_calls=134071 feedback_calls=134071 infer_wait=699.047s infer_obs_pack=12.441s env_step=69.187s feedback_total=55.368s feedback_obs_pack=14.101s feedback_info_pack=2.230s effective_fps=160.36 +2026-09-28 17:53:55.367 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=138916 infer_calls=138916 feedback_calls=138916 infer_wait=722.184s infer_obs_pack=12.929s env_step=71.979s feedback_total=57.575s feedback_obs_pack=14.655s feedback_info_pack=2.316s effective_fps=160.66 +2026-09-28 17:54:25.368 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=143362 infer_calls=143362 feedback_calls=143362 infer_wait=746.849s infer_obs_pack=13.313s env_step=74.074s feedback_total=59.351s feedback_obs_pack=15.110s feedback_info_pack=2.395s effective_fps=160.43 +2026-09-28 17:54:55.371 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=148476 infer_calls=148476 feedback_calls=148476 infer_wait=770.810s infer_obs_pack=13.747s env_step=76.491s feedback_total=61.316s feedback_obs_pack=15.609s feedback_info_pack=2.473s effective_fps=160.97 +2026-09-28 17:55:25.373 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=153740 infer_calls=153740 feedback_calls=153740 infer_wait=794.773s infer_obs_pack=14.191s env_step=78.882s feedback_total=63.309s feedback_obs_pack=16.113s feedback_info_pack=2.553s effective_fps=161.64 +2026-09-28 17:55:55.374 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=158657 infer_calls=158657 feedback_calls=158657 infer_wait=818.428s infer_obs_pack=14.655s env_step=81.440s feedback_total=65.354s feedback_obs_pack=16.653s feedback_info_pack=2.639s effective_fps=161.92 +2026-09-28 17:56:25.377 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163192 infer_calls=163192 feedback_calls=163192 infer_wait=842.032s infer_obs_pack=15.084s env_step=84.031s feedback_total=67.417s feedback_obs_pack=17.173s feedback_info_pack=2.714s effective_fps=161.81 +2026-09-28 17:56:55.384 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=167239 infer_calls=167239 feedback_calls=167239 infer_wait=865.612s infer_obs_pack=15.523s env_step=86.628s feedback_total=69.521s feedback_obs_pack=17.731s feedback_info_pack=2.800s effective_fps=161.23 +2026-09-28 17:57:25.387 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=171973 infer_calls=171973 feedback_calls=171973 infer_wait=889.470s infer_obs_pack=15.964s env_step=89.118s feedback_total=71.499s feedback_obs_pack=18.242s feedback_info_pack=2.876s effective_fps=161.32 +2026-09-28 17:57:55.388 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=176334 infer_calls=176334 feedback_calls=176334 infer_wait=913.641s infer_obs_pack=16.381s env_step=91.458s feedback_total=73.391s feedback_obs_pack=18.754s feedback_info_pack=2.952s effective_fps=161.05 +2026-09-28 17:58:25.390 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=180997 infer_calls=180997 feedback_calls=180997 infer_wait=937.172s infer_obs_pack=16.863s env_step=94.044s feedback_total=75.476s feedback_obs_pack=19.300s feedback_info_pack=3.034s effective_fps=161.09 +2026-09-28 17:58:55.395 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=186116 infer_calls=186116 feedback_calls=186116 infer_wait=961.425s infer_obs_pack=17.282s env_step=96.341s feedback_total=77.368s feedback_obs_pack=19.783s feedback_info_pack=3.107s effective_fps=161.50 +2026-09-28 17:59:25.396 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=191480 infer_calls=191480 feedback_calls=191480 infer_wait=985.405s infer_obs_pack=17.729s env_step=98.722s feedback_total=79.358s feedback_obs_pack=20.290s feedback_info_pack=3.186s effective_fps=162.10 +2026-09-28 17:59:55.995 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=196610 infer_calls=196610 feedback_calls=196610 infer_wait=1010.310s infer_obs_pack=18.146s env_step=100.974s feedback_total=81.243s feedback_obs_pack=20.776s feedback_info_pack=3.260s effective_fps=162.40 +2026-09-28 18:00:26.000 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=202128 infer_calls=202128 feedback_calls=202128 infer_wait=1034.641s infer_obs_pack=18.566s env_step=103.210s feedback_total=83.134s feedback_obs_pack=21.255s feedback_info_pack=3.336s effective_fps=163.07 +2026-09-28 18:00:56.001 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=207714 infer_calls=207714 feedback_calls=207714 infer_wait=1058.912s infer_obs_pack=18.990s env_step=105.475s feedback_total=85.033s feedback_obs_pack=21.739s feedback_info_pack=3.411s effective_fps=163.76 +2026-09-28 18:01:26.163 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=212994 infer_calls=212994 feedback_calls=212994 infer_wait=1083.669s infer_obs_pack=19.389s env_step=107.605s feedback_total=86.834s feedback_obs_pack=22.199s feedback_info_pack=3.483s effective_fps=164.16 +2026-09-28 18:01:56.163 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=218117 infer_calls=218117 feedback_calls=218117 infer_wait=1108.383s infer_obs_pack=19.782s env_step=109.690s feedback_total=88.590s feedback_obs_pack=22.645s feedback_info_pack=3.552s effective_fps=164.44 +2026-09-28 18:02:26.163 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=223774 infer_calls=223774 feedback_calls=223774 infer_wait=1132.405s infer_obs_pack=20.225s env_step=112.056s feedback_total=90.577s feedback_obs_pack=23.154s feedback_info_pack=3.633s effective_fps=165.11 +2026-09-28 18:02:56.167 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=228873 infer_calls=228873 feedback_calls=228873 infer_wait=1156.540s infer_obs_pack=20.650s env_step=114.386s feedback_total=92.523s feedback_obs_pack=23.647s feedback_info_pack=3.710s effective_fps=165.36 +2026-09-28 18:03:26.167 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=233529 infer_calls=233529 feedback_calls=233529 infer_wait=1180.725s infer_obs_pack=21.106s env_step=116.682s feedback_total=94.423s feedback_obs_pack=24.141s feedback_info_pack=3.783s effective_fps=165.28 +2026-09-28 18:03:56.174 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=239004 infer_calls=239004 feedback_calls=239004 infer_wait=1204.706s infer_obs_pack=21.552s env_step=119.069s feedback_total=96.418s feedback_obs_pack=24.660s feedback_info_pack=3.863s effective_fps=165.77 +2026-09-28 18:04:26.177 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=244657 infer_calls=244657 feedback_calls=244657 infer_wait=1228.898s infer_obs_pack=21.987s env_step=121.360s feedback_total=98.346s feedback_obs_pack=25.156s feedback_info_pack=3.938s effective_fps=166.37 +2026-09-28 18:04:56.180 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=249869 infer_calls=249869 feedback_calls=249869 infer_wait=1253.399s infer_obs_pack=22.393s env_step=123.536s feedback_total=100.165s feedback_obs_pack=25.622s feedback_info_pack=4.010s effective_fps=166.64 +2026-09-28 18:05:26.182 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=255308 infer_calls=255308 feedback_calls=255308 infer_wait=1277.313s infer_obs_pack=22.845s env_step=125.952s feedback_total=102.173s feedback_obs_pack=26.137s feedback_info_pack=4.093s effective_fps=167.06 +2026-09-28 18:05:56.183 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=260621 infer_calls=260621 feedback_calls=260621 infer_wait=1301.176s infer_obs_pack=23.294s env_step=128.380s feedback_total=104.199s feedback_obs_pack=26.657s feedback_info_pack=4.175s effective_fps=167.38 +2026-09-28 18:06:26.187 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=265107 infer_calls=265107 feedback_calls=265107 infer_wait=1324.606s infer_obs_pack=23.766s env_step=131.066s feedback_total=106.275s feedback_obs_pack=27.179s feedback_info_pack=4.257s effective_fps=167.18 +2026-09-28 18:06:56.189 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=269673 infer_calls=269673 feedback_calls=269673 infer_wait=1348.229s infer_obs_pack=24.229s env_step=133.630s feedback_total=108.367s feedback_obs_pack=27.702s feedback_info_pack=4.335s effective_fps=167.04 +2026-09-28 18:07:26.191 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=274461 infer_calls=274461 feedback_calls=274461 infer_wait=1372.472s infer_obs_pack=24.653s env_step=135.921s feedback_total=110.261s feedback_obs_pack=28.187s feedback_info_pack=4.411s effective_fps=167.02 +2026-09-28 18:07:56.249 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=279039 infer_calls=279039 feedback_calls=279039 infer_wait=1396.051s infer_obs_pack=25.179s env_step=138.560s feedback_total=112.356s feedback_obs_pack=28.710s feedback_info_pack=4.498s effective_fps=166.87 +2026-09-28 18:08:27.618 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=282626 infer_calls=282626 feedback_calls=282626 infer_wait=1420.437s infer_obs_pack=25.717s env_step=141.338s feedback_total=114.655s feedback_obs_pack=29.253s feedback_info_pack=4.568s effective_fps=166.04 +2026-09-28 18:08:57.624 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=287331 infer_calls=287331 feedback_calls=287331 infer_wait=1443.822s infer_obs_pack=26.233s env_step=143.974s feedback_total=116.769s feedback_obs_pack=29.800s feedback_info_pack=4.651s effective_fps=166.01 +2026-09-28 18:09:27.636 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=292045 infer_calls=292045 feedback_calls=292045 infer_wait=1467.239s infer_obs_pack=26.699s env_step=146.639s feedback_total=118.885s feedback_obs_pack=30.344s feedback_info_pack=4.734s effective_fps=165.99 +2026-09-28 18:09:57.639 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=296593 infer_calls=296593 feedback_calls=296593 infer_wait=1490.624s infer_obs_pack=27.155s env_step=149.331s feedback_total=121.024s feedback_obs_pack=30.914s feedback_info_pack=4.832s effective_fps=165.87 +2026-09-28 18:10:27.648 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=301655 infer_calls=301655 feedback_calls=301655 infer_wait=1514.191s infer_obs_pack=27.627s env_step=151.919s feedback_total=123.083s feedback_obs_pack=31.443s feedback_info_pack=4.914s effective_fps=166.03 +2026-09-28 18:10:57.648 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=306420 infer_calls=306420 feedback_calls=306420 infer_wait=1537.584s infer_obs_pack=28.101s env_step=154.594s feedback_total=125.181s feedback_obs_pack=32.002s feedback_info_pack=4.998s effective_fps=166.04 +2026-09-28 18:11:29.316 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=311298 infer_calls=311298 feedback_calls=311298 infer_wait=1562.489s infer_obs_pack=28.605s env_step=157.312s feedback_total=127.349s feedback_obs_pack=32.573s feedback_info_pack=5.087s effective_fps=165.96 +2026-09-28 18:11:59.318 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=315878 infer_calls=315878 feedback_calls=315878 infer_wait=1586.097s infer_obs_pack=29.063s env_step=159.899s feedback_total=129.380s feedback_obs_pack=33.101s feedback_info_pack=5.169s effective_fps=165.86 +2026-09-28 18:12:29.320 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=320393 infer_calls=320393 feedback_calls=320393 infer_wait=1609.888s infer_obs_pack=29.498s env_step=162.385s feedback_total=131.400s feedback_obs_pack=33.614s feedback_info_pack=5.257s effective_fps=165.73 +2026-09-28 18:12:59.326 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=325101 infer_calls=325101 feedback_calls=325101 infer_wait=1634.344s infer_obs_pack=29.891s env_step=164.613s feedback_total=133.206s feedback_obs_pack=34.075s feedback_info_pack=5.328s effective_fps=165.69 +2026-09-28 18:13:29.327 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=330192 infer_calls=330192 feedback_calls=330192 infer_wait=1658.470s infer_obs_pack=30.314s env_step=166.971s feedback_total=135.127s feedback_obs_pack=34.575s feedback_info_pack=5.404s effective_fps=165.85 +2026-09-28 18:13:59.329 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=335415 infer_calls=335415 feedback_calls=335415 infer_wait=1682.470s infer_obs_pack=30.757s env_step=169.354s feedback_total=137.102s feedback_obs_pack=35.084s feedback_info_pack=5.484s effective_fps=166.07 +2026-09-28 18:14:29.334 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=340038 infer_calls=340038 feedback_calls=340038 infer_wait=1706.371s infer_obs_pack=31.194s env_step=171.797s feedback_total=139.077s feedback_obs_pack=35.593s feedback_info_pack=5.563s effective_fps=166.00 +2026-09-28 18:14:59.335 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=345274 infer_calls=345274 feedback_calls=345274 infer_wait=1730.045s infer_obs_pack=31.653s env_step=174.322s feedback_total=141.146s feedback_obs_pack=36.129s feedback_info_pack=5.647s effective_fps=166.22 +2026-09-28 18:15:29.335 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=350592 infer_calls=350592 feedback_calls=350592 infer_wait=1754.268s infer_obs_pack=32.079s env_step=176.592s feedback_total=143.072s feedback_obs_pack=36.629s feedback_info_pack=5.725s effective_fps=166.47 +2026-09-28 18:15:59.336 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=355976 infer_calls=355976 feedback_calls=355976 infer_wait=1778.648s infer_obs_pack=32.496s env_step=178.805s feedback_total=144.942s feedback_obs_pack=37.104s feedback_info_pack=5.799s effective_fps=166.74 +2026-09-28 18:16:29.343 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=361010 infer_calls=361010 feedback_calls=361010 infer_wait=1803.442s infer_obs_pack=32.884s env_step=180.858s feedback_total=146.675s feedback_obs_pack=37.545s feedback_info_pack=5.868s effective_fps=166.84 +2026-09-28 18:16:59.348 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=366441 infer_calls=366441 feedback_calls=366441 infer_wait=1827.852s infer_obs_pack=33.305s env_step=183.048s feedback_total=148.550s feedback_obs_pack=38.023s feedback_info_pack=5.945s effective_fps=167.11 +2026-09-28 18:17:29.350 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=371111 infer_calls=371111 feedback_calls=371111 infer_wait=1851.596s infer_obs_pack=33.774s env_step=185.561s feedback_total=150.569s feedback_obs_pack=38.541s feedback_info_pack=6.027s effective_fps=167.05 +2026-09-28 18:17:59.357 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=376534 infer_calls=376534 feedback_calls=376534 infer_wait=1875.941s infer_obs_pack=34.200s env_step=187.781s feedback_total=152.460s feedback_obs_pack=39.029s feedback_info_pack=6.103s effective_fps=167.32 +2026-09-28 18:18:29.360 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=381578 infer_calls=381578 feedback_calls=381578 infer_wait=1900.384s infer_obs_pack=34.608s env_step=189.979s feedback_total=154.305s feedback_obs_pack=39.504s feedback_info_pack=6.178s effective_fps=167.41 +2026-09-28 18:18:59.364 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=386800 infer_calls=386800 feedback_calls=386800 infer_wait=1923.777s infer_obs_pack=35.103s env_step=192.595s feedback_total=156.480s feedback_obs_pack=40.061s feedback_info_pack=6.264s effective_fps=167.59 +2026-09-28 18:19:29.366 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=392104 infer_calls=392104 feedback_calls=392104 infer_wait=1947.875s infer_obs_pack=35.531s env_step=194.931s feedback_total=158.429s feedback_obs_pack=40.560s feedback_info_pack=6.342s effective_fps=167.80 +2026-09-28 18:20:00.590 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=397314 infer_calls=397314 feedback_calls=397314 infer_wait=1973.033s infer_obs_pack=35.972s env_step=197.344s feedback_total=160.410s feedback_obs_pack=41.066s feedback_info_pack=6.421s effective_fps=167.87 +2026-09-28 18:20:30.591 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=401901 infer_calls=401901 feedback_calls=401901 infer_wait=1996.668s infer_obs_pack=36.423s env_step=199.922s feedback_total=162.468s feedback_obs_pack=41.602s feedback_info_pack=6.511s effective_fps=167.77 +2026-09-28 18:21:00.595 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=406987 infer_calls=406987 feedback_calls=406987 infer_wait=2020.068s infer_obs_pack=36.892s env_step=202.564s feedback_total=164.633s feedback_obs_pack=42.148s feedback_info_pack=6.598s effective_fps=167.89 +2026-09-28 18:21:16.444 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=2031.168s infer_obs_pack=37.112s env_step=203.765s feedback_total=165.633s feedback_obs_pack=42.407s feedback_info_pack=6.638s effective_fps=168.03 +2026-09-28 18:21:16.446 | INFO  | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client-seed2.log b/experiments/e43-cross-machine-training/results/cross-clients/client-seed2.log new file mode 100644 index 0000000..ededd21 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client-seed2.log @@ -0,0 +1,95 @@ +2026-09-28 17:39:00.144 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 17:39:00.147 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 17:39:00.149 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 17:39:00.155 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 17:39:00.163 | WARNING  | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 17:39:00.474 | INFO  | __main__:main:83 - Starting env client exp_name=e43-cross-seed2 output_dir=runs\e43-cross-seed2 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 17:39:00.644 | INFO  | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://100.75.226.89:9792... +2026-09-28 17:39:00.700 | INFO  | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 17:39:30.716 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=822 infer_calls=822 feedback_calls=822 infer_wait=28.471s infer_obs_pack=0.108s env_step=0.655s feedback_total=0.492s feedback_obs_pack=0.110s feedback_info_pack=0.019s effective_fps=27.65 +2026-09-28 17:40:01.560 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=4097 infer_calls=4097 feedback_calls=4097 infer_wait=53.587s infer_obs_pack=0.560s env_step=2.875s feedback_total=2.403s feedback_obs_pack=0.605s feedback_info_pack=0.090s effective_fps=68.94 +2026-09-28 17:40:31.565 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=8608 infer_calls=8608 feedback_calls=8608 infer_wait=76.649s infer_obs_pack=1.063s env_step=5.633s feedback_total=4.654s feedback_obs_pack=1.172s feedback_info_pack=0.179s effective_fps=97.82 +2026-09-28 17:41:01.569 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=13288 infer_calls=13288 feedback_calls=13288 infer_wait=100.121s infer_obs_pack=1.578s env_step=8.226s feedback_total=6.775s feedback_obs_pack=1.714s feedback_info_pack=0.267s effective_fps=113.86 +2026-09-28 17:41:31.576 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=18144 infer_calls=18144 feedback_calls=18144 infer_wait=123.639s infer_obs_pack=2.053s env_step=10.862s feedback_total=8.860s feedback_obs_pack=2.239s feedback_info_pack=0.352s effective_fps=124.77 +2026-09-28 17:42:01.578 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=23032 infer_calls=23032 feedback_calls=23032 infer_wait=148.159s infer_obs_pack=2.456s env_step=13.044s feedback_total=10.671s feedback_obs_pack=2.702s feedback_info_pack=0.424s effective_fps=132.12 +2026-09-28 17:42:31.578 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=28065 infer_calls=28065 feedback_calls=28065 infer_wait=171.807s infer_obs_pack=2.921s env_step=15.591s feedback_total=12.738s feedback_obs_pack=3.236s feedback_info_pack=0.507s effective_fps=138.21 +2026-09-28 17:43:01.784 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=32770 infer_calls=32770 feedback_calls=32770 infer_wait=196.119s infer_obs_pack=3.344s env_step=17.957s feedback_total=14.661s feedback_obs_pack=3.722s feedback_info_pack=0.588s effective_fps=141.20 +2026-09-28 17:43:31.788 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=37488 infer_calls=37488 feedback_calls=37488 infer_wait=219.933s infer_obs_pack=3.804s env_step=20.436s feedback_total=16.679s feedback_obs_pack=4.218s feedback_info_pack=0.675s effective_fps=143.71 +2026-09-28 17:44:01.793 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=42322 infer_calls=42322 feedback_calls=42322 infer_wait=243.036s infer_obs_pack=4.301s env_step=23.221s feedback_total=18.901s feedback_obs_pack=4.783s feedback_info_pack=0.768s effective_fps=146.21 +2026-09-28 17:44:31.795 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=47573 infer_calls=47573 feedback_calls=47573 infer_wait=266.467s infer_obs_pack=4.769s env_step=25.858s feedback_total=21.053s feedback_obs_pack=5.326s feedback_info_pack=0.854s effective_fps=149.53 +2026-09-28 17:45:01.797 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=52153 infer_calls=52153 feedback_calls=52153 infer_wait=289.911s infer_obs_pack=5.230s env_step=28.510s feedback_total=23.164s feedback_obs_pack=5.872s feedback_info_pack=0.939s effective_fps=150.38 +2026-09-28 17:45:31.800 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=57345 infer_calls=57345 feedback_calls=57345 infer_wait=314.027s infer_obs_pack=5.647s env_step=30.862s feedback_total=25.102s feedback_obs_pack=6.370s feedback_info_pack=1.017s effective_fps=152.66 +2026-09-28 17:46:01.803 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=61774 infer_calls=61774 feedback_calls=61774 infer_wait=337.987s infer_obs_pack=6.067s env_step=33.303s feedback_total=27.050s feedback_obs_pack=6.875s feedback_info_pack=1.095s effective_fps=152.75 +2026-09-28 17:46:31.807 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=66671 infer_calls=66671 feedback_calls=66671 infer_wait=361.497s infer_obs_pack=6.537s env_step=35.925s feedback_total=29.155s feedback_obs_pack=7.403s feedback_info_pack=1.180s effective_fps=153.93 +2026-09-28 17:47:01.812 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=71362 infer_calls=71362 feedback_calls=71362 infer_wait=384.993s infer_obs_pack=7.003s env_step=38.566s feedback_total=31.246s feedback_obs_pack=7.937s feedback_info_pack=1.265s effective_fps=154.53 +2026-09-28 17:47:31.821 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=75993 infer_calls=75993 feedback_calls=75993 infer_wait=408.767s infer_obs_pack=7.459s env_step=41.096s feedback_total=33.249s feedback_obs_pack=8.429s feedback_info_pack=1.346s effective_fps=154.91 +2026-09-28 17:48:01.831 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=80980 infer_calls=80980 feedback_calls=80980 infer_wait=432.255s infer_obs_pack=7.918s env_step=43.764s feedback_total=35.339s feedback_obs_pack=8.965s feedback_info_pack=1.432s effective_fps=155.95 +2026-09-28 17:48:31.835 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=85967 infer_calls=85967 feedback_calls=85967 infer_wait=456.749s infer_obs_pack=8.317s env_step=45.962s feedback_total=37.169s feedback_obs_pack=9.423s feedback_info_pack=1.503s effective_fps=156.82 +2026-09-28 17:49:01.835 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=90570 infer_calls=90570 feedback_calls=90570 infer_wait=480.854s infer_obs_pack=8.731s env_step=48.366s feedback_total=39.062s feedback_obs_pack=9.906s feedback_info_pack=1.582s effective_fps=156.96 +2026-09-28 17:49:31.838 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=95389 infer_calls=95389 feedback_calls=95389 infer_wait=505.305s infer_obs_pack=9.130s env_step=50.596s feedback_total=40.911s feedback_obs_pack=10.363s feedback_info_pack=1.656s effective_fps=157.42 +2026-09-28 17:50:01.843 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=100678 infer_calls=100678 feedback_calls=100678 infer_wait=529.709s infer_obs_pack=9.531s env_step=52.835s feedback_total=42.760s feedback_obs_pack=10.832s feedback_info_pack=1.730s effective_fps=158.59 +2026-09-28 17:50:31.844 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=106054 infer_calls=106054 feedback_calls=106054 infer_wait=553.569s infer_obs_pack=9.969s env_step=55.289s feedback_total=44.787s feedback_obs_pack=11.341s feedback_info_pack=1.811s effective_fps=159.81 +2026-09-28 17:51:01.844 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=110762 infer_calls=110762 feedback_calls=110762 infer_wait=577.796s infer_obs_pack=10.397s env_step=57.595s feedback_total=46.676s feedback_obs_pack=11.819s feedback_info_pack=1.887s effective_fps=159.95 +2026-09-28 17:51:31.847 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=116053 infer_calls=116053 feedback_calls=116053 infer_wait=602.237s infer_obs_pack=10.799s env_step=59.801s feedback_total=48.536s feedback_obs_pack=12.292s feedback_info_pack=1.962s effective_fps=160.88 +2026-09-28 17:52:01.849 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=120959 infer_calls=120959 feedback_calls=120959 infer_wait=626.926s infer_obs_pack=11.170s env_step=61.933s feedback_total=50.294s feedback_obs_pack=12.743s feedback_info_pack=2.033s effective_fps=161.21 +2026-09-28 17:52:31.851 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=125190 infer_calls=125190 feedback_calls=125190 infer_wait=651.251s infer_obs_pack=11.603s env_step=64.199s feedback_total=52.153s feedback_obs_pack=13.216s feedback_info_pack=2.109s effective_fps=160.66 +2026-09-28 17:53:01.862 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=129988 infer_calls=129988 feedback_calls=129988 infer_wait=675.363s infer_obs_pack=12.011s env_step=66.585s feedback_total=54.092s feedback_obs_pack=13.704s feedback_info_pack=2.197s effective_fps=160.87 +2026-09-28 17:53:31.868 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=134880 infer_calls=134880 feedback_calls=134880 infer_wait=699.323s infer_obs_pack=12.481s env_step=69.004s feedback_total=56.059s feedback_obs_pack=14.215s feedback_info_pack=2.277s effective_fps=161.17 +2026-09-28 17:54:01.868 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=139294 infer_calls=139294 feedback_calls=139294 infer_wait=723.091s infer_obs_pack=12.918s env_step=71.516s feedback_total=58.073s feedback_obs_pack=14.728s feedback_info_pack=2.363s effective_fps=160.92 +2026-09-28 17:54:31.868 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=144435 infer_calls=144435 feedback_calls=144435 infer_wait=747.342s infer_obs_pack=13.353s env_step=73.785s feedback_total=59.982s feedback_obs_pack=15.218s feedback_info_pack=2.442s effective_fps=161.48 +2026-09-28 17:55:01.872 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=149523 infer_calls=149523 feedback_calls=149523 infer_wait=771.097s infer_obs_pack=13.798s env_step=76.270s feedback_total=62.051s feedback_obs_pack=15.735s feedback_info_pack=2.523s effective_fps=161.96 +2026-09-28 17:55:31.872 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=154920 infer_calls=154920 feedback_calls=154920 infer_wait=795.319s infer_obs_pack=14.231s env_step=78.547s feedback_total=63.976s feedback_obs_pack=16.221s feedback_info_pack=2.601s effective_fps=162.72 +2026-09-28 17:56:01.875 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=159298 infer_calls=159298 feedback_calls=159298 infer_wait=818.394s infer_obs_pack=14.723s env_step=81.374s feedback_total=66.207s feedback_obs_pack=16.792s feedback_info_pack=2.696s effective_fps=162.43 +2026-09-28 17:56:31.877 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163976 infer_calls=163976 feedback_calls=163976 infer_wait=842.908s infer_obs_pack=15.111s env_step=83.552s feedback_total=68.028s feedback_obs_pack=17.257s feedback_info_pack=2.772s effective_fps=162.42 +2026-09-28 17:57:01.881 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=167970 infer_calls=167970 feedback_calls=167970 infer_wait=866.369s infer_obs_pack=15.575s env_step=86.219s feedback_total=70.149s feedback_obs_pack=17.835s feedback_info_pack=2.858s effective_fps=161.77 +2026-09-28 17:57:31.885 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=172724 infer_calls=172724 feedback_calls=172724 infer_wait=890.435s infer_obs_pack=16.010s env_step=88.590s feedback_total=72.099s feedback_obs_pack=18.348s feedback_info_pack=2.940s effective_fps=161.86 +2026-09-28 17:58:02.012 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=177131 infer_calls=177131 feedback_calls=177131 infer_wait=914.229s infer_obs_pack=16.479s env_step=91.132s feedback_total=74.146s feedback_obs_pack=18.881s feedback_info_pack=3.016s effective_fps=161.62 +2026-09-28 17:58:32.015 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=182081 infer_calls=182081 feedback_calls=182081 infer_wait=937.839s infer_obs_pack=16.951s env_step=93.665s feedback_total=76.242s feedback_obs_pack=19.416s feedback_info_pack=3.101s effective_fps=161.89 +2026-09-28 17:59:02.018 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=186802 infer_calls=186802 feedback_calls=186802 infer_wait=962.636s infer_obs_pack=17.334s env_step=95.714s feedback_total=77.987s feedback_obs_pack=19.866s feedback_info_pack=3.172s effective_fps=161.92 +2026-09-28 17:59:32.020 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=191509 infer_calls=191509 feedback_calls=191509 infer_wait=987.397s infer_obs_pack=17.719s env_step=97.769s feedback_total=79.743s feedback_obs_pack=20.310s feedback_info_pack=3.243s effective_fps=161.94 +2026-09-28 18:00:02.020 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=196275 infer_calls=196275 feedback_calls=196275 infer_wait=1012.168s infer_obs_pack=18.101s env_step=99.824s feedback_total=81.479s feedback_obs_pack=20.754s feedback_info_pack=3.313s effective_fps=162.00 +2026-09-28 18:00:33.524 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=200706 infer_calls=200706 feedback_calls=200706 infer_wait=1039.178s infer_obs_pack=18.434s env_step=101.594s feedback_total=82.984s feedback_obs_pack=21.137s feedback_info_pack=3.376s effective_fps=161.57 +2026-09-28 18:01:03.531 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=205618 infer_calls=205618 feedback_calls=205618 infer_wait=1064.172s infer_obs_pack=18.803s env_step=103.550s feedback_total=84.667s feedback_obs_pack=21.564s feedback_info_pack=3.446s effective_fps=161.75 +2026-09-28 18:01:33.532 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=210751 infer_calls=210751 feedback_calls=210751 infer_wait=1088.955s infer_obs_pack=19.194s env_step=105.594s feedback_total=86.415s feedback_obs_pack=22.009s feedback_info_pack=3.521s effective_fps=162.10 +2026-09-28 18:02:03.537 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=215858 infer_calls=215858 feedback_calls=215858 infer_wait=1113.655s infer_obs_pack=19.589s env_step=107.663s feedback_total=88.196s feedback_obs_pack=22.466s feedback_info_pack=3.595s effective_fps=162.41 +2026-09-28 18:02:36.893 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=221186 infer_calls=221186 feedback_calls=221186 infer_wait=1141.375s infer_obs_pack=20.013s env_step=109.855s feedback_total=90.089s feedback_obs_pack=22.945s feedback_info_pack=3.672s effective_fps=162.48 +2026-09-28 18:03:06.895 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=225846 infer_calls=225846 feedback_calls=225846 infer_wait=1165.725s infer_obs_pack=20.411s env_step=112.077s feedback_total=91.984s feedback_obs_pack=23.429s feedback_info_pack=3.747s effective_fps=162.46 +2026-09-28 18:03:36.897 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=230764 infer_calls=230764 feedback_calls=230764 infer_wait=1190.082s infer_obs_pack=20.821s env_step=114.297s feedback_total=93.872s feedback_obs_pack=23.917s feedback_info_pack=3.824s effective_fps=162.62 +2026-09-28 18:04:06.900 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=236084 infer_calls=236084 feedback_calls=236084 infer_wait=1214.476s infer_obs_pack=21.238s env_step=116.493s feedback_total=95.755s feedback_obs_pack=24.401s feedback_info_pack=3.900s effective_fps=163.05 +2026-09-28 18:04:36.901 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=241326 infer_calls=241326 feedback_calls=241326 infer_wait=1239.104s infer_obs_pack=21.635s env_step=118.600s feedback_total=97.564s feedback_obs_pack=24.860s feedback_info_pack=3.974s effective_fps=163.40 +2026-09-28 18:05:06.973 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=245762 infer_calls=245762 feedback_calls=245762 infer_wait=1264.415s infer_obs_pack=21.979s env_step=120.486s feedback_total=99.145s feedback_obs_pack=25.265s feedback_info_pack=4.039s effective_fps=163.19 +2026-09-28 18:05:36.973 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=251053 infer_calls=251053 feedback_calls=251053 infer_wait=1288.689s infer_obs_pack=22.405s env_step=122.735s feedback_total=101.056s feedback_obs_pack=25.754s feedback_info_pack=4.117s effective_fps=163.56 +2026-09-28 18:06:06.975 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=255991 infer_calls=255991 feedback_calls=255991 infer_wait=1312.483s infer_obs_pack=22.865s env_step=125.186s feedback_total=103.103s feedback_obs_pack=26.286s feedback_info_pack=4.201s effective_fps=163.72 +2026-09-28 18:06:36.979 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=260216 infer_calls=260216 feedback_calls=260216 infer_wait=1336.160s infer_obs_pack=23.336s env_step=127.758s feedback_total=105.129s feedback_obs_pack=26.808s feedback_info_pack=4.282s effective_fps=163.41 +2026-09-28 18:07:06.982 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=264639 infer_calls=264639 feedback_calls=264639 infer_wait=1359.856s infer_obs_pack=23.783s env_step=130.282s feedback_total=107.189s feedback_obs_pack=27.340s feedback_info_pack=4.368s effective_fps=163.25 +2026-09-28 18:07:36.985 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=269799 infer_calls=269799 feedback_calls=269799 infer_wait=1384.003s infer_obs_pack=24.210s env_step=132.591s feedback_total=109.132s feedback_obs_pack=27.836s feedback_info_pack=4.446s effective_fps=163.52 +2026-09-28 18:08:06.989 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=272755 infer_calls=272755 feedback_calls=272755 infer_wait=1407.154s infer_obs_pack=24.781s env_step=135.345s feedback_total=111.372s feedback_obs_pack=28.505s feedback_info_pack=4.514s effective_fps=162.48 +2026-09-28 18:08:36.995 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=277131 infer_calls=277131 feedback_calls=277131 infer_wait=1430.549s infer_obs_pack=25.273s env_step=137.988s feedback_total=113.499s feedback_obs_pack=29.058s feedback_info_pack=4.602s effective_fps=162.32 +2026-09-28 18:09:06.996 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=281802 infer_calls=281802 feedback_calls=281802 infer_wait=1454.179s infer_obs_pack=25.734s env_step=140.523s feedback_total=115.598s feedback_obs_pack=29.590s feedback_info_pack=4.694s effective_fps=162.33 +2026-09-28 18:09:36.995 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=286197 infer_calls=286197 feedback_calls=286197 infer_wait=1477.557s infer_obs_pack=26.214s env_step=143.195s feedback_total=117.727s feedback_obs_pack=30.152s feedback_info_pack=4.775s effective_fps=162.18 +2026-09-28 18:10:07.367 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=290818 infer_calls=290818 feedback_calls=290818 infer_wait=1501.990s infer_obs_pack=26.655s env_step=145.564s feedback_total=119.665s feedback_obs_pack=30.651s feedback_info_pack=4.853s effective_fps=162.12 +2026-09-28 18:10:37.371 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=295610 infer_calls=295610 feedback_calls=295610 infer_wait=1525.681s infer_obs_pack=27.113s env_step=148.089s feedback_total=121.704s feedback_obs_pack=31.171s feedback_info_pack=4.935s effective_fps=162.19 +2026-09-28 18:11:07.371 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=300398 infer_calls=300398 feedback_calls=300398 infer_wait=1549.095s infer_obs_pack=27.591s env_step=150.755s feedback_total=123.808s feedback_obs_pack=31.719s feedback_info_pack=5.022s effective_fps=162.27 +2026-09-28 18:11:37.375 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=305223 infer_calls=305223 feedback_calls=305223 infer_wait=1572.774s infer_obs_pack=28.063s env_step=153.277s feedback_total=125.847s feedback_obs_pack=32.241s feedback_info_pack=5.107s effective_fps=162.36 +2026-09-28 18:12:07.376 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=309758 infer_calls=309758 feedback_calls=309758 infer_wait=1596.395s infer_obs_pack=28.514s env_step=155.857s feedback_total=127.883s feedback_obs_pack=32.763s feedback_info_pack=5.188s effective_fps=162.29 +2026-09-28 18:12:37.378 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=314086 infer_calls=314086 feedback_calls=314086 infer_wait=1620.581s infer_obs_pack=28.937s env_step=158.151s feedback_total=129.823s feedback_obs_pack=33.286s feedback_info_pack=5.262s effective_fps=162.11 +2026-09-28 18:13:07.384 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=318724 infer_calls=318724 feedback_calls=318724 infer_wait=1645.105s infer_obs_pack=29.323s env_step=160.335s feedback_total=131.624s feedback_obs_pack=33.742s feedback_info_pack=5.334s effective_fps=162.09 +2026-09-28 18:13:38.278 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=323586 infer_calls=323586 feedback_calls=323586 infer_wait=1670.517s infer_obs_pack=29.723s env_step=162.504s feedback_total=133.436s feedback_obs_pack=34.206s feedback_info_pack=5.408s effective_fps=162.10 +2026-09-28 18:14:08.281 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=328630 infer_calls=328630 feedback_calls=328630 infer_wait=1694.227s infer_obs_pack=30.188s env_step=164.994s feedback_total=135.494s feedback_obs_pack=34.733s feedback_info_pack=5.491s effective_fps=162.29 +2026-09-28 18:14:38.283 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=333534 infer_calls=333534 feedback_calls=333534 infer_wait=1717.713s infer_obs_pack=30.666s env_step=167.575s feedback_total=137.630s feedback_obs_pack=35.277s feedback_info_pack=5.580s effective_fps=162.42 +2026-09-28 18:15:08.287 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=338945 infer_calls=338945 feedback_calls=338945 infer_wait=1741.806s infer_obs_pack=31.099s env_step=169.902s feedback_total=139.607s feedback_obs_pack=35.780s feedback_info_pack=5.661s effective_fps=162.77 +2026-09-28 18:15:40.048 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=344066 infer_calls=344066 feedback_calls=344066 infer_wait=1768.025s infer_obs_pack=31.507s env_step=172.080s feedback_total=141.466s feedback_obs_pack=36.258s feedback_info_pack=5.736s effective_fps=162.83 +2026-09-28 18:16:10.049 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=349394 infer_calls=349394 feedback_calls=349394 infer_wait=1792.494s infer_obs_pack=31.919s env_step=174.252s feedback_total=143.317s feedback_obs_pack=36.731s feedback_info_pack=5.812s effective_fps=163.12 +2026-09-28 18:16:40.050 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=354646 infer_calls=354646 feedback_calls=354646 infer_wait=1817.131s infer_obs_pack=32.318s env_step=176.353s feedback_total=145.129s feedback_obs_pack=37.197s feedback_info_pack=5.887s effective_fps=163.36 +2026-09-28 18:17:10.055 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=359868 infer_calls=359868 feedback_calls=359868 infer_wait=1841.565s infer_obs_pack=32.732s env_step=178.542s feedback_total=146.982s feedback_obs_pack=37.668s feedback_info_pack=5.961s effective_fps=163.59 +2026-09-28 18:17:41.487 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=364546 infer_calls=364546 feedback_calls=364546 infer_wait=1866.935s infer_obs_pack=33.181s env_step=180.958s feedback_total=148.957s feedback_obs_pack=38.187s feedback_info_pack=6.041s effective_fps=163.47 +2026-09-28 18:18:11.489 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=369868 infer_calls=369868 feedback_calls=369868 infer_wait=1891.164s infer_obs_pack=33.603s env_step=183.229s feedback_total=150.886s feedback_obs_pack=38.684s feedback_info_pack=6.119s effective_fps=163.74 +2026-09-28 18:18:41.491 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=374903 infer_calls=374903 feedback_calls=374903 infer_wait=1914.731s infer_obs_pack=34.091s env_step=185.777s feedback_total=153.002s feedback_obs_pack=39.231s feedback_info_pack=6.206s effective_fps=163.88 +2026-09-28 18:19:11.495 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=380381 infer_calls=380381 feedback_calls=380381 infer_wait=1938.751s infer_obs_pack=34.531s env_step=188.127s feedback_total=155.005s feedback_obs_pack=39.740s feedback_info_pack=6.289s effective_fps=164.21 +2026-09-28 18:19:41.498 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=385099 infer_calls=385099 feedback_calls=385099 infer_wait=1962.936s infer_obs_pack=34.944s env_step=190.441s feedback_total=156.924s feedback_obs_pack=40.223s feedback_info_pack=6.367s effective_fps=164.20 +2026-09-28 18:20:11.504 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=390300 infer_calls=390300 feedback_calls=390300 infer_wait=1987.074s infer_obs_pack=35.366s env_step=192.772s feedback_total=158.865s feedback_obs_pack=40.727s feedback_info_pack=6.447s effective_fps=164.40 +2026-09-28 18:20:41.506 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=394828 infer_calls=394828 feedback_calls=394828 infer_wait=2010.433s infer_obs_pack=35.829s env_step=195.439s feedback_total=161.028s feedback_obs_pack=41.298s feedback_info_pack=6.534s effective_fps=164.32 +2026-09-28 18:21:11.512 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=399905 infer_calls=399905 feedback_calls=399905 infer_wait=2034.364s infer_obs_pack=36.259s env_step=197.845s feedback_total=163.050s feedback_obs_pack=41.813s feedback_info_pack=6.617s effective_fps=164.47 +2026-09-28 18:21:41.516 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=405296 infer_calls=405296 feedback_calls=405296 infer_wait=2057.844s infer_obs_pack=36.708s env_step=200.428s feedback_total=165.233s feedback_obs_pack=42.354s feedback_info_pack=6.700s effective_fps=164.74 +2026-09-28 18:22:11.977 | INFO  | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409600 infer_calls=409600 feedback_calls=409600 infer_wait=2079.798s infer_obs_pack=37.166s env_step=203.037s feedback_total=167.377s feedback_obs_pack=42.870s feedback_info_pack=6.778s effective_fps=164.67 +2026-09-28 18:22:11.978 | INFO  | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed0.json b/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed0.json new file mode 100644 index 0000000..ab8952a --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed0.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-cross-seed0", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 1814.383544921875, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 2034.7191001036845, + "infer_obs_pack_s": 37.187602982419776, + "env_step_s": 203.39059199477197, + "feedback_s": 166.08562310118577, + "feedback_obs_pack_s": 42.57605869916733, + "feedback_info_pack_s": 6.698729697585804, + "collect_time_s": 2441.382918182062, + "effective_fps": 167.77417296955738, + "infer_wait_frac": 0.8334289082430406, + "infer_obs_pack_frac": 0.015232187751240165, + "env_step_frac": 0.08330958264679907, + "feedback_frac": 0.06802932135892016 + } +} diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed1.json b/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed1.json new file mode 100644 index 0000000..461a46e --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed1.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-cross-seed1", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 2011.196533203125, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 2031.16819569387, + "infer_obs_pack_s": 37.11224539653631, + "env_step_s": 203.76540099648992, + "feedback_s": 165.632964603632, + "feedback_obs_pack_s": 42.40658090336365, + "feedback_info_pack_s": 6.638116796093527, + "collect_time_s": 2437.6788066905283, + "effective_fps": 168.02910985475054, + "infer_wait_frac": 0.8332386490455852, + "infer_obs_pack_frac": 0.015224419761404538, + "env_step_frac": 0.08358993007496686, + "feedback_frac": 0.06794700111804339 + } +} diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed2.json b/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed2.json new file mode 100644 index 0000000..0cffaaa --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client-summary-seed2.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-cross-seed2", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 903.5633544921875, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409600, + "infer_calls": 409600, + "feedback_calls": 409600, + "infer_wait_s": 2079.7975664928963, + "infer_obs_pack_s": 37.16626099994755, + "env_step_s": 203.03654488865868, + "feedback_s": 167.37722861059592, + "feedback_obs_pack_s": 42.87021679469035, + "feedback_info_pack_s": 6.777995801880024, + "collect_time_s": 2487.3776009920985, + "effective_fps": 164.6714193440633, + "infer_wait_frac": 0.8361406670476418, + "infer_obs_pack_frac": 0.01494194568011052, + "env_step_frac": 0.08162674810920421, + "feedback_frac": 0.06729063916304344 + } +} diff --git a/experiments/e43-cross-machine-training/results/cross-clients/client_side.out b/experiments/e43-cross-machine-training/results/cross-clients/client_side.out new file mode 100644 index 0000000..ea71042 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross-clients/client_side.out @@ -0,0 +1,9 @@ +client: C:/Users/75128/.claude/jobs/1fea619d/tmp/e43/client/.venv/Scripts/python.exe +server: 100.75.226.89 ports 9790+seed +out: D:/75128/Desktop/plugrl-work/plugrl-server/.claude/worktrees/fix-pi0-image-mask-batching/experiments/e43-cross-machine-training/results/cross-clients +start 2026-09-28 17:38:48 +seed 0 client rc=0 at 18:21:16 +seed 1 client rc=0 at 18:21:16 +seed 2 client rc=0 at 18:22:12 +end 2026-09-28 18:22:12 +CLIENTS_DONE diff --git a/experiments/e43-cross-machine-training/results/cross-machine.png b/experiments/e43-cross-machine-training/results/cross-machine.png new file mode 100644 index 0000000..980508f Binary files /dev/null and b/experiments/e43-cross-machine-training/results/cross-machine.png differ diff --git a/experiments/e43-cross-machine-training/results/cross.out b/experiments/e43-cross-machine-training/results/cross.out new file mode 100644 index 0000000..0c4003f --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross.out @@ -0,0 +1,14 @@ +arm: cross ports 9790+seed +server: /home/guangzhao/zuogou/plugrl/e43-reg/../plugrl-server/.venv/bin/python (src /home/guangzhao/zuogou/plugrl/e43-reg/src) +out: /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross +steps: 409600 seeds: 0 1 2 +start 2026-09-28 17:37:46 +READY seed 0 port 9790 at 17:37:49 +READY seed 1 port 9791 at 17:37:54 +READY seed 2 port 9792 at 17:37:59 +seed 0 server rc=0 at 18:20:28 +seed 1 server rc=0 at 18:20:28 +seed 2 server rc=0 at 18:21:24 +end 2026-09-28 18:21:24 +failed seeds: 0 +ARM_DONE cross diff --git a/experiments/e43-cross-machine-training/results/cross/server-seed0.log b/experiments/e43-cross-machine-training/results/cross/server-seed0.log new file mode 100644 index 0000000..3e36e9b --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross/server-seed0.log @@ -0,0 +1,23 @@ +17:37:48|INFO|plugrl_server version: 0.1.0 +17:37:48|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +17:37:48|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +17:37:48|INFO|Seeded python, numpy and torch with 0. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +17:37:48|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed0 +17:37:48|INFO|Policy created... +17:37:48|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +17:37:48|INFO|Algorithm created: + +17:37:48|INFO|Agent Server is listening on 0.0.0.0:9790 +17:47:21|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed0/81921 +17:55:44|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed0/163841 +18:03:46|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed0/245761 +18:12:28|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed0/327681 +18:20:28|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed0/409601 +18:20:28|INFO|Stopping server as the algorithm signaled to stop. +18:20:28|INFO|Shutdown started: aborting pending infer requests. +18:20:28|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +18:20:28|INFO|Shutdown closing 1 websocket connection(s). +18:20:28|INFO|Shutdown interrupted an in-flight inference request from ('100.65.224.34', 59143). +18:20:28|INFO|WebSocket server closed. +18:20:28|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/cross/server-seed1.log b/experiments/e43-cross-machine-training/results/cross/server-seed1.log new file mode 100644 index 0000000..2153e25 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross/server-seed1.log @@ -0,0 +1,23 @@ +17:37:53|INFO|plugrl_server version: 0.1.0 +17:37:53|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +17:37:53|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +17:37:53|INFO|Seeded python, numpy and torch with 1. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +17:37:53|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed1 +17:37:53|INFO|Policy created... +17:37:53|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +17:37:53|INFO|Algorithm created: + +17:37:53|INFO|Agent Server is listening on 0.0.0.0:9791 +17:47:21|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed1/81921 +17:55:42|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed1/163841 +18:03:45|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed1/245761 +18:12:28|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed1/327681 +18:20:28|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed1/409601 +18:20:28|INFO|Stopping server as the algorithm signaled to stop. +18:20:28|INFO|Shutdown started: aborting pending infer requests. +18:20:28|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +18:20:28|INFO|Shutdown closing 1 websocket connection(s). +18:20:28|INFO|Shutdown interrupted an in-flight inference request from ('100.65.224.34', 59146). +18:20:28|INFO|WebSocket server closed. +18:20:28|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/cross/server-seed2.log b/experiments/e43-cross-machine-training/results/cross/server-seed2.log new file mode 100644 index 0000000..8d0b78f --- /dev/null +++ b/experiments/e43-cross-machine-training/results/cross/server-seed2.log @@ -0,0 +1,23 @@ +17:37:58|INFO|plugrl_server version: 0.1.0 +17:37:58|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +17:37:58|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +17:37:58|INFO|Seeded python, numpy and torch with 2. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +17:37:58|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed2 +17:37:58|INFO|Policy created... +17:37:58|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +17:37:58|INFO|Algorithm created: + +17:37:58|INFO|Agent Server is listening on 0.0.0.0:9792 +17:47:22|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed2/81921 +17:55:43|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed2/163841 +18:04:18|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed2/245761 +18:13:14|INFO|Checkpoint saved at step 327680 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed2/327680 +18:21:24|INFO|Checkpoint saved at step 409600 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/cross/fpo/fpo-policy/halfcheetah-cross-seed2/409600 +18:21:24|INFO|Stopping server as the algorithm signaled to stop. +18:21:24|INFO|Shutdown started: aborting pending infer requests. +18:21:24|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +18:21:24|INFO|Shutdown closing 1 websocket connection(s). +18:21:24|INFO|Shutdown interrupted an in-flight inference request from ('100.65.224.34', 60912). +18:21:24|INFO|WebSocket server closed. +18:21:24|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/curves.json b/experiments/e43-cross-machine-training/results/curves.json new file mode 100644 index 0000000..759b740 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/curves.json @@ -0,0 +1 @@ +{"local": {"returns": [[-292.9, -265.8, -348.3, -326.5, -269.1, -309.9, -250.6, -319.2, -246.5, -209.5, -144.2, -119.4, -74.0, -33.3, -33.9, -12.3, 20.2, 81.3, -0.7, 171.6, 135.8, 255.5, 310.1, 406.0, 314.5, 326.6, 523.0, 375.7, 626.8, 703.5, -90.6, 12.0, 718.1, 588.5, 316.4, 843.0, 764.4, 605.5, 619.0, 128.6, 846.7, 667.8, 798.1, 614.4, 896.9, 859.8, 776.4, 934.0, 894.1, 462.3, 1256.7, 1244.2, 923.9, 625.9, 956.5, 585.0, 791.2, 1118.8, 788.8, 397.9, 622.8, 758.2, 878.7, 812.4, 573.9, 763.6, 1124.8, 645.8, 774.2, 434.0, 729.5, 818.3, 683.1, 1229.7, 1200.0, 849.3, 553.9, 966.7, 911.4, 1089.2, 1068.2, 522.2, 1433.1, 1173.5, 434.6, 849.3, 1543.5, 1461.3, 1118.3, 1651.5, 1188.1, 1570.7, 1320.2, 1318.6, 1587.1, 1553.3, 1202.9, 963.7, 996.0, 919.5], [-324.9, -240.9, -256.3, -332.4, -289.9, -263.6, -258.5, -217.1, -222.1, -144.2, -170.5, -112.6, -170.4, -143.1, -117.1, -41.2, -1.2, -166.0, 29.2, 93.2, 38.4, 254.9, 65.0, 200.8, 133.0, 98.6, 63.3, 375.7, 463.0, 422.3, 361.6, 389.2, 451.3, 549.7, 742.8, 309.9, 456.5, 550.2, 543.5, 197.4, 347.8, 551.6, 987.6, 359.5, 805.5, 366.0, 787.7, 510.9, 836.2, 824.1, 174.0, 523.2, 818.4, 635.6, 779.7, 368.4, 653.2, 729.1, 788.9, 509.7, 539.1, 1100.0, 1101.2, 1213.5, 578.8, 714.8, 483.6, 92.1, 596.8, 814.7, 992.6, 911.0, 988.3, 259.3, 498.3, 744.8, 427.8, 1023.0, 1032.8, 919.1, 846.2, 863.5, 880.1, 728.1, 531.2, 785.9, -39.0, 1205.6, 605.9, 772.9, 928.4, 839.2, 642.9, 848.8, 745.6, 1095.2, 105.0, 540.4, 532.2, 667.9], [-250.5, -366.4, -279.8, -279.1, -231.3, -280.0, -213.7, -206.8, -260.4, -227.5, -188.5, -204.6, -197.2, -156.8, -135.4, -110.9, -93.8, -124.6, -140.7, -128.5, -156.1, -123.4, -81.7, -64.4, -69.4, -4.1, -53.3, -74.7, 27.5, 25.7, 90.7, 32.1, 106.4, 223.2, 189.8, 156.4, 29.8, 39.6, 136.6, 119.3, 400.2, 310.2, 478.2, 436.1, 349.0, 249.3, 674.4, 704.0, 757.4, 721.6, 677.4, -63.4, 637.2, 133.1, 436.6, 897.6, 932.2, 1055.9, 93.0, 814.1, 629.6, 993.3, 525.8, 516.2, 716.9, 688.7, 400.3, 359.1, 337.7, 780.5, 442.1, 575.8, 415.5, 134.1, 679.7, 310.8, 445.0, 327.1, 991.1, 404.7, 567.3, 811.3, 171.5, 711.0, 417.2, 327.8, -29.4, 524.2, 384.8, 676.3, 731.1, 299.3, 636.0, 316.9, 585.9, 1052.6, 1052.1, 469.3, 965.6, 1144.4], [-321.3, -255.8, -229.9, -375.2, -261.7, -245.5, -186.8, -219.7, -199.5, -139.3, -166.4, -138.3, -103.9, -87.8, -11.8, -170.8, 3.0, 33.9, 95.1, 25.3, 142.8, 239.3, 123.1, 166.2, 271.5, 399.0, 462.9, 559.7, 579.3, 549.6, 748.2, 784.0, 821.8, 715.7, 883.2, 880.2, 968.4, 952.3, 983.1, 669.4, 1036.3, 994.3, 1082.1, 1191.7, 1177.8, 1052.8, 937.2, 1216.4, 1035.8, 1115.5, 1162.6, 1271.8, 1236.9, 1263.2, 1327.4, 1376.4, 1357.7, 1285.3, 1348.6, 1379.6, 1216.1, 1481.8, 1458.5, 1479.5, 1564.8, 1513.9, 1522.6, 1572.3, 1582.5, 1614.6, 1656.9, 1686.2, 1643.2, 1654.2, 1702.2, 1707.4, 1530.1, 1666.9, 1676.5, 1691.7, 1734.8, 1849.6, 1819.3, 1727.9, 1680.6, 1813.7, 1779.5, 1838.3, 1977.4, 1938.7, 1900.4, 1934.5, 1897.3, 1943.0, 2021.8, 1939.3, 1959.8, 2024.0, 2017.4, 2031.8], [-311.6, -293.7, -358.3, -264.5, -164.8, -205.2, -178.3, -157.0, -170.6, -141.6, -64.1, -80.6, 2.6, -43.8, 10.7, 75.0, 78.8, 122.0, 105.9, 239.4, 306.9, 342.5, 443.0, 558.7, 398.0, 641.2, 489.7, 571.4, 728.5, 750.6, 804.8, 814.8, 883.8, 959.5, 981.1, 1016.5, 1031.3, 1071.1, 1022.0, 1187.8, 1181.3, 1241.0, 1177.3, 1030.6, 1305.2, 1342.3, 1410.5, 1455.8, 1306.4, 1491.5, 1130.1, 1481.4, 1649.5, 1631.3, 1710.8, 1735.9, 1737.4, 1867.8, 1321.2, 1801.3, 1588.5, 1784.0, 1865.5, 1933.8, 1927.9, 1863.1, 2008.5, 2031.7, 1865.5, 2019.2, 2080.0, 2185.5, 2165.9, 2100.2, 2197.0, 2207.8, 2202.4, 1842.8, 2257.0, 2301.0, 2235.2, 2344.5, 2321.7, 2202.0, 2251.5, 1985.7, 2275.2, 2308.7, 2280.4, 2204.9, 2400.2, 2377.9, 2489.1, 2396.1, 2382.1, 1761.2, 2200.3, 2563.8, 2595.2, 2190.7], [-346.2, -234.7, -334.3, -298.1, -298.6, -304.2, -199.5, -234.2, -119.0, -180.2, -143.4, -129.3, -54.7, -45.2, 2.2, 94.8, 198.5, 265.5, 344.6, 260.8, 513.6, 312.3, 53.3, 32.9, 33.7, 398.9, 82.2, 524.3, 95.2, 358.0, 461.6, 362.1, 571.3, 188.2, 388.1, 175.9, 361.7, 289.2, 345.9, 532.2, 561.2, 366.4, 631.3, 651.0, 652.6, 524.1, 767.2, 437.9, 141.0, 649.0, 380.2, 594.1, 794.6, 707.3, 632.5, 875.3, 785.6, 742.9, 783.7, 639.5, 991.6, 1004.8, 562.3, 865.4, 845.0, 997.7, 669.1, 745.3, 1042.1, 562.9, 700.1, 629.0, 855.5, 1030.4, 335.3, 1176.5, 218.5, 860.1, 1114.0, 1186.7, 1197.7, 785.5, 875.2, 777.5, 762.8, 940.4, 807.4, 771.8, 1291.1, 1298.1, 668.2, 886.9, 1330.9, 1069.2, 925.8, 1108.6, 542.7, 1084.7, 627.5, 328.7]], "span_s": [1222.9, 1187.4, 1174.6, 1276.9, 1204.2, 1188.4]}, "cross": {"returns": [[-344.9, -305.7, -296.0, -347.8, -289.3, -342.7, -290.5, -294.1, -318.6, -245.1, -203.4, -193.2, -234.3, -195.2, -175.1, -171.6, -170.3, -57.4, -91.8, -64.6, -22.1, 84.2, 114.6, 237.9, 227.0, 232.6, 289.4, 247.0, 395.3, 500.4, 605.3, 617.9, 723.7, 806.8, 744.1, 806.1, 847.3, 906.9, 1010.3, 1003.5, 827.1, 1012.1, 854.6, 1148.1, 1048.6, 1096.1, 895.3, 782.1, 964.6, 1229.8, 853.5, 1112.5, 1172.3, 1279.1, 1391.0, 1437.8, 845.0, 1216.3, 1396.7, 1366.9, 1452.9, 1438.6, 1584.6, 1429.5, 1710.9, 1407.0, 1616.5, 1612.5, 1675.2, 1725.1, 1245.6, 1670.5, 1370.8, 1734.5, 1826.3, 1891.1, 1924.4, 1990.8, 1884.1, 921.9, 1200.5, 1778.0, 1929.0, 1427.0, 1774.2, 1930.7, 1516.5, 1635.0, 1284.8, 1709.7, 1893.6, 2039.4, 2097.3, 2111.2, 1625.6, 2138.8, 2191.0, 2307.1, 1886.9, 2305.5], [-318.4, -325.3, -226.1, -334.7, -328.2, -278.8, -142.7, -192.6, -173.9, -185.5, -114.2, -68.9, -51.7, -4.6, -63.9, 14.0, 51.1, 47.8, 149.6, 203.8, 323.4, 333.1, 195.5, 445.6, 563.3, 414.8, 570.1, 616.8, 649.1, 603.5, 770.5, 747.0, 558.8, 553.8, 859.0, 733.7, 572.0, 745.4, 767.6, 494.8, 900.7, 993.1, 1090.2, 999.9, 966.5, 1116.6, 1156.5, 1144.1, 1156.0, 1120.5, 1197.1, 1260.6, 1257.6, 1275.2, 1325.8, 1326.3, 1033.6, 1354.7, 1385.6, 1448.0, 1378.6, 1537.9, 1430.3, 1569.3, 1617.8, 1668.9, 1615.1, 1726.4, 1594.7, 1696.3, 1704.7, 1716.9, 1783.8, 1752.2, 1769.0, 1803.0, 1852.1, 1890.6, 1919.8, 1915.4, 1945.9, 1979.7, 1502.7, 1898.4, 2037.0, 2045.7, 1983.4, 2004.4, 2113.4, 2029.4, 2097.1, 2149.8, 2146.0, 2141.2, 2113.4, 2168.1, 1786.5, 2281.4, 2151.1, 2232.8], [-249.8, -302.3, -271.4, -310.5, -237.5, -273.5, -237.9, -192.1, -306.9, -235.4, -198.1, -250.9, -219.5, -179.3, -216.9, -165.7, -171.3, -127.8, -173.9, -138.3, -109.0, -79.1, -113.8, -54.2, -60.5, 72.5, 52.2, 12.9, -24.0, 143.8, 131.9, 208.6, 196.5, 167.1, 310.4, 295.4, 287.5, 382.0, 390.3, 432.1, 432.1, 374.3, 276.1, 481.4, 519.8, 562.5, 537.3, 478.7, 590.7, 628.5, 661.0, 630.8, 608.8, 689.7, 635.3, 651.7, 624.1, 666.3, 655.1, 630.7, 796.0, 679.7, 774.3, 763.6, 785.7, 771.8, 771.0, 704.5, 709.7, 625.7, 695.9, 812.2, 796.2, 869.0, 866.0, 830.0, 841.6, 879.5, 835.2, 852.7, 879.5, 801.1, 866.8, 825.7, 829.7, 878.2, 937.0, 914.1, 961.6, 960.4, 947.3, 989.9, 918.9, 927.0, 939.7, 962.3, 1019.7, 852.8, 928.4, 978.0], [-283.3, -352.5, -238.5, -303.9, -313.2, -247.7, -239.7, -190.6, -169.6, -152.3, -102.9, -58.3, -53.9, -34.8, -56.0, -60.4, -28.5, 71.0, 13.9, 79.4, 73.6, 173.2, 58.7, 256.1, 291.7, 333.4, 440.4, 525.9, 525.7, 675.5, 631.9, 503.9, 756.7, 596.4, 557.2, 791.2, 838.8, 863.5, 985.0, 1036.7, 1001.2, 1076.9, 1043.3, 1097.3, 1147.6, 1231.5, 1213.8, 1330.9, 1319.5, 1223.1, 948.3, 1361.2, 1304.4, 1298.3, 1356.0, 1450.9, 1511.2, 1390.2, 1537.2, 1458.8, 1453.7, 1605.0, 1705.8, 1644.1, 1813.8, 1690.0, 1825.9, 1547.8, 1780.4, 1800.8, 1808.7, 1823.3, 1903.3, 1962.9, 1931.6, 1895.5, 2052.3, 2033.1, 1594.0, 2039.3, 2087.0, 2102.7, 1610.7, 2082.0, 2042.5, 2102.7, 2081.2, 2134.5, 1608.2, 2125.9, 2264.4, 2282.5, 2242.2, 2311.9, 2330.1, 1905.2, 2372.3, 2305.8, 2367.7, 2283.3], [-314.1, -235.5, -355.4, -281.3, -107.2, -202.2, -164.1, -121.2, -173.8, -85.6, -85.9, -57.5, 65.9, 45.2, 45.6, 28.3, 87.1, 173.8, 225.6, 204.8, 321.7, 292.1, 456.8, 387.3, 355.6, 385.1, 147.2, 120.6, 621.7, 684.4, 756.7, 632.8, 612.9, 805.5, 899.5, 594.3, 688.2, 887.2, 474.6, 718.8, 36.5, 928.7, 882.4, 782.0, 775.7, 1107.9, 1140.7, 1145.0, 1137.7, 681.9, 1200.8, 788.0, 1215.0, 819.7, 1045.5, 883.2, 1186.5, 908.1, 920.7, 1275.6, 1088.8, 855.6, 1352.6, 955.3, 864.6, 1100.0, 1251.3, 1184.8, 1351.0, 837.6, 1003.6, 1285.8, 1378.0, 1394.8, 1470.1, 878.7, 1331.6, 1347.6, 1447.7, 1510.3, 1528.0, 1489.1, 1558.4, 1137.7, 1189.3, 1179.0, 1077.9, 1058.6, 1676.5, 1541.5, 1159.3, 1238.6, 1664.4, 1688.9, 1778.0, 1656.7, 1314.3, 1757.4, 1863.7, 1751.4], [-270.0, -317.2, -328.2, -317.3, -298.3, -186.0, -194.5, -226.6, -219.0, -157.3, -161.2, -115.5, -44.8, -28.6, 49.7, 77.6, 107.8, 116.4, 192.8, 360.8, -24.1, 245.4, 325.3, 440.1, 285.9, 322.6, 82.2, 509.0, 411.9, 365.1, 386.6, 667.7, 701.9, 537.8, 853.9, 849.0, 981.4, 948.9, 637.9, 169.2, 747.9, 686.5, 640.2, 1065.4, 535.3, 815.4, 1067.4, 413.5, 1153.0, 958.2, 813.3, 977.4, 843.9, 1264.6, 424.6, 959.3, 791.2, 999.2, 1272.8, 1250.9, 1261.4, 1097.2, 1352.2, 1334.2, 1326.7, 1279.4, 1503.7, 1477.1, 1080.2, 1133.6, 1219.0, 1462.4, 1037.9, 991.1, 1510.6, 1190.2, 1458.3, 885.3, 1452.8, 1504.2, 1323.5, 1418.3, 1452.4, 1144.7, 1270.0, 1194.5, 1323.9, 1501.2, 1516.6, 1560.3, 1537.7, 886.2, 1177.6, 1021.9, 1212.7, 513.8, 1568.4, 644.0, 1321.1, 932.8]], "span_s": [2475.8, 2475.4, 2530.4, 3151.5, 3189.5, 3151.5]}} diff --git a/experiments/e43-cross-machine-training/results/ladder.tsv b/experiments/e43-cross-machine-training/results/ladder.tsv new file mode 100644 index 0000000..6e66c6c --- /dev/null +++ b/experiments/e43-cross-machine-training/results/ladder.tsv @@ -0,0 +1,61 @@ +rung payload_kib rep exchanges pack_ms rtt_ms unpack_ms wall_s exchanges_per_s +lo 0 1 1000 0.0016 0.0974 0.0019 0.111 8991.7 +ts 0 1 1000 0.0019 0.1076 0.0031 0.131 7628.7 +phys 0 1 1000 0.0058 2.6360 0.0202 2.749 363.7 +lo 48 1 1000 0.0059 0.1483 0.0037 0.189 5283.8 +ts 48 1 1000 0.0058 0.1876 0.0041 0.225 4445.6 +phys 48 1 1000 0.0640 8.2662 0.0489 8.762 114.1 +lo 184 1 1000 0.0250 0.2390 0.0031 0.371 2694.3 +ts 184 1 1000 0.0272 0.2829 0.0045 0.428 2335.7 +phys 184 1 1000 0.2371 19.4763 0.0515 20.853 48.0 +lo 588 1 1000 0.1226 0.3866 0.0039 0.824 1213.6 +ts 588 1 1000 0.1266 0.6995 0.0067 1.178 848.8 +phys 588 1 1000 1.5953 68.0037 0.0604 73.301 13.6 +phys 0 2 1000 0.0089 3.0039 0.0331 3.185 314.0 +ts 0 2 1000 0.0015 0.0901 0.0017 0.103 9722.6 +lo 0 2 1000 0.0020 0.1075 0.0033 0.137 7321.5 +phys 48 2 1000 0.0488 7.9971 0.0456 8.443 118.4 +ts 48 2 1000 0.0061 0.1813 0.0040 0.219 4561.6 +lo 48 2 1000 0.0055 0.1711 0.0029 0.202 4949.6 +phys 184 2 1000 0.1773 19.9575 0.0520 21.070 47.5 +ts 184 2 1000 0.0258 0.3591 0.0047 0.502 1994.0 +lo 184 2 1000 0.0262 0.3435 0.0047 0.488 2048.8 +phys 588 2 1000 1.7267 65.7984 0.0636 71.393 14.0 +ts 588 2 1000 0.1296 0.9870 0.0088 1.494 669.4 +lo 588 2 1000 0.1306 1.0271 0.0094 1.537 650.8 +lo 0 3 1000 0.0021 0.0984 0.0034 0.127 7844.0 +ts 0 3 1000 0.0020 0.1128 0.0035 0.141 7083.4 +phys 0 3 1000 0.0075 2.6058 0.0257 2.747 364.0 +lo 48 3 1000 0.0061 0.1574 0.0039 0.204 4905.1 +ts 48 3 1000 0.0054 0.1725 0.0029 0.203 4930.9 +phys 48 3 1000 0.0519 7.8099 0.0481 8.274 120.9 +lo 184 3 1000 0.0263 0.2661 0.0042 0.407 2456.0 +ts 184 3 1000 0.0264 0.3059 0.0047 0.449 2225.3 +phys 184 3 1000 0.2664 21.3771 0.0521 22.641 44.2 +lo 588 3 1000 0.1413 0.9472 0.0088 1.480 675.9 +ts 588 3 1000 0.1260 0.3789 0.0039 0.818 1221.9 +phys 588 3 1000 1.6748 63.2057 0.0628 68.898 14.5 +phys 0 4 1000 0.0122 3.4053 0.0380 3.630 275.5 +ts 0 4 1000 0.0015 0.0957 0.0019 0.109 9161.7 +lo 0 4 1000 0.0019 0.1030 0.0033 0.128 7814.9 +phys 48 4 1000 0.0791 9.4465 0.0582 10.047 99.5 +ts 48 4 1000 0.0061 0.1557 0.0040 0.202 4961.8 +lo 48 4 1000 0.0059 0.1820 0.0032 0.216 4631.2 +phys 184 4 1000 0.2691 23.0144 0.0580 24.556 40.7 +ts 184 4 1000 0.0237 0.1803 0.0023 0.303 3300.2 +lo 184 4 1000 0.0261 0.3178 0.0048 0.464 2156.4 +phys 588 4 1000 1.7631 61.9649 0.0596 67.910 14.7 +ts 588 4 1000 0.1300 0.5129 0.0054 0.992 1008.5 +lo 588 4 1000 0.1312 0.3930 0.0039 0.865 1156.2 +lo 0 5 1000 0.0019 0.0990 0.0032 0.123 8157.0 +ts 0 5 1000 0.0015 0.1075 0.0019 0.121 8273.1 +phys 0 5 1000 0.0080 2.8148 0.0262 2.965 337.3 +lo 48 5 1000 0.0061 0.1607 0.0038 0.205 4884.8 +ts 48 5 1000 0.0062 0.1600 0.0038 0.205 4870.2 +phys 48 5 1000 0.0561 8.0063 0.0487 8.488 117.8 +lo 184 5 1000 0.0261 0.2928 0.0038 0.434 2305.2 +ts 184 5 1000 0.0260 0.3792 0.0048 0.519 1927.7 +phys 184 5 1000 0.2857 26.0893 0.0604 27.541 36.3 +lo 588 5 1000 0.1360 1.0013 0.0073 1.535 651.6 +ts 588 5 1000 0.1348 0.9064 0.0071 1.421 703.5 +phys 588 5 1000 1.5906 100.0883 0.0547 105.308 9.5 diff --git a/experiments/e43-cross-machine-training/results/local-b.out b/experiments/e43-cross-machine-training/results/local-b.out new file mode 100644 index 0000000..99621a8 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b.out @@ -0,0 +1,18 @@ +arm: local ports 9780+seed +server: /home/guangzhao/zuogou/plugrl/e43-reg/../plugrl-server/.venv/bin/python (src /home/guangzhao/zuogou/plugrl/e43-reg/src) +client: /home/guangzhao/zuogou/plugrl/e43-reg/../plugrl-env-client/.venv/bin/python +out: /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b +steps: 409600 seeds: 3 4 5 +start 2026-09-28 18:23:56 +READY seed 3 port 9783 at 18:23:59 +READY seed 4 port 9784 at 18:24:04 +READY seed 5 port 9785 at 18:24:09 +seed 5 client rc=0 at 18:44:10 +seed 5 server rc=0 at 18:44:10 +seed 4 client rc=0 at 18:44:20 +seed 4 server rc=0 at 18:44:21 +seed 3 client rc=0 at 18:45:28 +seed 3 server rc=0 at 18:45:29 +end 2026-09-28 18:45:29 +failed seeds: 0 +ARM_DONE local diff --git a/experiments/e43-cross-machine-training/results/local-b/client-seed3.log b/experiments/e43-cross-machine-training/results/local-b/client-seed3.log new file mode 100644 index 0000000..8b4aab2 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b/client-seed3.log @@ -0,0 +1,52 @@ +2026-09-28 18:23:59.978 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 18:23:59.978 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 18:23:59.979 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 18:23:59.979 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 18:23:59.979 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 18:24:00.016 | INFO | __main__:main:83 - Starting env client exp_name=e43-local-seed3 output_dir=runs/e43-local-seed3 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 18:24:00.054 | INFO | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://127.0.0.1:9783... +2026-09-28 18:24:00.057 | INFO | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 18:24:30.061 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=10836 infer_calls=10836 feedback_calls=10836 infer_wait=27.607s infer_obs_pack=0.156s env_step=0.982s feedback_total=0.743s feedback_obs_pack=0.195s feedback_info_pack=0.032s effective_fps=367.47 +2026-09-28 18:25:00.063 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=20887 infer_calls=20887 feedback_calls=20887 infer_wait=55.322s infer_obs_pack=0.311s env_step=1.921s feedback_total=1.460s feedback_obs_pack=0.381s feedback_info_pack=0.064s effective_fps=353.93 +2026-09-28 18:25:30.064 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=31413 infer_calls=31413 feedback_calls=31413 infer_wait=82.947s infer_obs_pack=0.462s env_step=2.911s feedback_total=2.194s feedback_obs_pack=0.574s feedback_info_pack=0.099s effective_fps=354.90 +2026-09-28 18:26:00.065 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41367 infer_calls=41367 feedback_calls=41367 infer_wait=110.723s infer_obs_pack=0.610s env_step=3.834s feedback_total=2.887s feedback_obs_pack=0.754s feedback_info_pack=0.136s effective_fps=350.40 +2026-09-28 18:26:30.066 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=52087 infer_calls=52087 feedback_calls=52087 infer_wait=138.245s infer_obs_pack=0.766s env_step=4.867s feedback_total=3.662s feedback_obs_pack=0.955s feedback_info_pack=0.175s effective_fps=353.04 +2026-09-28 18:27:00.068 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=61911 infer_calls=61911 feedback_calls=61911 infer_wait=166.021s infer_obs_pack=0.911s env_step=5.789s feedback_total=4.348s feedback_obs_pack=1.137s feedback_info_pack=0.210s effective_fps=349.64 +2026-09-28 18:27:30.070 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=72721 infer_calls=72721 feedback_calls=72721 infer_wait=193.540s infer_obs_pack=1.066s env_step=6.833s feedback_total=5.113s feedback_obs_pack=1.340s feedback_info_pack=0.249s effective_fps=352.07 +2026-09-28 18:28:00.071 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=82767 infer_calls=82767 feedback_calls=82767 infer_wait=221.234s infer_obs_pack=1.215s env_step=7.804s feedback_total=5.819s feedback_obs_pack=1.524s feedback_info_pack=0.285s effective_fps=350.60 +2026-09-28 18:28:30.074 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=93395 infer_calls=93395 feedback_calls=93395 infer_wait=248.805s infer_obs_pack=1.370s env_step=8.813s feedback_total=6.585s feedback_obs_pack=1.723s feedback_info_pack=0.323s effective_fps=351.67 +2026-09-28 18:29:00.077 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=102758 infer_calls=102758 feedback_calls=102758 infer_wait=276.715s infer_obs_pack=1.509s env_step=9.688s feedback_total=7.236s feedback_obs_pack=1.887s feedback_info_pack=0.355s effective_fps=348.16 +2026-09-28 18:29:30.081 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=111709 infer_calls=111709 feedback_calls=111709 infer_wait=304.770s infer_obs_pack=1.634s env_step=10.492s feedback_total=7.858s feedback_obs_pack=2.041s feedback_info_pack=0.387s effective_fps=343.98 +2026-09-28 18:30:00.082 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=120599 infer_calls=120599 feedback_calls=120599 infer_wait=332.800s infer_obs_pack=1.760s env_step=11.301s feedback_total=8.484s feedback_obs_pack=2.206s feedback_info_pack=0.423s effective_fps=340.34 +2026-09-28 18:30:30.082 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=129313 infer_calls=129313 feedback_calls=129313 infer_wait=360.895s infer_obs_pack=1.879s env_step=12.095s feedback_total=9.070s feedback_obs_pack=2.356s feedback_info_pack=0.454s effective_fps=336.81 +2026-09-28 18:31:00.082 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=137012 infer_calls=137012 feedback_calls=137012 infer_wait=389.235s infer_obs_pack=1.989s env_step=12.778s feedback_total=9.596s feedback_obs_pack=2.497s feedback_info_pack=0.481s effective_fps=331.27 +2026-09-28 18:31:30.084 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=145585 infer_calls=145585 feedback_calls=145585 infer_wait=417.371s infer_obs_pack=2.107s env_step=13.543s feedback_total=10.182s feedback_obs_pack=2.647s feedback_info_pack=0.514s effective_fps=328.48 +2026-09-28 18:32:00.086 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=154883 infer_calls=154883 feedback_calls=154883 infer_wait=445.339s infer_obs_pack=2.240s env_step=14.390s feedback_total=10.809s feedback_obs_pack=2.811s feedback_info_pack=0.545s effective_fps=327.60 +2026-09-28 18:32:31.554 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163842 infer_calls=163842 feedback_calls=163842 infer_wait=474.868s infer_obs_pack=2.367s env_step=15.200s feedback_total=11.413s feedback_obs_pack=2.970s feedback_info_pack=0.577s effective_fps=325.18 +2026-09-28 18:33:01.555 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=173189 infer_calls=173189 feedback_calls=173189 infer_wait=502.800s infer_obs_pack=2.508s env_step=16.058s feedback_total=12.061s feedback_obs_pack=3.139s feedback_info_pack=0.611s effective_fps=324.67 +2026-09-28 18:33:31.557 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=183307 infer_calls=183307 feedback_calls=183307 infer_wait=530.506s infer_obs_pack=2.660s env_step=17.001s feedback_total=12.794s feedback_obs_pack=3.330s feedback_info_pack=0.646s effective_fps=325.61 +2026-09-28 18:34:01.558 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=192589 infer_calls=192589 feedback_calls=192589 infer_wait=558.506s infer_obs_pack=2.795s env_step=17.821s feedback_total=13.430s feedback_obs_pack=3.497s feedback_info_pack=0.679s effective_fps=325.02 +2026-09-28 18:34:31.559 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=202589 infer_calls=202589 feedback_calls=202589 infer_wait=586.296s infer_obs_pack=2.943s env_step=18.716s feedback_total=14.142s feedback_obs_pack=3.676s feedback_info_pack=0.718s effective_fps=325.66 +2026-09-28 18:35:01.560 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=212742 infer_calls=212742 feedback_calls=212742 infer_wait=614.073s infer_obs_pack=3.089s env_step=19.629s feedback_total=14.843s feedback_obs_pack=3.854s feedback_info_pack=0.754s effective_fps=326.47 +2026-09-28 18:35:31.562 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=222034 infer_calls=222034 feedback_calls=222034 infer_wait=642.025s infer_obs_pack=3.228s env_step=20.468s feedback_total=15.489s feedback_obs_pack=4.023s feedback_info_pack=0.792s effective_fps=325.94 +2026-09-28 18:36:01.564 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=232165 infer_calls=232165 feedback_calls=232165 infer_wait=669.835s infer_obs_pack=3.373s env_step=21.373s feedback_total=16.179s feedback_obs_pack=4.201s feedback_info_pack=0.826s effective_fps=326.64 +2026-09-28 18:36:31.565 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=241715 infer_calls=241715 feedback_calls=241715 infer_wait=697.786s infer_obs_pack=3.511s env_step=22.219s feedback_total=16.829s feedback_obs_pack=4.373s feedback_info_pack=0.858s effective_fps=326.49 +2026-09-28 18:37:01.568 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=251801 infer_calls=251801 feedback_calls=251801 infer_wait=725.561s infer_obs_pack=3.656s env_step=23.137s feedback_total=17.527s feedback_obs_pack=4.562s feedback_info_pack=0.896s effective_fps=327.06 +2026-09-28 18:37:31.569 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=261493 infer_calls=261493 feedback_calls=261493 infer_wait=753.487s infer_obs_pack=3.792s env_step=23.994s feedback_total=18.182s feedback_obs_pack=4.735s feedback_info_pack=0.930s effective_fps=327.09 +2026-09-28 18:38:01.839 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=270338 infer_calls=270338 feedback_calls=270338 infer_wait=781.827s infer_obs_pack=3.920s env_step=24.791s feedback_total=18.797s feedback_obs_pack=4.892s feedback_info_pack=0.964s effective_fps=325.97 +2026-09-28 18:38:31.840 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=279947 infer_calls=279947 feedback_calls=279947 infer_wait=809.729s infer_obs_pack=4.058s env_step=25.667s feedback_total=19.455s feedback_obs_pack=5.069s feedback_info_pack=1.001s effective_fps=325.93 +2026-09-28 18:39:01.843 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=289151 infer_calls=289151 feedback_calls=289151 infer_wait=837.653s infer_obs_pack=4.203s env_step=26.519s feedback_total=20.099s feedback_obs_pack=5.240s feedback_info_pack=1.038s effective_fps=325.45 +2026-09-28 18:39:31.843 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=298776 infer_calls=298776 feedback_calls=298776 infer_wait=865.480s infer_obs_pack=4.342s env_step=27.415s feedback_total=20.789s feedback_obs_pack=5.418s feedback_info_pack=1.073s effective_fps=325.45 +2026-09-28 18:40:01.845 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=307425 infer_calls=307425 feedback_calls=307425 infer_wait=893.604s infer_obs_pack=4.468s env_step=28.199s feedback_total=21.377s feedback_obs_pack=5.570s feedback_info_pack=1.109s effective_fps=324.41 +2026-09-28 18:40:31.847 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=317166 infer_calls=317166 feedback_calls=317166 infer_wait=921.444s infer_obs_pack=4.608s env_step=29.098s feedback_total=22.051s feedback_obs_pack=5.745s feedback_info_pack=1.146s effective_fps=324.57 +2026-09-28 18:41:01.849 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=326819 infer_calls=326819 feedback_calls=326819 infer_wait=949.230s infer_obs_pack=4.751s env_step=30.002s feedback_total=22.761s feedback_obs_pack=5.926s feedback_info_pack=1.183s effective_fps=324.63 +2026-09-28 18:41:33.888 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=335874 infer_calls=335874 feedback_calls=335874 infer_wait=979.235s infer_obs_pack=4.889s env_step=30.840s feedback_total=23.399s feedback_obs_pack=6.090s feedback_info_pack=1.217s effective_fps=323.47 +2026-09-28 18:42:03.888 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=345463 infer_calls=345463 feedback_calls=345463 infer_wait=1007.066s infer_obs_pack=5.029s env_step=31.725s feedback_total=24.099s feedback_obs_pack=6.269s feedback_info_pack=1.256s effective_fps=323.49 +2026-09-28 18:42:33.888 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=355420 infer_calls=355420 feedback_calls=355420 infer_wait=1034.994s infer_obs_pack=5.165s env_step=32.579s feedback_total=24.759s feedback_obs_pack=6.443s feedback_info_pack=1.289s effective_fps=323.85 +2026-09-28 18:43:05.006 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=364546 infer_calls=364546 feedback_calls=364546 infer_wait=1064.186s infer_obs_pack=5.293s env_step=33.368s feedback_total=25.366s feedback_obs_pack=6.602s feedback_info_pack=1.321s effective_fps=323.12 +2026-09-28 18:43:35.006 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=374744 infer_calls=374744 feedback_calls=374744 infer_wait=1091.972s infer_obs_pack=5.439s env_step=34.267s feedback_total=26.085s feedback_obs_pack=6.782s feedback_info_pack=1.356s effective_fps=323.68 +2026-09-28 18:44:05.007 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=384395 infer_calls=384395 feedback_calls=384395 infer_wait=1119.919s infer_obs_pack=5.572s env_step=35.111s feedback_total=26.717s feedback_obs_pack=6.948s feedback_info_pack=1.391s effective_fps=323.75 +2026-09-28 18:44:35.892 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=393218 infer_calls=393218 feedback_calls=393218 infer_wait=1148.805s infer_obs_pack=5.708s env_step=35.929s feedback_total=27.339s feedback_obs_pack=7.107s feedback_info_pack=1.424s effective_fps=322.90 +2026-09-28 18:45:05.894 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=402885 infer_calls=402885 feedback_calls=402885 infer_wait=1176.601s infer_obs_pack=5.855s env_step=36.825s feedback_total=28.036s feedback_obs_pack=7.284s feedback_info_pack=1.458s effective_fps=323.00 +2026-09-28 18:45:28.692 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=1195.004s infer_obs_pack=5.954s env_step=37.457s feedback_total=28.532s feedback_obs_pack=7.409s feedback_info_pack=1.482s effective_fps=323.30 +2026-09-28 18:45:28.692 | INFO | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/local-b/client-seed4.log b/experiments/e43-cross-machine-training/results/local-b/client-seed4.log new file mode 100644 index 0000000..9618156 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b/client-seed4.log @@ -0,0 +1,50 @@ +2026-09-28 18:24:04.971 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 18:24:04.971 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 18:24:04.972 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 18:24:04.972 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 18:24:04.972 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 18:24:05.011 | INFO | __main__:main:83 - Starting env client exp_name=e43-local-seed4 output_dir=runs/e43-local-seed4 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 18:24:05.050 | INFO | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://127.0.0.1:9784... +2026-09-28 18:24:05.054 | INFO | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 18:24:35.055 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=10882 infer_calls=10882 feedback_calls=10882 infer_wait=27.537s infer_obs_pack=0.165s env_step=1.005s feedback_total=0.770s feedback_obs_pack=0.208s feedback_info_pack=0.036s effective_fps=369.17 +2026-09-28 18:25:05.056 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=20966 infer_calls=20966 feedback_calls=20966 infer_wait=55.200s infer_obs_pack=0.333s env_step=1.956s feedback_total=1.492s feedback_obs_pack=0.406s feedback_info_pack=0.068s effective_fps=355.47 +2026-09-28 18:25:35.056 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=31618 infer_calls=31618 feedback_calls=31618 infer_wait=82.785s infer_obs_pack=0.494s env_step=2.948s feedback_total=2.250s feedback_obs_pack=0.613s feedback_info_pack=0.102s effective_fps=357.36 +2026-09-28 18:26:05.058 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41648 infer_calls=41648 feedback_calls=41648 infer_wait=110.511s infer_obs_pack=0.642s env_step=3.898s feedback_total=2.958s feedback_obs_pack=0.806s feedback_info_pack=0.134s effective_fps=352.92 +2026-09-28 18:26:35.058 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=52530 infer_calls=52530 feedback_calls=52530 infer_wait=138.003s infer_obs_pack=0.808s env_step=4.952s feedback_total=3.725s feedback_obs_pack=1.012s feedback_info_pack=0.168s effective_fps=356.17 +2026-09-28 18:27:05.061 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=62615 infer_calls=62615 feedback_calls=62615 infer_wait=165.701s infer_obs_pack=0.963s env_step=5.918s feedback_total=4.431s feedback_obs_pack=1.200s feedback_info_pack=0.201s effective_fps=353.73 +2026-09-28 18:27:35.062 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=73512 infer_calls=73512 feedback_calls=73512 infer_wait=193.169s infer_obs_pack=1.132s env_step=6.983s feedback_total=5.207s feedback_obs_pack=1.404s feedback_info_pack=0.235s effective_fps=356.01 +2026-09-28 18:28:05.064 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=83700 infer_calls=83700 feedback_calls=83700 infer_wait=220.782s infer_obs_pack=1.287s env_step=7.984s feedback_total=5.936s feedback_obs_pack=1.599s feedback_info_pack=0.269s effective_fps=354.68 +2026-09-28 18:28:36.566 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=94210 infer_calls=94210 feedback_calls=94210 infer_wait=249.852s infer_obs_pack=1.449s env_step=9.005s feedback_total=6.684s feedback_obs_pack=1.803s feedback_info_pack=0.304s effective_fps=352.86 +2026-09-28 18:29:06.568 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=104606 infer_calls=104606 feedback_calls=104606 infer_wait=277.573s infer_obs_pack=1.598s env_step=9.950s feedback_total=7.399s feedback_obs_pack=1.990s feedback_info_pack=0.336s effective_fps=352.78 +2026-09-28 18:29:38.128 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=114690 infer_calls=114690 feedback_calls=114690 infer_wait=306.894s infer_obs_pack=1.748s env_step=10.895s feedback_total=8.092s feedback_obs_pack=2.175s feedback_info_pack=0.371s effective_fps=350.06 +2026-09-28 18:30:08.131 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=124346 infer_calls=124346 feedback_calls=124346 infer_wait=334.775s infer_obs_pack=1.890s env_step=11.757s feedback_total=8.773s feedback_obs_pack=2.352s feedback_info_pack=0.402s effective_fps=348.12 +2026-09-28 18:30:38.132 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=134441 infer_calls=134441 feedback_calls=134441 infer_wait=362.648s infer_obs_pack=2.038s env_step=12.624s feedback_total=9.450s feedback_obs_pack=2.528s feedback_info_pack=0.432s effective_fps=347.61 +2026-09-28 18:31:08.133 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=143730 infer_calls=143730 feedback_calls=143730 infer_wait=390.599s infer_obs_pack=2.176s env_step=13.472s feedback_total=10.081s feedback_obs_pack=2.700s feedback_info_pack=0.460s effective_fps=345.23 +2026-09-28 18:31:38.133 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=154080 infer_calls=154080 feedback_calls=154080 infer_wait=418.412s infer_obs_pack=2.329s env_step=14.367s feedback_total=10.758s feedback_obs_pack=2.885s feedback_info_pack=0.492s effective_fps=345.57 +2026-09-28 18:32:08.762 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163842 infer_calls=163842 feedback_calls=163842 infer_wait=446.908s infer_obs_pack=2.472s env_step=15.258s feedback_total=11.425s feedback_obs_pack=3.069s feedback_info_pack=0.521s effective_fps=344.16 +2026-09-28 18:32:38.765 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=174175 infer_calls=174175 feedback_calls=174175 infer_wait=474.650s infer_obs_pack=2.624s env_step=16.194s feedback_total=12.135s feedback_obs_pack=3.258s feedback_info_pack=0.554s effective_fps=344.49 +2026-09-28 18:33:10.799 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=184322 infer_calls=184322 feedback_calls=184322 infer_wait=504.358s infer_obs_pack=2.781s env_step=17.145s feedback_total=12.876s feedback_obs_pack=3.454s feedback_info_pack=0.588s effective_fps=343.14 +2026-09-28 18:33:40.801 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=194297 infer_calls=194297 feedback_calls=194297 infer_wait=532.157s infer_obs_pack=2.927s env_step=18.038s feedback_total=13.590s feedback_obs_pack=3.638s feedback_info_pack=0.622s effective_fps=342.85 +2026-09-28 18:34:10.803 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=204464 infer_calls=204464 feedback_calls=204464 infer_wait=559.874s infer_obs_pack=3.082s env_step=18.963s feedback_total=14.316s feedback_obs_pack=3.831s feedback_info_pack=0.655s effective_fps=342.93 +2026-09-28 18:34:40.805 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=214063 infer_calls=214063 feedback_calls=214063 infer_wait=587.779s infer_obs_pack=3.223s env_step=19.823s feedback_total=14.971s feedback_obs_pack=4.008s feedback_info_pack=0.684s effective_fps=342.06 +2026-09-28 18:35:10.806 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=224253 infer_calls=224253 feedback_calls=224253 infer_wait=615.536s infer_obs_pack=3.375s env_step=20.733s feedback_total=15.684s feedback_obs_pack=4.208s feedback_info_pack=0.717s effective_fps=342.20 +2026-09-28 18:35:40.808 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=233872 infer_calls=233872 feedback_calls=233872 infer_wait=643.392s infer_obs_pack=3.522s env_step=21.603s feedback_total=16.373s feedback_obs_pack=4.393s feedback_info_pack=0.747s effective_fps=341.47 +2026-09-28 18:36:10.811 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=244437 infer_calls=244437 feedback_calls=244437 infer_wait=671.089s infer_obs_pack=3.681s env_step=22.545s feedback_total=17.110s feedback_obs_pack=4.584s feedback_info_pack=0.779s effective_fps=342.14 +2026-09-28 18:36:40.812 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=254045 infer_calls=254045 feedback_calls=254045 infer_wait=698.989s infer_obs_pack=3.828s env_step=23.408s feedback_total=17.759s feedback_obs_pack=4.766s feedback_info_pack=0.808s effective_fps=341.47 +2026-09-28 18:37:10.812 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=264523 infer_calls=264523 feedback_calls=264523 infer_wait=726.741s infer_obs_pack=3.974s env_step=24.331s feedback_total=18.453s feedback_obs_pack=4.956s feedback_info_pack=0.842s effective_fps=341.98 +2026-09-28 18:37:41.294 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=274434 infer_calls=274434 feedback_calls=274434 infer_wait=755.115s infer_obs_pack=4.114s env_step=25.191s feedback_total=19.112s feedback_obs_pack=5.134s feedback_info_pack=0.872s effective_fps=341.53 +2026-09-28 18:38:11.295 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=284755 infer_calls=284755 feedback_calls=284755 infer_wait=782.804s infer_obs_pack=4.266s env_step=26.119s feedback_total=19.838s feedback_obs_pack=5.330s feedback_info_pack=0.904s effective_fps=341.83 +2026-09-28 18:38:42.816 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=294914 infer_calls=294914 feedback_calls=294914 infer_wait=812.047s infer_obs_pack=4.422s env_step=27.054s feedback_total=20.538s feedback_obs_pack=5.519s feedback_info_pack=0.935s effective_fps=341.31 +2026-09-28 18:39:12.818 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=305226 infer_calls=305226 feedback_calls=305226 infer_wait=839.755s infer_obs_pack=4.574s env_step=28.004s feedback_total=21.253s feedback_obs_pack=5.710s feedback_info_pack=0.972s effective_fps=341.57 +2026-09-28 18:39:43.430 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=315394 infer_calls=315394 feedback_calls=315394 infer_wait=868.201s infer_obs_pack=4.721s env_step=28.900s feedback_total=21.925s feedback_obs_pack=5.894s feedback_info_pack=1.005s effective_fps=341.43 +2026-09-28 18:40:13.430 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=326185 infer_calls=326185 feedback_calls=326185 infer_wait=895.923s infer_obs_pack=4.876s env_step=29.841s feedback_total=22.646s feedback_obs_pack=6.087s feedback_info_pack=1.041s effective_fps=342.17 +2026-09-28 18:40:43.602 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=335874 infer_calls=335874 feedback_calls=335874 infer_wait=923.934s infer_obs_pack=5.017s env_step=30.736s feedback_total=23.329s feedback_obs_pack=6.270s feedback_info_pack=1.072s effective_fps=341.68 +2026-09-28 18:41:13.603 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=346344 infer_calls=346344 feedback_calls=346344 infer_wait=951.527s infer_obs_pack=5.176s env_step=31.725s feedback_total=24.085s feedback_obs_pack=6.468s feedback_info_pack=1.104s effective_fps=342.06 +2026-09-28 18:41:44.060 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=356354 infer_calls=356354 feedback_calls=356354 infer_wait=979.748s infer_obs_pack=5.327s env_step=32.646s feedback_total=24.793s feedback_obs_pack=6.654s feedback_info_pack=1.136s effective_fps=341.82 +2026-09-28 18:42:14.063 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=366901 infer_calls=366901 feedback_calls=366901 infer_wait=1007.464s infer_obs_pack=5.485s env_step=33.602s feedback_total=25.489s feedback_obs_pack=6.839s feedback_info_pack=1.169s effective_fps=342.25 +2026-09-28 18:42:44.753 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=376834 infer_calls=376834 feedback_calls=376834 infer_wait=1036.076s infer_obs_pack=5.627s env_step=34.458s feedback_total=26.133s feedback_obs_pack=7.015s feedback_info_pack=1.199s effective_fps=341.86 +2026-09-28 18:43:14.754 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=387692 infer_calls=387692 feedback_calls=387692 infer_wait=1063.754s infer_obs_pack=5.782s env_step=35.396s feedback_total=26.887s feedback_obs_pack=7.208s feedback_info_pack=1.234s effective_fps=342.54 +2026-09-28 18:43:44.755 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=397712 infer_calls=397712 feedback_calls=397712 infer_wait=1091.639s infer_obs_pack=5.923s env_step=36.252s feedback_total=27.558s feedback_obs_pack=7.387s feedback_info_pack=1.264s effective_fps=342.45 +2026-09-28 18:44:14.756 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=407965 infer_calls=407965 feedback_calls=407965 infer_wait=1119.432s infer_obs_pack=6.079s env_step=37.160s feedback_total=28.241s feedback_obs_pack=7.569s feedback_info_pack=1.299s effective_fps=342.56 +2026-09-28 18:44:20.903 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=1123.272s infer_obs_pack=6.105s env_step=37.324s feedback_total=28.360s feedback_obs_pack=7.599s feedback_info_pack=1.305s effective_fps=342.74 +2026-09-28 18:44:20.903 | INFO | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/local-b/client-seed5.log b/experiments/e43-cross-machine-training/results/local-b/client-seed5.log new file mode 100644 index 0000000..c13ec7e --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b/client-seed5.log @@ -0,0 +1,49 @@ +2026-09-28 18:24:09.967 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 18:24:09.968 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 18:24:09.968 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 18:24:09.968 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 18:24:09.969 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 18:24:10.008 | INFO | __main__:main:83 - Starting env client exp_name=e43-local-seed5 output_dir=runs/e43-local-seed5 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 18:24:10.044 | INFO | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://127.0.0.1:9785... +2026-09-28 18:24:10.047 | INFO | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 18:24:40.049 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=10959 infer_calls=10959 feedback_calls=10959 infer_wait=27.503s infer_obs_pack=0.161s env_step=1.029s feedback_total=0.784s feedback_obs_pack=0.214s feedback_info_pack=0.038s effective_fps=371.78 +2026-09-28 18:25:10.049 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=20957 infer_calls=20957 feedback_calls=20957 infer_wait=55.210s infer_obs_pack=0.323s env_step=1.964s feedback_total=1.507s feedback_obs_pack=0.406s feedback_info_pack=0.071s effective_fps=355.18 +2026-09-28 18:25:40.049 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=31565 infer_calls=31565 feedback_calls=31565 infer_wait=82.848s infer_obs_pack=0.479s env_step=2.932s feedback_total=2.253s feedback_obs_pack=0.603s feedback_info_pack=0.102s effective_fps=356.62 +2026-09-28 18:26:10.052 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=41769 infer_calls=41769 feedback_calls=41769 infer_wait=110.576s infer_obs_pack=0.632s env_step=3.880s feedback_total=2.960s feedback_obs_pack=0.793s feedback_info_pack=0.133s effective_fps=353.83 +2026-09-28 18:26:40.052 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=52622 infer_calls=52622 feedback_calls=52622 infer_wait=138.062s infer_obs_pack=0.792s env_step=4.941s feedback_total=3.731s feedback_obs_pack=1.003s feedback_info_pack=0.168s effective_fps=356.70 +2026-09-28 18:27:10.054 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=62547 infer_calls=62547 feedback_calls=62547 infer_wait=165.837s infer_obs_pack=0.938s env_step=5.875s feedback_total=4.423s feedback_obs_pack=1.188s feedback_info_pack=0.200s effective_fps=353.23 +2026-09-28 18:27:40.055 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=73590 infer_calls=73590 feedback_calls=73590 infer_wait=193.278s infer_obs_pack=1.102s env_step=6.952s feedback_total=5.216s feedback_obs_pack=1.397s feedback_info_pack=0.234s effective_fps=356.29 +2026-09-28 18:28:10.056 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=83801 infer_calls=83801 feedback_calls=83801 infer_wait=220.917s infer_obs_pack=1.254s env_step=7.934s feedback_total=5.942s feedback_obs_pack=1.591s feedback_info_pack=0.265s effective_fps=355.02 +2026-09-28 18:28:41.393 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=94210 infer_calls=94210 feedback_calls=94210 infer_wait=249.923s infer_obs_pack=1.406s env_step=8.913s feedback_total=6.663s feedback_obs_pack=1.787s feedback_info_pack=0.297s effective_fps=352.97 +2026-09-28 18:29:11.394 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=105053 infer_calls=105053 feedback_calls=105053 infer_wait=277.598s infer_obs_pack=1.555s env_step=9.881s feedback_total=7.388s feedback_obs_pack=1.976s feedback_info_pack=0.330s effective_fps=354.40 +2026-09-28 18:29:41.394 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=114927 infer_calls=114927 feedback_calls=114927 infer_wait=305.412s infer_obs_pack=1.696s env_step=10.790s feedback_total=8.064s feedback_obs_pack=2.155s feedback_info_pack=0.361s effective_fps=352.58 +2026-09-28 18:30:11.395 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=125328 infer_calls=125328 feedback_calls=125328 infer_wait=333.116s infer_obs_pack=1.848s env_step=11.743s feedback_total=8.796s feedback_obs_pack=2.342s feedback_info_pack=0.394s effective_fps=352.54 +2026-09-28 18:30:41.397 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=135252 infer_calls=135252 feedback_calls=135252 infer_wait=361.012s infer_obs_pack=1.989s env_step=12.621s feedback_total=9.454s feedback_obs_pack=2.519s feedback_info_pack=0.425s effective_fps=351.23 +2026-09-28 18:31:11.399 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=145896 infer_calls=145896 feedback_calls=145896 infer_wait=388.677s infer_obs_pack=2.151s env_step=13.601s feedback_total=10.171s feedback_obs_pack=2.709s feedback_info_pack=0.461s effective_fps=351.90 +2026-09-28 18:31:41.401 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=155692 infer_calls=155692 feedback_calls=155692 infer_wait=416.516s infer_obs_pack=2.285s env_step=14.498s feedback_total=10.847s feedback_obs_pack=2.884s feedback_info_pack=0.492s effective_fps=350.54 +2026-09-28 18:32:11.401 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=166194 infer_calls=166194 feedback_calls=166194 infer_wait=444.197s infer_obs_pack=2.435s env_step=15.468s feedback_total=11.560s feedback_obs_pack=3.073s feedback_info_pack=0.527s effective_fps=350.87 +2026-09-28 18:32:41.817 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=176130 infer_calls=176130 feedback_calls=176130 infer_wait=472.398s infer_obs_pack=2.578s env_step=16.390s feedback_total=12.253s feedback_obs_pack=3.260s feedback_info_pack=0.557s effective_fps=349.73 +2026-09-28 18:33:11.821 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=186744 infer_calls=186744 feedback_calls=186744 infer_wait=500.034s infer_obs_pack=2.731s env_step=17.381s feedback_total=12.999s feedback_obs_pack=3.454s feedback_info_pack=0.590s effective_fps=350.27 +2026-09-28 18:33:42.026 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=196610 infer_calls=196610 feedback_calls=196610 infer_wait=528.020s infer_obs_pack=2.876s env_step=18.306s feedback_total=13.695s feedback_obs_pack=3.634s feedback_info_pack=0.621s effective_fps=349.28 +2026-09-28 18:34:12.029 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=207346 infer_calls=207346 feedback_calls=207346 infer_wait=555.614s infer_obs_pack=3.031s env_step=19.308s feedback_total=14.437s feedback_obs_pack=3.830s feedback_info_pack=0.654s effective_fps=350.02 +2026-09-28 18:34:42.030 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=217307 infer_calls=217307 feedback_calls=217307 infer_wait=583.401s infer_obs_pack=3.179s env_step=20.226s feedback_total=15.111s feedback_obs_pack=4.010s feedback_info_pack=0.684s effective_fps=349.41 +2026-09-28 18:35:12.033 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=227741 infer_calls=227741 feedback_calls=227741 infer_wait=611.088s infer_obs_pack=3.332s env_step=21.176s feedback_total=15.850s feedback_obs_pack=4.208s feedback_info_pack=0.715s effective_fps=349.59 +2026-09-28 18:35:42.033 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=237644 infer_calls=237644 feedback_calls=237644 infer_wait=638.890s infer_obs_pack=3.477s env_step=22.086s feedback_total=16.543s feedback_obs_pack=4.386s feedback_info_pack=0.746s effective_fps=348.97 +2026-09-28 18:36:12.034 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=248175 infer_calls=248175 feedback_calls=248175 infer_wait=666.568s infer_obs_pack=3.629s env_step=23.056s feedback_total=17.269s feedback_obs_pack=4.579s feedback_info_pack=0.778s effective_fps=349.29 +2026-09-28 18:36:42.239 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=258050 infer_calls=258050 feedback_calls=258050 infer_wait=694.610s infer_obs_pack=3.772s env_step=23.948s feedback_total=17.949s feedback_obs_pack=4.768s feedback_info_pack=0.809s effective_fps=348.58 +2026-09-28 18:37:12.239 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=268537 infer_calls=268537 feedback_calls=268537 infer_wait=722.341s infer_obs_pack=3.925s env_step=24.877s feedback_total=18.667s feedback_obs_pack=4.959s feedback_info_pack=0.841s effective_fps=348.84 +2026-09-28 18:37:42.269 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=278530 infer_calls=278530 feedback_calls=278530 infer_wait=750.161s infer_obs_pack=4.071s env_step=25.798s feedback_total=19.355s feedback_obs_pack=5.146s feedback_info_pack=0.872s effective_fps=348.43 +2026-09-28 18:38:12.270 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=289222 infer_calls=289222 feedback_calls=289222 infer_wait=777.839s infer_obs_pack=4.224s env_step=26.773s feedback_total=20.083s feedback_obs_pack=5.340s feedback_info_pack=0.904s effective_fps=348.91 +2026-09-28 18:38:42.271 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=299013 infer_calls=299013 feedback_calls=299013 infer_wait=805.680s infer_obs_pack=4.364s env_step=27.682s feedback_total=20.758s feedback_obs_pack=5.525s feedback_info_pack=0.933s effective_fps=348.30 +2026-09-28 18:39:12.272 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=309063 infer_calls=309063 feedback_calls=309063 infer_wait=833.387s infer_obs_pack=4.511s env_step=28.631s feedback_total=21.484s feedback_obs_pack=5.718s feedback_info_pack=0.963s effective_fps=348.04 +2026-09-28 18:39:44.093 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=319490 infer_calls=319490 feedback_calls=319490 infer_wait=862.944s infer_obs_pack=4.654s env_step=29.586s feedback_total=22.184s feedback_obs_pack=5.902s feedback_info_pack=0.999s effective_fps=347.51 +2026-09-28 18:40:14.095 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=330414 infer_calls=330414 feedback_calls=330414 infer_wait=890.561s infer_obs_pack=4.808s env_step=30.587s feedback_total=22.929s feedback_obs_pack=6.097s feedback_info_pack=1.033s effective_fps=348.21 +2026-09-28 18:40:44.204 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=339970 infer_calls=339970 feedback_calls=339970 infer_wait=918.530s infer_obs_pack=4.948s env_step=31.488s feedback_total=23.593s feedback_obs_pack=6.273s feedback_info_pack=1.063s effective_fps=347.42 +2026-09-28 18:41:14.204 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=350242 infer_calls=350242 feedback_calls=350242 infer_wait=946.178s infer_obs_pack=5.103s env_step=32.482s feedback_total=24.325s feedback_obs_pack=6.468s feedback_info_pack=1.095s effective_fps=347.43 +2026-09-28 18:41:46.026 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=360450 infer_calls=360450 feedback_calls=360450 infer_wait=975.684s infer_obs_pack=5.254s env_step=33.462s feedback_total=25.038s feedback_obs_pack=6.659s feedback_info_pack=1.126s effective_fps=346.77 +2026-09-28 18:42:16.026 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=371218 infer_calls=371218 feedback_calls=371218 infer_wait=1003.380s infer_obs_pack=5.405s env_step=34.416s feedback_total=25.764s feedback_obs_pack=6.851s feedback_info_pack=1.157s effective_fps=347.27 +2026-09-28 18:42:46.028 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=380968 infer_calls=380968 feedback_calls=380968 infer_wait=1031.328s infer_obs_pack=5.541s env_step=35.270s feedback_total=26.404s feedback_obs_pack=7.019s feedback_info_pack=1.185s effective_fps=346.79 +2026-09-28 18:43:16.031 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=391707 infer_calls=391707 feedback_calls=391707 infer_wait=1059.103s infer_obs_pack=5.689s env_step=36.206s feedback_total=27.088s feedback_obs_pack=7.200s feedback_info_pack=1.215s effective_fps=347.23 +2026-09-28 18:43:46.031 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=401593 infer_calls=401593 feedback_calls=401593 infer_wait=1087.067s infer_obs_pack=5.822s env_step=37.054s feedback_total=27.723s feedback_obs_pack=7.370s feedback_info_pack=1.246s effective_fps=346.90 +2026-09-28 18:44:10.138 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=1107.268s infer_obs_pack=5.937s env_step=37.806s feedback_total=28.278s feedback_obs_pack=7.519s feedback_info_pack=1.270s effective_fps=347.33 +2026-09-28 18:44:10.138 | INFO | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/local-b/server-seed3.log b/experiments/e43-cross-machine-training/results/local-b/server-seed3.log new file mode 100644 index 0000000..a8deec8 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b/server-seed3.log @@ -0,0 +1,23 @@ +18:23:58|INFO|plugrl_server version: 0.1.0 +18:23:58|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +18:23:58|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +18:23:58|INFO|Seeded python, numpy and torch with 3. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +18:23:58|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed3 +18:23:59|INFO|Policy created... +18:23:59|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +18:23:59|INFO|Algorithm created: + +18:23:59|INFO|Agent Server is listening on 0.0.0.0:9783 +18:27:57|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed3/81921 +18:32:31|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed3/163841 +18:36:44|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed3/245761 +18:41:07|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed3/327681 +18:45:28|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed3/409601 +18:45:28|INFO|Stopping server as the algorithm signaled to stop. +18:45:28|INFO|Shutdown started: aborting pending infer requests. +18:45:28|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +18:45:28|INFO|Shutdown closing 1 websocket connection(s). +18:45:28|INFO|Shutdown interrupted an in-flight inference request from ('127.0.0.1', 44274). +18:45:28|INFO|WebSocket server closed. +18:45:28|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/local-b/server-seed4.log b/experiments/e43-cross-machine-training/results/local-b/server-seed4.log new file mode 100644 index 0000000..f040c62 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b/server-seed4.log @@ -0,0 +1,23 @@ +18:24:03|INFO|plugrl_server version: 0.1.0 +18:24:03|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +18:24:03|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +18:24:03|INFO|Seeded python, numpy and torch with 4. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +18:24:03|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed4 +18:24:04|INFO|Policy created... +18:24:04|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +18:24:04|INFO|Algorithm created: + +18:24:04|INFO|Agent Server is listening on 0.0.0.0:9784 +18:28:00|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed4/81921 +18:32:08|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed4/163841 +18:36:16|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed4/245761 +18:40:19|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed4/327681 +18:44:20|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed4/409601 +18:44:20|INFO|Stopping server as the algorithm signaled to stop. +18:44:20|INFO|Shutdown started: aborting pending infer requests. +18:44:20|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +18:44:20|INFO|Shutdown closing 1 websocket connection(s). +18:44:20|INFO|Shutdown interrupted an in-flight inference request from ('127.0.0.1', 52100). +18:44:20|INFO|WebSocket server closed. +18:44:20|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/local-b/server-seed5.log b/experiments/e43-cross-machine-training/results/local-b/server-seed5.log new file mode 100644 index 0000000..eb1c35b --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local-b/server-seed5.log @@ -0,0 +1,23 @@ +18:24:09|INFO|plugrl_server version: 0.1.0 +18:24:09|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +18:24:09|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +18:24:09|INFO|Seeded python, numpy and torch with 5. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +18:24:09|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed5 +18:24:09|INFO|Policy created... +18:24:09|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +18:24:09|INFO|Algorithm created: + +18:24:09|INFO|Agent Server is listening on 0.0.0.0:9785 +18:28:05|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed5/81921 +18:32:05|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed5/163841 +18:36:05|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed5/245761 +18:40:07|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed5/327681 +18:44:10|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local-b/fpo/fpo-policy/halfcheetah-local-seed5/409601 +18:44:10|INFO|Stopping server as the algorithm signaled to stop. +18:44:10|INFO|Shutdown started: aborting pending infer requests. +18:44:10|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +18:44:10|INFO|Shutdown closing 1 websocket connection(s). +18:44:10|INFO|Shutdown interrupted an in-flight inference request from ('127.0.0.1', 53650). +18:44:10|INFO|WebSocket server closed. +18:44:10|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/local.out b/experiments/e43-cross-machine-training/results/local.out new file mode 100644 index 0000000..0402878 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local.out @@ -0,0 +1,18 @@ +arm: local ports 9780+seed +server: /home/guangzhao/zuogou/plugrl/e43-reg/../plugrl-server/.venv/bin/python (src /home/guangzhao/zuogou/plugrl/e43-reg/src) +client: /home/guangzhao/zuogou/plugrl/e43-reg/../plugrl-env-client/.venv/bin/python +out: /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local +steps: 409600 seeds: 0 1 2 +start 2026-09-28 17:37:46 +READY seed 0 port 9780 at 17:37:49 +READY seed 1 port 9781 at 17:37:54 +READY seed 2 port 9782 at 17:37:59 +seed 2 client rc=0 at 17:57:46 +seed 2 server rc=0 at 17:57:46 +seed 1 client rc=0 at 17:57:53 +seed 1 server rc=0 at 17:57:54 +seed 0 client rc=0 at 17:58:24 +seed 0 server rc=0 at 17:58:25 +end 2026-09-28 17:58:25 +failed seeds: 0 +ARM_DONE local diff --git a/experiments/e43-cross-machine-training/results/local/client-seed0.log b/experiments/e43-cross-machine-training/results/local/client-seed0.log new file mode 100644 index 0000000..a630f83 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/client-seed0.log @@ -0,0 +1,50 @@ +2026-09-28 17:37:49.620 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 17:37:49.620 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 17:37:49.620 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 17:37:49.620 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 17:37:49.621 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 17:37:49.663 | INFO | __main__:main:83 - Starting env client exp_name=e43-local-seed0 output_dir=runs/e43-local-seed0 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 17:37:49.700 | INFO | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://127.0.0.1:9780... +2026-09-28 17:37:49.704 | INFO | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 17:38:19.706 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=10548 infer_calls=10548 feedback_calls=10548 infer_wait=27.657s infer_obs_pack=0.148s env_step=0.968s feedback_total=0.723s feedback_obs_pack=0.192s feedback_info_pack=0.033s effective_fps=357.61 +2026-09-28 17:38:50.110 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=20482 infer_calls=20482 feedback_calls=20482 infer_wait=55.824s infer_obs_pack=0.298s env_step=1.883s feedback_total=1.420s feedback_obs_pack=0.376s feedback_info_pack=0.066s effective_fps=344.66 +2026-09-28 17:39:20.112 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=30911 infer_calls=30911 feedback_calls=30911 infer_wait=83.454s infer_obs_pack=0.448s env_step=2.865s feedback_total=2.158s feedback_obs_pack=0.570s feedback_info_pack=0.100s effective_fps=347.61 +2026-09-28 17:39:50.758 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=40962 infer_calls=40962 feedback_calls=40962 infer_wait=111.770s infer_obs_pack=0.594s env_step=3.833s feedback_total=2.875s feedback_obs_pack=0.763s feedback_info_pack=0.133s effective_fps=344.01 +2026-09-28 17:40:20.759 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=51443 infer_calls=51443 feedback_calls=51443 infer_wait=139.327s infer_obs_pack=0.750s env_step=4.856s feedback_total=3.631s feedback_obs_pack=0.963s feedback_info_pack=0.168s effective_fps=346.27 +2026-09-28 17:40:51.183 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=61442 infer_calls=61442 feedback_calls=61442 infer_wait=167.434s infer_obs_pack=0.900s env_step=5.836s feedback_total=4.337s feedback_obs_pack=1.149s feedback_info_pack=0.199s effective_fps=344.20 +2026-09-28 17:41:21.184 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=71945 infer_calls=71945 feedback_calls=71945 infer_wait=194.938s infer_obs_pack=1.060s env_step=6.892s feedback_total=5.102s feedback_obs_pack=1.353s feedback_info_pack=0.232s effective_fps=345.90 +2026-09-28 17:41:51.286 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=81922 infer_calls=81922 feedback_calls=81922 infer_wait=222.734s infer_obs_pack=1.205s env_step=7.862s feedback_total=5.823s feedback_obs_pack=1.541s feedback_info_pack=0.263s effective_fps=344.76 +2026-09-28 17:42:21.287 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=92470 infer_calls=92470 feedback_calls=92470 infer_wait=250.296s infer_obs_pack=1.359s env_step=8.891s feedback_total=6.574s feedback_obs_pack=1.742s feedback_info_pack=0.295s effective_fps=346.17 +2026-09-28 17:42:51.690 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=102402 infer_calls=102402 feedback_calls=102402 infer_wait=278.411s infer_obs_pack=1.506s env_step=9.843s feedback_total=7.281s feedback_obs_pack=1.932s feedback_info_pack=0.326s effective_fps=344.74 +2026-09-28 17:43:21.692 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=112957 infer_calls=112957 feedback_calls=112957 infer_wait=305.985s infer_obs_pack=1.661s env_step=10.857s feedback_total=8.033s feedback_obs_pack=2.131s feedback_info_pack=0.359s effective_fps=345.92 +2026-09-28 17:43:52.413 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=122882 infer_calls=122882 feedback_calls=122882 infer_wait=334.481s infer_obs_pack=1.807s env_step=11.790s feedback_total=8.713s feedback_obs_pack=2.315s feedback_info_pack=0.390s effective_fps=344.41 +2026-09-28 17:44:22.415 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=133442 infer_calls=133442 feedback_calls=133442 infer_wait=362.069s infer_obs_pack=1.962s env_step=12.799s feedback_total=9.456s feedback_obs_pack=2.517s feedback_info_pack=0.422s effective_fps=345.45 +2026-09-28 17:44:52.619 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=143362 infer_calls=143362 feedback_calls=143362 infer_wait=390.005s infer_obs_pack=2.113s env_step=13.730s feedback_total=10.175s feedback_obs_pack=2.703s feedback_info_pack=0.453s effective_fps=344.60 +2026-09-28 17:45:22.620 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=153879 infer_calls=153879 feedback_calls=153879 infer_wait=417.622s infer_obs_pack=2.272s env_step=14.719s feedback_total=10.912s feedback_obs_pack=2.895s feedback_info_pack=0.485s effective_fps=345.39 +2026-09-28 17:45:53.594 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=163842 infer_calls=163842 feedback_calls=163842 infer_wait=446.275s infer_obs_pack=2.422s env_step=15.692s feedback_total=11.625s feedback_obs_pack=3.086s feedback_info_pack=0.517s effective_fps=344.20 +2026-09-28 17:46:23.595 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=174348 infer_calls=174348 feedback_calls=174348 infer_wait=473.821s infer_obs_pack=2.579s env_step=16.719s feedback_total=12.383s feedback_obs_pack=3.289s feedback_info_pack=0.551s effective_fps=344.90 +2026-09-28 17:46:54.684 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=184322 infer_calls=184322 feedback_calls=184322 infer_wait=502.527s infer_obs_pack=2.731s env_step=17.721s feedback_total=13.122s feedback_obs_pack=3.485s feedback_info_pack=0.583s effective_fps=343.82 +2026-09-28 17:47:24.685 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=194668 infer_calls=194668 feedback_calls=194668 infer_wait=530.078s infer_obs_pack=2.888s env_step=18.747s feedback_total=13.880s feedback_obs_pack=3.687s feedback_info_pack=0.618s effective_fps=344.18 +2026-09-28 17:47:56.136 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=204802 infer_calls=204802 feedback_calls=204802 infer_wait=559.127s infer_obs_pack=3.041s env_step=19.750s feedback_total=14.624s feedback_obs_pack=3.887s feedback_info_pack=0.651s effective_fps=343.32 +2026-09-28 17:48:26.137 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=215039 infer_calls=215039 feedback_calls=215039 infer_wait=586.760s infer_obs_pack=3.191s env_step=20.743s feedback_total=15.355s feedback_obs_pack=4.080s feedback_info_pack=0.683s effective_fps=343.49 +2026-09-28 17:48:57.624 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=225282 infer_calls=225282 feedback_calls=225282 infer_wait=615.895s infer_obs_pack=3.342s env_step=21.721s feedback_total=16.083s feedback_obs_pack=4.272s feedback_info_pack=0.715s effective_fps=342.87 +2026-09-28 17:49:27.627 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=235796 infer_calls=235796 feedback_calls=235796 infer_wait=643.478s infer_obs_pack=3.495s env_step=22.728s feedback_total=16.836s feedback_obs_pack=4.474s feedback_info_pack=0.747s effective_fps=343.46 +2026-09-28 17:49:59.580 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=245762 infer_calls=245762 feedback_calls=245762 infer_wait=673.188s infer_obs_pack=3.641s env_step=23.671s feedback_total=17.538s feedback_obs_pack=4.652s feedback_info_pack=0.777s effective_fps=342.27 +2026-09-28 17:50:29.582 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=255738 infer_calls=255738 feedback_calls=255738 infer_wait=700.968s infer_obs_pack=3.790s env_step=24.596s feedback_total=18.237s feedback_obs_pack=4.833s feedback_info_pack=0.806s effective_fps=342.08 +2026-09-28 17:50:59.582 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=265768 infer_calls=265768 feedback_calls=265768 infer_wait=728.769s infer_obs_pack=3.934s env_step=25.507s feedback_total=18.935s feedback_obs_pack=5.016s feedback_info_pack=0.837s effective_fps=341.98 +2026-09-28 17:51:29.583 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=274781 infer_calls=274781 feedback_calls=274781 infer_wait=756.752s infer_obs_pack=4.069s env_step=26.351s feedback_total=19.549s feedback_obs_pack=5.180s feedback_info_pack=0.865s effective_fps=340.61 +2026-09-28 17:51:59.584 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=285189 infer_calls=285189 feedback_calls=285189 infer_wait=784.387s infer_obs_pack=4.218s env_step=27.350s feedback_total=20.283s feedback_obs_pack=5.375s feedback_info_pack=0.898s effective_fps=341.04 +2026-09-28 17:52:30.716 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=294914 infer_calls=294914 feedback_calls=294914 infer_wait=813.254s infer_obs_pack=4.367s env_step=28.306s feedback_total=20.968s feedback_obs_pack=5.560s feedback_info_pack=0.929s effective_fps=340.20 +2026-09-28 17:53:00.721 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=305213 infer_calls=305213 feedback_calls=305213 infer_wait=840.854s infer_obs_pack=4.518s env_step=29.305s feedback_total=21.722s feedback_obs_pack=5.756s feedback_info_pack=0.964s effective_fps=340.49 +2026-09-28 17:53:32.784 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=315394 infer_calls=315394 feedback_calls=315394 infer_wait=870.534s infer_obs_pack=4.671s env_step=30.309s feedback_total=22.449s feedback_obs_pack=5.952s feedback_info_pack=0.996s effective_fps=339.88 +2026-09-28 17:54:02.785 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=325871 infer_calls=325871 feedback_calls=325871 infer_wait=898.106s infer_obs_pack=4.828s env_step=31.327s feedback_total=23.200s feedback_obs_pack=6.152s feedback_info_pack=1.030s effective_fps=340.35 +2026-09-28 17:54:34.547 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=335874 infer_calls=335874 feedback_calls=335874 infer_wait=927.514s infer_obs_pack=4.977s env_step=32.318s feedback_total=23.924s feedback_obs_pack=6.347s feedback_info_pack=1.064s effective_fps=339.70 +2026-09-28 17:55:04.550 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=346150 infer_calls=346150 feedback_calls=346150 infer_wait=955.174s infer_obs_pack=5.129s env_step=33.308s feedback_total=24.649s feedback_obs_pack=6.542s feedback_info_pack=1.096s effective_fps=339.94 +2026-09-28 17:55:34.551 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=356162 infer_calls=356162 feedback_calls=356162 infer_wait=982.899s infer_obs_pack=5.282s env_step=34.245s feedback_total=25.355s feedback_obs_pack=6.727s feedback_info_pack=1.129s effective_fps=339.92 +2026-09-28 17:56:04.551 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=365026 infer_calls=365026 feedback_calls=365026 infer_wait=1010.930s infer_obs_pack=5.409s env_step=35.062s feedback_total=25.967s feedback_obs_pack=6.886s feedback_info_pack=1.155s effective_fps=338.81 +2026-09-28 17:56:34.552 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=374947 infer_calls=374947 feedback_calls=374947 infer_wait=1038.729s infer_obs_pack=5.553s env_step=35.978s feedback_total=26.663s feedback_obs_pack=7.065s feedback_info_pack=1.186s effective_fps=338.73 +2026-09-28 17:57:04.553 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=384585 infer_calls=384585 feedback_calls=384585 infer_wait=1066.592s infer_obs_pack=5.693s env_step=36.873s feedback_total=27.329s feedback_obs_pack=7.244s feedback_info_pack=1.218s effective_fps=338.40 +2026-09-28 17:57:34.554 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=394046 infer_calls=394046 feedback_calls=394046 infer_wait=1094.430s infer_obs_pack=5.836s env_step=37.772s feedback_total=28.012s feedback_obs_pack=7.418s feedback_info_pack=1.248s effective_fps=337.93 +2026-09-28 17:58:04.555 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=403761 infer_calls=403761 feedback_calls=403761 infer_wait=1122.230s infer_obs_pack=5.976s env_step=38.696s feedback_total=28.706s feedback_obs_pack=7.595s feedback_info_pack=1.277s effective_fps=337.70 +2026-09-28 17:58:24.614 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=1138.156s infer_obs_pack=6.065s env_step=39.255s feedback_total=29.146s feedback_obs_pack=7.702s feedback_info_pack=1.300s effective_fps=337.78 +2026-09-28 17:58:24.615 | INFO | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/local/client-seed1.log b/experiments/e43-cross-machine-training/results/local/client-seed1.log new file mode 100644 index 0000000..42d077c --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/client-seed1.log @@ -0,0 +1,49 @@ +2026-09-28 17:37:54.620 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 17:37:54.620 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 17:37:54.621 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 17:37:54.621 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 17:37:54.621 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 17:37:54.665 | INFO | __main__:main:83 - Starting env client exp_name=e43-local-seed1 output_dir=runs/e43-local-seed1 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 17:37:54.701 | INFO | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://127.0.0.1:9781... +2026-09-28 17:37:54.703 | INFO | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 17:38:24.705 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=10819 infer_calls=10819 feedback_calls=10819 infer_wait=27.578s infer_obs_pack=0.155s env_step=0.996s feedback_total=0.763s feedback_obs_pack=0.204s feedback_info_pack=0.037s effective_fps=366.86 +2026-09-28 17:38:54.705 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=21007 infer_calls=21007 feedback_calls=21007 infer_wait=55.225s infer_obs_pack=0.317s env_step=1.957s feedback_total=1.496s feedback_obs_pack=0.401s feedback_info_pack=0.070s effective_fps=356.08 +2026-09-28 17:39:24.707 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=31819 infer_calls=31819 feedback_calls=31819 infer_wait=82.737s infer_obs_pack=0.476s env_step=3.003s feedback_total=2.256s feedback_obs_pack=0.605s feedback_info_pack=0.105s effective_fps=359.65 +2026-09-28 17:39:54.708 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=42031 infer_calls=42031 feedback_calls=42031 infer_wait=110.370s infer_obs_pack=0.625s env_step=3.987s feedback_total=2.989s feedback_obs_pack=0.798s feedback_info_pack=0.141s effective_fps=356.29 +2026-09-28 17:40:24.708 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=52808 infer_calls=52808 feedback_calls=52808 infer_wait=137.872s infer_obs_pack=0.780s env_step=5.041s feedback_total=3.758s feedback_obs_pack=1.004s feedback_info_pack=0.175s effective_fps=358.14 +2026-09-28 17:40:54.711 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=62993 infer_calls=62993 feedback_calls=62993 infer_wait=165.446s infer_obs_pack=0.934s env_step=6.067s feedback_total=4.499s feedback_obs_pack=1.208s feedback_info_pack=0.207s effective_fps=356.00 +2026-09-28 17:41:26.112 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=73730 infer_calls=73730 feedback_calls=73730 infer_wait=194.326s infer_obs_pack=1.098s env_step=7.128s feedback_total=5.271s feedback_obs_pack=1.419s feedback_info_pack=0.243s effective_fps=354.77 +2026-09-28 17:41:56.114 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=84435 infer_calls=84435 feedback_calls=84435 infer_wait=221.771s infer_obs_pack=1.259s env_step=8.211s feedback_total=6.053s feedback_obs_pack=1.630s feedback_info_pack=0.280s effective_fps=355.82 +2026-09-28 17:42:26.115 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=94533 infer_calls=94533 feedback_calls=94533 infer_wait=249.442s infer_obs_pack=1.409s env_step=9.176s feedback_total=6.783s feedback_obs_pack=1.826s feedback_info_pack=0.315s effective_fps=354.31 +2026-09-28 17:42:56.116 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=105371 infer_calls=105371 feedback_calls=105371 infer_wait=276.957s infer_obs_pack=1.564s env_step=10.213s feedback_total=7.546s feedback_obs_pack=2.030s feedback_info_pack=0.349s effective_fps=355.65 +2026-09-28 17:43:26.118 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=115015 infer_calls=115015 feedback_calls=115015 infer_wait=304.729s infer_obs_pack=1.705s env_step=11.145s feedback_total=8.227s feedback_obs_pack=2.212s feedback_info_pack=0.380s effective_fps=353.02 +2026-09-28 17:43:56.119 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=125408 infer_calls=125408 feedback_calls=125408 infer_wait=332.323s infer_obs_pack=1.865s env_step=12.151s feedback_total=8.975s feedback_obs_pack=2.408s feedback_info_pack=0.413s effective_fps=352.95 +2026-09-28 17:44:26.862 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=135170 infer_calls=135170 feedback_calls=135170 infer_wait=360.789s infer_obs_pack=2.017s env_step=13.098s feedback_total=9.680s feedback_obs_pack=2.595s feedback_info_pack=0.445s effective_fps=350.56 +2026-09-28 17:44:56.863 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=145609 infer_calls=145609 feedback_calls=145609 infer_wait=388.359s infer_obs_pack=2.177s env_step=14.117s feedback_total=10.428s feedback_obs_pack=2.796s feedback_info_pack=0.482s effective_fps=350.80 +2026-09-28 17:45:27.574 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=155650 infer_calls=155650 feedback_calls=155650 infer_wait=416.718s infer_obs_pack=2.327s env_step=15.098s feedback_total=11.156s feedback_obs_pack=2.992s feedback_info_pack=0.515s effective_fps=349.54 +2026-09-28 17:45:57.574 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=166228 infer_calls=166228 feedback_calls=166228 infer_wait=444.214s infer_obs_pack=2.485s env_step=16.147s feedback_total=11.920s feedback_obs_pack=3.200s feedback_info_pack=0.551s effective_fps=350.13 +2026-09-28 17:46:27.575 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=176131 infer_calls=176131 feedback_calls=176131 infer_wait=471.881s infer_obs_pack=2.632s env_step=17.126s feedback_total=12.637s feedback_obs_pack=3.391s feedback_info_pack=0.583s effective_fps=349.27 +2026-09-28 17:46:57.575 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=186734 infer_calls=186734 feedback_calls=186734 infer_wait=499.340s infer_obs_pack=2.788s env_step=18.201s feedback_total=13.412s feedback_obs_pack=3.596s feedback_info_pack=0.619s effective_fps=349.86 +2026-09-28 17:47:27.576 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=196661 infer_calls=196661 feedback_calls=196661 infer_wait=526.943s infer_obs_pack=2.946s env_step=19.213s feedback_total=14.144s feedback_obs_pack=3.787s feedback_info_pack=0.653s effective_fps=349.16 +2026-09-28 17:47:57.578 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=207390 infer_calls=207390 feedback_calls=207390 infer_wait=554.404s infer_obs_pack=3.110s env_step=20.279s feedback_total=14.931s feedback_obs_pack=4.003s feedback_info_pack=0.688s effective_fps=349.89 +2026-09-28 17:48:27.578 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=217394 infer_calls=217394 feedback_calls=217394 infer_wait=582.092s infer_obs_pack=3.259s env_step=21.241s feedback_total=15.643s feedback_obs_pack=4.191s feedback_info_pack=0.719s effective_fps=349.38 +2026-09-28 17:48:57.578 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=228241 infer_calls=228241 feedback_calls=228241 infer_wait=609.586s infer_obs_pack=3.421s env_step=22.295s feedback_total=16.414s feedback_obs_pack=4.396s feedback_info_pack=0.754s effective_fps=350.22 +2026-09-28 17:49:27.579 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=238217 infer_calls=238217 feedback_calls=238217 infer_wait=637.217s infer_obs_pack=3.570s env_step=23.268s feedback_total=17.161s feedback_obs_pack=4.593s feedback_info_pack=0.787s effective_fps=349.69 +2026-09-28 17:49:57.581 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=248654 infer_calls=248654 feedback_calls=248654 infer_wait=664.811s infer_obs_pack=3.723s env_step=24.285s feedback_total=17.912s feedback_obs_pack=4.794s feedback_info_pack=0.822s effective_fps=349.86 +2026-09-28 17:50:27.582 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=258417 infer_calls=258417 feedback_calls=258417 infer_wait=692.655s infer_obs_pack=3.864s env_step=25.190s feedback_total=18.587s feedback_obs_pack=4.972s feedback_info_pack=0.852s effective_fps=349.07 +2026-09-28 17:50:57.583 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=268648 infer_calls=268648 feedback_calls=268648 infer_wait=720.346s infer_obs_pack=4.017s env_step=26.159s feedback_total=19.311s feedback_obs_pack=5.166s feedback_info_pack=0.885s effective_fps=348.97 +2026-09-28 17:51:28.097 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=278530 infer_calls=278530 feedback_calls=278530 infer_wait=748.680s infer_obs_pack=4.158s env_step=27.075s feedback_total=19.988s feedback_obs_pack=5.343s feedback_info_pack=0.913s effective_fps=348.21 +2026-09-28 17:51:58.099 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=289126 infer_calls=289126 feedback_calls=289126 infer_wait=776.211s infer_obs_pack=4.321s env_step=28.111s feedback_total=20.756s feedback_obs_pack=5.553s feedback_info_pack=0.948s effective_fps=348.60 +2026-09-28 17:52:28.101 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=299118 infer_calls=299118 feedback_calls=299118 infer_wait=803.855s infer_obs_pack=4.469s env_step=29.107s feedback_total=21.480s feedback_obs_pack=5.746s feedback_info_pack=0.979s effective_fps=348.25 +2026-09-28 17:52:58.104 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=309680 infer_calls=309680 feedback_calls=309680 infer_wait=831.370s infer_obs_pack=4.633s env_step=30.137s feedback_total=22.258s feedback_obs_pack=5.955s feedback_info_pack=1.016s effective_fps=348.58 +2026-09-28 17:53:28.105 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=319740 infer_calls=319740 feedback_calls=319740 infer_wait=859.105s infer_obs_pack=4.782s env_step=31.089s feedback_total=22.948s feedback_obs_pack=6.141s feedback_info_pack=1.046s effective_fps=348.33 +2026-09-28 17:53:58.108 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=330645 infer_calls=330645 feedback_calls=330645 infer_wait=886.617s infer_obs_pack=4.945s env_step=32.118s feedback_total=23.733s feedback_obs_pack=6.346s feedback_info_pack=1.081s effective_fps=349.00 +2026-09-28 17:54:28.109 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=340455 infer_calls=340455 feedback_calls=340455 infer_wait=914.324s infer_obs_pack=5.093s env_step=33.082s feedback_total=24.444s feedback_obs_pack=6.537s feedback_info_pack=1.112s effective_fps=348.49 +2026-09-28 17:54:58.110 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=350968 infer_calls=350968 feedback_calls=350968 infer_wait=941.870s infer_obs_pack=5.250s env_step=34.112s feedback_total=25.203s feedback_obs_pack=6.740s feedback_info_pack=1.145s effective_fps=348.72 +2026-09-28 17:55:28.113 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=360712 infer_calls=360712 feedback_calls=360712 infer_wait=969.734s infer_obs_pack=5.388s env_step=35.012s feedback_total=25.863s feedback_obs_pack=6.915s feedback_info_pack=1.177s effective_fps=348.18 +2026-09-28 17:55:58.115 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=371124 infer_calls=371124 feedback_calls=371124 infer_wait=997.475s infer_obs_pack=5.537s env_step=35.959s feedback_total=26.569s feedback_obs_pack=7.106s feedback_info_pack=1.207s effective_fps=348.30 +2026-09-28 17:56:28.702 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=380930 infer_calls=380930 feedback_calls=380930 infer_wait=1025.907s infer_obs_pack=5.680s env_step=36.847s feedback_total=27.251s feedback_obs_pack=7.284s feedback_info_pack=1.237s effective_fps=347.66 +2026-09-28 17:56:58.703 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=391143 infer_calls=391143 feedback_calls=391143 infer_wait=1053.631s infer_obs_pack=5.831s env_step=37.799s feedback_total=27.964s feedback_obs_pack=7.475s feedback_info_pack=1.268s effective_fps=347.61 +2026-09-28 17:57:29.861 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=401410 infer_calls=401410 feedback_calls=401410 infer_wait=1082.405s infer_obs_pack=5.985s env_step=38.801s feedback_total=28.700s feedback_obs_pack=7.676s feedback_info_pack=1.301s effective_fps=347.27 +2026-09-28 17:57:53.865 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=1102.688s infer_obs_pack=6.106s env_step=39.583s feedback_total=29.288s feedback_obs_pack=7.829s feedback_info_pack=1.326s effective_fps=347.81 +2026-09-28 17:57:53.866 | INFO | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/local/client-seed2.log b/experiments/e43-cross-machine-training/results/local/client-seed2.log new file mode 100644 index 0000000..0aa9672 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/client-seed2.log @@ -0,0 +1,49 @@ +2026-09-28 17:37:59.617 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.classic.classic_env: pygame is not installed. Please install it with pip install "plugrl-env-client[classic]". +2026-09-28 17:37:59.617 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.atari.atari_env: Atari is not installed. Please install it with the 'atari' extra, e.g. 'pip install plugrl-env-client[atari]' +2026-09-28 17:37:59.618 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.d4rl.d4rl_env: d4rl is not installed. Please install it with pip install "plugrl-env-client[d4rl]". +2026-09-28 17:37:59.619 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.libero.libero_env: libero is not installed. Please install it with pip install "plugrl-env-client[libero]". +2026-09-28 17:37:59.619 | WARNING | plugrl_env_client.envs:_load_env_modules:27 - Skip loading env module plugrl_env_client.envs.robomimic.robomimic_env: Robomimic is not installed. Please install it with the 'robomimic' extra, e.g. 'pip install plugrl-env-client[robomimic]' +2026-09-28 17:37:59.661 | INFO | __main__:main:83 - Starting env client exp_name=e43-local-seed2 output_dir=runs/e43-local-seed2 recorder={'episode_freq': 0, 'thread0_only': True, 'record_video': False, 'video_fps': 30.0, 'record_full_rollout': False, 'record_obs_stats': True, 'record_episode_metrics': True, 'metric_window': 100} +2026-09-28 17:37:59.695 | INFO | plugrl_env_client.agent.websocket_env_client_agent:_wait_for_server:67 - Waiting for server at ws://127.0.0.1:9782... +2026-09-28 17:37:59.697 | INFO | plugrl_env_client.runner.run:report_server_metadata:56 - Server metadata: {'protocol_version': 1, 'server': 'plugrl-server', 'server_version': '0.1.0', 'algorithm': 'FPOAlgorithm', 'policy': 'FPOPolicy', 'action_dim': 6, 'action_horizon': 1} +2026-09-28 17:38:29.701 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=10830 infer_calls=10830 feedback_calls=10830 infer_wait=27.570s infer_obs_pack=0.154s env_step=0.988s feedback_total=0.780s feedback_obs_pack=0.203s feedback_info_pack=0.035s effective_fps=367.22 +2026-09-28 17:38:59.702 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=20998 infer_calls=20998 feedback_calls=20998 infer_wait=55.247s infer_obs_pack=0.315s env_step=1.939s feedback_total=1.494s feedback_obs_pack=0.396s feedback_info_pack=0.070s effective_fps=355.93 +2026-09-28 17:39:29.703 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=31802 infer_calls=31802 feedback_calls=31802 infer_wait=82.750s infer_obs_pack=0.476s env_step=2.970s feedback_total=2.279s feedback_obs_pack=0.604s feedback_info_pack=0.104s effective_fps=359.45 +2026-09-28 17:39:59.705 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=42189 infer_calls=42189 feedback_calls=42189 infer_wait=110.410s infer_obs_pack=0.625s env_step=3.941s feedback_total=3.005s feedback_obs_pack=0.799s feedback_info_pack=0.136s effective_fps=357.59 +2026-09-28 17:40:29.705 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=53115 infer_calls=53115 feedback_calls=53115 infer_wait=137.955s infer_obs_pack=0.783s env_step=4.969s feedback_total=3.762s feedback_obs_pack=1.004s feedback_info_pack=0.171s effective_fps=360.18 +2026-09-28 17:40:59.706 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=63517 infer_calls=63517 feedback_calls=63517 infer_wait=165.544s infer_obs_pack=0.934s env_step=5.979s feedback_total=4.509s feedback_obs_pack=1.203s feedback_info_pack=0.205s effective_fps=358.92 +2026-09-28 17:41:29.893 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=73730 infer_calls=73730 feedback_calls=73730 infer_wait=193.355s infer_obs_pack=1.087s env_step=6.974s feedback_total=5.235s feedback_obs_pack=1.399s feedback_info_pack=0.240s effective_fps=356.79 +2026-09-28 17:41:59.894 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=84679 infer_calls=84679 feedback_calls=84679 infer_wait=220.869s infer_obs_pack=1.250s env_step=8.014s feedback_total=5.999s feedback_obs_pack=1.603s feedback_info_pack=0.274s effective_fps=358.61 +2026-09-28 17:42:29.896 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=94802 infer_calls=94802 feedback_calls=94802 infer_wait=248.502s infer_obs_pack=1.402s env_step=9.013s feedback_total=6.729s feedback_obs_pack=1.794s feedback_info_pack=0.307s effective_fps=356.87 +2026-09-28 17:42:59.897 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=105670 infer_calls=105670 feedback_calls=105670 infer_wait=275.973s infer_obs_pack=1.566s env_step=10.077s feedback_total=7.500s feedback_obs_pack=1.999s feedback_info_pack=0.340s effective_fps=358.06 +2026-09-28 17:43:29.899 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=115758 infer_calls=115758 feedback_calls=115758 infer_wait=303.662s infer_obs_pack=1.719s env_step=11.046s feedback_total=8.214s feedback_obs_pack=2.188s feedback_info_pack=0.372s effective_fps=356.57 +2026-09-28 17:43:59.900 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=126568 infer_calls=126568 feedback_calls=126568 infer_wait=331.163s infer_obs_pack=1.884s env_step=12.088s feedback_total=8.989s feedback_obs_pack=2.391s feedback_info_pack=0.411s effective_fps=357.41 +2026-09-28 17:44:29.901 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=136704 infer_calls=136704 feedback_calls=136704 infer_wait=358.846s infer_obs_pack=2.037s env_step=13.059s feedback_total=9.703s feedback_obs_pack=2.579s feedback_info_pack=0.445s effective_fps=356.33 +2026-09-28 17:45:01.167 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=147458 infer_calls=147458 feedback_calls=147458 infer_wait=387.642s infer_obs_pack=2.194s env_step=14.098s feedback_total=10.462s feedback_obs_pack=2.780s feedback_info_pack=0.478s effective_fps=355.84 +2026-09-28 17:45:31.168 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=158187 infer_calls=158187 feedback_calls=158187 infer_wait=415.173s infer_obs_pack=2.354s env_step=15.117s feedback_total=11.237s feedback_obs_pack=2.983s feedback_info_pack=0.511s effective_fps=356.37 +2026-09-28 17:46:01.170 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=168278 infer_calls=168278 feedback_calls=168278 infer_wait=442.837s infer_obs_pack=2.502s env_step=16.092s feedback_total=11.955s feedback_obs_pack=3.174s feedback_info_pack=0.542s effective_fps=355.48 +2026-09-28 17:46:31.172 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=179246 infer_calls=179246 feedback_calls=179246 infer_wait=470.277s infer_obs_pack=2.666s env_step=17.166s feedback_total=12.737s feedback_obs_pack=3.380s feedback_info_pack=0.576s effective_fps=356.46 +2026-09-28 17:47:01.173 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=189378 infer_calls=189378 feedback_calls=189378 infer_wait=497.915s infer_obs_pack=2.822s env_step=18.164s feedback_total=13.455s feedback_obs_pack=3.571s feedback_info_pack=0.610s effective_fps=355.74 +2026-09-28 17:47:31.176 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=200087 infer_calls=200087 feedback_calls=200087 infer_wait=525.389s infer_obs_pack=2.984s env_step=19.228s feedback_total=14.227s feedback_obs_pack=3.778s feedback_info_pack=0.644s effective_fps=356.14 +2026-09-28 17:48:01.178 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=210298 infer_calls=210298 feedback_calls=210298 infer_wait=553.032s infer_obs_pack=3.140s env_step=20.199s feedback_total=14.952s feedback_obs_pack=3.973s feedback_info_pack=0.677s effective_fps=355.64 +2026-09-28 17:48:31.180 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=221028 infer_calls=221028 feedback_calls=221028 infer_wait=580.563s infer_obs_pack=3.304s env_step=21.227s feedback_total=15.717s feedback_obs_pack=4.178s feedback_info_pack=0.713s effective_fps=356.03 +2026-09-28 17:49:01.182 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=231283 infer_calls=231283 feedback_calls=231283 infer_wait=608.238s infer_obs_pack=3.456s env_step=22.193s feedback_total=16.435s feedback_obs_pack=4.370s feedback_info_pack=0.744s effective_fps=355.64 +2026-09-28 17:49:32.093 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=241666 infer_calls=241666 feedback_calls=241666 infer_wait=636.731s infer_obs_pack=3.609s env_step=23.197s feedback_total=17.186s feedback_obs_pack=4.568s feedback_info_pack=0.779s effective_fps=355.01 +2026-09-28 17:50:02.094 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=252055 infer_calls=252055 feedback_calls=252055 infer_wait=664.387s infer_obs_pack=3.771s env_step=24.166s feedback_total=17.919s feedback_obs_pack=4.762s feedback_info_pack=0.810s effective_fps=354.89 +2026-09-28 17:50:33.147 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=262146 infer_calls=262146 feedback_calls=262146 infer_wait=693.166s infer_obs_pack=3.921s env_step=25.114s feedback_total=18.625s feedback_obs_pack=4.953s feedback_info_pack=0.841s effective_fps=353.86 +2026-09-28 17:51:03.147 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=272677 infer_calls=272677 feedback_calls=272677 infer_wait=720.899s infer_obs_pack=4.068s env_step=26.065s feedback_total=19.322s feedback_obs_pack=5.140s feedback_info_pack=0.872s effective_fps=353.96 +2026-09-28 17:51:33.427 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=282626 infer_calls=282626 feedback_calls=282626 infer_wait=749.008s infer_obs_pack=4.209s env_step=26.958s feedback_total=20.018s feedback_obs_pack=5.325s feedback_info_pack=0.902s effective_fps=353.20 +2026-09-28 17:52:03.432 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=293529 infer_calls=293529 feedback_calls=293529 infer_wait=776.503s infer_obs_pack=4.371s env_step=28.002s feedback_total=20.798s feedback_obs_pack=5.532s feedback_info_pack=0.937s effective_fps=353.79 +2026-09-28 17:52:33.433 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=303446 infer_calls=303446 feedback_calls=303446 infer_wait=804.206s infer_obs_pack=4.519s env_step=28.948s feedback_total=21.524s feedback_obs_pack=5.721s feedback_info_pack=0.969s effective_fps=353.17 +2026-09-28 17:53:03.434 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=314149 infer_calls=314149 feedback_calls=314149 infer_wait=831.743s infer_obs_pack=4.684s env_step=29.963s feedback_total=22.289s feedback_obs_pack=5.923s feedback_info_pack=1.003s effective_fps=353.50 +2026-09-28 17:53:33.435 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=324067 infer_calls=324067 feedback_calls=324067 infer_wait=859.453s infer_obs_pack=4.832s env_step=30.908s feedback_total=23.009s feedback_obs_pack=6.113s feedback_info_pack=1.037s effective_fps=352.94 +2026-09-28 17:54:03.435 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=334910 infer_calls=334910 feedback_calls=334910 infer_wait=886.971s infer_obs_pack=4.991s env_step=31.944s feedback_total=23.780s feedback_obs_pack=6.321s feedback_info_pack=1.070s effective_fps=353.40 +2026-09-28 17:54:33.437 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=344839 infer_calls=344839 feedback_calls=344839 infer_wait=914.660s infer_obs_pack=5.141s env_step=32.907s feedback_total=24.496s feedback_obs_pack=6.507s feedback_info_pack=1.101s effective_fps=352.88 +2026-09-28 17:55:03.438 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=355405 infer_calls=355405 feedback_calls=355405 infer_wait=942.280s infer_obs_pack=5.298s env_step=33.886s feedback_total=25.241s feedback_obs_pack=6.700s feedback_info_pack=1.135s effective_fps=353.04 +2026-09-28 17:55:33.439 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=364952 infer_calls=364952 feedback_calls=364952 infer_wait=970.208s infer_obs_pack=5.435s env_step=34.748s feedback_total=25.884s feedback_obs_pack=6.870s feedback_info_pack=1.163s effective_fps=352.18 +2026-09-28 17:56:03.441 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=375284 infer_calls=375284 feedback_calls=375284 infer_wait=997.922s infer_obs_pack=5.583s env_step=35.697s feedback_total=26.586s feedback_obs_pack=7.055s feedback_info_pack=1.193s effective_fps=352.12 +2026-09-28 17:56:34.142 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=385026 infer_calls=385026 feedback_calls=385026 infer_wait=1026.421s infer_obs_pack=5.741s env_step=36.602s feedback_total=27.263s feedback_obs_pack=7.237s feedback_info_pack=1.222s effective_fps=351.29 +2026-09-28 17:57:04.143 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=395438 infer_calls=395438 feedback_calls=395438 infer_wait=1054.099s infer_obs_pack=5.892s env_step=37.563s feedback_total=28.004s feedback_obs_pack=7.422s feedback_info_pack=1.254s effective_fps=351.33 +2026-09-28 17:57:34.145 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Intermediate rollout timing summary: env_steps=405567 infer_calls=405567 feedback_calls=405567 infer_wait=1081.817s infer_obs_pack=6.042s env_step=38.518s feedback_total=28.711s feedback_obs_pack=7.609s feedback_info_pack=1.286s effective_fps=351.11 +2026-09-28 17:57:46.029 | INFO | plugrl_env_client.runner.rollout:log_timing_summary:98 - Final rollout timing summary: env_steps=409601 infer_calls=409601 feedback_calls=409601 infer_wait=1091.056s infer_obs_pack=6.102s env_step=38.899s feedback_total=28.982s feedback_obs_pack=7.684s feedback_info_pack=1.298s effective_fps=351.58 +2026-09-28 17:57:46.030 | INFO | plugrl_env_client.runner.run:run:156 - Server signalled the end of the run; collection stopped after 409/419 episodes. diff --git a/experiments/e43-cross-machine-training/results/local/client-summary-seed0.json b/experiments/e43-cross-machine-training/results/local/client-summary-seed0.json new file mode 100644 index 0000000..f8c860f --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/client-summary-seed0.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-local-seed0", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 1130.982177734375, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 1138.1556630530395, + "infer_obs_pack_s": 6.06455692043528, + "env_step_s": 39.254850561730564, + "feedback_s": 29.146010104566813, + "feedback_obs_pack_s": 7.702043684199452, + "feedback_info_pack_s": 1.299566077068448, + "collect_time_s": 1212.6210806397721, + "effective_fps": 337.78152676011274, + "infer_wait_frac": 0.9385913548959209, + "infer_obs_pack_frac": 0.005001197008084053, + "env_step_frac": 0.03237190181537989, + "feedback_frac": 0.02403554628061516 + } +} diff --git a/experiments/e43-cross-machine-training/results/local/client-summary-seed1.json b/experiments/e43-cross-machine-training/results/local/client-summary-seed1.json new file mode 100644 index 0000000..acd130d --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/client-summary-seed1.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-local-seed1", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 728.420166015625, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 1102.6878664772958, + "infer_obs_pack_s": 6.10606533754617, + "env_step_s": 39.583459712099284, + "feedback_s": 29.287589349318296, + "feedback_obs_pack_s": 7.829329504165798, + "feedback_info_pack_s": 1.3255081390962005, + "collect_time_s": 1177.6649808762595, + "effective_fps": 347.80774384174197, + "infer_wait_frac": 0.9363340885425872, + "infer_obs_pack_frac": 0.005184891659937837, + "env_step_frac": 0.033611816904538176, + "feedback_frac": 0.024869202892936853 + } +} diff --git a/experiments/e43-cross-machine-training/results/local/client-summary-seed2.json b/experiments/e43-cross-machine-training/results/local/client-summary-seed2.json new file mode 100644 index 0000000..c05cb92 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/client-summary-seed2.json @@ -0,0 +1,27 @@ +{ + "exp_name": "e43-local-seed2", + "process_id": 0, + "total_processes": null, + "completed_episodes": 409, + "total_episode_count": 409, + "metric_window": 100, + "mean_return": 577.3640747070312, + "mean_success_rate": 0.0, + "timing": { + "env_steps": 409601, + "infer_calls": 409601, + "feedback_calls": 409601, + "infer_wait_s": 1091.0561992097646, + "infer_obs_pack_s": 6.102110865525901, + "env_step_s": 38.89879705617204, + "feedback_s": 28.9821773073636, + "feedback_obs_pack_s": 7.684091416187584, + "feedback_info_pack_s": 1.2983896555379033, + "collect_time_s": 1165.0392844388261, + "effective_fps": 351.5769858329677, + "infer_wait_frac": 0.9364973471562398, + "infer_obs_pack_frac": 0.0052376867862143845, + "env_step_frac": 0.033388399494965304, + "feedback_frac": 0.02487656656258049 + } +} diff --git a/experiments/e43-cross-machine-training/results/local/server-seed0.log b/experiments/e43-cross-machine-training/results/local/server-seed0.log new file mode 100644 index 0000000..546b729 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/server-seed0.log @@ -0,0 +1,23 @@ +17:37:48|INFO|plugrl_server version: 0.1.0 +17:37:48|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +17:37:48|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +17:37:48|INFO|Seeded python, numpy and torch with 0. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +17:37:48|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed0 +17:37:48|INFO|Policy created... +17:37:48|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +17:37:48|INFO|Algorithm created: + +17:37:48|INFO|Agent Server is listening on 0.0.0.0:9780 +17:41:51|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed0/81921 +17:45:53|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed0/163841 +17:49:59|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed0/245761 +17:54:09|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed0/327681 +17:58:24|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed0/409601 +17:58:24|INFO|Stopping server as the algorithm signaled to stop. +17:58:24|INFO|Shutdown started: aborting pending infer requests. +17:58:24|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +17:58:24|INFO|Shutdown closing 1 websocket connection(s). +17:58:24|INFO|Shutdown interrupted an in-flight inference request from ('127.0.0.1', 36784). +17:58:24|INFO|WebSocket server closed. +17:58:24|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/local/server-seed1.log b/experiments/e43-cross-machine-training/results/local/server-seed1.log new file mode 100644 index 0000000..e2a4c99 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/server-seed1.log @@ -0,0 +1,23 @@ +17:37:53|INFO|plugrl_server version: 0.1.0 +17:37:53|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +17:37:53|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +17:37:53|INFO|Seeded python, numpy and torch with 1. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +17:37:53|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed1 +17:37:53|INFO|Policy created... +17:37:53|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +17:37:53|INFO|Algorithm created: + +17:37:53|INFO|Agent Server is listening on 0.0.0.0:9781 +17:41:49|INFO|Checkpoint saved at step 81921 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed1/81921 +17:45:51|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed1/163841 +17:49:50|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed1/245761 +17:53:50|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed1/327681 +17:57:53|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed1/409601 +17:57:53|INFO|Stopping server as the algorithm signaled to stop. +17:57:53|INFO|Shutdown started: aborting pending infer requests. +17:57:53|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +17:57:53|INFO|Shutdown closing 1 websocket connection(s). +17:57:53|INFO|Shutdown interrupted an in-flight inference request from ('127.0.0.1', 39242). +17:57:53|INFO|WebSocket server closed. +17:57:53|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/local/server-seed2.log b/experiments/e43-cross-machine-training/results/local/server-seed2.log new file mode 100644 index 0000000..1dbde60 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/local/server-seed2.log @@ -0,0 +1,23 @@ +17:37:58|INFO|plugrl_server version: 0.1.0 +17:37:58|INFO|Algorithm: fpo, Config: FPOAlgoConfig(global_steps=409600, buffer_size=4096, output_mode='u_but_supervise_as_eps', fpo_playground_trick=True, treat_truncated_as_done=False, discounting=0.995, reward_scaling=10.0, gae_lambda=0.95, batch_size=1024, num_updates_per_batch=16, learning_rate=0.0003, value_loss_coeff=0.25, clipping_epsilon=0.05, normalize_advantage=True, n_critic_warmup_itrs=0, max_policy_drift=0.0, n_samples_per_action=8, discretize_t_for_training=True, average_losses_before_exp=True, cfm_loss_steps=None, cfm_loss_dims=None, cfm_loss_sum_over_steps=False, ratio_per_sample=False, critic_learning_rate=None, adam_eps=1e-08, weight_decay=0.0, actor_adam_beta2=0.999, max_grad_norm=None, normalize_advantage_per_minibatch=False, cfm_loss_huber_delta=None, save_interval=20, policy_checkpoint_path=None, restore='all', master_weights_device=None) +17:37:58|INFO|Policy: fpo-policy, Config: FPOPolicyConfig(algo='fpo', device=device(type='cpu'), obs_dim=17, action_dim=6, flow_steps=10, timestep_embed_dim=8, action_horizon=1, feather_std=0.0, policy_mlp_output_scale=0.25, normalize_observations=True, freeze_obs_stats=False, hidden_dims=(32, 32, 32, 32), value_hidden_dims=(256, 256, 256, 256, 256), state_keys=('obs',)) +17:37:58|INFO|Seeded python, numpy and torch with 2. One env client is then reproducible; several are not, because batch composition depends on when their requests arrive. +17:37:58|INFO|Checkpoint Manager created: + at /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed2 +17:37:58|INFO|Policy created... +17:37:58|INFO|Initialized RolloutBuffer buffer_size=4096 action_shape=(4096, 1, 6) value_shape=(4096, 1) +17:37:58|INFO|Algorithm created: + +17:37:58|INFO|Agent Server is listening on 0.0.0.0:9782 +17:41:53|INFO|Checkpoint saved at step 81920 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed2/81920 +17:45:48|INFO|Checkpoint saved at step 163841 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed2/163841 +17:49:44|INFO|Checkpoint saved at step 245761 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed2/245761 +17:53:44|INFO|Checkpoint saved at step 327681 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed2/327681 +17:57:46|INFO|Checkpoint saved at step 409601 to /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/results/local/fpo/fpo-policy/halfcheetah-local-seed2/409601 +17:57:46|INFO|Stopping server as the algorithm signaled to stop. +17:57:46|INFO|Shutdown started: aborting pending infer requests. +17:57:46|INFO|Shutdown cleanup finished: pending_futures=1, drained_requests=1 +17:57:46|INFO|Shutdown closing 1 websocket connection(s). +17:57:46|INFO|Shutdown interrupted an in-flight inference request from ('127.0.0.1', 43114). +17:57:46|INFO|WebSocket server closed. +17:57:46|INFO|Scheduler task cancelled and cleaned up. diff --git a/experiments/e43-cross-machine-training/results/pilot.txt b/experiments/e43-cross-machine-training/results/pilot.txt new file mode 100644 index 0000000..1d8307d --- /dev/null +++ b/experiments/e43-cross-machine-training/results/pilot.txt @@ -0,0 +1,40 @@ +E43 pilot, 2026-09-28 17:13-17:25 EDT, before PROTOCOL.md + +Machines + server guangzhao: Linux workstation at Cornell, wired (enp128s31f6), 24 cores, CPU only + client Windows 11 laptop at Cornell: Intel Wi-Fi 6 AX201, link rate 459 Mbps + link Tailscale, peer-to-peer (guangzhao sees the laptop at 10.48.91.111:41641, + not relayed); guangzhao's public address refuses inbound (TCP to + 128.84.216.41:22 timed out), so Tailscale is the path + code server: exp/e43-cross-machine-training (main 16df640 + this directory) + env client 931ab56 on both machines; plugrl-protocol 3a59528 for the ladder + mujoco 3.14.0, gymnasium 1.3.0, numpy 2.3.2 on both; laptop Python 3.11.14 (uv) + +Training, 8,192 steps (2 iterations), seeds 0 and 1, both arms at once + per step, iteration 1 -> 2 on the server's clock (4,096 steps): + local 2.950 2.936 ms + cross 5.736 5.671 ms difference 2.76 ms per step (mean) + per call, from the env client's summary.json: + infer wait env step feedback send + local s0 2.548 0.091 0.072 ms + local s1 2.527 0.090 0.069 + cross s0 4.632 0.563 0.453 + cross s1 4.582 0.543 0.443 + return per iteration: + local s0 -292.8568 -265.7560 s1 -324.9155 -240.8591 + cross s0 -344.8546 -305.6646 s1 -318.3506 -325.2844 + local seed 0 run twice more: -292.8568 -265.7560 both times (identical) + +Physics, HalfCheetah-v5 from seed 0 under one fixed action sequence, no network + max |laptop - guangzhao| over the observation: + step 0 0 step 1 8.9e-16 step 2 2.7e-15 step 10 1.2e-12 + step 50 4.0e-7 step 100 0.82 step 200 8.3 step 1000 6.4 + +Ladder, 200 exchanges, one repetition (mean ms per exchange) + payload lo rtt ts rtt phys rtt phys pack phys unpack + state 0.122 0.119 3.781 0.008 0.036 + 48 KiB 0.176 0.158 9.103 0.027 0.040 + 184 KiB 0.347 0.372 21.757 0.141 0.047 + 588 KiB 0.891 0.676 55.687 1.529 0.052 + +Throughput, laptop -> guangzhao, 64 MiB each: 19.0 21.7 23.5 MB/s diff --git a/experiments/e43-cross-machine-training/results/throughput.tsv b/experiments/e43-cross-machine-training/results/throughput.tsv new file mode 100644 index 0000000..f7cb3e7 --- /dev/null +++ b/experiments/e43-cross-machine-training/results/throughput.tsv @@ -0,0 +1,6 @@ +rep mib seconds mb_per_s +1 64.0 3.273 20.5 +2 64.0 3.724 18.0 +3 64.0 3.427 19.6 +4 64.0 4.224 15.9 +5 64.0 4.675 14.4 diff --git a/experiments/e43-cross-machine-training/results/verdicts.txt b/experiments/e43-cross-machine-training/results/verdicts.txt new file mode 100644 index 0000000..606ba2d --- /dev/null +++ b/experiments/e43-cross-machine-training/results/verdicts.txt @@ -0,0 +1,79 @@ +P1 both arms run end to end + local seed 0 HOLDS (100 iterations, 5 checkpoints, client rc 0) + local seed 1 HOLDS (100 iterations, 5 checkpoints, client rc 0) + local seed 2 HOLDS (100 iterations, 5 checkpoints, client rc 0) + cross seed 0 HOLDS (100 iterations, 5 checkpoints, client rc 0) + cross seed 1 HOLDS (100 iterations, 5 checkpoints, client rc 0) + cross seed 2 HOLDS (100 iterations, 5 checkpoints, client rc 0) + +V1 the arms are what they say + local seed 0 client log names 127.0.0.1: yes + local seed 1 client log names 127.0.0.1: yes + local seed 2 client log names 127.0.0.1: yes + cross seed 0 client log names 100.75.226.89: yes + cross seed 1 client log names 100.75.226.89: yes + cross seed 2 client log names 100.75.226.89: yes + six server configurations identical: yes + PASS + +status learns: iterations 91-100 at least +200 over iteration 1, on 2 of 3 + local seed 0 -292.9 -> 1262.0 (+1554.9) it50 462.3 + local seed 1 -324.9 -> 694.5 (+1019.5) it50 824.1 + local seed 2 -250.5 -> 725.3 (+975.8) it50 721.6 + cross seed 0 -344.9 -> 2059.6 (+2404.5) it50 1229.8 + cross seed 1 -318.4 -> 2126.7 (+2445.1) it50 1120.5 + cross seed 2 -249.8 -> 946.4 (+1196.2) it50 628.5 + +P2 the cross arm learns + 3 of 3 HOLDS + +P3 the local arm learns + 3 of 3 HOLDS + +P4 the cross arm's extra time, iteration 1 -> 100, against 1119 s (±25%) + seed 0 local 1223 s cross 2476 s extra 1253 s in + seed 1 local 1187 s cross 2475 s extra 1288 s in + seed 2 local 1175 s cross 2530 s extra 1356 s in + 3 of 3 HOLDS + +V2 ladder runs complete + every rung and payload has 5 valid runs: yes + +P5 lo and ts agree within 0.25 ms + 0 KiB lo 0.099 ts 0.107 |d| 0.008 + 48 KiB lo 0.161 ts 0.172 |d| 0.012 + 184 KiB lo 0.293 ts 0.306 |d| 0.013 + 588 KiB lo 0.947 ts 0.700 |d| 0.248 + HOLDS + +P6 phys = phys(states) + 2 x bytes / B, B = median throughput 18.0 MB/s (20.5, 18.0, 19.6, 15.9, 14.4) + 48 KiB predicted 8.28 ms measured 8.01 ms in + 184 KiB predicted 23.72 ms measured 21.38 ms in + 588 KiB predicted 69.72 ms measured 65.80 ms in + HOLDS + +reported: ladder medians, pack / rtt / unpack ms + lo 0 KiB 0.002 / 0.099 / 0.003 + lo 48 KiB 0.006 / 0.161 / 0.004 + lo 184 KiB 0.026 / 0.293 / 0.004 + lo 588 KiB 0.131 / 0.947 / 0.007 + ts 0 KiB 0.002 / 0.107 / 0.002 + ts 48 KiB 0.006 / 0.172 / 0.004 + ts 184 KiB 0.026 / 0.306 / 0.005 + ts 588 KiB 0.130 / 0.700 / 0.007 + phys 0 KiB 0.008 / 2.815 / 0.026 + phys 48 KiB 0.056 / 8.006 / 0.049 + phys 184 KiB 0.266 / 21.377 / 0.052 + phys 588 KiB 1.675 / 65.798 / 0.060 + +reported: env client timing per call (infer wait / env step / feedback send, ms) + local seed 0 2.779 / 0.096 / 0.071 + local seed 1 2.692 / 0.097 / 0.072 + local seed 2 2.664 / 0.095 / 0.071 + cross seed 0 4.968 / 0.497 / 0.405 + cross seed 1 4.959 / 0.497 / 0.404 + cross seed 2 5.078 / 0.496 / 0.409 + +reported: E10's ratio with phys at 184 KiB: 21.4 ms against pi0.5's 100 ms forward = 21% + +wrote /home/guangzhao/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/summary.tsv diff --git a/experiments/e43-cross-machine-training/server_side.sh b/experiments/e43-cross-machine-training/server_side.sh new file mode 100644 index 0000000..8fbec96 --- /dev/null +++ b/experiments/e43-cross-machine-training/server_side.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# E43's training servers, on guangzhao: the fpo-policy · FPO · HalfCheetah +# cell of the coverage figure, E16 Phase A's command lines unchanged. +# +# bash server_side.sh local STEPS [OUT_DIR] # servers and env clients here +# bash server_side.sh cross STEPS [OUT_DIR] # servers only; the env clients +# # connect from another machine +# # (client_side.sh) +# +# One server per seed, as in E16. The server seed initialises the policy and +# the client seed the environment, and the two arms use the same pairs. +set -uo pipefail + +ARM="$1" +STEPS="$2" +HERE="$(cd "$(dirname "$0")" && pwd)" +OUT="${3:-$HERE/results/$ARM}" +case "$OUT" in /*) ;; *) OUT="$(pwd)/$OUT" ;; esac +case "$ARM" in + local) PORT_BASE="${PORT_BASE:-9780}" ;; + cross) PORT_BASE="${PORT_BASE:-9790}" ;; + *) echo "arm must be local or cross" >&2; exit 2 ;; +esac + +SEEDS="${SEEDS:-0 1 2}" +BUFFER="${BUFFER:-4096}" +SERVER_DIR="$(cd "$HERE/../.." && pwd)" +CLIENT_DIR="${PLUGRL_ENV_CLIENT:-$SERVER_DIR/../plugrl-env-client}" +SERVER_PY="${PLUGRL_SERVER_PYTHON:-$SERVER_DIR/../plugrl-server/.venv/bin/python}" +CLIENT_PY="${PLUGRL_CLIENT_PYTHON:-$CLIENT_DIR/.venv/bin/python}" +export OMP_NUM_THREADS=1 +mkdir -p "$OUT" + +echo "arm: $ARM ports $PORT_BASE+seed" +echo "server: $SERVER_PY (src $SERVER_DIR/src)" +[ "$ARM" = local ] && echo "client: $CLIENT_PY" +echo "out: $OUT" +echo "steps: $STEPS seeds: $SEEDS" +date '+start %F %T' + +run_seed() { + local seed="$1" port=$((PORT_BASE + $1)) + (cd "$SERVER_DIR" && exec env PYTHONPATH="$SERVER_DIR/src" "$SERVER_PY" -m plugrl_server.cli \ + fpo-policy default fpo default \ + --port "$port" --seed "$seed" --policy.device cpu \ + --algo.global-steps "$STEPS" --algo.buffer-size "$BUFFER" \ + --algo.save-interval 20 \ + --no-show-progress-bar --no-show-metric-table \ + --checkpoint-base-dir "$OUT" --exp-name "halfcheetah-$ARM-seed$seed" --overwrite \ + > "$OUT/server-seed$seed.log" 2>&1) & + local server_pid=$! + + local waited=0 + until grep -q "is listening on" "$OUT/server-seed$seed.log" 2>/dev/null; do + sleep 1 + waited=$((waited + 1)) + if [ "$waited" -ge 180 ]; then + echo "seed $seed: server never listened; see $OUT/server-seed$seed.log" >&2 + return 1 + fi + done + echo "READY seed $seed port $port at $(date '+%T')" + + if [ "$ARM" = local ]; then + (cd "$CLIENT_DIR" && "$CLIENT_PY" -m plugrl_env_client.cli mujoco-v1 \ + --server-port "$port" --server-host 127.0.0.1 \ + --num-envs 1 --num-episodes $((STEPS / 1000 + 10)) \ + --runner.replan-steps 1 --runner.seed "$seed" \ + --exp-name "e43-local-seed$seed" \ + > "$OUT/client-seed$seed.log" 2>&1) + echo "seed $seed client rc=$? at $(date '+%T')" + fi + wait "$server_pid" + echo "seed $seed server rc=$? at $(date '+%T')" +} + +PIDS=() +for seed in $SEEDS; do + run_seed "$seed" & + PIDS+=($!) + sleep 5 +done +FAILED=0 +for pid in "${PIDS[@]}"; do wait "$pid" || FAILED=$((FAILED + 1)); done + +date '+end %F %T' +echo "failed seeds: $FAILED" +echo "ARM_DONE $ARM" diff --git a/experiments/e43-cross-machine-training/summarise.py b/experiments/e43-cross-machine-training/summarise.py new file mode 100644 index 0000000..6b28ad1 --- /dev/null +++ b/experiments/e43-cross-machine-training/summarise.py @@ -0,0 +1,243 @@ +"""Read E43's verdicts, in PROTOCOL.md's order. + + python summarise.py [results-dir] + +Expects, under results/: + local/ cross/ the servers' runs and logs (server_side.sh) + local.out cross.out server_side.sh's own output + local/client-seed{s}.log the local arm's client logs + local/client-summary-seed{s}.json + cross-clients/ client_side.sh's output, client logs, and + client-summary-seed{s}.json from the laptop + throughput.tsv ladder.tsv Part B, from the laptop +Writes summary.tsv next to this file. +""" + +import csv +import json +import pathlib +import re +import statistics +import sys + +from tensorboard.backend.event_processing.event_accumulator import EventAccumulator + +HERE = pathlib.Path(__file__).resolve().parent +R = pathlib.Path(sys.argv[1]).resolve() if len(sys.argv) > 1 else HERE / "results" +SEEDS = (0, 1, 2) +ITERS = 100 +GAIN = 200.0 +TS_IP = "100.75.226.89" +PREDICTED_EXTRA_S = 99 * 4096 * 2.76e-3 +IMAGE_BYTES = { + 48: 128 * 128 * 3, + 184: 224 * 224 * 3 + 112 * 112 * 3, + 588: 448 * 448 * 3, +} + + +def read(p: pathlib.Path) -> str: + return p.read_text(encoding="utf-8", errors="replace") if p.exists() else "" + + +def rewards(arm: str, seed: int): + run = R / arm / "fpo" / "fpo-policy" / f"halfcheetah-{arm}-seed{seed}" + events = sorted((run / "tensorboard").glob("events.*")) + if not events: + return run, [] + acc = EventAccumulator(str(events[-1]), size_guidance={"scalars": 0}) + acc.Reload() + return run, acc.Scalars("rollout/reward") if "rollout/reward" in acc.Tags()[ + "scalars" + ] else [] + + +def client_rc(arm: str, seed: int): + out = ( + read(R / "local.out") + if arm == "local" + else read(R / "cross-clients" / "client_side.out") + ) + m = re.search(rf"seed {seed} client rc=(\d+)", out) + return int(m.group(1)) if m else None + + +def client_log(arm: str, seed: int) -> str: + return read( + R / ("local" if arm == "local" else "cross-clients") / f"client-seed{seed}.log" + ) + + +def main() -> int: + runs = {} + print("P1 both arms run end to end") + p1 = True + for arm in ("local", "cross"): + for seed in SEEDS: + run, r = rewards(arm, seed) + log = read(R / arm / f"server-seed{seed}.log") + saves = ( + [p for p in run.iterdir() if p.name.isdigit()] if run.exists() else [] + ) + rc = client_rc(arm, seed) + ok = ( + len(r) >= ITERS + and len(saves) >= 5 + and bool(log) + and "Traceback" not in log + and rc == 0 + ) + p1 &= ok + runs[(arm, seed)] = (r, log) + print( + f" {arm:5s} seed {seed} {'HOLDS' if ok else 'FAILS'} ({len(r)} iterations, {len(saves)} checkpoints, client rc {rc})" + ) + + print("\nV1 the arms are what they say") + configs = set() + v1 = True + for (arm, seed), (_, log) in runs.items(): + m = re.search(r"Algorithm: fpo, Config: (FPOAlgoConfig\(.*?\))\s*$", log, re.M) + configs.add(m.group(1) if m else None) + host = TS_IP if arm == "cross" else "127.0.0.1" + connected = host in client_log(arm, seed) + v1 &= connected + print( + f" {arm:5s} seed {seed} client log names {host}: {'yes' if connected else 'NO'}" + ) + same = len(configs) == 1 and None not in configs + v1 &= same + print(f" six server configurations identical: {'yes' if same else 'NO'}") + print(f" {'PASS' if v1 else 'FAIL'}") + + print( + f"\nstatus learns: iterations 91-100 at least +{GAIN:.0f} over iteration 1, on 2 of 3" + ) + gains, spans = {}, {} + lines = ["arm\tseed\tfirst\tit50\tlast10\tgain\tspan_s"] + for arm in ("local", "cross"): + for seed in SEEDS: + r, _ = runs[(arm, seed)] + if len(r) < ITERS: + gains[(arm, seed)] = float("nan") + spans[(arm, seed)] = float("nan") + continue + v = [e.value for e in r] + first, last10 = v[0], statistics.mean(v[90:100]) + gains[(arm, seed)] = last10 - first + spans[(arm, seed)] = r[99].wall_time - r[0].wall_time + print( + f" {arm:5s} seed {seed} {first:8.1f} -> {last10:8.1f} ({last10 - first:+.1f}) it50 {v[49]:.1f}" + ) + lines.append( + f"{arm}\t{seed}\t{first:.1f}\t{v[49]:.1f}\t{last10:.1f}\t{last10 - first:+.1f}\t{spans[(arm, seed)]:.1f}" + ) + ok_read = p1 and v1 + for name, arm in ( + ("P2 the cross arm learns", "cross"), + ("P3 the local arm learns", "local"), + ): + n = sum(gains[(arm, s)] >= GAIN for s in SEEDS) + verdict = "NOT READ" if not ok_read else ("HOLDS" if n >= 2 else "FALSIFIED") + print(f"\n{name}\n {n} of 3 {verdict}") + + print( + f"\nP4 the cross arm's extra time, iteration 1 -> 100, against {PREDICTED_EXTRA_S:.0f} s (±25%)" + ) + within = 0 + for seed in SEEDS: + extra = spans[("cross", seed)] - spans[("local", seed)] + hit = 0.75 * PREDICTED_EXTRA_S <= extra <= 1.25 * PREDICTED_EXTRA_S + within += hit + print( + f" seed {seed} local {spans[('local', seed)]:.0f} s cross {spans[('cross', seed)]:.0f} s extra {extra:.0f} s {'in' if hit else 'OUT'}" + ) + print( + f" {within} of 3 {'NOT READ' if not ok_read else ('HOLDS' if within >= 2 else 'FALSIFIED')}" + ) + + print("\nV2 ladder runs complete") + rows = list( + csv.DictReader(open(R / "ladder.tsv", encoding="utf-8"), delimiter="\t") + ) + cells = {} + for row in rows: + if int(row["exchanges"]) == 1000: + cells.setdefault((row["rung"], int(row["payload_kib"])), []).append(row) + v2 = all( + len(cells.get((rung, k), [])) == 5 + for rung in ("lo", "ts", "phys") + for k in (0, 48, 184, 588) + ) + print(f" every rung and payload has 5 valid runs: {'yes' if v2 else 'NO'}") + + def med(rung: str, k: int, field: str = "rtt_ms") -> float: + return ( + statistics.median(float(x[field]) for x in cells.get((rung, k), [])) + if cells.get((rung, k)) + else float("nan") + ) + + print("\nP5 lo and ts agree within 0.25 ms") + p5 = True + for k in (0, 48, 184, 588): + d = abs(med("ts", k) - med("lo", k)) + p5 &= d <= 0.25 + print( + f" {k:3d} KiB lo {med('lo', k):.3f} ts {med('ts', k):.3f} |d| {d:.3f}" + ) + print(f" {'NOT READ' if not v2 else ('HOLDS' if p5 else 'FALSIFIED')}") + + tp = [ + float(x["mb_per_s"]) + for x in csv.DictReader( + open(R / "throughput.tsv", encoding="utf-8"), delimiter="\t" + ) + ] + b = statistics.median(tp) * 1e6 + print( + f"\nP6 phys = phys(states) + 2 x bytes / B, B = median throughput {b / 1e6:.1f} MB/s ({', '.join(f'{x:.1f}' for x in tp)})" + ) + p6 = True + base = med("phys", 0) + for k, nbytes in IMAGE_BYTES.items(): + pred = base + 2 * nbytes / b * 1000 + got = med("phys", k) + hit = 0.75 * pred <= got <= 1.25 * pred + p6 &= hit + print( + f" {k:3d} KiB predicted {pred:6.2f} ms measured {got:6.2f} ms {'in' if hit else 'OUT'}" + ) + print(f" {'NOT READ' if not v2 else ('HOLDS' if p6 else 'FALSIFIED')}") + + print("\nreported: ladder medians, pack / rtt / unpack ms") + for rung in ("lo", "ts", "phys"): + for k in (0, 48, 184, 588): + print( + f" {rung:4s} {k:3d} KiB {med(rung, k, 'pack_ms'):.3f} / {med(rung, k):.3f} / {med(rung, k, 'unpack_ms'):.3f}" + ) + print( + "\nreported: env client timing per call (infer wait / env step / feedback send, ms)" + ) + for arm, d in (("local", R / "local"), ("cross", R / "cross-clients")): + for seed in SEEDS: + p = d / f"client-summary-seed{seed}.json" + if not p.exists(): + print(f" {arm:5s} seed {seed} no summary") + continue + t = json.loads(p.read_text())["timing"] + print( + f" {arm:5s} seed {seed} {1000 * t['infer_wait_s'] / t['infer_calls']:.3f} / " + f"{1000 * t['env_step_s'] / t['env_steps']:.3f} / {1000 * t['feedback_s'] / t['feedback_calls']:.3f}" + ) + print( + f"\nreported: E10's ratio with phys at 184 KiB: {med('phys', 184):.1f} ms against pi0.5's 100 ms forward = {med('phys', 184) / 100:.0%}" + ) + + (HERE / "summary.tsv").write_text("\n".join(lines) + "\n", encoding="utf-8") + print(f"\nwrote {HERE / 'summary.tsv'}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/experiments/e43-cross-machine-training/summary.tsv b/experiments/e43-cross-machine-training/summary.tsv new file mode 100644 index 0000000..2f40d0c --- /dev/null +++ b/experiments/e43-cross-machine-training/summary.tsv @@ -0,0 +1,7 @@ +arm seed first it50 last10 gain span_s +local 0 -292.9 462.3 1262.0 +1554.9 1222.9 +local 1 -324.9 824.1 694.5 +1019.5 1187.4 +local 2 -250.5 721.6 725.3 +975.8 1174.6 +cross 0 -344.9 1229.8 2059.6 +2404.5 2475.8 +cross 1 -318.4 1120.5 2126.7 +2445.1 2475.4 +cross 2 -249.8 628.5 946.4 +1196.2 2530.4 diff --git a/experiments/e43-cross-machine-training/throughput.py b/experiments/e43-cross-machine-training/throughput.py new file mode 100644 index 0000000..c3367b6 --- /dev/null +++ b/experiments/e43-cross-machine-training/throughput.py @@ -0,0 +1,49 @@ +"""Raw TCP throughput between the two machines, to read the ladder against. + + python throughput.py sink PORT # on guangzhao: receive and time + python throughput.py send HOST PORT MIB # on the laptop: send MIB MiB + +The sink prints MiB received, seconds, and MB/s (10^6 bytes), timed from the +first byte to the connection closing. +""" + +import socket +import sys +import time + + +def sink(port: int) -> None: + srv = socket.socket() + srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + srv.bind(("0.0.0.0", port)) + srv.listen(1) + print("listening", flush=True) + conn, _ = srv.accept() + total, start = 0, None + while True: + chunk = conn.recv(1 << 20) + if not chunk: + break + if start is None: + start = time.perf_counter() + total += len(chunk) + if start is None: + print("0.0 0.000 0.0", flush=True) + return + secs = time.perf_counter() - start + print(f"{total / 2**20:.1f}\t{secs:.3f}\t{total / secs / 1e6:.1f}", flush=True) + + +def send(host: str, port: int, mib: int) -> None: + s = socket.create_connection((host, port)) + block = bytes(1 << 20) + for _ in range(mib): + s.sendall(block) + s.close() + + +if __name__ == "__main__": + if sys.argv[1] == "sink": + sink(int(sys.argv[2])) + else: + send(sys.argv[2], int(sys.argv[3]), int(sys.argv[4])) diff --git a/experiments/e43-cross-machine-training/throughput.sh b/experiments/e43-cross-machine-training/throughput.sh new file mode 100644 index 0000000..0a1fe06 --- /dev/null +++ b/experiments/e43-cross-machine-training/throughput.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Laptop -> guangzhao TCP throughput over Tailscale, REPS times, MIB each. +# bash throughput.sh REPS MIB OUT_TSV +set -uo pipefail +REPS="$1" +MIB="$2" +TSV="$3" +HERE="$(cd "$(dirname "$0")" && pwd)" +REMOTE="${REMOTE:-guangzhao-memeseeks}" +TS_IP="${TS_IP:-100.75.226.89}" +LOCAL_PY="${PLUGRL_CLIENT_DIR:?set PLUGRL_CLIENT_DIR}/.venv/Scripts/python.exe" +R_SCRIPT='~/zuogou/plugrl/e43-reg/experiments/e43-cross-machine-training/throughput.py' +printf 'rep\tmib\tseconds\tmb_per_s\n' > "$TSV" +for rep in $(seq "$REPS"); do + port=$((9860 + rep)) + ssh "$REMOTE" "(nohup python3 $R_SCRIPT sink $port > /tmp/e43tp_$port.log 2>&1 &); for i in \$(seq 50); do grep -q listening /tmp/e43tp_$port.log && exit 0; sleep 0.1; done; exit 1" || { echo "sink failed" >&2; continue; } + "$LOCAL_PY" "$HERE/throughput.py" send "$TS_IP" "$port" "$MIB" + sleep 1 + line=$(ssh "$REMOTE" "tail -1 /tmp/e43tp_$port.log") + printf '%s\t%s\n' "$rep" "$line" | tee -a "$TSV" +done