strands-arm frame_dolly: the whole leader arm keeps a subject framed and drives the omni base as its dolly
Six servo deltas per 0.1 s (ids 1..6, clipped +-5 deg) plus three base velocity commands (vx, vy, wz in [-5, 5], mapped onto the drive guard caps 0.15 m/s, 0.15 m/s, 0.6 rad/s) from what the Nicla reports (FOMO box centre,
size, visible, ToF), the six servo readings, the last command and a shot spec (where in the frame,
how big), plus the commanded base velocity (vx, vy, wz). Trained with PPO in a
MuJoCo twin of this arm (SO-101 leader + printed head, lens as a camera at the firmware FOV, identified
STS3215 servo model, calibrated limits) with occlusion rays, one-frame detector delay, pixel noise and
randomized FOV, and the base following the commanded velocity through a ramp; the size spec is a dolly offset the base can deliver (the arm alone cannot: 0.002 size per servo degree, see rl/sim/sensitivity.py).
Baseline: the head-only P step watch uses today, arm frozen at the fold.
Top: head-only P. Bottom: this policy, same subject. Left = the Nicla's view (green = shot spec, yellow = detection), right = third person. MP4.
Eval (rl/eval_frame.py, 4 seeds x 64 envs x 15 s, 256 episodes)
| controller | visible | framed | centre err | size err | contacts/step | travel deg/step |
|---|---|---|---|---|---|---|
| zero action | 0.27 | 0.00 | 1.010 | 0.080 | 0.004 | 0.10 |
| head-only P (watch.py) | 0.34 | 0.02 | 0.664 | 0.078 | 0.503 | 1.91 |
| this policy | 0.87 | 0.70 | 0.120 | 0.045 | 0.014 | 5.32 |
| this policy, arm frozen | 0.59 | 0.11 | 0.469 | 0.073 | 0.940 | 1.36 |
| frame_mobile (arm only, never drives) | 0.72 | 0.30 | 0.240 | 0.080 | 0.138 | 7.76 |
PASS: keeps the subject visible more often than the head-only P step. framed = centre error < 0.15 and size error < 0.1. The arm-frozen row is the same network with servos 1..4 held: the difference is what the arm buys. The frame_mobile row is the shipped arm-only cameraman dropped into the same env with the base parked: the difference is what driving buys. Ship rule for this policy: size err < 0.12 and visible > frame_mobile.
Contract
Input obs [1, 33] float32, output act [1, 9] float32 = servo deg per 0.1 s step, ids 1..6; base_*_cmd in [-5, 5] -> [-cap, cap] (0.15 m/s, 0.15 m/s, 0.6 rad/s).
| i | obs |
|---|---|
| 0 | s1_n |
| 1 | s2_n |
| 2 | s3_n |
| 3 | s4_n |
| 4 | s5_n |
| 5 | s6_n |
| 6 | g1_n |
| 7 | g2_n |
| 8 | g3_n |
| 9 | g4_n |
| 10 | g5_n |
| 11 | g6_n |
| 12 | ex |
| 13 | ey |
| 14 | size |
| 15 | vis |
| 16 | last_ex |
| 17 | last_ey |
| 18 | last_size |
| 19 | age_n |
| 20 | ex_star |
| 21 | ey_star |
| 22 | size_star |
| 23 | tof_n |
| 24 | a1_prev |
| 25 | a2_prev |
| 26 | a3_prev |
| 27 | a4_prev |
| 28 | a5_prev |
| 29 | a6_prev |
| 30 | base_vx_n |
| 31 | base_vy_n |
| 32 | base_wz_n |
Runner: strands_arm.policy.FramePolicy (image ey flipped to the sim's up-positive frame, size = box
height / half frame, ToF mm -> m, targets clamped through kine.clamp_pose). Never run on the real
arm yet; shadow mode (real pose, detections and ToF, nothing written) is logged in the repo's RL.md.
Source: github.com/cagataycali/strands-arm 56caa5b, rl/envs/frame_env.py, rl/sim/. PPO: {"n_envs": 256, "steps_per_env": 50, "epochs": 4, "minibatches": 8, "lr": 0.0003, "gamma": 0.98, "lam": 0.95, "clip": 0.2, "ent_coef": 0.002, "vf_coef": 0.5, "max_grad_norm": 0.5, "hidden": 128, "act_scale": 5.0, "act_dim": 9, "total_updates": 2500, "device": "cpu", "seed": 0}
Env overrides: {"w_act": 0.1, "w_jerk": 0.1, "w_size": 2.0}
- Downloads last month
- 5