Comp,
I cannot find anything to back your response. The rulebook says the Lookback rule is in effect when: The ball is live, the batter-runner has touched 1st base or has been declared out, and the pitcher has possession and control of the ball within the pitcher's circle. Once those three items are fulfilled it then goes on to say in part 2. Once the runner stops at a base for any reason, the runner will be declared out if leaving the base.
I say the runner is out.
In ASA, look up Rule Supplement 34-E:
A pitcher fielding a ball in the circle is an infielder and runners can leave their base. If runners leave their base, the same rule applies while the pitcher holds the ball in the circle: once the runner stops, they must decide which way to continue or be called out.