It's funny that you mention having seen something similar in a novel, because that's where I got the idea from - not the novel that you mentioned, though, but one just published by Baen, in 2009. The novel is called 'Fledgling', and is set in the Liaden universe of Steve Miller and Sharon Lee.
You can get all of the Liaden novels from Baen, in electronic format (no DRM) for $6 each OR LESS - most of them were included in bundles that brought the cost down to about $4 each. The only ones that you'd really have to purchase separately are "The Tomorrow Log", "Dragon Tide", and "Fledgling", and the forthcoming "Saltation". There is that which is Not Traveller in the novels - and yet, they feel as much like Traveller as Moon's "Vatta's War" or Drake's "The Republic of Cinnabar Navy" series do, and I could easily see myself begging Mongoose to negotiate with Baen to get the rights to Travellerize both Liaden and RCN. Or myself begging permission and information from Steve and Sharon to turn the Liadens into a Contact!/Club Room article or twelve.
At any rate, there are a couple of scenes in Fledgling where the focus character gets involved in playing a game called 'bowli ball', which has a ball that reacts sort of as I described; the actual mechanism was not really described; it reads as a handwave (so I assume that the ball has an alloy of unobtanium and handwavium in it, to allow the mass to be manipulated). I thought the idea of the game was one that would be worth Travellerizing for Freelance Traveller, along with another recreational feature described in the novel, so I posted the question.
(N.B. from now on, and when/if it appears in Freelance Traveller, I'll be calling it a "varia-ball". I'll happily acknowledge Mr Miller and Ms Lee as the source of the inspiration, but no way in hell am I going to use the name that they created for their novel, not without their permission...)
The biggest problems with the non-grav versions of the varia-ball is that the effective mass doesn't change, and neither do the external forces on the ball, so the trajectory is still going to be fundamentally ballistic and predictable, which detracts from the game. Ideally, I'd want something about the size of a soccer ball or basketball, or maybe down to half that volume would work, that is generally as featureless as a spaldeen, but can be made to behave like a wiffle.
o: 
