Kind of true, but not exactly. Mathematically almost anything other than straight down will result in an orbit around the center of mass as long as you ignore air resistance, it will just be a highly elliptical orbit. However, the two factors of 'around the center of mass' and 'ignoring air resistance' are critically important.
Technically if you threw a baseball it would orbit around the center of mass if you ignore air resistance. The problem is that the highly elliptical around the center of mass intersects the surface of the Earth, which is obviously bad. Even if you were to fire a baseball at a velocity of approximately 27 times the speed of sound (orbital velocity at the surface of the Earth) it wouldn't stay orbiting because the air resistance would slow it down (well, mathematically it would slow it down. It would actually burst into flames and be destroyed) and it would crash back to Earth.
So if you shoot a rocket 'straight up' it will be dragged into a highly elliptical orbit (I'm not sure the Coriolis effect is the correct force, but things such as Earth's own orbit around the Sun will mess up the perfectly 'straight up' angle) but unless you put tremendous amounts of thrust into it that orbit won't be stable as perigee will probably be below the surface of the Earth.
Even if you did put enough energy into the rocket such orbits are of very limited use. We much prefer to use circular orbits.