Wall Street traders and Las Vegas gamblers have a lot in common. But when a Wall Street firm set up shop taking Vegas bets, both sides got a surprise.