You have to set value in the shopping cart item first.
ShoppingCartItemInfo product = shoppingService
This custom data is transferred from cart item to order item automatically, when the order is submitted (when you create order from shopping cart). Internally they stored as xml inside the DB.
Same goes for cartInfo and orderInfo. So you can decide where you want to keep your custom data at the order level or at the item level.