In terms of innovation, I’ve observed that users will modify your product (if possible) to suit their particular needs. Some of their modifications have been very creative and brilliant. You can get some great concepts and ideas to bring back to your design team. Innovation does not have to start from you, Marketing, or Engineering alone. In fact, there's a whole area of study on this by Professor Erc Von Hippel called "User Innovation". An example is soldiers mounting machine guns on cargo planes in Vietnam out of their own ingenuity, which led to flying gunships and attack helicopters. You will also see and learn where you product and its usage fits in the total universe of the customer. Learn the user’s workflow, his/her bottlenecks, and challenges. Remember: people don’t buy products for features; they buy products to solve a problem that they have.
A very important side benefit is that customers usually very much appreciate that you are there in a non-selling situation mainly to learn about their situation, make their lives better, and stand behind your products and them. You also get to know them on a personal level and develop references. There is nothing better than knowing and treating your customers as friends. All things being equal, people will prefer to do business with someone whom they like.
Note: Eric von Hippel is an amazing person who truly beieves in open and democratic innovation, including free and open source software (FOSS). He has a very informative website where his papers and books can be downloaded for free! The man walks the walk.