Agencies get to work on a lot of different things, surrounded by lots of other creative people. Companies often hire agencies to take off the edge work, something to gain an outer perspective, and to explore ideas. If you like that kind of work, it could be very fresh and exciting, but it also means you might not be fully let into their plans, and not able to contribute all of your good ideas. Often, these exploratory ideas also don't get shipped, and you need to be okay with that.
Working in-house on the other hand allows you to see projects through, and develop a deep expertise in a particular area. If you have both of these experiences, you’ll get the best of both worlds and be more impactful in each situation.