I am a recent college grad who has fallen for a working man who is recently divorced and has one kid. ?

I am 24 and I recently graduated from college. I can go where ever I want, to start working, I feel as if the world is my oyster. However, I have just met a wonderful man who has changed me and has made me fallen head over heels in love like never before. He is 24 also and just got divorced from a woman who got pregnant and forced marriage onto him if he wanted to have any part in his sons life. He loves his son more than life it's self and I love that. I just feel like I am always going to be second which is fine because children should come first, but he works so hard just to make ends meet and still comes up short. I just feel like I can do anything work wherever, travel the world and living a stress free life. With him things just seem like it's going to be a little bit of a struggle. I have decided to get a job in the town I have been going to school in and just stick around here for a while to see how things go between us. Is this stupid? Should I just go on with my life and forget about this man with so much baggage? I could see myself being with him forever and having his children someday. I just am confused. I could work at it and be with him or take the easy route and hit the highway.