Options: Leave him out of our lives with the mistress he left me for and her kids OR Try and make it work?

Long story short: My x of 5 years and I tried working it out for out 9 month old son. Appears I was the only one working on the relationship and not him...I caught him last year cheating on me with his coworker. Shes known to be a home wrecker....and premisquous. Shes mid 40s and not attractive, stays with her mother and her 3 kids who are all ruthless. I am hurt not just because he cheated but he lied, and he cheated lower than me. Of course add wood to the fire after I kick him out I find out im pregnant a month later. He kept his distance from me up until I had my son he came for a bit. About March is when we started communicating somewhat that led to our 2nd try at the relationship. They still work together and work night shift now and he left me again for the same woman. She was upset and had her bill collectors come to my address for payments?!?! We was together 2 weeks then he broke it off we agreed he would stay for our son and wed be roomates thats it. Then few days later the woman texts me etc blah blah blah and I guess they been having for the last 2 weeks he was at my place...I was like wow, she even told me that she knew he was below her and she dont care about me and that he came after her. I dont know why or how this came about everyone friends relatives coworkers both sides ask what happend and why her? I dont know I am hurt and disappointed for myself but the fact my son got use to having him around and he just up and left us. I thought we were doing ok working on it day by day and we even went on a mini family vacation, please help with some insight or opinion on this situation. On an ending note her kids 15-19 all talk about me and they reflect their mother. He would like to have our son on the weekends but im not sure thats a good idea, she told me i was a bad mother and dont know how to raise a child. i have been raising my child by myself and working full time i dont get help from friends or family either and it bothers me that she said that. I know he has some feelings with me he told me he did and i could feel it too. He says he doesnt know why he chooses her or why he does the things he does he asks himself on a daily basis is what he told me...I just think hes taking the easy way out and not ready to be a dad, he told me he loved that other woman the other night he finally told me that. I dont know what to do...