Is Milo with you during your off time? I think that is the main question you have to answer. As long as you guys are together and learning new things and being active, I think you are probably doing a good job.
There are all different ways to be a family. As long as you both are physically and mentally active, I wouldn't worry to much.