Swimming is resistance work. Having the body fight agaisnt water mass, will create potential mild muscle gain, and a thermogenic fat burning effect, as it will increase Heart rate and increase Metabolism during and for a few hours after the swimming.
In terms of giving you good physical condition, it's not effective enough in my opinion, you would need weight training too. Any competitive swimmers do weights, fact, as it helps them to be faster and more endurance based athletes.
In your case the emphasis on both those factors is reduced, as you aren't doing it for competitive reasons.
Swimming is good Cardio, and being resistance work, might stimulate mild muscle increases / fat loss, but downsides could include muscle loss with any Fat, as it's much more taxing Cardio than something like Jogging or Cycling, becasue the muscles must fight the water weight, increasing fatigue, and also potentially Glycogen usage too, compared to outdoor or machine based Cardio types. Though such a comparison should be made over the same time frame.
You couldn't compare 20 minutes of Swimming, to 40 minutes of Cycling, and make such assumptions as that's lopsided.
Then you have issues with your stroke action, speed of strokes, and the way you swim, e.g. crawl, backstroke, butterfly, combinations etc etc, which are all variables to consider reagrds your muscle stimulus.
So you can do it, but you should do weights, as you'll probably get better muscle condition. Any potential muscle increases from such resistance work like Swimming, could be uneven, depending on your execution of your sessions, and therefore lead to postural inadequacies.