The plain sweater vest isn't particularly offensive. It's simple and unimposing. I think some people may not like the sweater vest because they feel it can look a bit "dorky" or something like in that direction. I understand why they feel that way. I understand that sense that is possible to get, if worn in a certain way. I think some men that may be a bit more stout shy away from the sweater vest because they feel it accentuates and draws attention to their mid section. That is a fair point and I think I would feel the same. I think these different feelings and aversions are understandable. I understand them.
That being said, I wear sweater vests like the one pictured above. Unless it is a day I am spending lots of time outside, I think I actually prefer a sweater vest over a sweater. The sweater constricts your arms and your underarms are covered by multiple layers of fabric which almost always leads to more perspiration. This is more uncomfortable and over a long period of time, the underarms of the shirt will become discolored quicker. The sweater vest avoids both these problems.
I sometimes see men wearing a sweater vest as a part of a very put-together outfit. Shirt, tie, sweater vest, sport coat. I generally don't wear the sweater vest in this way. My personal style tends to drift toward being a bit more relaxed and nonchalant, even when wearing a tie and sport coat. The sweater vest on top of the tie and under the coat feels very prim, perfect and put-together and so I personally don't end up doing this too often. It just isn't where my personal style orbits. It can absolutely look fantastic. I see guys wearing this and they look great. Super intentional, thoughtful, put-together, almost dashing. It's just not my personal style.
I tend to wear the sweater vest on a casual day, like today. Sleeves rolled up, working inside, no sport coat. That is where I am always drawn. To compliment the midnight forest green sweater vest, I chose a pair of mustard yellow socks with oxblood penny loafers. Simple, casual and natural.