Hello Ske, before I get into what I think let's talk about what is a Patreon, it's a monthly subscription used for raising money for, in your case, a server. The cost could be anything really, ranging from $1 to I don't even care, I sincerely doubt your going to need a $10 monthly payment, your not that big of a streamer... Yet.
Now time for what I think, your not made of money, and I think that a Patreon is a great way of sustaining a server. While most people would be angered, annoyed or even infuriated, I for one think that this can be beneficial to you as the streamer, your content is higher quality than all of the streamers your size, it's free to watch and you constantly talk to the viewers.
I myself don't get why people get so pissed off over something like this, My guess is that they think we are rich or something. My opinion: Go for it, it will separate your true subscribers from those who are there to ask the 'smart' question of: "Can you give me a shout out?"
There you have my opinion on the matter, originally I had written this more aggressively but this is a community forum so I decided to tone my aggressiveness down a notch.