Saying most Muslims are not terrorists is like me saying most Americans are not lottery winners or murders. Even if there are 50,000 active muslim terrorists in the world (a pretty high estimate I'd hope you'd agree), that means 99.99% of Muslims are not terrorists and don't deserve to be looped into with those assholes. There were 660,000 murders in the US from 1960 to 1996 (36 years), does that mean most murderers are US citizens? Do you as a US citizen want to be associated with murderers?
Muslims in the US have the right to practice their religion without a bunch of bigoted bullshit.
And we have the right to choose to not be around them. That's all we're saying buddy.