I have a genuine question for which I will use an example taken to the extreme to test your principle:
Let's say that most people on earth like to consume a particular substance from which a by-product is created that reduces the amount of air available to breathe. Let's say that it is clearly determined at what point air will run out and everyone will die. In this case would it still be right to allow these people to continue to choose - to have the freedom - to consume this product? Or would it be right to use force to stop them doing so?
That's an extreme example. But yeah, they would be initiating violence. It's like asking "if someone is strangling me would I be allowed to use force to stop them?"
Perhaps the real question here is, when do a person's actions become their responsibility such that force is justified. E.g. If a man attacks another man, use of force by the second is justified. So if a bunch of people ruin a lake by leaving plastic bottles in it, what is justified against them? Direct/Indirect etc.
People own themselves and their property. They have the right to defend both. You can use other means besides violence to defend yourself. Under our current system it'd be a matter for civil courts. In a stateless society you'd most likely have private defense resolution organizations - or something similar, whatever the market produces.
I assume you're asking in terms of a stateless society.
If one man attacks another with violence, then the victim has the right to defend himself using necessary force.
The lake example would be a property rights issue. If multiple people own the lake, and someone starts polluting it - they have the right to file a civil suit against that person, most likely requesting cleanup and compensation for any damages.
If he refuses to comply - he'll likely be punished by the community denying him access to roads, shops, credit cards, etc. until the situation is resolved.
I think it's a great question and worthy of a proper response - there are members here who could do a much better job answering the question - as well as entire books written on this topic.
It's a deep rabbit hole.
A good start would be digging through here...
Subject: Property
And Stefan Molyneux gives some good ideas on how dispute resolution would work in a stateless society here...
http://www.freedomainradio.com/free/books/FDR_5_PDF_Practical_Anarchy_Audiobook.pdf
Short answer - you have the right to defend yourself with force if force is initiated against you. Most aggressions like theft and property damage would most likely be a civil matter.
Hope that helps.