My first guess is that that is a tomato hornworm. I zoomed right in on your picture and think I can see spots on the sides but some close up images of the caterpillars would be useful to confirm if you can add them. Does the caterpillar have a prominent spike, or horn, protruding from it's rear end?

**************Update******************

I moved the info you posted here and replaced the original image with the higher res one. You can edit your original question by logging in and going to the bottom of your post and clicking on the small edit button. You can then add new text and images.

It could be tomato fruitworm which also like to feed on peppers. Regardless, a dusting of diatomaceous earth might help or spraying the plants with Bacillus thurengiensis (Bt). Both of these treatments would be natural ways to control the larvae. If it is the tomato fruitworm then you should remove the fruit with holes. But, if it is another pest then the hole may actually be an exit hole rather than an entry hole in which case this wouldn't help to reduce the numbers. If you can get some close ups of the larvae then that would be helpful for diagnosing the pest.