This could be due to lack of pollination. If there are not enough bees around to pollinate the flowers, they will fall off without producing fruit. Another cause may be temperature. I usually get lots of blossom production and fruit set later in the summer when temperatures are warm during the day and night, but don't usually get a lot of production from flowers set early in the season. Excess nitrogen fertilizer will also pre-dispose plants to allocate resources to leaf and shoot growth over flowers or fruit. Sometimes shifting the fertilizer regime to favor less nitrogen and more phosphorus (the "P" in "N-P-K" on your fertilizer label) will also help the plant to allocate more to blooms than leaves/shoots.
Peppers are also susceptible to blossom end rot, which is when immature fruits get what looks like a rotten splotch on the end where the flower petals connected, and may eventually fall off. This could be part of the problem you are experiencing. It is due to the inability of the plant to take up Calcium from the soil. You can see what the calcium levels are in your soil with a home soil test (usually available at gardening supply stores). To help the plants take up calcium effectively and allocate it to the developing fruits, again don't fertilize too much with nitrogen, and make sure plants are in nice soil that does not restrict root growth. Also make sure the plants don't dry out a lot in between waterings, since that messes with the consistency of root uptake of nutrients.