It could be orange tortrix insect damage. Generally the first generation larvae of orange tortilla (Argyrotaenia franciscana) feeds on leaves, whereas the second generation larvae feeds on young fruits (usually around button area) resulting in appearance of outward scar. The subsequent generation larvae feeds on ripening fruits by making holes into the rind. This opening makes way for saprophytic fungus/bacteria resulting in decaying and early falling of fruits.

By the way did you see any larvae or adult insect on the tree? Particularly look for nests spun by larvae around fruit clusters.

This insect will have three to four generation per year.

http://ipm.ucanr.edu/PMG/r107302411.html