Cardamom fruits have a tendency to ripen very irregularly and so it is recommended that the fruits are harvested individually. In saying that, you don't want to pick them when the are very ripe as the fruit tends to split and the seeds do not dry well. What color are the seeds when the fruits fall? Have they changed color? Thank you for including lots of detail in your question, I hope we can find an answer for you.