Firstly, the seerah (biography) of the Imams shows that the Imams did not struggle for power nor did they have a craving for power. If at times they were defending their rights, it was because they believed it was their obligation to do so.
This saying of Imam Ali (a.s) "Leave me and go to others" is a good proof of the above claim.
Secondly, the hereditariness of Imamate (leadership), if accompanied with superiority to others is not a matter of objection but it is also endorsed and ratified by reason and Shari'ah. In other words, as prophethood was determined directly by God's order and He knows with whom to lay His mission, if on the basis of the same order Imamate and prophethood are decreed in a single lineage, we should accept it as was the case with Ibrahim and his two sons Ismail and Isaac who became prophets after their fathers. Similarly, after Isaac his son Jacob and then Jacob's son, Yusuf became the prophets.
There are two points to take into consideration in regard to the foregoing question:
First: If you look carefully at the lives of the Infallible Imams (a.s.), you will come to know that the Imams did not struggle for power nor did they have the desire to attain power. If at times they were defending their rights, it was because they considered it to be their obligation to do so. Go through the following two examples in this regard:
1. After the second caliph appointed six people to constitute the Council of Caliphate, Zubair gave his vote to Ali (a.s); Talha gave his vote to Utman and Sa'ad Waqas gave his to Abdur Rahman. Eventually, three people were left. If any of these people relinquished in favor of the other, he would become the caliph. However, at that time, Abdur Rahman went to Ali (a.s.) and said that he would pledge allegiance to him provided that he should act according to God's book, the Prophet's Sunnah (tradition) and Seerah of Sheikhayn (Abu Bakr and Umar). But Ali (a.s.) said that if he were to become a caliph he would act according to the Book of God and the Sunnah of His Prophet (s) in the way he understood and perceived them; thus, he rejected the caliphate because he would not act according to the Seerah of Sheikhayn. 
Some politicians are of the view that the Imam should have accepted apparently the three conditions and then he could have ignored them. In fact, every world and power-seeking individual might think in the said way but we see that the Imam (a.s) rejected the caliphate outright.
2. After the death of Uthman, the third caliph, the people swarmed to Ali's house. Addressing them, the Commander of the Faithful (a.s) said, "It is better for me to be your vizier and advisor rather than a ruler… leave me and go to others."  Does one who seeks power behave in such a way? Is it in any way compatible with any of the standards of seeking power?
We are certain that if it were not for the sake of justice, the Imams would never seek power. As the Commander of Faithful, Ali (a.s.) said, "If it were not for justice, in my view this world of yours is no better than the sneezing of a goat." 
Second: The hereditariness of prophethood, caliphate and government can be studied from two aspects:
1. The power is transferred from one individual to another due to family relationship without the second person being eligible for it. This kind of inheritance is not acceptable religiously nor is it reasonable.
2. Those who come to power owing to their eligibility not because of their family bonds. When it comes to such people, we should study the hereditariness of power by going through the history of the divine prophets and of the Imams thereafter.
1-2. After Ibrahim was appointed by God for Imamate, he prayed to God to let Imamate continue with his offspring and God also said that His covenant (Imamate) would not reach the unjust. 
2 – 2. Prophet Moses asked God to make his brother Haroon his vizier and successor. 
3-2. Sulayman son of Dawood inherited his father. 
Therefore, in all the above cases which involve inheritance, the eligibility of the individuals who replace their fathers or brothers is justifiable according to the Quran.
When it comes to Imam Ali (a.s) and the pure Imams from his offspring, there are special textual proofs from the Prophet (s) about their right to caliphate and leadership. In addition, the Prophet (s) said about Ali (a.s): "You are my heir." 
As for the Imamate of the Imams, it should be said that this inheritance is of the second type. The criterion for the caliphate and Imamate is a divine command. Because He knows best whom to lay His message with.  If on the basis of the same order Imamate and prophethood are decreed in a single lineage, we should accept it as was the case with Ibrahim and his two sons Ismail and Isaac who became prophets after their fathers. Similarly, after Isaac his son Jacob and then Jacob's son, Yusuf became prophets.
For further information, read the following indexes:
"The Imams (a.s.) and Political Leadership", question 7163 (site: 8068)
"The Political Wilayah of Ali (a.s) at Ghadir Khum", question 2570 (site: 2742).
 - See: Kulayni, Usul Kaafi, Kamraei, Muhammad Baqir, vol.2, pg. 721, Uswah Publication, 3rd edition, Qom, 1996.
 - Allamah Majlisi, Behar al-Anwar, vol.32, pg. 7 and 8, Al-Wafa Institute, Beirut, 1404 A.H.
 - Nahjul-Balaghah, pg.50, Dar al-Hijrah, Qom.
 - Al-Baqarah: 124
وَ إِذِ ابْتَلى إِبْراهیمَ رَبُّهُ بِکَلِماتٍ فَأَتَمَّهُنَّ قالَ إِنِّی جاعِلُکَ لِلنَّاسِ إِماماً قالَ وَ مِنْ ذُرِّیَّتی قالَ لا یَنالُ عَهْدِی الظَّالِمین.
 - Taha: 29 and 30. وَ اجْعَلْ لی وَزیراً مِنْ أَهْلی هارُونَ أَخی.
 - Al-Naml: 16, وَ وَرِثَ سُلَیْمانُ داوُد.
 - Sheikh Saduq, Amali, pg. 362, Islamic Library Publications, 1983. قال رسول الله ص لعلی أنت وارثی.
 - Al- An'am: 124, اللَّهُ أَعْلَمُ حَیْثُ یَجْعَلُ رِسالَتَه.