Najm (النجم) means 'star' or a plant that doesn't have a stem. In the verse, considering that shajar or 'tree' has been mentioned before najm, it can be concluded that what is meant by najm is 'herb' and not 'star'.
Essentially speaking, this term means to rise, and the reason why plants without stems have been called najm, is because it rises from the soil. The same applies to why stars are called najm; it is because they rise and set too.
Shajar denotes any plant with a trunk or stem, like fruit trees, wheat stalks, etc. Therefore, and also considering the fact that in the verse, shajar and najm have been mentioned next to each other, most commentators believe the prostration of the two to be their obedience before God's command; meaning that it is out of His command that they rise from the soil and out of His command that they grow.
Nonetheless, some commentators have taken najm to mean its main meaning, which is 'star', and shajar to mean 'tree' and the reason for mentioning these two together to be because they symbolize the farthest and nearest objects to us in nature.
To conclude, whether we take najm to mean 'star' or 'herb' and 'plant', their prostration will not be that of man's (i.e. putting the forehead to the ground); it will mean their total submission and humility before Allah Almighty, the same way all of creation is submitted to His will and is praising Him, each with their own tongue.
In order that the meaning of the verse and what is meant by the prostration of najm and shajar become clear to us, we must first analyze the meanings of these two verses, and then engage in interpreting the verse and see what is meant by their prostration.
Najm: As Raghib says in his book of Mufradaat: "أصل النَّجْم، الکوکب الطالع، و جمعه نُجُومٌ، و نَجَمَ، طَلَعَ، ... و منه شُبِّهَ به طلوعُ النّبات،...".
Najm originally means 'star', and sometimes refers to a plant without a stem. This term essentially means to rise, and the reason why such plants are called najm is because they rise from the soil, and the reason why stars are called najm is because they rise and set as well. Here, in surah Rahman, considering the fact that shajar (tree) has also been mentioned in the verse, the second meaning seems to be better for the term najm, meaning a plant without a stem. Therefore, in the verse, najm denotes small plants such as the squash or cucumber vine.
Shajar: "الشَّجَرُ من النّبات، ما له ساق", and "الشجر: ما قام علی ساق...". Shajar refers to all plants with a stem, including trees (because of their trunk).
Therefore, although the main meaning of najm is 'star', many commentators of the Quran have said that since in this verse, shajar and najm have been mentioned next to each other, najm will denote a plant without a stem or trunk.
Having put this introduction and the meanings of these two terms behind, we will delve into the interpretation of the verse at hand so that it becomes clear what is meant by the prostration of plants and trees.
Most commentators of the Quran, after interpreting najm as plants without stems, interpret their prostration as their total and unquestioned submission to Allah Almighty. What is meant by their submission is their compliance with the laws of nature and creation and not breaking the laws God has set for them in this world.
When quoting the major view of commentators on the meaning of najm, Allamah Tabatabai says: "… this is a good meaning to choose; what corroborates it is the fact that the words najm and shajar have been mentioned together, although shams (sun) and qamar (moon) have been mentioned before this verse and this may imply that what is meant by najm here is 'star'." He goes on to say that this noble verse states that plants and trees prostrate before God, and what is meant by prostration, is the humility and submission of these two before God's decree; they rise from the soil out of His command and it is out of His command that they grow…".
Some commentators though, have taken najm to mean its main meaning of 'star', saying: "Najm is located in the sky and shajar (which means tree) is located on the ground, and although these things seem to be soulless, they have enough understanding and feeling to be inclined towards worship of the Lord: "وَ إِنْ مِنْ شَیْءٍ إِلَّا یُسَبِّحُ بِحَمْدِهِ وَ لکِنْ لا تَفْقَهُونَ تَسْبِیحَهُمْ" (There is not a thing but celebrates His praise, but you do not understand their glorification).
Due to prostration symbolizing the highest point of humility and submission, the prostration of these things needs to be defined in a way that signifies their utmost humility and submission, and that would be their total compliance with the laws of nature that God has set for them; no star is seen to come out of its orbit, and no tree is seen that bears fruit other than its own.
Conclusion: Regardless of whether najm means 'star' or 'plant' and 'herb', their prostration does not resemble that of us (putting the forehead to the ground), rather, it denotes total submission and humility before Allah Almighty, the same way all creatures and objects are submissive to and praising of Him. The only difference is that each does it in its own tongue; each and every thing has its own way of prostrating and it is us who cannot see this with our physical eyes or understand their prostration, as the verse: "وَ لکِنْ لا تَفْقَهُونَ تَسْبِیحَهُمْ" says.
For further information in regard to this subject, please refer to the commentaries on this verse (Rahman:6).
 Raghib Isfahani, Al-Mufradat fi Gharib al-Quran, vol. 1, pp 791-792, the term "نجم".
 Ibid, vol. 1, p. 446, the term "شجر".
 Tarihi, Fakhruddin, Majma' al-Bahrayn, vol. 6, p. 173, the term "شجر".
 Tabatabai, Sayyid Muhammad Husayn, Al-Mizan, vol. 19, p. 96, Islamic Publications Office, Qum 1417 AH.
 A group of translators, Tafsir Hidayat, vol. 14, p. 291, Islamic Research Foundation of Astane Quds Razavi, Mashad, first edition, 1998.