If we follow Paul's narrative in the sequence he gave it to the church there, reading through chapter 3 and on into chapter 4 in the order he wrote it, we do not come across any reference to idols until 4:8.
Galatians 4:8.. Howbeit then, when ye knew not God, ye did service unto them which by nature are no gods.
You claim that is what elements oo the world and tutors and governors refer to in verses 2-3, or at least elements of the world (if perhaps you recognize tutors and governors of verse 3 as LAW). That is by no means the antecedent to verses written before it. Note what is written before verse 8.
Galatians 4:1-7.. Now I say, That the heir, as long as he is a child, differeth nothing from a servant, though he be lord of all; ..(2).. But is under tutors and governors until the time appointed of the father. ..(3).. Even so we, when we were children, were in bondage under the elements of the world: ..(4).. But when the fulness of the time was come, God sent forth his Son, made of a woman, made under the law, ..(5).. To redeem them that were under the law, that we might receive the adoption of sons. ..(6).. And because ye are sons, God hath sent forth the Spirit of his Son into your hearts, crying, Abba, Father. ..(7).. Wherefore thou art no more a servant, but a son; and if a son, then an heir of God through Christ.
To follow Paul's context, you cannot take verse 8 and claim that is the antecedent of what verses 2 and 3 refer to as tutors and governors, or the elements of the world. How on earth can tutors and governors refer to the same thing that "service unto them which by nature are no gods" refers to?..
Tutors and governors are the schoolmaster that Paul mentioned in chapter 3! And that is LAW! How can you not see that? LAW is the antecedent for the reference to elements of the world, and tutors and governors.
To take something written AFTER verses 3 and 4, such as verse 8's reference to "service unto them which by nature are no gods," and claim that is "the elements of the world" is to violate all grammar and comprehensive reading. ANTE in ANTECEDENT means BEFORE. You do not write about something, such as "elements of the world" and "governors and tutors" and mean idolatry and paganism that is not written until afterwards, if those phrases actually refer to idolatry. You look at what is written before verses 3 and 4 of chapter 4 to find the antecedent, and lo and behold that is LAW in Chapter 3. In fact, it repeats LAW in 4:5.
To say that his readers knew what he meant about idolatry, as though idolatry was what those phrases in verse 3 and 4 actually meant, without Paul having to actually write the reference to idolatry before verses 3 and 4 is grand assumption and inserting into the text things which you cannot prove whatsoever.
The reason he mentioned "service unto them which by nature are no gods" is because these pagans weer involved in the Old Covenant as converts to it before they came to Christ, and Pau noted that they were returning to bondage of law, which is what the warder references of chapter 3 already stated quite clearly.
He, therefore, mentioned "service unto them which by nature are no gods" as a side thought that they were orginally pagans before they came into Judaism, after which they were saved by Christ, and not the basic reference he's using to compare to tutors and governors who oversee an heir.