If what you have stated is true, your husband is not balancing his duties towards his wife with his duties towards his parents. He is certainly sacrificing your rights over the right of his parents. You have every right to ask him what you have mentioned. We also balance our duties and never do one at the expense of the other; the Prophet (peace be upon him) said, "Your eyes have rights; your body has rights, and your wife has rights; and give each one his or her due." (Reported by Timidhi on the authority of Abd Allah b. Amar b. al-Aas).

So, even as your husband must respect and honor his parents, he does not have to spend all of his time with them, he needs to respect your rights, as well. By failing in his essential duty towards you,  he is guilty of a major offence.

If he is persiting on this behaviour, I advise you to seek marriage counselling.