Since your user groups are dynamic (i.e. users can invite other users to a thread at any time), there is no strong reason to enforce the new thread validation. I would give up one of the requirements. Either make the user group static (i.e. a new user should define the group members at creation time) or allow users to create any threads they want. If the restriction on new threads is there to prevent spamming, use something else like CAPTCHAs (or a limit on # of new threads per hour per user) to accomplish the same objective.
You are going to run into SQL performance issues if you try to satisfy both requirements, since you'll be running multiple full-table scans on the database every time someone tries to start a new thread. Also, at any time you can end up in a situation where 2 threads have the same users. E.g. Thread 1 has users A, B and C. Thread 2 has users A and B. What happens when user B invites C to thread 2?