I can just include dummy variables but that just gets impossible when the number of groups increases
By "impossible," do you mean "computationally impossible"? If so, check out the plm
package, which was designed to handle cases that would otherwise be computationally infeasible, and which permits fixed-effects IV.
Start with the plm
vignette. It will quickly make clear whether plm
is what you're looking for.
Update 2018 December 03: the estimatr
package will also do what you want. It's faster and easier to use than the plm
package.