Yes, there's no problem doing what you suggest. Each ZMQ socket operates completely distinct from the other sockets in your code. It often makes sense to mix connection protocols to optimize communication the way you're looking to do.
One assumption I'm making here is that your broker has 2 sets of sockets: client facing sockets that you can connect via TCP and worker facing sockets that you can connect via some other protocol. If both client and worker are connecting to the same socket on the broker, then they must connect via the same protocol.
The only thing to consider is whether your workers will always reside in the same process as your broker, or if it might grow to a point where it makes sense to separate them. But, if you define your socket connections in some sort of configurable way, rather than baking it into the code, even that could be a relatively easy fix if you decide to change things down the line.