When you open a file, your ability to do so is determined by your effective uid and gid at the time you open the file.
When you change your effective uid or gid, it has no effect on any open file descriptors that you may have.
In most cases, if you have a valid file descriptor, that's all you need to read or write the resource that descriptor is connected to. The fact that you hold the valid file descriptor is supposed to be all the proof you need that you have permission to read/write the underlying resource.
When you read or write using an ordinary file descriptor, no additional authorization checks are performed. This is partly for efficiency (because those authentication checks would be expensive to perform each time), and partly so that -- this may be exactly what you are trying to do -- you can open a privileged resource, downgrade your process's privileges, and continue to access the open resource.
Bottom line: Yes, it's entirely possible for a process to use an open file descriptor to read or write a file which (based on its current uid/gid) it would not be able to open.
Footnote: What I've said is true for ordinary Unix file descriptors connected to ordinary resources (files, devices, pipes, network streams, etc.). But as @Mark Plotnick reminds in a comment, some file descriptors and underlying resources are "different" -- NFS and Linux /proc
files are two examples. For those, it's possible for additional checks to be performed at the time of read/write.