Use glReadPixels
to read the image from the framebuffer. You can either pack directly to client memory, or you may use a intermediary Pixel Buffer Object to allow for asynchronous operation and memory mapping.
Do memory map from GPU to CPU cost large time?
You've to copy the data from GPU memory to system memory anyway. If it's done through a memory map (i.e. triggering DMA transfer by page fault) or explicitly by glGetBufferSubData (using a PBO) or glReadPixels makes no big difference regarding bandwidth utilization and performance. Most important is, that the pixel format requested matches the format of the data on the GPU side, otherwise a costly conversion has to be performed.
Since you want to feed the data to libx264
I'd say using a mapped buffer would work the best, so that you can interleave libx264 processing of the previous frame with asynchronous transfer of the next frame.