My thought is probably not.
Photoshop defines the Overlay formula as:
Multiplies or screens the colors, depending on the base color. Patterns or colors overlay the existing pixels while preserving the highlights and shadows of the base color. The base color is not replaced but is mixed with the blend color to reflect the lightness or darkness of the original color.
So what you've really got here is two separate blend modes (multiply and screen), being chosen based on the base layer color. I'm not aware of any way to dynamically switch blend modes based on the base layer color.
As answered in this question, even implementing screen and multiply is not trivial in OpenGL, so I doubt there's a way you can do them both at the same time.