If I'm understanding you correctly you want the following:
- Have a cursor that is limited to the lower green box in your screenshot
- Still allow the native cursor to be usable(ungrabbed) when navigating outside of the game window (i.e users can click the x button and such)
I've achieved this before by rendering my own cursor and hiding the native one while in the game window. There's a few steps to accomplishing this. I should mention that this was done with a slightly older version than the current LWJGL offerings, and I'm not sure which version you are running.
Step 1: Disabling the native cursor
You want to disable the native cursor when inside your game window. Grabbing was too aggressive in my case since it disabled the mouse for a windowed game entirely. When initializing your game, try the following:
try {
final Cursor emptyCursor = new Cursor(1, 1, 0, 0, 1, BufferUtils.createIntBuffer(1), null);
Mouse.setNativeCursor(emptyCursor);
} catch (final LWJGLException e) {
//TODO: Consider better exception handling
e.printStackTrace();
}
What this does is render the native mouse as nothing while in your game window. This means the arrow you normally see will disappear when you mouse over your game. This does NOT mean that the mouse is not functional, we have just hidden it.
Step 2: Track mouse movement when gathering keyboard/mouse input
public void collectInput() {
while(Mouse.next()){
this.mouseX = Mouse.getX();
final int newMousePosition = Mouse.getY();
//Only move the cursor on the Y axis if it is between the upper and lower bounds
if(newMousePosition < this.upperBound && newMousePosition > this.lowerBound){
this.mouseY = newMousePosition;
}
this.mouseY = Display.getHeight()- Mouse.getY();
//TODO: Consider adding logging here
}
}
I had a collectInput() method that would grab keypresses/mouse events I thought were necessary. All I'm doing here is grabbing the current X/Y of the mouse in my game window. The Y coordinate is subtracted from the total display height to deal with render coordinates vs mouse coordinates later on. In my case I just used the bounds of my entire window, but you may want to consider not updating X/Y coordinates if they go outside of your green box.
EDIT
Based on your comment, I updated the code block here to not update the Y coordinate if the new coordinate is not within some boundaries. You will want to determine what your x and y boundaries are, and only update your mouse X/Y coordinates if the new coordinates gathered from the Mouse class lie within them.
Step 3: Render your own cursor
Finally, you will want to render your cursor. This looks like any other render you would do with LWJGL. My example assumes you have a texture for this:
//Render mouse texture
Color.white.bind();
this.mouseTexture.bind();
GL11.glEnable(GL11.GL_TEXTURE_2D);
GL11.glBegin(GL11.GL_QUADS);
GL11.glTexCoord2f(0, 0);
GL11.glVertex2f(this.mouseX, this.mouseY);
GL11.glTexCoord2f(1, 0);
GL11.glVertex2f(this.mouseX + this.cursorHeight, this.mouseY);
GL11.glTexCoord2f(1, 1);
GL11.glVertex2f(this.mouseX + this.cursorHeight, this.mouseY + this.cursorWidth);
GL11.glTexCoord2f(0, 1);
GL11.glVertex2f(this.mouseX, this.mouseY + this.cursorWidth);
GL11.glEnd();
GL11.glDisable(GL11.GL_TEXTURE_2D);
What this results in is that when I am using the mouse in the game window, my custom cursor is rendered. It's hard to tell from the image, but the white circle is now my cursor, and it moves wherever I move the mouse. It's important to note here that the regular native cursor is not being shown due to step 1:

But when I leave the game window, the native mouse is shown, and my custom cursor is left in the last place it received input:
