0

I'm looking into using the Kinect to play video games. This requires the emulation of both keyboard and mouse events. I've figured out that keyboard events with ctypes' SendInput() works in both Skyrim and Minecraft. I've also found that ctypes' mouse_event() works in both Skyrim and Minecraft for emulating mouse buttons. Those are the solutions I've found so far that seem to work for both Skyrim and Minecraft.

The main issue I'm having is with moving the player's camera around within Skyrim. I'm able to move the camera around in Minecraft using ctypes' SetUserPos() and also ctypes' mouse_event(). But neither solutions work for Skyrim. I've also tried using SendInput() to move the player's camera in Skyrim, but every time the cursor gets moved with SendInput() (even outside of Skyrim) my monitor goes blank for a moment and the camera still doesn't move (I have no idea why something like that is happening...)

So is there anyway that I could possibly be able to emulate the camera movement within Skyrim or other games that probably handle their mouse inputs similarly? I'm willing to use C/C++ for this task, but straight Python would be more preferable. Thanks for any and all suggestions you may have!

I'm also going to leave the code I used to make my monitor go blank. I'm thinking SendInput() probably won't work for camera movement within Skyrim, but maybe I just did something terribly wrong. (I also got most of the below code from a multitude of threads online.)

import ctypes
import time

# C struct redefinitions
PUL = ctypes.POINTER(ctypes.c_ulong)

class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]


class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


class POINT(ctypes.Structure):
        _fields_ = [("x", ctypes.c_ulong),
                    ("y", ctypes.c_ulong)]


class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]


class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Actuals Functions


def PressKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, hexKeyCode, 0x0008, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))


def ReleaseKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, hexKeyCode, 0x0008 | 0x0002, 0, ctypes.pointer(extra))  # lint:ok
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def MoveMouse(x, y):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    x = int(x*(65536/ctypes.windll.user32.GetSystemMetrics(0))+1)
    y = int(y*(65536/ctypes.windll.user32.GetSystemMetrics(1))+1)
    ii_.mi = MouseInput(x, y, 0, 0x0001 | 0x8000, 1, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(0), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def main():
    mouse = Mouse()
    time.sleep(3)
    MoveMouse(100, 100)

main()
MattDMo
  • 100,794
  • 21
  • 241
  • 231
MerrrLocke
  • 1
  • 1
  • 2

3 Answers3

1

You can install the PyAutoGUI GUI automation module from PyPI (run pip install pyautogui) and then call the pyautogui.moveTo() to click on a certain X and Y coordinates of the screen:

>>> import pyautogui
>>> pyautogui.moveTo(50, 100)

Behind the scenes on Windows, it makes similar ctypes code you are using, so it might work to control the camera in a game like Skyrim.

PyAutoGUI works on Windows, Mac, and Linux, and on Python 2 and 3. It also can emulate the keyboard, do mouse drags, take screenshots, and do simple image recognition of the screenshots. Full docs are at https://pyautogui.readthedocs.org/

Al Sweigart
  • 11,566
  • 10
  • 64
  • 92
  • As of 10-11-15, `pyautogui` for Python 2.7 uses the deprecated ctype function `mouse_event` to implement `pyautogui.click()`, and therefore using it will throw a WindowsError: `'File "C:\Python27\lib\site-packages\pyautogui\_pyautogui_win.py", line 480, in _sendMouseEvent raise ctypes.WinError() WindowsError: [Error 2] The system cannot find the file specified.` which I couldn't see until putting it in a `try...finally` block. I would suggest using pyautogui's `hotkeys` to imitate a keyboard shortcut instead. – Britney Smith Oct 11 '15 at 06:20
  • When the code is provided, the answer should be within the same code discussing the changes/modification to make it work. In such scenario suggestion can be part of answer but not the answer itself. – Vinod Srivastav Nov 15 '21 at 12:32
1
            import ctypes

            SendInput = ctypes.windll.user32.SendInput

            PUL = ctypes.POINTER(ctypes.c_ulong)
            class KeyBdInput(ctypes.Structure):
                _fields_ = [("wVk", ctypes.c_ushort),
                            ("wScan", ctypes.c_ushort),
                            ("dwFlags", ctypes.c_ulong),
                            ("time", ctypes.c_ulong),
                            ("dwExtraInfo", PUL)]

            class HardwareInput(ctypes.Structure):
                _fields_ = [("uMsg", ctypes.c_ulong),
                            ("wParamL", ctypes.c_short),
                            ("wParamH", ctypes.c_ushort)]

            class MouseInput(ctypes.Structure):
                _fields_ = [("dx", ctypes.c_long),
                            ("dy", ctypes.c_long),
                            ("mouseData", ctypes.c_ulong),
                            ("dwFlags", ctypes.c_ulong),
                            ("time",ctypes.c_ulong),
                            ("dwExtraInfo", PUL)]

            class Input_I(ctypes.Union):
                _fields_ = [("ki", KeyBdInput),
                            ("mi", MouseInput),
                            ("hi", HardwareInput)]

            class Input(ctypes.Structure):
                _fields_ = [("type", ctypes.c_ulong),
                            ("ii", Input_I)]

            def PressKey(hexKeyCode):
                extra = ctypes.c_ulong(0)
                ii_ = Input_I()
                ii_.ki = KeyBdInput( 0, hexKeyCode, 0x0008, 0, ctypes.pointer(extra) )
                cmd = Input( ctypes.c_ulong(1), ii_ )
                ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))

            def ReleaseKey(hexKeyCode):
                extra = ctypes.c_ulong(0)
                ii_ = Input_I()
                ii_.ki = KeyBdInput( 0, hexKeyCode, 0x0008 | 0x0002, 0, ctypes.pointer(extra) )
                cmd = Input( ctypes.c_ulong(1), ii_ )
                ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))

            def MoveMouse(x, y):
                x = 1 + int(x * 65536./1440.) # <-- monitor 1440x900
                y = 1 + int(y * 65536./900.)
                extra = ctypes.c_ulong(0)
                ii_ = Input_I()
                ii_.mi = MouseInput(x, y, 0, (0x0001 | 0x8000), 0, ctypes.pointer(extra))
                cmd = Input(ctypes.c_ulong(0), ii_)
                ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))

            def LeftClick():
                extra = ctypes.c_ulong(0)
                ii_ = Input_I()
                ii_.mi = MouseInput(0, 0, 0, 0x0002, 0, ctypes.pointer(extra))
                cmd = Input(ctypes.c_ulong(0), ii_)
                ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))

            def LeftRelease():
                extra = ctypes.c_ulong(0)
                ii_ = Input_I()
                ii_.mi = MouseInput(0, 0, 0, 0x0004, 0, ctypes.pointer(extra))
                cmd = Input(ctypes.c_ulong(0), ii_)
                ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))
Gerson José
  • 11
  • 1
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jan 30 '23 at 06:18
0

I have fixed your code and it's working for me in win10. Please see the changes and try.

This will move the cursor relative to the current position

import ctypes
import time

# C struct redefinitions
PUL = ctypes.POINTER(ctypes.c_ulong)

class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]


class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


class POINT(ctypes.Structure):
        _fields_ = [("x", ctypes.c_ulong),
                    ("y", ctypes.c_ulong)]


class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]


class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Actuals Functions


def PressKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, hexKeyCode, 0x0008, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))


def ReleaseKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, hexKeyCode, 0x0008 | 0x0002, 0, ctypes.pointer(extra))  # lint:ok
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def MoveMouse(x, y):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    #x = int(x*(65536/ctypes.windll.user32.GetSystemMetrics(0))+1)
    #y = int(y*(65536/ctypes.windll.user32.GetSystemMetrics(1))+1)
    ii_.mi = MouseInput(x, y, 0, 0x0001, 0, ctypes.pointer(extra))
    cmd = Input(ctypes.c_ulong(0), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))

def main():
    #mouse = Mouse()
    MoveMouse(1,15)
    time.sleep(3)
    MoveMouse(1,-15)
    time.sleep(3)
    MoveMouse(1,15)
    time.sleep(3)
    MoveMouse(2,15)

main()

But you can also use ctypes.windll.user32.SetCursorPos(x,y) to move directly to a pixel position.

This one will move the cursor to the pixel position in the x/y axis on the screen

import ctypes

# Move cursor to x 500 and y 500
ctypes.windll.user32.SetCursorPos(500,500)
Vinod Srivastav
  • 3,644
  • 1
  • 27
  • 40