As I mentioned in the comments, your type doesn't really hold any state, so a StateMonad instance would be nonsensical for it.
However, since this is just an exercise (also based on the comments), I guess it's ok to implement the instance technically, even if it doesn't do what you'd expect it to do.
First, the compiler error you're getting tells you that the MonadState class actually takes two arguments - the type of the state and the type of the monad, where monad has to have kind * -> *, that is, have a type parameter, like Maybe, or list, or Identity.
In your case, the monad in question (not really a monad, but ok) is IO, and the type of your "state" is Char, since that's what you're getting and putting. So the declaration has to look like this:
instance MonadState Char IO where
Second, the state method doesn't have the signature (s -> s) -> m s as you claim, but rather (s -> (a, s)) -> m a. See its definition. What it's supposed to do is create a computation in monad m out of a function that takes a state and returns "result" plus new (updated) state.
Note also that this is the most general operation on the State monad, and both get and put can be expressed in terms of state:
get = state $ \s -> (s, s)
put s = state $ \_ -> ((), s)
This means that you do not have to implement get and put yourself. You only need to implement the state function, and get/put will come from default implementations.
Incidentally, this works the other way around as well: if you define get and put, the definition of state will come from the default:
state f = do
s <- get
let (a, s') = f s
put s'
return a
And now, let's see how this can actually be implemented.
The semantics of the state's parameter is this: it's a function that takes some state as input, then performs some computation based on that state, and this computation has some result a, and it also may modify the state in some way; so the function returns both the result and the new, modified state.
In your case, the way to "get" the state from your "monad" is via GetChar, and the way in which is "returns" the Char is by calling a function that you pass to it (such function is usually referred to as "continuation").
The way to "put" the state back into your "monad" is via PutChar, which takes the Char you want to "put" as a parameter, plus some IO a that represents the computation "result".
So, the way to implement state would be to (1) first "get" the Char, then (2) apply the function to it, then (3) "put" the resulting new Char, and then (3) return the "result" of the function. Putting it all together:
state f = GetChar $ \c -> let (a, c') = f c in PutChar c' (Done a)
As further exercise, I encourage you to see how get and put would unfold, starting from the definitions I gave above, and performing step-by-step substitution. Here, I'll give you a couple first steps:
get = state $ \s -> (s, s)
-- Substituting definition of `state`
= GetChar $ \c -> let (a, c') = (\s -> (s, s)) c in PutChar c' (Done a)
-- Substituting (\s -> (s, s)) c == (c, c)
= GetChar $ \c -> let (a, c') = (c, c) in PutChar c' (Done a)
= <and so on...>
