Start with STB high. Bring it low before every command, and bring it high after you send the command before sending the next.
Since you're using SG past 11, you'll need to change the display mode (Commands 1 in the datasheet) before you write data. This can be done by sending 0b00000000 to set the controller to 4-digit 14-segment mode. (This is only necessary because you are using the SG12 pin. It also means less addresses to write to later.)
Next, you want to set the data settings (Commands 2). This starts with 0b0100 and has 4 more bits. The next bit is a 0 because we're not in test mode. You'll want to increment the address each data write, so do a 0 next. The display controller is also designed to read a key matrix, but you aren't using that, so we want to add a final 00. This gives 0b01000000.
The next part is the hard part, because you want to set address you need to write to. This is where I can't easily help you, you need to understand this. Commands 3 shows all the addresses you can write to. You want to specifically affect segments 10 through 12 of every digit. This means that you want to write to bytes 1 to 3 of all the bytes at odd-numbered addresses. Since you aren't using the other pins, the easiest and fastest way to do this is to go through all the addresses. Since you don't care what you write to even addresses, you can write the same you do to odd ones. For simplicity, you'll want to start by setting the address to 0 by sending the code for addresses: 0b1100 followed by your address 0b0000. This gives 0b11000000. Now, without pulling the STB back high, start sending your data bytes. On every write, the address will be incremented. Since we previously set the number of digits to 4, we only need to go up to address 0x7. You can do this by sending 8 bytes that contain 0b0000RBG0, where the letters represent the state of the colours. Once you do this, you can set the strobe high agatin. (This assumes that the cathodes are all connected to GND so that the digit canthodes are ignored)
This is of course the very minimum. It would be far better to only write the bits you need, and leave the rest off, but I'm trying to keep things simple, even if it's not the right way of doing things.
You'll want to set some other display control settings (Commands 4). These start with 0b1000. You then want a 1 to keep the display on, followed by 111 to keep the display lit as much as possible. This gives 0b10001111.
This component is not designed for beginner use, and you are using it for an unintended purpose that far overcomplicates things. You are abusing the purpose of the part. While it's a good learning experience, you have to actually try to learn. Take the time to read the datasheet because it's likely that it won't work based off what I've said depending on how it's set up. I've tried to explain why each command is sent, not just what it is. You need to understand the commands before you use this. Go through the datasheet, and look at every command to see why I recommended to do that.