The answer is similar to changing the layout for Radio Button Groups or Combo Boxes. See this question for using individual components rather than the group component.
The other option is to use a custom renderer. That question has a link to another StackOverflow question covering it, or you can look at the answer by Tim Tripcony to this question or this XSnippet which uses a custom renderer to remove table tags from Radio Button Groups. All will give you the basics, then it's a case of identifying the correct Java class for the component (the Java code for your XPage in the Local folder in Package Explorer may tell you that) and creating the relevant HTML output, inserting the using a for loop to create a new table cell and row where appropriate.
Although the renderer for the core CheckBox is not open sourced yet, the renderer for the DojoCheckbox and DojoFormWidget (which it extends) are open source.