As far as I know you cannot do multi-key sorting in Netsuite's version of Freemarker
What you can do is use the sequence built-ins and do your own grouping and sorting.
a bit more detail in code taken from a working example is here: Does Oracle NetSuite Advanced PDF Template have "Group by" and "SUM" Functions?
A quick bit of code for your case might look like:
<#if record.item?has_content>
<table class="itemtable" style="width: 100%; margin-top: 10px;"><!-- start items -->
<thead>
<tr>
<th colspan="5">Style</th>
<th align="left" colspan="11">Description</th>
<th align="center" colspan="7">Size/Quantity</th>
<th align="center" colspan="4">Unit Price</th>
<th align="center" colspan="3">Quantity</th>
</tr>
</thead>
<#assign depts = []>
<#list record.item?sort_by("department") as lineitem>
<#assign item_dept = lineitem.department>
<#if depts?seq_contains(item_dept)>
<#else>
<#assign depts = depts + [item_dept]>
</#if>
</#list>
<#list depts?sort as dept>
<#list record.item?sort_by("cust_brand") as lineitem>
<#if linteitem.department == dept>
<tr>
...
</tr
</#if>
</#list>
</table>
</#if>