You can leverage the table DDL view AWS published a few months ago (https://github.com/awslabs/amazon-redshift-utils/blob/master/src/AdminViews/v_generate_tbl_ddl.sql) by picking out the constraint component and parsing out the key columns:
select schemaname,tablename, substring(ddl,charindex('(',ddl)+1, charindex(')',ddl)-1-charindex('(',ddl))
from
(
SELECT
n.nspname AS schemaname
,c.relname AS tablename
,200000000 + CAST(con.oid AS INT) AS seq
,'\t,' + pg_get_constraintdef(con.oid) AS ddl
FROM
pg_constraint AS con
INNER JOIN pg_class AS c
ON c.relnamespace = con.connamespace
AND c.relfilenode = con.conrelid
INNER JOIN pg_namespace AS n
ON n.oid = c.relnamespace
WHERE c.relkind = 'r'
ORDER BY seq
)
Note that this query also gives you foreign key columns. It's easy enough to filter those out by appending the query with
where ddl like '%PRIMARY KEY%'