You can't exchange arbitrary objects directly like that. However, they did add the ability to run an agent with runWithDocumentContext
to pass in a designated in-memory Document to use as context, which is a bit more flexible than the older NoteID parameter. You can use that to create a doc in-memory, pass it to the agent, have the agent fill it with info (say, the note IDs of the collection as a multi-value text or RT item), and then read the value back out XPages-side.
Granted, this is more awkward than your immediate desire, but it's the tool we have that doesn't resort to Ajax round-trips.