I did not come across any specific program examples, but within the GTK source code I did find a usage of the "g_menu_set_attribute" function. The thing I noticed is that it did not include your final parameter of NULL. You might want to revise your program and call the function as follows:
g_menu_set_attribute (item, "custom", "s", "target");
That function command allows for a variable number of parameters based upon the format parameter (e.g. "s"), but for what you are doing, the NULL probably is providing some confusing information to the function.
If that still does not address your issue, you may continue reading what I did to experiment with the addition of a child widget to a popover menu item.
This may not be an ultimate solution, but I tried out various methods of adding a child widget to the popover menu, and only one method worked for me that appeared to produce the results you are after.
I could not seem to get the "g_menu_item_set_attribute (item, “custom”, “s”, “target”, NULL);" statement to work. So, instead what I did was utilize an existing menu model "ui" build file and inserted the "custom" property manually as noted in the following snippet (I named the ID as "widgets").
<item>
<attribute name="label">Cut</attribute>
<attribute name="action">app.cut</attribute>
<attribute name="icon">edit-cut-symbolic</attribute>
<attribute name="custom">widgets</attribute>
</item>
Then in a sample program, after the popover menu object was built using the menu model defined in the "ui" file (I called it "popmenu"), I added a GTK check button to the menu item with the "Cut" label as noted in the following code snippet.
GtkWidget *colorbutton;
. . . /* Other code here */
colorbutton = gtk_colorbutton_button_new();
. . . /* Other code here */
gtk_popover_menu_add_child (GTK_POPOVER_MENU(popmenu), colorbutton, "widgets");
. . . /* Other code here */
When I ran the program and clicked on the widget to which I attached the popover menu, I got the color button to appear. Actually, the color button replaced the menu item's label, so I don't know if that is what you ultimately want. To get both text and the color button to appear, I placed a label and the color button within a GTK box and then added the box as a child. The following sample illustration was the result.

Anyway, you might try using a "ui" file that contains the menu model and build your popover menu that way.
Regards.