Without seeing any of your source code it's hard to offer specific details, but in general, what you need to do is get the graph's stylesheet, and then modify the edge related parameters. An example is:
mxGraph mxgraph = new mxGraph();
Object parent = mxgraph.getDefaultParent();
mxgraph.getModel().beginUpdate();
mxStylesheet stylesheet = mxgraph.getStylesheet();
Hashtable<String, Object> style = new Hashtable<>();
stylesheet.putCellStyle("ROUNDED", style);
Map<String, Object> vertexStyle = stylesheet.getDefaultVertexStyle();
vertexStyle.put(mxConstants.STYLE_FILLCOLOR, "#FFFFFF");
vertexStyle.put(mxConstants.STYLE_STROKECOLOR, "#000000");
vertexStyle.put(mxConstants.STYLE_AUTOSIZE, 1);
vertexStyle.put(mxConstants.STYLE_SPACING, "10");
vertexStyle.put(mxConstants.STYLE_ORTHOGONAL, "true");
vertexStyle.put(mxConstants.STYLE_SHAPE, mxConstants.SHAPE_ELLIPSE);
Map<String, Object> edgeStyle = stylesheet.getDefaultEdgeStyle();
edgeStyle.put(mxConstants.STYLE_EDGE, mxConstants.EDGESTYLE_ORTHOGONAL);
edgeStyle.put(mxConstants.STYLE_SHAPE, mxConstants.SHAPE_CURVE);
edgeStyle.put(mxConstants.STYLE_ENDARROW, mxConstants.ARROW_CLASSIC);
...set up your edges and vertices here, where the last parameter is "ROUNDED" (the name of the stylesheet)
mxgraph.getModel().endUpdate();