According to my research and testing, you can follow these steps to create Modal Dialog Pop Up In SharePoint Online:
1.Create a new Wiki page.

2.Click Inset ->> Web Part ->> Media and Content ->> Script Editor ->> Click Add

3.Click Edit Web Part ->> Click EDIT SNIPPET

4.Input the code in the attachment, then click Inset and OK.
<button onclick="showModalPopUp();" type="button">Click meβ</button>
<script language="javascript" type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.8.1/jquery.min.js"></script>
<script language="javascript" type="text/javascript">
function showModalPopUp() {
//Set options for Modal PopUp
var options = {
url: '/sites/xxx/Lists/List_test2', //Set the url of the page
title: 'SharePoint Modal Pop Up', //Set the title for the pop up
allowMaximize: false,
showClose: true,
width: 600,
height: 400
};
//Invoke the modal dialog by passing in the options array variable
SP.SOD.execute('sp.ui.dialog.js', 'SP.UI.ModalDialog.showModalDialog', options);
return false;
}
</script>
Output:
