When creating your own components, if you're closely replicating some behavior that is already in an extension library component, I highly recommend you extend that component and just add what's needed to accommodate your different functionality. This makes things much easier and you don't have to code around every little scenario that the component might be placed in.
But, if you are developing a component that is nothing like any of the extension library or core components then just ensure your component extends UIComponent or UIComponentBase. If going this route, you'll also need to create your own renderer which extends Renderer. This is what will build the on-screen representation of your component. Again, if there's already something in the core components or extension library components that closely mimics what you need then make your renderer extend that renderer. Also, don't forget to include the renderer definition in the faces-config file and the component definition in the xsp-config file or your component won't work.
As for initiating controls from the extlib.... I assume you mean can you inject them onto the page at runtime. If so the answer is absolutely yes. To add an input text field to the page where there is a container (i.e. panel, div, span, whatever) with an ID of "someContainer"
XspInputText input = new XspInputText();
input.setValue("someValue");
input.setId("someID");
UIComponent container = FacesContext.getCurrentInstance().getViewRoot().findComponent("someContainer");
container.getChildren().add(input);
To see the api for all of the core and extension library components take a look at the XPages Controls Documentation. For a more complete tutorial on creating your own components take a look at my blog for creating a custom component inside an nsf, the steps are pretty much the same for putting them into a library:
Part 1,
Part 2 and there is an example database in the Part 2 post.