SchemDraw uses matplotlib's fontfamily and the fonts installed in the system. In order to add a custom font using .ttf file, we have to add the font to matplotlib using its font_manager as shown below.
from matplotlib import font_manager
fpath = '/path/to/custom_font.ttf'
font_manager.fontManager.addfont(fpath)
The above snippet caches the font and then the name of the font can be used in SchemDraw directly.
For example, schemdraw.config(font='custom')
to config the font globally for the whole drawing.
Thanks to Collin J. Delker for answering this on BitBucket. You can find the original answer here!