It looks like the tool you're using is doing a good job at removing the weeds and their deep tap roots, which is essential to preventing them from growing back. Short of using chemicals, you've already found the most efficient way of removing them.
If you want to fill in the holes, tap the loose soil off of the weed roots. Pack that soil back into the hole, and push the dirt around the edges of the hole into the hole. If necessary, you could get a bag of topsoil to fill in the holes. (Note that topsoil is a different product from potting soil and compost.) However, if you can tolerate having those holes in your lawn, they may actually be beneficial. Lawns can benefit from occasionally being aerated to reduce soil compaction and thatch buildup.
Your lawn doesn't look especially healthy. The grass looks rather thin and clumpy, with bare dirt visible between clumps of grass. A healthy lawn with dense grass cover makes it harder for weeds to get established. Taking some basic steps to improve your lawn health will help prevent new weed problems in the future. It might benefit from raking out the dead leaves and grass, maybe aerating the soil some more, then over-seeding with good quality grass seed. Mix the new seed with good quality top soil and apply them together, or put a thin layer of soil on either before or after the seed, then water liberally and regularly until the new grass is well established.
If you did want to use chemicals, you would need a systemic broadleaf weedkiller sold for lawn use. Systemic means it kills the roots as well as the leaves (the opposite of a systemic herbicide is a contact herbicide). Since these weeds have substantial taproots, a contact herbicide would not kill them permanently. Broadleaf means it kills broadleafed plants (also called dicots), but not narrow-leaved plants (also called monocots) like grasses and sedges. Wildflowers are also dicots, so a broadleaf herbicide will kill any wildflowers you get it on. However, I'm not seeing any plants other than grass in the photos you showed. Most systemic broadleaf weedkillers intended for use on lawns will only kill the weeds you put them directly on, and shouldn't spread out through the soil to affect nearby plants. Look for a product whose main ingredient is 2,4-D. If it has additional active ingredients, double-check the label to make sure those ingredients are not herbicides that will linger in the soil and prevent seeds from germinating. Key words would be "pre-emergent" and "soil residual."
Your weeds might be cutleaf teasel seedlings. Although I'm not 100% certain in that ID, I'm confident that your weeds are the rosette form of a weed that will eventually "bolt," or grow a tall stem which will produce a flower. That flower will make a large number of seeds. You don't want that to happen, because those seeds will produce a huge number of new weeds all over your lawn, undoing all your hard work so far. So the deciding factor between using chemicals and continuing to remove the weeds painstakingly by hand, is whether you can get rid of them before they produce seeds. Flowering time depends on the species, but if they're teasels the process will probably start in late May or June, with seeds within a few weeks after flowers. If your weeds do start blooming, it will probably be too late to kill them with just a systemic herbicide before they set seed. At that point, you could cut down the flowering stems and apply herbicide, or mow them repeatedly every time they start to regrow the flowering stems. At some point a few plants will probably manage to grow flowers on a very short stem below the level of the mower blades, so watch out for those and remove the entire plant before they go to seed.
