Place all the scooped out filling into a bowl. Add the butter, milk (start with less), salt, pepper (start with less of the salt and pepper), nutmeg and cheddar cheese. Blend with a fork till rustically smooth (it doesn't have to be perfect, without lumps! If you need more milk to break up the potatoes, only add 1 tbsp at a time, you don't want them to become too loose. You want just a little firmer than classic mashed potatoes. The size of your potatoes will determine how much milk and even cheese to add.
Taste and re-season as desired.
Divide the mashed potato mixture between the potatoes. Don't be stingy, but not so much that they teeter over! Fluff the tops up lightly with a fork, creating a pattern of sorts, this is where the browning will happen. If you have too much filling, no worries, just make mini cakes and fry them up for breakfast the next morning.
Bake until the cheeses have melted and are starting to bubble, the exterior is nice and golden brown, and the filling is warmed through, about 20-25 minutes.
Feel free to use ground turkey or chicken instead.
The filling can be made in advance and stored in the fridge till needed. If you make the full batch, feel free to freeze half and make only four large potatoes at a time.
Leftover baked stuffed potatoes can be stored in a sealed container in the fridge. Re-heat in a toaster oven or oven at 325F till warmed through. If exterior starts to brown too much, just cover with foil.