Preheat the oven to 350°F (175°C). Grease a 9x5-inch loaf pan and line it with parchment paper.
Cook the sweet potatoes until completely tender. Remove the skins and mash until smooth. Allow them to cool before using.
In a medium bowl, whisk together flour, baking soda, baking powder, salt, cinnamon, nutmeg, and ginger.
In a large bowl, whisk together melted butter, granulated sugar, brown sugar, eggs, vanilla extract, and milk.
Add the mashed sweet potato and whisk until smooth.
Gradually add the dry ingredients to the sweet potato mixture.
Gently stir with a spatula until just combined. Do not overmix.
Pour the batter into the prepared loaf pan and smooth the top.
If desired, sprinkle chopped pecans and coarse sugar over the batter.
Bake for 50–65 minutes, or until a toothpick inserted into the center comes out clean or with a few moist crumbs.
If the top browns too quickly, loosely cover the loaf with foil during the final part of baking.
Allow the bread to cool in the pan for 15 minutes.
Transfer to a wire rack and cool completely before slicing.
For the optional glaze, whisk powdered sugar, milk, and vanilla until smooth.
Drizzle the glaze over the cooled bread before serving.