In a large bowl, combine the mashed potatoes with the other casserole ingredients (except the toppings) and mix until incorporated. Pour the sweet potato mixture into the prepared baking dish and sprinkle with the toasted pecans. Bake for 40-45 minutes or until the sweet potato casserole is bubbly around the edges and set in the middle. If the pecans begin to burn, cover very lightly with foil. Transfer dish to a cooling rack to cool 5 minutes.
¼ cup light brown sugar, ¼ cup evaporated milk or cream, 2 tbsp butter, 2 tbsp orange juice, 1 tsp vanilla extract, ½ tsp salt, ½ tsp orange zest, ½ tsp ground cinnamon, ¼ tsp freshly ground nutmeg, pinch ground mace, pinch ground cloves, ¼-½ cup pecans