Add the sugar, cornstarch, milk, and salt to a double boiler (or a heatproof bowl set over a saucepan) placed over medium heat.
½ cup granulated sugar, ⅓ cup cornstarch, 2½ cups whole milk, ½ teaspoon kosher salt
Whisk frequently until the mixture begins to thicken, about 10-15 minutes.
Place the egg yolks in a separate bowl.
3 egg yolks
Add about ½ cup of the thickened milk mixture to the egg yolks and whisk to temper the eggs.
Add the tempered eggs to the milk mixture and continue to whisk until thickened, about 3-5 minutes. It may look slightly thinner than pudding at this point, but it will thicken further in the refrigerator. (See notes for further explanation).
Remove from the heat then whisk in the vanilla extract, butter, and food coloring (if using) until the butter has melted.
2 teaspoons pure vanilla extract, 3 tablespoons salted butter, 4 drops yellow food coloring
Arrange an even layer of vanilla wafers on the bottom of an 8x8-inch baking dish.
11 ounces Nilla Wafers
Top the vanilla wafers with banana slices. Pour half of the pudding over the cookies and banana slices.
2 large bananas
Smooth the pudding in an even layer, then top with more cookies and banana slices.
Pour the remaining pudding on top in an even layer.
Let cool for 15 minutes, then cover with plastic and refrigerate for at least 4-6 hours.
Top with whipped topping before serving. Crush any remaining cookies and use to garnish along with additional banana slices.
8 ounces whipped topping