Bring the milk to a boil, stirring often. Once it comes to a boil, remove from the heat and set aside.
2 cups whole milk
While the milk heats, in a large saucepan, whisk together the yolks, brown sugar, cornstarch, cinnamon, nutmeg, and salt. Whisk until the mixture is thick and completely blended.
6 large egg yolks, ½ cup brown sugar, ⅓ cup cornstarch, ½ teaspoon ground cinnamon, ⅕ teaspoon freshly grated nutmeg, ⅛ teaspoon kosher salt
While whisking quickly and nonstop, drizzle in ¼ cup of the hot milk to temper the yolks so they won’t curdle. While still quickly whisking, slowly add the remaining milk.
Place the pan over medium heat. Whisk constantly while bringing the mixture to a boil. Boil and continue whisking, being sure to get the edges of the pan, for 1-2 minutes before removing from the heat.
Continue whisking and add the vanilla. Leave alone for 5 minutes, then whisk in the butter pieces. Stir until all butter is incorporated and the custard is smooth and silky.
1 teaspoon pure vanilla extract, 3 tablespoons unsalted butter
Cover the custard with plastic wrap that has been sprayed with nonstick spray (place the sprayed side on top of the custard) and refrigerate until cold, about 1 hour. Once the custard and pie crust are completely cooled, it is time to assemble the pie. (See NOTE for quicker cooling method)
To assemble the pie, peel the bananas and cut them into ¼-inch thick, diagonal slices. Whisk the cold custard to loosen it and spread ¼ of it over the bottom of the pie crust. The layer will be thin. Top the custard with half of the banana slices. Spread half of the remaining custard over the banana layer and top with the remaining banana slices. Spread the remaining half of the custard evenly over the bananas.