Preheat the oven to 375 degrees. Bring a large saucepan of water to a boil.
Score the stem end of each peach and place the peaches in the boiling water. Boil for about 1 minute, or until the skins soften.
Transfer to a bowl of cold water to cool, then peel, halve and pit the peaches.
In a 9-inch cast-iron skillet, combine 1/3 cup of the sugar with 1 tablespoon of the butter. Cook over medium heat for 3 to 5 minutes, or until the sugar begins to melt.
Add the peaches to the skillet, cut-side up, in one layer (the fruit should fit tightly).
Remove the pan from the heat and set aside.
In a medium bowl, combine the flour, baking powder, baking soda, cinnamon and salt; set aside.
In a large bowl, with an electric mixer at medium speed, beat the remaining 1/3 cup sugar and 1 tablespoon butter with the oil until combined.
Add the egg, beating until smooth, then beat in the vanilla and almond extract. With the mixer at low speed, add the buttermilk and the reserved flour mixture, beating until just incorporated.
Spoon the batter evenly over the peaches in the skillet, place the skillet in the oven and bake, uncovered, at 375 degrees F (190 degrees C), for 20 to 25 minutes, or until a cake tester inserted into the center of the cake comes out clean.
Transfer the skillet to a wire rack to cool for 3 to 4 minutes. Loosen the edges of the cake with a knife. Invert the cake onto a serving plate. If any of the peaches stick to the skillet, remove them with a knife and replace them on the cake.
