Combine the first 4 ingredients in a bowl. Spoon into a 9-inch round cake pan coated with cooking spray.
Place 2/3 cup sugar and next 4 ingredients (2/3 cup sugar through egg) in a large bowl; beat with a mixer at medium speed until well blended (for about 5 minutes). Lightly spoon flour into dry measuring cups; level with a knife.
Combine flour, baking powder, baking soda, and salt, stirring well with a whisk.
Add the flour mixture to sugar mixture alternately with buttermilk, beginning and ending with flour mixture; mix after each addition.
Spoon batter over peach mixture in pan.
Bake at 350 for 45 minutes or until a wooden pick inserted in center comes out clean. Cool for 10 minutes in pan on a wire rack.
Place a plate upside down on top of cake, and invert onto plate.
Serve warm with frozen yogurt and the caramel syrup.