Using a pastry brush, brush oil on both sides of bread.
Spread jam evenly over 1 side of each slice of bread; sprinkle with rosemary.
Cut mozzarella into 8 slices; place 2 slices of cheese on each of 4 bread slices.
Sprinkle a pinch of salt over cheese, if desired; top with remaining 4 slices of bread, jam side down.
Sprinkle tops with brown sugar.
Grill panini in a panini press until cheese has melted and bread is golden and crispy (about 3 minutes). If you do not have a panini press or indoor grill, use a ridged grill pan: Put sandwiches in pan (in batches, if necessary); place a weight, such as a brick wrapped in aluminum foil or a heavy cast-iron skillet, on top to press them down. Brown the first side (about 2-3 minutes), flip the sandwich, replace the weight, and brown the other side (about 2-3 minutes) to finish melting the cheese.
