Whisk water, ketchup, vinegar, brown sugar, butter, salt, Worcestershire sauce, mustard, and chili powder together in a bowl; pour mixture over drumsticks. Cover with aluminum foil.
Bake in preheated oven until no longer pink at the bone and the juices run clear, about 1 hour, turning chicken about halfway through. An instant-read thermometer inserted near the bone should read 165 degrees F (74 degrees C).
