Today, online privacy is the domain of regulatory measures and privacy-enhancing technologies. Transparency in the form of external and public assessments has been proposed for improving privacy and security because it exposes otherwise hidden deficiencies. Previous work has studied privacy attitudes and behavior of consumers. However, little is known on how organizations react to measures that employ public "naming and shaming" as an incentive for improvement. We performed the first study on this aspect by conducting a qualitative survey with 152 German health insurers. We scanned their websites with PrivacyScore.org to generate a public ranking and confronted the insurers with the results. We obtained a response rate of 27%. Responses ranged from positive feedback to legal threats. Only 12% of the sites - mostly non-responders - improved during our study. Our results show that insurers struggle due to unawareness, reluctance, and incapability, and demonstrate the general difficulties of transparency-based approaches.