Gosport Borough Council has rejected an application relating to a plot of land that used to form part of the grounds of Anglesey Lodge.
Following a huge fire, the Grade II-listed building is being restored and turned into a care home.
The bungalow proposal sparked eight letters of objection from people living or working nearby.
A council report said large, bulky dormers and other aspects of the design meant the scheme would result in a cramped and overly-dominant form of development between two existing homes.
The report also said that the building’s “stark” white-painted finish would be an incongruous addition to the street scene.
It added: “As a result of the combined length, height and proximity of the dwelling, and its projection along the full length of the neighbours garden, it would feel extremely intrusive and overbearing.”
The report said concerns about the scheme outweighed its “modest” contribution to housing supply.
