"I know Castle Hill House is of importance but from the outside Castle Hill House looks much the same as many other listed buildings of that age." - quite!
"Not sure if you quite realise the importance of Castle Hill House, it would not have been upgraded to Grade II* listed otherwise." - yup.
Dover's only Grade II* listed residential house, Castle Hill House was upgraded in September 2014 by the Secretary of State to Grade II* for the following principal reasons:
* Architectural interest: for its quality of composition, detailing, distinctive plan form and outstanding
interior joinery, rococo plaster ceilings and marble fireplaces, of more than special interest;
* Intactness: little altered externally, except for the addition of an early C19 curved bay.
Internally nine or ten rooms retain significant C18 or early C19 fittings;
* Historic interest: the home of numerous mayors and town clerks of Dover and the constituency home in the 1930s of John Jacob Astor, the newspaper proprietor;
* Rarity of type: it is the only large detached C18 house in Dover.
See also (apologies for the wonky link, but you get the drift):
https://en.m.wikipedia.org/wiki/Grade_II*_listed_buildings_in_Dover_(district)