Castle of the bishops of Lausanne, then baillivale residence and prefecture