Some buildings grow, others shrink. And it's quite interesting to follow their transformations as they track cultural (social, political) and physical transformations in the city.
When Juan de Garay, the founder of Buenos Aires, distributed land, he set aside the entire block east of the square for the fort and a portion of the one to the west of the square for the Cabildo, the seat of the government. But as soon as the local government broke with the crown in 1810, the first thing that they did was to relocate to the fort. And that would be the location of successive ruling bodies and presidents all the way to the present.
The Primera Junta (as the first government is called in history books) moved to the north side of the fort. Eventually the southern half of the fort was demolished and in 1873 replaced by a monumental Central Post Office. The new building dwarfed the two-story remnants of the fort, now the presidential palace, the Casa Rosada (named after the color of its paint, a mix of lime and ox blood) and it was replaced by a twin of the recently completed building, different only by a long upper balcony (made famous after 1945 by Juan Domingo and Eva Perón.) A few years later the post office was annexed to the new Casa Rosada linking the two buildings with a three-story archway pavilion that filled the void and moved the compositional emphasis to the center.
That's the story... or almost, because in 1938 the southern wing of the post office part was demolished (I believe that the plan was raze the whole building, but demolition was halted in the nick of time.) And yes, that's the Casa Rosada that you see today.