Just checked again and I can build in the slot i cancelled, it seems to keep the previous displayed item shown even though its removed.