Walls are not all about immigrants. The notoriously brutal Berlin Wall was not about keeping people out but keeping people in. Other brutal regimes also maintained formidable obstacles to movement. In the Confederate States, a system of passports limited movement as much as it dd among the citizens of the Soviet Union and other authoritarian regimes.
To learn more about how boundaries work, read Boundary Control by Edward L. Gibson. Order the book below.