The difference here is that for classical liberals, the 'people in control' refers only to the government, whilst for both the far-left and the far-right, it refers to anyone who has any influence on society, including corporations, universities, professional bodies, churches, and more. This is why they don't respect other people's freedoms.