The Migration Advisory Committee (MAC) discovered there were 23 fewer UK jobs for every 100 migrants from outside the EU.
It is at odds with findings from the National Institute of Economic and Social Research (NIESR), which says immigration has had little impact.
The committee estimates 160,000 British-born workers’ jobs have been “displaced” following non-EU immigration between 1995 and 2010.
However, migrants who had been in the UK for more than five years were not linked with the displacement of British-born workers, it said.
The committee also found wages for the most well-paid people went up, while those at the bottom went down between 1995 and 2010.
The report also added that EU migration had had “little or no impact” on the native employment rate.
Mac chairman Professor David Metcalf said the information and communications technology, hospitality and retail as sectors, could see the most impact from non-EU migration. He said: “Assessing the impacts of migration is not a simple decision and our conclusions will require careful consideration by the government.”
Considering overall GDP did not present a “true picture”, he said. “Instead, the impact of migration on the economic well-being of the resident population should be the focus.
“Impact assessments must also consider wider effects such as the effects of skills transfer from migrants and their impacts on public finances, employability of UK workers, housing and transport.
“Although difficult to measure, these will ensure we can better understand the effects of migration.”
The report comes at a time when the government is trying to reduce net migration to the UK from “hundreds of thousands to tens of thousands”.
Dr Scott Blinder, senior researcher at the Migration Observatory at the University of Oxford, said the report highlights the need of the government to clarify whose needs it is prioritising when developing immigration policy.
“Trade-offs need to be confronted head on.
“Without more debate and clarity about whose interests policy is trying to maximise, we cannot hope to reach more agreement about the costs and benefits of specific policies.”
Immigration Minister Damian Green said: “Controlled immigration can bring benefits to the UK, but uncontrolled immigration can put pressure on public services, on infrastructure and on community relations.
“This report makes clear that it can also put pressure on the local labour market.”
The study from the National Institute of Economic and Social Research (NIESR) looked at the number of migrants given National Insurance numbers between 2002-3 and 2010-11 and compared them with the number claiming unemployment benefits.
It showed a “very small negative and generally insignificant” link between the two.
A spokesperson said: “For all practical purposes, these results suggest that migration has essentially no impact on claimant count unemployment.”
However, the organisation could not conclude whether an increase in the number of migrants coming to the UK leads to a fall in the number of low-skilled jobs for British workers, which is masked by more jobs for highly skilled Britons.
