BOSTON (AP) – Massachusetts is losing workers at an alarming rate at both ends of the economic scale and must act quickly to stem the bleeding, according to a report to be released this week.

Between 2003 and 2005, Massachusetts saw its labor force – those who have jobs or who are looking for jobs – shrink by 1.7 percent while the labor force nationwide expanded by 3.1 percent, according to the study by the Massachusetts Institute for a New Commonwealth, or MassINC.

“The economic track record of the past three years in terms of the labor force is very troubling,” MassINC president Ian Bowles said.

Bowles also points to recent data that he said indicates the state may be on the path to its fourth consecutive year of a shrinking labor force – unprecedented in Massachusetts in the post-World War II era. He urged the state to adopt a “no worker left behind” philosophy.

“A growing labor force is a key indication of a healthy economy,” he said. “The fact that it’s shrinking is quite worrisome.”

The contrast is even sharper compared to neighboring New England states.

Since 2000, Massachusetts’ labor force has remained essentially static, while all other New England states saw their labor forces grow – from 4.6 percent in Connecticut to 6 percent in Vermont.

There are two main reasons for the decline, according to the authors of the report.

First is the difficulty that lower-skilled and less-educated men are having finding and keeping jobs as the state’s manufacturing base is replaced by high tech and other jobs requiring more education and skills. Massachusetts is seeing male workers withdrawing from the work force at a steeper rate than the nation as a whole, according to the report.

By contract, female workers have had fewer problems as jobs traditionally dominated by women in the retail and health care sectors have remained steady.

“Men have suffered from the economic restructuring of our state,” said Andrew Sum, one of the report’s authors. “We as a society pay a high fiscal cost of keeping these men out of the labor force.”

Massachusetts also is struggling as rising housing prices and the state’s cost of living drive more highly educated workers into the hands of other states.

Of even more concern is that the decision to flee Massachusetts is highest among younger workers – 16 to 24 year olds and 35 to 54 year olds.

Those numbers are reflected in the state’s overall loss of population. From 2000 to 2005, according to the study, Massachusetts lost a net 233,000 residents to other states, or about 3.6 percent of its 2000 population.

The report’s authors say one way of reversing the trend is to improve job training and education programs for lower-skilled workers before they lose their jobs.

“We need a much more invigorated skills and education agenda in Massachusetts,” Bowles said.

That must be coupled with a renewed effort to bring housing costs under control, according to Sum, who said Massachusetts has the third highest housing costs when compared to median household income in the nation, after California and Hawaii.

“We don’t have the Hawaiian beaches, but we have the Hawaiian housing prices,” Sum said.

Romney spokesman Eric Fehrnstrom defended the administration’s efforts to bolster the economy, saying the state is on the rebound.

But Fehrnstrom acknowledged that as the economy recovers “one of the biggest challenges facing us in the future is a potential labor shortage.”

“The way to address it is with policies that keep taxes low, encourage the creation of more housing and emphasize math and science skills in the education of our young people,” he said.

The job of trying to reverse the trend falls to incoming Democratic Gov.-elect Deval Patrick.

Patrick spokesman Richard Chacon said he recognized the problem of people leaving the state. One of Patrick’s top priorities is beefing up the production of new and affordable housing, he said.

“He fully recognizes the need for more affordable housing especially for younger people who may be just getting into the job market, maybe who have just graduated from college or are starting a family,” Chacon said.

Patrick also is looking to improve job training programs, including creating partnerships between community colleges and local companies.