Spare a thought for the small-business owner. Restauranteurs, hotel operators, movie theatre owners and so many other entrepreneurs have seen their businesses decimated by the pandemic. The housing impact on workers in these sectors has been felt most acutely in the rental and multifamily space, but many small business owners also own their homes. While their neighbors telecommute to salaried office jobs and refinance their homes at record-low rates, these small business owners have had to make heart-wrenching cuts to their businesses and even their own incomes to stay afloat. When they go to their broker or loan officer, hoping a refi might offer them some personal relief, they’re being punished for a situation wholly out of their control.

“It’s been really hard to see some of these hardest-hit people, these small business owners who worked so hard to build their businesses and have had to close for no fault of their own,” said Jonathon Auer (pictured), SVP of mortgage lending for Guaranteed Rate in Plymouth, Michigan. “They’re the ones that probably need help the most, but if they didn’t make any money, their profits and losses are bad, or they had to lay themselves off to pay employees, when they try to get a mortgage it doesn’t work.”

Read more: Cherry Creek’s president expects consolidation in the mortgage industry

Auer noted the particular pain of those small-business owners who cut or entirely eliminated their personal salaries to keep their employees on the payroll. Those owners, who did the right thing in the face of a seismic shock to their business, could really use the savings that would come with a refi at today’s rates. According to Auer, however, current lending guidelines are keeping them from accessing any lower rates.

Even those owners who continued to pay themselves at nearly their pre-pandemic salary are feeling the pain when they try to refinance. Kirk Tatom, president of Tatom Lending in Dallas, Tx, explained that self-employed borrowers are assessed based on both personal income and business earnings because of “COVID overlays” in agency guidelines. Because their earnings fell off a cliff in 2020, those borrowers are still losing out.

Tatom and Auer both shared how heartbreaking some of these conversations with small business owners can be. As loan officers, it’s in their nature to try and help everyone but, when the 2020 numbers look brutal, they said there’s very little they can do.

“Working with self employed borrowers this year we’ve had to put a lot of people on ice and say, ‘let’s wait until later in 2021,’” Tatom said, citing the example of a local restauranteur who was shut down for six months in 2020, before reopening and enjoying solid sales numbers ever since. Pushing that borrower back a few months, provided their business isn’t closed again, could mean they have a profit-loss balance that makes them eligible again. The risk in that delay, however, is that if rates do tick up more meaningfully by the back half of the year, these borrowers might miss out on a chance to secure a historically low rate.

Read more: Hot sellers’ market drives improvement in housing sentiment

It’s important to note, however, that not all small business owners are actually struggling in the same way those in the hardest-hit industries are. Yury Shraybman, broker with Innovative Mortgage Brokers in Philadelphia, told the story of a client from the relatively unaffected insurance industry who was being turned down because his application said “self-employed.”

“He went to several different lenders and as soon as they heard that he was self employed, they didn’t want to touch this loan,” Shraybman said. “When he came to me, I realized that this was just a regular loan, there is absolutely no reason why this loan should not work.”

Shraybman, Tatom, and Auer all agreed that the incredibly sector-specific impacts of the pandemic mean that not all self-employed borrowers can be painted with the same brush. Some business owners are thriving and it’s up to a loan officer to look past the top line of the application and find out how they can help.

As for those self-employed borrowers coming from hard-hit industries, Shraybman, Tatom, and Auer all agreed that some kind of policy solution is needed. They noted that agency “COVID overlays” lack forgiveness for the difficult situations so many small business owners are being put in. Auer noted that a program like the Home Affordable Refinance Program (HARP), offered in 2008, could make a significant difference for these borrowers.

“I’m hoping that officials and decision makers will take a look at the small business owners that are absolutely getting crushed,” Auer said. “I hope they can give them the ability to take advantage of these lower rates while they’re still here. But the longer they wait, the less impact they will have because the rates keep ticking up.”