For the past six years, there has been a quiet revolution in the mortgage market: Big banks like JPMorgan (NYSE: JPM), Bank of America (NYSE: BAC) and Citibank (NYSE: C) have moved out and nonbank lenders such as Quicken, loanDepot and Caliber Home Loans have moved in — in a big way.

The revolution went largely unchallenged, but that may be about to change if the Trump administration removes regulations on the big banks and stops sending bad loans back to the banks for repayment. Deregulation would open the door for big banks to move back in.

Paul Miller, a banking analyst at FBR Capital Markets, said he believes big banks will return to the market, "but they will need solid protections on reps and warrants" — the financial due diligence that's done on both sides of the transaction before a deal can close. He added that banks will also need rules concerning the ability to repay bad loans to be relaxed.

The growth in nonbank lending is truly staggering, from just 10 percent of the mortgage origination market in 2010 to half of it today, according to Inside Mortgage Finance. Its share is dominant in government-insured lending by the Federal Housing Administration (FHA), which was particularly aggressive in holding big banks accountable for any mistakes in loan underwriting.

FHA loans are riskier by definition because they require just a 3.5 percent down payment. The big banks moved out of FHA loans almost entirely. Banks that are still in the market are much less competitive in pricing and have therefore dropped in market share.

With the exception of Wells Fargo (NYSE: WFC), which is the largest mortgage lender by a wide margin, big banks have become far less competitive in the overall mortgage market because they have had to shell out billions of dollars in fines and legal settlements resulting from the financial crisis.

"It's become very unattractive from a regulatory perspective," Josh Rosner, managing director at Graham Fisher, said in an interview on CNBC's "Sqawk Box."

Big banks also continue to bear the brunt of mistakes in underwriting and, as a result, protect themselves by charging slightly more for home loans. Wells Fargo declined requests for a comment on potential deregulation.

The Trump administration has not put forward any specifics yet, but the banking sector is expecting changes in some of the more stringent rules put on mortgage lending following the financial crisis. If deregulation happens, nonbank lenders who were able to grow market share so easily will face tougher competition.

"I fully expect that the banks will also step up their exposure to lending a lot more, and I expect it to be more from the regional banks than from the major banks in terms of the sequencing of how lending will grow," said Sanjiv Das, CEO of Texas-based Caliber Home Loans.

Das headed Citigroup's CitiMortgage unit from 2008 to 2013 during the worst of the housing and mortgage crisis. Caliber has been one of the first lenders to begin offering loans to borrowers with less-than-perfect credit. Das said even with deregulation, he believes nonbank lenders will still have the edge.

"The independent lender will continue down the path of continuing to gain market share because they are substantially more nimble and more focused," said Das.

Nonbanks are subject to all Dodd-Frank consumer protections, and they mostly sell their loans to Fannie Mae and Freddie Mac, so they are subject to strict underwriting rules. The same with FHA, which nonbanks have moved into the most. Deregulation could create more competition, but, at the same time, it could grow the mortgage business as a whole.

"I think it will benefit us because what I think it's going to do is provide a higher level of volume, so I think industry growth overall is going to be good for the industry and for the consumer," said Anthony Hsieh, CEO of loanDepot, a nonbank lender. "We are prepared for any direction that the new administration would want to take, but certainly I think providing more available credit to the country really is a good direction."

The government still provides 90-plus percent of mortgage financing, with Fannie, Freddie, FHA and Veterans Administration loans. That is expected to change in the new administration, and new regulations on the capital coming into the market will change along with that. Smaller regional banks will surely want to take advantage, even if the big banks hold back.

"I don't expect a lot of competition from the super big lenders, but I do expect that the regional banks will step up their exposure to lending a lot more because the U.S. consumer is still very strong, home buying is very strong, and everyone has to show growth for their shareholders and investors," said Caliber Home's Das.

Unfortunately for all lenders, rising mortgage rates will take overall mortgage origination volume down in 2017, deregulation or not. Refinance volume is already at half the level it was one year ago, and while home sales are expected to rise very slightly, that will not make up for the huge downshift in refinancing.