How the land is used
Fannie Mae does not buy mortgages on agricultural properties such as farms or ranches, and USDA's handbook for its Section 502 home loans rules out land used principally to produce income. A home where the land is mostly for living on, with a garden, some animals or a few rows of vines, is a different property from a farm whose crops are the business. Where your property falls is the first thing a lender asks, and each lender draws its own line.
The appraisal and the size of the site
Fannie Mae's Selling Guide says a site should be of a size, shape and topography generally conforming and acceptable in its market area, and that the appraisal must include the actual size of the site, not a hypothetical portion of it. On a large rural parcel that can mean few comparable sales nearby, which the appraiser has to work through, and which can shape what a lender will lend.
Water, septic, access and fire
A rural home may draw its water from a well or a small water district and treats its wastewater with a septic system, and a lender and an appraiser look at both. Legal access to the parcel, and its fire hazard severity zone, bear on the value and on the insurance the lender requires, which is part of the payment you are qualified on.
A Williamson Act contract
California's Williamson Act lets a county or city and a landowner sign a contract restricting land to agricultural or related open space use in exchange for a lower property tax assessment, and the Department of Conservation says the contract binds every later buyer. San Luis Obispo County's rules say a new residence on contracted land must be incidental to and in support of the agricultural enterprise on the property, so check for a contract before you plan to build.
Personal home or farm business
A loan for a home you will live in is consumer credit, with the federal disclosures and the ability-to-repay rule that go with it. Credit primarily for an agricultural purpose is business credit outside Regulation Z, under the lender's own rules for that kind of loan, and a lender on a working farm generally looks at the operation itself: its records of what it produces and sells, its income and expenses, and the land and water it depends on. Which one your file is depends on what the loan is mainly for, not on the label on the listing.