An ambitious plan by Jared Kushner's family to recast its indebted Fifth Avenue office building as a luxury architectural trophy is collapsing, setting off a chain of events that may imperil the Kushners' ownership of a property central to their real estate empire.

Their partner, Vornado Realty Trust, is telling brokers to plan for a much more mundane renovation that would leave the property as an office building, according to three people familiar with the matter. Vornado Chairman and Chief Executive Officer Steve Roth was never enthusiastic about the Kushner plan although until now he hadn't stood in its way.

Putting an end to the Kushner effort — to salvage their overpriced investment by turning it into a Midtown jewel with expensive condos, a hotel and five-floor mall — could have profound ramifications for the family. Vornado, which owns 49.5 percent of 666 Fifth Ave., is unlikely to invest further in the property without first being reassured of its future, said three people familiar with Roth's thinking. That means returning to the negotiating table with lenders — a battle that could result in Kushner Cos.' losing control of the building, said the people, who asked not to be named discussing private deals.