The police raid was launched two days after a member of the Salhiyeh family took to the roof of the house, threatening to blow it up with gas canisters if they were forced out.
Police had withdrawn after the standoff on Monday but returned before dawn on Wednesday, saying in a statement it was enforcing a court-approved eviction order of "illegal buildings built on grounds designated for a school for children with special needs".
A tree-lined area of sandstone homes, foreign consulates and luxury hotels, Sheikh Jarrah has become an emblem of what Palestinians regard as an Israeli campaign to force them out of East Jerusalem.
The family's case has been monitored by diplomats and rights activists.
In a joint statement with the Jerusalem municipality, police said several people were arrested for questioning on suspicion of violating a court order and disturbing the peace. Witnesses said some 25 Palestinians, including five family members, were detained.
At the site, about a dozen police officers removed gas canisters, with a mechanical digger used in the demolition leaving behind a mound of concrete and twisted metal.
The municipality had expropriated the plot, in an area Israel captured and occupied in a 1967 war, along with the rest of East Jerusalem, and later annexed.
The home was located about 1 kilometre north of Jerusalem's Old City walls, where clashes erupted last year between Palestinians and Jewish settlers.
"I am devastated. You see livelihoods being destroyed in front of your eyes, and now the house is gone," said one of the activists, who asked not to be identified, and was part of a group that held an overnight vigil in the home and its garden.
The home had stood across from the British Consulate in East Jerusalem, which said on Monday that evictions in occupied territory, in all but the most exceptional circumstances, were against international humanitarian law.
It urged the Israeli government to "cease such practices which only serve to increase tensions on the ground".
