The source said the talks were preliminary as Israel had not formally faulted its prison authorities in the death of Ben Zygier, which was made public this week by an Australian television expose that described him as a Mossad officer.
A Mossad link has been neither denied nor confirmed by Australia or Israel, where military censorship and court gag orders kept many details of the case from the media.
The silence has fanned media speculation that Israel believes the 34-year-old Melbourne Jew had betrayed its intelligence agency's high-stakes work abroad.
The office of Israeli Prime Minister Benjamin Netanyahu, which oversees the Mossad, did not respond for a request for comment on the matter.
Israel's Haaretz daily said the state agreed to pay "several million shekels" in damages to Zygier's family around six weeks ago, when an internal inquest declared his death a suicide.
The inquest result was disclosed by the Justice Ministry on Wednesday, in Israel's only official statement on the case. The statement, which did not identify Zygier by name, said a judge had also ordered an "evaluation regarding issues of negligence".
A source briefed on the affair denied there had been any agreement to compensate Zygier's family for the failure of staff to prevent his suicide at Ayalon prison, where he had been held for months, under alias and in isolation from other inmates.
"There's no decision on negligence yet, so there's no compensation in any form in that regard," the source told Reuters on condition of anonymity. "What there have been are initial inquiries by the deceased's representatives about compensation."'Grave crime'
A Zygier family lawyer, Moshe Mazur, declined to comment, citing the sensitivity of the case.
So did Israel's Prisons Service. But one of its officials voiced skepticism about the idea of compensation being agreed with Zygier's family, saying such payouts in negligence cases could take "years" to negotiate.
Avigdor Feldman, an Israeli lawyer with whom Zygier briefly consulted while in prison, said he knew of no compensation deal.
Were the state to pay damages for negligence, he said, it would not reflect any official position on Zygier's guilt or innocence: "Even convicted criminals are eligible for compensation if their jailers fail to provide for their well-being as required."
Feldman said Zygier died after being indicted for "grave crimes" but before being tried. Zygier had denied the charges against him but was considering a plea bargain, Feldman said.
Australian Foreign Minister Bob Carr said on Thursday that Canberra was told Zygier had been held over "serious offenses under Israeli national security legislation".
Feldman told Israeli radio on Thursday that a "Mossad liaison" contact had arranged his meeting with Zygier.
The Age, a Melbourne newspaper, said in a report citing Australian security officials that Zygier may have been in contact with the intelligence services of his native country and "been about to blow the whistle" about Mossad operations - including their possible fraudulent use of Australian passports.
A veteran intelligence officer who declined to be identified by name or nationality said there was a possibility that, had Zygier indeed served Mossad, the agency would have paid death benefits to his family -- regardless of the charges against him.
"If he was never tried, then he was never found guilty, and he may be considered to have died while in active service," the intelligence veteran said. "That would make his next-of-kin eligible to the various relevant payouts."
The Hebrew word for compensation, "pitzuim", can also be used for benefits paid without claims of misconduct.
JERUSALEM (Reuters) -- Relatives of an Australian immigrant to Israel who killed himself in 2010 while secretly jailed on charges of violating national security are seeking compensation from the state, a source briefed on the affair said Friday.