A senior source told the BBC that the prime minister had “brushed aside” an official and had failed to acknowledge a dance reception.
They added that they “expected better”.
But Mr Cameron told MPs on Monday that his visit had succeeded in raising awareness of human rights issues.
In May 2009, Sri Lanka’s army defeated the separatist Tamil Tigers after almost 30 years of brutal and bloody civil war.
The international community has since focused on the final phase of the conflict, as civilians were hemmed into a thin strip of land on the north-eastern coast. Both sides have been accused of atrocities.
One UN report estimates that as many as 40,000 civilians were killed in that final phase, mostly by government shelling.
‘Declined to sign’
Mr Cameron had come under pressure not to take part in the biennial Commonwealth Heads of Government Meeting in the capital Colombo in protest.
However, he attended and, during his stay, visited the northern region of Jaffna to see the situation facing the country’s Tamil minority.
A Sri Lankan government source said Mr Cameron had “breached protocol several times”.
It was alleged that he had “brushed aside” the main Tamil local official in visiting a displacement camp, when he should have been escorted by him.
The source added that Mr Cameron had “declined to sign the national visitors’ book” or “acknowledge a dance reception on arrival as he should have done”.
They said: “We thought Britain would do better. They’re the ones who taught us protocol.”
The Sri Lankan newspaper Mawbima reports that government minister Champika Ranawaka said Mr Cameron had “no right” to address the Tamil issue as he was not taking part in a bilateral meeting between the two countries but a Commonwealth summit.
On Monday in the House of Commons, the prime minister defended himself, saying it was his duty to “deal with, not ignore” controversial issues.
He called for Sri Lanka to begin an inquiry into alleged war crimes by March or face international intervention.
“I had a choice at this summit to stay away and allow President Rajapaksa to set the agenda he wanted or to go and shape the agenda by advancing our interests with our Commonwealth partners,” he said.
“I chose to go, to stand up for our values and do all I could to advance them. That was, I believe, the right decision for the Commonwealth, Sri Lanka and for Britain.” (BBC)