America may be the land of the free but it also has the highest incarceration rate in the world. Its prisons contain more people than big cities such as Philadelphia or Dallas and express systemic racism in concrete and steel. But solutions do not necessarily lie with the White House.

“People think that criminal policy in the United States is set by the federal government,” says film-maker Yoni Brook. “But in actuality, when you think about mass incarceration, there’s over 2,000 district attorneys and state attorneys and local prosecutors with a huge amount of unchecked discretion to decide what is a crime and what isn’t a crime. We really need to think about who we want in those offices.”

Brook and his co-director, Ted Passon, have made a fly-on-the-wall documentary series about one such district attorney in their home city of Philadelphia. Larry Krasner is part of a new wave of “progressive prosecutors” who are winning elections in Boston, Los Angeles and other major cities with a mandate to reform criminal justice from within.

Krasner’s steady but steely assault on the status quo, including elements in the police fiercely resistant to change, is traced by Philly DA, which ended on PBS on Tuesday night and is streaming on its website. He is seen addressing hearings, community meetings and town halls, taking heat from conservatives who say he is pushing too far and radicals who say he is not pushing far enough.

One activist seeking to abolish the cash bail system – which disproportionately keeps poor defendants confined before trial – tells Krasner at a meeting shown in the final episode: “I think we’ve been patient as a working group understanding all of the different challenges of us as crazy-ass leftists or whatever, but we do want to say that we’re not taking off the table for ourselves the idea of publicly saying that we don’t think you guys have done enough.”

Krasner appears unfazed by such criticism from left or right. The 60-year-old tells the Guardian: “I’ve had a thick skin for a long time, but I can say it gets thicker.”

He is tested in episode three by the case of Robert Wilson, a Black police officer murdered while attempting to thwart an armed robbery. Two Black brothers plead guilty to the killing. John McNesby of a local police union responds: “If there’s ever a classic case for a death penalty case, this is it.” After all, one of Krasner’s predecessors, Lynne Abraham, became known as “the country’s deadliest DA” because of her aggressive pursuit of capital punishment.

But there is a new, more liberal sheriff in town. Krasner has always opposed the death penalty, which was used 17 times in America last year alone. He served on a jury in one such case when he was 23, before going to law school, where he studied the punishment carefully. Then he became a civil rights lawyer who defended people in death penalty cases.

“It’s quite an experience to be standing in front of a jury when the prosecutor’s purpose is to kill your client,” he explains by phone. “That’s quite a moment. So it’s an issue I’ve thought a lot about and it’s an issue I feel strongly about and that particular episode covers a scenario in which basically every one of my predecessors would have sought the death penalty for reasons that were at least as political as they were philosophical.”

He adds: “I did the thing that was not politically normal at all but which I thought was justice, and that was we did not pursue the death penalty in a case involving the killing of a police officer during a robbery. It was a specific decision about a specific case.

“It was a decision that reflected, among other things, that the two different mothers [of Wilson’s two sons] were not supportive of the death penalty. But their position was not the traditional one and therefore we were really up against the police union trying to use the entire scenario in a ghoulish way for their own politics.”

The film poses the classic question of whether all idealists are destined, as the much-quoted phrase goes, to campaign in poetry and govern in prose. Krasner was seen as an outsider candidate in 2017 – when Philadelphia had one of the highest incarceration rates of any major city – but pulled off a landslide victory with huge support from African American voters. Does he feel that he has been able to stay true to his principles or were there moments when he bowed to pragmatism?

“I think in many ways the reason we got so much resistance in the first term is because we did what we said we would do,” he replies firmly. “Doing what you say you will do does not necessarily mean you always succeed.

“We said we were going to cut mass incarceration; we cut future years of incarceration in half. We said we were going to cut mass supervision on probation and parole; we cut that by two-thirds. I can list a few more enormous achievements, including exonerating 20 people at this point, almost all of whom are completely innocent, that stand in stark contrast to our predecessors.

“But we also said we were going to push against the cash bail system. We did push against the cash bail system but ultimately realised that it’s going to take state legislation to eliminate the use of cash bail in order for us to eliminate it completely because we don’t control that. Bail is not set by us; bail is set by judges.”

Voters evidently agree that Krasner kept his promises and his soul. Last month, despite the police union throwing its weight and money against him, he won resoundingly in a Democratic primary election seen as a test of progressive prosecutors’ sustainability.

“So even where there is frustration, the voters understood and Philadelphia understood, so long as you have tried to do the right thing, you have tried to be fair, you have not simply compromised out of some sort of political expediency, they respect that.

“I think the real story of our first term is that our opponents came at us with flamethrowers because we did what we said we would do, we accomplished so much, and therefore they felt it was necessary to turn us back because we were accomplishing what we said we would do.”

No one wielded an angrier flamethrower than John McNesby, president of the Fraternal Order of Police (FOP), Lodge 5. But Krasner is careful to distinguish this union, which he says has always been led by conservative white men (“people who came up at a time when policing was all about brutality and racism”), from the police department as a whole, which is more representative of Philadelphia’s racial diversity than it used to be.

“There are some deeply troubling white supremacist elements in the police department and they are coddled if not encouraged by the head of the police union, a guy named John McNesby, who also referred to Black Lives Matter as ‘a pack of rabid animals’. I feel like our relationship with rank-and-file police officers is improving and, with some of them, it’s quite good.”

Krasner, who insists he does not harbour ambitions for higher office, is optimistic about the future of criminal justice reform. “Number one, 10% of the United States has now elected and re-elected, in many cases, a progressive prosecutor. That’s a lot and they’ve done it usually in big cities that have an outsized level of control over mass incarceration.

“So in the same way that in Philadelphia we were able to slam on the brakes on mass incarceration and cut in half the number of years of jail and prison being generated in our county court, that sort of thing is happening in places like Chicago and Los Angeles and Brooklyn, and it will continue to happen as more and more DAs are elected.

“The second thing is that progressive prosecutors are winning elections like crazy and we’re not winning them because we’re so magnetically wonderful. We’re winning them because people want this. There’s a pretty good argument that the most effective political party in the United States is not the Democrats or the Republicans who win some and lose some; it’s progressive prosecutors.”

With investment in prevention rather than policing, Krasner argues, history will come to regard America’s addiction to locking people up as a grim anomaly with origins in Richard Nixon’s cynical politics, structural racism and the war on drugs. “Everybody’s going to look back on the period of mass incarceration as the radical experiment that failed.

“They’re going to look at the decade that preceded it and the decades that hopefully we’re setting in motion as being normal and an appropriate level of incarceration in which we see declining crime, declining numbers of people in jail, because we’ve reinvested in the things that actually prevent crime and make us safer.”

And he has no regrets about allowing “a couple of scrappy Philly guys” to film his successes and failures for the sake of transparency and demystifying the process for other outsiders who might consider running to be a district attorney.

Passon, 40, became curious about the criminal justice system when he was young after three family members, including his brother, got locked up at different times. He had heard Krasner’s name about town but never dreamed the activist lawyer could win an election.

“He shocked the city, including us, when he actually won and so immediately the story became way bigger. Well, now we get to know if you’re actually going to be able to do any of this stuff and why or why not.”

Brook, 38, says: “You could look at the series and, to some degree, you’d think that we had planned out how this was going to go in terms of the characters and the access to the office. But the truth is Larry Krasner walks in, turns on the lights and we’re there with him and there was such chaos and tumult in the office.

“All these prosecutors who had worked there for 30 years had just been vilified in the campaign by Larry Krasner. They’d been his courtroom opponent for 30 years and he walks in and there’s two guys with cameras with him. Up was down. Down was up. And so in that moment, as film-makers and journalists, we said to ourselves we’re just going to keep showing up until somebody tells us not to.

“I really believe that Larry Krasner is a storyteller at heart and I think he knows that one of the biggest tools he has in his office is the bully pulpit to be able to change the narrative of what people perceive to be crime and punishment in Philadelphia, and maybe the nation.”

But he adds with candid humour: “Sometimes when you ask him and his team about the series, they sometimes project an aura of knowing what they were doing but that’s total bullshit. He didn’t know how to be a DA. We didn’t know how to even make a film about this. So it was all people learning together, to be quite honest.”

• Philly DA is all available to stream on PBS.org and via the PBS app with a UK date to be announced