In 2004, the book Down and Dirty Pictures: Miramax, Sundance, and the Rise of Independent Film painted a brash, memorable portrait of Harvey Weinstein. Written by Peter Biskind, a former editor of the now defunct Premiere magazine and a contributing editor to Vanity Fair, it chronicled the independent film scene of the 1990s. Weinstein, along with his brother Bob, was a champion of this era, distributing films like Steven Soderbergh’s Sex, Lies, and Videotape and Quentin Tarantino’s Pulp Fiction. Biskind met with Weinstein quite a few times as he was writing the book, eventually depicting him as a brutish, violent man who nevertheless managed to charm all the right people in Hollywood and release a series of groundbreaking, Oscar-winning films.

But in light of the recent allegations about Weinstein’s sexual misconduct—at least 30 women thus far have accused him of sexual harassment or assault—the producer’s legacy has gone down in a blaze of nitrate flames. This sordid element of Weinstein’s life isn’t mentioned in Down and Dirty Pictures, though the book does contain numerous anecdotes about Weinstein’s grotesque behavior toward employees (slamming tables, tearing phones out of walls, telling a subordinate to jump off balcony). Still, at the time he was reporting it, Biskind tells V.F. he did hear whispers about Weinstein’s alleged sexual misconduct, which he says was “an open secret” in Hollywood.

“I would hear things occasionally, but they were rumors,” Biskind said. Why didn’t they merit a mention in his book? “I wasn’t writing a biography of Harvey; I was writing about the explosion of independent [films] in the 90s . . . I didn’t feel that whatever truth there was to this rumors about Harvey’s personal life had really much relevance to what I was writing about.”

At the time, Weinstein was also one of the most powerful figures in Hollywood—and one of the most terrifying. In the preface to Down and Dirty Pictures, Biskind recalls working on an investigative piece about Miramax at Premiere—until the Weinsteins threatened to pull ads from the magazine. The story was killed. “Next thing I knew, Harvey was writing columns for Premiere and I was his editor,” Biskind writes.

The preface also details the first meeting Biskind had with Bob and Harvey while researching the book. It had an “odor of menace,” he writes, with Harvey forcefully trying to persuade Biskind that his book wouldn’t make any money; at best, he would get pats on the back at cocktail parties. He then asked Biskind if there were any other books he was interested in writing, perhaps something that Miramax might be interested in publishing. Biskind, who still vividly remembers Weinstein’s bullish tone (and the fact that there was a baseball bat resting in the corner of the room), turned the offer down.

Despite Weinstein’s bullying tactics, industry insiders did talk openly about his alleged sexual harassment of women, even before that alleged conduct was public knowledge. In an interview with The New York Times, Gwyneth Paltrow said that Weinstein had sexually harassed her. She also said that Brad Pitt, her boyfriend at the time, confronted Weinstein about his alleged behavior at a premiere. Biskind says he first learned of that incident while reporting his 2001 cover on Brad Pitt, but wasn’t able to mention it in his story because Pitt had gone off-the-record.

“He made me turn the tape recorder off . . . but he did say he liked Harvey, despite the advances he had made on Gwyneth Paltrow,” Biskind says now. The actor added that he had never done a Miramax movie, and “felt it was his duty” to confront Weinstein. (Representatives for Pitt had no comment, but a source close to him says the actor has no recollection of ever making a comment like that about Weinstein, particularly given the situation with Paltrow and the manner in which Pitt confronted Weinstein.)

After Down and Dirty Pictures was published, Weinstein was “very unhappy,” Biskind says—as was Ben Affleck, whose breakthrough film Good Will Hunting was produced by Miramax. Both Matt Damon and Affleck are quoted in the book, particularly in an excerpt from it published in Vanity Fair in 2004. Shortly after that excerpt was published, says Biskind, Affleck wrote a letter to the magazine, essentially “attacking” the writer.

“He said I had misled him . . . that I ended up writing a hatchet job on Harvey,” he says. (Representatives for Affleck have not yet responded to V.F.’s request for comment.)

This echoes a pattern: when David Carr wrote a story about Weinstein for New York magazine in 2001, stars including Paltrow, Nicole Kidman, and Paul Newman called him out of the blue to offer up warm anecdotes about Weinstein’s softer side. Meanwhile, The Wrap editor Sharon Waxman has said that Matt Damon and Russell Crowe both called her in 2004 when she was reporting for The New York Times on Weinstein’s behavior—particularly his relationship with Fabrizio Lombardo, who was rumored “to take care of Weinstein’s women needs” in Europe. Waxman wrote that Damon and Crowe both heaped praise on Lombardo, and speculated that their influence kept any allegations of sexual misconduct out of her finished article. (Damon, for his part, disputes this account: “I just remember it being a negative piece, a hit job on Fabrizio, was what Harvey was saying,” he told Deadline Tuesday. “Basically, that he had no professional experience. Harvey said, you worked with him. Can you tell her that he was a professional and you had a good experience, and that was it. I didn’t mind doing it, because that was all true.”)

Weinstein “became untouchable,” Biskind says. “People were afraid to say anything about him other than ‘Thank you, thank you, Harvey’ at the Academy Awards.”

Why, then, has that aura of invincibility vanished? Biskind thinks that piercing Weinstein’s protective armor is easier now in the wake of revelations about other powerful men and their alleged sexual misconduct, including Roger Ailes, Bill Cosby, and Bill O’Reilly. Perhaps as importantly, Weinstein is losing his golden touch—recent projects like Tulip Fever have been high-profile, money-losing flops—and the independent movie scene is becoming more corporatized.

As of now, the Weinstein Company is on the edge of destruction, its legacy tarnished by Harvey’s alleged actions. Weinstein himself is also likely finished in the industry, although Biskind wouldn’t be surprised if the producer eventually tries to come back from the scandal.

“There’s so many second acts in Hollywood . . . certainly he’s got enough money to retire and he’s old enough to retire, but he’s an extremely competitive person,” he says. And yet, Biskind can’t think of any figure in Hollywood history who could serve as a blueprint for the disgraced producer. He does think, though, that Weinstein’s downfall will help the industry weed out other sexual predators.

“People feel empowered to speak,” Biskind said. “And they should, because it really is out of control.”