In the oral arugments for Barenblatt v. United States (1958), a case about the investigations conducted by the House Un-American Activities Committee, Justice Potter Stewart objected to a particular line of reasoning by saying, "It would certainly be false, an oversimplification, to say that the right of privacy is a constitutional right." At the time, no one seemed to object, but then most of the justices weren't as vigorous in oral arguments as most of them are now. It struck me as a pretty bold statement of a view that probably seemed somewhat uncontroversial at the time but now sounds so jarring because of how the Supreme Court has talked about rights to privacy for decades.
That it could be said with no controversy in 1958 perhaps says something about the method of judicial interpretation of those who suddenly found it to be so clearly present in the Constitution a mere seven years later. Justice Stewart was one of the dissenters to Griswold v. Connecticut in 1965, which first declared there to be exactly such a right, but he lost that argument. By the Roe v. Wade decision in 1972, he had apparently given up entirely, since he voted with the majority in that case. I suspect the reason is that he was always more concerned about going along with precedent than the rightness of a decision on the merits, just as his compatriot Justice John Marshall Harlan II was.
Both Justices Harlan and Stewart were seen as conservatives in those days (although Stewart later came to be seen as a moderate once there were enough Republicans' appointees on the Court again that the 50% of the Republicans' nominees who were actually conservative were able to bring conservative jurisprudence back into the mainstream after decades of the influence of FDR appointees.) Justice Stevens and Justice Breyer have been the strongest defenders of precedents they don't agree with in more recent years, so this tendency to favor precedent runs across traditional liberal-conservative lines. But Justice Stevens, to this day, insists that he is a judicial conservative, specifically on the ground that he respects precedent above other considerations, because that's what he sees being a judicial conservative amounts to, ignoring other ways of being judicially conservative (e.g. sticking closer to the constitutional or statutory text rather than policy arguments, ruling narrowly according to the smallest issue that would decide the case in question rather than making wide-ranging proclamations against laws not at issue in the immediate case, favoring states' rights over federal power, ruling in a way that politically conservative views would favor, and so on).
I don't really have an argument to make here. I just thought it was interesting to hear a comment like that in a pretty different context during a time when such a comment was less politically-loaded and could go relatively unquestioned by the other justices once it was put forward by the junior-most justice of the time.