The Pilgrim House opened for business around 1810 and counted Henry David Thoreau among its guests. The original structure, set so far back from the street that there was room for a gazebo or bandstand in its front yard, might have dated to the late 1700s.

The house was purchased in 1810 by Benjamin Gifford to be run as a hostelry. The wife of the seller, Phineas Nickerson, was so opposed to the idea of letting go of this that she wished openly that it would be destroyed by an earthquake. “Singularly enough, one did happen, which shook and rattled the dishes and did a slight amount of damage to the building,” Herman A. Jennings wrote in Provincetown, Or Odds and Ends from the Tip End (1890).

From Benjamin Gifford, the property passed in 1847 to his son, James Gifford, who also owned the Gifford House on Carver St. The younger Gifford, Jennings reports, “made large and extensive alterations and improvements.” The Giffords controlled the property until 1873 when it was leased from Isaiah Gifford by Samuel Sands Uncle Sam Smith. By 1890, as the result of years of alterations and expansions, “very little remains of the original building, excepting the framework,” Jennings wrote. At this time, the building was still denominated 313 Commercial Street. W. H. Potter was the proprietor by 1901.

Like its modern counterpart, the Pilgrim House doubled as a nightspot. In the 1950s, a dance orchestra played on Saturday nights in what was then called the Sea Dragon Club. In the 1960s, what was now known as the Madeira Club played host to all kinds of talent, including — in the summer of 1967 — a young comedian with a “plastic face,” who was gaining fame as a performer on The Garry Moore Show on CBS-TV. She was Lily Tomlin and her very successful run at the Pilgrim House occurred two years before she joined the cast of Rowan & Martin’s Laugh-In on NBC, which brought true national stardom. Her booking brought her to Provincetown for the first time. “I love it here,” she told The Advocate (20 July 1967), “but it’s frustrating not being able to grow a beard.”

Though much transformed, the building managed to last until October 1990, when it was destroyed in a four-alarm blaze that required more than 100 firefighters from seven Cape towns to extinguish and injured more than a dozen people. The Pilgrim House was a complete loss. Donald R. Edwards rebuilt the Pilgrim House. While it occupied roughly the same footprint, in its new incarnation the property was more about entertainment than accommodation, though it did have 20 guest rooms. Edwards thought that Provincetown was missing a women’s club that could compete with the successful Pied Piper. Christel Krahulik wrote that shortly after Vixen opened in the mid-90s, it surpassed the Pied piper as the most popular women’s bar in Provincetown.