The Reformed Church of Roxbury was organized August 30, 1802. It was known as "The Church of Beaverdam". Worship took place in a barn north of the village. In 1825 the church was recognized as the First Dutch Reformed Church of Roxbury. The church that is standing today was the forth attempt at a church being built in Roxbury. The first two were destroyed by high winds in 1867, and the third burned down on November 22, 1891. Efforts to raise money for a new church were discouraging until Jay Gould came into the picture. He promised liberal aid if a stone building was erected. Mr. Gould died before he was able to fulfill his promise to help build the new church, but his children stepped in and took over the plans. The church was built and dedicated to God in 1894 as a memorial to their father. The gift came with a new name The Jay Gould Memorial Reformed Church.
The three windows shown on the left were placed in memory of Jay Gould. The savior is in the center window, to the left is Mary, and to the right there is an angel.