Plans for "The Big House"; as it came to be known, were drawn up for Mr. and Mrs. C.G. Segerstrom by the Stockton engineering and architectural firm of Davis-Pearce in 1926. The general contract was nearly one-hundred pages long and called for the strictest standards of construction. Phrases unheard of in modern contracts such as "put the work together in a perfect and workmanlike manner", "all workmanship shall be of the highest grade" and "all workmen shall be skilled in their respective duties", were used. Plans called for a four story structure to be built on the very highest point of the rocky hilltop known as Knowles Hill. The full basement would be carved out of natural slate, schist and quartz and be lined with reinforced concrete fourteen inches thick. This open space was then bridged with steel and wooden girders to create a rigid platform on which the remaining three stories were constructed of the finest available clear pine, fir, redwood, oak and mahogany. Every detail was carefully spelled out from the compressive strength of the reinforced concrete, to the consistency of mortar sand in the "Joseph's Coat" brick veneer, to the acceptable brands of paint. In a day when many fine homes were constructed for less than a thousand dollars, the hardware budget alone for the Segerstrom home exceeded that amount.

By late 1927 the The Big House was ready for the Segerstrom family which included Charles and Carrie Inch, and their five children, Charles, Jr., Martha, Richard, Donald and William. They moved from the historic 1877 Burden-Inch home which still stands at 84 Washington Street where they had lived since 1905 and they settled into a home that one grand-daughter described as "a house that enclosed you with warmth and good feeling. There was always noise and chaos, but the capacious structure seemed to embrace it. There was always room for one more person, one more dog and one more thing. You couldn't help but love the place fiercely." The size of the home and its welcoming interior made it perfect for all sorts of gatherings. A guestbook, which dates from June 15, 1935 includes signatures and letters of former United States President Herbert Hoover (1936 and 1940), Episcopal Bishop Louis Childes Sanford (1936), Wendell Willkie (1940, 1941, 1943), Frank Merriam, Lt. Governor CA (1938), San Francisco Mayor, James Rolph Jr., Host of the Amateur Hour Major Ed Bowes (1939), Daughter of Wm. Jennings Bryan Ruth Bryan Owens (1937), Football and Baseball Coach Amos Alonzo Stagg (1942), actor Wallace Beery, California Senator Hiram Johnson (1940), California Governor Earl Warren, Ohio Senator Robert Taft (1940), NH Senator Styles Bridges (1940), Helen Keller (1943) and Herbert Hoover Jr.

The Segerstroms were instrumental in preserving the town of Columbia as a State Park, and when California Governor Earl Warren came to participate in the opening ceremonies on July 15, 1945, he spent the night at Knowles Hill.

The house faced east and commanded a 360 degree view of Sonora and the rolling hills. As the well planned landscaping matured, it became surrounded by cedars, pines, oaks, a sequoia, coastal redwoods, crepe myrtles, lawns and hedges, daylilies, myrtle, rockrose, azaleas, creeping fig, ivy, camellias and roses. On the east slope toward old Sonora, a rose garden grew up intersected with paths of slate and stone and trails of dirt extending below the old water canal that crossed the hill in the 1850s to supply water for the Sonora Mill. To the south was the 1908 home (still standing) of pharmacist Otto Mouron where attorney Melvin Belli, the King of Torts, was raised. On the north was Tuolumne County's first tennis court and beyond that was the Fairview Pavilion, built by June Knowles and Otto Mouron in 1909. Here, dances and other social events were held. Patrons could stroll along the scenic water canal. This pavilion became "the lab" when the Segerstroms purchased the property. After being severely damaged by a falling tree, the pavilion was replaced by a private home on the footprint of the original structure.

Following a brief use of Knowles Hill by daughter Martha and her husband Edward Cronwall (who was managing the Sonora Inn, a Segerstrom property), Donald I. Segerstrom, Charles' third son, and his wife Mary Etta moved into the house in 1953 and raised a second family of four boys and one girl on Knowles Hill. Once again the sights and sounds of young children with all of their friends and pets filled the house. By 1996 Mary Etta had lived in the house for forty-three years. Although the children had grown and moved into homes of their own, The Big House had been a center of joyful gatherings and wonderful memories for five generations of Segerstroms. The house was sold and became a Bed and Breakfast in 2001.