Sag Harbor was a historically black beach enclave on the East End and for generations offered a refuge from racism and a retreat for middle-class African-American families.

It flourished as a result of Jim Crow-era segregation and was one of the only a few black beach communities along the East Coast.

"I couldn't wait to get to some place where I could just fit in, and [Sag Harbor] is where that could happen," said Terrance Webber, who inherited his home in the community. "It was like a big extended family."

Three districts in the enclave of Sag Harbor still stand today – Ninevah Beach, Azurest and Sag Harbor Hills. Markers along Route 114 serve as a reminder for each area's roots. For many of the African-American families who bought homes there when the subdivisions were first built, owning waterfront property elsewhere was just not an option.

The presence of African-Americans in the area goes back centuries. In 1836, most free blacks in Sag Harbor were laborers. They created a community called Eastville. It was in the 1940s when the idea of beach living was ushered in. At the time, mortgages were not easily given to African-Americans, so they would buy their own lot and build with their own money.

Many community members, like 90-year-old Jackie Vaughn, told News 12 that they feel their heritage is slipping away.

"I realized that our uniqueness is gone and not coming back,” said Vaughn. She said she no longer knows her neighbors in the community.

Recently, real estate prices have skyrocketed in the black neighborhoods of Azurest, Ninevah and Sag Harbor Hills, and the change has been met with resistance.

But some say they welcome the large properties that dwarf the small cottages.

"To me, it's fantastic, because all it does is improve my asset. At the end of the day, my house is much more valuable," said one homeowner.

Webber says he welcomes newcomers, but wants them to respect and preserve the area's culture.

"With these three little communities along this stretch of beach, they came about with blood, sweat and tears of our parents and grandparents. It wasn't easy.”