we usually call it quits, or make the decision when winds hit 15+ knots. after that point your gonna be drifting like a bat outta hell, unless you can get wind against tide setup, but then you'll have a little extra roughness. specially take consideration to the inlets on high wind days against tide. barnegat can get real snotty with such situations, as im sure others do to.
for wave height, 4 and over will end the trip. but then it also depends on the wave period. i've gone out in what had to be 8+ swells that where spaced far apart and made for a nice day actually, lotta up and down and couldnt see the boat next to ya when they dropped into the swell, but a nice day.
then i've had days where 2-3's ended the day because they where tight together and you got thrown all over the place.
all in all, any situation is your call, as only you know your limits and ability's. always start your trip by watching the weather a couple days in advance. a few days of a high pressure system will be nice weather. low pressure comin will be more prone to foul weather.
good resource to find out whats happening around you, just for an idea of whats comin is here
http://www.ndbc.noaa.gov/