probably it didn't work as desired because of the limit values. From 5.7 admin guide page 596:
4. Select the SYN Rate Limits Mode:
•Off: SYN Rate Limits are disabled. This is the default setting.
•Automatic: This is the recommended mode. The engine automatically calculates the number of Allowed SYNs per Second (the number of allowed SYN packets per second)
and the Burst Size (the number of allowed SYNs before the engine starts limiting the SYN rate) for the interface based on the engine’s capacity and memory size.
•Custom: Enter the desired values for Allowed SYNs per Second and Burst Size. We recommend that the Burst Size be at least one tenth of the Allowed SYNs per Second
value. If the Burst Size is too small, SYN Rate Limits do not work. For example, if the value for Allowed SYNs per Second is 10000, the Burst Size must be at least 1000.
Now you have it the other way around, allowed SYNs is one tenth of burst size. So now you'd need over 100 SYNs per second to trigger the rate limiting. In testing environment you could change them to e.g. 10 allowed and 5 burst or similar so it would start blocking after 10 SYN/sec. In production the values should be given more thought.
I have tried to adjust syn rate allowed as your suggestion but firewall can't block syn. how should i do ? or Is it need to enable ips ?
Thanks in advance
sorry for the long pause, it should work without inspection too. I'd start with latest 5.5, 5.7 or 5.8 release and test different thresholds and hping values.