the problem is indeed the appliance version, the 256-bit encryption requires FW 5.5 version.
From the 5.5 release notes (same is in admin guide):
"It is now possible to use 256-bit encryption for the connection between Security Engines and the Management Server. This requires both the engines and the Management Server to be version 5.5 or higher. You must also use an Internal ECDSA Certificate Authority to sign certificates for system communication."
Maybe easiest solution would be to upgrade the appliance locally to 5.5 (assuming the appliance can run 5.5 and the license is v5.5 or newer) and then do initial contact. See this article: McAfee | License Center.
Thank you, friend!
I upgraded my appliance locally to 5.5 version, then enable option 256 bit encryption. Appliance succesfully authenticated into SMC.