I faced with that error during running DBCA in silent mode. Complete error message is:
ORA-27104: system-defined limits for shared memory was misconfigured
I have been checked all related parameter such as /etc/sysctl.conf & /etc/security/limits.conf
After investigate issue I found MOS 1922934.1 which is refer to bug.
To resolve this issue , please apply or ask for one-off patch for Bug 14202016.
OR
As a workaround, increase the value for kernel parameter shmall to 9272480