This is from the link bequimão posted: "Note: Currently, kwallet-pam has at least two limitations: first, it's not compatible with GnuPG keys, so KDE Wallet must use the standard blowfish encryption. Also, the wallet name must be "kdewallet" (that's the default name). If, for some reason, you create a new wallet, you need to use this name (so you will probably need to rename the old wallet too)."
So it is related to encryption used? Because I switched to blowfish and it stopped asking me for a password.