Ok, yeah I see what I did wrong.
Since I didn't set any perms for the "earned access" area, I had assumed that it would not matter, so I left all of those settings to their default status.
I had expected that the "default" would take over because of that and I should be able to get in. As "Earned" was set to "None", I was being locked out. Changed earned to "Score" and all works.
So, what makes sense to me anyway is that if all "Optional" (That's the keyword there and why I didn't realize those settings mattered), are left at their blank/default settings, then the "Default Access" should be used.