I've only used CA glue on woodturnings or fixing plastic Chinese toys. I never used it for joining boards. I have a feeling that you might be correct, the glue is brittle.
I've seen some folks use a combination of CA glue and wood glue. They glue up the joint like normal, but use CA glue to tack it in place while the woodglue cures. Spread wood glue, then a dab of CA glue on one side, accelerator spray on the other piece, then press together? You'd have to do it right the first time, because once that accelerator and CA glue meet, it's a done deal. Might be worth experimenting with.