It is a wise idea to leave them on.
My mate would often claim that he could see my rear triangle flexing under the strain, especailly when it was wet...
(Sounds like I'm writing for a top shelf magazine)
...With the boosters on, it makes a more positve feel, with very little flex.
I use carbon boosters on my other bikes...no weight to carry! I don't use them on the front with my Pace RC36's because the forks have enough bracing to cope. And carbon boosters aren't cheap as a rule.