Usability isn’t something you just can cook up in any one phase of design, but must be developed and refined throughout the entire process. If you want the best end product, you have to anticipate real user scenarios from the prototyping phase. Usability testing should be the last place to start thinking about usability.
Why worry about usability testing so early in the process when prototyping already has a big enough to-do list? Because unless your prototype is usable, all your testing will tell you is that people don’t like terrible products.