What weight linen is best for wide-leg trousers? I keep buying the wrong thing.

Ive made three attempts at linen trousers and two of them were too floppy and one was too stiff. Im aiming for something that has structure but still drapes well. What GSM range am I looking for and does the weave structure matter?