It would not be unusual with today's high strength lines to use 8 mm or even 6 mm sheets with a light air sail such as a gennaker. If the winches are #40s or larger, the self tailing jaws very possibly might not grab the sheet. Contacting the winch manufacturer, if they are still in business, might provide a solution.
Another possibility is to purchase outer covering to splice over the working portion of the smaller diameter line to make it easier to handle as well as provide a better bite on the winch. (It is possible to buy outer cover only from some of the suppliers that cater to sailers in particular).