Can I fill in the COGS for all the product variants at once?

Our app has an option for that! It can be done on this page here. This will allow you to update all variants of a product at once.

That page can also be accessed on the Manual COGS page here with the "Edit Product by Name" button.