What made the vellum wrinkle? When you printed on it or when you were trying to put it on glass?
And are you using actual vellum?
Any vellum that I have used is fairly thick and not easily wrinkled....
When I print on vellum, I select the paper type accordingly in printer settings....depending on your printer, you might select "other transparency" or something like that for paper type.
After it prints, let it sit for awhile to dry...it takes several minutes for the ink to dry on vellum paper if you are using an inkjet printer.
After it's dry, spread a thin layer of Mod Podge on the back of the vellum and a thin layer on the glass....carefully apply the vellum, smoothing it out as you go
After that, you can paint on another layer or layers of Mod Podge for protection if that is what it calls for.....