Can I cut a rug into a custom rug shape, and what will finishing cost?

Yes—pros will cut and bind/serge edges. Expect roughly \$2–\$12 per linear foot for binding/serging, depending on material and profile.