No! You can continue to use your Special Card on your Apple Watch if you have removed the card from your iPhone.