I was able to get our fastpasses for our November trip this weekend and now feel like I'm in the final countdown!

We have a two year old that will be going with us, so obviously we'll be utilizing the rider swap option for the rides that have height requirements. Here's my question. Our entire group of six adults have fastpasses for the same ride - say, Big Thunder. If we go up to the FastPass line, tell them we're doing the rider swap and three of us go onto the ride with they other three entertain the two year old, does the rider swap then override the fastpass, or do those three people that didn't use their fast pass have both their fast pass and the rider swap?

Thank you! It's our first time using rider swap so while we've been reading up on it we have no practical experience.