Raftery attended Saint Cecilia High School in Kearny, New Jersey, where he starred as a 3-sport athlete in baseball, soccer and basketball. He became the all-time leading scorer in state history with 2,192 point earning all-state honors and led his team to a State Championship.
Raftery played at La Salle University during his freshman year he scored a freshman record 370 points, followed by a team leading 17.8 points per game in his sophomore year. Raftery was selected in the 14th round (82nd overall) of the 1963 NBA draft by the New York Knicks but never played in the NBA.
Raftery began his coaching career at Fairleigh Dickinson University where he was the head basketball coach from 1963 to 1968. Also, Raftery coached golf and served as associate athletic director. From 1970 to 1981, he was the head coach of Seton Hall University, where he posted a 154–141 record and led the Pirates to four ECAC post-season tournaments and two National Invitational Tournament appearances. In 1979, he was named Coach of the Year by the New Jersey Sports Writers Association.His 154 wins as a coach places is fourth on the all-time list at Seton Hall.
Raftery has been an analyst and play-by-play announcer for CBS Sports' college basketball coverage for over 33 years. On June 27, 2013, Raftery signed with FOX to call Big East basketball games on Fox Sports 1. His trademark quotes are "Onions!", "Send It In Big Fella!" and, "A Little Lingerie On The Deck!" He is especially remembered for "Send It In, Jerome!", his call immediately after Jerome Lane of the University of Pittsburgh shattered the backboard with a dunk in 1988.
Starting with the 2014–15 collegiate basketball season, CBS/Turner Sports partnered Raftery with Jim Nantz and Grant Hill to make up the primary announcing team for the remainder of the regular season, all the way through the NCAA men's basketball tournament and the Final four. On June 8, 2015, Raftery was inducted into the National Sportscasters and Sportswriters Association's Hall of Fame. He also won the Sports Emmy Award for Outstanding Sports Event Analyst in 2015 and 2016.
New Jersey Sports Writers Association Coach of the Year (1979)