The Same Watari As Always

I'm working on one of the WIPs that Will Not End, and I have a narrative problem. Most of the story is over Character A's shoulder. But I realized I have a scene with Character B where A cannot be there. Is it jarring to switch POV for just that one scene? Should I start doing it more often? What do y'all think?