Hi there! It’s actually not thaaat weird — for one thing, they’re both introverted feelers. For another thing, INFP has Si (ISFJ’s dominant function) in their stack (tertiary; 3rd.)
The big thing is whether you’re dripping in and living and breathing words like a.) feelings, emotions, dreams, color, magic, special, unique, creativity, ideas, authenticity… or b.) certainty, clarity, stability, having thing settled, anxious
Both may relate to both sets, but ISFJ it will feel predominantly “logical” about the (b) set, and for INFP it will feel predominantly “emotional.”