I think you are probably right, he probably had no racist intention at all, but the term itself is racist - my mom was Japanese and I work in a predominantly Japanese-American community and it still brings a lot of pain to hear the term used. But trust me, I see a lot of reverse racism too so when I hear JAs using racist Japanese terms against white people, etc., I call them on it too!