Jealous of boyfriends brother..?

It seems petty but I am struggling with the fact that my boyfriends brother is his #1 confidant, and I am not. He shares everything with him while he is not so open with me. We've been together for 4 years and I am tired of coming in 2nd to his brother. I try to talk about it and it turns around on me, saying if he tells me everything I'll just throw it back in his face later. This hurts me immensly and I really can't be completely happy knowing I'm always taking a backseat to his brother. What to do?