Let me sit with you (or sit with me)

Something said often at the singles ward. It was nice at 1st. As I am often shy at 1st when I am in a new situations. My 1st day there someone freaked me out by saying this. I'm like ok. Other times people say sit with us/ me.