My boyfriend tells me white lies, not whoppers but it's enought to make me uneasy. For example, he used to pop?

For example he used to pop into our local pub for a pint after work most nights, which I never have had a problem with. However when I met up with him once on one of these occasions, I walked into the pub and saw him chatting away to a barmaid very amicably, nothing wrong with that but when they both saw me they became very animated and there was definitely tension in the air! I chatted to him and her equally but there was still a weird atomosphere, I was not imagining it. I said nothing but later on at home, feeling very ashamed of myself i checked his mobile phone and sure enough, her number was in there. Again nothing really wrong with that, he can pick and choose his friends but he's never mentioned this girl to me and he usually would say if he had struck up a new friendship with someone, be it male or female. I then started to get really paranoid as a week earlier I had had to take our two kids on a long weekend break on my own as he said he could not really have the time off work to come and didn't really want to (he had recently lost his mother) and wanted some quiet time on his own and also save a bit of money. I was upset but wanted to respect his feelings so went and gave him the space I thought he wanted. However when I returned from this 3 day break he told me he had been out on the Friday and Saturday night with mates and got pretty wrecked on pub crawls - not really keeping his head down and having some quiet time! I started to convince myself he had stayed behind to see this barmaid. One night after a lot of wine I confronted him and he told me he only had her number as he was going to try to match make her with one of his friends and if i was worried he would delete her number which he then did. Everything was fine until last week when he went into the pub again and I am ashamed to say I checked his phone and sure enough her number was put back in! I confessed to him that i had snooped but why did he have her number? He said she took his phone off him and put it in. I was so angry I text her and told her politely that he wasn't interested, to which she replied that it was he who had asked for her number! Don't know what to think or do anymore. Does anyone have any opinions or advice? Would be very grateful.