Costa, who drilled home from 20 yards in the 89th minute, escaped punishment for a late tackle on Hammers goalkeeper Adrian having already been booked for dissent.
James Collins looked to have earned Bilic’s side a point by cancelling out Eden Hazard’s penalty.
Bilic told Sky Sports: “He had already been booked and it wasn’t reckless and I don’t think that it was deliberate but it was quite late, the ball was already gone and he could have, or should have, been booked again, which would mean a red card.
“But, to be fair, I’m not frustrated because of that, I’m frustrated because we conceded a very late goal and that’s what makes me sad.”
Antonio Conte believes there is more to come from Chelsea after his first competitive game in charge ended in victory.
“The team played a very good game, with a great intensity and a good spirit,” he said. “We know that we can improve and we must improve a lot with the work.
“Today I’m pleased for the attitude and behaviour of my players and the victory is, for me, deserved. Now we have to prepare for the next game. Today; good, we are happy but from tomorrow we have to think to the next game.”