It shall be unlawful for any person who caused, owns or controls said encroachment, to fail to remove, or commence to remove and diligently prosecute its removal prior to the expiration of the ten (10) days from the service of said notice. If the encroachment is such as to effectually obstruct and prevent the use of the sidewalk for pedestrians or the street for vehicles, the City building inspector must forthwith remove the same.