Broadcom Inc. is in talks to buy SAS Institute Inc., according to people familiar with the matter, in the latest move by the acquisitive technology giant to beef up in enterprise software.