Accumulating evidence indicates a link between diabetes and cancer. Selective estrogen receptor modulators (SERMs) may increase diabetes risk via antiestrogen effects. This study investigated incident diabetes risk of SERM treatment and its effects on metastatic cancer and death prevention in breast cancer survivors. This retrospective cohort study included female patients with early-stage breast cancer, treated with or without SERMs, between 2008 and 2020 in a tertiary care hospital in Korea. Four propensity score-matched comparison pairs were designed: SERM use versus non-use, long-term use (≥1500 days) versus non-use, tamoxifen use versus non-use, and toremifene use versus non-use; then, logistic regression analysis was performed for risk analysis. SERMs in general were not associated with an elevated risk of diabetes; however, when used for ≥1500 days, SERMs—especially toremifene—substantially increased diabetes risk in breast cancer patients (OR 1.63, p = 0.048). Meanwhile, long-term SERM treatment was effective at preventing metastatic cancer (OR 0.20, p < 0.001) and death (OR 0.13, p < 0.001). SERM treatment, albeit generally safe and effective, may increase diabetes risk with its long-term use in women with breast cancer. Further studies are required to verify the association between toremifene treatment and incident diabetes.