Rav Chaim Kanievsky shlita answers because Tzizit is a “Chovas Gavra” (obligation on the individual’s person) that shouldn’t be neglected (though if you’re not wearing a garment one would’t be obligated), but the Torah specifically obligated the Kohanim to wear different clothing so they don’t have to wear Tzizit.
After witnessing the second plane hit the World Trade Center, I traveled to Bnei Brak to be with Rav Avrohom zt”l. Besides feeling the comfort of his presence, he quoted me a verse from Tanach that indicated physically built structures are not permanent in nature.
Shabbos usually unfolded in the following fashion. For mincha, he would walk me to daven with a chutz la’aretz minyan of young yeshiva bachurim. I would return and accompany him to the shiur he would give, which usually took place in a different shul each time. On the way to the shiur, he would pose a question to me and then I knew what the shiur was about. He would ask my opinion on the matter and ask for a sevara. I would usually take a seat near the front of the shul where he spoke. He was famously known to end every drasha by using the lamdus in the shiur to give mussar. On the way home, he usually asked me how I liked the drasha. On the table for Shabbos at night, he had garlic and after hamotzei gave challah to each person so as not to have it handed over. He often told stories at the table. After the meal, at some point he would do shnayim mikra v’echad targum and ask the rebbetzin to be his shomer as he held chumras when it came to electricity. Shabbos day I often joined him to daven at Sanz. If we ever walked to shul and a group of females were walking towards us, he would grab onto my jacket. As written in many places when people saw him walking in the street they would stop and bow their heads and he would give them a Shalom aleichem. At lunch we usually learned “Kitzur Hilchos Shabbos” which he said was a wonderful book. He also always told me to have extra kavana in Ein Kelokainu (as it relates to the avodah). By shalosh seudos, during the summer months, we would learn Pirkei Avot. Also it was well known that Rav Avrohom zt”l reviewed seven blot of gemara, rashi and tosofot every day. At shalos seudos I would often see him turning pages of a gemara so in my estimation that was when he reviewed those pages for that day. After Shabbos, the rebbetzin always made a great melava malka and she packed a large bag of cakes and other food items for me to take back to yeshiva. Though I tried to resist, Rav Avrohom always walked me to the cab station on motzei Shabbos and always inquired with the cab driver that he was going to the place I had to go. These were magical times that I got to spend with a Gadol. I can only reminisce about it now missing every nuance and emotion that accompanied these Shabbosim.